Title: DiViNeT: 3D Reconstruction from Disparate Views using Neural Template Regularization
Venue: NeurIPS 2023
Accepted: yes
Keywords: Multi-view Neural 3D Reconstruction

Sparse and Disparate Views

Neural Rendering

Volume Rendering
Abstract: We present a volume rendering-based neural surface reconstruction method that takes as few as three disparate RGB images as input. Our key idea is to regularize the reconstruction, which is severely ill-posed and leaving significant gaps between the sparse views, by learning a set of neural templates that act as surface priors. Our method, coined DiViNet, operates in two stages. The first stage learns the templates, in the form of 3D Gaussian functions, across different scenes, without 3D supervision. In the reconstruction stage, our predicted templates serve as anchors to help “stitch” the surfaces over sparse regions. We demonstrate that our approach is not only able to complete the surface geometry but also reconstructs surface details to a reasonable extent from few disparate input views. On the DTU and BlendedMVS datasets, our approach achieves the best reconstruction quality among existing methods in the presence of such sparse views and performs on par, if not better, with competing methods when dense views are employed as inputs.

DiViNeT: 3D Reconstruction from Disparate Views via Neural Template Regularization

Abstract

We present a volume rendering-based neural surface reconstruction method that takes as few as three disparate RGB images as input. Our key idea is to regularize the reconstruction, which is severely ill-posed and leaving significant gaps between the sparse views, by learning a set of neural templates to act as surface priors. Our method, coined DiViNet, operates in two stages. It first learns the templates, in the form of 3D Gaussian functions, across different scenes, without 3D supervision. In the reconstruction stage, our predicted templates serve as anchors to help 'stitch' the surfaces over sparse regions. We demonstrate that our approach is not only able to complete the surface geometry but also reconstructs surface details to a reasonable extent from a few disparate input views. On the DTU and BlendedMVS datasets, our approach achieves the best reconstruction quality among existing methods in the presence of such sparse views and performs on par, if not better, with competing methods when dense views are employed as inputs.

1 Introduction

3D reconstruction from multi-view images is a fundamental task in computer vision. Recently, with the rapid advances in neural fields [51, 31] and differentiable rendering [18], many methods have been developed for neural 3D reconstruction [34, 55, 23, 28, 48, 22, 54, 36, 62, 50]. Among them, volume rendering-based methods have achieved impressive results. Exemplified by neural radiance fields (NeRF) [31], these approaches typically employ compact MLPs to encode scene geometry and appearance, with network training subjected to volume rendering losses in RGB space.

The main drawback of most of these methods is the requirement of dense views with considerable image overlap, which may be impractical in real-world settings. When the input views are sparse, these methods often fail to reconstruct accurate geometries due to radiance ambiguity [61, 49]. Despite employing smoothness priors during SDF/occupancy prediction as an inductive bias [34, 55, 48, 54], the limited overlap between regions in the input images causes little-to-no correlation to the actual 3D surface, resulting in holes and distortions, among other artifacts, in the reconstruction.

Several recent attempts have been made on neural reconstruction from sparse views. SparseNeuS [25] learns generalizable priors across scenes by constructing geometry encoding volumes at two resolutions in a coarse-to-fine and cascading manner. While the method can handle as few as 2 or 3 input images, the corresponding views must be sufficiently close to ensure significant image overlap and quality reconstruction. MonoSDF [58] relaxes on the image overlap requirement and relies on geometric monocular priors such as depth and normal cues to boost sparse-view reconstruction. However, such cues are not always easy to obtain, e.g., MonoSDF relied on a pre-trained depth estimation model requiring 3D ground-truth (GT) data. Moreover, depth prediction networks often rely on cost volumes [52, 14] which are memory expensive and thus restricted to predicting low-resolution depth maps. Because of this, the results obtained typically lack fine geometric details [58].

Input images

Neus

Ours

Scan 24

Scan 55

Scan 40

In this paper, we target neural surface reconstruction from sparse and disparate views. That is, not only are the input RGB images few in number, they also do no share significant overlap; see Figure 1. Our key idea is to regularize the reconstruction, which is severely ill-posed and leaves significant gaps to be filled in between the few disparate views, by learning a set of neural templates that act as surface priors. Specifically, in the first stage, we learn the neural templates, in the form of 3D Gaussian functions, across different scenes. Our network is a feedforward CNN encoder-decoder, trained with RGB reconstruction and auxiliary losses against the input image collection. In the second stage, surface reconstruction via volume rendering, our predicted templates serve as anchors to help 'stitch' the surfaces over sparse regions without negatively impacting the ability of the signed distance function (SDF) prediction network from recovering accurate geometries.

Our two-stage learning framework (see Figure 2) is coined DiViNet for neural 3D reconstruction from Disparate Views via NEural Templates regularization. We conduct extensive experiments on two real-world object-scenes datasets, viz., DTU [16] and BlendedMVS [53], to show the efficiency of our method over existing approaches on the surface reconstruction task, for both sparse and dense view input settings. Through ablation studies, we validate the design choices of our network in the context of different optimization constraints employed during training.

2 Related Work

View-based 3D Reconstruction. Reconstructing 3D surfaces from image inputs is a non-trivial task mainly due to the difficulty of establishing reliable correspondences between different regions in the input images and estimating their correlation to the 3D space. This is especially pronounced when the number of views is limited and disparate . Multi-view stereo (MVS) techniques, both classical [1, 3, 6, 9, 44, 12, 4, 41, 45], and learning-based [27, 47, 59, 39, 15, 52, 57], while using an explicit 3D representation, address the problem of multi-view 3D reconstruction either by estimating depth via feature matching across different input views or by voxelizing 3D space for shape reconstruction.

Arecent exploration of neural implicit functions for 3D shape/scene representation [29, 37, 30, 24, 46, 40, 17] has led to an explosion of 3D reconstruction from multi-view images, where two prominent directions exist - one based on surface rendering techniques [34, 55, 23], which require accurate 2D masks at the input, and the other based on volume rendering techniques [28, 48, 22, 54, 36, 62, 50],

. . .

𝑠𝑖

𝑐𝑖

MLP

MLP

Volume rendering

Implicit Rep.

𝑧

(

r

)

RGB,

𝑐

(

r

)

w(

r

)

peak along

depth

𝑧

𝑧 𝑡

(

r

)

ℒ𝑑𝑒𝑝𝑡ℎ

ℒ𝑟𝑔𝑏

GT Image, c(

r

)

Camera ray,

r

SDF constraint,

Template

Parameters,

Φ

pi

ct

s

t

Stage 1: Neural Template Learning

Stage 2: Surface Reconstruc

�

on using Volume Rendering

f

i

which require no such object masks. The latter techniques combine the representational power of neural implicit with the recently proposed NeRF models [31], achieving better surface reconstructions over the former techniques. A major drawback of these works is that they all require dense multiview images as input. Two recent techniques, Sparse NeuS [25] and Mono-SDF [58] learn 3D reconstruction from just three input images. In the case of [25], the three input images have a significant amount of overlap, and can not be employed when the views are disparate. [58], on the other hand, leverages additional cues such as ground truth depths and normals, allowing reconstruction from disparate views. Our work differs from these in the sense that we do not make use of any explicit cues in the form of ground truth depths and normals, while still being able to faithfully reconstruct 3D surfaces from just three disparate RGB images as input.

Regularizing Neural Surface Reconstruction. Our ability to reconstruct 3D surfaces from sparse, disparate view images is made possible through template-based regularization that provides helpful surface priors during the volume rendering stage. Regularization has been explored in the context of novel-view synthesis [33, 7], where patch-based regularization for both the scene geometry and color is performed [33] or additional cues such as depth supervision are incorporated [7] while handling sparse inputs in both cases. In the context of surface reconstruction, MonoSDF [58] uses depth and normal supervision as a part of unsaid regularization to obtain surface geometry from sparse input images. Other works [5, 60, 8] study the effect of regularization on the volumetric rendering-based surface reconstruction framework. Their goal is to improve surface reconstruction by imposing multi-view photometric and COLMAP [43] constraints on the Signed Distance Field (SDF) prediction during training. Such methods show significant improvements over vanilla approaches for surface reconstruction. However, under sparse scenarios, because of significant view differences, photometric and COLMAP constraints are difficult to satisfy, resulting in poor reconstruction quality. In contrast to these methods, inspired by [11, 10, 35], we propose to learn surface priors in the form of templates across a data distribution and use these templates to guide the reconstruction process.

3 Method

Our method achieves 3D reconstruction of solid objects from a small set of RGB images { I i } N -1 i =0 with very less visual overlap, where N can be as less as 3 images and I i ∈ [0 , 1] H × W × 3 . We assume that camera extrinsics and intrinsics are known for all the images. As shown in Figure 2 our approach is realized in two stages: (1) Learning a network for predicting shape templates (see Section 3.1), and (2) Leveraging the predicted templates, learning a volumetric surface reconstruction network with depth and SDF constraints (see Section 3.2).

3.1 Stage 1: Learning Surface Priors with Neural Templates

Template Representation. We represent the shape with a bunch of N t local implicit functions called templates, where the influence of i th template is represented as g i ( p, φ i ) and is given by,

where p ∈ R 3 is the 3D location of a point in the volume and φ i is the template parameter. Specifically, each template φ i is represented as a scaled, anisotropic 3D Gaussian, φ i ∈ R 7 , which consists of a scaling factor s i , a center point c i ∈ R 3 , and per-axis radii r i ∈ R 3 . Using this local implicit function, we can find the probability of a point near the surface by summing up the influences of all N t templates, i.e. G ( p, Φ) = ∑ N t i =1 g i ( p, φ i ) , where Φ is the set of all the template parameters.

Network Architecture. Figure 3 shows the template training stage. Given a fixed set of N t templates ( Φ ) and input images set { I i } , we aim to optimize a network T θ for the task of template prediction which comprises of an encoder ( E ) and decoders ( D geo , D vox ). The encoder is a feedforward Convolutional Neural Network (CNN), which we first use to extract 2D image features from individual input views. We then obtain per template latent code z φ i ∈ R 64 using a bi-linear interpolation step as shown in Figure 4. For this step, depending on the number of templates,

Reconstructed

Image,

𝐼

(

𝒓

)

GT Image,

𝐼

(

𝒓

Loss

Blending

𝑐 𝑖

𝑤𝑖

G(.)

. . .

Template

Parameters,

Φ

(Nt

)

…

Nt

MLP

𝑓 𝑖

Template Local

Feature Volumes

Feature

Maps,

𝑧𝜑𝑖

𝜀

𝐷𝑔𝑒𝑜

M

M

M

)

we create a uniform grid of points, which we then use for the interpolation of features obtained from convolutional layers. Each latent code of the template, z φ i is then decoded through two decoders D geo and D vox . D geo is the geometry decoder with simple MLP layers predicting the template parameters φ i . That is, D geo : R 64 → R 7 . D vox , on the other hand, is a feature decoder comprised of transposed convolutional layers. The objective of D vox is to map the template feature representation, z φ i , to a local volumetric feature grid for each individual template.

Decoding all the template latent codes z φ i at once through D vox results in a dense voxel grid V ∈ R ( C∗M ) × ( M∗ √ N t ) × ( M∗ √ N t ) , which we then rearrange to N t local volumes of size R C×M 3 , where C is the feature dimension of each voxel and M is the resolution of the feature grid. In our experiments, we set N t = 576 , C = 8 and M = 16 . For more details refer to supplementary.

Network Training. Due to the lack of ground truth 3D information of objects in the training set, we use RGB reconstruction loss along with auxiliary losses to optimize the template parameters. Given the predicted feature volume from the template network, we can query a continuous field at any 3D query point p i by passing its tri-linearly interpolated feature f vox ( p i ) , along with the location p i to a small MLP, consisting of 2 layers and 256 hidden dimensions. The output of the MLP is the color prediction at that particular query point, p i in the 3D space. In addition to color, we compute each point's blending weights w i in 3D space using our predicted templates. This blending weight is computed by summing up the local influences of

𝑁𝑡

𝑁𝑡

H/16

W/16

each template on the query point. We then integrate the predicted colors and weights to obtain the predicted image ˆ I . For K query points sampled along a ray, this can be written as:

Here, p i is a query point along a ray, ϵ is a small perturbation to prevent w i from diverging to infinity when G ( p i , Φ) takes small values, whereas G ( p i , Φ) = ∑ i ∈ N t g i ( p i , ϕ i ) . We then optimize our template prediction network using RGB reconstruction loss. To make sure that the predicted templates are plausible and near the surface, we propose auxiliary loss functions. The total loss by combining all the losses is given as:

Here, L rgb is the mean square error loss between the predicted pixel ˆ I ( r ) and ground truth pixels I ( r ) which is defined as:

L chamfer is the chamfer distance loss, which minimizes the distance between the template centers { C t : c 1 , c 2 , · · · , c N t ∈ R 3 } , with the corresponding sparse reconstructed point cloud { S c : s 1 , s 2 , · · · , s N ∈ R 3 } obtained using triangulation, from COLMAP[43]. This loss makes sure that the predicted templates from the network are near the surface. This is given by:

L cov is the coverage loss, which is defined over the entire sparse points, S c . This ensures that the predicted templates cover the entire object. It is defined as,

where σ ( . ) is the sigmoid function. In addition to this, we also use radius L radius and variance loss L var , which penalizes large and skewed template radii. This is given by:

Here, R t is the set of all template radii. r i is the individual template radii, and ¯ r t is the mean radius across all the templates, N t .

3.2 Stage 2: Surface Reconstruction using Volume Rendering

Geometry Representation and Volume Rendering. Following the recent works of surface reconstruction with volume rendering [48, 54, 36], we represent the volume contained by the object to be reconstructed using a signed distance field ˜ s i , which is parameterized using an MLP f θ s , with learnable parameters θ s i.e ˜ s i = f θ s ( γ ( x i )) . Here γ ( . ) is the positional encoding of point in 3D space [31]. Given the signed distance field, the surface S of the object is represented as its zero-level set i.e.

To train the SDF network in volume rendering framework, we follow [48], and use the below function, to convert the SDF predictions, f θ s ( p i ) to α i at each point, p i in the 3D space.

Here, Φ s ( x ) = (1 + e -sx ) -1 is the sigmoid function and s is learned during training.

Along with the signed distance field, ˜ s i , we also predict color, ˜ c i for each sampled point in 3D space using a color MLP, f θ c . Following [55] the input to the color MLP is a 3D point p i , a viewing

direction ˆ d , analytical gradient ˆ n i of our SDF, and a feature vector z i computed for each point p i by f θ s . Hence, the color MLP is the following function, ˜ c i = f θ c ( p i , ˆ d, ˆ n i , z i ) . We optimize these networks using volume rendering, where the rendered color of each pixel is integrated over M discretely sampled points { p i = o + t i ˆ d | i = 1 , · · · , M, t i < t i +1 } along a ray traced from camera center o and in direction ˆ d . Following [31] the accumulated transmittance T i for a sample point p i is defined as T i = ∏ i -1 j =1 (1 -α j ) , where α j is the opacity value defined in Eq. 9. Following this, we can compute the pixel color and depth as follows:

Optimization. The overall loss function we use for optimizing for the surface is:

L color , is the L1 loss between the predicted and ground truth colors which is given as:

Following [13] we use Eikonal loss to regularize the SDF in 3D space, which is given as:

In addition to these, we propose the following regularization terms in order to aid the surface reconstruction process, especially in sparse reconstruction scenarios.

Depth Loss : We sample depth cues z t ( r ) along each ray using the templates predicted by our template network. These depth cues are obtained by finding the depth locations where the template weight function w ( x ) at a point x , mentioned in 2 attains a maximum value among all the points sampled along a ray. We then minimize the L1 loss L depth between the predicted depth, ˆ z ( r ) and the computed depth cues z t ( r ) as:

Here, R valid denotes the set of rays that intersect the templates in the 3D space. We find these rays using ray sphere intersection.

SDF Loss : In addition to depth, we also constraint our SDF to pass through the template centers which we assume are near the surface. This is done by minimizing the L1 loss between the SDF at the template centers and the zero-level set. Here, N t is the number of template parameters.

Implementation Details. Our implementation is based on the PyTorch framework [38]. For both stages, we use the Adam optimizer [20] to train our networks and set a learning rate of 5 e -4 . For stage-1, we set λ cd , λ c , λ r and λ v to 1 . 0 , 0 . 1 , 0 . 1 , and 1 . 0 , respectively. And for stage-2, we set λ 1 , λ 2 and λ 3 to 1 . 0 , 0 . 8 and 0 . 8 , respectively. As the templates are not perfect, we follow [58] and use an exponentially decaying loss weight for both SDF and depth regularization for the first 25 k iterations of optimization. During training in both stages, we sample a batch of 512 rays in each iteration. During both stages, we assume that the object is within the unit sphere. Our SDF network f θ s , is an 8 layer MLP with 256 hidden units with a skip connection in the middle. The weights of the SDF network are initialized by geometric initialization [2]. The color MLP is a 4 layer MLP with 256 hidden units. The 3D position is encoded with 6 frequencies, whereas the viewing direction is

Ours

Ours

encoded with 4 frequencies. We train the network for 300 k iterations which takes roughly 8 hours on NVIDIA RTX 3090 Ti GPU. After training, we extract the mesh using marching cubes [26] at a 512 3 resolution. See the Supplementary material for more details.

4 Results and Evaluation

Datasets. We use two commonly employed real-world object-scenes datasets, viz., DTU [16] and BlendedMVS dataset [53]. DTU dataset contains multi-view images (49-64) of different objects captured with a fixed camera and lighting parameters. For training sparse view reconstruction models, we use the same image IDs as in [33, 58, 56]. In addition to this, in order to show the generalization ability of our template prediction network we experiment with our method on the MobileBrick dataset [21], which consists of high-quality 3D models along with precise ground-truth annotations.

For the template learning stage, our training split consists of objects that do not overlap with the 15 test scans of the DTU dataset. We provide the list of scans used for training in the supplementary. We evaluate our method on standard 15 scans used for evaluation by [34, 55, 54, 48, 36]. The resolution of each image in the dataset is 1200 × 1600 . Unlike MonoSDF [58] which resizes the images to 384 × 384 resolution for training, we input the actual image resolution to our pipeline. The other dataset, BlendedMVS, consists of 113 scenes captured from multiple views, where each image is of resolution 576 × 768 . Unlike DTU, BlendedMVS dataset has not been employed before in the sparse view setting, and as such, specific view IDs to train and test are not available. We therefore manually select 3 views that have little overlap.

Baselines. For sparse views, we compare against the classical MVS method, COLMAP [42], and recent volume rendering-based methods such as NeuS [48], VolSDF [54] and MonoSDF [58]. We exclude SparseNeuS [25] evaluation in disparate settings because of its inability to reconstruct 3D geometry when the input images have significantly less overlap. For dense views, we additionally compare against the recent grid-based method, Voxurf [50]. A mesh from COLMAP point cloud output is obtained using Screened Poisson Surface Reconstruction [19].

Evaluation Metrics. Following standard evaluation protocol, we use Chamfer Distance (CD) between the ground truth point cloud and predicted meshes to quantify the reconstruction quality. On the DTU dataset, following prior works [54, 48, 58], we report CD scores. On the BlendedMVS dataset, we simply show qualitative results as is commonly done in the literature. In addition to this, for evaluation on the MobileBrick dataset, we use F1 score computed using the ground-truth mesh.

4.1 Results on DTU

Figure 1 and Table 1 show qualitative and quantitative results, respectively, in the presence of sparse view inputs (three, to be specific) on the DTU dataset. We observe from Table 1 that our method outperforms all other competing methods, including MonoSDF [58], on nine of the fifteen scenes in the test set, and achieves the lowest mean CD score (the next best adds an additional 0.09 CD score).

Scan 110

Scan 112

NeuS

Voxurf

Ours

On the other six scans, our method is the second-best performing method behind MonoSDF. This is because our predicted templates are not always perfect for all objects, meaning, regions on the surface with no predicted templates potentially suffer from incomplete reconstructions. Other methods such as VolSDF [54] and NeuS [48] fail to even beat the classical MVS method, COLMAP [43], indicating that these methods either require dense view inputs[54] or require a significant amount of overlap on the input views [25] to faithfully reconstruct the surface. This emphasizes the effectiveness of our method on the disparate views case. Further, Figure 5 compares surface details on the reconstructed geometry against, MonoSDF [58], where it can be observed that our method produces sharper details, attributed mainly to the geometry guidance from learned template priors.

On dense view inputs (4964), Figure 6 and Table 3 present qualitative and quantitative results, respectively, over the aforementioned techniques. We additionally compare against a recent method, Voxurf [50]. In this setting, all neural surface reconstruction

methods outperform COLMAP, indicating that the presence of dense view inputs is necessary to tap into the learning ability of neural rendering techniques for improved geometry reconstruction. We observe that our method outperforms the rest on five scans, which is higher than any other method (Voxurf-three scans, NeuS-four, MonoSDF-two, and VolSDF-one). Overall, our method achieves the second-best mean CD score, closely trailing behind Voxurf (a difference of 0.03 CD score). This is expected as feature-grid-based approaches tend to reconstruct sharper details compared to MLP [58]. But otherwise, we beat all other existing works owing to our neural template regularization scheme. More qualitative results on dense view inputs are provided in the Supplementary material.

4.2 Results on BlendedMVS

We also run our pipeline on a more challenging, also real-world, BlendedMVS dataset [53]. We use the sparse input setting and show comparisons against the same set of prior works, i.e., NeuS[48] and MonoSDF [58]. Figure 7 shows qualitative results on 4 object-scenes with complex geometries. Clearly, our method obtains the best surface reconstruction. NeuS is not able to reconstruct the

Chamfer Distance (CD) (

↓

)

Scan ID

→

24

37

40

55

63

65

69

83

97

105

106

110

114

118

122

Mean

COLMAP[41]

0.81

2.05

0.73

1.22

1.79

1.58

1.02

3.05

1.4

2.05

1.0

1.32

0.49

0.78

1.17

1.36

NeuS[48]

1.0

1.37

0.93

0.43

1.1

0.65

0.57

1.48

1.09

0.83

0.52

1.2

0.35

0.49

0.54

0.84

1.14

1.26

0.81

0.49

1.25

0.7

0.72

1.29

1.18

0.7

0.66

1.08

0.42

0.61

0.55

0.86

0.83

1.61

0.65

0.47

0.92

0.87

0.87

1.3

1.25

0.68

0.65

0.96

0.41

0.62

0.58

0.84

Voxurf [50]

0.91

0.73

0.45

0.34

0.99

0.66

0.83

1.36

1.31

0.78

0.53

1.12

0.37

0.53

0.51

0.76

Ours

0.78

1.29

0.68

0.42

0.96

0.70

0.68

1.43

1.30

0.827

0.62

0.94

0.34

0.49

0.49

0.79

Sculpture

Clock

Buddha

Man

Input images

Neus

Ours

geometry faithfully for any of the objects due to disparate view inputs. MonoSDF, on the other hand, despite using ground truth depth and normal cues, fails to reconstruct accurate geometry, especially for Clock and Sculpture objects. In contrast to this, our method can accurately reconstruct the geometric details owing to template guidance coming from Stage 1.

4.3 Generalization on MobileBrick Dataset

We test the generalization ability of our template prediction network (TPN) on a new dataset, MobileBrick [21], by comparing the reconstruction results when the neural templates were learned by a pre-trained TPN (on DTU) vs. when they were trained on MobileBrick. Note that overall, the models from these two datasets are quite different in terms of geometry and structure. The results in Table 2 and qualitative results in Figure 8a (a) show that the reconstruction qualities under the two scenarios are comparable, attesting to the generalizability of our TPN. As mentioned before, the metric used in this evaluation is F1 score as reported by MobileBrick dataset.

4.4 Ablation Studies

Ours with

pre-trained TN

Ours with

trained TN

Bridge

Camera

Colosseum

Castle

(a)

5

10

15

20

25

30

35

40

45

Number of Input Views

1.0

1.5

2.0

2.5

3.0

3.5

4.0

4.5

5.0

Chamfer Distance (CD)

Ours

NeuS

60%

0

100

200

300

400

500

600

700

800

900

1000

# of gaussians

1.450

1.475

1.500

1.525

1.550

1.575

1.600

1.625

Chamfer Distance (CD)

(b)

Referencelmage

OnlySDFConstraint

OnlyDepthConstraint

SDF+Depth Constraint

In Figure 9, we demonstrate the need for employing different loss functions as described in Section 3.2 and depicted on the right side of Figure 2 and Table 4. We observe how the reconstructed geometry changes when L sdf and L depth are separately employed. Evidently, only SDF constraints cannot reconstruct surfaces due to the coarse supervision from template centers. On the other hand, depth regularization significantly improves the reconstruction but holes

still exist. Our full pipeline includes both the above constraints, resulting in improved reconstruction, as can be seen in the last column of Figure 9 and Table 4. We also study reconstruction quality as a function of the number of input views. Figure 8b depicts the behavior of two such methods i.e. NeuS and Ours, up to fifty views. Our method achieves the same Chamfer Distance (CD) score for twelve views as NeuS achieves for around thirty. This demonstrates the efficacy of our method to 'catch' surface geometry even when sufficient view inputs are not present. In addition to this, we measure the effect on reconstruction quality by varying the number of Gaussians. As it can be seen in Figure 8b the surface reconstruction quality significantly improves as we increase the number of gaussians. The reconstruction quality saturates after 576 gaussians and hence we use 576 gaussians throughout our experiments.

5 Discussions, Limitations and Future Work

We present DiViNet for sparse multi-view 3D reconstruction. Our core idea is a simple twostage framework with neural Gaussian templates learned across different scenes serving as surface priors to regularize the volume-rendering-based reconstruction. Experiments demonstrate quality reconstructions by DiViNet in both dense and sparse+disparate view settings, with clearly superior performance over state-of-the-art in the latter.

One current limitation of our method is its reconstruction speed. While comparable to NeuS [48], VolSDF [54], and MonoSDF [58], it is still slow due to the two-stage approach, especially when generalizing to a large number of real-world scenes. In addition to addressing these issues, e.g., dynamic templates, we would also like to replace the MLPs in DiViNet with more efficient gridbased techniques, e.g., instant NGP [32], and explore more general primitives such as convexes and super-quadrics in place of Gaussians.

References

M. Agrawal and L. S. Davis. A probabilistic framework for surface reconstruction from multiple images. In Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001 , volume 2, pages II-II. IEEE, 2001.

M. Atzmon and Y. Lipman. Sal: Sign agnostic learning of shapes from raw data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2565-2574, 2020.

M. Bleyer, C. Rhemann, and C. Rother. Patchmatch stereo-stereo matching with slanted support windows. In Bmvc , volume 11, pages 1-11, 2011.

A. Broadhurst, T. W. Drummond, and R. Cipolla. A probabilistic framework for space carving. In Proceedings eighth IEEE international conference on computer vision. ICCV 2001 , volume 1, pages 388-393. IEEE, 2001.

F. Darmon, B. Bascle, J.-C. Devaux, P. Monasse, and M. Aubry. Improving neural implicit surfaces geometry with patch warping. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6260-6269, 2022.

J. S. De Bonet and P. Viola. Poxels: Probabilistic voxelized volume reconstruction. In Proceedings of International Conference on Computer Vision (ICCV) , volume 2, page 3, 1999.

K. Deng, A. Liu, J.-Y. Zhu, and D. Ramanan. Depth-supervised NeRF: Fewer views and faster training for free. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12882-12891, 2022.

Q. Fu, Q. Xu, Y. S. Ong, and W. Tao. Geo-neus: Geometry-consistent neural implicit surfaces learning for multi-view reconstruction. Advances in Neural Information Processing Systems , 35:3403-3416, 2022.

Y. Furukawa, C. Hernández, et al. Multi-view stereo: A tutorial. Foundations and Trends® in Computer Graphics and Vision , 9(1-2):1-148, 2015.

K. Genova, F. Cole, A. Sud, A. Sarna, and T. Funkhouser. Local deep implicit functions for 3d shape. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4857-4866, 2020.

K. Genova, F. Cole, D. Vlasic, A. Sarna, W. T. Freeman, and T. Funkhouser. Learning shape templates with structured implicit functions. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 7154-7164, 2019.

M. Goesele, B. Curless, and S. M. Seitz. Multi-view stereo revisited. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06) , volume 2, pages 2402-2409. IEEE, 2006.

A. Gropp, L. Yariv, N. Haim, M. Atzmon, and Y. Lipman. Implicit geometric regularization for learning shapes. arXiv preprint arXiv:2002.10099 , 2020.

X. Gu, Z. Fan, S. Zhu, Z. Dai, F. Tan, and P. Tan. Cascade cost volume for high-resolution multiview stereo and stereo matching. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 2495-2504, 2020.

P.-H. Huang, K. Matzen, J. Kopf, N. Ahuja, and J.-B. Huang. Deepmvs: Learning multi-view stereopsis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 2821-2830, 2018.

R. Jensen, A. Dahl, G. Vogiatzis, E. Tola, and H. Aanæs. Large scale multi-view stereopsis evaluation. In 2014 IEEE Conference on Computer Vision and Pattern Recognition , pages 406-413. IEEE, 2014.

Y. Jiang, D. Ji, Z. Han, and M. Zwicker. Sdfdiff: Differentiable rendering of signed distance fields for 3d shape optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1251-1261, 2020.

H. Kato, D. Beker, M. Morariu, T. Ando, T. Matsuoka, W. Kehl, and A. Gaidon. Differentiable rendering: A survey. CoRR , abs/2006.12057, 2020.

M. Kazhdan and H. Hoppe. Screened poisson surface reconstruction. ACM Transactions on Graphics (ToG) , 32(3):1-13, 2013.

D. P. Kingma, J. A. Ba, and J. Adam. A method for stochastic optimization. arxiv 2014. arXiv preprint arXiv:1412.6980 , 106, 2020.

K. Li, J.-W. Bian, R. Castle, P. H. Torr, and V . A. Prisacariu. Mobilebrick: Building lego for 3d reconstruction on mobile devices. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4892-4901, 2023.

L. Liu, J. Gu, K. Zaw Lin, T.-S. Chua, and C. Theobalt. Neural sparse voxel fields. Advances in Neural Information Processing Systems , 33:15651-15663, 2020.

S. Liu, Y. Zhang, S. Peng, B. Shi, M. Pollefeys, and Z. Cui. Dist: Rendering deep implicit signed distance function with differentiable sphere tracing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2019-2028, 2020.

S. Lombardi, T. Simon, J. Saragih, G. Schwartz, A. Lehrmann, and Y. Sheikh. Neural volumes: Learning dynamic renderable volumes from images. arXiv preprint arXiv:1906.07751 , 2019.

X. Long, C. Lin, P. Wang, T. Komura, and W. Wang. SparseNeuS: Fast generalizable neural surface reconstruction from sparse views. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXII , pages 210-227. Springer, 2022.

W. E. Lorensen and H. E. Cline. Marching cubes: A high resolution 3d surface construction algorithm. ACM siggraph computer graphics , 21(4):163-169, 1987.

W. Luo, A. G. Schwing, and R. Urtasun. Efficient deep learning for stereo matching. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 56955703, 2016.

N. Max. Optical models for direct volume rendering. IEEE Transactions on Visualization and Computer Graphics , 1(2):99-108, 1995.

L. Mescheder, M. Oechsle, M. Niemeyer, S. Nowozin, and A. Geiger. Occupancy networks: Learning 3d reconstruction in function space. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 4460-4470, 2019.

M. Michalkiewicz, J. K. Pontes, D. Jack, M. Baktashmotlagh, and A. Eriksson. Deep level sets: Implicit surface representations for 3d shape inference. arXiv preprint arXiv:1901.06802 , 2019.

B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng. NeRF: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM , 65(1):99-106, 2021.

T. Müller, A. Evans, C. Schied, and A. Keller. Instant neural graphics primitives with a multiresolution hash encoding. ACM Transactions on Graphics (ToG) , 41(4):1-15, 2022.

M. Niemeyer, J. T. Barron, B. Mildenhall, M. S. Sajjadi, A. Geiger, and N. Radwan. RegNeRF: Regularizing neural radiance fields for view synthesis from sparse inputs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5480-5490, 2022.

M. Niemeyer, L. Mescheder, M. Oechsle, and A. Geiger. Differentiable volumetric rendering: Learning implicit 3d representations without 3d supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3504-3515, 2020.

C. Niu, M. Li, K. Xu, and H. Zhang. RIM-Net: Recursive implicit fields for unsupervised learning of hierarchical shape structures. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11779-11788, 2022.

M. Oechsle, S. Peng, and A. Geiger. Unisurf: Unifying neural implicit surfaces and radiance fields for multi-view reconstruction. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 5589-5599, 2021.

J. J. Park, P. Florence, J. Straub, R. Newcombe, and S. Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 165-174, 2019.

A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer. Automatic differentiation in pytorch. 2017.

G. Riegler, A. O. Ulusoy, H. Bischof, and A. Geiger. Octnetfusion: Learning depth fusion from data. In 2017 International Conference on 3D Vision (3DV) , pages 57-66. IEEE, 2017.

S. Saito, Z. Huang, R. Natsume, S. Morishima, A. Kanazawa, and H. Li. Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization. In Proceedings of the IEEE/CVF international conference on computer vision , pages 2304-2314, 2019.

J. L. Schönberger and J.-M. Frahm. Structure-from-motion revisited. In Conference on Computer Vision and Pattern Recognition (CVPR) , 2016.

J. L. Schönberger, E. Zheng, J.-M. Frahm, and M. Pollefeys. Pixelwise view selection for unstructured multi-view stereo. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part III 14 , pages 501-518. Springer, 2016.

J. L. Schönberger, E. Zheng, M. Pollefeys, and J.-M. Frahm. Pixelwise view selection for unstructured multi-view stereo. In European Conference on Computer Vision (ECCV) , 2016.

S. M. Seitz, B. Curless, J. Diebel, D. Scharstein, and R. Szeliski. A comparison and evaluation of multi-view stereo reconstruction algorithms. In 2006 IEEE computer society conference on computer vision and pattern recognition (CVPR'06) , volume 1, pages 519-528. IEEE, 2006.

S. M. Seitz and C. R. Dyer. Photorealistic scene reconstruction by voxel coloring. International journal of computer vision , 35:151-173, 1999.

V. Sitzmann, M. Zollhöfer, and G. Wetzstein. Scene representation networks: Continuous 3d-structure-aware neural scene representations. Advances in Neural Information Processing Systems , 32, 2019.

B. Ummenhofer, H. Zhou, J. Uhrig, N. Mayer, E. Ilg, A. Dosovitskiy, and T. Brox. Demon: Depth and motion network for learning monocular stereo. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 5038-5047, 2017.

P. Wang, L. Liu, Y. Liu, C. Theobalt, T. Komura, and W. Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. arXiv preprint arXiv:2106.10689 , 2021.

Y. Wei, S. Liu, J. Zhou, and J. Lu. Depth-guided optimization of neural radiance fields for indoor multi-view stereo. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2023.

T. Wu, J. Wang, X. Pan, X. Xu, C. Theobalt, Z. Liu, and D. Lin. Voxurf: Voxel-based efficient and accurate neural surface reconstruction. arXiv preprint arXiv:2208.12697 , 2022.

Y. Xie, T. Takikawa, S. Saito, O. Litany, S. Yan, N. Khan, F. Tombari, J. Tompkin, V. Sitzmann, and S. Sridhar. Neural fields in visual computing and beyond. Computer Graphics Forum , 2022.

Y. Yao, Z. Luo, S. Li, T. Fang, and L. Quan. Mvsnet: Depth inference for unstructured multiview stereo. In Proceedings of the European conference on computer vision (ECCV) , pages 767-783, 2018.

Y. Yao, Z. Luo, S. Li, J. Zhang, Y. Ren, L. Zhou, T. Fang, and L. Quan. Blendedmvs: A large-scale dataset for generalized multi-view stereo networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1790-1799, 2020.

L. Yariv, J. Gu, Y. Kasten, and Y. Lipman. Volume rendering of neural implicit surfaces. Advances in Neural Information Processing Systems , 34:4805-4815, 2021.

L. Yariv, Y. Kasten, D. Moran, M. Galun, M. Atzmon, B. Ronen, and Y. Lipman. Multiview neural surface reconstruction by disentangling geometry and appearance. Advances in Neural Information Processing Systems , 33:2492-2502, 2020.

A. Yu, V. Ye, M. Tancik, and A. Kanazawa. PixelNeRF: Neural radiance fields from one or few images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4578-4587, 2021.

Z. Yu and S. Gao. Fast-mvsnet: Sparse-to-dense multi-view stereo with learned propagation and gauss-newton refinement. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1949-1958, 2020.

Z. Yu, S. Peng, M. Niemeyer, T. Sattler, and A. Geiger. Monosdf: Exploring monocular geometric cues for neural implicit surface reconstruction. arXiv preprint arXiv:2206.00665 , 2022.

S. Zagoruyko and N. Komodakis. Learning to compare image patches via convolutional neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 4353-4361, 2015.

J. Zhang, Y. Yao, S. Li, T. Fang, D. McKinnon, Y. Tsin, and L. Quan. Critical regularizations for neural surface reconstruction in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6270-6279, 2022.

K. Zhang, G. Riegler, N. Snavely, and V. Koltun. Nerf++: Analyzing and improving neural radiance fields. arXiv preprint arXiv:2010.07492 , 2020.

X. Zhang, S. Bi, K. Sunkavalli, H. Su, and Z. Xu. NeRFusion: Fusing radiance fields for large-scale scene reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5449-5458, 2022.

Peer reviews:
comments: SUMMARY:
This paper proposes a two-stage framework for neural 3D reconstruction from disparate views via neural templates regularization. In the first stage, a network is trained for predicting shape templates. After that the volumetric surface reconstruction network with depth and SDF constraints is trained with the templates prior. Compared to other 3D reconstruction methods, DiViNet specifically targets disparate input images and achieves SOTA in sparse image views.

STRENGTHS:
1.The proposed DiViNet gets rid of the require of explicit cues, unlike former methods. Moreover, only a small set of images with small overlaps are used.

2.This proposed designs are technically sound.

WEAKNESSES:
1. Although DiViNet performs best with sparse view inputs, with dense view inputs, it's worse than the latest methods. According to Table 2, DiViNet is the second-best method. 
The DiViNet achieves SOTA in too many constraints to other methods. MonoSDF[1] using multi-resolution feature grids gets 0.73 on DTU dataset.

2. As mentioned in the paper, the neural templates lacks generalization in different data distribution. It matters how big the influence of the data distribution is. The authors claim the drawback of other methods is the requirement of dense views with overlap. The generalization to new data distribution is also important in consideration of real-world setting.

[1]MonoSDF: Exploring Monocular Geometric Cues for Neural Implicit Surface Reconstruction

QUESTIONS:
Please refer to Weaknesses.

LIMITATIONS:
Please refer to Weaknesses.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-30

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes DiViNet for sparse multi-view reconstruction, which specifically targets on sparse input as few as three disparate RGB images. The key is to regularize the reconstruction process by learning a set of neural templates as surface priors, which is basically a set of 3D gaussian functions with optimizable features.  Extensive experiments are conducted to demonstrate the quality of DiViNet in sparse and disparate view settings.\n\nSTRENGTHS:\n•\tThis paper focuses on an important and practice problem. Priors works fail to reconstruct accurate geometries when input views are sparse, while this paper proposes novel neural temporal regularization method to achieve good quality even with only three disparate images as input.\n•\tThe idea of learning surface priors with neural templates is novel, and it’s effectiveness is also validated through the experiment section. \n•\tSome interesting observations are draw from the experiment part, for example, under sparse view input cases, VolSDF and NeuS even fail to bear COLMAP, while under dense input cases, all the neural methods outperform COLMAP.\n\nWEAKNESSES:\n•\tThe ability of learned neural template to generalize to new scenes is not very clear. On one hand they are learned across different scenes, on the other hand the author claimed the need to be learned again when deployed to datasets from a different data distribution. A clarification and explanation on under what scenario can the learned template functions be reused will be very helpful.\n•\tSince each learned template is represented as a scaled, anistropic 3D gaussian, a visualization showing the learned gaussian from a scene and the corresponding sparse reconstructed pointcloud from Colmap will be helpful. I am especially curious about the positional distribution and the scale of Nt gaussian, whether the query points are affected by only a small number of 3D gaussians or not.\n\nQUESTIONS:\n•\tThere are some point-based neural representation works which also consider 3D gaussian as their main geometry primitive, I feel the neural template function in this paper share a lots of insight with those papers. Including those paper in related works and providing some discussion will make the paper more stronger.\no\t3D Gaussian Splatting for Real-Time Radiance Field Rendering. Siggraph 2023\no\tNeural Point Catacaustics for Novel-View Synthesis of Reflections. ToG 2022\n\nLIMITATIONS:\nThe authors have discussed potential negative social impacts of this paper.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the problem of surface reconstruction from spase input views. The authors adopt a SDF representation parameterized by an MLP. They propose learning a set of neural templates (in the form of 3D Gaussian functions) to serve as anchors in the reconstruction process to help stitch the surfaces over sparse regions. Reconstruction is carried out by optimizing the SDF (MLP) through minimizing the rendering loss and the Eikonal loss. The authors introduce a depth loss and SDF loss, both computed based on the estimated neural templates, to regularize the optimzation. Experimental results on the DTU and BlendMVS datasets show the proposed approach can reconstruct surface details to a reasonable extent from few disparate input views.\n\nSTRENGTHS:\n+ The template prediction network can be trained end-to-end using RGB reconstruction loss without any 3D supervision.\n+ The predicted templates, which approxmiate points on object surface, help to regularize the SDF optimization, allowing the proposed approach to produce more complete reconstructions with reasonable surface details.\n+ The depth loss, which computes the difference between the rendered depth and depth cues obtained from the predicted templates, and the SDF loss, which evaluates the signed distance at template centers, both sound logical and correct in regularizing the SDF optimization. Their effectiveness has been validated through ablation study.\n\nWEAKNESSES:\n- It is not clear about the generalization ability of the template prediction network. No training details have been provided in the paper. In the evaluations, are the same datasets being used in both training and testing the template prediction network? What would be the performance if the template prediction network trained on one dataset is applied to a completely different dataset? How will this (negatively) affect the reconstruction?\n- Quite often the reconstructions also include incorrect background surface. Is that caused by incorrect template predictions? No discussions or analysis have been included in the paper.\n- The description of the template prediction network architecture is very confusing (both in the main paper and the supplementary material). For instance, it is not clear how per-template feature are being sampled using bilinear interpolation from the extracted feature maps. There is also no explanations for the design of the dimension (C x M x sqrt(N_t) . M x sqrt(N_t) . M) for the volumetric feature grid.\n- The formulation in (2) may be problematic. Note a ray will in general intersect an object surface in at least 2 points. This implies more than one template will produce a large value for w_k and therefore colors at multiple surface points will be mixed (as depth is not being considered).\n\nQUESTIONS:\nPlease refer to the points in weaknesses.\n- L_{rgb} in (3) should be L_{color} instead.\n- \\epsilon in (2) is undefined.\n- \\sigma in (9) is undefined. How is \\sigma and the signed distance being related?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

comments: SUMMARY:
This work presents a volume rendering-based sparse view neural surface reconstruction method. For the hard sparse view reconstruction ,the authors propose to learn neural templates as surface priors to guide the learning of neural fields. The results on DTU and Blended MVS are better than NeuS and MonoSDF .

STRENGTHS:
The idea of learning templates for enhancing spase view reconstrucion is novel and make sense.

WEAKNESSES:
This paper did not compare with the the most related works on sparse view reconstruction like SparseNeuS and VolRecon. Comparing only with NeuS / MonoSDF are not convincing, since this methods do not have special designes for sparse view reconstruction. 

The results on dense view reconstruction are quite poor, shown in Fig. 4. My concern lies in the poor results in dense view reconstruction, i.e. the artifacts in Fig.4, where all the baselines perform better. The quantitative comparisons shown in Tab.2 further deepen my concerns, where the SOTAs this paper not compared already reduce CD to less than 0.55 (e.g. Geo-NeuS), where this paper can only achieves comparable results with NeuS (0.84 vs. 0.79), which was published two years ago.

I do not think that choosing MonoSDF as the main baseline is convincing, since MonoSDF is mainly designed for scene-level reconstruction, where the monolar priors of MonoSDF are not suitable for object level reconstruction (e.g. DTU).

QUESTIONS:
Why not follow the experimental settings of SparseNeuS(ECCV22)?

Is the proposed method also suitable for scene-level multi-view reconstrucion (e.g. Replica/ScanNet)?

LIMITATIONS:
See the weaknesses above.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a framework for sparse view 3D reconstruction from disparate views. A two stage approach is presented for reconstruction of a scene from posed sparse images. In the first stage a template is predicted from the sparse images, represented by a number of parametric 3D gaussians. The second stage uses the predicted template to reconstruct the scene from sparse views. An SDF representation is used to represent the geometry. State of the art performance is shown on shape reconstruction and novel view synthesis.\n\nSTRENGTHS:\n1. **Clarity** : The paper is very well written with great attention to detail. Each component is adequately motivated. The approach section is built up in a very methodical and thorough manner. \n2. **Reproducibility**: The exact implementation details and training specifics are made clear, aiding in the reproducibility of the proposed approach. \n3. **Results**: The qualitative results especially for shape recovery is very compelling, particularly for scenes with wide baselines. \n5. **Novelty**: The use of gaussian templates to guide the reconstruction of the scene under sparse setting is a simple and elegant idea that is also easy to incorporate into existing frameworks as a form of regularization. This approach would serve as an important and strong baseline for sparse view 3D reconstruction method.\n6. **Quantitative analysis**: The approach has been validated against a variety of contemporary approaches and state of the art performance is shown on chamfer distance for recovered surface.\n\nWEAKNESSES:\n1. **Assumption on 3D information for stage 1**: The losses to train the template prediction network pretrains this network against dataset that have point clouds from COLMAP. Does this limits the applicability mainly to kind of scenes where COLMAP provides enough reconstruction information? Providing some more details about this prior is helpful. \n2. **Evaluation**: Although quantitative metrics are provided for geometry, also include image level metrics like PSNR/ SSIM/ LPIPS for the novel view synthesis task is potentially helpful to strengthen the narrative of the evaluation section.\n3. **Effect of number of template gaussians**: An ablation study showing how the number of gaussians are chosen and the effect that this number has on the reconstruction quality is instructive.\n4. **Additional ablations**: Quantitative ablations are provided for CD as a function of number of input views. However, the manuscript would benefit from the ablations below:\n> - *Quantitative ablation* showing the effect of different regularization terms (particularly the SDF constraint and the depth constraints).  \n> - *Quantitative/ Qualitative ablation* showing the direct optimization of posed sparse views without needing stage 1.  \n> - *Qualitative ablation* showing the importance of $L_{cov}$ , $L_{radius}$ and $L_{var}$ in Stage 1. \n5. **Video Results**: Although not strictly necessary, including turntable video results of the recovered geometry in the supplm will help demonstrate the efficacy of the approach better.\n\nQUESTIONS:\n1. $D_{geo}$ essentially takes the feature map and predicts the center, radius and scale. Does this imply that the template is only based on global level information of the scene, since all local information is lost?\n2. The exact specifics of how the spatial resolution of the feature map is flattened for $D_{geo}$ is unclear. Is there a pooling happening between the feature layer and the geometry decoder?\n\nLIMITATIONS:\nAdequate treatment of the limitations of the approach has been provided.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}