Voxurf: Voxel-based Efficient and Accurate Neural Surface Reconstruction
Authors: Tong Wu
Jiaqi Wang
Xingang Pan
Xudong XU
Christian Theobalt
Ziwei Liu
Dahua Lin
Venue: ICLR 2023 notable top 25% (2023)
Keywords: Surface Reconstruction
Neural Radiance Field

Abstract: Neural surface reconstruction aims to reconstruct accurate 3D surfaces based on multi-view images. Previous methods based on neural volume rendering mostly train a fully implicit model with MLPs, which typically require hours of training for a single scene. Recent efforts explore the explicit volumetric representation to accelerate the optimization via memorizing significant information with learnable voxel grids. However, existing voxel-based methods often struggle in reconstructing fine-grained geometry, even when combined with an SDF-based volume rendering scheme. We reveal that this is because 1) the voxel grids tend to break the color-geometry dependency that facilitates fine-geometry learning, and 2) the under-constrained voxel grids lack spatial coherence and are vulnerable to local minima. In this work, we present Voxurf, a voxel-based surface reconstruction approach that is both efficient and accurate. Voxurf addresses the aforementioned issues via several key designs, including 1) a two-stage training procedure that attains a coherent coarse shape and recovers fine details successively, 2) a dual color network that maintains color-geometry dependency, and 3) a hierarchical geometry feature to encourage information propagation across voxels. Extensive experiments show that Voxurf achieves high efficiency and high quality at the same time. On the DTU benchmark, Voxurf achieves higher reconstruction quality with a 20x training speedup compared to previous fully implicit methods. Our code is publicly available at https://github.com/wutong16/Voxurf/.

Published as a conference paper at ICLR 2023
VOXURF: VOXEL-BASED EFFICIENT AND ACCURATE NEURAL SURFACE RECONSTRUCTION
Tong Wu! Jiaqi Wang 1 Xingang Pan Xudong Xu², Christian Theobalt", Ziwei Liu", Dahua Lin 1:255 Shanghai AI Laboratory. "hhe Chinese University of Hong Kong, Maa Planck Institute for Informatics, LS-Lab, Nanyang Technological University, 'Centre of Perceptual and Interactive Intelligence 020, xx018, dhlin] cuhk. edu.bk, wang wang]iaql jlab. org.en, Ixpan, (enoball/empi-iee mpg.de, Aeee..lu@nttu wtw.uuntuuuuuu..e...
ABSTRACT
Neural surface reconstruction aims to reconstruct accurate 3D surfaces based multi-view images. Previous methods based on neural volume rendering mostly train fully implicit model with MLPs, which typically require hours of training for a single scene. Recent efforts explore the explicit volumetric representation accelerate the optimization via memorizing significant information with learn- able voxel grids. However, existing voxel- methods often struggle in recon- structing fine-gres fine-grained geometry, even when combined with an SDF-based volume rendering scheme. We reveal that this is because 1) the voxel grids tend to break the color-geometry dependency that facilitates fine-geometry learning, and 2) the under-constrained voxel grids lack spatial coherence and are vulnerable to local minima. In this work, we present Voxurf, a voxel-based surface reconstruction approach that is both efficient and accurate. Voxurf addresses the aforementioned issues via several key designs, including 1) a two-stage training procedure that attains coherent coarse shape and recovers fine details successively, 2) dual color network that maintains color-geometry dependency, and 3) a hierarchical geometry feature to encourage information propagation across voxels. Extensive experiments show that Voxurf achieves high efficiency and high quality at the same time. On the DTU benchmark, Voxurf achieves higher reconstruction qual- ity with 20x training speedup compared to previous fully implicit methods.
INTRODUCTION
Neural surface reconstruction based on multi-view images has recently seen dramatic progress, In- spired by the success of Neural Radiance Fields (NeRF) (Mildenhall et al., 2020) on Novel View Synthesis (NVS), recent works follow the neural volume rendering schemne to represent the 3D ge- ometry with signed distance function (SDF) or occupancy field via a fully implicit model (Oechsle) al. 2021 Yariv al.] 2021 Wang al., 2021). These approaches train deep multilayer per- ceptron (MLP), which takes in hundreds of sampled points on each camera ray and outputs the corresponding color and geometry information. Pixel-wise supervision is then applied by measur- ing the difference between the accumulated color on each ray and the ground truth. Struggling with learning all the geometric and color details with pure MLP-based framework, these methods require hours of training for a single scene, which substantially limits their real-world applications.
Recent advances in NeRF accelerate the training process with the aid of an explicit volumetric rep- resentation (Sun et al. 2022at] Yu et al. 20222) Chen et al.) 2022). These works directly store and optimize the geometry and color information via explicit voxel grids. For example, the density of queried point be readily interpolated from the eight neighboring points, and the view-dependent color is either represented with spherical harmonic coefficients (Yu et al. 2022 or predicted by shal- low MLPs that take learnable grid features as auxiliary inputs (Sun et al. 2022a). These approaches achieve competitive rendering performance at a much lower training cost (< 20 minutes). However, their 3D surface reconstruction results cannot faithfully represent the exact geometry, suffering from
M Corresponding authors. Our code is availahle

---

Published as a conference paper at ICLR 2023
Ground Truth
(a) at 4 mins.
(b) NeuS hours. (c) DVGD+NeuUS mins. (d)
Figure I: Comparisons among different methods for surface reconstruction and novel view synthesis.
(a) DVGO (v2) (Sun et al. 2022a b) benefits from the fastest convergence but suffers from poor surface; (b) NeuS (Wang et al. (022)) produces decent surfaces after a long training time, while high-frequency details are lost in both the geometry and the image; (c) the straightforward combination of DVGO and NeuS produces continuous but noisy surfaces; (d) our method achieves around 20x speedup than NeuS and recovers high- quality surfaces and images with fine details. All the training times are tested on a single Nvidia A GPU.
conspicuous noise and holes (Fig. (a). is due to the inherent ambiguity of the density-based vol- ume rendering scheme, and the explicit volumetric representation introduces additional challenges.
In this work, we aim to take advantage of the explicit volumetric representation for efficient training and propose customized designs to harvest high-quality surface reconstruction. straightforward idea for this purpose is to embed the SDF-based volume rendering scheme (Wang et al. 2021 Yariv al. 202T) into explicit volumetric representation frameworks (Sun et al., [202aa). However, we find this naïve baseline model not working well by losing most of the geometry details and producing undesired noise (Fig. (c)). We reveal several critical issues for this framework as follows. First, in fully implicit models, the color network takes surface normals as inputs, effectively building color- geometry dependency that facilitates fine-geometry learning. However, in the baseline model, the color network tends to depend more the additional under-constrained voxel feature grid input, thus breaking color-geometry dependency. Second, due to a high degree of freedom in optimizing a voxel grid, it hard to maintain a globally coherent shape without additional constraints. Individual optimization for each voxel point hinders the information sharing across the voxel grid, which hurts the surface smoothness and introduces local minima. We'll 'll unveil these effects and introduce the insight for our architecture design via an empirical study in Sec. 4
To tackle the challenges, we introduce Voxurf, an efficient pipeline for accurate Voxel-basee surface reconstruction: 1) We leverage two-stage training process that attains a coherent coarse shape and recovers fine details successively. 2) We design a dual color network that capable of representing complex color field via a voxel grid and preserving the color-geometry dependency with two sub- networks that work in synergy. 3) We also propose a hierurchical geometry feature based on the SDF voxel grid to encourage information sharing in a larger region for stable optimization. 4) We introduce several effective regularization terms to boost smoothness and reduce noise.
We conduct experiments on the DTU (Jensen et al. 2014) and BlendedMVS (Yao al. 2020 datasets for quantitative and qualitative evaluations. Experimental results demonstrate that Voxuri achieves lower Chamfer Distance
the DTU (Jensen al. 2014) benchmark than competitive fully implicit method NeuS (Wang et al.] (021) with around 20x speedup. also achieves remark- able results on the auxiliary task of NVS. As illustrated in Fig. [ our method is shown to be superior in preserving high-frequency details in both geometry reconstruction and image rendering compared the previous approaches. In summary, our contributions are highlighted below:
1. Our approach enables around 20x speedup for training compared to the SOTA methods, reducing the training time from over hours 15 minutes on a single Nvidia A100 GPU.
2. Our approach achieves higher surface reconstruction fidelity and novel view synthesis qual- ity, which is superior in representing fine details for both surface recovery and image ren- dering compared to previous methods.
2

---

Published as a conference paper at ICLR 2023
3. Our study provides insightful observations and analysis of the architecture design of the explicit volumetric representation framework for surface reconstruction.
RELATED WORKS
Multi-view 3D reconstruction Recently, implicit representations that encode the geometry and
appearance of a 3D scene by neural networks have gained attention (Park et al. 2019] [Chen &
Zhang 2019 Lombardi et al. 2019 Mescheder et al.] 2019 Sitzmann et al. [019b] [Saito et al.)
2019 Atzmon et al. 2019 Jiang et al. 2020] Zhang et al. 2021] Toussaint et al. [2022.. Among
them, plethora of papers have explored neural surface reconstruction from multi-view images.
Methods based on surface rendering (Niemeyer et al. 2020; Yariv et al.) 2020: Liu et al.) 2020b]
Kellnhofer al., 2021 regard the color of an intersection point of the ray and the surface as the final rendered color, while requiring accurate object masks and careful weight initialization. Recent approaches (Wang al. 2021; Yariv et al., 2021 Oechsle al., 202T Darmon et al. 2022 Zhang
et al. 2022: Liu et al.) 2020a Sitzmann et al.) 2019a) based on volume rendering (Max) 1995) for-
mulate the radiance fields and implicit surface representations in a unified model, thereby achieving
the merits of both techniques. However, encoding the whole scene in pure MLP networks requires long training time. In a departure from these works, we leverage learnable voxel grids and shallow color networks for quick convergence, well as pursue fine details in surfaces and rendered images.
Explicit volumetric representation Despite the success of implicit neural representations in 3D modeling, recent advances have integrated explicit 3D representations, e.g., point clouds, voxels, and MPIs (Mildenhall
2019), and received growing attention (Wizadwongsa et 2021 Xu al.,
2022 Lombardi et al.) 2019 Wang et al.] 2022,] Fang et al. (022). Instant- ngg (Müller et al. 2022]
uses multi-resolution hashing for efficient encoding and implements fully-fused CUDA kernels. Plenoxels (Yu et al. 2022) represent scene
sparse 3D grid with spherical harmonics and are
optimized two orders of magnitude faster than NeRF (Mildenhall et al., 2020). TensoRF (Chen) ct al. (022) considers the full volume field 4D tensor and factorizes it into multiple compact low-rank tensor components for efficiency. The method most related to ours is DVGO (Sun et al.) 2022a), which adopts hybrid architecture design including voxel grids and shallow MLP. Despite their remarkable results on novel view synthesis, none of them is designed to faithfully reconstruct the geometry of the scene. In contrast, target not only rendering photo-realistic images from novel viewpoints but also reconstructing high-quality surfaces with fine details.
3 PRELIMINARIES
Volume rendering with SDF representation. NeuS (Wang al.. 2021) represents a scene as an implicit SDF field parameterized by an MLP. The ray emitting from the camera center o through an image pixel in the viewing direction can be expressed as (p(t) o + tult 2 0). The rendered color for the image pixel is integrated along the ray with volume rendering (Max 1995), which is approximated by N discrete sampled points {pi = + +all = I..,, N, 4 4+11 1 on the ray:
N
i-1
C(r) = Σ Tass T. II (1-a)),
where os is the opacity value, and T, is the accumulated transmittance. The key difference between NeuS and NeRF is the formula of az, In NeuS, a; is formulated as:
T.. (f(((()))) T. (f(((()))))) 4, (f((((t))
a1 max
Here, fax is the SDF function, and (x) (1+e -- is the Sigmoid function, where the value is learned or manually updated during training.
Explicit volumetric representation. DVGO (Sun et al. 2022a) represents the geometry with applies a hybrid architecture for
explicit density voxel grids (ddensity) E RIXN,XX,xNV color prediction that comprises a shallow MLP parameterized by O and a feature voxel grid Given a 3D position p and the viewing direction e, the volume den-
V(feat) E sity and color are estimated with:
interp(p, V(density)),
(3)
3

---

Published as a conference paper at ICLR 2023
good case
7t: normal
(a)
(b)
(d)
surface
normal
feature (vis)
fi feature
n(x) [(x)
nlv) ((x)
n(x) [(v]
n(v) fl
nlVv
f(x)
None
case (1)
nlv
MM
Chamfer Distance
2,30
2.44
1.43
nlv
case (2)
(xx
None
n(v)
f(V)
Chamfer Distance
2,07
0.67
0.91
0.89
Figure 2: Reconstruction results from different architecture designs. The surface normal and learnable feature are both optional inputs to the color network. We show results of two left, and we zoom to analyze the surfaces, normal fields, and feature fields on the right. Case (1) (a, c) and (b, d) show that the feature f helps maintain a coherent shape, while case (2) (b, d) reveal that it discourages the reconstruction of geometry details since it disturbs the color-geometry dependency built by the normal n.
under four settings the
MLPe (intepnt V(feat)), ), ,, p,v,,
where 'interp' denotes the trilinear interpolation. Following NeRF (Mildenhall et al. 2020 Tancik et al. 2020), the positional encoding for both p and v is applied in Equ.
Naïve Combination. A straightforward combination of the two techniques is to replace the volume rendering in DVGO with the SDF-based volume rendering scheme as in Eqn. and Eqn. 2 It serves as the naive baseline in this work, which can hardly produce satisfactory results, as shown in Fig. (c). We will cast light this phenomenon via empirical study in the next section.
4 STUDY ON ARCHITECTURE DESIGN FOR GEOMETRY LEARNING
In this section, we carry out some prior experiments with variants of the baseline model, aiming to figure out the key factors for architecture design this task. Specifically, we employ an SDF voxel grid ((adf) and apply Eqn. for calculation with a manually defined schedule for s. We start with shallow MIP the color network, where 1) the local feature f interpolated from V(feat) and 2) the normal vector n calculated by (ssdf) are both optional inputs. A decent surface reconstruction expected to possess a coherent coarse structure, accurate fine details, and a smooth surface. We will next focus on these factors and analyze the effects of different architecture designs.
The key to maintaining a coherent coarse shape. Intuitively, the capacity of a shallow MLP is limited, and it can hardly represent a complex scene with different materials, high-frequency textures, and view-dependent lighting information. When the ground truth image encounters a rapid color-shifting, the volume rendering integration over an under-fitted color field results in a corrupted geometry, as shown in Fig. 2 case (1) (a) and (b). Incorporating the local feature ff enables fast color learning and increases the representation ability of the network, and the problem is noticeably alleviated, as shown in Fig. 2 case (1), the differences between (a) and (c), (b) and (d).
The key to reconstructing accurate geometry details. We then introduce another case in Fig. 2 case (2). Its texture changes moderately, and the color is largely correlated with the surface normal due to diffuse reflection. Although the geometry still collapses given neither normal n or feature as input in Fig. 2 case (2) (a), we can observe reasonable reconstruction even with some geometry details in Fig. 2 case (2) (b) with only n as the input. Incorporating the feature f does not further reduce the Chamfer Distance (CD); instead, geometry details are missing since the learnable feature if disturbs the geometry-color dependency, i.e., the relationship built between the color and the surface normal, as shown in Fig. 2 case (2), the differences between (b) and (d). Considering the cons and pros of the learnable local feature, it is valuable to design a framework that leverages it for coherent shape and keeps color-geometry dependency for fine details.
4

---

Published as a conference paper at ICLR 2023
Hierarchical geometyy feature f
Multi level sampling
Bay r
SDF value
Level 0.0
By
Normal vector ng
Level 0.5
V
Level 1.
Sare
Iewel 1.5
color (r))
Level 2,0
MLP
Readerr
Optimization region for one point
Position p.
detuch
deooch
Lewen
V'eeodt
F.
ML
Reader
((P)
Learnable feature /
The color denoteslevel
Explicit volumetric representation
Dual color network
Hierarchical geometry feature
Overview key components our model. We adopt an explicit volumetric representation with SDF voxel grid (((df) and feature voxel grid ((feat), In the middle, we show the design for dual color network, where f!eat is the interpolated feature from Vifeat) at point and f denotes the hierarchical feature constructed on the right. Here we show the multi-level sampling scheme and the region of grids that is affected by one point during optimization with different settings of levels.
Figure 3:
The reason for noisy surfaces. For all the cases above, the results suffer from obvious noise on the surface. Compared with learning implicit representation globally, the mnder-constrained voxel grids lack spatial coherence and are vulnerable to local minima, which hurts the continuity and smoothness of the surface. An intuitive idea is leveraging geometry from region rather than local point, which can be introduced in model inputs, network components, and loss functions.
5 METHODOLOGY
we propose several key designs: 1) we adopt two- stage training procedure that attains a coherent coarse shape (Sec. [.I and recovers fine details (Sec. 5.2) successively; 2) we propose a dual color network to maintain color-geometry dependency and recover precise surfaces and novel-view images; 3) we design hierarchical geometry feature to encourage information propagation across voxels for stable optimization; 4) we also introduce smoothness priors, including a gradient smoothness loss better visual quality (Sec. 5.3
Inspired by the insight revealed in Sec. 4
5.1 COARSE SHAPE INITIALIZATION
We initialize our SDF voxel grid (adff with an ellipsoid-like zero level set inside a prepared region for reconstruction in (Sun et al., 2022a). We then perform coarse shape optimiza- train shallow MLP tion with the aid of V(feat) introduced in Sec. 4 Specifically, with both normal vector n and local feature f as inputs, along with the embedded position To encourage stable training process and smooth surface, we pro-
and viewing direction pose to conduct the interpolation In particular. we denote G(V, ky. og)) as applying 3D convolution on the voxel grid V with Gaussian kernel, whose weight matrix follows Gaussian distribution: K.J., exp (-(ii- - 2 + (j Lkg/ /2]2 (k 2 Z denotes a normalization term, g denotes the kernel size, and 0g denotes the standard deviation. Querying smoothed SDF value d' of an arbitrary point p thus becomes:
smoothed voxel grid rather than the raw data of V(adf),
1/Z x .ij,k E {0, 1,.,,, kg 1}. where
d' interp(p, (paterp(,.(((vdd)) ligs 0g))
(5)
We d' for the ray marching integration following Eqn. land Eqn. 2 and calculate the reconstruc- tion loss. We also apply several smoothness priors as to be introduced in Sec. [5.
5.2 FINE GEOMETRY OPTIMIZATION
At this stage, we aim to recover accurate geometry details based on the coarse initialization. We note that the challenges are two-fold: 1) The study in Sec. 4reveals trade-off introduced by the feature voxel grid, i.e., the representation capacity of the color field improved at the sacrifice of color- geometry dependency. 2) The optimization of the SDF voxel grid is based trilinear interpolation
5

---

Published as a conference paper at ICLR 2023
to query 3D point. The operation brings in fast convergence, while it also limits information sharing across different locations, which may lead to local minima with degenerate solutions and a sub-optimal smoothness. We propose a dual color network and hierarchical geometry feature to address these two issues, respectively.
Dual color network. The observation in Sec. 4 encourages to design dual color network that takes advantage of the local feature J.feat interpolated from the learnable feature voxel grid V(feat) without losing the color-geometry dependency. As shown in Fig. 3 we train two shallow MLPs with different additional imputs besides the embedded position and view direction. The first MLP Igeo takes the hierarchical geometry feature _ā which will be introduced later, to build the color- geometry dependency: the second one fet takes both simple geometry feature (i.e, the surface normal n,) and the local feature fi f:fent as inputs to enable a faster and more precise color learning, which will in turn benefit the geometry optimization. The two networks are combined in a residual manner with detaching operations: the output of Igeo. denoted by Co. is detached before input to feoat and the output is added back to detached copy of co to get the final color prediction c.
Outputs of both Igeo and Gfeat are supervised by reconstruction loss between the ground truth image and the integrated color along the ray. Specifically, the rendered colors from them are denoted as Co (r) and C(r), and the overall reconstruction loss is formulated as:
Crecon
1 - R
Σ ((C())- 0()) + Aol(Co(r) - -r())1B. FE
(6)
where C(r) denotes the ground truth color, and Ao denotes a loss weight. ((feet) and the MLP Ifeat fit the scene field rapidly, while the MLP Igeo fits the scene at a relatively slower pace. The detaching operations promote stable optimization of Ageo guided by the reconstruction loss of itself, which helps preserve color-geometry dependency.
Hierarchical geometry feature. Using the surface normal n as the geometry feature for the color networks is straightforward choice, while it takes in information only from adjacent grids of ((SDF), In order to enlarge the perception area and encourage information propagation across voxels, we propose to look at a larger region of the SDF field and take the corresponding SDF values and gradients
auxiliary condition to the color networks. Specifically, for given 3D position p = (r,y. y, 2), we take half of the voxel size U2 as the step size and define its neighbours along the X, Y, Z axis both sides. Taking the X axis as example, the neighbouring coordi- nates are defined as ph- = (x' UU and //= = (x'+, 2), where x' = max - vs, 0). rrft min(x: 1 Us; 1g ), E [0.5, 1.0, 1.5, ...] denotes the "level' of neighbour area, and u/ denotes the maximum of the voxel grid on x axis. We then extend the definition to hierarchical manner by concatenating the neighbours from different levels together as formulated below:
dk [dh df [interp(p'
1- ((ad))) intter((pp ++ V(edf)),, E N.,
(7)
do. p (1) [d do.5 .e...... ..... ,.. Lz df d.
where d' denotes the SDF values queried from ((ddf) at locations p/- and ps+, When 0. fael (0) d", which is exactly the SDF value at the location itself. Then, we also incorporate the gradient information into the geometry feature. Specifically, we can obtain the gradient vector
5% = (d== d22) v.). We normalize the 8 6y, 8%] to L2-norm of I, denoted as n' E R3
The hierarchical version of the normal is formulated as:
Pp pnormal (l) = [,. '''..
(8)
Finally, the hierarchical geometry feature at point p for a predefined level E [0.5, 1.0, 1.5, ...] is to combine the information above by:
(rea()) (I) = [ (I), Jp pnormal (1))
(9)
As shown in Fig. 3 fg is input to the MLP Iyeo to assist geometry learning.
5.3 SMOOTHNESS PRIORS
We incorporate two effective regularization terms to facilitate surface smoothness during training. (1) First, we adopt a total variation (TV) regularization (Rudin & Osher 1994):
Crv(V)= = Σ
A3 d) A3(V, d) + A3(V,d),
(10)
dE[D]
6

---

Published as a conference paper at ICLR 2023
Table 1: Quantitative evaluation on DTU dataset.
Scan
24
37
40
55
63
65
83
97
105
106
110
114
118
122
Mildenhall el
1.83
2.39
0.66
1.79
2.61
0.99
1.54
IDR Yarv et ni AT4D
1.63
1.87
0.63
0.48
1.04
0.79
0.77
1.33
1.16
0.76
0.67
0.90
0.42
0.51
0.53
0.90
SvG
2022a
1.83
1.74
1.70
1.53
1.91
1.91
1.77
2.60
2.08
1.79
1.76
2.12
1.60
1.80
1.58
1.85
PointNeRF Xu ct 20
0.87
2.06
1.20
1.01
1.01
1.39
0.80
1.04
0.92
0.74
0.97
0.76
0.56
0.90
1,05
1,02
NeuSi Wang et 4121
0.83
0.98
0.56
0.37
1.13
0.59
0.60
1.45
0.95
0.78
0.52
1.43
0.36
0.45
0.45
0.77
DVGO
0.87
Ours
0.65
0.74
0.39
0.35
0.96
0.64
0.85
1.58
1.01
0.68
0.60
1.11
0.37
0.45
0.47
0.72
Table 2: An overall comparison surface reconstruction, novel view synthesis, and training time on DTU.
PSNR
SSIM 1
LPIPS
CD
Time (Nvidia A100)
DVGO (Sun al.
2022a
31.64
0.916
0.159
1.85
4 mins
NeuS
Wang ot al
2021
29.63
0.892
0.199
0.77
5.5 hours
Ours
32.16
0,929
0.144
0.72
15
where (V. denotes the squared difference between the value of dth channel in voxel == (i:j;k) k) and the dth value in voxel (i + 1;j; k), which can be analogously extended to A and (V; d). We apply the TV term above to the SDF voxel grid, denoted by LTv (V(6d)),, which encourages continuous and compact geometry.
(2) We also assume the surface to be smooth in local area, and we follow the definition of the Gaussian convolution Sec. 5. and introduce smoothness regularization formulated as:
Ssmooth (V) || ky. V12.
(11)
We apply the smoothness term above the gradient of SDF voxel grid for a gradient smoothness loss, denoted by Lemoohh [ (>d()) It encourages smooth surface and alleviates the issue of noisy points in the free space. Notice that we can also naturally conduct post-processing on the SDF field after training, thanks to its explicit representation. For example, applying the Gaussian kernel above before extracting the geometry can further boost surface smoothness for better visualization.
Finally, the overall training loss is formulated as:
L = Lrenon + At (V(a4)) AaLsmooth (VV(s))),
(12)
where Ato and D denote the weights for the corresponding loss terms.
EXPERIMENTS
Experimental setup. We use the DTU (Jensen al. 2014) dataset for quantitative and qualitative
comparisons and show qualitative results on several challenging scenes from the BlendedMVS ( Yao
al., (2200) dataset. We include several baselines for comparisons:
IDR Yarriv et al. 2020))
2) NeuS (Wang et al.] 2021). 3) NeRF (Mildenhall et al.) 2020). 4) DVGO (Sun et al.) 2022a). 5)
Point-NeRF (Xu et al. 2022). The results of 1), 2), and 3) are taken from the original papers ( Yariiv the neural point for evaluation. We provide a clean background for all the methods for a fair comparison. Experimental results with non-empty back- grounds and comparisons with more methods (Schönberger et al. 2016 Oechsle al. 2021 Yariv et al. 2021) are included in the supplementary materials. Please also refer to the supplementary materials for further descriptions of the datasets, baseline methods, and implementation details.
et al. 2020: Wang al.) 202I); for 5),
6.1 COMPARISONS
The quantitative results for surface reconstruction on DTU are reported in Table 1 Quantitative experimental results show that we achieve lower Chamfer Distances than previous methods under the same setting. We conduct qualitative comparisons on both DTU and BlendedMVS in Fig. and Fig. 5 respectively. DVGO shows poor reconstruction quality with noise and holes since is designed for novel view synthesis rather than surface reconstruction. NeuS and ours show accurate and continuous surface recovery in a variety of cases. In comparison, NeuS, as a fully implicit model, naturally benefits from the intrinsic continuity and encourages smoothness in local areas,

---

Published as a conference paper at ICLR 2023
Reference Image
DVGO
NeuS
Ours
Sean 24 (DTU)
Scan 65 (DTU)
Sean 110 (DTU)
Figure 4: Qualitative comparisons on the DTU dataset. See more scenes in supplementary materials.
Reference Image
DVGO
NeuS
Ours
Man (BlendedMyvS)
Clock (BlendedMVS)
Figure 5: Qualitative comparisons the BlendedMVS dataset. See more scenes in supplementary materials.
while sometimes fails to recover very thin geometry details due to over-smoothing In contrast, our method is superior recovering fine geometry details thanks to our designs in Sec.
We further perform more extensive evaluation of our method on surface reconstruction, novel view synthesis, and training time in Table 2 Our method outperforms DVGO and NeuS on both surface reconstruction and novel view synthesis by a clear margin on all the metrics. Notably, our method achieves around 20x speedup compared to NeuS for producing high-quality surface reconstruction.
6.2 ANALYSIS
In this section, we carry out series of ablation studies to evaluate each technical component.
The effect the dual color network and a hierarchical geometry feature. As shown in Table both techniques individually work well on the baseline model, and combination of them produces the best result. 1) The effect dual color network can be directly sensed the improvement of image rendering quality, can be seen from the comparison of roof textures in Fig. 6 An accurate color field and the color- moor-ggemetry dependency will promote geometry learning, as can be observed from the comparison of roof geometries (viewed in normal images) in Fig, 6 Experimental results in Table 4 also validate the effectiveness of the design introduced in Sec. 5.2, including the residual color and detachment. 2) Hierarchical geometry feature directly promotes accurate
8

---

Published as a conference paper at ICLR 2023
GT (test view)
w/o dual color network
w/o geometry feature
Full model
33
Figure 6: The dual color network learns the color field for complex scenes well and preserves color-geometry dependency, which facilitates geometry learning (see the roofs); the hierarchical geometry feature promotes accurate surface reconstruction (see the windows).
Table 3: Ablation over the effect of dual color network and hierarchical geometry feature.
Table Ablation over Residual and Detach designs of the dual color network (Sec. 5.2).
CD
0.91
0.79
0.77
0.72
CD
0.77
0.75
0.75
0.72
Dual
Residual Detach
Hierarchical
surface reconstruction, as demonstrated by results in Table 3 and the difference between normal images of Fig. 6 We also explore different design details, including the level selection and the effects of gradient and SDF value in supplementary materials.
Ablation over smoothness priors. We make efforts to encourage the continuity and smoothness of the reconstructed surface at different stages. As shown in Fig. 7 (a), during the coarse shape ini- tialization stage, the naive solution produces holes and noises. Applying the Gaussian convolution substantially alleviates the problem and leads to a more compact geometry. Regularization terms including the TV and gradient smoothness loss would further encourage clean and smooth surface to provide a good initialization for the next stage. Fig- (b) shows that during the fine geom- etry optimization stage, the regularization terms also help maintain surface smoothness. Finally, as shown in Fig. (c), post processing on trained model can promote surface smoothness for better visualization quality and maintain an accurate structure at the same time. An ablation study on the effects of the SDF TV term and our gradient smoothness loss is in the supplementary materials.
(a) Training stage-1
(b) Training stage-2
(c) Inference stage
baseline
wi` Gauasian
w) Gaussian regularication Gradient smoothness loss
w/o SDF-TV
w| SDF-TV & Gradient smoothness loss
who post wi post processing processing
Figure 7: Studies on technical components that encourage surface smoothness during the (a) coarse shape initialization, (b) fine geometry optimization, and (c) post-processing stage.
CONCLUSION
This paper proposes Voxurf. a voxel-based approach for efficient and accurate neural surface recon- struction. It includes several key designs: the two-stage framework attains coherent coarse shape and recovers fine details successively; the dual color network helps maintain color-geomett de- pendency, and the hierarchical geometry feature encourages information propagation across voxels; effective smoothness priors including a gradient smoothness loss further improve the visual quality. Extensive experiments show that Voxurf achieves high efficiency and high quality the same time.

---

Published as a conference paper at ICLR 2023
8 ACKNOWLEDGEMENT
This work is supported by Shanghai Al Laboratory, NTU NAP, MOE AcRF Tier 2 (MOE- T2EP20221-0012), and under the RIE2020 Industry Alignment Fund Industry Collaboration Projects (IAF-ICP) Funding Initiative.
10

---

Published as a conference paper at ICLR 2023
REFERENCES
Matan Atzmon, Niv Haim, Lior Yariv, Ofer Israclov, Haggai Maron, and Yaron Lipman. Controlling neural level sets. Advances in Neural Information Processing Systems, 32, 2019.
Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. Tensorf: Tensorial radiance fields. In European Conference on Computer Vision, 2022.
Zhiqin Chen and Hao Zhang. Learning implicit fields for generative shape modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5939-5948, 2019.
François Darmon, Bénédicte Bascle, Jean-Clément Devaux, Pascal Monasse, and Mathieu Aubry. Improving neural implicit surfaces geometry with patch warping. Proceedings of the IEEE/CVF International Conference on Computer Vision, 2022.
Jiemin Fang, Taoran Yi, Xinggang Wang, Lingxi Xie, Xiaopeng Zhang, Wenyu Liu, Matthias Nießner, and Qi Tian. Fast dynamic radiance fields with time-aware neural voxels.
arxiv:2205. 11..005..512855 15285, 2022.
Rasmus Jensen, Anders Dahl, George Vogiatzis, Engin Tola, and Henrik Aanæs. Large scale multi- view stereopsis evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp. 406 413, 2014.
Yue Jiang, Dantong Zhizhong Han, and Matthias Zwicker. Sdidiff: Differentiable rendering of signed distance fields for 3d shape optimization. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
Petr Kellnhofer, Lars C Jebe, Andrew Jones, Ryan Spicer, Kari Pulli, and Gordon Wetzstein. Neural lumigraph rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4287-4297, 2021.
Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt. Neural sparse voxel fields. Advances in Neural Information Processing Systems (NeurIPS), 2020a.
Shaohui Liu, Yinda Zhang, Songyou Peng, Boxin Shi, Mare Pollefeys, and Zhaopeng Cui. Dist: Rendering deep implicit signed distance function with differentiable sphere tracing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 2019-2028, 2020b.
Stephen Lombardi, Tomas Simon, Jason Saragih, Gabriel Schwartz, Andreas Lehrmann, and Yaser Sheikh. Neural volumes: Learning dynamic renderable volumes from images. ACM Transactions Graphics (TOG), 2019.
Nelson Max. Optical models for direct volume rendering. IEEE Transactions Visualization and Computer Graphics, 1(2)999-10,, 1995.
Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Oc- cupancy networks: Learning reconstruction in function space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4460-4470, 2019.
Ben Mildenhall, Pratul P. Srinivasan, Rodrigo Ortiz-Cayon, Nima Khademi Kalantari, Ravi Ra- mamoorthi, Ren Ng, and Abhishek Kar. Local light field fusion: Practical view synthesis with prescriptive sampling guidelines. ACM Transactions on Graphics (TOG), 2019.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi. and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In European conference on computer vision, pp. 405 421. Springer, 2020.
Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics prim- itives with multiresolution hash encoding. ACM Trans. Graph., 2022.
Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger. Differentiable volumet- ric rendering: Learning implicit 3d representations without 3d supervision. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 3504-3515, 2020.

---

Published as a conference paper at ICLR 2023
Michael Occhsle, Songyou Peng, and Andreas Geiger. Unisurf: Unifying neural implicit surfaces and radiance fields for multi-view reconstruction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5589-5599, 2021.
Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 163-174, 2019.
Leonid I Rudin and Stanley Osher. Total variation based image restoration with free local constraints. In Proceedings of 1st international conference on image processing, volume 1, pp. 31-35. IEEE,
1994.
Shunsuke Saito, Zeng Huang, Ryota Natsume, Shigeo Morishima, Angjoo Kanazawa, and Hao Li. Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2304-2314, 2019.
Johannes I. Schönberger, Enliang Zheng, Jan-Michael Frahm, and Marc Pollefeys. Pixelwise view selection for unstructured multi-view stereo. European Conference on Computer Vision, pp. 501-518. Springer, 2016.
Vincent Sitzmann, Justus Thies, Felix Heide, Matthias Nießner, Gordon Wetzstein, and Michael Zollhofer. Deepvoxels: Learning persistent 3d feature embeddings. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2437-2446, 2019a.
Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. Scene representation networks: Con- tinuous 3d-structure-awarn neural scene representations. In Advances in Neural Information Processing Systems, 2019b.
Cheng Sun, Min Sun, and Elwan--Tzone Chen. Direct voxel grid optimization: Super-fast con- vergence for radiance fields reconstruction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2022a.
Cheng Sun, Min Sun, and Swaan-Tzzon Chen. Improved direct voxel grid optimization for radiance fields reconstruction, 2022b.
Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions low dimensional domains. Advances in Neural Information Processing Systems, 33:7537-7547, 2020.
Briac Toussaint, Maxime Genisson, and Séan-Sébstiin Franco. Fast Gradient Descent for Surface Capture Via Differentiable Rendering. 3DV 2022 International Conference on 3D Vision, pp. 1-10, September 2022.
Jingwen Wang, Tymoteusz Bleja, and Lourdes Agapito. Go-surf: Neural feature grid optimization for fast, high-fidelity rgb-d surface reconstruction. 2022 International Conference on 3D Vision (3DV). IEEE, 2022.
Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. Advances in Neural Information Processing Systems (NeurIPS). 2021.
Suttisak Wizadwongsa, pakkapon Phongthawee, Jiraphon Yenphraphai, and Supasorn Suwa- janakorn. Nex: Real-time view synthesis with neural basis expansion. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 8534 8543, 2021.
Qiangeng Xu, Zexiang Xu. Julien Philip, Sai Bi, Zhixin Shu, Kalyan Sunkavalli, and Ulrich Neumann. Point-nerf: Point-based neural radiance fields. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognitio, 2022.
Yao Yao, Zixin Luo, Shiwei Li, Jingyang Zhang, Yufan Ren, Lei Zhou, Tian Fang, and Long Quan. Blendedmvs: A large-scale dataset for generalized mult stereo networks. In Proceedings of the IEER/CVF Conference Computer Vision and Pattern Recognition, pp. 1790-1799, 2020.
12

---

Published as a conference paper at ICLR 2023
Lior Yariv, Yoni Kasten, Dror Moran, Meirav Galun, Matan Atzmon, Basri Ronen, and Yaron Lipman. Multiview neural surface reconstruction by disentangling geometry and appearance.
Advances in Neural Information Processing Systems (NeurIPS), 33, 2020.
Lior Yariv, Jiatao Gu, Yoni Kasten, and Yaron Lipman. Volume rendering of neural implicit surfaces. In Thirty-Fifth Conference on Neural Information Processing Systems, 2021.
Alex Yu, Sara Fridovich-Keil, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels: Radiance fields without neural networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2022.
Jingyang Zhang, Yao Yao, and Long Quan. Learning signed distance field for multi-view surface reconstruction. International Conference on Computer Vision (ICCV), 2021.
Xiaoshuai Zhang, Sai Bi, Kalyan Sunkavalli, Hao Su, and Zexiang Xu. Nerfusion: Fusing radi- ance fields for large-scale scene reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
13


Reviews:
[{"review_id": "mzT-TajJja", "reviewer": "Reviewer_kZqD", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a fast, voxelized algorithm for implicit surface reconstruction and novel view synthesis. It addresses limitations of previous methods in preserving fine geometric details by a two-stage training procedure starting with a coarse resolution grid and moving on to finer resolution, a two-branch color network, and a hierarchical geometry feature that promotes inter-voxel coherence. Surface reconstruction and novel view synthesis results are presented on the DTU and BlendedMVS datasets, along with comparisons to relevant methods. Extensive ablation studies are also presented.", "summary_of_the_review": "At least two of the key ideas of the paper, specifically the dual color network and the hierarchical feature design, are novel, but it is unclear to me how effective they are beyond the context of the algorithm presented here. I was hoping that a thorough evaluation, including comparisons with previous work, would answer this question. The results presented in the paper are extensive, but they fall short of this because they are not performed on the same datasets as the most relevant previous methods. Important relevant work, such as PointNeRF, is omitted from the evaluation. I consider this kind of evaluation necessary for making a decision in favor of acceptance.\n\n+++ UPDATE DURING DISCUSSION +++\n\nI have been convinced by the authors’ responses and I have upgraded my rating of the paper. \n", "strength_and_weaknesses": "Strengths\n\nThe paper presents an approach for neural surface reconstruction employing an SDF representation decomposed in voxels. To accelerate convergence, the authors adopt a two-stage training procedure. I consider this a strength of the paper, but not a major one. It is not surprising that such a scheme is fast and also effective, as long as the coarse grid is of sufficient resolution.\n\nThe dual color network is novel to the best of my knowledge.  It comprises two branches that aim to synthesize accurate colors in novel viewpoints. The design is motivated by the observation that color and geometry drift apart as larger contexts are considered for geometric optimization. The two branches are based on different feature inputs, but also share information, to synthesize correct views while contributing to geometry optimization and fast convergence.\n\nThe hierarchical geometry feature enables information propagation across voxels by considering neighborhoods of different size, mitigating a limitation of voxel-based formulations. In my opinion, this is the most important contribution of the paper.\n\nThe smoothness priors are shown to be effective experimentally. This indicates that the dual color network and the hierarchical geometric features do not oversmooth. It may have been possible to achieve sufficient regularization with the other components of the network, but since this is not the case, the additional terms defined in Section 5.3 contribute. (The impact of all network components and their interactions on convergence time is unclear.)\n\nThe proposed approach is faster than many alternatives relying on global representations, but see also below for a relevant weakness.\n\nAn important strength of the paper is that ablations studies for all aspects are presented in the main paper and the supplement.\n\nWeaknesses\n\nA minor question, not necessarily a weakness, in my opinion is the loss function associated with the dual color network. It is unclear to me why the loss aims at having both branches achieve perfect view synthesis independently. It may be more advantageous to require that the combination of the two branches achieves this goal.\n\nThe method is fast, but it is much slower than the naïve baseline constructed by the authors at the end of Section 3, specifically DVGO with an SDF representation. This is not surprising due to the second stage of training, but weakens the argument that the proposed method somehow advances the efficiency of neural surface reconstruction methods. It seems to be on the efficiency-fidelity curve of voxel-based approaches.\n\nAn important weakness in the evaluation is that the comparison with DVGO, which is similar to the naïve baseline, is performed on different datasets than those used by the authors of DVGO. In particular, four of the five datasets used in the DVGO paper are missing, with only BlendedMVS used in both papers. The four missing datasests are: Synthetic-NeRF (Mildenhall et al., ECCV 2020), Synthetic-NSVF (Liu et al., NeurIPS 2020 – see below), Tanks & Temples (Knapitsch et al., SIGGRAPH 2017) and DeepVoxels (Sitzmann et al., CVPR 2019). Some of these papers are not even cited. (See below.) PointNeRF by Xu et al., which is cited, presents results on the DTU dataset with better PSNR, SSIM and LPIPS than the proposed method, while also being faster.\n\nThe surface reconstruction results on DTU, as measured by the average of the Chamfer distances between the reconstructed and ground truth points in both directions, are much weaker than those of recent MVS methods. MVS reconstructions are in the order of 0.3-0.35 mm while the proposed method is at 0.72 mm. (MVS papers use the same test set on DTU, which may be different than the one used here and by DVGO, but I assume that the statistics should be similar on average.) Chamfer distance may not measure interesting properties of the surfaces such as smoothness and watertightness, but the authors accept it as a metric. \n\nBased on the above, novel view synthesis should be the focus of the evaluation, since NeRF-style methods are not ready to be used for explicit surface reconstruction yet. Therefore, additional experiments on datasets widely used in the NeRF literature are needed to demonstrate the advantages of the proposed method.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "m9Puhr4m4L", "reviewer": "Reviewer_7aTz", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a hybrid voxel-grid- and MLP-based neural surface reconstruction method called Voxurf, that is both fast and accurate. It uses the volumetric representation of a voxel grid to expedite training, while proposing a two-stage training process, a novel network that captures the relationship between color and geometry, and a more expressive SDF feature, all in order to improve reconstruction accuracy. \n\nThe paper also provides a study on architecture design for hybrid volume/MLP geometry estimation models. This study motivates the insight behind the design decisions in the proposed method, and also provides clarity on how this work differentiates itself from prior research, such as DVGO (Sun et al., 2021) and NeuS (Wang et al., 2021).\n\nExperiments are performed on two common, relevant datasets, DTU and BlendedMVS, and the results demonstrate improvements in surface reconstruction quality with faster convergence times. Furthermore, ablation studies are included to test individual aspects of the proposed architecture.", "summary_of_the_review": "This paper effectively makes the case for Voxurf as an efficient solution for high quality neural surface reconstruction. The paper is well motivated, sufficiently evaluated, and clear with its insights. There are areas the paper could be improved, specifically in building out some of the experimental analysis further. Nevertheless, I would recommend this paper for acceptance to ICLR.", "strength_and_weaknesses": "**Strengths**\n\n* The choice of baseline as the straightforward combination of DVGO (Sun et al., 2021) and NeuS (Wang et al., 2021) is effective for demonstrating the performance improvements that Voxurf offers. The improved speed and reconstruction accuracy results helps to answer the gut-response question of \"why not just combine them?\".\n\n* The architecture design analysis in Section 4 and references to it in the methodology explanation in Section 5 is very helpful to motivate and explain design decisions. This paper takes great pains to justify the network architecture and loss function. In many papers, this insight is often omitted or overly brief, which limits the ability of future researchers to learn from the work. \n\n* The proposed idea to low-pass filter the SDF volume with a Gaussian kernel, in particular, seems to have an important effect on producing smooth surface reconstructions. Although, it may be fair to question whether the emphasis on smooth outputs is overdone, as Figure 7 appears to show fine detail being smoothed away when combining both TV regularization and the Gaussian filtering. Evaluating the impact of each of the two smoothing components is an ablation study that is missing, and the paper would benefit from including.\n\n\n**Weaknesses**\n\n* Section 4 describes the key to maintaining a coherence coarse shape to be the local feature, pointing out the poor geometry in Figure 2 case (1)(b) as an example of how ignoring the local feature leads to noisy reconstructions. On the other hand, Figure 2 case (2)(b) uses the same setup but provides top performance (and also qualitatively looks pretty clean). Moreover, it strongly preserves the coarse shape and maintains fine detail. This figure seems to undercut the point of the claim.\n\n* Table 2 presents a comparison of surface reconstruction and novel view synthesis on the DTU dataset. Although Voxurf is proposed to combine the speed benefits of DVGO with the output quality of NeuS, it is an interesting result to see that DVGO outperforms NeuS on novel view synthesis metrics at baseline. Moreover, Voxurf is only marginally better than DVGO for novel view synthesis despite taking 3.5x as long to train. Now, in terms of surface reconstruction metrics, Voxurf supports its claims by outperforming NeuS significantly in reconstruction quality in a fraction of the time. However, this discrepancy ought to be explained. Is it a problem with the expressiveness of the PSNR and SSIM metrics for NVS? Perhaps there is diminishing returns in NVS as reconstruction accuracy increases?\n\n* Both DTU and BlendedMVS consist of fairly small scenes. One of the stated benefits of volumetric representations is the faster convergence speed, especially for large scenes with many cameras. It would improve the case for the proposed method to evaluate it on larger scene datasets such as ScanNet (Dai et al., 2017). That would provide stronger support to differentiate it from methods like Instant-NGP (Muller et al., 2021) or NerfFusion (Zhang et al., 2022).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "_mATVF3naH", "reviewer": "Reviewer_VHoF", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper deals with the problem of fast and accurate surface reconstruction by combining the explicit volumetric based representation with the implicit function. They further improve acurracy of surface geometry reconstruction by 1) enforcing color-geometry dependency with a dual color network, 2) including hierachical geometry feature, 3) incorporating smoothness constraints in the SDF field. They have demonstrated superior performance on the reconstructed geometry and rendered images, and achieved relatively fast convergence as compared with baseline method on Nerf.", "summary_of_the_review": "In general, I think this paper has achieved good performance as compared with SOTA methods. My major concerns are: 1) the gemoetry rendering branch is not clear; 2) whether smoothness plays major roles in achieving good geometry reconstruction; 3) the smoothness term is a bit tricky to tune.", "strength_and_weaknesses": "Strength:\n-- First of all, this paper has addressed a very critical issue when training Nerf, which is the extremely high computational cost. They have adopted the Direct Voxel representation and the key insight is to design a dual color network to model the color-geometry dependency. They have achieved better performance on fast convergence as compared with NeuS, and more accurate geometry as compared with DVGO and the naive combination of DVGO and NeuS.\n\n\nWeakness:\n-- I have some concerns about the assumption on color-geometry dependency. In some cases or scenes, the surface texture is not correlated with the geometry/normal. For example, we could simply have a plane with printed textures. I wonder how could this kind of situation being handled by the proposed framework.\n-- After checking all the implementation details, I still didn't get the idea of how to render images just from geometry branch. It is rather unclear for this part.\n-- The smoothness term is a bit tricky. How to tune the weighting parameters to stop over-smoothing? \n-- A straightfoward implementation is to directly replace the density prediction with SDF in DVGO paper. And the authors have demonstrated this baseline in Fig.1. But I want to know what if we apply this smoothness constraints in this baseline method, will we also get very good geometry as well as fast convergence. For my understanding, it seems that the smoothness constraints have played an important role to have good reconstruction on the geometry. \n-- It would be better to also discuss limitations of the proposed method.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "4D7hozLLqkN", "reviewer": "Reviewer_8Ao6", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces voxurf - a method for surface reconstruction that combines explicit voxel representations with SDF-based volume rendering. On top of naive combination of existing techniques, the key ideas are: employing a coarse-to-fine strategy, such that coarse geometry is estimated first; a dual color network, which is supposed to encourage better consistency between appearance and geometry, adding \"hierarchical features\" and smoothness priors, which are meant to improve propagation of information across different voxels and make optimization easier. Quantitative results are convincing, and method seems to provide a significant speed up over (some) competitive. \n", "summary_of_the_review": "This work presents a well-performing and efficient method for neural surface reconstruction, but it is not fully clear if it contains enough non-trivial ideas. I still believe that the overall method is interesting, and could be valuable to the community. Thus I am leaning more towards accept.\n---\nAfter going through author's rebuttal, and looking at the additional ablation studies they pointed out, I am recommending accepting this work to ICLR. ", "strength_and_weaknesses": "Strength:\n+ The overall approach is reasonable - it makes sense that one needs more capacity and some notion of locality to optimize \n+ Quantitative results are pretty convincing: the method seems to be outperforming the strong baselines methods it builds upon - NeuS and DVGO.\n+ Authors claim 20x improvement over fully implicit representation - which is not surprising (also see quality/misc below) but still impressive. \n\nWeaknesses:\n- The method does \"just\" combine NeuS objective with explicit voxel-based representation from DVGO, plus some additional (primarily smoothness and consistency promoting) inductive biases and priors in architecture and optimization process. \n- The effects of smoothness priors are not thoroughly evaluated - in particular it would be important to know if using naive combination of DVGO+NeuS and smoothness already leads to good results.\n- (minor) Qualitative results on NVS are quite similar to DVGO (it is worth looking at the supplementary material). In fact, I find on some of the examples DVGO has better specularities (see Scan 114) .\n- (minor) Similarly, geometry results seem to be very similar to NeuS, in some cases (it seems) NeuS providing less noisy reconstructions: snowman's nose on Scan 69, can on Scan 97.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%