HIVE: HIerarchical Volume Encoding for Neural Implicit Surface Reconstruction
Authors: Xiaodong Gu
Weihao Yuan
Heng Li
Zilong Dong
Ping Tan
Venue: Submitted to ICLR 2023 (2023)
Keywords: neural implicit surface reconstruction
multi-view surface reconstruction
hierarchical volume encoding

Abstract: Neural implicit surface reconstruction has become a new trend in reconstructing a detailed 3D shape from images. In previous methods, however, the 3D scene is only encoded by the MLPs which do not have an explicit 3D structure. To better represent 3D shapes, we introduce a volume encoding to explicitly encode the spatial information. We further design hierarchical volumes to encode the scene structures in multiple scales. The high-resolution volumes capture the high-frequency geometry details since spatially varying features could be learned from different 3D points, while the low-resolution volumes enforce the spatial consistency to keep the shape smooth since adjacent locations possess the same low-resolution feature. In addition, we adopt a sparse structure to reduce the memory consumption at high-resolution volumes, and two regularization terms to enhance results smoothness. This hierarchical volume encoding could be appended to any implicit surface reconstruction method as a plug-and-play module, and can generate a smooth and clean reconstruction with more details. Superior performance is demonstrated in DTU, EPFL, and BlendedMVS datasets with significant improvement on the standard metrics. The code of our method will be made public.

Under review as a conference paper at ICLR 2023
HIVE: HIERARCHICAL VOLUME ENCODING FOR NEURAL IMPLICIT SURFACE RECONSTRUCTION
Anonymous authors Paper under double-blind review
ABSTRACT
Neural implicit surface reconstruction has become a new trend in reconstructing detailed 3D shape from images. In previous methods, however, the 3D scene is only encoded by the MLPs which do not have an explicit 3D structure. To better represent 3D shapes, we introduce volume encoding explicitly encode the spatial information. We further design hierarchical volumes to encode the scene structures in multiple scales. The high-resolution volumes capture the high- frequency geometry details since spatially varying features could be learned from different 3D points, while the low-resolution volumes enforce the spatial consis- tency to keep the shape smooth since adjacent locations possess the same low- resolution feature. In addition, consumption at high-resolution volumes, and two regularization terms to enhance results smoothness. This hierarchical volume encoding could be appended to any implicit surface reconstruction method plug blug-and-un -ad---dd-pa play module, and can gen- erate a smooth and clean reconstruction with more details. Superior performance is demonstrated in DTU, EPFL, and BlendedMVS datasets with significant im- provement on the standard metrics. The code of our method will be made public.
adopt a sparse structure reduce the memory
INTRODUCTION
Surface reconstruction, image-based modeling (Tan 2021). from multi-view images is heavily studied classic task in computer vision. Given multiple images from different views of an object as well as the corresponding camera parameters, this task aims recover the accurate 3D surface of the target object. Traditional methods (Fuhrmann et al.) (014)) usually solve a depth map for each input image and then fuse (Kazhdan et al. (2066) those depth images build a complete surface model. After the arising of deep networks, many methods try to exploit the neural networks to directly learn the mapping from 2D images 3D surfaces (Murez et al. 2020 Sun et al. 2021). These learning- based methods skip the intermediate depth map estimation and are highly efficient even on unseen objects and scenes. But they usually only recover coarse scale geometry and miss most of the high frequency surface details.
Recently, many methods Yariv et al. 2020 Wang et al. 2021 Yariy et al. 2021 Darmon al. 2022) have achieved highly accurate reconstruction results based on neural implicit surface. These methods represent the shape with an implicit function, such as occupancy (Oechsle et al. 2021)
100
Image
NeuS
NeuS+Ours
Figure I: Visualization of normal maps to highlight our advantages in recovering shape details.

---

Under review as a conference paper at ICLR 2023
or signed distance fields (SDF) WWang et al. 2021 and then leverage the neural radiance field (NeRF) (Mildenhall et al. 2020) to render the implicit geometry into color images. Thus, the differ- ence between the rendered images and the input images could optimize the neural radiance field as well as the implicit geometry. Strong results have been demonstrated. However, in these methods, the 3D shape is implicitly encoded in the multi-layer perceptrons (MIPss.. Although ML.ss are com- pact and memory efficient, they do not have an explicit 3D structure, which may cause difficulties in optimizing the target 3D shape as is observed in mesh and point-cloud processing (Chibane al.) 2020 Peng et al. 2020). Furthermore, it is also known (Sun et al.) (02T) that compact MLPs are hard to encode all the geometry
geometryy details, such that the recovered surface prone to be over-smooth.
To solve this problem, some methods employ feature volumes or feature hash tables to facilitate ML.Ps to encode the 3D space Fridoviich-Kil et al. (2022); p: Sun et al.] (2022a); Chen et al. (2022);
Müller et al. (202)), which could directly encode the geometry of each 31D position faithfully and unambiguously However, there are some problems
existing methods. For the volume encoding
methods Fridovich- Keil et al. (2022): Sun et al. (2022a); Chen et al. (2022). there is usually only one high-resolution volume in their frameworks, in which case each voxel is updated in isolation.
Due to the high degree of freedom in optimizing one voxel, it is hard to maintain globally coherent shape, as is shown in Figure 2 (c). For the hash-table encoding methods [uüler et al.) (2022). there are hash collisions which could cause some geometry defects, is shown in Figure (a)
(b). To address these, we introduce a hierarchical volume encoding to encode the 3D space. This hierarchical structure has three advantages. First, the high-resolution feature volume helps to reason geometry details in the corresponding locations. Secondly, the voxels in the low- resolution volumes contain the information of large space, which could enforce spatial consistency to keep the shape smooth. Thirdly, this hierarchical structure allows us to use low-dimensional features in the high-resolution volume, which helps reduce memory consumption. To further reduce memory consumption, we sparsify high-resolution volumes with the preliminary surface reconstruction computed from low-resolution volumes. We only keep voxels nearby the surface of the preliminary results. Finally, we design two smoothness terms to facilitate the optimization to make the reconstructed shape clean.
In the experiments, we demonstrate that by simply adding the proposed hierarchical volume encod- ing, the performance of different methods are all improved significantly. Specifically, the error of NeuS (Wang et al. 2021) is reduced by 25% from 0.84 mm to 0.63 mm, the error of VoISDF (Yariv 2021) is reduced by 23% from 0.86 mm to 0.66 mm, and the error of Neural NeuralWarp (Darmon 2022 is reduced by 10% from 0.68 mm to 0.61 mm the DTU (Jensen et al. 2014) dataset.
et al.
al.
More than that, the error of Neural Warp is reduced by 31 % on the EPFL (Strecha et al.
2008) dataset
with the "full" metric. Figure I shows an example from the DTU dataset. The color coded normal
map clearly demonstrates our method can significantly boost NeuS (Wang et al.) 2021 to recover more geometry details while keeping the surface smooth and clean.
The main contributions of this work are summarized in the following:
We propose a hierarchical volume encoding, which can significantly boost the performance of neural implicit surface reconstruction as plug-and- -layy module.
The hierarchical volume encoding is further improved by employing a sparse structure which reduces the memory consumption, and by enforcing two regularization terms that keep the recon- structed surface smooth and clean.
We demonstrate superior performance in three datasets.
2 RELATED WORK
Traditional multi-view surface reconstruction Traditional methods typically follow a long
pipeline of structure-fromt fronteeffrommmaee motionooooonnooooooo (i.e. camera calibration) (Schönberger & Frahm 2016), depth map estimation (Barnes et al. 2009), and multi-depth fusion (Kazhdan et al.] (006) to reconstruct
surface models from images. Many advanced geometric methods (Jiang et al. 2013 Lhuillerr &
Quan 2005] Furukawa & Ponce 2009 Galliani et al.) 2015); SSchönberger et al. 2016 have been developed to enhance these different steps in the last two decades.
After the arising of the deep networks, almost all steps in the conventional pipeline have been rev- olutionized, including feature extraction and matching (DeTone et al. 2018, (arlin al. 2020
2

---

Under review as a conference paper at ICLR 2023
DIEE EELEE HELEE
EIEGE
(a) Instant-NGP
(b) NeuS+Hash
(c) NeuSnelenoxels
(d) NeuS+Ours
Figure 2: Visual comparison of different encoding on DTU scan-24.
structure- froeee-ffommmeeee from-motion (Ummenhofer et al. 2017 Tang & Tan 2019), and depth map estimation
Yao al. 2018 Gu et al. 2020 Tang al. 2022).
To skip the intermediate depth map estimation, given multiple images with known camera poses, some methods try to directly regress volumetric prediction end-to-end like occupancy volume (Ji al. (017) or a TSDF volume (Murez et al.) [020] [Snn et al. [202T]. While these methods simplify the overall pipeline and can be generalized to unseen objects and scenes, they often only learn to recover coarse scale geometry and miss many surface details.
Implicit surface reconstruction neural radiance field (NeRF) encodes 3D scene (Mildenhall itt al. (020) implicitly in a neural network. The network is optimized to match the rendered images to the input images such that it can generate high quality novel view synthesis results. However, since there is no constraint imposed on the 3D geometry, the surface extracted from the implicit network usually has some defects due to tuning of the density threshold (Oechsle et al., 2021). To address this problem, some methods first represent the 3D shape with an implicit geometry network, like occupancy grid (Niemeyer et al. 2020; Oechsle et al. [021] 1) or signed distance fields Yariiv let al. 2020 Wang et al. [2021] Yariv et al.) 2021; Darmon et al. 2022 [unn et al.) 2022b Long jet al. 2022 Fu et al. 2022] Yu et al. (022). and then transfer the output of the geometry network to density function, with which the radiance network could render color images. In this way, the radiance network as well as the geometry network can be optimized together by matching the rendered and input images. In these methods, the 3D geometry is directly encoded in the MLPs without any explicit 3D spatial information. To facilitate MLPs, some methods propose to encode the 3D space with single geometric volume -Fridovich-Kee Keil al. (2022); Sun et al. (2022a); Chen et al. (022); Martel et al. (202T) or hash tables Müller et al. (2022). However, single-volume may cause noise due to the high degree of freedom in optimizing one voxel, and hash tables may
cause hash collisions which leads to defects, as displayed in Figure 2 In this paper, we introduce
hierarchical volume encoding to explicitly encode the 3D spatial information, thus our method can reconstruct more surface details while keeping the shape globally coherent. A similar structure is proposed in TTakikawa et al.] (202T) for the SDF encoding task, which sums up the features from a lurge-feature-chene octree, while we concatenate the features from multiple small-ceature-chamdd volumes, in which case our method consumes less memory.
3 METHODS
3.1
OVERVIEW
The overall framework our method is illustrated in Figure 3 Given multiple images [ of an object and corresponding camera poses. the task is to reconstruct the 3D surface of this object. The 3D shape is represented by an implicit SDF network, with which another implicit color network render images using the neural volume rendering. To enhance the representation ability of the spatial information, we add a bierarchical volume encoding as the input of the SDF network, which can embed the 3D space in multiple scales. To save the memory consumption, we also introduce sparse structure for high-resolution volumes. Finally, two implicit networks as well as the volume encoding are optimized by minimizing the difference between the rendered images and the input images as well as minimizing two regularization terms.
3.2 IMPLICIT SURFACE RECONSTRUCTION BASED ON VOLUME RENDERING
Neural implicit volume rendering is first introduced in Miilenhall al.) 2020 for novel view rendering and then used in (Wang et al. 2021 Yariv et al.) [221] Darmon et al. 2022 for surface
reconstruction. Our method could be added to any of them for better reconstruction. Here take
3

---

Under review as a conference paper at ICLR 2023
Hisrarhicaal Volume
Cat fedl
Neural Impplel Surfices
Deasly
MEP
Feature Velume 256
Cake
Spanity
Neural Impleel Surfacw
Deasky
Sparse 512
MET
SDF
513
Spanity
Neural Implieit Surfaces
Deskky
Spanie 1934
1824
SDF
Color
Shand Weights
Figure 3: Method overview. In the first stage, compute initial result use features from volumes with resolution from to 256. In a later stage, we finalize the result use features from sparsified high resolution volumes with a resolution of 512 or 1024.
NeuS (Wang et al. 2021) as an example, which uses two multi-layer perceptrons (MLPs) to serve as two functions for representing an object: one is the SDF network sdf R + R that maps spatial position x E Rs to its SDF value of the object surface, and the other one is the color network e: IR3 S + that maps the spatial point x and a viewing direction v E S² to a color value. The surface S is then represented by the zero-level set of the SDF function as:
= {x E I Isdf (x) = =}.
(I)
To optimize the SDF network, images are rendered from the color network well as weighting function computed from the SDF network. The volume rendering function for generating colors is calculated
c(o.v)= w(t)c(p(t), v)dt,
(2)
where w is the weighting function computed by the density from the SDF network.
To make the weighting function unbiased and coclusion-aware it is calculated
(t) (()),()),
(3)
where
T(t)= - fo p(u)du
(4)
and
(sdf (p(1))
p(t) max
(sdf 0), 2.(x) = (11)= am
(5)
3.3 HIERARCHICAL VOLUME ENCODING
Previous methods usually encode all the information of a 3D object in the MLPs. To assist the MLPs, we build 3D feature volume as the input of the MLPs, which explicitly encodes the 3D spatial information. This feature volume can naturally encode the knowledge about the 3D space of the object, while being optimized as well as the MLPs from the rendering loss.
To enhance the representation ability of the encoding. we also employ a hierarchical mechanism, where different scales of the 3D feature volumes are adopted, as is shown in Figure 4 In the experiments, we find combination of multi-scale volumes works better than a single large-width volume. This is reasonable since in low-resolution volumes, one voxel represent a large space, such that this space can have the same code. The same code could smooth this space and prevent crushing 3D shape, which is important in surface reconstruction.
In a basic setting, we employ eight feature volumes for the 3D space encoding, whose resolutions increase from 2 x 2 x x 4 to 256 256 256 x 4. When a point x is being rendered, eight features of the corresponding position from these eight volumes are trilinearly interpolated and concatenated to form F(x), which works as the input of the MIPPs. Therefore, the SDF function becomes
S {x E 3 sdf (F(x)) 0).
4

---

Under review as a conference paper at ICLR 2023
Single volume eucoding
Hlerarchical volume encoding
Feature
1x Pb
H
W
P
q
q
Figure 4: A 2D toy example of the hierarchical volume encoding. Left: single high resolution volume. Features with 3 channels used to encode two locations and Note it only captures spatial variant features. Right: hierarchical volume with lower dimensionality. Features have just channel and the memory consumption is much less. The high resolution volume encodes spatial variant features, while low resolution volume enforces spatial smoothness.
3.4 SPARSE HIGH RESOLUTION VOLUME
is evident that the higher resolution of the volume is, the more details would be recovered, a volume larger than 256 256 x 256 would consume much more memory. To solve this problem, a multi-stage optimization scheme is adopted with high-resolution sparse volumes. In general, a lot of voxels in a dense volume unoccupied and invalid. Thus, we could save memory consumption a lot by using the shape reconstructed from the coarse volumes to cull voxels far from the surface in the high- resolution volumes.
Centruc bedec
Figure 5: Sparse high-resolution volume. The index of
-1 fetches the last embedding (in dark gray) in T..
To do this, we propose a multi-stage optimization framework. In the first stage, we use the above- mentioned basic structure, where resolution of the largest volume is 256. After the first stage, we obtain coarse surface reconstruction S.. In the second stage, we utilize S. to cull unnecessary voxels to obtain the valid voxels Vealid the high-resolution volume. Specifically, we first dilate the surface S, to ensure all the valid voxels nearing the surface are included. Therefore, we only need to optimize the embeddings of n valid voxels Vealid instead of all voxels. Here, we use a simple data structure, embedding table Te, to store these embeddings.
Given a floating-Po point three dincessiinniin position x, we obtain the surrounding eight integer coordi- nates through the rounding operation, take out the embedding of these integer points, and then fuse them through trilinear interpolation to serve as the encoding of x. In order to efficiently extract the corresponding embedding from the embedding table Tg, construct another index table T, which stores the indexes of Te. The values in table T; are all initialized to -1, which corresponds to the last embedding in table T.. The length of the embedding table T. is only n (n N such that the memory consumption is reduced lot due to the sparse structure of the high-resolution volumes. More details are given in the appendix.
3.5 LOSS FUNCTION
We equip three previous methods (Wang et al. 2021 Yariv et al. 2022 Darmon et al. 2022) with our hierarchical volume encoding. To optimize the model, we the losses in their work without changes, i.e. a rendering loss Lcotor which minimizes the difference between the rendered colors and input colors, and Eikonal loss Lesk (Gropp et al. 2020) which encourages unit norm of the SDF function gradients. Besides, for NeuralWarp (Darmon et al. 2022). an additional warping color loss Lwarp adopted, which warps views on each other to enforce multi-view photo-consistency.
5

---

Under review as a conference paper at ICLR 2023
Method
24
37
40
55
63
65
69
83
97
105
106
110
114
118
122
Mean
DR et 2020
1.63
1.87
0.63
0.48
1.04
0.79
0.77
1.33
1.16
0.76
0.67
0.90
0.42
0.51
0.53
0.90
MVSD Pham a 2011
0.83
1.76
0.86
0.44
1.11
0.90
0.75
1.26
1.02
1.35
0.87
0.84
0.34
0.47
0.46
0.88
COLMAPIS COLMAPPSAoOIn
2016
NeKF Middeaalll al.
X00
1.90
1.80
1.47
1.07
2.33
1.06
1.15
1.49
UNISCKI
1.32
0.50
1.47
NeuS Wank 21 2021
1.00
1.37
0.93
0.43
1.10
0.65
0.57
1.48
1.09
0.63
0.52
1:20
0.35
0.49
0.54
0.84
VotS Tary de 2041
1.14
1.26
0.81
0.49
1.25
0.70
0.72
1.29
1.18
0.70
0.80
1.U5
0.42
0.61
0.35
0.56
NeuralWarppann
2022
0.49
0.71
0.81
1.20
0.06
0.74
0.41
0.63
0.51
0.68
NeuS-
1.26
1.67
0.84
0.39
1.19
0.78
0.64
1.41
1.24
0.72
0.58
1.49
0.34
0.49
0.52
0.90
NeuS-Ours
OuSDFFOurs NeuralWarp4Ours
0.51
0.77
0.38
0.75
0.67
0.71
0.57
0.89
0.38
0.52
0.44
0.69
0.36
0.38
0.63
0.64
0.69
1.13
0.99
0.68
0.51
0.81
0.36
0.53
0.52
0.61
Table 1: Quantitative results on the DTU dataset.
In addition, we add two additional regularization terms Ltv and Lnormal make the reconstructed surfaces smooth and clean. The total variation (TV) (Rudin & Osher (994) regularization is applied to each embedding volume to make adjacent voxels have similar characteristics, in which case the geometry could be more continuous and compact. It is computed as:
C.- --= -E (VHLLAA-- Vi,aa)*+ + (V V.,xa² + (Vii,t+I - Visn)3 WT L.J.k
where is the number of the hierarchical volumes.
Another regularization term Lnormat is smoothness constraint for normal. For each pixel of the image, we calculate the accumulated normal gradients along the marching ray as
N Ng v) Jo
(8)
where (Igra(()) is the gradient the normal at p()) and computed as
ngrrad (1) (ddf((F(p(())
The normal regularization Lnormat then defined as:
Enormal 1
EllNgradlee
(10)
Npsz piz
where Npir is the number of pixels in optimization.
Thus, the final loss C is defined as:
L - Cori + Nto Ltv + Anoemal * Lnormal,
(11)
where Cori is the original loss of each method, Aet and Nnormat weighting factors.
4 EXPERIMENTS
4.1 IMPLEMENTATION DETAILS
This work is implemented in Pytorch and experimented on Nvidia 2080Ti GPUs. The Adam optimizer (0.9, 0.999) used update the network weights. The learning rate for the MLPs is set to 5e 4 and decreased to 1/20, while the learning rates for the volumes with resolu- tions 2, 4, 8, 16, 32, 64, 128, 256, 512, 1024 are set to 0.01, 0.01, 0.01, 0.01, 0.01, 0.001, 0.001, 1e-1, 1e 1e and decreased to 1/100. We adopt three- stage training strategy by default, and the number of optimization iterations set to the same as the original framework. Taking "NeuS Ours" as example, there 300K iterations in total. In the first stage, eight volumes of resolution from [23, 4] to |2563, 4] are employed for 80K-iteration optimization. In the second stage, a sparse volume with the resolution of [5123 4) is added for another C0K-iteration optimization. Finally, sparse volume of [10243, 4] is appended for the remaining 200K-iton ittertioo optimization. The length of the index tables for the second and the third stages to 2563.

---

Under review as a conference paper at ICLR 2023
DTU
LERR DILDJ
FELLE CEBERE BEEEE
DTU
BlendedM
BlendedMVVS
BlendedMVS
EPFL
Image
Neural
NeuS
NeuS+Ours
Figure Visual comparison of the reconstructed meshes.
4.2 EVALUATION
For fair comparison, follow previous methods evaluate our method DTU (Jensen al., 2014) EPFL (Strecha et al., (2008) and BlendedMVS (Yao et al. 2020) benchmark. The Chamfer LI distance is used for evaluating the accuracy of the recovered surfaces. This metric is the average of the accuracy, which measures the distance from the reconstructed surface to the ground-truth surface, and the completeness, which measures the distance in reverse.
We first evaluate our method on the DTU dataset and report the quantitative results in Table I For a fair comparison, the meshes all methods are extracted by marching cubes with resolution of 512. We follow previous work (Darmon et al.] 2022. Oechsle et al., 2021) to clean the predicted mesh by visibility masks for more reasonable evaluation. As shown Table [ the accuracy of previous methods is improved significantly by adding the hierarchical volume encoding. To be specific, the error of NeuS (Wang et al. 2021 is reduced by 25%. from 0.84 to 0.63, while the error of Weur (Darmon et al. 2022 is reduced by 10%. from 0.68 to 0.61. The reconstructed meshes are shown in Figure 6) from where we can see that the surfaces of our method are smooth and clean, while containing accurate geometry details. For instance, the details of the house in the
7

---

Under review as a conference paper at ICLR 2023
(b)
Figure 7: Ablation study. (a) Sparse high-res volume. The results of without and with sparse high-resolution volumes displayed in the first and second rows, respectively. The top row shows the normal maps and the bottom row shows the reconstructed meshes. (b) Regularization terms. The color image. results of without and with regularization terms are displayed. (c) HauS+Hash vs. NeuS+Ours.
first scene in Figure are recovered, especially the shape of the windows and the bricks of the roof, which is more remarkable in the normal map in Figure
We then evaluate our method on the EPFL dataset. For fair comparison, we follow Neu- ralWarp to use both the "full" chamfer distance and the "center" chamfer distance to evaluate the reconstructed surfaces. The "center" metric only evaluates the center of the scene cropped by a manually defined box, which focuses more on the precision of the reconstruction, while the "full" metric is also influenced by the ground plane and the rarely seen points, which thus also considers the completeness of the recon- struction. As shown Table 2 the performance of each framework is improved significantly by adding our method. Especially, the "full" metric of NeuralWarp is decreased by 31% to 5.72, which is also a 21% improvement compared to COLMAP (Schönberger et al. 2016).
Method
Fountain-P1I Full
Herzjesu-P7 Full
Mean Center
Center
Center
Full
COLMAP UNISURF
6.47
2.45
7.95
2.31
7.21
2.38
26.16
27.22
MVSDF VoISDF
6.87
2.26
11.32
2.72
9.10
2.49
12.89
2.99
13.61
4.58
13.25
3.78
NeuS
7.64
2.25
19.03
14.02
13.33
7.79
NeuralWarp
7.77
1.92
8.88
2.03
8.32
1.97
NeuS-Ours
7.39
1.76
11.23
3.69
9.31
2.73
NemnlWarp-Ous
3.31
1.73
8.13
1.71
5.72
1.72
Table 2: Quantitative results on the EPFL dataset.
Finally, the evaluation is performed the BlendedMVS dataset, which contains objects of more complex shapes. As shown Figure 6, our method can recover more detailed shapes than previous methods. These illustrate that hierarchical volume encoding can facilitate MLPs to encode more complex shapes and improve the neural implicit surface reconstruction.
4.3 ABLATION STUDY
We choose "NeuS + Ours" to perform extensive ablation studies to validate the proposed method. All the results are reported from the 15 scans of DTU dataset unless otherwise specified.
study the effectiveness the hierarchical volume encoding. equip the basic set of eight volumes to NeuS (Wang et al. (2021) without any other change. As shown in Table the error of the recovered surfaces is significantly reduced from 0.84 to 0.70.
Hierarchical volume encoding.
Regularization terms. We also ablate the two smoothness terms study their effectiveness. As shown in Table 3 they help to further reduce the error metric down to 0.66. Figure (b) shows visual comparison without (middle) and with (right) the regularization terms. The reconstructed mesh is smoother and more complete with the regularization terms.
Sparse high resolution volume. We analyze the effect of the resolution of volumes. As shown in Table 3 sparse high resolution embedding volume (resolution of 1024 here) further improves the accuracy to 0.63. We also visualize the reconstructed surfaces without (left) and with (right) sparse
8

---

Under review as a conference paper at ICLR 2023
1.4
1.33
Mean dh distaace
1.01
2.4
chamter
6.4
0.77
0.38
0.73
0.7
0.49
Number of the Vetames
Resolation of the smallest volume
Figure Ablation study about the number of the volumes and the resolution combination of volumes.
high resolution embedding Figure a
clear that the high-resolution embedding generates more shape details, c.g.. clearer facial shapes.
Setup of embedding volumes. To further inspect the ef- fect of different combinations of the volume encoding, we perform two experiments on the volume setups. (1) We study the number of the volumes. We first compare our hierarchical setup to a large volume of 256 32, the number of whose learnable parameters is about 8 times larger than ours. However, its performance is much worse than ours, as presented in the left of Figure 8 Then we
Hierarchical
Regular Terms
Sparse High-Res
Mean
Volume
X
X
0.84
/
0.70
/
X
0.66
/
0.63
Table Ahlation study the DTU dataset. keep the number of feature channels fixed at 32 and increase the volume number. The resolution of the first volume is set 256, and the resolution of the remaining volumes decreased by 1/2 one by one. As shown in Figure X left, the accuracy of the reconstructed surface is gradually improved with the arising of the volume number. (2) Next, we study the impact of the low-resolution volume. We keep the resolution of the largest volume fixed at 256, the number of the volumes fixed 8, and vary the resolution sequentially enlarged, and the enlargement factor is (256/ min 1/8, From the results shown in Figure right, we can see the performance is decreased when the low-resolution volume begins from a larger resolution. This is intuitive because the low-resolution volume enforces spatial smoothness. Starting from lower resolution helps to enforce smoothness across larger areas.
smallest volume from to 64. The resolutions of intermediate volumes are
Layer number MLPs. Finally, we experiment with layer number of MLPs in the SDF network to inspect its effect. When only half of the layers used, the chamfer distance increases to 0.71. More details are included in the appendix.
4.4 COMPARISON TO HASH ENCODING
A similar hierarchical feature encoding is adopted in Instant-NGP (üüller et al. 2022), but the features are stored in hash tables such that the hash collision is inevitable. Although this is efficient in both memory and convergence speed, the hash collision may result in defects in the implicit geometry. To make comparison to hash encoding, we perform two experiments: one is the original version of Instant-NGP, and the other one is adding the hash encoding NeuS framework. Some of the results are presented in Figure (c) and Table [ For more details and results please see Hash+NeuS could obtain accurate novel view synthesis, there are some defects in their surface. However, the convergence speed of Instant-NGP is faster than method, which is due to the fact that still employ the large MLPs used in NeuS.
the appendix. From the results, we can that although Instant-NGP
5 CONCLUSION
We propose explicitly encode the 3D shape surface by hierarchical volumes to facilitate ML.s in neural implicit surface reconstruction methods. The spatially varying features can be obtained from high resolution volume to reason more details for each query 3D point, while the feature from low- resolution volumes could reason spatial consistency to keep shapes smooth. We also design sparse structure to reduce the memory consumption high-resolution volumes, and two regularization terms to enhance surface smoothness. Our hierarchical volume encoding could be appended to any implicit surface reconstruction method as a plug-and and-play module to significantly boost their performance, which is demonstrated in three datasets.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Connelly Barnes, Eli Shechtman, Adam Finkelstein, and Dan B Goldman. Patchmatch: A random- ized correspondence algorithm for structural image editing. ACM Transactions on Graphics, 28 (3):24, 2009.
Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. Tensorf: Tensorial radiance fields. In Proceedings of the European Conference on Computer Vision, 2022.
Julian Chibane, Thiemo Alldieck, and Gerard Pons-Moll. Implicit functions in feature space for 3d shape reconstruction and completion. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2020.
François Darmon, Bénédicte Bascle, cean-Clément Devaux, Pascal Monasse, and Mathicu Aubry. Improving neural implicit surfaces geometry with patch warping. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6260-6269, 2022.
Daniel DeToone, Tomasz Malisiewicz, and Andrew Rabinovich. Superpoint: Self-supervised interest. point detection and description. In CVPR, pp. 224-236, 2018.
Sara Fridovich-Keil. Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels: Radiance fields without neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5501-5510, 2022.
Qiancheng Fu, Qingshan Xu, Yew-Soon Ong, and Wenbing Tao. Geo-neus: Geometry-consister neural implicit surfaces learning for multi-view reconstruction. arXiv preprint arXiv:2205.1555 15848, 2022.
Simon Fuhrmann, Fabian Langguth, and M. Goesele. Mve multi-view reconstruction environ- ment. In Proceedings of the Eurographics Workshop Graphics and Cultural Heritage, 2014.
Yasutaka Furukawa and Jean Ponce. Accurate, dense, and robust multiview stereopsis. IFEE Trans- actions Pattern Analysis and Machine Intelligence, 32(8):1362-1377 2009.
Silvano Galliani, Katrin Lasinger, and Konrad Schindler. Massively parallel multiview stereopsis by surface normal diffusion. In Proceedings of the International Conference on Computer Vision, pp. 873-881, 2015.
Amos Gropp, Lior Yariv, Niv Haim, Matan Atzmon, and Yaron Lipman. Implicit geometric reg- ularization for learning shapes. In Proceedings of Machine Learning and Systems 2020, pp- 3569-3579. 2020.
Xiaodong Gu, Zhiwen Fan, Siyu Zhu, Zuozhuo Dai, Feitong Tan, and Ping Tan. Cascade cost volume for high-resolution multi-view stereo and stereo matching. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 2495-2504, 2020.
Rasmus Jensen, Anders Dahl, George Vogiatzis, Engil Tola, and Henrik Aanæs. Large scale multi- view stereopsis evaluation. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 406-413. IEFE, 2014.
Mengqi Ji, Juergen Gall, Haitian Zheng, Yebin Liu, and Lu Fang. Surfacenet: An end-to-end 3d neural network for multiview stereopsis. In Proceedings of the International Conference on Com- puter Vision, pp. 2307-2315, 2017.
Nianjuan Jiang, Zhaopeng Cui, and Ping Tan. A global linear method for camera pose registration. In Proceedings of the International Conference on Computer Vision, 2013.
Michael Kazhdan, Matthew Bolitho, and Hugues Hoppe. Poisson surface reconstruction. In Pro- ceedings of the fourth Eurographics symposium Geometry processing, volume 7, 2006.
Maxime Lhuillier and Long Quan. A quasi-dense approach surface reconstruction from uncali- brated images. IEEE Transactions on Pattern Analysis and Machine Intelligence, 27:418 433, 2005.
10

---

Under review as a conference paper at ICLR 2023
Xiaoxiao Long, Cheng Lin, Peng Wang, Taku Komura, and Wenping Wang. Sparseneus: Fast generalizable neural surface reconstruction from sparse views. arXiv preprint arXiv:2206.05737, 2022.
Julien NP Martel, David Lindell, Connor Z Lin, Eric R Chan, Marco Monteiro, and Gordon Wetzstein. Acorn: Adaptive coordinate networks for neural scene representation. arXiv preprint arXiv:2105.02788, 2021.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In Proceedings of the European Conference on Computer Vision, pp. 405-421, 2020.
Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics prim- itives with multiresolution hash encoding. ACM Trans. Graph. 41(4):102:1-102:15: July 2022. doi: 10.114555522233533177.. URL httpp::/d6....rrg110.14445/ http./o/rrg00111//2222.2. 3530127
Zak Murez, Tarrence van As, James Bartolozzi, Ayan Sinha, Vijay Badrinarayanan, and Andrew Rabinovich. Atlas: End-to-end 3d scene reconstruction from posed images. In Proceedings of the European Conference on Computer Vision, pp. 411 Springer, 2020.
Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger. Differentiable volumet- ric rendering: Learning implicit 3d representations without 3d supervision. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 3504-3515, 2020.
Michael Oechsle, Songyou Peng, and Andreas Geiger. Unisurf: Unifying neural implicit surfaces and radiance fields for multi-view reconstruction. In Proceedings of the International Conference on Computer Vision, pp. 5589-5599, 2021.
Songyou Peng, Michael Niemeyer, Lars Mescheder, Mare Pollefeys, and Andreas Geiger. Convo- lutional occupancy networks. In Proceedings of the European Conference on Computer Vision, 2020.
Leonid Rudin and Stanley Osher. Total variation based image restoration with free local constraints. In Proceedings of 1tt international conference image processing, volume 1, pp. 31-35. IEEE, 1994.
Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. Superglue: Learning feature matching with graph neural networks. In CVPR, pp. 4938-4947, 2020.
Johannes L Schönberger, Enliang Zheng, Jan-Michael Frahm, and Marc Pollefeys. Pixelwise view selection for unstructured multi-view stereo. In Proceedings of the European Conference on Computer Vision, pp. 501-518. Springer, 2016.
Johannes Lutz Schönberger and Jan-Michael Frahm. Structure- frreeefroommeeee motion revisited. In Confer- Computer Vision and Pattern Recognition (CVPR), 2016.
Christoph Strecha, Wolfgang Von Hansen, Luc Van Gool, Pascal Fua, and Ulrich Thoennessen. On benchmarking camera calibration and multi-view stereo for high resolution imagery. In Proceed- ings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 1-8. leee, 2008.
Cheng Sun, Min Sun, and Awann-Tzone Chen. Direct voxel grid optimization: Super-fast vergence for radiance fields reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5459-5469, 2022a.
Jiaming Sun, Yiming Xie, Linghao Chen, Xiaowei Zhou, and Hujun Bao. Neuralrecon: Real-time coherent 3d reconstruction from monocular video. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 15598-15607, 2021.
Jiaming Sun, Xi Chen, Qianqian Wang, Zhenggi Li, Hadar Averbuch-Elor. Xiaowei Zhou, and Noah Snavely. Neural 3d reconstruction in the wild. AMACM SIGGRAPH 2022 Conference Proceedings, pp. 1-9, 2022b.
11

---

Under review as a conference paper at ICLR 2023
Towaki Takikawa, Joey Litalien, Kangxue Yin, Karsten Kreis, Charles Loop, Derek Nowrouzezahrai, Alec Jacobson, Morgan McGuire, and Sanja Fidler. Neural geometric level of detail: Real-time rendering with implicit 3d shapes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11358-11361 2021.
Ping Tan. Image-based modeling. In Computer Vision: A Reference Guide, pp. 653-656. Springer, 2021.
Chengzhou Tang and Ping Tan. Ba-net: Dense bundle adjustment network. ICLR, 2019.
Shitao Tang. Jiahui Zhang, Siyu Zhu, and Ping Tan. Quadtree attention for vision transformers.
ICLR, 2022.
Benjamin Ummenhofer, Huizhong Zhou, Jonas Uhrig, Nikolaus Mayer, Eddy Tlg, Alexey Dosovit- skiy, and Thomas Brox. Demon: Depth and motion network for learning monocular stereo. In CVPR. pp. 5038-5047, 2017.
Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. Advances in Neural Information Processing Systems, 34:27171-27182 2021.
Yao Yao, Zixin Luo, Shiwei Li, Tian Fang, and Long Quan. Mvsnet: Depth inference for unstruc- tured multi-view stereo. In Proceedings of the European Conference Computer Vision, pp- 767-783, 2018.
Yao Yao, Zixin Luo, Shiwei Li, Jingyang Zhang, Yufan Ren, Lei Zhou, Tian Fang, and Long Quan. Blendedmvs: large-scale dataset for generalized multi-view stereo networks. In Proceedings of the IEEE Conference on Computer Vision and Patter Recognition, pp. 1790-1799, 2020.
Lior Yariv, Yoni Kasten, Dror Moran, Meirav Galun, Matan Atzmon, Basri Ronen, and Yaron Lip- man. Multiview neural surface reconstruction by disentangling geometry and appearance. Ad- vances in Neural Information Processing Systems, 33:2492 21-2422-5502, 2020.
Lior Yariv, Jiatao Gu, Yoni Kasten, and Yaron Lipman. Volume rendering of neural implicit surfaces. Advances in Neural Information Processing Systems, 34:4805-4815, 2021.
Zehao Yu, Songyou Peng, Michael Niemeyer, Torsten Sattler, and Andreas Geiger. Monosdf: Exploring monocular geometric cues for neural implicit surface reconstruction. arXiv preprint arXiv:2206. 206626006.0065 2022.
Jingyang Zhang, Yao Yao, and Long Quan. Learning signed distance field for multi-view surface reconstruction. In Proceedings of the International Conference on Computer Vision, pp. 6525- 6534, 2021.
12

---

Under review as a conference paper at ICLR 2023
A APPENDIX
A.1 DATASETS
DTU (Jensen et al. 2014 is well-used dataset for multi-view reconstruction, consisting of 124 scans of various objects. The images are obtained from an RGB camera and a structured light industrial robot arm. Each scene is captured from 49 or 64 views with a resolution of 1600 x 1200. For a fair comparison, we follow previous methods and select 15 scenes from DTU for evaluation. BlendedMVS (Yao et al. 2020) is another dataset for multi-view reconstruction composed of 113 scenes including architectures, sculptures, and small objects. Each scene consists of dozens hundreds images with resolution of 768 576. EPFL (Strecha al., 2008) is small dataset composed of two outdoor scenes, Fountain and Herzjesu, which contain I1 and high-resolution images, respectively, and the accurate ground truth meshes.
scanner mounted
A.2 DETAILS OF SPARSE VOLUME STORING
We look up the values of Vualid from table T. through T. Specifically, we first define mapping function f:
f(x)= = x + y N
(12)
where N is the resolution of the volume, and 2;, y: 2 the coordinates of the position x. We utilize f to map each voxel vj in Vealid to the index in table T; and then T; further converts it to the index j in Te:
(x)( j, j = I...n
(13)
where n is the number of valid voxels Vealid The length of the embedding table Te is only n + (n N such that the memory consumption is reduced lot due to the sparse structure of the high-resolution volumes.
A.3 LIMITATIONS
(I) Our framework is memory bounded and has difficulties in reconstructing large-scale scenes. This is also common problem of volume-based methods. (2) The voxels pruned at low resolution cannot recovered at high resolution. To address this, we dilate the low-resolution surface, such that usually all the useful voxels would be included. But there is still chance that the voxels would be pruned by mistake.
A.4 MORE ABLATION STUDY
Layer number of MLPPs We perform experiments on the layer number of MLPs in the SDF network to inspect its effect. As shown in Table the mean chamfer distance decreases with the increase of the layer number of MLPs. NeuS (Wang et al. 2021) adopts layers of MLPs and gets an error of 0.84, while the error of our method already decreases to 0.78 with only layers.
Method
Num. 24
37
40
55
63
65
69
83
97
105
106
110
114
118 122 Mean
NeuS
1.00
1.37
0.93
0.43
1.10
0.65
0.57
1.48
1.09
0.83
0.52
1.20
0.35
0.49
0.54
0.84
1.59
1.67 0.56 0.94 0.96 0.73 0.97
1.95
1.82
0.95
1.48
1.67
1.73
0.83
0.78
1.24
0.53 0.78 0.73 0.36 0.90 0.66 0.74
1.48
1.10
0.72
0.55
1.59
0.36
0.57
0.64
0.78
NeuS+Ours
0.49 0.75 0.72 0.37 0.87 0.58 0.63
1.34
1.11
0.74
0.49
1.58
0.34
0.42
0.46
0.73
0.51 0.74 0.58 0.36 0.89 0.58 0.66
1.39
1.05
0.78
0.51
1.47
0.36
0.41
0.42
0.71
0.42 0.77 0.38 0.34 0.81 0.61 0.65 1.26
1.10 0.65 0.49
1.60
0.32
0.58 0.51
0.70
Table 4: Quantitative results for the different layer number of MLPs on the DTU datasct.
Ablation study on Neurallarpp- To better inspect the effectiveness of our modules on differ- ent frameworks, we present the ablation study on the Neural NeuralWarp Darmon et al. (2022 framework, shown in Table and Figure
13

---

Under review as a conference paper at ICLR 2023
Hicrarchical Volume
Regular Terms
Sparse High-Res
Mean
0.68
0.64
X
0.63
V
0.61
Table Ablation study SarralWarp+Ours on the DTU dataset.
Without High-res
With High-res
Without Regularization
With Regularization
Figure 9: Ablation study Neuralwarp+Ours on the DTU dataset.
A.5 RESULTS ON NORMAL CONSISTENCY
We report the normal consistency metric in Table 6 This score is obtained by first calculating the mean absolute dot product of the normals in the reconstructed mesh and the normals at the corresponding nearest neighbors in the ground-truth mesh, and then calculating that in reverse.
Method
55
106
110
122
Mom
0.955
0.935
0.822
0.918
Neural Warp
0.917 0.908 0. 0.984 0. 0.959 0.931 0.944 0.898 0.826 0.840 0.825
0.947
0.896 0.942
0.942 0.940 0.912
NeuS+Ours
1.9942 895 0.968 0.957 0.938 0.953 0.906 0.817 0.858 0.817 0.956 0.897 0.943 0.951 0.927 0.915
NeuralWaap--uus
0.920 0.909 0.968 0.955 0.947 0.944 0.910 0.829 0.852 0.822 0.953 0.901 0.940 0.943 0.938 0.915
Table 6: Quantitative results for the normal consistency metric on the DTU dataset.
A.6 COMPARISON TO PLENOXELS AND INSTANT-NGP.
To better see the difference between our method and Plenoxels or Instant-NGP, we perform some detailed experiments on the Scan-24 of DTU dataset Jensen al. (2014).
The first one is the original version of Instant-NGP with hash tables of size 224 The batch size (the number of sampled points in one iteration) set 65536 for fair comparison of the efficiency. Other parameters are kept the same as the original code, while the images and camera poses are kept same as in our experiments. The result this experiment can seen in Figure II where the accuracy of novel view synthesis is high but the surface is noisy.
The second one is NauS+Hash, where we add to NeuS with the hash tables of the same settings as the Instant-NGP experiment. We add the feature embedding from the hash tables to NeuS while keeping other parameters of NeuS unchanged. The result of this experiment can seen Figure [I] where NeuS+ Hash could obtain the surface with high quality in most regions, but there still some defects on the surface, especially in the less-seen regions, which take a weak place in the hash collision.
The third one is NeuS+Plenoxels,, where we add to NeuS with a single-volume encoding reso- lution 2503 We still keep layers of MI.ss, because the result is mess without any MI.ss. We
14

---

Under review as a conference paper at ICLR 2023
NeuS+Plenoxer Image
Ed1
10277
1983
1211
5255
PSNR: 20.59
PSNR: 24.14
PSNR: 25.83
NeuS CeuS+Plenoxels Surface
FUEEE DEECE
NeuS+Ours Image
180
156
PT1
1951
PSNR: 21.60
PSNR: 24.20
PSNR: 26.09
NeuS+Ours Surface
BDECE
BEER
10K
50K
300K
Figure 10: Visual comparison to Plenoxels DTU scan-22 24. The rendered novel view and extracted mesh are displayed with respect to the iteration number.
also tried the original version of Plenoxels, which performs badly on this dataset. From the results displayed in Figure 10 the results of NeuS+ is noisy, despite that the image quality is high.
However, the convergence speed of Instant-NGP is faster than ours. This because we still use large MLPs equipped in NeuS, which requires more time optimize. Also, the hash tables memory efficient, such that our method requires about three times more memory compared to Instant-NGP NeuS+Hast these experiments.
15

---

Under review as a conference paper at ICLR 2023
Instant- -GGP Image
125
1123
1722
PSNR: 25.89
PSNR: 26.07
PSNR: 27.01
Instant-NGP Surface
NeuS++Hash Hash Image
115
117A
PSNR: 20.26
PSNR: 24.67
PSNR: 25.95
NeuS+ Hash Surface
NeuS+OOurs Image
P1SS
1855
1221
LACE
AURLCS
PSNR: 21.60
PSNR: 24.20
PSNR: 26.09
Ours Surface
ECEEE BRERE
10K
50K
300K
16
Figure 11: Visual comparison to Instant-NGP on DTU scan-24. The rendered novel view and the extracted mesh are displayed with respect the iteration number.

---

Under review as a conference paper at ICLR 2023
A.7 EVALUATION NOVEL VIEW SYNTHESIS
To evaluate the effectiveness of our method in novel view synthesis, we perform the experiments and report the results in Table 7 Table and Figure 12 We select one of each images from the original image set of DTU as the test views, and the remaining images as the training views. The accuracy of novel view synthesis on scans of DTU reported in Table 7 while the PSNR result with respect to iteration number is reported in Table g From the results in these two tables and the images displayed in Figure 12 we can see both the accuracy and convergence speed of the novel view synthesis benefit from our hierarchical volume encoding.
Method
106
114
118
Mean
24.54
25.04
27.97
29.
17.90
25. 22
31.44
33.14
NeuS=Cuus
26.00
24.98
26.76
26.94
27.06
30.08
28.44
27.1 20.03
27.68
32.37
(3.81
29.73
33.70
34.45
28.84
Table 7: PSNR results for the novel view synthesis on the test images of DTU dataset.
Method 5K
10K
20K
30K
50K
100K
200K
300K
NeuS
18.22
19.01
20.38
20.88
21.49
22.25
23.84
24.54
NeuS+Ours
20.03
21.87
23.04
23.67
23.68
24.84
25.84
26.00
Table 8: PSNR results for the novel view synthesis with respect the iteration number on 7 test images of DTU scan-24.
1222
PSNR: 18.51
PSNR: 21.93
PSNR: 24.31
m
12
1:3
3:30
3303
PSNR: 21.60
PSNR: 24.20
PSNR: 26.09
1OK
50K
300K
Figure 12: Novel view synthesis on DTU scan-24. The top row is the results of NeuS while the bottom row the results of NeuS+Ours.
A.8 VISUALIZATION OF VOLUME ENCODINGS
To inspect how the volumes help encode the 3D space, we visualize the volumes in Figure 33 Due to the 4D structure of the volumes, we perform the marching cubes with threshold to do the visualization. Following the threshold selection of NeRF, the threshold is set to 0 since the values of most of the voxels are distributed around 0, as shown in the histogram of Figure 13 From the displayed histograms and volumes, we can see the values in the volumes are set to be distributed randomly (μ 0, a 0.02) before the training, and distributed close to the object surface after the training. Also, the volumes in different resolutions represent different fineness of the scene.
17

---

Under review as a conference paper at ICLR 2023
8000
800
800
400
3000
2000
200
1000
-0.04
-0.01
0.02
0.03
-20
20
#Iter 0, Resolution = 32
#iter = 300K, Resolution = =32
#lter 0, Resolution 3
#lter 300K, Resolution 32
#lter = 300K, Resolution = 3 #Iter = 300K, Resolution = 163
Blended with Res. 16 and 323
volume encodings. The top row presents the histograms values of voxels in the volume with resolution of 32 of which the first one the initialized volume before training, and the second one is the final volume after the training. The corresponding meshes are displayed in the second row, and more meshes displayed in the third row.
Figure 13: Visualization
18


Reviews:
[{"review_id": "SZ9iZixMgI", "reviewer": "Reviewer_hdwP", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents an improved design of spatial encoding for neural surface reconstruction methods, e.g. NeuS and VolSDF. Instead of using frequency positional encoding as in NeRF, the authors propose to explicitly encode the spatial information with multi-scale voxels storing feature vectors. In particular, the encoding consists of dense voxel grids up to the resolution of 256^3, whereas subsequent higher resolutions (512^3 and 1024^3) are derived from sparsified results from the previous levels. Experiments are focused on the DTU benchmark with additional visualizations on BlendedMVS and EPBL, showing improvements over frequency encoding.", "summary_of_the_review": "I think this is a nice paper that presents surface reconstruction results that have shown observable improvements, but I have major concerns on the novelty of this work. While I understand novelty is not the sole evaluation criterion, I think there are insufficient discussions on (a) how this work differ from the others, (b) deeper insights on the technical parts of the methods, (c) limitations of the proposed method. I am thus leaning towards a reject at this point.\n", "strength_and_weaknesses": "Strengths:\n+ This paper presents a simple and effective improvement over frequency encoding used in neural surface reconstruction methods like NeuS and VolSDF. Using explicit (hybrid) representations have recently shown success to accelerate convergence and improve view synthesis results [A,B,C]; this paper applies the same concept to surface reconstruction.\n+ The proposed HIVE adds significant improvement over baseline models (NeuS and VolSDF) in reconstruction quality.\n+ The method description is easy to follow and understand.\n\nWeaknesses:\n- The authors have missed very important references that improves NeRF with explicit (hybrid) encodings, listed as [A,B,C]. In particular, Plenoxels [A] have already proposed explicit encoding with a voxel pruning strategy. In some sense, this paper proposed almost exactly the same concept, but only applied to surface reconstruction. In addition, the authors have ignored a large body of prior works on using octrees to represent 3D scenes with hierarchical voxel grids. A survey can be found in [D].\n- I think it would be good to have discussions on the drawbacks of the proposed HIVE. One limitation I can see is that HIVE is currently limited to the resolution of 1024^3. Although the higher resolutions can be initialized from sparsified voxels (with an additional dilation step), the amortized memory growth will be N^2 (proportional to the surface area) and eventually it will still be memory bounded. So it doesn't seem like HIVE is much more scalable.\n- Another drawback of HIVE is that granular details are not recoverable if certain voxels were pruned at resolution 256^3. So there is a heavy assumption that the coarse shapes should be fully captured at 256^3. This is not a limitation for more naïve frequency encoding from NeRF.\n- The improvement on BlendedMVS scenes seems significant over NeuS and somewhat better than NeuralWarp, but the resolution of the recovered surfaces do not seem to be as high resolution as expected (1024^3?). It would be great if the authors could clarify.\n- The NeuS+hash results seem questionable. From the results of Muller et al., Instant NGP has even more representation capability than the original NeRF, and artifacts shown in Fig 6(c) are not to be expected. It would be good if the authors could clarify; otherwise I would not think this is a fair and faithful comparison.\n- Minor: Fig 6 is hard to understand without sufficient descriptions in the captions. \n- Minor: I don't think equations 7 and 8 are necessary; they are just binary dilation and index lookup operations.\n\n[A] Yu et al. \"Plenoxels: Radiance Fields without Neural Networks.\" CVPR 2022\n[B] Sun et al. \"Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstruction.\" CVPR 2022\n[C] Chen et al. \"TensoRF: Tensorial Radiance Fields.\" ECCV 2022\n[D] Knoll. \"A survey of octree volume rendering methods.\" 2006", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "MENGuPaPs-K", "reviewer": "Reviewer_frLo", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The work tackles on reconstructing the implicit surface from images. The main contribution of the work is to propose a hierarchical volume encoding for neural implicit surface reconstruction, which can be easily be plug-and-played on top of other existing works. Although the hierarchical volume encoding scheme has been widely used in computer vision community, experiments on widely used datasets, such as DTU, BlendedMVS show that the work clearly enhances other methods.", "summary_of_the_review": "Although the proposed hierarchical volume encoding itself is not novel in the field of surface reconstruction, I believe that the pros of having better performance while being simple is very practical and will be beneficial to community if the additional figures and experiments are done as stated in the weakness section. I would like to hear what other reviewers before making the final decision. \n", "strength_and_weaknesses": "**Strength**\n\n- Performance\n\nThe work shows impressive performance both quantitatively and qualitatively. I was impressed by figure1 & 5 reconstructing high frequency details from images.\n\n- Simple and efficient scheme\n\nUsing hierarchical features for implicit reconstruction makes sense and is straightforward. I also enjoyed a simple idea of obtaining sparse high resolution volume by using multi-stage optimization and dilating the low resolution reconstructed implicit function. \n\n\n**Weakness** \n\n- Using hierachical feature volume encoding has been used in the field of computer vision for a long time with various tasks, including surface reconstuction, such as [1]. \n\nThis idea is not brand new, however, I did enjoy the performance increase with respect to the simplicity of the method.\n\n\n- The work can be strengthened by adding more experiments for NeuralWarp + HIVE. \n\nThe authors have reported all scores for both Neus and NeuralWarp with and without HIVE for table 1 & 2. I think adding restuls for NeuralWarp + HIVE in Figure 6 and ablation study can further strengthen the paper, because this paper shows enhancement of the results based on previous works. Because this work is an add on to other methods, I believe these results are necessary to show the performance increase. \n\n- The work can be strengthened by adding more quantitative metric than Chamfer distance. \n\nTable 1, 2 shows the Chamfer distance compared to other methods. While the Chamfer is a good metric to compare the similarity of to points clouds, there has been many questions [2] whether the Chamfer distance could capture the high frequency details or overall ditributional similarity between two point clouds. I would suggest adding normal consistency as in [3] or some other metrics that show that the method can capture the high frequency surfaces.\n\n[1] Chibane et al. Implicit functions in feature space for 3d shape reconstruction and completion. CVPR, 2020.\n\n[2] Wu et al. Density-aware Chamfer Distance as a Comprehensive Metric for Point Cloud Completion. NeurIPS, 2021\n\n[3] Mescheder et al. Occupancy Networks: Learning 3D Reconstruction in Function Space. CVPR 2019\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "782hk90GlL5", "reviewer": "Reviewer_ZvSi", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a multi-scale learnable feature volume encoding for neural implicit surface reconstruction. \nThe main idea of the paper is to train a neural implicit surface function from coarse to fine in 3 stages. In each stage, a learnable volume feature encoding is concatenated to the input of the implicit function. The authors also proposed a volume sparsify procedure to discard the empty voxel for memory efficiency. \nThe proposed strategy can be widely adopted in different coordinate-based neural implicit surface papers.\n\nThe underlying technique of this paper is very simple and easy to understand, yet it significantly improves the surface reconstruction quality of the previous state-of-the-art methods in various public available datasets.", "summary_of_the_review": "\nI found the idea of this paper very simple yet very effective.\nThe proposed method can also be easily applied to other works. The paper achieves state-of-the-art performance on surface reconstruction. \nBut some claims in the paper is not properly addressed and nor proofed via experiments. Please see the weakness. \n\nI think the paper at least provides good empirical results on utilizing volume encoding in improving SDF surface reconstruction. On the other hand, it would be good if the authors could also provide more insights on how such encoding assists the geometry reconstruction, and whether such encoding can be applied to broader use (such as novel synthesis).", "strength_and_weaknesses": "Strength\n\n- The proposed strategy can be widely adopted in different coordinate-based neural implicit surface papers.\n\n- The underlying technique of this paper is very simple and easy to understand, yet it significantly improves the surface reconstruction quality of the previous state-of-the-art methods in various public available datasets.\n\n\n\nWeaknesses\n\nPlease explain more on the claims in the paper : \n- In page 4, Sec 3.3, \"This works well for novel view rendering but has problems in surface reconstruction, because the simple MLP architecture makes learning and optimization inefficient\".\nWhy a simple MLP makes the learning inefficient? Why is it inefficient in geometry learning, but good at RGB learning? \n\n- In page 5, on top: \"This feature volume can naturally encode the knowledge about the 3D space of the object\".\nWhy is a randomly initialized volume encoding can \"naturally\" encode the knowledge about an object? How is this volume encoding looks like when fully trained? More visualization would be helpful here in revealing the use of such encoding.\n\n\nAdditional experiments suggestions:\n\nI am curious to see whether the proposed method can improve the RGB reconstruction or not? It will be good to showcase some novel view synthesis evaluations. And it would be good to explain more on why this encoding can or cannot help RGB reconstruction.\n\nTypos: \nIn page 4, first line, \"with which another implicit\", ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "3ygTS6gyZU6", "reviewer": "Reviewer_wTJb", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper describes a series of experiments in which a hierarchical feature volume is used to encode a latent representation of a scene. The volume defines a continuous feature field through linear interpolation of the features associated with discrete voxel locations. The resultant feature at any point in space can then be passed as input to a neural network or networks to render images of the encoded scene. A variety of rendering methods can make use of this same scene representation, such as NeRF, NeuS, etc. The paper also describes a method of progressively optimizing the feature volumes in a coarse-to-fine order, and a sparsification strategy for the higher-resolution volumes to avoid using too much memory.", "summary_of_the_review": "The manuscript provides new experiments showing that the addition of a sparse hierarchical volume encoding to a neural scene renderer can improve performance of a variety of existing baseline methods. However, the technical novelty over other previously published sparse hierarchical scene representations is very small, and the manuscript not only fails to make a case for the use of the proposed technique over the highly similar existing approaches.\n", "strength_and_weaknesses": "The paper identified a valid issue with many neural scene representations, i.e. the inductive bias towards lower-frequency radiance, density, and/or signed distance fields which often results in overly-smooth renderings. The paper also proposes a sensible strategy to mitigate this issue, and show that their technique significantly reduces reconstruction errors when applied to a variety of scene rendering models. The ablation studies provide interesting insights into the value added by the hierarchical volumes and the influence of various hyperparameters.\n\nThe key weakness of the paper is that it lacks technical novelty. However, it is not clear from reading the manuscript this is the case. There are important gaps in the related work cited, and the related work that is cited is given a fairly cursory treatment. Instant Neural Graphics Primitives introduces a very similar approach in which a scene is volumetrically encoded, and it can be used interchangeably with the proposed hierarchical volumes. However, Instant-NGP is only mentioned at the very end of the manuscript. An experiment is performed to compare the performance of Instant-NGP vs. HIVE and the results indicate that HIVE leads to superior performance. However, the experiment is not described in detail and as such it is hard to interpret the results. The manuscript mentions an advantage of HIVE over Instant-NGP, namely that it does not experience hash collisions which can impair performance, but does not mention the corresponding advantage of Instant-NGP over HIVE which is that the memory cost can be kept to O(1) in the scene volume whereas HIVE is O(N) in the scene volume (with some scalar multiplier resulting from sparsification). Was the amount of memory used to represent the scenes balanced between the two approaches for the experiment presented in Table 1? The manuscript also provides no runtime analysis of the proposed approach, so we do not know how it compares to Instant-NGP which is known to enable fast inference of scene representations.\n\nThe manuscript also completely fails to mention a number of highly related works on hierarchical sparse representation of scenes for neural reconstruction, such as Neural Geometric Level of Detail (Takikawa et al. 2021), PlenOctrees (Yu et al. 2021), and ACORN (Martel et al. 2021). NGLoD in particular introduces an extremely similar scene representation. Unfortunately the two approaches are evaluated on different datasets and thus we do not know which performs better or why.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject