HF-NeuS: Improved Surface Reconstruction Using High-Frequency Details
Authors: Yiqun Wang
Ivan Skorokhodov
Peter Wonka
Venue: NeurIPS 2022 Accept (2022)
Keywords: NeRF-based surface reconstruction
Multi-view surface reconstruction
High-frequency details
Signed distance fields

Abstract: Neural rendering can be used to reconstruct implicit representations of shapes without 3D supervision. However, current neural surface reconstruction methods have difficulty learning high-frequency geometry details, so the reconstructed shapes are often over-smoothed. We develop HF-NeuS, a novel method to improve the quality of surface reconstruction in neural rendering. We follow recent work to model surfaces as signed distance functions (SDFs). First, we offer a derivation to analyze the relationship between the SDF, the volume density, the transparency function, and the weighting function used in the volume rendering equation and propose to model transparency as a transformed SDF. Second, we observe that attempting to jointly encode high-frequency and low-frequency components in a single SDF leads to unstable optimization. We propose to decompose the SDF into base and displacement functions with a coarse-to-fine strategy to increase the high-frequency details gradually. Finally, we design an adaptive optimization strategy that makes the training process focus on improving those regions near the surface where the SDFs have artifacts. Our qualitative and quantitative results show that our method can reconstruct fine-grained surface details and obtain better surface reconstruction quality than the current state of the art. Code available at https://github.com/yiqun-wang/HFS.

HF-NeuS: Improved Surface Reconstruction Using High-Frequency Details
Yiqun Wang KAUST
Ivan Skorokhodov KAUST
Peter Wonka KAUST
Abstract
Neural rendering can be used to reconstruct implicit representations of shapes without 3D supervision. However, current neural surface reconstruction methods have difficulty learning high-frequency geometry details, so the reconstructed shapes are often over-smoothed. We develop HF-NeuS, novel method improve the quality of surface reconstruction in neural rendering. We follow recent work to model surfaces as signed distance functions (SDFs). First, offer a derivation to analyze the relationship between the SDF, the volume density, the transparency function, and the weighting function used in the volume rendering equation and propose to model transparency as transformed SDF. Second, we observe that attempting to jointly encode high-frequency and low-frequency components in a single SDF leads to unstable optimization. We propose to decompose the SDF into base and displacement functions with a coarse-to-fin strategy to increase the fighffrequency details gradually. Finally, we design an adaptive optimization strategy that makes the training process focus on improving those regions near the surface where the SDFs have artifacts. Our qualitative and quantitative results show that our method can reconstruct fine-grained surface details and obtain better surface reconstruction quality than the current state of the art. Code available at https: /tttp::/github.. com/yiquu..nooccc wang/HF
Introduction
3D reconstruction from a set of images is a fundamental challenge computer vision 9.. In the recent past, the seminal framework NeRF [19] inspired lot of follow up work by modeling 3D objects as a density function (xx) and view-dependent color c(z, v) for each point x E R in the volume. The density function and view-dependent color function are implicit functions modeled by a neural network. The results of this approach are very strong and therefore NeRF inspired a large
amount of follup up work, e.g. [T8 17 24
35
20 2
In particular, one direction of work tries to constrain the density field to make it more consistent with density field stemming from surface. In the original formulation, almost arbitrary densities can be modeled by the neural network and there is no guarantee that a meaningful surface can be extracted from the density. Two noteworthy recent approaches, Neus [30] and VoISDF (32), proposed to embed a signed distance field in the volume rendering equation. Therefore, instead modeling the density with neural network, these approaches model signed distance function f with neural network. This leads to greatly improved surface reconstruction.
We build on this exciting recent work and seek further improvement in the quality of surfaces that are being reconstructed. To this end, we propose our method HF-NeuS consisting of three new building blocks. First, we analyze the relationship between the signed distance function on the one hand and the volume density, the transparency, and the weighting function on the other hand. We conclude from our derivation that it would be best to model function that maps signed distances to the transparency and propose a class of functions that fulfill the theoretical requirements. Second, we observe that
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Imiage
NeRF
VoISDF
NeuS
OURS
Figure Qualitative evaluation on the Lego, Robot, and Ficus models. First column: reference images. Second to the fifth column: NeRF, VoISDF, NeuS, and OURS.
Reference Image
NeuS
NeuS with High Frequency
OURS
Figure 2: The challenge of using high- frehufrrquencies directly in the NeuS framework. First column: reference image. Second the fourth column: NeuS, NeuS with high-frequency details, and OURS.
is challenging to learn high-frequency details directly with single signed distance function as shown in Fig. 2 We therefore propose to decompose the signed distance function into a base function and displacement function following related work. We adapt this idea the differentiable NeRF rendering framework and the NeRF training scheme. Third, the functions that translate distance to transparency can be chosen have a parameter, which we call scale 8. It controls the slope of function (or the deviation of the derivative), which further controls the localization precision of the surface and how much out-of-surfia colors influence the result. In previous work, this parameter is set globally but is trainable, it change from iteration to iteration. We propose novel spatially adaptive weighting scheme to influence this parameter, so that the optimization focuses more problematic regions in the distance field. The three building blocks are the three main contributions of the paper. In the results, we can see that HF-NeuS has clear improvement surface reconstruction. On the 15 scene DTU benchmark (NeuS) and 0.86 (VoISDF) to 0.77 the Chamfer distance (See Figs. and for a visual comparison). The benchmark well the metric proposed by previous work.
improve from the current best values of 0.87
2 Related Work
Multi-view 3D reconstruction. 3D reconstruction based on multiple views is a fundamental challenge in the field of 3D vision. Classical 3D reconstruction algorithms usually reconstruct discrete 3D representations. The methods can be roughly categorized into voxel-based methods and point-based methods. Voxel-based methods 6
27 14 II 22 first discretize the three-
dimensional space uniformly into voxels, and then decide whether the surface occupies particular voxel. Point-based methods reconstruct depth maps and fuse multiple depth maps into point cloud. The point cloud needs to be
26 25 g usually use the correlation between multiple views to

---

subsequently reconstructed into mesh model using explicit algorithms like ball-pivoting 3 and [T]] or implicit algorithms like Poisson surface reconstruction
Delaunay trianglulation
Neural implicit surfaces. Recently, neural implicit representations have received a lot of attention. The corresponding methods aim to reconstruct continuous implicit function representations of shapes directly from 2D images. A required building block is differentiable rendering, which maps the 3D 2D image for a given camera pose. DVR (21) utilizes surface rendering to model the occupancy function of a 3D shape, which uses rool search approach to obtain the location of the surface and predicts 2D image. IDR 33 models the signed distance function of the shape and uses a sphere tracking algorithm to render 2D images. A significant milestone in 3D reconstruction was the development of NeRF [9]. It uses volume rendering to map 3D density field and a 3D directional color field to a 2D image. The proposed representation is flexible enough so that realistic images can be synthesized. To model more complex scenes, NeRF++ (35) proposes to model the background scene with an additional neural radiance field, which handles the foreground and background separately, and achieves better results for large scenes. However, the density function is not as easy to control as the occupancy function the signed distance function, and it is difficult to guarantee the smoothness of the generated 3D shape. Subsequently, UNISURF [23 embeds the occupancy function into the volume rendering equation of NeRF. They use a decay strategy to control which region to sample around the surface during training without explicitly modeling volumetric density. Using signed distance functions, VolSDF (32 embeds signed distance function into the density formulation and proposes a sampling strategy that satisfies a derived error bound on the transparency function. NeuS [30] derive unbiased density function equation using logistic sigmoid functions and introduce a learnable parameter to control the function's 's slope during rendering and sampling. Concurrent to our work, NeuralPatch (5) uses the homography matrix to warp the source patches adjacent to the reference image to constrain colors in the volume to come from closeby patches. However, the calculation of patch warping relies on the accurate surface normal, so it cannot be trained from scratch. Therefore, it is only used as a fine-tuning or post-processing method for other algorithms to optimize the surface. We consider VolSDF and NeuS as the current state of the art and we will compare to these two methods.
scene representation
High-frequency detail reconstruction. It is generally difficult for neural networks to learn high- frequency information from raw signals. Inspired by the field of natural language processing, positional encoding [19 29] is used to guide the network reconstruct high-frequency details. Positional encoding spreads the original signal into different frequency bands using sine and cosine functions of different frequency. Subsequently, SIREN (28 proposes to the sin function as activation function in the network. MipNeRF (2) presents an integrated positional encoding to control frequency in different scales. Park et al. [24] proposed use coarse-to-fine learning strategy to gradually increase high-frequency information, which was subsequently used for pose estimation (17].
Hertz er al. [10] further propose a spatially adaptive progressive coding strategy. For surface reconstruction, implicit displacement fields were proposed for single-view 3D reconstruction [16). Based on the supervision of ground truth SDF values of sampled points, the method utilizes separate networks to model the base SDF and implicit displacement fields. Subsequently, Wang et al. [34] utilize the SIREN network to learn the base implicit function and implicit displacement function, respectively, for point cloud reconstruction tasks. In contrast to our proposed algorithm, these methods require 3D supervision. Further, they do not involve the NeRF formulation or volume rendering. In our work, we build on these ideas to develop new state-of-the-art algorithm for multi-view reconstruction.
3 Method
As input we consider a set of N images -= (1,, IN J. and their corresponding intrinsic and { uT....TN J. HF-NeeS aims to reconstruct the representation of 3D surface S as implicit functions. Specifically, we encode surfaces as signed distance fields. We will explain our method in three parts: 1) First, we show how to embed the signed distance function into the formulation of volume rendering and discuss how to model the relationship between distance and transparency. 2) Then, we propose to utilize an additional displacement signed distance function to add high-frequency details to the base signed distance function. 3) Finally, we observe that the function that maps signed distances to transparency is controlled by a parameter s that determines the slope of the function. We propose scheme to set this parameter in spatially varying manner
I2...IN,
extrinsic camera parameters
3

---

depending on the gradient norm of the distance field, rather than keeping it constant for the complete volume within single training iteration.
3.1 Modeling transparency as transformed SDF
We first review the integral formula for volume rendering and derive a relationship between trans- parency and the weighting function (the product between density and transparency). Based this analysis, we discuss the criteria for functions that are suitable to map signed distances to transparency and propose class of functions that fulfill the theoretical requirements.
Given a ray r(1) o + td, the volume rendering equation is used to calculate the radiance of the pixel corresponding to the ray The volume rendering equation is integral along the ray and involves the following quantities defined for each point in the volume: the volume density o and the (directional) color In addition, the volume has compact support and the boundaries of the volume are encoded by t, and tz.
C(r) -(+)= l.
r(t)o(r(())((((t)c dddt
(1)
The transparency T(t) is derived from the volume density as explained below. The function T(t) denotes the accumulated transmittance along the ray from in to
T(I) = exp (-f," (((()))ss)
(2)
and T(t) is monotonic decreasing function with starting value of Ttt,,) 1. The product (r()) can be regarded as a weighting function w (t) in the volume rendering equation as in Eq. I
In order to involve a signed distance function f. we have to define a function V to transform a signed distance function so that it can be used compute the density related terms in the rendering equation. One way is to directly model a density function (I(()) = V/ (f (r(t))) as proposed by VOLSDF 32]. Taking this approach, a sampling method is required to satisfy
error bound of the sampling to make it less than an error threshold by gradually reducing the scale parameter. Another way is model the weighting function ve((t) V (f (r(t))) as proposed by NeuS. The NeuS paper showcases complex derivation to get the expression for the density function ø.
We rethink this problem to obtain a simplified derivation by focusing transparency instead of the weighting function and also a better understanding of the problem, as follows:
d d((()) -T(())(((())
(3)
dt
An interesting observation is that the derivative of the transparency function T'()) is the negative weighting function. The weighting function has the property of having a maximum on the surface. We take the derivative of the weighting function and set it to 0 to find the extrema (maxima), as follows.
d (T((W)((())) - di (T(I) dt Assuming a planar surface and a single ray-plane intersection, we can see that the extremum point, denoted as f., of the weighting function wt)) will also be the extremum point of the derivative of the transparency function (I). The point lg is expected to the intersection of the ray and the surface. Therefore, we consider defining the transparency function directly as T(t) V (f (r(t))). If the transparency function is designed in such a way that its derivative (t) reaches a minimum on the surface, it follows that the weighting function has maximum the surface. Therefore, one can directly model a transparency function under the condition that its derivative has a minimum on the surface. This is conceptually simpler than modeling the weighting function w(t) proposed by NeuS. We compute the derivative of T (f (I(t)) as follows.
d (I) =
di2
dt
( (S (r(1))) V (r(t))
of dr =w' (F(t)) dd
(5)
dt
dr dt
where V (r(()) is the product of the surface normal and the ray direction, which a constant in case of planar surface and single ray-plane intersection. The signed distance function is zero

---

Reference
Neus
VoISDF
OURS without displacement fuuction
Figure 3: Comparing NeuS and VoISDF with our transparency model. Ground truth on the top left. For each method, the left shows the reconstructed image and the right the reconstructed surface.
on the surface. Hence V/' has an extremum at f = O. This also means has the steepest slope at the surface of the shape. On the other hand, the signed distance function is positive outside of the object, and negative when entering the interior of the object. We generally assume that / = ln is outside that the signed distance starts positive and decays monotonic decreasing function. According to the characteristics of transparency T(t) = (f (r(1))) the transparency starts
negative value along a ray, which is
& fu and is a monotonic decreasing function to 0 inside the object. This inverse property results in the v function being a monotonic increasing function from 0 to 1. Therefore, have our design criteria for V/: y should monotonic increasing function from 0 to 1, with the steepest slope at O.
A very intuitive idea to satisfy this criteria is
sigmoid function and normalize the function to have an output in the interval [0, 1]. We simply use the logistic sigmoid function proposed by NeuS 30 for fair comparison. However, idea is general and other sigmoid functions could be used. Our designed transparency function is as follows,
T(t) M. (f (r(t))) 1+
where V,, the logistic sigmoid function with parameter s controlling the slope of the function. Note that the parameter s is also the standard deviation of the function V. We will use this fact later when discussing the adaptive version of the framework.
Given the differentiable transparency function T(t). the volume density a can be casily calculated following Eq.
(t)
(r(()) = T())
For discretization, we bring Eq. 5 and Eq. 6 into 7 and take advantage of the properties of the derivative of the logistic sigmoid function V s. V.). We get the formula for the discretization computation:
(r(tt)) (v. (f (r(t))) V (r(tt)
Then the volume rendering integral can be approximated using a-composition, where &, = exp (-o; (t++1 - t.)). For multiple surface intersections, we follow the same strategy NeuS [30] where clamp (a+, 0, 1). Compared with NeuS, we obtain simpler formula for the density o for the discretization computation, reducing the numerical problems caused by division NeuS. Furthermore, our approach does not need to involve two different sampling points, namely section points and mid-points, which makes it easier to satisfy the unbiased weighting function. Since there is no need to calculate the SDF and the color separately for the two different point sets, the color and the geometry are more consistent compared to NeuS. Compared to VoISDF (32), since the transparency function is explicit, our method can use inverse distribution sampling computed with the inverse CDF to satisfy the approximation quality. Thus no complex sampling scheme as in VoISDF is required. A visual comparison is shown in Fig.
5

---

3.2 Implicit displacement field without 3D supervision
In order to enable a multi-scal fitting framework, we propose to model the signed distance function as combination of base distance function and a displacement function [34 along the normal of the base distance function. The implicit displacement function is an additional implicit function. The reason for this design is that it is difficult for a single implicit function to learn wwwfrequency and high-frequency information at the same time. The implicit displacement function can complement the base implicit function, so that it is easier to learn fighunryy information.
Compared with the task of learning implicit functions from point clouds, reconstructing 3D shapes from multiple images makes it more difficult to learn high-frequency content. We propose to use neural networks to learn frequencies multiple scales, and to gradually increase the frequency content in a coarse-to- to-fine manner.
Suppose f is the combined implicit function that represents the surface we want obtain. The function fo is the base implicit function that represents the base surface. Following [3), the displacement implicit function fa is used to map the point 2% on the base surface the surface point x along the normal ni, and vice versa fa is used to map the point & on the base surface to the surface point 26 along the normal ns, thus fa (xx) = fa(x). Because of the nature of implicit functions, the relationship between the two functions can be expressed as follows,
((xx)) = [xx + fa (xx) n1) = 0
(xs) [x is the normal of x,, on the base surface. To compute the expression for
where x the implicit function f. we bring the formula X3 = x - far (xx) n, into the Eq. (9) and obtain the expression for the combined implicit function:
(xx==I(xx - Sa nz)
(10)
Therefore, we can use the base implicit function and the displacement implicit function to represent the combined implicit function. However, two challenges arise. First, the Eq. [0 only satisfied the point is the surface. Second, the normal at the point x is difficult to estimate when only knowing the position x. We rely on two assumptions to solve the problem. One assumption is that In this
this deformation can be applied to all iso-surfaces, i.c. fx,)) = fx, + fa' (xx) ns) way the equation is assumed to be valid for all points in the volume and not only on the surface. Another assumption is that X;, and x are not too far away, thus n, be replaced with normal on the point x in the Eq. (TO). We control the magnitude of the implicit displacement function using a displacement constraint (f6))
To precisely control the frequency, we use positional encoding to encode the base implicit function and the displacement implicit function separately. We would like to note some differences to [4 We use positional encoding instead of Siren (28), so that the frequency can be explicitly controlled by coarse-to- fine strategy rather than simply using two Siren networks with two different frequency levels. This is useful when 3D supervision is not given. More details are shown in the supplementary. Positional encoding decomposes the input position into multiple selected frequency bands.
?(x) = ho 71 (x), 7t- (x)]
(11)
where each component consists of a sin and cos function with different frequency.
Y (x) sin (2/xx) cos (29mx))
(12)
Directly learning figh-freeuency positional encoding makes the network susceptible to noise, because wrongly learned high-frequencies hinder the learning of low frequencies. This problem is less pronounced if 3D supervision is available, however high-frequency information of images is easily introduced into the surface generation as noise. We use the coarse-to-fing strategy proposed by Park et al. 24 to gradually increase the frequency of the positional encoding.
75 a) = wy (a) (x) = (1 (1-cos cos (clamp (aL - j. 0, 1))))) (x(x)
(13)
2
where a E [0, 1] is the parameter to control the frequency information involved. In each iteration, a is increased by 1/Tmax until it touches 1, where Ramx is the maximum number of iterations.

---

We utilize two kinds of positional encoding 7(x, 0b), 7xx, a²)) with different parameter as and Odt- We set ag 0.5ag and only control ag for simplicity. We also use MLP functions MLPp, MLPa for fiting the base and displacement functions.
f(x) = MLP((1(x,0)) (anim))-4.......
(14)
where n
f that can be computed by the gradient of MLP, and (.())) = [x)
V (ML(((( a,))). The of the displacement constraint should be clamped during training. show how to control the adaptive in the supplemental materials.
We bring this implicit function into Eq. 66) for calculating the transparency so that the radiance (color) C. of images can be computed by the volume rendering equation.
To train the network, employ the loss function L Lrad + Greg. which includes the radiance loss and the Eikonal regularization loss of the signed distance functions. For the regularization loss, constrain both the base implicit function and the detailed implicit function.
Σ 1 N 1 E c:l1, == 1 M
[(VV))))))l 11² (IVF(xx))l 1)²]
(15)
3.3 Modeling an adaptivate transparency function
In previous subsections, the transparency function is parametrized as sigmoid function controlled by the scale s. This parameter controls the slope of the sigmoid function and it is also the standard deviation of the derivative. We can also say that controls the smoothness of the function. When is large, the value of the sigmoid function drops sharply as the position moves away from the surface. On the contrary, the value decreases smoothly when is small. However, choosing a single parameter 8 per iteration gives the same behavior at all spatial locations in the volume.
Since two signed distance functions need to be reconstructed, especially after the high frequency is superimposed, it easy to break the Eikonal constraint, i.c., make the SDF's gradient norm deviate from in some positions. Even with the regularization loss, it is impossible to avoid this problem.
We propose to use the gradient norm of the signed distance field weight the parameter spatially varying manner, increasing when the gradient norm along the ray direction is larger than The intuition is that the implicit function with the larger gradient norm undergoes more abrupt changes, which indicates region that should be improved. Making larger in such regions makes the distance function more precise by magnifying its errors, especially near the surface. In order to adaptively modify the scale 8, we propose the following equation:
EXD
T(t)= = 1 1
(16)
where Vf is the gradient of the signed distance function, and K is the number of sampling points, Wi is the normalized V (ffs) as the weight and Σ
W =1
While this method can be used to control the transparency function, it can also be used for the hierarchical sampling stage proposed by standard NeRF [9]. By locally increasing s, more samples will be generated near the surface where the signed distance values change more rapidly. This mechanism also helps to optimization to focus on these regions in the volume.
Experiments
Baselines. We compare HF-NeuS to the following three state-of-the-art baselines: (1)NeuS 30 the most relevant baseline for our work. We consider it to be the best published method. (2)St 32 concurrent work to NeuS. We consider be the second best published method. Overall also performs very well. Ne) focuses image synthesis and is included for completeness. NeRF
7

---

Table 1: Quantitative results the DTU dataset.
Method
NoRF
1.90
1,60
1,85
0.58
2.28
1.27
1.47
1.67
2,05
1.07
0.88
2.53
1.06
1.15
0.96
1.49
Fidelity
VOL.DE
1.14
1.26
0.81
1.25
0.70
1.29
1.18
0.70
0.66
1.08
0.42
0.61
0.55
0.86
NeuS
1.37
1.21
0.73
0.40
1.20
0.70
0.72
1.01
1.16
0.82
0.66
1.69
0.39
0.49
0.51
0.87
OURS
1.32
0.70
0.39
1.06
0.63
0.63
1.12
0.52
1.22
0.33
0.49
0.50
0.77
NeRF
26.24
25.74
26.79
27.57
31.96
31.50
29.58
32.78
28.35
32.08
33.49
31.54
31.0
35.39
35.51
30.65
VOLSDF
26.28
25.61
26.55
26.76
31.57
31.50
29.38
33.25
28.03
32.13
33.16
31.49
30.33
34.90
34.75
30.38
NeuS
28.20
27.10
28.13
28.80
32.05
33.75
30.96
34.47
29.57
32.98
35.07
32.74
31.69
36.97
37.07
31.97
OURS
29.15
27.33
28.37
28.88
32.89
33.84
3L17
34.83
30,66
33.77
38.44
33,09
32.12
37.13
37.32
32.33
Image
Figure 4: Qualitative evaluation on DTU (first and third rows) and BlendedMVS (second row).
not really a surface reconstruction method and does not reconstruct high-quality surfaces, but it is very good in image-based metrics. We use threshold of 25 (as proposed by NeuS (30) to extract surfaces for the comparisons. For three methods, we use the default parameters and number of iterations recommended in their respective papers. We do not include older methods in the comparison, such as UNISURF or IDR 333, because NeuS and VoISDF have better results.
Datasets. We conduct experiments on the DTU dataset [2] We follow previous work and choose the same 15 models for comparison. DTU is multi-view stereo dataset. Each scene consists of 49 or 64 views with 1600 x 1200 resolution. We further choose 9 challenging scenes from other datasets: scenes from the NeRF-synthetic dataset [99 and 3 scenes from BlendedMVS [I License). The image resolution of NeRF synthetic dataset [19] is 800 x 800 and 100 views are provided for each scene. The dataset contains objects with very obvious detailed and sharp features. such the Lego and Microphone scenes. We chose this dataset for the analysis of reconstructions of high-frequency details. The BlendedMVS dataset is similar to DTU dataset, but with a richer background. This dataset provides image resolution of 768 x 576. We also select models with nigh-frequeney details or sharp features which are difficult to reconstruct. In three datasets, ground truth surfaces and camera poses provided.
Evaluation metries. To evaluate the quality of the reconstruction, we follow previous work and used Chamfer distance (lower values are better) and PSNR (higher values are better). For the DTU dataset, we use the official evaluation protocol, which means computing the mean of accuracy (distance from the reconstructed surface to the ground truth surface) and completeness (distance from the ground truth surface to reconstructed surface). For DTU and BlendedMVS, the background not part of the ground truth surface. Therefore, we remove the background for computing the Chamfer distance, following previous work. The NeRF-synthetic dataset [9] has no background, so we only remove disconnected parts for all competing methods.

---

Table 2: Quantitative results on NeRF-synthetic and BlendedMVS datasets.
Metrio[10 -3)
Method
Chair
Ficus
Lego
Materials
Mic
Ship
Mean
Bread
Dog
Robot
Mean
NcRF
2.12
5.17
3.05
1.51
4.77
3.54
3,36
0.102
0.693
2.325
1.07
Fidelity
VOLSDF
1.26
2.83
1.35
3.62
2,92
0.074
0.354
1.453
0.63
NeuS
0.74
1.21
2.35
1.30
3.89
2.33
1.97
0.068
0.173
1.036
0.43
OURS
0.69
1.12
0.94
1,08
0.72
2.18
1.12
0.066
0.155
0.922
0.38
NeRF
33.00
30.15
32.54
29.62
32.91
28.34
31.27
27.46
25.33
VOLSDF
25.91
24.41
26.99
28.83
29. 29,46
26.86 86
31.05
28.24
25,46
28.25
PSNR
NeuS
27.95
25.79
29.85
29.36
29.89
25.46 .6
28.05
31.32
28.71
25.87
28.63
OURS
28.69
26.46
30,72
29.87
30.35
25.87
28.66
31.89
29.42
26.15
29.15
Table 3: Ablation study results.
Chamfer Distance
PSNR
Datasets
Base
Base+H
Base+C2F IDF+H IDF+C2F
FULL
Base
Base+Hf
Base+C2F
IDF+H
IDF+C2F
FULL
DTU
1.08
1.20
1.07
1.25
0.89
0.78
31.77
32.73
32.56
32.69
32.13
32.49
NcRF-Syuthetic
2955
2.83
28.39
30.52
29.88
BlendedMVS
0.43
fail
0.63
0.47
0.41
0.38
28.63
fail
27.35
28.20
28.95
29.15
Implementation details. We use MLPs to model two signed distance functions fn. and fa. Each MI.P consists of layers. Related work like NeuS (30] and IDR (33) also use MLPs with 8 layers. We use Adam with learning rate Se 4 for the network training using NVIDIA TITAN A100 40GB graphics cards. For adaptive sampling, the SDF and its gradient at these points. We utilize the Eq. 16 to calculate the gain the s parameter, and then adaptively update the weight according the gain and sample additional 64 points. For the coarse-ter to- to-fine strategy, we observe that using a = at the beginning for surface reconstruction produces smoothed results.
first uniformly sample 64 points the ray, then calculate
utilize a = 0.5 and af = 0.5a = 0.25 for both signed distance functions. We set I, 16 for the parameter of the frequency band of positional encoding. For other parameter settings. please see the supplemental materials.
Comparison. In table D we show quantitative results with other competitors on 15 scenes of the DTU dataset 12.. The values shown the upper part of the table measure the fidelity of the surface reconstruction, the Chamfer distance. The numbers indicate that HF-NeuS significantly outperforms NeRF. most scenes, HF-NeuS better than VoISDF and NeuS that the overall average distance is also improved. In the lower part of the table we show the PSNR values. It can be seen that our PSNR surpasses all other methods. We further compare the visual quality achieved by different methods. As shown in Fig. 4 HF-NeuS can reconstruct high-frequency details. For example, the windows have better geometric details, and the feathers of the bird are more distinct.
Most of the scenes in the DTU dataset have smooth surfaces, and figh-frequency details are not obvious. We selected 9 challenging models from the NeRF-synthetic dataset (19 and BlendedMVS dataset (3I], which have more high-frequeney details. For example, the Lego model has uneven repeating bumps, and the power cord of the Mic model has a very thin structure (Fig. D. The robot model has richer edge and corner features (Fig. 4 second row). As shown in Table 2 the gap between our surface reconstruction quality and that of all other methods widens. This shows that HF-NeuS is especially advantageous for surface reconstructions with high-frequency information. We can also observe that NeRF very good the image-based metric (PSNR) while performing poorly in the surface reconstruction metric (Chamfer distance). This observation is consistent with previous work. Compared with the NcRF-Synthetic dataset, the BlendedBMS dataset has a more complex background, this also restricts the performance of NeRF to a certain extent. Besides outperforming other baselines in terms of quantitative error, we also achieve better results in terms qualitative visual effects. As shown in Fig. T HF-NeuS can more accurately reconstruct the details of each Lego block and even some of the tiny holes that are not reconstructed by any other method. For the Robot scene, He-NeuS can reconstruct more accurate facial contours and sharper horns. Finally, for the Mie model, He-NeuS can clearly reconstruct the power cord, while other methods will mess up this structure.
Ablation study. We verify the influence of different modules on the reconstruction results, including the coarsc-to-fine module, the implicit displacement function module, and the aosition-adaptive

---

control module. In Table E "Base "refers to the baseline method, which is NeuS. "H" means we use high-frequency positional encoding. Here we L=16 to represent high frequencies. "C2F" refers to the coarse-to- to-feee optimization strategy with high-frequency positional encoding. We set the initial 0.5. "IDF" represents using the implicit displacement function in reconstruction. For each dataset, we chose the mean of the three scenes as the quantitative metric. From the results of the BlendedM dataset, we can observe that the divergence of network training can be prevented based on the coarse-to-fine strategy. From the DTU and NeRF-synthetic datasets, introducing high-frequency directly can easily lead to overfitting on these datasets. This means that an increase in PSNR cannot guarantee the improvement of the fidelity of surface reconstruction. Although the coarse-to-fie to-fine module can alleviate this mismatch to some degree, it is difficult further improve the performance. However, adding the implicit displacement function component improves the fidelity of the surface reconstruction and PSNR at the same time. During reconstruction, the network with adaptive can help to improve the reconstruction quality upon more complex scenes.
Limitation. As shown in Fig. 5 our method still has challenges. We show a reference ground truth image, our corresponding reconstructed image, and our reconstructed surface. For the grid of ropes of the ship, some overfitting to ground-truth radiance is still observed. Specifically, the grid of ropes is visible in the image, but the surface is not reconstructed accurately. Another limitation is that the individual thin ropes are missing. We also visualize bad case of Table [ where the error is larger than that of the other methods as shown in Fig. [4 DTU Bunny in the supplementary material. In this case, the lighting of this model varies and the texture is not as pronounced, thus it is difficult to reconstruct the details of the belly. Further, integrating proposed IDF increases computation time.
Reference
OURS-Image
OURS-Surface
Figure 5: Limitation. First column: the reference ground truth images. Second column: our synthetic
images. Last column: reconstructed surface.
5 Conclusion
We introduce HF-NeuS, new method for multi-view surface reconstruction with high-frequency details. We propose a new derivation to explain the relationship between signed distance and transparency and propose class functions that can be used. By decomposing the signed distance field into combination of two independent implicit functions, and using adaptive scale constraints to focus on optimizing the regions where the implicit function distribution is not ideal, a more refined surface can be reconstructed compared to previous work. The experimental results show that the method outperforms the current state of the art in terms of quantitative metrics and visual inspection. An interesting direction for future work is to explore the reconstruction of scenes under different lighting modalities. Finally, we do not expect negative social impacts that will be directly linked to research. Negative social impacts of surface reconstruction in general are possible though.
Acknowledgements
We would like to acknowledge support from the SDAIA-KAUST Center of Excellence in Data Science and Artificial Intelligence and the NSFC No.62202076.
10

---

References
C. Barnes, F. Shechtman, A. Finkelstein, and D. B. Goldman, Patchmatch: randomized correspondence algorithm structural image editing. ACM Trans. Gruph., 28(3):24, 2009.
[2] J. T. Barron, B. Mildenhall, M. Tancik, Hedman, R. Martin-Brualla, and P. P. Srinivasan. Mip-nerf:
A multiscale representation for anti-aliasing neural radiance fields. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages
2021.
[3] F. Bernardini, J. Mittleman, H. Rushmeier, C. Silva, and G. Taubin. The ball- hall-pivoting algorithm for surface reconstruction. IEEE transactions on visualization and computer graphics, 5(4):349 1999.
[4] A. Broadhurst, T. W. Drummond, and R. Cipolla. probabilistic framework for space carving. In Proceedings eighth IEEE international conference on computer vision. ICCV 2001, volume 1, pages 388-393, IEEE, 2001.
[5] F. Darmon, B. Bascle, J.-C. Devaux, P. Monasse, and M. Aubry. Improving neural implicit surfaces geometry with patch warping. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6260-6266 2022.
[6] J. S. De Bonet and P. Viola. Poxels: Probabilistic voxelized volume reconstruction. In Proceedings of International Conference on Computer Vision (ICCV), pages 418-425, 1999.
[7] Y. Furukawa and Ponce. Accurate, dense, and robust multiview stereopsis. IEEE transactions on pattern analysis and machine intelligence, 32(8):1362 1376, 2009.
[8] S. Galliani, K. I Lasinger, and K. Schindler. Gipuma: Massively parallel multi-view steren reconstruction. Publikationen der Deutschen Gesellschaft für Photogrummetrie, Fernerkundnng und Geoinformation V, 25(361-369)2, 2016.
[9] R. Hartley and A. Zisserman. Multiple view geometry in computer vision. Cambridge university press, 2003.
[10] A. Hertz, Perel, R. Giryes, Horkine-Hornung, and D. Cohen-Or. Sape: Spatially-adaptic progressive encoding for neural optimization. In Advances in Neural Information Processing Systems, 2021.
[11] S. Izadi, D. Kim, O. Hilliges, D. Molyncaux, R. Newcombe, P. Kohli, J. Shotton, S. Hodges, D. Freeman, A. Davison, al. Kinectfusion: real-time 3d reconstruction and interaction using moving depth camera. In Proceedings of the 24th annual ACM symposium User interface software and technology. pages 559-568, 2011.
[12] R. Jensen, A. Dahl, G. Vogiatzis, E. Tola, and Aanæs. Large scale multi view stereopsis evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 406 0666441,,2 2014.
[13] M. Kazhdan, M. Bolitho, and H. Hoppe. Poisson surface reconstruction. In Proceedings of the fourth Eurographics symposiun Geometry processing, volume 7, 20066
[14] N. Kutulakos and M. Seitz. theory shape by space carving. International journal of computer vision, 38(3):199 218, 2000.
[15] P. Labatut, J..P. Pons, and R. Keriven. Efficient multi-view reconstruction of large-scale scenes using interest points, delaunay triangulation and graph cuts. In 2007 IEEE 11th international conference on computer vision, pages 1-8. IEEE, 2007.
[16] and Zhang. D2im-net: Learning detail disentangled implicit fields from single images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10246- 10255, 2021.
[17] C.-H. Lin, W.C. Ma. A. Torralba, and S. Lucey. Barf: Bundle-adjusting neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5741-5751, 2021. 2021.
[18] L. Liu, Gu, K. Zaw Lin, T.S.. Chua, and C. Thcobalt. Neural sparse voxel fields. Advances Neural Information Processing Systems, 33:15651-15663. 2020.
[19] B. Mildenhall, P. P. Srinivasan, M. Tancik, T. Barron, R. Ramamoorthi, and R. Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In European conference on computer vision, pages 405 421. Springer, 2020.

---

[20] M. Niemeyer and A. Geiger. Giraffe: Representing scenes as compositional generative neural feature fields. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 11453-11464, 2021.
[21] M. Niemeyer, L. Mescheder, M. Oechsle, and A. Geiger. Diffenentiable volumetric rendering: Learning implicit representations without supervision. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3504-3515, 2020.
[22] Nießner. M. Zollhöfer, S. Izadi, and Stamminger. Real-time seal-time reconstruction at scale using voxel hashing. ACM Transactions on Graphics (ToG), 32(6):1-11, 2013.
[23] M. Occhsle, S. Peng, and A. Geiger. Unisurf: Unifying neural implicit surfaces and radiance fields for multi-view reconstruction. In Proceedings of IEEE/CVF International Conference on Computer Vision, pages 5589-5599, 2021.
[24] Park, U. Sinha, Barron, S. Bouaziz, B. Goldman, M. Seitz, and R. Martin-Bruall Brualla. Nerlies: Deformable neural radiance fields. International Conference on Vomputer Vision, 2021.
[25] Schonberger and J.M. Frahm. Structure- centtturr-fr----eee trom-motion revisited. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4104 4113, 2016.
[26] L.. Schönberger, E. Zheng, J.-M. Frahm, and M. Pollefeys. Pixelwise view selection for unstructured multi-view stereo. In European Conference on Computer Vision, pages 501-518. Springer, 2016.
[27] S. M. Seitz and C. R. Dyer. Photorealistic scene reconstruction by voxel coloring. International Journal of Computer Vision, 35(2):151-173, 1999.
[28] V. Sitzmann, Martel, A. Bergman, D. Lindell, and G. Wetzstein. Implicit neural representations with periodic activation functions. Advances Neural Information Processing Systems, 33:7462-7473, 2020.
[29] M. Tancik, Srinivasan, B. Mildenhall, S. Fridovich-Keil, N. Raghavan, U. Singhal, R. Ramamoorthi, Barron, and R. Ng. Fourier features networks learn high frequency functions in low dimensional domains. Advances in Neural Information Processing Systems, 33:7537-7547, 2020.
[30] P. Wang, L. Liu, Y. Liu, Theobalt, Komura, W. Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. Advances in Neural Information Processing Systems, 2021.
[31] Y. Yao, Z. Luo, S. Li, J. Zhang, Y. Ren, L. Zhou, T. Fang, and L. Quan. Blendedmvv: A large-scale dataset for generalized multi-view stereo networks. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 1790-1799, 2020.
[32] L. Yariv, J. Gu, Y. Kasten, and Y. Lipman. Volume rendering of neural implicit surfaces. Advances in Neural Information Processing Systems, 34, 2021.
[33] L. Yariv, Y. Kasten, D. Moran, M. Galun, M. Atzmon, B. Ronen, and Lipman. Multiview neural surface reconstruction by disentangling geometry and appearance. Advances in Neural Information Processing Systems, 33:2492-2502, 2020.
[34] W. Yifan, L. Rahmann. and O. Sorkine hommung Geonetry-consistent consistent neural shape representation with implicit displacement fields. In International Conference Learning Representations, 2022.
[35] K. Zhang, G. Riegler, N. Snavely, and V. Koltun. Nerf++: Analyzing and improving neural radiance fields. arXiv preprint arXiv:0010.07492, 2020.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
12

---

(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proof's of all theoretical results? [Yes]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material URI.)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report enror bars (c.g., with respect the random seed after running experi- ments multiple times)? [N/A]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] Section
4. If you using existing assets (e.g., code, data, models) curating/releusing assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes] See Section
(c) Did you include any
either in the supplemental material URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [Yes] All datasets are public.
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [Yes] See Section
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
13


Reviews:
[{"review_id": "m1R7Bb9W26c", "reviewer": "Reviewer_YNsW", "summary": "The paper improves the detail representation of geometric details for implicit surface reconstruction using multiview images. \nIt contains 3 contributions: \n\n1. an analysis of the SDF and values in the volumetric rendering, which leads to formally deriving the suitable transformation function to use SDF for volumetric rendering.\n2. an improved implicit displacement field for better detail reconstruction, which contains several changes to the original implicit displacement field to robustly work with 2D supervision\n3. a strategy to improve the optimization by adapting the mapping function in (1) spatially.", "questions": "1. Regarding the displacement constrain $\\Psi'_s$, in the supplemental, the authors mentioned using some progressive strategy to gradually increase the constraint. If I understand correctly, this means gradually requiring the displacement to be spatially sparse and more focused on the surface area. The motivation is not very well explained. Shouldn't I want the opposite - by requiring a tighter spatial bound around the base surface at the beginning of the training, the base SDF can quickly converge, so that when optimizing the displacement $f_d$, $f_b$ remains stable and hopefully the composed function $f$ converges faster. Why is this not the case?\n2. Can I understand the adaptive transparency function described in (15) as a form of hard example mining? What do you think $\\|\\nabla f\\|$ would be a good indicator? BTW, should it be $\\\\|\\nabla f\\\\|$ instead of $\\|\\nabla f\\|$?\n3. Table 3: Is X+C2F adding C2F to X+H?  FULL is adding adaptive transparency to IDF+H+C2F? If this is not the case, then the IDF+adaptive s is missing\n", "limitations": "Yes.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths:\nThis paper is very well written and shows impressive improvements compared to the state-of-the-art for reconstructing geometric details.\nThe three steps are well motivated and the ablation shows that they all contribute to the improvement. The entirety of the paper provides a good basis for NeRF-based multi-view 3D reconstruction.\n\n# Weakness:\nI think while this paper was very clear in stating the 3 contributions (section 3.1-section 3.3), it's not very clear what exactly is novel (proposed by the paper) and what is from prior work. This is particularly true in section 3.2. L175-201 seem to be purely prior work [32], which should be clarified and condensed. More effort should be put into explaining and motivating the differences.\n\nThere is a small section in the supplemental (B.2) explaining hierarchical sampling based on $cs$. But I don't see any result demonstrating the effectiveness of this sampling strategy. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "T38rDcJnxG9", "reviewer": "Reviewer_t73h", "summary": "The authors address the objective of improving the accuracy of 3D scene reconstruction based on neural rendering approaches. For this purpose, they first provide a theoretical discussion on how the signed distance function (SDF) can be embedded into volume rendering as well as the relationship between the SDF, the transparency function, the volume density and the weighting function. In addition, criteria for functions that are suitable to map signed distances to transparency and a respectively suitable class of functions is discussed, but the choices follow previous work (in particular, the integration of the SDF follows VolSDF, and the selection of a logistic sigmoid function with learnable scale parameter for Psi follows NeuS). This is used in a multi-scale fitting framework where the SDF is modeled as a combination of a base distance function and a (implicit) displacement function (similar to references [reference 15] and [reference 32]) along the normal of the base distance function, and positional encoding is used for each of these functions separately, where the frequency of the positional encoding is increased according to the coarse-to-fine strategy by Park et al. [reference 22]. Both functions are additionally represented by separate MLPs, and the used loss includes a radiance term and an additional Eikonol regularization of the SDF, where the norm of the gradient of the SDF is used to weight the scale parameter in a spatially varying manner.\n\nExperiments on the DTU Dataset, NeRF Synthetic Dataset and BlendedMVS dataset with comparisons to NeuS and VolSDF show the potential of the proposed approach.", "questions": "## Exposition:\n- As mentioned above under 'strengths/weaknesses', some selections of reference lists seem arbitrary. For NeRF improvements, it might be helpful to cluster references regarding the aspect the techniques are focused on, e.g. faster training/inference, less constrained conditions, video inputs, generalization capabilities, multi-scale representations, etc.. The lists of papers for 3D reconstruction and surface reconstruction also seem arbitrary in the current formulation. Referring to the respective works or respective surveys could solve this.\n\n## Evaluation:\n- I would recommend to also provide the equation for the Chamfer distance, as there are different definitions used in literature.\n- Regarding Figures 1 and 3, showing the local errors on the surface with respect to the ground truth might additionally better highlight where deviations are larger.\n- As mentioned above, the MipNeRF approach focuses on scene representation at continuously-valued scale which improves the representation of fine-grained details. Why has this approach has only been mentioned in the list in the introduction, but not in the discussion of related work or among the competing techniques used for evaluation.\n- As mentioned above, training and inference times could be mentioned.\n- As mentioned above, an ablation study of the effect of the regularizers would be interesting as well.\n- As mentioned above, the stability of the reconstructions of fine-grained structural details could be shown with respect to the number of views. Maybe the proposed approach gets the same quality of the reconstruction of NeRF with significant less input views.\n- As mentioned above, there is no detailed discussion of problematic cases or failure cases in the main paper. Limitations regarding inaccurate reconstructions of linear structures (i.e. ropes) mentioned in the supplemental should rather be mentioned in the main paper. In addition, e.g., for the cases where the proposed approach does not outperform its competitors in Table 1, respective visualizations could be provided to demonstrate the kind of errors and where these are larger than for the other methods. This would provide insights on further improvement potential.\n- A discussion or comparison to the recent approaches Instant-NGP (Muller et al., Instant neural graphics primitives with a multiresolution hash encoding) and SVLF (Tremblay et al., RTMV: A Ray-Traced Multi-View Synthetic Dataset for Novel View Synthesis) would be interesting, however, it seems as if these works were not yet published at the NeuRIPS submission deadline.\n\n## Reproducibility:\n- Will code be release upon paper acceptance?", "limitations": "The authors have adequately addressed the potential negative societal impact of their work. Limitations in terms of the need for optimizing an additional implicit function or overfitting have only been shortly discussed in the last section on conclusions. As mentioned above, the discussion of limitations needs to be improved.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Originality/novelty and technical soundness:\npro:\n+ The approach seems reasonable.\n+ The major novelty seems combining these concepts in the NeRF framework and using a spatially varying scale parameter which, together with the combination of all the concepts, leads to accuracy improvements over the other approaches (NeuS, VolSDF) in the comparison. \n\ncon:\n- However, there is a larger overlap with existing work regarding the use of separate networks to represent a base SDF and displacement fields [references 15 and 32], the integration of the SDF into volume rendering [reference 30], using an unbiased density function using logistic sigmoids and learnable control parameter [reference 28]).\n\n\n\n## Clarity and exposition:\npro:\n+ The paper is well-structured and easy to follow. Arguments are clear.\n+ Figures/Tables and the respective captions are informative.\n+ The relevance of the topic is sufficiently motivated.\n+ The supplemental provides further implementation details in more detail.\n\ncon:\n- While text quality is mostly ok, there are typos, e.g. 'view-dependent' appears several times as 'view dependent', 'low/high frequency' should be written as 'low/high-frequency' if it is followed by a noun it refers to (e.g., 'components', 'content', 'surfaces', etc.), 'state of the art algorithm' should read 'state-of-the-art algorithm', 'coarse to fine optimization' should read 'coarse-to-fine optimization'.\n\n- Some selections of reference lists seem arbitrary, e.g., the reference lists on NeRF improvements (in the introduction) and multi-view stereo reconstruction (in the related work section). It is not clear why only these references have been used. For NeRF improvements, it might be helpful to cluster references regarding the aspect the techniques are focused on, e.g. faster training/inference, less constrained conditions, video inputs, generalization capabilities, multi-scale representations, etc.. Regarding voxel-based 3D reconstruction approaches there are many more variants, e.g.:\n- - Izadi et al., KinectFusion: real-time 3D reconstruction and interaction using a moving depth camera, and its extensions such as:\n- - Niessner et al., Real-time 3D reconstruction at scale using voxel hashing\n- - Whelan et al., Kintinuous: Spatially extended kinectfusion\n- - Dai et al., Bundlefusion: Real-time globally consistent 3d reconstruction using on-the-fly surface reintegration\n- - Klingensmith et al., Chisel: Real Time Large Scale 3D Reconstruction Onboard a Mobile Device using Spatially Hashed Signed Distance Fields\n- - Stotko et al., Efficient 3D Reconstruction and Streaming for Group-Scale Multi-Client Live Telepresence\n\nIn addition, there are more variants for surface meshing such as:\n- - Fuhrmann and Goesele, Floating Scale Surface Reconstruction\n- - Calakli and Taubin, SSD: Smooth signed distance surface reconstruction\n- - Berger et al., A survey of surface reconstruction from point clouds\n\n\n\n## Evaluation:\npro:\n+ The authors provide qualitative and quantitative results with comparisons to reasonable baselines (VolSDF and NeuS), where the proposed approach is demonstrated to lead to improvements in the reconstruction result.\n+ An ablation study shows the effects of high-frequency positional encoding, coarse-to-fine optimization and the use of the implicit displacement function.\n+ The supplemental provides further ablations on the coarse-to-fine parameter alpha_d and the number of frequency bands L as well as ablations for adaptive transparency.\n\ncon:\n- The MipNeRF approach that focuses on scene representation at continuously-valued scale also improves the representation of fine-grained details. It is not clear why this approach has only been mentioned in the list in the introduction, but not in the discussion of related work or among the competing techniques used for evaluation.\n- Training and inference times could be mentioned.\n- An ablation study of the effect of the regularizers would be interesting as well.\n- Additionally, the stability of the reconstructions of fine-grained structural details could be shown with respect to the number of views. Maybe the proposed approach gets the same quality of the reconstruction of NeRF with significant less input views.\n- There is no detailed discussion of problematic cases or failure cases in the main paper. Limitations regarding inaccurate reconstructions of linear structures (i.e. ropes) mentioned in the supplemental should rather be mentioned in the main paper. In addition, e.g., for the cases where the proposed approach does not outperform its competitors in Table 1, respective visualizations could be provided to demonstrate the kind of errors and where these are larger than for the other methods. This would provide insights on further improvement potential.\n\n\n\n## Reproducibility:\npro:\n+ The information in the paper together with the attached code seems sufficient to re-implement the approach.\n\ncon:\n- It is not entirely clarified, whether the code will also be made available afterwards.\n\n\n\n## References:\ncon:\n- See comment under 'Exposition'.\n- For several of the references, only the arxiv reference is provided and not the final conference/journal reference. This should be revised accordingly.\n\n\n\n## Post-rebuttal recommendation:\nThe paper addressed my major concerns in terms of ...\n... the detailed discussion of the works of VolSDF [30], NeuS [28] and Yifan et al.'s approach [32] which helps to better see the embedding of the proposed approach in the context of related work,\n... the improvements regarding the exposition (typos, also mentioning MipNeRF in the related work, mentioning the used Chamfer distance formula, etc.),\n... the promised code release,\n... the additional details regarding training/inference times,\n... the shift of limitations into the main paper,\n... the added visualization of local errors.\nAs a result, I increase my rating from borderline accept to weak accept.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "RyCn8cKqI6c", "reviewer": "Reviewer_AK1b", "summary": " \n This paper builds on the recent efforts in multi-view neural implicit surface reconstruction and improves in the reconstruction quality by introducing a different way of modelling SDFs, applying the implicit displacement fields to this task, and also proposing locally-adaptive scaling factors. They show decent improvements on multiple object-level dataset for both surface reconstruction and view synthesis.\n", "questions": "- I wonder if you simply change the way of modelling SDFs of NeuS to yours, how much benefit can you obtain (without the displacement field) for NeuS?\n \n \n - No enough acknowledgement to the IDF paper in 3.2. For example, $\\mathbf{x}_b = \\mathbf{x} - f_d(\\mathbf{x}_b)\\mathbf{n}_b$ (the key idea of implicit displacement field) mentioned already in IDF. \n \n - Why not do the same as in the IDF paper where they use two SIRENs for the base model and displacement model but using different frequencies?\n \n \n - PSNR calculation. Do you calculate the PSNR on the training views or the held-out testing views?\n \n - Ablation study. Table 3 confuses me since it lacks of discussion of different componenets. Why Base+H is much worse than only Base? Why changing from only base to IDF does not gurantee better results (DTU experiment)? And it seems to me that most beneficial components are coming from the coarse-to-fine strategy and the locally-adaptive scales.\n \n", "limitations": "Limitations and societal impact are briefly discussed in the conclusion. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "*Strengths*\n - I like the idea of locally-adaptive scaling factor s, which makes a lot of sense and I think can be a general add-on for other methods.\n - The paper is well written overall, easy to follow.\n \n *Weaknesses*\n- In 3.1, what is the motivation of modelling SDFs with transparency? I don't see the clear advantage of doing so compared to modelling with the density function as done in VolSDF, or weighting function in NeuS. Indeed, modelling with the transparency is indeed conceptually simper than NeuS, but fundamentally it is very similar to the combination of NeuS and VolSDF.\n- I don't think the method has fully utilize the benefit of implicit displacement field. For example, intuitively speaking, using a base model to model the coarse shape and another network to model the details should speed up the optimization process a lot, since the coarse shape can be very quickly obtained, and learning displacements is an easier optimization task than learning fully SDFs. Any experiments show the runtime benefit?  \n- According to both quantitative and qualitative results, even after adding 3 different contributions and course-to-fine strategies, the proposed approach is only marginally better. I am not fully convinced by the usefulness of your different contributions. Moreover, because of the use of an extra MLPs, you even need more computational resources.\n \n \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "KWpgoXFWOFn", "reviewer": "Reviewer_XTFd", "summary": "The paper tackles the task of surface reconstruction from images. They use the volume rendering based approach popularized by NeRF, and thus are also interested in maintaining high novel view synthesis performance. In particular they follow recent work (IDF, VolSDF) that embed an SDF into the volume rendering pipeline in order to get more accurate surfaces, and propose an approach for achieving high frequency detail. To do this, they incorporate the SDF in a new fashion, use implicit displacement fields [32] and use a coarse to fine strategy. ", "questions": "In section 3.2 the work describing the decomposition into a base and displacement field is not cited at all (should be [32]). Given that it is a major part of the approach and the use of that work is not cited anywhere except in related work, this is quite important to do.", "limitations": "Both limitations and potential negative societal impacts have been discussed in the conclusion.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- Good explanation of previous ways that SDFs have been incorporated into volume rendering\n- Good explanation for the proposed way to incorporate SDFs and the intuition behind it.\n- Experimented on well established datasets/benchmarks and shows significant improvements on surface reconstruction while maintaining similar or better view synthesis quality\n- Good ablation for the high frequency detail methods, e.g. basic encoding vs displacement field method, and the benefit of the coarse to fine approach.\n\nWeaknesses\n- No theoretical explanation of the benefits of the new SDF incorporation method compared to previous methods, except for simplicity of the derivation. You have done an excellent job motivating your approach mathematically, but haven't clearly explained what benefits that has over the other methods. You should explain in more detail why directly modelling the transparency function would be better.\n- No ablations showing how the new way to incorporate SDFs differs in performance to the way VolSDF/NeuS does it.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "EyKHQu24ZXs", "reviewer": "Reviewer_iTh1", "summary": "The paper present s new method for surface reconstruction using the recently popular neural volume rendering approach. The authors suggest improving existing methods, which often lack high-frequency details. To do so, the authors present various changes. First, they propose transforming a learned sign distance function into the transparency function, rather than the physical density or the weighting function as done in relevant baselines. Additionally, following previous works, the authors decompose the sign distance function into base and displacement functions, accompanied by a gradually increasing frequency learning strategy. Lastly, they propose improving certain regions by weighing based on the properties of the sign distance function. The paper presents reconstruction results compared to existing baselines at three different datasets.", "questions": "Except for the mentioned concerns raised in the previous section, I would appreciate it if the authors would answer the following questions:\n\n1. Could the authors explain their choice of the logistic sigmoid function for Psi?\n2. (Relates to the above) Have the authors considered using the Gaussian or Laplace distributions CDF as the transformation function Psi? Why would one choice will be better than the other?\n3. I suspect equation 9 has a typo, and it should be f_d(x_b)? If not, please explain the transition from equation 8.\n4. The intention behind equation 15 is unclear. Could the authors please explain?", "limitations": "The authors addressed the method limitations. However, I encourage the authors to move the limitations section to the main paper in the revised version.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. The need for high-frequency details improvement is well motivated and understood.\n2. I find some of the suggested improvements innovative and contributive to future works. More specifically, I like the idea of transforming the transparency and examining its derivate and relation to the weighting function, and the idea of the adaptive weighting strategy. I mostly appreciate the concept of weighting based on the eikonal equation satisfaction.\n3. The related work section is concrete, well written, and covers the paper area for newcomers.\n\nWeaknesses:\n\n1. The contribution of each component is not straightforward. Although the authors conducted an ablation study, they have not presented qualitative results or discussed the potential implications of each component for the overall improvement. \n2. Following but separately from the above, I’m concerned with the need for the base and displacement decomposition. Because this decomposition is not unique, and since high frequencies encoding adds noise, more possible wrongly reconstructions exist that overfit the rendering. I believe that a visualization of the decomposition would emphasize its benefits. Another alternative is to show an ablation using the decomposition (“IDF”) with frequencies as the baseline NeuS.\n3. There is a clear difference between the presented VolSDF baseline results shown in this paper and other resources. I direct the authors to the original VolSDF paper for results of the DTU and BlendedMVS datasets (figures 5 and 10), and to the recent work RefNeRF (https://dorverbin.github.io/refnerf/) for the synthetic NeRF results (see video). I would appreciate it if the authors could explain this difference.\n4. There are some missing methodology implementation details. For example, it is unclear what are the discretization formulas for the transparency and the volume rendering integral approximations, given a set of samples along the ray. Also, I believe the authors should state how multiple surface intersections are handled (line 173). Additionally, it is not certain how the displacement constraint is imposed, nor its relation to equation 13.\n5. The superiority of the presented methods is mostly given quantitively but not qualitatively. It is also apparent in the PSNR results, surpassing all other methods, but the paper does not show rendering comparisons. Similar to the ablation study, I believe that additional qualitative results (even in the supplementary) would help the paper statements.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept