RayDF: Neural Ray-surface Distance Fields with Multi-view Consistency
Authors: Zhuoman Liu
Bo Yang
Yan Luximon
Ajay Kumar
Jinxi Li
Venue: NeurIPS 2023 poster (2023)
Keywords: implicit shape representations
multi-view consistency
novel view synthesis
TL;DR: We introduce a new ray-based 3D representation, which is 1000x faster in rendering than coordinate-based representations such as NeRF/SDF/OF/UDF.

Abstract: In this paper, we study the problem of continuous 3D shape representations. The majority of existing successful methods are coordinate-based implicit neural representations. However, they are inefficient to render novel views or recover explicit surface points. A few works start to formulate 3D shapes as ray-based neural functions, but the learned structures are inferior due to the lack of multi-view geometry consistency. To tackle these challenges, we propose a new framework called RayDF. It consists of three major components: 1) the simple ray-surface distance field, 2) the novel dual-ray visibility classifier, and 3) a multi-view consistency optimization module to drive the learned ray-surface distances to be multi-view geometry consistent. We extensively evaluate our method on three public datasets, demonstrating remarkable performance in 3D surface point reconstruction on both synthetic and challenging real-world 3D scenes, clearly surpassing existing coordinate-based and ray-based baselines. Most notably, our method achieves a 1000x faster speed than coordinate-based methods to render an 800x800 depth image, showing the superiority of our method for 3D shape representation. Our code and data are available at https://github.com/vLAR-group/RayDF

RayDF: Neural Ray-surface Distance Fields with Multi-view Consistency
Zhuoman Liu Bo Yang' Yan Luximon
Ajay Kumar Jinxi Li vLAR Group, The Hong Kong Polytechnic University
(zhuo-man liu, jinxi (axxi.1i)@commi..... 1cnnnenneeett.......oo... www.eoonnnnettt.......eooeoeooee hk
bo. yangppooly. edu. .kk
Abstract
In this paper, we study the problem of continuous 3D shape representations. The majority of existing successful methods are coordinate-based implicit neural repre- sentations. However, they are inefficient to render novel views or recover explicit surface points. A few works start to formulate 3D shapes ray- ray-based neural functions, but the learned structures are inferior due to the lack of multi-view geom- etry consistency. To tackle these challenges, we propose new framework called RayDF. consists three major components: 1) the simple ray-surface distance field, 2) the novel dual-ray visibility classifier, and 3) multi-view consistency optimization module to drive the learned ray- ray-surface distances to be multi-view geometry consistent. We extensively evaluate our method on three public datasets, demonstrating remarkable performance in 3D surface point reconstruction on both synthetic and challenging real-world 3D scenes, clearly surpassing existing coordinate- baseeee--ededeeee and ray-based baselines. Most notably, our method achieves a 1000x faster speed than coordinate-basee methods render an 800 800 depth image, showing the superiority of our method for 3D shape representation. Our code and data available https://github.. www.eo/o//eeeeeeeeeeeeeee Arogg
Introduction
Learning accurate and efficient 3D shape representations is crucial for many cutting-edge applica- tions in the fields of machine vision and robotics. Recent advances in 3D coordinate-hased neural representations including occupancy fields (OF) [46, 11], un/signed distance fields (U/SDF) [54, 14], and radiance fields (NeRF) [47], have shown great potential to recover complex shapes from RGB/D images and/or point clouds. Although these methods and their variants have achieved excellent performance in wide range of downstream tasks such shape reconstruction [55, 58, 60), novel view synthesis [4, 72, 51, 5], and scene understanding [85, 86, 73], obtaining 3D shapes or 2D views from their trained networks is computationally expensive, due to the requirement of extensive network evaluations to regress surface points.
Very recently, number of works start to represent 3D models as ray-based neural functions. By simply taking individual light rays as input, the methods LFN [64] and NeuLF [40] learn directly predict the radiance (RGB) values, while PRIF [23] and DDF [3] straightly estimate the surface hitting points of input rays. Compared with mordinate-basse representations, these ray-based methods are clearly more efficient to infer 3D geometry and render 2D views, as every single ray only needs to query the trained network once forward. Nevertheless, the learned 3D geometries by these methods are still lack of fidelity, primarily because they fail to explicitly take into account geometry consistency across multiple views, resulting in the network likely over-fitting individual training rays but unable to generalize to unseen rays in testing.
In this paper, we aim address this key issue of ray-based neural representations by explicitly integrating multi-view geometry consistency into the network design. In particular, we introduce
*Corresponding Author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

mutual visibility ray I
ray 2
distance
ray-surface distances
mutual visibility
Multi-view Consistency Optimization
ray
hitting point
Ray-surfuce Distance Network
Dual-ray Visibility Classifier
Figure 1:
The general workflow and components of our framework.
It consists of two independent neural networks together with a particular training module: 1) the main ray-surface distance network, 2) an auxiliary dual-ray visibility classifier, and 3) multi-view consistency optimization module.
new pipeline shown in Figure 1.
The main network simply takes a ray as input and directly infers the distance between ray origin and its hitting point on the surface. Basically, this network elegantly represents 3D shapes ray-based implicit functions, denoted as ray-surface distance fielld, keeping the unique advantage in efficiency for shape extraction and rendering. The auxiliary network takes a pair of rays input and predicts their mutual visibility. In fact, this is a simple binary classifier, aiming at distinguishing whether any two rays hit at the same surface point not, i.e., the mutual visibility. Having trained auxiliary network at hand, the multi-view consistency optimization module specifies how to effectively leverage the learned dual-ray visibility to train the main network, thus driving learned ray-surface distances to be multi-view consistent from any seen or unseen viewing angles.
Since our ray-surface distance fields primarily aim at representing accurate 3D shapes, the whole pipeline is designed to trained on depth images, although light fields (radiance) can be optionally learned in parallel if color images are also available in training. In this regard, the closest works to ours are PRIF [23] and DDF [3], neither of which explicitly considers the multi-view geometry consistency. Overall, compared with all existing coordinate- wordinntee-bsed representations, our method keeps the superiority in efficiency thanks to the ray-based formulation. Compared with the existing ray- based approaches, ours excels at learning accurate 3D geometries thanks to the multi-view consistency for ray-surface distance fields. Our method is called RayDF and our contributions are:
We employ a straightforward ray-surface distance field for representing 3D shapes. This formulation is inherently more efficient than existing coordinate-basse representations.
We design a new dual-ray visibility classifier learn the spatial relationships of any pair of rays, enabling the learned ray-surface distance fields to be multi-view geometry consistent.
We demonstrate superior 3D shape reconstruction accuracy and efficiency on multiple datasets, showing significantly better results than existing coordinate-ban based and ray-based baselines.
2 Related Work
Explicit 3D Shape Representations: Classic methods recover explicit 3D geometry of objects and scenes mainly include SIM [53] and SLAM [7]systems such as Colmap [62] and ORB-SLAM [49]. Another classic method is space carving [45, 37], which involves the process of carving out the voxel grid by utilizing multiple distinct views to obtain robust approximation within the voxel space. To model explicit 3D structures, deep learning- based methods have shown impressive progress in recovering voxel grids [15, 80, 81], point clouds [21], octree [71], polygon meshes [34, 26] and shape primitives [87]. comprehensive survey of these methods can be found in [6, 28]. Thanks to the large-scale datasets [10] for training sophisticated neural networks [29, 31], these methods demonstrate excellent results in many downstream tasks such as shape reconstruction [76, 70],
generation [38], and semantic scene perception [67, 17, 25]. However, the fidelity of these discrete shape representations is primarily limited by their spatial resolutions and memory footprint.
Coordinate-basee Implicit 3D Shape Representations: To avoid the discretization issue of tra- ditional 3D representations, there has been a growing interest in developing implicit neural 3D representations using simple MLPs to represent continuous 3D shapes. Inspired by the seminal coordinate-bused methods [11, 46, 54] which have shown great success in encoding high-quality 3D shapes, a plethora of follow-up works have been developed to tackle various vision tasks in [65, 79, 2, 12, 13, 20, 33, 43, 42, 27, 83]. These coordinate-base representations can be generally classified as: 1) occupancy fields (OF) [46, 11], 2) signed distance fields (SDF) [54], 3) unsigned

---

d.
gin oft goot gone
|
gonly
(0in m)))
d ray-surface
input ray T fe
distance
Figure 2:
The left block illustrates the spherical parameterization of an input ray and the right block shows our simple MLP-based ray-surface distance field fe.
distance fields (NDF) [14, 74], and 4) radiance fields (NeRF) [47]. More related methods can be found in surveys [24, 18]. Among them, the OF/SDF/NDF based methods demonstrate exceptional accuracy in recovering continuous 3D structures thanks the simple level-set formulation, while the NeRF based methods achieve an unprecedented level of fidelity in rendering 2D views thanks to the successful volume rendering equation. However, all these coordinate- inevitably require dense 3D location sampling and evaluations on trained networks to regress explicit surface points, though advanced techniques [41, 57, 69, 39, 30, 56, 50, 82, 48] can mitigate this issue somewhat but at the expense of a large memory increase. In this paper, our RayDF only needs single network evaluation regress every surface point, while still achieving on par or better reconstruction accuracy with existing OF/SDF/NDF methods.
representations
Ray-based Implicit 3D Shape Representations: To overcome the inefficiency of coordinate- -based 3D representations. number of works start to formulate 3D shapes as ray- ray-based neural functions via MLPs. Ray-based methods simply take individual rays as input and directly output either radiance values (RGB), i.e. light fields, or surface points, i.e. distance fields. LFN [64] and NeuLF [40] are among the early works of light fields to encode 3D representations from observed color images. Other light field works include [22, 61. 78, 52, 68, 75, 9, 1]. PRIF [23]. DDF [3] and DRDF [36] are the early ray-based distance fields to learn 3D shapes from observed depth scans. Although these methods have shown very encouraging results for novel view rendering or surface reconstruction, they usually limited to datasets with small baselines across multi views. Basically, this is because the multi- geometry consistency is not taken into account in their network designs. By contrast, in our RayDF pipeline, we explicitly introduce dual-ray visibility classifier to aid our ray-surface distance fields to be multi-view shape consistent during training.
3
RF
3.1 Overview
Our pipeline consists of two networks and an optimization module. As shown in Figure 2, for the main network: ray-surface distance field fe. it takes a single oriented ray as input, and directly regresses the distance d between ray starting point and its surface hitting point. For the input ray r, we opt for the conventional spherical parameterization [32], as supports querying from 360 viewing angles. In particular, fix-sized sphere is predefined with a relatively large diameter D as convex hull in which the target 3D scene is bounded. Each surface point p of the target scene can be regarded as directional ray penetrating the sphere with two intersection points the sphere. Each intersection point is parameterized by two variables specifying the angles with regard to the sphere center. Then, for each ray, we have 4D parameterization r= (0in, d godd @oul ). Formally, the ray-surface distance field is defined below and implementation details are in Appendix A.I.
fo(o( where r (0 gow oour), ) + are trainable parameters of MLPs
(1)
For the auxiliary network: dual-ray visibility classifier ha. it takes pair of rays as input and simply predicts their mutual visibility, aiming at explicitly modeling the mutual spatial relationships between any pairs of rays. This network, once well-trained, will play key role in the third component: multi-view consistency optimization. Detailed designs are discussed in Sections 3.283.3.
3.2 Dual-ray Visibility Classifier
The main ray-surface network alone can indeed well fit many training ray-distance pairs, but there is no mechanism driving its output distances, i.e., surface geometry, to be consistent across multiple views, especially for unseen views. For example, illustrated in the leftmost block of Figure 3, if
3

---

r1
3
di
d3
T2
mutual invisible
22
02
31
h4
Ogue
dout
I: mutual visible
Figure 3: The leftmost block illustrates the mutual visibility of pair of rays. The remaining block shows the symmetric design of our dual-ray visibility classifier he.
two input rays and r2 are mutually visible in the scene, both rays must hit at the same surface point, then the corresponding ray-surface distances d1 and d2 must satisfy transformation equation ماا - pm - and r'
''n cos 0" sin sin sin
+
T + d222. where م
{in, (More
cos
details in Section A.1.3), according to both rays' angles. Similarly, if input rays rı and r3 are mutually invisible, then they should never satisfy the transformation equation because both never meet the same surface point. From this fundamental principle, we can see that the mutual visibility between any rays is crucial inform the ray-surface distance network to be multi-view consistent.
To this end, we design a binary classifier to discriminate the visibility an input pair rays. A naive idea is to simply concat two rays input, feeding into MLPs to regress 0/1, where I represents visible and 0 otherwise. However, such a design fails to retain the symmetry of two rays. Here symmetry means that the visibility of two rays must be invariant to the input order of two rays. With this point, as illustrated in the right block of Figure 3, our visibility classifier h&p is formally defined as:
hop MLPs 8(0)" 0tth 0TTh 11tH) + g (( 02" 0.th Dou
(K()).). 0/1
2
where (0)" 6f"h 0 (rnh) and (0 "h 0 )0"") the parameterizaaions of two input rays and 12 respectively; s() is a shared single fully-connected layer followed by an average pooling, thus guaranteeing the symmetry of input rays. Note that, the surface hitting point of rı, i.e., (11, is also encoded via separate single fully-connected layer k(), followed by a concatenation operation denoted by @B,, thus enhancing the pooled ray features, we empirically find that such enhancement could notably improve the classifier's 's accuracy. Implementation details are in Appendix A.I.
3.3 Multi-view Consistency Optimization
With the designed main ray-surface distance network so and the auxiliary dual-ray visibility classifier he at hand, we introduce the crucial multi-view consistency optimization module to train both networks. In particular, given K posed depth images (H W) of statie 3D scene as the whole training data, our training module consists of two stages.
Stage 1- Training Dual-ray Visibility Classifier
The key to training this classifier is to create correct data pairs. First of all, all raw depth values are converted to ray-surface distance values. For specific 1dh ray (pixel) in the 4 image, we project its ray-surface point back to the remaining (K 1) scans, obtaining the corresponding (K 1) distance values. We set 10 millimeters the closeness threshold to determine whether the projected (K -11) 1) rays are visible the (K - images. In total, we generate K H W * (K pairs of rays together with 0/1 labels. The standard cross-entropy loss function is adopted to optimize our dual-ray visibility classifier. More details on training data generation and implementation are Appendix A.1.3.
Note that, this classifier is trained in scene-specific fashion. Once the network is well-trained, it basically encodes the relationships between any two rays of specific scene into network weights.
Stage Training Ray-surface Distance Network
The ultimate goal of our whole pipeline is to optimize the main ray-surface network and drive it to be multi-view geometry consistent. However, this non-trivial as simply fitting the network with ray-surface data points cannot generalize to unseen rays, which can be seen in our ablation study in Section 4.5. In this regard, we fully leverage the well-trained visibility classifier aid our training of ray-surface distance network. Particularly, this stage consists of the following key steps:

---

Step 1: All depth images are converted to ray-surface distances, generating K * H x W training ray-distance pairs for a specific 3D scene.
Step 2: As illustrated in Figure 4, for a specific training ray (r, d), called primary ray, we uniformly called multi-view rays, in ball centering at the surface point p. We then calculate the distance between surface point p and the bounding sphere along each of M This can be easily achieved according to the given distance in the training set. M is simply set 20 and more details are in Appendix A.4.
sample M rays (r!...."' ... r),,
rays, obtaining multi-view distances {d] (c'...dh ....d....d dd))
r,r")) r}}} and then feed them into Step 3: We establish pairs of rays ((r.,,))..... (r.p, (r. p,
the well-trained visibility classifier hæ, inferring their visibility scores {ul {u!...,>" Th ).)
Step 4: We feed the primary ray and all sampled M multi- rays (r,rl...,,**
M) into the Since the network fe is randomly initialized, thus the estimated distances are inaccurate in the beginning.
ray-surface distance network fe. estimating their surface distances {d, d' dm
Step 5: We design the following multi-view consistency loss function to optimize the ray-surface distance network until convergence:
M Σ n==1
D 1 +1 (ld- dd
(Iem - dim v(m)
Basically, this simple loss drives the network to not only fit the primary ray-surface distance (seen rays in the training set), but also satisfy that the visible multi-view rays (unlimited unseen rays in the training set) also have accurate distance estimations.
.M
d.
d
di
p
3.4 Surface Normal Derivation and Outlier Points Removal
In the above Sections 3.,.3333. we have two network de- signs and an optimization module to train them separately. Nev- ertheless, we empirically find that the main ray-surface distance network may predict inaccurate distance values, particularly for rays near sharp edges. Essentially, this is because the actual Figure 4: Multi-view ray sampling. sur-surface distances may be discontinuous at sharp edges given extreme viewing angle changes. This shape discontinuity is actually a common challenge for almost all existing implicit neural representations, because modern neural networks can only model continuous functions in theory.
dm
Fortunately, a nice property of our ruy-surface distance field that the normal vector at every estimated 3D surface point can be easily derived in closed-form expression using auto differentiation of the network. In particular, given an input ray r = (0 ptr gou " and its estimated ray-surface distance from network fe. the corresponding normal vector n of that estimated surface point can be derived as a specific function shown below.
== e ad n
r.D.. details of the function Q and derivation are in Appendix A.2.
(4)
With this normal vector, we may choose to add an additional loss to regularize the estimated surface points to be as smooth possible. Yet, we empirically find that the overall performance improvement on an entire 3D scene is rather limited, as these extremely discontinuous cases are actually sparse.
In this regard, we turn to simply removing the predicted surface points, i.e., outliers, whose normal vectors' Euclidean distances are larger than threshold in the network inference stage. In fact, PRIF [23] also adopts a similar strategy to filter out outliers. Note that, advanced smoothing or interpolating techniques may be integrated to improve our framework, which is left for future exploration.
Experiments
Our method is evaluated on three types of public datasets: 1) the object-level synthetic Blender dataset from the original NeRF paper [47], 2) the scene-level synthetic DM-SR dataset from the recent DM-NeRF paper [73], and 3) the scene-level real-world ScanNet dataset [16].
We carefully select the following six successful and representative implicit neural shape representations our baselines: 1) OF [46], 2) DeepSDF [54], 3) NDF [14], 4) NeuS [77]. 5) DS- NeRF [19], 6) LFN [64], and 7) PRIF [23]. The OF/DeepSIF/NDF/Nea. methods are coordinate- based level-set methods, showing outstanding performance in modeling 3D structures. DS-NeRF
Baselines:
5

---

is a depth-supervised NeRF [47], inheriting excellent capability in rendering 2D views. LFN and PRIF are two ray-based methods with superior efficiency in generating 2D views. We note that there are many sophisticated variants of these baselines, achieving SOTA performance on various datasets. Nevertheless, we do not intend to comprehensively compare with them, basically because many of their techniques such as more advanced implementations, adding additional conditions, replacing with more powerful backbones, etc., can be easily integrated into our framework as well. We leave these potential improvements for future exploration but only focus our vanilla ray-surface distance field in this paper. For a fair comparison, all baselines are supervised with the same amount of depth scans ours, carefully trained from scratch in the same scene- scene-specific fashion. More details about the implementation of all baselines and possible minor adaptations are in Appendix 1.3.1.
Metries: For evaluation metrics of shape reconstruction, we report: 1) the per ray-surface absolute distance error (ADE) in centimeters averaged across all testing images, 2) the Chamfer distance (CD) [21] between the whole reconstructed structure and the ground truth shape for each scene. As to our method, we use the existing TSDF fusion [84] to obtain full mesh from predicted depths at the testing views, on which we uniformly sample 30K points. Another 30K points are uniformly sampled from the ground truth 3D mesh for calculating CD. The outlier point removal is only applied to clean the reconstructed point clouds when calculating CD. No additional post-processing steps are employed when computing all ADE scores. Apparently, ADE is more accurate to evaluate the surface estimation of our method, because the CD scores may be biased due to the extra TSDF fusion. In this regard, the CD scores are just presented as a complementary metric to show the general shape quality in the whole 3D scene space. More details about how to obtain the reconstructed 3D full shape for each baseline and our method are in Appendix A.3.2. For evaluation metrics of appearance reconstruction, i.e. novel view synthesis of 2D color images, the standard PSNR, SSIM, and LPIPS scores are reported following NeRF [47]. We present Accuracy (%) and F1-Score (%) as evaluation metrics of the dual-ray visibility classifier. For qualitative results, refer Figure 5 and Appendix A.7.
4.1 Efficiency of RayDF
Table 1: Rendering time
Similar to the existing ray-based methods LFN and PRIF, our RayDF also has the superiority in efficiency generate 2D images or recover the explicit surface points. To quantitatively compare the efficiency with baselines, we conduct a simple experiment to generate an 800 800 depth image on a computer with single NVIDIA RTX 3090 GPU card and a CPU of AMD Ryzen 7. As shown in Table 1, it can be seen that, not surprisingly, the coordinate- based methods OF/ DeepSDF/ NDF/ NeuS/ DS-NeRF are extremely slow to render a high-resolution depth image. In particular, OF [46] needs large number of small steps to gradually approach the surface point along a given light ray direction, while DeepSDF [54] and NDF [14] rely on expensive sphere tracing to regress the points. NeuS [77] and DS-NeRF [19] need extra post-processing steps to calculate depth
consumption (scconds).
Time
OF [46]
286.057
DeepSDF [54]
17.590
NDF [14]
28.310
NeuS [77]
32.793
DS-NeRF [19]
25.612
LFN [64]
0.017
PRIF [23]
0.013
RayDF (Ours)
0.019
values from densities. By contrast, our RayDF and the existing ray- ray-based methods achieve more than 1000x faster speed to render a dense depth view.
4.2 Evaluation on Blender Dataset
The Blender dataset from NeRF [47] consists of pathtraced images of 8 synthetic objects with complicated geometry and realistic materials. Each object has 100 views for training and 200 novel views for testing. Each image has 800 800 pixels. Since the released dataset does not include depth images, we just use the provided Blender files to generate additional depth scans of the same resolution exactly following the original 2D view poses. Note that, the physical sizes of these models are about 2.5 meters in length/height/width, so the sphere diameter /) is set as 3 meters in our method. For the baselines and our method, we conduct the following two groups of experiments.
Group - 3D Shape Representation Only: Since our RayDF needs depth scans in training to learn continuous 3DD shape representations, we conduct this group of experiments only on multi-view depth images. All baselines are also trained on the same number of depth views.
Group 2 - 3D Shape and Appearance Representations: Our RayDF is also flexible to add a parallel branch to output radiance field, i.e., RGB values, so that the continuous appearance representations can be simultaneously learned. In this regard, we conduct this group of experiments on both RGB images and depth images. The baselines NeuS/ DS-NeRF/ LFN/ PRIF are also trained on the same RGBD images for comparison. The other three methods OF/ DeepSDF/ NDF not

---

Table 2:
Quantitative results of all baselines and our method in experiments of Groups 1&2. All scores are averaged out across 8 scenes of the Blender dataset [47].
Group
Group
CD (x 10
CD (x10-
ADE
ADE
PSNR1
SSIM f LPIPS
mean / median
mean / median
OF [46]
10.57
2.982 / 0.706
DeepSDF [54]
12.95
3.382 / 0.679
NDF [14]
12.14
2.976 / 0.831
NeuS [77]
11.88
4.756 / 0.907
12.10
4.662 / 0.938
27.19
0.910
0.100
DS-NeRF [19]
13.22
117.270 1.135
14.64
143.295
1.760
26.63
0.933
0.063
LFN [64]
24.47
89.425 / 15.681
12.33
60.289
1.230
23.20
0.888
0.125
PRIF [23] 14.68
20.764 1.677
14.56
21.279
1.693
23.31
0.874
0.152
RayDF (Ours)
7.97
3.388 / 0.663
8.17
3.295 / 0.755
26.52
0.910
0.099
included here, because is non-trivial add RGB supervision due their level-set formulation. Details of the parallel branch for all methods are in Appendix A.3.
Analysis: Table shows the quantitative comparison. We can see that: 1) Our RayDF achieves significantly better results for explicit surface recovering on both groups of experiments, especially on the most important ADE metric, demonstrating the clear superiority over both coordinate and ray based baselines. 2) Our method also achieves comparable performance with DS-NeRF for novel view synthesis, being better than LFN and PRIF and showing the flexibility of our framework.
4.3 Evaluation on DM-SR Dataset
The DM-SR dataset from the recent DM-NeRF paper [73] consists of 8 synthetic complex 3D indoor rooms. The room types and designs follow Hypersim dataset [59], and the rendering trajectories for both training and testing images follow the Blender dataset of NeRF [47]. For each 3D scene, there are 300 views for training and 100 novel views for testing. Each view has 400 x 400 pixels. Each scene has a physical size of about 10 meters in length/heighh/wwithh, so the sphere diameter D is set as 11 meters in our method. Similarly, we conduct the following two groups of experiments.
Group 3D Shape Representation Only: We conduct this group of experiments only multi-view depth images for all methods.
Group 2 3D Shape and Appearance Representations: We conduct this group of experiments on both RGB images and depth images for NeuS/ DS-NeRF/ LFN/ PRIF and our method.
Table 3: Quantitative results of all basclines and our method in experiments of Groups 1&2. All scores are averaged out across 8 scenes of the DM-SR dataset [73].
Group 1 CD (x )
Group 2
CD (x10-3)4 mean median
ADE
ADE|
PSNR SSIM 1 LPIPS 4
mean / median
OF [46]
15.83
11.402 4.888
DeepSDF [54]
16.97
11.281 / 5.087
NDF [14]
22.41
12.300 / 5.911
NeuS [77]
9.94
12.744/ / 4.620
11.66
15.017/5.308
33.22
0.965
0.054
De-NeRF [19]
10.77
25.380 / 6.032
10.77
25.548
6.102
31.83
0.977
0.026
LFN [64]
18.30
13.673 / 5.372
18.51
14.085 5.349
30.86
0.930
0.111
PRIF [23]
11.89
25.993 / 5.159
11.77
24.842 5.156
31.01
0.933
0.111
RayDF (Ours)
7.41
14.272 / 5.353
7.97
14.251 5,300
30.32
0.940
0.113
Analysis: From Table 3, it can be seen that our method again surpasses all baselines on the most critical ADE metric in both groups of experiments, though the rough metric CD scores are just comparable with baselines potentially due to the inaccuracy incurred by external TSDF fusion. In the meantime, our method still obtains high-quality novel view synthesis, without noticeably sacrificing shape representation when trained with both RGB images and depth scans in Group 2.
4.4 Evaluation on ScanNet Dataset
We further evaluate our method on the challenging real-world 3D dataset ScanNet [16). We randomly select 6 scenes for evaluation. Originally, each scene has more than 2000 RGBD scans in relatively
7

---

Table 4:
Quantitative results of all baselines and our method in experiments of Groups 1&2. All scores are averaged out across 6 scenes of the ScanNet dataset [16].
Group
Group
CD (x 34
CD (x 3r4 mean median
ADE
ADE
PSNRT
SSIM f LPIPS
mean / median
OF [46]
15.84
DeepSDF [54]
12.07
17.200 / 6.255
NDF [14]
22.15
30.436 / 8.348
NeuS [77]
17.20
30.475 3.771
24.12
67.504
5.658
27.00
0.817
0.253
DS-NeRF [19]
6.64
9.653 / 2.318
7.84
14.642 2..900
25.03
0.855
0.197
LFN [64]
6.53
7.651 / 1.864
6.77
31,928
2.116
28.14
0.814
0.262
PRIF [23]
10.32
19.662 / 2.828
10.65
22.902 2.804
21.14
0.731
0.388
RayDF (Ours)
5.42
10.517 / 1.848
5.31
10.739 2.009
31.58
0.856
0.224
long sequence, whose camera poses are usually facing outwards within indoor rooms. Such long camera trajectories pose great challenge to our method due to sphere parameterization of input rays. To overcome this issue, we take the divide and conquer strategy to process a subset of the full video frames for each scene. In particular, for each scene, only pick up subset of 300 continuous RGBD scans in our experiment, where 200 are uniformly sampled for training, and 100 for testing. As to these partial 3D scenes, the physical sizes are between 4 ~ 16 meters in length/height/wide the sphere diameter D is set as 16 meters in our method. Similarly, we conduct the following two groups of experiments.
so
Group 1 3D Shape Representation Only: We conduct this group of experiments only multi-view depth images for all methods.
Group 2 3D Shape and Appearance Representations: We conduct this group of experiments on both RGB images and depth images for NeuS/ DS-NeRF/ LFN/ PRIF and our method.
Analysis: Table compared all methods on the challenging real-world scenes. Remarkably, our RayDF clearly outperforms all baselines in almost all evaluation metrics in both groups of experiments, showing a distinct advantage in representing noisy and complex real-world 3D scenes. This confirms that niew geometry consistency is indeed essential in learning 3D representations.
4.5 Ablation Study
Since our framework just has two MLP-based neural networks, there is few hyperparameter to tune. The key component here is the dual-ray visibility classifier. To evaluate effectiveness, we mainly conduct the following ablations on the classifier on Blender dataset [47] only with depth images.
(1) Removing the dual-ray visibility classifier: Without the classifier, the multi-view consistency loss function becomes the simple & loss: lm. ld id-d d) for rays in the training depth scans.
(2) Replacing position (x) by ray distance in function k(): This to evaluate the enhancement
of explicitly adding surface point position.
(3) Removing the function k() and only using two rays: This ablation is to demonstrate the necessity of adding surface information for accurate classification.
(4) Concatenating two rays instead of average pooling: This ablation aims to evaluate the effect of the symmetry of the input rays on our classifier.
(5) Adding noises to the well-trained visibility classifier: Various levels of noise are introduced
the well-trained classifier to assess the reliability of our visibility classifier. In particular, we directly add random noise drawn from normal distribution (0, σ²) to the visibility score with clip between and and then use the noisy score our multi-vier consistency loss En in Eq. to optimize our ray surface distance network.
Table 5: Ablation studies the classifier.
Table 6: Ablation study on the classifier with noises.
Acc.t FIf ADEL
CD 10 L mean median
Noise level
Acc.t FI1
ADE|
CD (x mean median
(a²)
(1) w/o classifier hy
47.08
1328.934 1073.704
1.0
62.51
61.46
17.03
4.032 / 0.935
(2) function A 81.56 77.67
10.82
4,984 / /0.994
(3) remove function k
81.93 76.77
10.23
4.753 / 0.890
0.5
66.63
65.78
15.54
3.716 0.869
(4) g(r))((( ((()))
95.06 50.17
4.740
0.1
77.09
74.61
12.64
3.454 0.850
Full RayDF
86.34 84.23 7.97
3,388 / 0.663
(R)yDF)
86.34
84.23
7.97
3,388 0.663

---

Analysis: Table 5 (1) clearly shows that, without the aid of our dual-ray visibility classifier, the main ray-surface distance field completely collapses and cannot predict reasonable distance values for novel rays in the test set. In Table 5 (2) and (3), with the input of surface point position, the classifier achieves higher accuracy and higher F1 Fl-score, thus providing the ray-surface distance network with more accurate visibility information to predict precise distance values. In addition, concatenating two imput rays directly disrupts the inherent symmetry of input rays. This is shown in Table (4), where the classifier trained in this way achieves high accuracy but exhibits low Fl-score. This indicates that such a classifier is significantly less robust than the one trained with symmetric input rays. Table 6 demonstrates that less robust classifier results in the ray-surface distance network predicting inaccurate distances.
In addition to the dual-ray visibility classifier, we also perform various ablation studies on our entire pipeline on the Blender dataset.
Ablation on Multi-view Rays Sampling: We conduct an ablation study with different values of M 200,20,4 40) to de- termine the optimal number of multi-view rays for training a ray-surface distance network. Table 7 demonstrates that in- creasing the sampled multi-view rays enhances the accuracy of the ray-surface distance network. Although M 40 achieves lower ADE and CD values, the substantial GPU memory and extended training time outweigh the limited performance im- provement. Therefore, we opt to sample multi-view rays with M 20 train our ray-surface distance network in all main experiments.
Table 7: Ablation study on the number of multi-view rays sampling.
CD (x10-3) 3)- mean / median
M
ADE
10
10.12
4.469 0.713
20 (RayDF)
7.97
3.388/ / 0.663
40
7.72
2.711 / 0.621
Ablation on Sparse Depth Supervision: With the ad- vancement of existing techniques of depth estimation from RGB images, it is quite feasible to obtain sparse depth signals using existing techniques such SfM learning- based monocular depth estimators. In this regard, we ad- ditionally provide experimental results using sparse depth supervision. In Table 8, our method maintains satisfac- tory performance, even with only 1 1% depth values during training. We hypothesize that such robustness comes from our multi-view consistency constraint, because many depth values in the training set may be redundant thanks to our effective classifier.
Table Ablation study on sparse depth supervision.
CD (x 10 10-3) 4
Depth Sparsity
ADE|
mean / median
1%
8.54
3,301 / 0.937
5%
8.70
3,250 / 0.920
10%
8.68
3.313 / 0.924
100% (RayDF)
7.97
3.388 / 0.663
Ablation of One-stage RayDF: In the paper, we adopt a two-stage training strategy for our dual-ray visibility clas- sifier and ray-surface distance network. For a comparison, we simply train both networks simultancously. However, not surprisingly, the performance of one-stage training drops noticeably as shown in Table 9. This drop in perfor- mance primarily because the classifier inaccurate at the early stage and unlikely to provide effective constraints for the ray-surface distance network, given the similar number of training steps. Nevertheless, exploring one-stage training remains an intriguing direction for our future research efforts.
Table 9: Ablation study one/two-stage training.
CD (x10 })_ mean median
ADE|
one-stage two-stage (RayDF)
12.41
4.032/ / 0.659
7.97
3.388 0.663
For more comprehensive analysis, we provide extensive ablation studies and additional qualitative as well as detailed quantitative results of each scene in the Blender dataset in Appendix A.4.
5 Conclusion
In this paper, we have shown that it is truly possible to efficiently and accurately learn 3D shape representations by using a multi-view consistent ray-based framework. In contrast to the existing coordinate-based methods, we instead use a simple ray-surface distance field to represent 3D shape geometries, which is further driven by novel dual-ray visibility classifier to be multi-viww shape consistent. Extensive experiments on multiple datasets demonstrate the extremely high rendering effi- ciency and outstanding performance of our approach. It would be interesting to extend our framework with more advanced techniques such as faster implementation and additional regularizations.
9

---

Distance Map
Mesh
Radiance Map
OF
DeepSDF
NDF
NeuS
DS-NeRF NeuS
DS-NeRF
LFN
PRIF
Ours
LI
DDe
NeuS
PRIF
GT
Blender Lego
Ours
DS-N
LFNN
PRIF
Ours
GTT
DeepSS DIF
Neo'S
NDF
NeuS
NRRN OF
DS-NeRF
LFN
PRIF
Ours
GT
LFN
PRIF
DM-SR Dinning
DeepSDF
NeuS
Ours
GT
www.igpireiieeei Wonrennnenn www.eIISSSORRRIIIS www. www.w//wwwwww...w.......
DS-NNRR
LFN DeepSDr
NDF
NeuS
DS-NERE OF
DS-N
PRIF
Ours
GT
ScanNer Scenedorr 00
DeepSo
NDF
NeuS
Ours
GT
Nel
Figure 5: Qualitative results of all methods the three datasets. More qualitative results can be
found in Appendix A.7 and our project page: http://larrgroup.ithhihee iithub.
10

---

Acknowledgement: This work was supported in part by Research Grants Council of Hong Kong under Grants 15225522 & 25207822 & 15606321, in part by National Natural Science Foundation of China under Grant 62271431, and in part by The Hong Kong Polytechnic University under Grant P0031501.
References
[1] B. Attal, Huang, C. Richardt, M. Zollhoefer, J. Kopf, M. O'Toole, and C. Kim. HyperRecl: High- fidelity 6-DoF video with ray- conditioned sampling. CVPR, 2023.
[2] M. Atzmon and Y. Lipman. SAL: Sign Agnostic Learning of Shapes from Raw Data. CVPR, 2020.
[3] T. Aumentado- Armstron,, S. Tsogkas, S. Dickinson, and A. Jepson. Representing 3D Shapes with Probabilistic Directed Distance Fields. CVPR, 2022.
[4] J. T. Barron, B. Mildenhall, M. Tancik, P. Hedman, R. M.-b. Pratul, and C. Mar. Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields. ICCV, 2021.
[5] J. T. Barron, B. Mildenhall, D. Verbin, P. P. Srinivasan, and P. Hedman. Mip-NeRI 360: Unbounded Anti- Aati-Aliased Neural Radiance Fields. CVPR, 2022.
[6] M. Berger, A. Tagliasacchi, L. M. Seversky, P. Allicz, J. A. Levine, A. Sharf, and C. T. Silva. A Survey of Surface Reconstruction from Point Clouds. Computer Gruphics Forum, 36(1):301-329, 2017.
[7] C. Cadena, L. Carlone, H. Cariilo,, Y. Latif, D. Scaramuzza, J. Neira, I. D. Reid, and J. J. Leonard. Past, Present, and Future of Simultaneous Localization and Mapping: Towards the Rohust-Perception Age. TRO, 2016.
[8] E. Camahort, A. Lerios, and D. S. Fussell. Uniformly sampled light fields. Eurographics Workshop Rendering Techniques, 1998.
[9] J. Cao. H. Wang, P. Chemerys, V. Shakhrai. J. Hu, Y. Fu, D. Makoviichuk, S. Tulyakov, and Ren. Real- Time Neural Light Field on Mobile Devices. CVPR, 2023.
[10] A. X. Chang, T. Funkhouser, L. Guibas, P. Hanrahan, Q. Huang, Z. Li, S. Savarese, M. Savva, S. Song, Su, Xiao, L.. Yi, and F. Yu. ShapeNet: An Information-Rich 3D Model Repository. arXiv: 1512.00013 2015.
[11] Z. Chen and H. Zhang. Learning Implicit Fields for Generative Shape Modeling. CVPR, 2019.
[12] Z. Chen, Y. Zhang. Genova, S. Fanello, Bouaziz, C. Haene, Du, C. Keskin, T. Funkhouser, and
D. Tang. Multiresolution Deep Implicit Functions for 3D Shape Representation. ICCV, 2021.
[13] Chibane, T. Alldieck, and G. Pons-Moll. Implicit Functions Feature Space Shape Reconstruction and Completion. CVPR, 2020.
[14] Chibane, A. Mir, and G. Pons-Moll, Neural Unsigned Distance Fields for Implicit Function Learning. NewrlPS, 2020.
[15] C. B. Choy, D. Xu, J. Gwak, K. Chen, and S. Savarese. 3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction. ECCV, 2016.
[16] A. Dai, X. Chang, M. Savva, M. Halber. T. Funkhouser, and M. NieBner. ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes. CVPR, 2017.
[17] A. Dai, D. Ritchie, M. Bokeloh, S. Reed, J. Sturm, and M. Nießner. ScanComplete: Large-Scale Scene Completion and Semantic Segmentation for 3D Seans. CVPR, 2018.
[18] L. De Luigi, A. Cardace, R. Spezialetti, P. Z. Ramirez, S. Salti, and L. Di Stefano. Deep Learning on Implicit Neural Representations of Shapes. ICLR, 2023.
[19] K. Deng, A. Liu, J.-Y. Zhu, and D. Ramanan. Depth superviee NeRF; Fewer Views and Faster Training for Free. CVPR, 2022.
[20] Y. Duan, H. Zhu, H. Wang, L. Yi, R. Nevatia, and L. J. Guibas. Curriculum DeepSDF. ECCV, 2020.
[21] H. Fan, H. Su, and L. Guibas. A Point Set Generation Network for 3D Object Reconstruction from a Single Image. CVPR, 2017.

---

[22] B. Y. Feng and A. Varshney. Signet: Efficient neural representation for light fields. ICCV, 2021.
[23] B. Feng, Y. Zhang, D. Tang, and R. Du. PRIF: Primary Ray-based Implicit Function. ECCV, 2022.
[24] A. T. O. Fried, J. T. V. Sitzmann, S. L. K. Sunkavalli, and R. M.-b. T. Saragih, State the Art on Ncural Rendering. Computer Graphics Forum, 2020.
[25] G. Gkioxari, J. Malik, and J. Johnson. Mesh R-CNN. ICCV, 2019.
[26] T. Groueix, M. Fisher, V. Kim, B. Russell, and M. Aubry. A Papier-Mache Approach to Learning Surface Generation. CVPR, 2018.
[27] Guandao Yang, Belongie, Hariharan, and V. Koltun. Geometry Processing with Neural Fields. NewrlPS, 2021.
[28] X.-f. Han, H. Laga, and M. Bennamoun. Image-based 3D Object Reconstruction: Statc-of-the- Art and Trends the Deep Learning Era. TPAMI, 2019.
[29] K. Hc, X. Zhang, S. Ren, and Sun. Deep residual learning for image recognition. CVPR, 2016.
[30] Hedman, P. Srinivasan, Mildenhall, T. Barron, and Debevec. Baking Neural Radiance Fields for Real-Time View Synthesis. ICCV, 2021.
[31] Н. Huang, D. Yang, Dai, Han, Y. Wang, K.-M. Lam, Yang. S. Huang. Liu, M. Agigan: Unpaired image translation for photographic ancient character generation. ACM MM, 2022.
[32] Ihm, Park, K. Lee, and M.-g. Scoul. Rendering Spherical Light Fields. PG, 1997.
[33] Y. Jiang, D. Ji, Han, and M. Zwicker. SDFDiff: Diffcrentiable Rendering of Signed Distance Fields for
3D Shape Optimization. CVPR, 2020.
[34] Kato, Ushiku, Harada. Neural 3D Mesh Renderer. CVPR, 2018.
[35] D. Kingma and J. Ba. Adam: method for stochastic optimization. ICLR, 2015.
[36] N. Kulkarni, J. Johnson, and D. F. Fouhey. Directed Ray Distance Functions for 3D Scene Reconstruction. ECCV, 2022.
[37] K. N. Kutulakos and S. M. Seitz. A theory of shape by space carving. IJCV, 2000.
[38] C.-H. Lin. C. Kong. and S. Lucey. Learning Efficient Point Cloud Generation for Dense 3D Object Reconstruction. AAAI, 2018.
[39] D. B. Lindell, J. N. P. Martel, and G. Wetzstein. AutoInt: Automatic Integration for Fast Neural Volume Rendering. CVPR, 2021.
[40] C. Liu, Z. Li, J. Yuan, and Y. Xu. NeuLF; Efficient Novel View Synthesis with Neural 4D Light Field. Eurographics Symposium Rendering, 2022.
[41] I. Liu, J. Gu, K. 7. Lin, T.S. Chua, and C. Theobalt. Neural Sparse Voxel Fields. NeurlPS, 2020.
[42] S. Liu, S. Saito, W. Chen, and H. Li. Learning to Infer Implicit Surfaces without 3D Supervision. NewrIPS, 2019.
[43] S. Liu, Y. Zhang. S. Peng. B. Shi. M. Pollefeys, and Z. Cui. DIST: Rendering Deep Implicit Signed Distance Function with Differentiable Sphere Tracing. CVPR, 2020.
[44] I. Loshchilov and F. Hutter. Sgdr: Stochastic gradient descent with warm restarts. ICLR, 2017.
[45] W. Matusik, C. Buchler, R. Raskar, S. J. Gortler, and L.. McMillan. Image-based visual hulls. SIGGRAPH,
2000.
[46] L. Mescheder, M. Oechsle, M. Niemeyer, S. Nowozin, and A. Geiger. Occupancy Networks: Learning 3D Reconstruction in Function Space. CVPR, 2019.
[47] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng. NcRF: Representing Scenes as Neural Radiance Fields for View Synthesis. ECCV, 2020.
[48] T. Müller, A. Evans, C. Schied, and A. Keller. Instant Neural Graphics Primitives with a Multiresolution Hash Encoding. SIGGRAPH, 2022.
12

---

[49] R. Mur-Artal, J. M. M. Montiel, and J. D. Tardos. ORB-SLAM: a versatile and accurate monocular SLAM system. TRO, 31(5), 2015.
[50] T. Neff. P. Stadlbauer, M. Parger, A. Kurz, C. R. A. Chaitanya, A. Kaplanyan, and M. Steinberger. DONeRF: Towards Real-Time Rendering of Neural Radiance Fields using Depth Oracle Networks. EGSR, 2021.
[51] M. Niemeyer, J. T. Barron, B. Mildenhall, M. S. M. Sajjadi, A. Geiger, and N. Radwan. RegNeRF: Regularizing Neural Radiance Fields for View Synthesis from Sparse Inputs. CVPR, 2022.
[52] J. Ost, I. Laradji, Newell, Y. Bahat, and F. Heide. Neural Point Light Fields. CVPR, 2022.
[53] O. Ozyesil, V. Voroninski, R. Basri, and A. Singer. A Survey of Structure from Motion. Acta Numerica, 2017.
[54] J. J. Park, Florence, Strauh, R. Newcombe, and Lovegrove. DeepSDF: Learning Continuous Signed Distance Functions Shape Representation. CVPR, 2019.
[55] S. Peng, M. Niemeyer, L. Mescheder, M. Pollefeys, and A. Geiger. Convolutional Occupancy Networks. ECCV, 2020.
[56] D. Rebain, W. Jiang, Yazdani, K. Li, K. M. Yi, and Tagliasacchi. DeRF: Decomposed Radiance Fields. CVPR, 2021.
[57] C. Reiser, S. Peng, Y. Liao, and A. Geiger. KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPs. ICCV, 2021.
[58] E. Remelli, A. Lukoianov. S. R. Richter, B. Guillard, T. Bagautdinov, P. Baque, and P. Fua. MeshSDF: Differentiable Iso-Surface Extraction. NewrIPS, 2020.
[59] M. Roberts and N. Paczan. Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding. ICCV, 2021.
[60] S. Saito, Z. Huang, R. Natsume, S. Morishima, A. Kanazawa, and H. Li. PIFu: Pixcl-Aligned Implicit Function for High- Righ-Resslution Clothed Human Digitization. ICCV, 2019.
[61] M. S. M. Sajjadi, H. Meyer, E. Pot, U. Bergmann, K. Greff, N. Radwan, S. Vora, M. Lucic, D. Duckworth, A. Dosovitskiy, J. Uszkoreit, T. Funkhouser. and A. Tagliasacchi. Scene Representation Transformer: Geometry- Novel View Synthesis Through Set-Latent Scene Representations. CVPR, 2022.
[62] L. Schonberger and J.-M. Frahm, Structure-fromd frot-Motinn Revisited. CVPR, 2016.
[63] V. Sitzmann, J. N. P. Martel. A. W. Bergman, D. B. Lindell, and G. Wetzstein. Implicit Neural Representa- tions with Periodic Activation Functions. NewrIPS, 2020.
[64] V. Sitzmann, S. Rezchikov, W. T. Freeman, J. B. Tenenbaum, and F. Durand. Light Field Networks: Neural Scene Representations with Single-Evaluation Rendering. NeurIPS, 2021.
[65] V. Sitzmann, M. Zollhöfer, and G. Wetzstein. Scene Representation Networks: Continuous 3D-Structure- Aware Neural Scene Representations, NeurlPS, 2019.
[66] L. Smith and Topin. Super-convergence: Very fast training of neural networks using large learning rates. Artificial intelligence and machine learning for multi-domain operations applications, 2019.
[67] S. Song, F. Yu, A. Zeng, A. X. Chang, M. Savva, and T. Funkhouser. Semantic Scene Completion from a Single Depth Image. CVPR, 2017.
[68] M. Suhail, C. Esteves, L. Sigal, and A. Makadia. Light Field Neural Rendering. CVPR, 2022.
[69] T. Takikawa, Litalien, K. Yin, K. Kreis, C. Loop, D. Nowrouzezahrai, A. Jacobson, M. McGuire, and
S. Fidler. Neural Geometric Lovel of Detail: Real-time Rendering with Implicit 3D Shapes. CVPR, 2021.
[70] J. Tang, X. Han, J. Pan, K. Jia, and X. Tong. A Skeleton-bridged Deep Learning Approach for Generating Meshes of Complex Topologies from Single RGB Images. CVPR, 2019.
[71] M. Tatarchenko, A. Dosovitskiy, and T. Brox. Octree Generating Networks: Efficient Convolutional Architectures for High-resolution 3D Outputs. ICCV, 2017.
[72] A. Trevithick and B. Yang. GRF: Learning a General Radiance Field for 3D Representation and Rendering.
ICCV, 2021.
13

---

[73] B. Wang, L. Chen, and B. Yang. DM-NeRF: 3D Scene Geometry Decomposition and Manipulation from 2D Images. ICLR, 2023.
[74] B. Wang, 7. Yu, B. Yang, J. Qin, T. Breckon, L. Shao, N. Trigoni, and A. Markham. RangeUDF: Semantic Surface Reconstruction from 3D Point Clouds. arXiv:2204001138, 2022.
[75] H. Wang. J. Ren, Z. Huang, K. Olszewski, M. Chai, Y. Fu, and S. Tulyakov. R2L: Distilling Neural Radiance Field to Neural Light Field for Efficient Novel View Synthesis. ECCV, 2022.
[76] N. Wang, Y. Zhang. 7. Li, Y. Fu, W. Liu, and Y.-G. Jiang. Pixel2Mesh: Generating 3D Mesh Models from Single RGB Images. ECCV, 2018.
[77] P. Wang, L. Liu, Y. Liu, C. Theobalt, T. Komura, and W. Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. NeurIPS, 2021.
[78] P. Wang, Y. Liu, G. Lin, J. Gu, L. Liu, T. Komura, and W. Wang. Progressively-comeeeed Light Field Network for Efficient View Synthesis, arXiv:2207.04466 2022.
[79] Q. Xu, W. Wang, D. Ceylan, R. Mech, and U. Neumann. DISN: Deep Implicit Surface Network for High-quality Single-view 3D Reconstruction. NeurlPS, 2019.
[80] B. Yang, S. Rosa, A. Markham, N. Trigoni, and H. Wen. Dense 3D Object Reconstruction from Single Depth View. TPAMI, 2019.
[81] B. Yang, S. Wang, A. Markham, and N. Trigoni. Robust Attentional Aggregation of Deep Feature Sets for Multi-view 3D Reconstruction. IJCV, 128:53-73, 2020.
[82] A. Yu, S. Fridovich-keil, and A. Kanazawa. Plenoxels: Radiance Fields without Neural Networks. CVPR, 2022.
[83] S. Zakharov, W. Kehl, A. Bhargava, and A. Gaidon. Autolabeling 3D Objects with Differentiable Rendering of SDF Shape Priors. CVPR, 2020.
[84] A. Zeng. Song. M. Niessner, M. Fisher, Xiao, and T. Funkhouser. 3DMatch: Learning Local Geometric Descriptors From RGB-D Reconstructions. CVPR, 2017.
[85] C. Zhang, Z. Cui, Y. Zhang, B. Zeng. M. Pollefeys, and S. Liu. Holistic 3D Scene Understanding from Single Image with Implicit Representation. CVPR, 2021.
[86] S. Zhi. Laidlow, S. Leutenegger, and A. Davison. In-Place Scene Labelling and Understanding with Implicit Scone Representation. /CCV, 2021.
[87] C. Zou, E. Yumer, J. Yang, D. Ceylan, and D. Hoiem. 3D-PRNN: Generating Shape Primitives with Recurrent Neural Networks. ICCV, 2017.
14

---

A
Appendix
A.1 Network Architecture and Training Details
We provide the details of our ray-surface distance field and the auxiliary dual-ray visibility classifier.
A.1.1 Ray-surface Distance Field
We use a 13-layer SIREN [63] with 1024 hidden units learn raa--suffcce distance field. We set the batch size 8192 and train the network for 10 epochs with the Adam [35] optimizer and cosine annealing strategy [44] with the learning rate initialized as 10 and decayed 10-8.
To learn a radiance field, we take an additional 2-layer SIREN with 1024 hidden units the radiance branch and output RGB values with sigmoid activation. For optimization together with the distance field, we adopt the same multi-view consistency as the distance field with mean-squared loss and the loss weight
Radiance Branch:
A.1.2 Dual-ray Visibility Classifier
We take a SIREN layer as the ray-feature encoder g() and another SIREN layer as the point-feature encoder kO. followed by a 7-layer SIREN with 512 hidden units and output the visibility with sigmoid activation. Both the ray-feature encoder and the point-feature encoder are with 512 hidden units. The batch sizes are 2048 for Blender dataset and ScanNet dataset, and 1024 for DM-SR dataset. We train the network for epochs with the Adam optimizer and a cycle annealing strategy [66] with the maximum learning rate 10
A.1.3 Training Data Generation
For scan with extrinsic parameters [R|t] and intrinsic parameters K, all rays of this scan start from the camera position o (2). The ray orientation of pixel (u, v) can be denoted as mn = RK--(()) (G).. and the unit ray direction is mo/ |mol]- Given oriented ray starting from o and a pre-defined sphere with a diameter D, we follow the two-sphere parameterization [8] to compute two intersections p" paur and construct our input parameters = (@ ,6" 6odd ,
To obtain the ray-surface distance from the raw depth value d of pixel (u, we have d = do. where f,cy, C2 are the camera intrinsic parameters (i.e., focal length, the center coordinates at y-axis and x-axis of image plane), and do = ||p" -ol--
(v + (,(=-c)2++(6
For a ray r at the pixel (u, v) of a specific scan, we have the first ray ray-sphere intersection p" and ray direction m, then we can compute the ray-surface point p p" + dm. To reproject this point the remaining (K - 1) scans, for example, we calculate the pixel coordinate (I² 1²) of the 1 scan using its extrinsic parameters [R|1] and intrinsic parameters Kt:
(5)
Then we can query the raw depth value at (u²,, v²) from the 4hhh scan, and obtain its ray-surface distance " and the ray parameters r "t (simplified as d A) following the previous steps. Besides, calculate the multi-view distance d between the ray-surface point and the first intersection D of ray r²: dh ||p 1. We set closeness threshold 10 millimeters to determine whether the projected 1thh ray is visible:
{1 0 otherwise
if lat d S
For both training and inference, the values of input parameters are normalized to be [-1, 1]. In particular, the latitude 0 = 20 / - and the longitude @ == / /. The input surface hitting point (xy 21) of our visibility classifier is also normalized to [-1, 1] by subtracting the sphere center and then dividing by the sphere radius D/2. The ground-truth ray-surface distance and the
multi-vics view distance d are scaled to be [0, 1] by dividing by the sphere diameter D.
15

---

A.2 Derivation of Surface Normal
In this section, we derive the formula surface normal from our ray-surface distance field.
As mentioned in Section A. given a ray starting from o with direction m and a fix-sized sphere with a diameter D centered at the coordinate origin, we can construct the input parameters (e 6tn gow x mui) from a ray-sphere intersect functiont D) =r.
By converting m to the spherical coordinate system, we have år and år which is the gradient of the above intersection function F() w.E.f. the ray direction m.
d8m d6m"
As shown Figure 6, to compute the surface normal of a predicted surface point along the ray direction m. we define sphere of radius R Dd + dg centered the camera position using spherical coordinates:
D mm mm)
(Rsin Om cos @m. Rsin On sin @m. Coos 0m)
30 dbm
ath dum 200 dr
In general, the formula for a unit normal vector is n
From Eq. 7, we have
aie J0m
ad cos om
(d 4o sin dm sin On
70m
D)
( dd sin 6m + (d $ cos dm sin 6m 70m
(8)
pm
ad cos Pm d0m
( dd sin 9m (d do 70
D
cos 6m cos Pm
2Φ
D
dd sin Pm (d do cos 6m) sin @m
2 mm
70
dd cos 6m (d + D sin 08m
dg
dd ad år
and dd dd dr the surface normal be denoted as
Here, replace 70m är 28m
dam dr d0m
function Q 2 D.d)..
A.3 Additional Implementation Details
In this section, we provide details of implementation and some minor adaptations of baselines, as well as the workflow for 3D shape reconstruction.
A.3.1 Baselines
Dd
9
OF/DeepSDE/NDFF Pre-processing on mesh before training is required for OF/DeepSDF/NDE, For fair comparisons, we use all depth images from training views reconstruct the training mesh which is then used for sampling occupancies/signed dis- tances/unsigned distances follow the official settings of hyperparameters and network ar- chitecture to train on the three datasets. For inference, we apply sphere tracing for DeepSDF and NDF to render depth images from testing views. The sphere tracing will stop after 100 iterations or when the predicted distance is smaller than a threshold at 0.005 meters. To avoid missing the intersections, marching with a f(p) with a = 0.6. Since sphere tracing for OF can only take fixed step size (e.g.. 0.005 meters), it requires about 1000 iterations to march the surface, which is time-consuming We thus construct a testing mesh by querying a value grid of occupancies and use ray-mesh intersections to render depth images for evaluation.
n
n.
1
256 voxel grid for training. We
Figure Sphere for surface normal
computation.
follow the damped sphere tracing [14] from NDF, i.e.,
NeuS/DS-NeRF: In general, we follow the same network architecture, training schedule, and color supervision from NeuS and DS-NeRE The vanilla DS-NeRF adopts sparse 3D points from structure- from-motion (SFM) and uses the reprojection error depth supervision. For fair comparison, we provide the same dense depth supervision from training views as ours on all three datasets for both
16

---

NeuS and DS-NeRK D..N For the experiments of Group 1, i.e., 3D shape representation only, we remove the color supervision of NeuS and DS-NeRF and leave depth supervision only.
We adopt the same network architec- ture and hyperparameters from the official settings of LFN/PRIF. In particular, we train LFN/PRIF for 10 epochs on each dataset, which the same as ours. Since the proposed depth computation from LFN makes it difficult ify the output of the last MLP layer to predict depth value with supervision. For the experiments of Group 2, i.e., 3D Shape and Appearance Representations, we extend 2-layer color branch both for I.FN and PRIF with appearance supervision. Note that, the LFN/PRIF results are obtained without our dual-ray visibility classifier. We additionally conduct experi- ments on LFN/PRIF with our own classifier for com- parison. Figure and Table 10 show that our method still achieves the best performance.
LFN/PRIF:
LFN Our Visibility Classifier
PRIF - Visibility Classifier
Ours
obtain dense depths, we mod-
Figure 7: Qualitative results of LFN/PRIF with our visibility classifier.
A.3.2
3D Shape Reconstruction for Evaluation
Table 10: Quantitative results of LFN/PRIF with our visibility classifier.
For fair comparison, initially render depth im- ages of testing views for all baselines and ours. For OF/DecpSDF/NDF,
CD mean / median
ADE
set the depth value to 0 for those unconverged rays (i.e., no intersection during sphere tracing). For PRIF and ours, apply spe- cific outlier removal strategies. Subsequently, we use TSDF fusion the processed depth images obtain complete meshes for further evaluation.
LFN + Our Visibility Classifier PRIF Visibility Classitier
90.82
121.289/ / 44.564
RayDF
A.4
Additional Ablation Studies
In this section, show more details of ablation studies, including ablations dual-ray visibility classifier, ray-surface distance network, training strategies, and post-processing.
A.4.1 Ablation Studies on the Dual-ray Visibility Classifier
The key issue of a naive ray based distance function the lack of generalization ability across novel views during testing. fundamentally because training a single ray based network (without classifier) tends fit all ray distance data pairs of the training set, but cannot guarantee the consistency of surface distances between seen (training) rays and unseen (testing) rays. As demonstrated in Tables 11, 12, 13 and Figure 8, the absence the classifier significantly impacts testing performance, resulting in notably inferior results compared to those optimized with the well-trained classifier. To provide better clarity, we also present visualizations of the visibility predictions Figure 9, Moreover, the results of our classifier with noises in Tables 14, 15, 16 and Figure 10 demonstrate that our visibility classifier is reliable for multi-view consistency.
w/o classifier one-stage
Ours
Novel view
w-O 000,0000.000000 6.2m
Mesh
670.872..-6600 6.07mm
Man 6004--
Figure 8: Qualitative results RayDF without visi- bility classifier and one-stage trained RayDF.
Figure 9: Visualization visibility predictions, w denotes visibility, d denotes distance difference.
17

---

Table 1 I:: Ablation study on the visibility classifier. Accuracy (%)) and F1-Score (%)} on 8 scenes and the average scores of the Blender dataset.
Chair
Drums
Ficus
Hondoy
Lego
Materials
Mic
Ship
Avg.
Acc. F1
Acc.
F1
Apc.
F1
Avc.
F]
Apc.
F]
Ace.
F1
Acc.
Aec.
F1
Ace.
F1
weo classifier
in
91.82
81.87
93.11
74.98
51
81.53
82.81
85.06
71.67
(3) remove function
91.37 91.55
80.05
78.51
61.10
18.07
93.33
91.14
81,55
75.
80.55
60.55
82.27
82.88
81.71
$2.32
81.93
76.77
(4) use ((P) it gtPs)
97.56 64.57
93.96
54.01
95.53
23.75
96.31 51.63
93,82
52. 37
96.04
31.77
97.56 64.04
89.67
59.24
95.06
50.17
Dual-rny Visibility Classilier
92.92 93.22
84.98
84,73
70.34
61.3
9361
94.67
87.74
85.24
89.95
81.88
87.00 87.78
84.16 K5.26
86.34
84.23
Table 12: Ablation study on the visibility classifier. ADEL on scenes and the average score of the Blender dataset.
Chair
Drums
Ficus
Hotdog
Lego
Materials
Mic
Ship
Avg.
(1) wio classifier ño
30.39
66,12
54,93
23.92
32.19
84.10
52.54
32.46
47.08
(2) function
6.14
4.15
12.49
9.19
13.15
(3) remove function &
6.24
11.57
15.39
4.08
12.08
10.42
8.64
13.45
10:23
(4) use g(rs) (D ((2))
1.89
11.09
14.29
4,70
11.01
7.91
8.84
13.85
9.77
Foll RayDF
4.59
9.24
3,06
7.55
13,07
Table 13: Ablation study on the visibility classifier. CD (x 10 the average score of the Blender dataset.
(x110-)4 3)ا (mean median) on 8 scenes and
(Toak dlausiiee 3
110 raew 277
1130 113. Log.
2009 2010
1050 oc0y So1
1879 Ste/
1344 S9a
1007 0192.773
1338 via
113
964 5.999 (USKS
Chamanee d
332 F0L437
5.8001.00 1.37
7,577 BUTT
3,000 B364:
2567 /0.55
2613/1.01 0.000
4.04/ FUSTT
4755 74 B50
Pell Hag
86777000 00.312
3.380/0899
2894/10.43
1,255 8.733
1,855 8,702
1.04 (0.67
3.54 0,712
3,38 /8.663
Table 14: Ablation study on adding noises to the well-trained visibility classifier. Accuracy (%)f and F1-Score (%)f on 8 scenes and the average scores of the Blender dataset.
Noise level (a')
Chair
Drums
Ficus
Hotdog
Lego
Materials
Mic
Ship
Avp.
Acc.
Fl
Acc.
F1
Acc.
F1
Acc.
F1
Acc.
F1
Acc. Fl
Acc.
F1
Acc.
F1
Acc.
Fl
1.0
65,04 66.57
61.75 63.31 55.19
51.58
65.91 70.55 62,56
59.06 63.79 50.11 62.92
65.24
62.92
65.24
62.51
61.46
0.5
70.46 71.61
65.99
66.82 $7.04
51.57
71.64
75.67
63,99
67.76 68.75
$3.99
67.59
69.42
67.59
69.42
66.63
65.78
0.1
85,02 85.37
77.31
76.32
61.78
49.78
86.96
88.96
68.01
72.59
81.98
66.75
80.23
81.01
75.44
76.08
77.09
74.61
(RayDF)
92.92
93.22
84.98 84.73 70.34 61.83 93,61
94.67
87.74
81.08
87.00
87.78
85,26
86.34
84.23
Table 15: Ablation study on adding noises to the well-trained visibility classifier. ADE on 8 scenes and the average score of the Blender dataset.
Noise level (o")
Chair
Drums
Ficus
Hotdog
Lego
Materials
Mic
Ship
1. 0.5
10.31
17.08
17.07
18.21
23.09
19.02
11.33
20.09
17.03
9.11
15.67
16.28
16.33
21.16
15.92
10.60
19.24
15.54
0.1
7.05
12.91
15.04
11.92
16.83
10.51
9.46
17.39
12.64
0 (RayDF)
4.59
9.24
12.56
3.06
7.98
5.68
7.55
13.07
7.97
Table 16: Ablation study on adding noises to the well-trained visibility classifier, CD (x (x10-)) (mean/ / median) on 8 scenes and the average score of the Blender dataset.
Noise (a+))
(at)
Chair
Drums
Ficus
Hotdog
Lego
Materials
Mic
Ship
Avg.
1.0
3.954/ 0.329 3,353 / 1,299 3.068/ 0.630 7.474/0.963 2.844/ 0.700 3,310/ 1,045 2.841 /1.564 5.415/0.943
4.032 0:935
0.
3185 0.287 3.87 1,221
2.793/ 0.533 7.080/ 0.933
2.313 0.658 2:923 (9959
2817/1.42
0.933
3,716/0
0.1
3,508 0.276 3.183/ 1.103 2.395 / 5-5888 6.730 0.857
L.739/ / 0.622 2,840/ 1.012
2.883/ 1.510
/
3.154/0.850
0 (Fu
3.807/0.30
3.310/0.958 2.91400544 7.255 (0,733 1,095 /0,702 1,604/00 0.617 3.571 /0.64
3.544/0.792
3,388 0.663
A.4.2 Ablation Studies on the Ray-surface Distance Network
We perform the following ablation studies on the ray-surface distance network.
(1) Multi-view Rays Sampling: Here we report the detailed quantitative results of each scene on Blender dataset in Tables 17 and 18.
(2) Network Architecture: We conduct additional experiments on the Blender dataset as shown in Table 19. We can see that our framework tends to favor wider deeper network. However, to extensively explore an optimal network architecture is non-trivial and we leave it for our future work.
(3) Multi-view Consistency Loss f In our multi-view consistency loss function (Eq. 3), we simply change it from & to Ea optimization, and conduct an additional experiment on Reception of DMSR dataset. From Table 20 and Figure 12, we can see that & based loss function can better reconstruct thin structures but with more noisy results than E based loss. In this regard, it of great interest to fully address the recovering task of thin structures and we leave it for future exploration.
18

---

Ours(
10%
Ours (109%)
Dinusee
Traia view
Novel view
Mesh
Figure 10: Qualitative results of RayDF using visi- bility classifier with noises.
Figure 11: Qualitative results of using sparse depth values in supervision.
Discance
w/o post-proesssing
poss-procsssing
Nrumal
Figure 12: Qualitative results of RayDF on Reception in DM-SR dataset. Red: RayDF-I1, Green; RayDF-I2.
Figure 13: Qualitative results of the predicted mesh by RayDF with and without post-processing.
Table 17: Ablation study on the number of multi-view rays sampling. ADE on scenes and the average of the Blender dataset.
M
Chair
Drams
Ficus
Hotdog
Lego
Materials
Mic
Ship
Avg.
4.89
9.82
13.27
3.14
8.27
6.44
7.71
13.29
10.12
20 (FayDF)
4.59
9.24
12.56
3.06
7.98
5.68
7.55
13.07
7,97
4.46
8.84
12.05
2.90
7.78
5.37
7.48
12.91
7.72
Table 18: Ablation study on the number of multi-view rays sampling. CD (x10-3) 9) (mean / median) on scenes and the average score of the Blender dataset.
Chair
Drams
Ficus
Hotdog
Lego
Materials
Mic
Ship
10
6.341 /0.328 5,680/ LOST 5.789 / /0.63 7.052 / 0.697
1.235/0. /.7334 1.743 /0.675
3.724/ 1 0.789
4.190/0.799
4.469 /0.713
20 (RuyDF)
3.807 0.312 3.310 0.958 2.914/ 0.544 7.255 /0.733
1.095 0,702
1.604/ /0611
3.571 0.644
3.544 0.792
3,388 0.663
40
3,445 /0.314 2.395 / 0.865 2.666 / 0.508 /0.742
1,080 0.672
0,918 0.531 1.508 / 0.579 2.463/ 0.758
2.711 0.621
Table 19; Ablation study the network architecture ray-surface distance network.
Table 20:
Ablation study the multi-view consis- tency loss function on Reception in DM-SR dataset.
CD (x10-=) mean
CD (x 10 mean / median
ADE
ADE|
median
layers, 512 units layers. 1024 units 13 layers, 512 units
9.13
4,311 / 0.954
6.56
12.141 6.853
8.52
4.010 0.805
(RayDF)
6.96
10.632 5.714
8.80
4.085 0.854
13 layers, 1024 units (FayDF)
7.97
3,388 0.663
A.4.3 Ablation Studies on Training Strategies
We additionally provide experimental results of our method using sparse depth values supervision in Tables 21, 22 and Figure 11.
(1) Sparse Depth Supervision:
(2) One-stage RayDF: We present detailed quantitative and qualitative results of one-stage training of our pipeline, the performance drops noticeably as shown in Tables 23, 24 and Figure 8.
A.4.4 Ablation Studies on Ray Parameterization
(1) Choices of Ray Parameterization: We discuss our ray parameterization in detail in Section
A.1.3. fact, our pipeline supports various ray parameterizations. To verify this, we conduct ablation studies by replacing our spherical coordinate with the one used in LFN/PRIF. As shown in Table 25 and Figure 14, PRIF parameterization can also achieve satisfactory performance, while LFN parameterization is inferior in our pipeline.
19

---

Table 21: Ablation study on sparse depth supervision. ADE., on 8 scenes and the average score of the Blender dataset.
Depth Sparsity
Chair
Drums
Ficus
Hotdog
Logo
Materials
Mic
Ship
Avg.
1%
4.29
11.06
15.06
3.21
8.69
4.84
8.33
12.87
8.54
14.44
4.58
8.44
8.70
10%
4.85
10.15
14.36
3,99
8.06
4.78
8.22
15.01
8.68
100% (RayDF)
9.24
12.56
3.06
7.98
5.68
7.55
13.07
7.97
Table 22: Ablation study on sparse depth supervision. CD (x (x10-3)4 (mean median) on 8 scenes and the average score of the Blender dataset.
Depth Sparsity
Chair
Drums
Ficus
Hotdog
Lego
Materiils
Mic
Ship
Avg.
1%
3.625 0.499 3,276 LO00 2.365 10,544 6.844 00.912
1.517/0.917
2.446/ 1,48 2,922/ 0022/1.55 11556 3.415 3.419/
9.9937
5%
3,536 / 0.348 3.067/ / L.77 2.523/ / 0.578
6,577 10.875
1.522/0.928
2/422/ 1.091 2.927/ / 1.532
3.623/ /0,930
3,250 0.920
10%
3,486 / 0.3311 3.068 / 088 2.M07 / 0.593 6,867 0.887
1,503 00.3/00.552
2,381 / 17776 2.922/
1,533
3,909 00,934
3,313 /09924
100%
(RayDF)
3,807 3.310/
2.914/
1.09500.700
16604 0.617
0.644
3.544/0.792
3.388/ 0.663
Table 23: Ablation study on onc/two-staee training. ADE on 8 scenes and the average score of the Blender dataset.
Chair
Drums
Ficus
Hotdog
Lego
Materials
Mic
Ship
Avg.
one-stage twu-stuge (RayDF)
6.69
12.35
17.37
10:94
15.88
9.20
9.27
17.61
12.41
4.59
9.24
12.56
3,06
7.98
5.68
7.55
13.07
7,97
Table 24: Ablation study on ote/two-stage training. CD (x 10 }) (mean median) on 8 scenes and the average score of the Blender dataset.
Chair
Drums
Ficus
Hotdog
Lego
Materials
Mic
Ship
Avg.
ono-stige 4.771 / 0.201 3.499/ / 1.202 2.9% 0.310 10.068 0.557 1,515 / 0.616 2.875/ 1,830 2,881/ / 1.450 3.727 / 00.700 4.032/ 0.659
two-stnge (RayDF) 3.807 /0311 3,310/ .3110/0.958 0.958 2.914/0.566 7.255 10.733 1,095 /.700 1,644/ / (1617 3,571 / (..44 3.544/0.792
3.388 / 0.063
Table 25: Ablation studies on ray parameterization on Blender dataset.
CD (x 4
ADEL
mean median
LFN param. + (Ray-surface Distance Network Visibility Classifier)
89.53
82.102/ 10.298
PRIF param. (Ray-surface Distance Network + Visibility Classifier)
8.42
3.409 / 0.621
(L.NN param. + Ray-surface Distance Network) (Our param. + Visihility Classifier)
91.30
142.570 74.345
(PRIF param. Ray-surface Distance Network) + (Our param. Visibility Classifier)
8.44
3.526 0.653
RayDF
7.97
3.388 0.663
Ray-surface
Ray-surface Distance Network + (Our Param. + Visibility Classifier)
Ray-surface
Ray-surface Distance Network Visibility Classifier
Distance Network + (Our Param, + Visibility Classifier)
Distance Network
Ours
Visibility Classifier
LFN Param.
PRIF Param.
Figure 14: Qualitative results using different ray parameterizations.
(2) Artifacts from Spherical Coordinate: The artifacts of fixed patterns in the predicted distance maps (as shown in Figure 15 (red)) are caused by the choice of spherical coordinate. We conduct additional experiments given two different choices of ray parameterizations. As shown in Table 25, our applied spherical coordinate can achieve the best ADE scores, and the caused artifacts can be easily fixed by using PRIF's ray parameterization as shown in Figure 15 (green). We leave it for our future work to fully address such an issue
20

---

Table 26: Ablation study on post post-processing. CD (x10- (mean / median) 8 scenes and the average score of the Blender dataset. Post-pmoessing
Chair
Drams
Ficus
Hordog
Lego
Materials
Mic
Ship
Avg.
4.378 / 0.203 3/14/ L,58 4.539/ 0.236 9.846 0..00 1.749 / 0.832 2,546 1,159 2,988/ / 1.540
3,829 / 0.678
4.157 /07777
3,807 /0.312 3,310/ 0.958 2.914/04 00543 7.255/0.733 1,095/ 00,702 1.604 / 0.617 3,571 00.644 3,544 0.799
3,388 0,663
A.4.5 Ablation Studies on Post-processing
We follow the outlier removal step in Section 3.4 as post- processing to obtain clean 3D point clouds for further mesh reconstruction and evaluation. In all experiments, we set the outlier threshold as 5 for post-procces posteprocessig. We conduct additional experiments w/ and w/o the outlier removal step on the Blender dataset. In Table 26, the experiment without post-pre pros-pnnsssingg yields higher CD errors compared to the one with processocssnn.. The qualitative results are shown in Figure 13.
Ours
Ours PRIF Param.
Figure 15: Visualizations of distance predic-
tion using different ray parameterization.
A.5 Limitations of RayDF
Since RayDF takes the conventional spherical parameterization for the input ray, i.e., sphere is predefined to bound the target 3D scene, our RayDF with such a parameterization cannot represent the 3D scene with rays starting from a position inside the sphere. However, the proposed distance field with a visibility classifier of our RayDF is flexible to different ray parameterizations. How to integrate distinct parameterizations of input rays for different types of 3D scenes with our ray-surface distance field and the dual-ray visibility classifier can be future exploration direction.
A.6 Additional Quantitative Results
In this section, we report the detailed quantitative results of each scene on the three datasets.
In Tables 28 and 29, we show the quantitative results of the experiments in Group 1. Our RayDF achieves significantly better performance than the other baselines on most scenes, especially on complex scenes (e.g.. Drums, Ficus, and Ship). In Tables 30, 31, 32, 33 and 34, we report the geometry and appearance metrics in Group Our method achieves comparable performance with DS-NeRF for novel view synthesis while recovering the 3D shape with fewer outliers.
Blender:
DM-SR: In Tables 35 and 36, we report the quantitative comparisons in Group and our RayDF performs better than the other baselines on those indoor scenes with more complex objects. As shown in Tables 37, 38, 39, 40 and 41, our method still achieves high-quality results for novel view synthesis on all scenes in Group 2 experiments.
ScanNet: In Tables 42 and 43, our RayDF performs significantly better than the other baselines on the worldworld scenes. To represent these real-world 3D scenes with much more noise and complex geometry, our method outperforms all baselines while obtaining photo-realistic novel view synthesis results, as shown in Tables 44, 45, 46, 47 and 48.
Table 27:
Optimization time consumption (hours).
Moreover, we provide comparisons of optimization time with baselines in Table 27. We compare the average training time of our method and all baselines on each scene of Blender dataset, using single NVIDIA RTX 3090 GPU card and a CPU of AMD Ryzen 7. Since we apply a two-stage training strategy, our method is not as fast as baselines during optimization. Nevertheless, once our ray-surface distance network is optimized, it can achieve superior efficiency in rendering novel views, shown in Table 1.
Time
OF [46]
0.2
DeepSDF [S4]
0.6
NDF [14]
2.1
NeuS [77]
10.2
Ne-N [19]
22.8
LFN [64]
0.9
PRIF [23]
2.2
RayDF (Ours)
24.9
A.7 Additional Qualitative Results
We provide additional qualitative results on the three datasets follows, including distance map, mesh from Group 1, and radiance map, textured mesh from Group 2. We also report the visualizations of our surface normal, which is computed following Section A.2.
Blender: Figures 16 and 17, we show the qualitative results on Materials and Drams. In particular, Materials is scene that contains empty spaces between multiple objects. Our RayDF predicts
21

---

distance with fewer outliers near the object surfaces. Our surface normal indicates that our method recovers the 3D scene with smoother surface while preserving more details of the complex geometry.
DM-SR: In Figures 18 and 19, we report the qualitative results on Reception and Study. Our RayDF predicts accurate distances and hence is capable of reconstructing more detailed structures.
ScanNet: In Figures 20 and 21, we show the qualitative results on Scene0005_ 1_00 and Scene0030 which are challenging real-world scenes. Our method recovers the 3D real-world scenes with reasonable details while predicting realistic radiance results.
Distance
OF
DeepSDF
NDF
NeuS
GT
DS-NeRF
LFN
PRIF
Ours
Mesh
OF
DeepSDF
NDF
NeuS
GT
DS-NeRF
LFN
PRIF
Ours
Textured Mesh Raidance
GT
NeuS
DS-NeRF
LFN
PRIF
Ours
Normal
DecpSDF
NDF
NeuS
DS-NeRF
Ours
Figure 16: Qualitative results of all methods on Materials in Blender dataset.
22

---

Distance
OF
DeepSDF
NDF
NeuS
GT
DS-NeRF
LFN
PRIF
Ours
Mesh
OF
DeepSDF
NDF
NeuS
GT
DS-NeRF
LFN
PRIF
Ours
Textured Mesh Raidance
GT
NeuS
DS-NeRF
LFN
PRIF
Ours
Normal
DeepSDF
NDF
NeuS
DS-NeRF
Ours
Figure 17: Qualitative results of all methods Drums in Blender dataset.
23

---

Distance
OF
DeepSDF
NDF
NeuS
GT
DS-NeRF
LFN
PRIF
Ours
Mesh
OF
Dee
NDF
NeuS
GT
DSRN
LFN
PRIF
Ours
Raidance
GT
Textured Mesh
NeuS
DS-NeRF
LFN
PRIF
Ours
Normal
DeepSDF
NDF
NeuS
DS-NeRF
Ours
Figure 18: Qualitative results of all methods Reception in DM-SR dataset.
24

---

Distance
OF
DeepSDF
NDF
NeuS
GT
DS-NeRF
LFN
PRIF
Ours
Mesh
OF
DeepSDF
NDF
NeuS
GT
DS-NeRF
LFN
PRIF
Ours
Textured Mesh Raidance
GT
NeuS
DS-NeRF
LFN
PRIF
Ours
Normal
DccpSDF
NDF
NeuS
DS-NcRF
Ours
Figure 19: Qualitative results of all methods on Study in DM-SR dataset.
25

---

Distance
OF
DeepSDF
NDF
NeuS
GT
DS-NeRF
LFN
PRIF
Ours
Mesh
OF
DeepSDF
NDF
NeuS
GT
DS-NeRF
LFN
PRIF
Ours
Raidance
Textured Mesh
NeuS
DS-NcRF
LFN
PRIF
Ours
GT
Normal
DeepSDF
NDF
NeuS
DS-NeRF
Ours
Figure 20: Qualitative results of methods on Scene0005 00 ScanNet dataset.
26

---

Distance
OF
DeepSDF
NDF
NeuS
GT
DS-NeRF
LFN
PRIF
Ours
Mesh
OF
DeepSDF
NDF
NeuS
GT
DS-NeRF
LFN
PRIF
Ours
Textured Mesh Raidance
NeuS
DS-NeRF
LFN
PRIF
Ours
GT
Normal
DeepSDF
NDF
NeuS
DS-NeRF
Ours
Figure 21: Qualitative results of all methods on Scene(0030000 in ScanNet dataset.
27

---

Table 28: ADE scores of all baselines and our method in experiments of Group I on the 8 scenes of Blender dataset.
Chair
Drums
Ficus
Hotdog
Lego
Materials
Mic
Ship
Avg.
OF [46]
6.52
14.37
19.88
4.35
9.85
6.06
8.36
15.20
10.57
DeepSDF [54]
6.84
16.55
22.80
8.37
12.18
9.45
9.31
18.07
12.95
NDF [14]
7.83
14.93
17.26
7.92
11.42
11.61
9.77
16.39
12.14
NeuS [77]
6.87
10.61
19.02
3.36
8.43
7.43
13.92
25.37
11.88
DS-NeRF [19]
11.73
19.42
18.75
2.22
11.01
5.77
19.20
17.66
13.22
LFN [64]
16.33
28.09
31.56
14.02
20.27
36.93
20.89
27.65
24.47
PRIF [23]
7.65
19.57
25.73
5.83
12.86
15.69
13.35
16.72
14.68
DayDF (Ours)
4.59
9.24
12.56
3.06
7.98
5.68
7.55
13.07
7.97
Table 29: CD (x10 1x-10>1 -2 (mean median) scores of all baselines and our method in experiments of Group 1 on the 8 scenes of Blender dataset.
Chair
Druns
Ficus
Hosdog
Logo
Ship
Avg.
OF
1,491
10.435
0.554
Dea [54]
4253/0..55
2.324/1
6.80/0.554
8.42//0.72
1.600/0.766
0.252 00.89
LA35
3.
[14
/0994
2.187/11
2.215/0663
0.84
77.4//0.934
1.083
16657
2.821
2.976
NeuS [77]
3831 0331/00277
2.582/0.885
3.376/0.920
8.022/6.00
1,252 00,66
1,771 00.918
12:238 1.816
4,943 1.083
4,756 0.907
DS-NaRF [19]
220 220.546/0.685 0.683
207.953/2002
4.471 10:900
8,22/007113
259.806
(1955
0.735 (1.441
203.95:
1.556
30.520 1.401
117.270/ 1.135
LFN [64]
64.599/ 64.599/0.517
84.757 222
182.662
21 21.466/2.888
65.227
3.671
152.122
668.91
84.327
88.184
M11.24
4.952
89.4
02.425 15.681
PRIF [23]
12.528 /0.46
21.911 14.507
20.889/0.58
22.712/ .091
12. 12.088/ /0.56
27. 121 1221//200 /2.129
24. 01.37072220 2.664
24.450
/0.0000000000
20. 764 1.577
RayDF (Ours)
1,000 / /0,312
3.310/0.958
2.914/0.543
/.755/
1,095 0.7702
1,604 70.677
3:571 70.644
3.544 /..49/
3.888 0.663
Table 30: ADE scores of all baselines and our method in experiments of Group 2 on the 8 scenes of
Blender dataset.
Chair
Drums
Ficus
Hotdog
Lego
Materials
Mic
Ship
Avg.
NeuS [77]
7.15
10.97
18.78
3.28
7.97
7.53
13.95
27.13
12.10
DS-NeRF [19]
9.81
15.47
18.02
6.72
10.90
18.77
19.78
17.62
14.64
LFN [64]
6.70
15.46
20.28
4.40
13.67
12.35
10.85
14.95
12.33
PRIF [23]
7.51
19.20
25.25
5.37
13.02
16,24
13.29
16.57
14.56
DayDF (Ours)
4.92
9.46
12.85
2.96
8.11
6.23
7.54
13.25
8.17
Table 31: CD (x10-3)) -3 (mean median) scores of all baselines and our method in experiments of Group 2 the 8 scenes of Blender dataset.
Chai
Drnos
Hotdog
Lego
Matesials
Ship
Avg.
[7]]
2.655
3.230/00
1.175
DS- Na-N [19]
295.4457 1.931
(42.057/2 2 2.192
2
0.775
214.536/
1.603
0.9976
666//2000
2.072
4,697
247.558/3.80
143.295
7,760
LPN [64]
63.002 10.516
8,036
1,962
37.45%
0.880
14.783
0.755
120,065
U:992
71.93
1.367
134,498
2.648
29.944/ 1.122
60.259
1,230
PRIP 123]
14.12/00896
20,627
4.991
20.91/00307
20.297 0.889
12,963
0,897
26.497
2.308
24.919/
29.265 L.037
21.279
1,693
RayDF (Ours)
3,29 0.306
3,433 /1.278
3.419/0.587
7.253/0.755
1.372: 0.802
1.551 (1689
1.798 0,513
3.907 0.858
3.295 10755
Table 32: PSNR1 scores of all baselines and our method in experiments of Group 2 the scenes of Blender dataset.
Chair
Drums
Ficus
Hotdog
Lego
Materials
Mic
Ship
Avg.
NeuS [77]
27.47
23.23
30.11
32.82
25.85
24.97
29.81
23.24
27.19
DS-NeRF [19]
29.77
23.38
23.02
32,80
28.58
23.21
28.11
24.18
26.63
LFN [64]
25.30
19.96
21.04
29.69
21.58
20.79
26.32
20.95
23.20
PRIF [23]
24.72
20.07
22.69
28.85
21.65
20.00
26.01
22.46
23.31
DayDF (Ours)
27.56
23.30
29.16
31.58
25.92
22.03
28.32
24.30
26.52
Table 33: SSIM scores of all baselines and our method in experiments of Group 2 on the 8 scenes of
Blender dataset.
Chair
Drums
Ficus
Hotdog
Lego
Materials
Mic
Ship
Avg.
NeuS [77]
0.914
0,911
0.968
0.957
0.873
0.900
0.966
0.800
0.910
NS-NeRF [19]
0,947
0.976
0.963
0.934
0,940
0,966
0.805
0,933
LFN [64]
0.890
0.890
0.963
0.948
0.815
0.855
0.954
0.785
0.888
PRIF [23]
0.878
0.866
0,934
0.933
0.807
0.847
0.950
0.780
0.874
RayDF (Ours)
0.925
0.915
0,966
0.959
0.877
0.872
0.961
0.810
0.910
Table 34: LPIPS scores of all baselines and our method in experiments of Group 2 on the scenes of Blender dataset.
Chair
Drums
Ficus
Hotdog
Lego
Materials
Mic
Ship
Avg.
NeuS [77]
0.108
0.096
0.029
0.0668
0.1 14
0.080
0.037
0.250
0.100
DS-NeRF [19]
0.055
0.058
0.016
0.044
0.036
0.034
0.037
0.223
0.063
LFN [64]
0.140
0.117
0.032
0.074
0.187
0.117
0.063
0.267
0.125
PRIF [23]
0.157
0.181
0.075
0.090
0.207
0.149
0.062
0.292
0.152
RayDF (Ours)
0.100
0.093
0.030
0.063
0.125
0.103
0.039
0.239
0.099
28

---

Table 35: ADE scores of all baselines and our method in experiments of Group 1 on the 8 scenes of DM-SR dataset.
Bathroom
Bedroom
Dinning
Kitchen
Office
Reception
Restroom
Study
Avg.
OF [F[4
14.15
10.22
14.44
16.09
16.44
12.50
27.48
15.28
15.83
DeepSDF [54]
15.61
11.25
15.09
16.98
17.01
15.92
26.88
17.01
16.97
NDF [14]
19.27
12.02
22.24
23.92
21.91
20.53
37.93
21.44
22.41
NeuS [77]
7.77
4,34
9.57
12.12
6.94
17.43
13.08
8.28
9.94
DS-NeRF [19]
11.25
8.83
13.13
13.41
7.51
18.93
10.03
10.77
LPN [64]
14.71
16.16
8.11
25.07
16.58
17.20
23.88
24.65
18.30
PRIP [23]
12.06
5,06
10:22
14.10
12.74
10.15
18.71
12.10
11.89
RayDF (Ours)
6.24
5.30
6.96
7.41
Table 36: CD (x '% (mean / median) scores of all baselines and our method in experiments of Group 1 on the 8 scenes of DM-SR dataset.
Ruthmom
Dinning
Office
Reception
Study
Avg.
OF [46]
7,817 3 285
14.923
4.309
4.305 281
16,764
6.271
8939/ /3-45
9.391 151111
19.926 / 5.575
9.484 5.066
11 4027 4,888
DecpSDE [54]
6,929
3.441
14,601
1,888
4.555
2,505
15.881 7.119
/.7663
9.998 5,070
19.524
8:405
9.959 /5.501
11.281 / 5.087
NDF [14]
7,096 259
15.821
/4.784
4.386 3.102
17,745/7.73
7.737
9.745/4.472
10.468
5.432
22,141
10.802
12.300 10.999/6.662
NeuS [77
7.524 2.814 2.814
17.435/3.90
4.968
/2573
08.510/6.008
8.361
/3,25
11.481/
4.945
21,992/
/.192/8.144
4.880 11.684
12.744/ 4,620
NR-NeRF
9.58/ 3.601
16.893
/4.418
2.624
12.072
27.548/4400 /4034
28.64/55
46.375/9.550
/6.57
25. 150
8640/
20.162/7-927
9..17/37100
PRIF [23]
16.864/3.365
25.057/44
/4:437
19.999/
2.437
14.584/77444
33.728
26.009 5.160
32,373 K,452
29.733
25 93.9993/55
RayDF (Ours)
7,499/ 5.140
17.766 4.755
4.226 2,778
18,463/7 7.699
9.14/33628
10.632/5.714
20.698 7878
24.777 6.99
/
Table 37: ADEJ, scores of all baselines and our method in experiments of Group 2 on the 8 scenes of DM-SR dataset.
Bathroom
Bedroom
Dinning
Kitchen
Office
Reception
Restroom
Study
Avg.
NeuS [77]
11.31
4.87
9.75
14.23
8.94
17.32
17.43
9.45
11.66
DS-NeRF [19]
11.14
3,48
8.36
13.83
13.10
7.75
18.92
9.56
10.77
LHN [64]
13.90
16.21
7.98
24.92
17.36
19.25
24.02
24.47
18.51
PRIF [23]
12.02
5.06
9.32
13.96
12.54
10.21
18.85
12.18
11.77
RayDF (Ours)
6.97
4.29
5.65
11.25
7.14
8.30
11.56
8.60
7.97
Table 38: CD (x10- (mean / median) scores of all baselines and our method in experiments of Group 2 on the scenes of DM-SR dataset.
Bedroom
Dinning
Reception
Study
Arg.
NeuS [72] 7.389 2,938
18.841 4.064
4.844/2.00
33.349/
11.617
8.665
/32338
11.375 /44921
23.56
8,307
12.11 4,978
15.017/ 5.308
DS-NeRF [19]
9.552
17.0%
4.385
4 2.476
40.004
13.066
58,091
26,4001 5.244
/5.422/
$2.650
6.210
25
602
LFN [64]
8.324/3.325
19.550/4
791
4.984
/
21 34.443
7 158
10.024
3.073
11,081
5.720
20.770 / 8.516
16.191
5,844
14,085
5,349
PRIF [23]
23.450/4
302
24
24.105
2.368
24 322
1 452
27.253
3434
24 24.142/5
3.3142/53366
29.527 8.167
22,524
5.813
24.842 5.156
17.220
10.589/5.675
21,189
14.251/5.0 5.300
Table 39: PSNR1 scores of baselines and our method experiments Group the scenes of DM-SR dataset.
Bathroom
Bedroom
Dinning
Kitchen
Office
Reception
Restroom
Study
Avg.
NeuS [77]
33.73
37.71
29.98
33.97
36.21
30.69
31.08
32.39
33.22
DS- AS-N [19]
31.26
36.67
32.36
27.73
32.78
32.26
31.02
30.56
31.83
LFN [64]
30.77
33.82
32.29
30.48
32.61
29.47
29.08
28.35
30.86
PRIF [23]
31.15
35.09
31.62
30.89
32.71
29.89
28.19
28.57
31.01
RayDF (Ours)
27.98
31.07
31.23
25.03
35.91
29.27
31.47
30.57
30.32
Table 40: SSIM scores of all baselines and our method in experiments of Group 2 on the 8 scenes of
DM-SR dataset.
Bathrom
Bedroom
Dinning
Kitchen
Office
Reception
Restroom
Study
Avg.
NeuS [77]
0.972
0.978
0.919
0.973
0.981
0.966
0.965
0.963
0.965
DS-NeRF [19]
0.982
0.989
0.967
0.958
0.982
0.981
0.979
0.976
0.977
LFN [64]
0.953
0.879
0.924
0.913
0.930
PRIF [23]
0.951
0.961
(1.755
0.949
0,955
0,930
0.917
0.925
0.933
RayDF (Ours)
0.946
0.967
0.886
0.940
0.963
0.934
0.936
0.945
0.940
Table 41: LPIPS.. scores of all baselines and method in experiments of Group 2 the scenes of DM-SR dataset.
Bathroom
Bedroom
Dinning
Kitchen
Office
Reception
Restroom
Study
Avg.
NeuS [77]
0.038
0.037
0.126
0.052
0.028
0.052
0.054
0.043
0.054
DS-NeRF [19]
0,018
0.011
0,030
0.073
0.026
0.016
0.018
0.018
0.026
LFN [64]
0.088
0.095
0.201
0.099
0.064
0.107
0.110
0.122
0.111
PRIF [23]
0.083
0.080
0.212
0.092
0.069
0.112
0.131
0.111
0.111
RayDF (Ours)
0.092
0.217
0.107
0.113
0.093
0.113
29

---

Table 42: ADEL scores of all baselines and method in experiments of Group 1 on scenes of ScanNet dataset.
Scenabil Scene0004_00 00
Scene0005_00 _000
Secne0009_00 _00
Secne0010 000
Scene0030 00
Secne0031 .00
Avg.
OF [46]
11.76
DeepSDF [54]
19:47
11.86
10.64
9.92
14,60
5.94
12.07
NDF [14]
31.13
20.47
21.89
17.25
29.38
12.79
22.15
NeuS [77]
25.75
15.76
17.79
15.58
17.50
10.82
17.20
DS-NeRF [19]
10.85
6.01
6.42
7.11
5.67
3.75
6.64
[64]
5,58
PRIP [23]
13.98
9.36
9.66
9.17
12.54
7.21
10.32
RayDF
5.29
Table 43: CD (x (x10-))4 (mean / median) scores of all baselines and method in experiments of Group on 6 scenes of ScanNet dataset.
Scenc0004_00 _00
Scene0005_00 .00
Socne0009_00 _0
Scene0010.
00
Scene0030_00 _0
Socne0031 __000)
Avg.
OF [46]
15.039 5.642
11.443
1.655
5.157 2.638
1446/ 0.844
13,921
2,364
/.721
8.991/ 2.810
DecpSDF
65.574/
14.168
11.173/
7.344/2.618
1661
/
9.869
1.866
/2.842
17.200
6.255
NDF [14]
88.485 27.523
5.893 / 3.227
13.861 /64474
2.973/ 1,400
20.873
13.503/ 5.331
30.436/ 8,348
NeuS [77]
47.561 /9.331
55.437 /22791
25.592 3.288
15.536 .809
26.199 2.500
12.526 2,908
30.475 3.771
DS-NeRF
28.287 6.293
8.407 1.283
6.720 2,044
2.014 0.842
9.085
3,407 22004
2.318
LFN [64]
17.777 4,443
5.931 / 1.125
3,921 1.699
3.256 / 0,704
10.824
1.189
4.194
/.0026
7,651 / 1.864
PRIF [23]
62.527 7.660
/5,450/ 1.966
9.499 2.386
6.030 0.909
12,951
1.557
11.515
2.511/22.53
19. 19.662 2.828
RayDF (Ours)
33.872 4.804
5.755 0.963
4.863 1.724
0.567
11,931 1.100
5.842 1.930
10.517 1,488
Table 44: ADE,, scores of all baselines and our method in experiments of Group 2 on 6 scenes of ScanNet dataset.
Scem00004_000
Sccad1005__0
Secnel009_00
Scene(011_00
Scene0030_00
Scene0031_00
Avg.
NeuS [77]
33.43
24.05
24.18
25.84
26.57
10.67
24.12
DS-NeRF [19]
12.18
7.01
8.00
LFN [64]
10.75
5.49
6.17
7.34
5.87
5.00
6.77
RayDF (Ours)
8.49
4.16
5.22
5.14
4.48
4.39
5.31
Table 45: CD (x10-3) } (mean median) scores of all baselines and our method in experiments of Group 2 6 scenes of ScanNet dataset.
Scene0004 00
Scene0005__0 _.
Scene0009 (00
Scene0010_00 000
Scene00330.00
Scene0031_00
Avg.
NeuS [77]
73.825
11.200
109.468 / 8.325
46.224 4,593
27.426
2.857
43.060/3.475
104.02
3,500
67.504 / 5.658
NeRN [19]
39.89 7.466
6.186/ 1.487
7.052/2.17
2.170
0.992
11.084
1.484
5.434 22663
14.642 2.950
LFN
18.970
44588
3,838 1.104
3,901 1.697
2.882 00.700
157.685
2.888
4,290
2.048
31,928
2.12
PRIF [23]
69.959
7.888
22,100/ 1.412
10.498
/ 2,272
6.226 0,9948
16.469
1.679
12.162
2.622
22.902 2,804
RayDF (Ours)
35.163 5.502
6,273 0.988
5.658 1.782
0.590
10.231 1.150
6.165 2.040
10.739 2.009
Table 46: PSNR scores of all baselines and our method in experiments of Group 2 6 scenes of SeanNet dataset.
Secr00004_00
Sccne0005 Secne0009_00 000
Secne0010. 00
Scene0030 00
Scene0031 00
Avg.
NeuS
28.01
27.91
25.62
26.06
30.29
27.00
DS-NeRF [19]
24.50
24.42
25.26
24.70
25. 18.18
11
25.03
LFN [64]
26.91
28.71
29.12
27.96
28.27
27.87
28.14
PRIF [23]
20.83
20.64
22.45
20.03
21.06
21.80
21.14
RayDF (Ours)
30.61
33.48
31.31
32.10
32.37
29.59
31,58
Table 47: SSIM? scores of all baselines and our method in experiments of Group 2 on scenes of ScanNet dataset.
Scene1004_00
Scene0005_ .0
Scene0009 00
Secne0010. 00
SeeneC0330 00
Sccne(003 000
Avg.
NeuS 1771
0.826
0.837
0.769
0.835
0.806
0.831
0.817
DS-NeRF [19]
0,861
0.889
0.796
0.884
0.862
0,839
0.855
LFN [64]
0,794
0.770
0.858
0.817
PRIP [23]
0.724
0.785
0.711
0.729
0.731
0.700
0.731
RayDF (Ours)
0.912
0.797
0.903
0.870
0.856
Table 48: LPIPS., scores of all baselines and our method in experiments of Group 2 on 6 scenes of ScanNet dataset.
Scene0004_00
Scenel1005_00
ScenellOU9_0u
SceneUU10_Pg
Scene00330.0
SceneOki31_00
Avg.
NeuS [?7]
0,266
0.225
0.347
0.197
0.209
0.270
0,253
DS-NeRI
LFN [64]
0.326
0.191
0.345
0.171
0,206
0.334
0.262
PRIF
0.427
0.350
0.420
0.347
0.319
0.465
0.388
RayDF (Ours)
0.261
0.146
0.328
0.135
0.157
0.319
0.224
30


Reviews:
[{"review_id": "JRvBtx6dNX", "reviewer": "Reviewer_fezd", "summary": "This paper proposes a framework MucRays for 3D shape representation. Specifically, the authors formulate 3D shapes as ray-based neural functions and incorporate multi-view geometry consistency to improve the performance. For the learning of multi-view geometry consistency, an auxiliary network is introduced to classify the mutual visibility of two sampled rays. Experiments are conducted on various datasets, showing good performance.", "questions": "See weaknesses.", "limitations": "Negative impacts are mentioned in the paper.", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is easy to follow and understand.\n2. The authors propose an effective framework for 3D shape representation, showing good performance in various datasets.", "weaknesses": "1. This paper only compares rendering time, but training/optimization time is not compared.\n2. The ablation study is not convincing. The authors should compare with and without postprocessing (Section 3.2).\n3. Table 5, the performance of \"w/o classifier\" is quite worse than the full model, and the error is very high. It would be better to discuss some possible reasons.\n4. 2-stage training is not elegant. Is that possible to train these two networks simultaneously?\n5. Line 57, \"ray-surface distance field\" representation is introduced by previous works and cannot be summarized as one contribution of this work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's thoughtful comments and address the main concerns below.\n\n**Q1: This paper only compares rendering time, but training/optimization time is not compared.**\n\n**A1:** Thanks for the suggestion. The following Table compares the average training time of our method and all baselines on each scene of Blender dataset, using a single NVIDIA RTX 3090 GPU card and a CPU of AMD Ryzen 7. Since we apply a two-stage training strategy, our method is not as fast as baselines during optimization. Nevertheless, once our ray-surface distance network is optimized, it can achieve superior efficiency in rendering novel views, as shown in Table 1 in the main text. We will clarify this point in the next version.\n\n|                    | time (hours) |\n|--------------------|:--------------:|\n| OF                 | **0.2**      |\n| DeepSDF            | 0.6          |\n| NDF                | 2.1          |\n| DS-NeRF            | 22.8         |\n| NeuS               | 10.2         |\n| LFN                | 0.9          |\n| PRIF               | 2.2          |\n| **MucRays (Ours)** | 24.9         |\n\n**Q2: The ablation study is not convincing. The authors should compare with and without postprocessing (Section 3.2).**\n\n**A2:** Section 3.2 does not mention any postprocessing step. In Section 3.4, we discuss the removal of potential outlier 3D points with the aid of our derived closed-form surface normals. Note that, this postprocessing step is only applied to clean the reconstructed 3D point clouds when calculating CD scores. We do not use any postprocessing step when calculating all DAE scores. As shown in the following Table, we conduct additional experiments w/ and w/o the outlier removal step on the Blender dataset. Figure 2 in the submission file shows the qualitative results. We will slightly rephrase lines 197-198 in Section 3.4 to clarify how this simple postprocessing step can be used when obtaining explicit 3D point clouds. \n\n|                    | DAE$\\downarrow$| CD ($\\times10^{-3}$) $\\downarrow$ (mean / median)  |\n|--------------------|:---------:|:-----------------:|\n| w/o post-processing|**7.97** | 4.157 / 0.777   |\n| **MucRays (w/ post-processing)** | **7.97**| **3.388** / **0.663**|\n\n**Q3: Table 5, the performance of \"w/o classifier\" is quite worse than the full model, and the error is very high. It would be better to discuss some possible reasons.**\n\n**A3:** This is a very good point. The key issue of a naive ray based distance function is the lack of generalization ability across novel views during testing, fundamentally because training a single ray based network (without classifier) tends to fit all ray distance data pairs of the training set, but cannot guarantee the consistency of surface distances between seen (training) rays and unseen (testing) rays. \nHowever, with the help of our well-trained dual ray visibility classifier, the ray distance network must satisfy a transformation equation (Appendix A.1.3). This is easily achieved by our multiview consistency loss function $\\ell_{mv}$ (Eq. 3 of main text), which drives the consistency of surface distances between (unlimited) seen and unseen rays.  We show the qualitative comparisons in Figure 3 in the submission file.\n\n**Q4: 2-stage training is not elegant. Is that possible to train these two networks simultaneously?**\n\n**A4:** We agree that an ideal strategy is to train both networks simultaneously. As shown in the following Table and Figure 3 in the submission file, we simply train our two networks at the same time on the Blender dataset. However, not surprisingly, the performance of one-stage training drops noticeably, primarily because the classifier is inaccurate at the early stage and unlikely to provide effective constraints for the ray distance network given the similar number of training steps. Nevertheless, it is an interesting direction for our future work.\n\n|                    | DAE$\\downarrow$| CD ($\\times10^{-3}$) $\\downarrow$ (mean / median)  |\n|--------------------|:---------:|:-----------------:|\n| one-stage training |  12.41  | 4.032 / 0.659   |\n| **two-stage training** | **7.97**| **3.388** / **0.663**|\n\n**Q5: Line 57, \"ray-surface distance field\" representation is introduced by previous works and cannot be summarized as one contribution of this work.**\n\n**A5:** Thank you for this advice. We will rephrase lines 57-58 or alternatively combine lines 57-60 in the next version."}, {"review_id": "1YeEVNcEJr", "reviewer": "Reviewer_4cJL", "summary": "The paper proposed a ray-based neural rendering method that is able to achieve good reconstruction quality from depth maps or RGBD inputs using only one network evaluation per pixel during evaluation. \nThe method requires two networks: a ray-surface distance network, and a dual-ray visibility classifier. In the first stage, the visibility classifier is trained on the multi-view depth maps. In the second stage, the ray-surface distance network can be trained with both ground truth rays from the depth maps, and random sampled novel rays. the visibilities of the random rays are determined by te visibility classifier, and only the distance of the visible rays are supervised. The proposed approach is extremely fast, and achieves competitive quality on three RGBD datasets.", "questions": "There appear to be some fixed-pattern noise artifacts in the demo video. I wonder what is the cause?", "limitations": "The limitations and broader impacts of the works are adequately discussed in the paper.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The proposed two-stage method is novel.\n* The proposed method is extremely fast during evaluation -- only one network evaluation is needed per ray.\n* The method achieves competitive performance on both synthetic and real datasets.\n* The paper provided useful insights on the derivation of surface normals from the proposed ray distance field.", "weaknesses": "* The method requires ground truth depth information for training, which is not the case for light field works such as [57]. This can significantly limit its use cases.\n* There lacked comparisons with some good performing light field methods such as [57].\n* The method relies on the visibility classifier to generalize to novel rays, which can potentially be unreliable.\n* Ther performance of the proposed method is lacking, especially according to the results in the appendix.\n* It might help understand the effect of visibility network better if there are visualizations of the visibility network predictions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's thoughtful comments and address the main concerns below.\n\n**Q1: The method requires ground truth depth information for training, which is not the case for light field works such as [57]. This can significantly limit its use cases.**\n\n**Q2: There lacked comparisons with some good performing light field methods such as [57].** \n\n**A1-A2:** The primary goal of our method is to model 3D surface geometry, instead of learning radiance fields for novel view RGB rendering. LFNR [57] is an image-conditioned light field method for RGB rendering, which is dramatically different and not directly comparable with ours. Unarguably, both pipelines have their own merits and applications, and our method has great potential in robotics mapping, navigation, obstacle avoidance, etc.\n \nOur method indeed requires depth supervision in the current submission. However, we argue that, with the advancement of existing techniques of depth estimation from RGB images, it is quite feasible to obtain sparse depth signals using existing techniques such as SfM or learning-based monocular depth estimators. In this regard, we additionally provide experimental results of our method using sparse depth values in supervision. From the following Table which reports the metrics on *Lego* in Blender dataset, we can see that our method still achieves satisfactory performance even though there are only 1\\% depth values in training. We hypothesize that such robustness comes from our simple multiview consistency constraint, because many depth values in the training set may be redundant thanks to our effective classifier. Figure 4 in the submission file shows the qualitative results. We will include these results in the next version. \n\n|Sparsity of depth supervision| DAE$\\downarrow$| CD ($\\times10^{-3}$) $\\downarrow$ (mean / median)  |\n|--------------------|:---------:|:-----------------:|\n| 1% |  8.69  | 1.517 / 0.937   |\n| 5% |  8.55  | 1.522 / 0.928   |\n| 10% | 8.06  | 1.503 / 0.952   |\n| **100% (MucRays)** | **7.98**| **1.095** / **0.702**|\n\n**Q3: The method relies on the visibility classifier to generalize to novel rays, which can potentially be unreliable.**\n\n**A3:** This is a very interesting point. To further evaluate the reliability of our pipeline, we conduct a series of experiments to add different levels of noise into our classifier. In particular, we directly add a random noise drawn from a normal distribution (0, $\\sigma$) to the visibility score with a clip between 0 and 1, and then use the noisy score in our multiview consistency loss $\\ell_{mv}$ in Eq. 3 to optimize our ray surface distance network.\n\nThe following Table shows the accuracy and F1 scores of our visibility classifier and the final DAE/CD scores on *Lego* in Blender dataset. We can see that, once the accuracy of the visibility classifier degrades, the reconstruction performance decreases sharply due to the large amount of mismatched training signals. This highlights that our classifier plays a crucial role in the pipeline. Figure 5 in the submission file shows qualitative results. We will include these results in the next version. \n\n|Noise level ($\\sigma^2$)|Acc. (\\%)$\\uparrow$ | F1 (\\%)$\\uparrow$ |DAE$\\downarrow$| CD ($\\times10^{-3}$) $\\downarrow$ (mean / median)  |\n|--------------------|:---------:|:-----------------:|:-----:|:-----:|\n| 1 | 62.56 | 59.06 | 23.09 | 2.713 / 0.960   |\n| 0.5 | 63.99 | 67.76 | 21.16 | 2.248 / 0.889   |\n| 0.1 | 68.01 | 72.59 | 16.83  | 1.850 / 0.860   |\n| **0 (MucRays)** |  **87.74** | **85.24**  | **7.98**| **1.095** / **0.702**|\n\n\n**Q4: The performance of the proposed method is lacking, especially according to the results in the appendix.**\n\n**A4:** Our method is extensively evaluated on three public datasets together with a series of ablation studies to verify the effectiveness of our design, achieving a clear advantage in surface point regression both in terms of accuracy and efficiency. We are always open to conducting more experiments at the request of reviewers.\n\n**Q5: It might help understand the effect of visibility network better if there are visualizations of the visibility network predictions.**\n\n**A5:** We highly appreciate this suggestion. To better understand the classifier, we initially present quantitative results in Table 3 in Appendix. In addition, we further provide qualitative results of the classifier (query point + sampling rays with visibility predictions) in Figure 6 in the submission file, which will be added to the paper in the next version.\n\n**Q6: There appear to be some fixed-pattern noise artifacts in the demo video. I wonder what is the cause?**\n\n**A6:** Thank you for pointing out this issue. The artifacts are caused by the choice of spherical coordinate.\n\nAs also suggested by Reviewer sym2, we conduct additional experiments given two different choices of ray parameterizations on the Blender dataset. As shown in the following Table, our applied spherical coordinate can achieve the best DAE scores, and the caused artifacts can be easily fixed by using PRIF's ray parameterization as shown in Figure 7 in the submission file. We leave it for our future work to fully address such an issue.\n\n|                    | DAE$\\downarrow$| CD ($\\times10^{-3}$) $\\downarrow$ (mean / median)  |\n|--------------------|:---------:|:-----------------:|\n| LFN param. + (Ray-surface Distance Network + Visibility Classifier) |  89.53  | 82.102 / 10.298   |\n| PRIF param. + (Ray-surface Distance Network + Visibility Classifier) |  8.42  | 3.409 / **0.621**   |\n| (LFN param. + Ray-surface Distance Network) + (Our param. + Visibility Classifier)|  91.30  | 142.570 / 74.345   |\n| (PRIF param. + Ray-surface Distance Network) + (Our param. + Visibility Classifier) |  8.44  | 3.526 / 0.653   |\n| **MucRays** | **7.97**| **3.388** / 0.663|"}, {"review_id": "iHzuYulMNX", "reviewer": "Reviewer_sym2", "summary": "This paper presents a new strategy to design ray-based neural representation of 3D shapes. Ray-based approaches towards shape representation is a recently emerging idea that bypasses extensive point-based evaluation required by conventional methods like signed distance function. A key missing component in existing ray-based neural representations is the lack of multi-view consistency. This paper proposes training a dual-ray visibility classifier and using that network's output to guide the training of the network for ray-distance prediction. The method is extensively evaluated on synthetic and real-world datasets, showing its superiority over previous ray-based methods in predicting ray hit points from novel views. Moreover, this paper also includes evaluations of color prediction of this ray-based method, showing it achieving comparable performance to previous efficient representations.", "questions": "- In Table 5, the performance drops significantly after removing the classifier, and it actually becomes much worse than the various baselines in Table 2. This is confusing and relates to the above-mentioned weakness about \"Not totally clear how the baselines are implemented\". Do the baseline methods included in Table 2, specifically LFN and PRIF, involve a classifier? That would totally change how to correctly interpret the results in Table 5.\n- If the baseline methods do not involve a classifier, and they are only just about a different ray parametrization with slightly smaller network, then the natural question becomes: how does LFN/PRIF + classifier perform?\n- In general, how does the ray parametrization affect the performance? It appears that it would have minor impact compared to the role of classifier, but the paper does not provide enough information to draw any conclusion.", "limitations": "Yes", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Addresses a key issue of multi-view inconsistency in current ray-based scene representations.\n- Clear writing and figure presentation.\n- Extensive evaluation showing the efficacy of the proposed strategy, outperforming previous baselines without slowing down inference", "weaknesses": "- The proposed method operates under the setting where depth maps are available for all views, which is okay, but the paper would be more complete if it discusses how bad the results would become if depth is unknown, which is actually the setting of LFN.\n- Not totally clear how the baselines are implemented. For example, when using LFN and PRIF, the supplementary material mentions that it uses the same architecture as the official setup. However, it is not clear whether the LFN/PRIF results are obtained still with the proposed classifier. At least based on the writing, it does not eliminate the ambiguity.\n- The proposed method also adopts a different ray parametrization than LFN and PRIF. It is not clear what the reason is behind this design choice.\n- The proposed method is implemented as a 13-layer SIREN with 1024 hidden channels, which is different than previous baselines. The paper does not include any analysis on the impact of varying layer depth and hidden channels.\n- In L123, the paper says the network predictions \"must satisfy a transformation equation\", without defining what this equation is.\n- Related work misses any discussion on space carving / visual hull (e.g., [Matusik 2000], [Kutulakos 2000]), which are important early concepts behind the idea of visibility checking proposed in this work. Related work also misses some recent papers on light field / ray-based representation (e.g. HyperReel [CVPR 2023], SRT [CVPR 2022], SIGNET [ICCV 2021]).\n- L237 typo: \"RTX 30390 GPU\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's thoughtful comments and address the main concerns below.\n\n**Q1: The proposed method operates under the setting where depth maps are available for all views, which is okay, but the paper would be more complete if it discusses how bad the results would become if depth is unknown, which is actually the setting of LFN.** \n\n**A1:** We thank the reviewer for this suggestion. Our method indeed operates with the requirement of depth supervision. Nevertheless, with the advancement of existing techniques of depth estimation from RGB images, it is quite feasible to obtain sparse depth signals using existing techniques such as SfM or learning-based monocular depth estimators. In this regard, we additionally provide experimental results of our method using sparse depth values in supervision. From the following Table which reports the metrics on *Lego* in Blender dataset, we can see that our method still achieves satisfactory performance even though there are only 1\\% depth values in training. We hypothesize that such robustness comes from our simple multiview consistency constraint, because many depth values in the training set may be redundant thanks to our effective classifier. Figure 4 in the submission file shows the qualitative results. We will include these results in the next version. \n\n|Sparsity of depth supervision| DAE$\\downarrow$| CD ($\\times10^{-3}$) $\\downarrow$ (mean / median)  |\n|--------------------|:---------:|:-----------------:|\n| 1% |  8.69  | 1.517 / 0.937   |\n| 5% |  8.55  | 1.522 / 0.928   |\n| 10% | 8.06  | 1.503 / 0.952   |\n| **100% (MucRays)** | **7.98**| **1.095** / **0.702**|\n\n**Q2: Not totally clear how the baselines are implemented. For example, when using LFN and PRIF, the supplementary material mentions that it uses the same architecture as the official setup. However, it is not clear whether the LFN/PRIF results are obtained still with the proposed classifier. At least based on the writing, it does not eliminate the ambiguity.**\n\n**A2:** The LFN/PRIF results are obtained without our classifier. Here, we further conduct experiments on LFN/PRIF by using our own classifier for comparison. As shown in the following Table, our method still achieves the best performance. Figure 1 in the submission file shows the qualitative results.\n\n|                    | DAE$\\downarrow$| CD ($\\times10^{-3}$) $\\downarrow$ (mean / median)  |\n|--------------------|:---------:|:-----------------:|\n| LFN + Our Visibility Classifier |  90.82  | 121.289 / 44.564   |\n| PRIF + Our Visibility Classifier |  9.97  | 4.187 / 0.891   |\n| **MucRays** | **7.97**| **3.388** / **0.663** |\n\n**Q3: The proposed method also adopts a different ray parametrization than LFN and PRIF. It is not clear what the reason is behind this design choice.**\n\n**A3:** In fact, our pipeline is amenable to different types of ray parameterizations. To verify this, we conduct additional ablation studies by replacing our spherical coordinate with that used in LFN and PRIF. The following Table shows the results on the Blender dataset. We can see that PRIF parameterization can also achieve satisfactory performance, while LFN parameterization is inferior in our pipeline. Figure 1 in the submission file shows qualitative results.\n\n|                    | DAE$\\downarrow$| CD ($\\times10^{-3}$) $\\downarrow$ (mean / median)  |\n|--------------------|:---------:|:-----------------:|\n| LFN param. + (Ray-surface Distance Network + Visibility Classifier) |  89.53  | 82.102 / 10.298   |\n| PRIF param. + (Ray-surface Distance Network + Visibility Classifier) |  8.42  | 3.409 / **0.621**   |\n| (LFN param. + Ray-surface Distance Network) + (Our param. + Visibility Classifier)|  91.30  | 142.570 / 74.345   |\n| (PRIF param. + Ray-surface Distance Network) + (Our param. + Visibility Classifier) |  8.44  | 3.526 / 0.653   |\n| **MucRays** | **7.97**| **3.388** / 0.663|\n\n**Q4: The proposed method is implemented as a 13-layer SIREN with 1024 hidden channels, which is different than previous baselines. The paper does not include any analysis on the impact of varying layer depth and hidden channels.** \n\n**A4:** Thanks for the suggestion to analyze different layers and channels. We conduct additional experiments as shown in the following Table. We can see that our framework tends to favor a wider or deeper network. However, to extensively explore an optimal network architecture is non-trivial and we leave it for our future work.\n\n|                    | DAE$\\downarrow$| CD ($\\times10^{-3}$) $\\downarrow$ (mean / median)  |\n|--------------------|:---------:|:-----------------:|\n| 8 layers, 512 hidden channels |  9.13  | 4.311 / 0.954   |\n| 8 layers, 1024 hidden channels |  8.52  | 4.010 / 0.805   |\n| 13 layers, 512 hidden channels |  8.80  | 4.085 / 0.854   |\n| **MucRays** | **7.97**| **3.388** / **0.663**|\n\n**Q5: In L123, the paper says the network predictions \"must satisfy a transformation equation\", without defining what this equation is.**\n\n**A5:** Thank you for pointing it out. The transformation equation is originally presented in Appendix A.1.3. and we will move it to the main text in the next version.\n\n**Q6: Related work misses any discussion on space carving / visual hull (e.g., [Matusik 2000], [Kutulakos 2000]), ...... Related work also misses some recent papers on light field / ray-based representation (e.g. HyperReel [CVPR 2023], SRT [CVPR 2022], SIGNET [ICCV 2021]).** \n\n**Q7: L237 typo: \"RTX 30390 GPU\".** \n\n**A6-A7:** Thank you for sharing the related works. We will include and discuss them in our next version. Typos will be corrected as well.\n\n**Q8: In Table 5, the performance drops significantly after removing the classifier ......** \n\n**A8:** Responded in **Q2**.\n\n**Q9: If the baseline methods do not involve a classifier ......**\n\n**A9:** Responded in **Q2**.\n\n**Q10: In general, how does the ray parametrization affect the performance? ......** \n\n**A10:**  Responded in **Q3**."}, {"review_id": "t7lX3y95Io", "reviewer": "Reviewer_MPar", "summary": "This paper proposes MucRays, which imposes ray-based neural functions with multi-view geometry consistency. This framework contains three-parts: ray-surface distance field, dual-ray visibility classifier, and multi-view consistency optimization strategy. Quantitative results of MucRays surpass the existing coordinate- or ray- based networks for synthetic or real-world scene modeling. This ray-based method achieves 1000x faster for depth image inference. ", "questions": "Main questions are in Weakness part. \n\nMore detailed comments: \n\nThe authors should use consistent abbreviation. For example, L. 80 use UDF for paper [13], while the remaining contents use NDF for [13]. ", "limitations": "Yes.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1.\tThe dual-ray visibility classifier is interesting. It is well-motivated and acts as the foundation of the multi-view consistency optimization. \n2.\tThe quantitative results are good. The high performance of MucRays on DAE metric proves its ability to render more accurate depth images. \n", "weaknesses": "1.\tThe qualitative results does not satisfy me. It seems that, MucRays has difficulty in representing thin structures. For example, in the Reception scene in Figure 4 (Appendix), the arm of the desk lamp is missing in both distance map and mesh. I would like to see a discussion about such phenomenon. \n2.\tI think NeuS (NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction) is a better baseline than DeepSDF because it could use both depth and color for supervision. \n3.\tI assume that this paper focused on better geometry modeling. However, it seems that the geometric result (mesh result) on Lego (Figure 5) is over smoothed compared with DS-NeRF or NDF. Is it caused by TSDF or the nature of ray distance representation itself? \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's thoughtful comments and address the main concerns below.\n\n**Q1: The qualitative results does not satisfy me. It seems that, MucRays has difficulty in representing thin structures. For example, in the Reception scene in Figure 4 (Appendix), the arm of the desk lamp is missing in both distance map and mesh. I would like to see a discussion about such phenomenon.**\n\n**A1:** Unarguably, recovering thin structures is particularly challenging for almost all methods, including ours and baselines. For our method, there could be multiple potential factors that lead to difficulty in reconstructing thin structures.\n\nIn our multi-view consistency loss function $\\ell_{mv}$ (Eq. 3), we simply change it from $\\ell_1$ to $\\ell_2$ optimization, and conduct an additional experiment on *Reception* of DMSR dataset. The following Table and Figure 8 in the submission file show the results. We can see that $\\ell_2$ based loss function can better reconstruct thin structures but with more noisy results at the same time. In this regard, it is of great interest to fully address this issue and we leave it for future exploration.\n\n|                    | DAE$\\downarrow$| CD ($\\times10^{-3}$) $\\downarrow$ (mean / median)  |\n|--------------------|:---------:|:-----------------:|\n| MucRays-$\\ell_2$|**6.56** | 12.141 / 6.853   |\n| **MucRays-$\\ell_1$** | 6.96| **10.632** / **5.714**|\n\n**Q2: I think NeuS (NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction) is a better baseline than DeepSDF because it could use both depth and color for supervision.**\n\n**A2:** Thank you for this suggestion and we conduct additional experiments for NeuS on the Blender dataset. From the following Table, our method is clearly better than NeuS in both reconstruction accuracy and novel view rendering speed. Figure 9 in the submission file shows the qualitative results. We will add this new baseline into the main paper in next version.\n\n|                    | DAE$\\downarrow$| CD ($\\times10^{-3}$) $\\downarrow$ (mean / median)  | PSNR $\\uparrow$| SSIM $\\uparrow$ | LPIPS $\\downarrow$ | Rendering time (seconds)\n|--------------------|:---------:|:-----------------:|:---:|:---:|:---:|:---:|\n| NeuS| 12.10 | 4.662 / 0.938   | **27.19** | **0.911** | **0.100** | 32.793 |\n| **MucRays** | **8.17** | **3.295** / **0.755**| 26.52 | 0.910 | 0.099 | **0.019**|\n\n**Q3: I assume that this paper focused on better geometry modeling. However, it seems that the geometric result (mesh result) on Lego (Figure 5) is over smoothed compared with DS-NeRF or NDF. Is it caused by TSDF or the nature of ray distance representation itself?**\n\n**A3:** This is a very interesting question. The observed over-smoothing effects may be caused by multiple potential factors, including the choice of ray parameterizations, the choice of loss functions, the postprocessing of TSDF, etc. Additionally, the nature of continuous ray-distance functions (MLPs) is also likely to result in smoothing predictions given similar input rays. In this regard, we hope that our method could inspire more advanced research works to tackle these core challenges in the future. \n\n**Q4: The authors should use consistent abbreviation. For example, L. 80 use UDF for paper [13], while the remaining contents use NDF for [13].** \n\n**A4:** Thanks. We will fix it in the next version."}]

Author rebuttal:
We appreciate all insightful comments. After carefully improving the quality of our work, we present a document containing additional experimental results. And the reponses to the comments include:

- Clarification of our dual-ray visibility classifier.
- Evaluations of our method using sparse depth supervision.
- Comparisons of our method using different ray parameterizations.
- Comparisons of adopting our dual-ray visibility classifier to other baselines.
- Additional ablations on our dual-ray visibility classifier, post-processing, and the training strategy.
- Additional ablations on our ray-surface distance netowrk, including network architecture and thin structure reconstruction.
- Additional baseline comparisons.

Meta-review: This paper received generally positive reviews. While there are clear limitations e.g. requirement of depth and long training time, the presented approach does allow accurate and efficient inference. Moreover, the technical contributions in the two-stage training of the ray-distance network are also interesting. On the balance, the AC would side with the reviewers recommending acceptance, but would encourage the authors to incorporate the reviewer suggestions to improve the presentation in the final version.
Decision: Accept (poster)