Title: RayDF: Neural Ray-surface Distance Fields with Multi-view Consistency
Venue: NeurIPS 2023
Accepted: yes
Keywords: implicit shape representations

multi-view consistency

novel view synthesis
Abstract: In this paper, we study the problem of continuous 3D shape representations. The majority of existing successful methods are coordinate-based implicit neural representations. However, they are inefficient to render novel views or recover explicit surface points. A few works start to formulate 3D shapes as ray-based neural functions, but the learned structures are inferior due to the lack of multi-view geometry consistency. To tackle these challenges, we propose a new framework called RayDF. It consists of three major components: 1) the simple ray-surface distance field, 2) the novel dual-ray visibility classifier, and 3) a multi-view consistency optimization module to drive the learned ray-surface distances to be multi-view geometry consistent. We extensively evaluate our method on three public datasets, demonstrating remarkable performance in 3D surface point reconstruction on both synthetic and challenging real-world 3D scenes, clearly surpassing existing coordinate-based and ray-based baselines. Most notably, our method achieves a 1000x faster speed than coordinate-based methods to render an 800x800 depth image, showing the superiority of our method for 3D shape representation. Our code and data are available at https://github.com/vLAR-group/RayDF

RayDF: Neural Ray-surface Distance Fields with Multi-view Consistency

Abstract

In this paper, we study the problem of continuous 3D shape representations. The majority of existing successful methods are coordinate-based implicit neural representations. However, they are inefficient to render novel views or recover explicit surface points. A few works start to formulate 3D shapes as ray-based neural functions, but the learned structures are inferior due to the lack of multi-view geometry consistency. To tackle these challenges, we propose a new framework called RayDF . It consists of three major components: 1) the simple ray-surface distance field, 2) the novel dual-ray visibility classifier, and 3) a multi-view consistency optimization module to drive the learned ray-surface distances to be multi-view geometry consistent. We extensively evaluate our method on three public datasets, demonstrating remarkable performance in 3D surface point reconstruction on both synthetic and challenging real-world 3D scenes, clearly surpassing existing coordinate-based and ray-based baselines. Most notably, our method achieves a 1000 × faster speed than coordinate-based methods to render an 800 × 800 depth image, showing the superiority of our method for 3D shape representation. Our code and data are available at https://github.com/vLAR-group/RayDF

1 Introduction

Learning accurate and efficient 3D shape representations is crucial for many cutting-edge applications in the fields of machine vision and robotics. Recent advances in 3D coordinate-based neural representations including occupancy fields (OF) [46, 11], un/signed distance fields (U/SDF) [54, 14], and radiance fields (NeRF) [47], have shown great potential to recover complex shapes from RGB/D images and/or point clouds. Although these methods and their variants have achieved excellent performance in a wide range of downstream tasks such as shape reconstruction [55, 58, 60], novel view synthesis [4, 72, 51, 5], and scene understanding [85, 86, 73], obtaining 3D shapes or 2D views from their trained networks is computationally expensive, due to the requirement of extensive network evaluations to regress surface points.

Very recently, a number of works start to represent 3D models as ray-based neural functions. By simply taking individual light rays as input, the methods LFN [64] and NeuLF [40] learn to directly predict the radiance (RGB) values, while PRIF [23] and DDF [3] straightly estimate the surface hitting points of input rays. Compared with coordinate-based representations, these ray-based methods are clearly more efficient to infer 3D geometry and render 2D views, as every single ray only needs to query the trained network once forward. Nevertheless, the learned 3D geometries by these methods are still lack of fidelity, primarily because they fail to explicitly take into account geometry consistency across multiple views, resulting in the network likely over-fitting individual training rays but unable to generalize to unseen rays in testing.

In this paper, we aim to address this key issue of ray-based neural representations by explicitly integrating multi-view geometry consistency into the network design. In particular, we introduce a

Ray-surface Distance Network

Dual-ray Visibility Classifier

ray

hitting point

distance

mutual

visibility

Multi-view

Consistency

Optimization

ray-surface

distances

mutual

visibility

ray 1

ray 2

new pipeline as shown in Figure 1. It consists of two independent neural networks together with a particular training module: 1) the main ray-surface distance network, 2) an auxiliary dual-ray visibility classifier, and 3) a multi-view consistency optimization module.

The main network simply takes a ray as input and directly infers the distance between ray origin and its hitting point on the surface. Basically, this network elegantly represents 3D shapes as ray-based implicit functions, denoted as ray-surface distance fields , keeping the unique advantage in efficiency for shape extraction and rendering. The auxiliary network takes a pair of rays as input and predicts their mutual visibility. In fact, this is a simple binary classifier, aiming at distinguishing whether any two rays hit at the same surface point or not, i . e ., the mutual visibility. Having a trained auxiliary network at hand, the multi-view consistency optimization module specifies how to effectively leverage the learned dual-ray visibility to train the main network, thus driving learned ray-surface distances to be multi-view consistent from any seen or unseen viewing angles.

Since our ray-surface distance fields primarily aim at representing accurate 3D shapes, the whole pipeline is designed to be trained on depth images, although light fields (radiance) can be optionally learned in parallel if color images are also available in training. In this regard, the closest works to ours are PRIF [23] and DDF [3], neither of which explicitly considers the multi-view geometry consistency. Overall, compared with all existing coordinate-based representations, our method keeps the superiority in efficiency thanks to the ray-based formulation. Compared with the existing raybased approaches, ours excels at learning accurate 3D geometries thanks to the multi-view consistency for ray -surface d istance f ields. Our method is called RayDF and our contributions are:

Weemploy a straightforward ray-surface distance field for representing 3D shapes. This formulation is inherently more efficient than existing coordinate-based representations.

We design a new dual-ray visibility classifier to learn the spatial relationships of any pair of rays, enabling the learned ray-surface distance fields to be multi-view geometry consistent.

We demonstrate superior 3D shape reconstruction accuracy and efficiency on multiple datasets, showing significantly better results than existing coordinate-based and ray-based baselines.

2 Related Work

Explicit 3D Shape Representations: Classic methods to recover explicit 3D geometry of objects and scenes mainly include SfM [53] and SLAM [7]systems such as Colmap [62] and ORB-SLAM [49]. Another classic method is space carving [45, 37], which involves the process of carving out the voxel grid by utilizing multiple distinct views to obtain a robust approximation within the voxel space. To model explicit 3D structures, deep learning-based methods have shown impressive progress in recovering voxel grids [15, 80, 81], point clouds [21], octree [71], polygon meshes [34, 26] and shape primitives [87]. A comprehensive survey of these methods can be found in [6, 28]. Thanks to the large-scale datasets [10] for training sophisticated neural networks [29, 31], these methods demonstrate excellent results in many downstream tasks such as shape reconstruction [76, 70], generation [38], and semantic scene perception [67, 17, 25]. However, the fidelity of these discrete shape representations is primarily limited by their spatial resolutions and memory footprint.

Coordinate-based Implicit 3D Shape Representations: To avoid the discretization issue of traditional 3D representations, there has been a growing interest in developing implicit neural 3D representations using simple MLPs to represent continuous 3D shapes. Inspired by the seminal coordinate-based methods [11, 46, 54] which have shown great success in encoding high-quality 3D shapes, a plethora of follow-up works have been developed to tackle various vision tasks in [65, 79, 2, 12, 13, 20, 33, 43, 42, 27, 83]. These coordinate-based representations can be generally classified as: 1) occupancy fields (OF) [46, 11], 2) signed distance fields (SDF) [54], 3) unsigned

r

(

θ

in

, φ

in

)

(

θ

out

, φ

out

)

d

. . .

f

Θ

p

D

d









θ

in

φ

in

θ

out

φ

out









input ray

r

ray-surface

distance

distance fields (NDF) [14, 74], and 4) radiance fields (NeRF) [47]. More related methods can be found in surveys [24, 18]. Among them, the OF/SDF/NDF based methods demonstrate exceptional accuracy in recovering continuous 3D structures thanks to the simple level-set formulation, while the NeRF based methods achieve an unprecedented level of fidelity in rendering 2D views thanks to the successful volume rendering equation. However, all these coordinate-based representations inevitably require dense 3D location sampling and evaluations on trained networks to regress explicit surface points, though advanced techniques [41, 57, 69, 39, 30, 56, 50, 82, 48] can mitigate this issue somewhat but at the expense of a large memory increase. In this paper, our RayDF only needs a single network evaluation to regress every surface point, while still achieving on par or better reconstruction accuracy with existing OF/SDF/NDF methods.

Ray-based Implicit 3D Shape Representations: To overcome the inefficiency of coordinate-based 3D representations, a number of works start to formulate 3D shapes as ray-based neural functions via MLPs. Ray-based methods simply take individual rays as input and directly output either radiance values (RGB), i . e . light fields , or surface points, i . e . distance fields . LFN [64] and NeuLF [40] are among the early works of light fields to encode 3D representations from observed color images. Other light field works include [22, 61, 78, 52, 68, 75, 9, 1]. PRIF [23], DDF [3] and DRDF [36] are the early ray-based distance fields to learn 3D shapes from observed depth scans. Although these methods have shown very encouraging results for novel view rendering or surface reconstruction, they are usually limited to datasets with small baselines across multi-views. Basically, this is because the multi-view geometry consistency is not taken into account in their network designs. By contrast, in our RayDF pipeline, we explicitly introduce a dual-ray visibility classifier to aid our ray-surface distance fields to be multi-view shape consistent during training.

3 RayDF

3.1 Overview

Our pipeline consists of two networks and an optimization module. As shown in Figure 2, for the main network: ray-surface distance field f Θ , it takes a single oriented ray r as input, and directly regresses the distance d between ray starting point and its surface hitting point. For the input ray r , we opt for the conventional spherical parameterization [32], as it supports querying from 360 ◦ viewing angles. In particular, a fix-sized sphere is predefined with a relatively large diameter D as a convex hull in which the target 3D scene is bounded. Each surface point p of the target scene can be regarded as a directional ray penetrating the sphere with two intersection points on the sphere. Each intersection point is parameterized by two variables specifying the angles with regard to the sphere center. Then, for each ray, we have a 4D parameterization r =( θ in , φ in , θ out , φ out ) . Formally, the ray-surface distance field is defined below and implementation details are in Appendix A.1.

For the auxiliary network: dual-ray visibility classifier h Φ , it takes a pair of rays as input and simply predicts their mutual visibility, aiming at explicitly modeling the mutual spatial relationships between any pairs of rays. This network, once well-trained, will play a key role in the third component: multi-view consistency optimization . Detailed designs are discussed in Sections 3.2&3.3.

3.2 Dual-ray Visibility Classifier

The main ray-surface network alone can indeed well fit many training ray-distance pairs, but there is no mechanism driving its output distances, i . e ., surface geometry, to be consistent across multiple views, especially for unseen views. For example, as illustrated in the leftmost block of Figure 3, if

r

1

r

2

r

3

1

: mutual visible

0

: mutual invisible

d

1

d

2

d

3

(

x

1

, y

1

, z

1

)

⊕

h

Φ

0 / 1

g

k









θ

in

1

φ

in

1

θ

out

1

φ

out

1















θ

in

2

φ

in

2

θ

out

2

φ

out

2







. . .



x

1

y

1

z

1



∑

2









two input rays r 1 and r 2 are mutually visible in the scene, both rays must hit at the same surface point, then the corresponding ray-surface distances d 1 and d 2 must satisfy a transformation equation details in Section A.1.3), according to both rays' angles. Similarly, if input rays r 1 and r 3 are mutually invisible, then they should never satisfy the transformation equation because both never meet at the same surface point. From this fundamental principle, we can see that the mutual visibility between any two rays is crucial to inform the ray-surface distance network to be multi-view consistent.

To this end, we design a binary classifier to discriminate the visibility of an input pair of rays. A naïve idea is to simply concat two rays as input, feeding into MLPs to regress 0/1, where 1 represents visible and 0 otherwise. However, such a design fails to retain the symmetry of two rays. Here symmetry means that the visibility of two rays must be invariant to the input order of two rays. With this point, as illustrated in the right block of Figure 3, our visibility classifier h Φ is formally defined as:

where ( θ in 1 , φ in 1 , θ out 1 , φ out 1 ) and ( θ in 2 , φ in 2 , θ out 2 , φ out 2 ) are the parameterizations of two input rays r 1 and r 2 respectively; g () is a shared single fully-connected layer followed by an average pooling, thus guaranteeing the symmetry of input rays. Note that, the surface hitting point of r 1, i . e ., ( x 1 , y 1 , z 1 ) , is also encoded via a separate single fully-connected layer k () , followed by a concatenation operation denoted by ⊕ , thus enhancing the pooled ray features, as we empirically find that such an enhancement could notably improve the classifier's accuracy. Implementation details are in Appendix A.1.

3.3 Multi-view Consistency Optimization

With the designed main ray-surface distance network f Θ and the auxiliary dual-ray visibility classifier h Φ at hand, we introduce the crucial multi-view consistency optimization module to train both networks. In particular, given K posed depth images ( H × W ) of a static 3D scene as the whole training data, our training module consists of two stages.

Stage 1 - Training Dual-ray Visibility Classifier

The key to training this classifier is to create correct data pairs. First of all, all raw depth values are converted to ray-surface distance values. For a specific i th ray (pixel) in the k th image, we project its ray-surface point back to the remaining ( K -1 ) scans, obtaining the corresponding ( K -1 ) distance values. We set 10 millimeters as the closeness threshold to determine whether the projected ( K -1 ) rays are visible in the ( K -1 ) images. In total, we generate K ∗ H ∗ W ∗ ( K -1 ) pairs of rays together with 0/1 labels. The standard cross-entropy loss function is adopted to optimize our dual-ray visibility classifier. More details on training data generation and implementation are in Appendix A.1.3.

Note that, this classifier is trained in a scene-specific fashion. Once the network is well-trained, it basically encodes the relationships between any two rays of a specific scene into network weights.

Stage 2 - Training Ray-surface Distance Network

The ultimate goal of our whole pipeline is to optimize the main ray-surface network and drive it to be multi-view geometry consistent. However, this is non-trivial as simply fitting the network with ray-surface data points cannot generalize to unseen rays, which can be seen in our ablation study in Section 4.5. In this regard, we fully leverage the well-trained visibility classifier to aid our training of ray-surface distance network. Particularly, this stage consists of the following key steps:

Step 1: All depth images are converted to ray-surface distances, generating K ∗ H ∗ W training ray-distance pairs for a specific 3D scene.

Step 2: As illustrated in Figure 4, for a specific training ray ( r , d ), called primary ray , we uniformly sample M rays { r 1 · · · r m · · · r M } , called multi-view rays , in a ball centering at the surface point p . We then calculate the distance between surface point p and the bounding sphere along each of M rays, obtaining multi-view distances { ˜ d 1 · · · ˜ d m · · · ˜ d M } . This can be easily achieved according to the given distance d in the training set. M is simply set as 20 and more details are in Appendix A.4.

Step 4: We feed the primary ray and all sampled M multi-view rays { r , r 1 · · · r m · · · r M } into the ray-surface distance network f Θ , estimating their surface distances { ˆ d , ˆ d 1 · · · ˆ d m · · · ˆ d M } . Since the network f Θ is randomly initialized, thus the estimated distances are inaccurate in the beginning.

Step 3: We establish M pairs of rays { ( r , p , r 1 ) · · · ( r , p , r m ) · · · ( r , p , r M ) } and then feed them into the well-trained visibility classifier h Φ , inferring their visibility scores { v 1 · · · v m · · · v M } .

Step 5: We design the following multi-view consistency loss function to optimize the ray-surface distance network until convergence:

Basically, this simple loss drives the network to not only fit the primary ray-surface distance (seen rays in the training set), but also satisfy that the visible multi-view rays (unlimited unseen rays in the training set) also have accurate distance estimations.

3.4 Surface Normal Derivation and Outlier Points Removal

In the above Sections 3.1&3.2&3.3, we have two network designs and an optimization module to train them separately. Nevertheless, we empirically find that the main ray-surface distance network may predict inaccurate distance values, particularly for rays near sharp edges. Essentially, this is because the actual

d

p

r

ray-surface distances may be discontinuous at sharp edges given extreme viewing angle changes. This shape discontinuity is actually a common challenge for almost all existing implicit neural representations, because modern neural networks can only model continuous functions in theory.

Fortunately, a nice property of our ray-surface distance field is that the normal vector at every estimated 3D surface point can be easily derived in a closed-form expression using auto differentiation of the network. In particular, given an input ray r =( θ in , φ in , θ out , φ out ) , and its estimated ray-surface distance ˆ d from network f Θ , the corresponding normal vector n of that estimated surface point can be derived as a specific function shown below.

With this normal vector, we may choose to add an additional loss to regularize the estimated surface points to be as smooth as possible. Yet, we empirically find that the overall performance improvement on an entire 3D scene is rather limited, as these extremely discontinuous cases are actually sparse.

In this regard, we turn to simply removing the predicted surface points, i . e ., outliers, whose normal vectors' Euclidean distances are larger than a threshold in the network inference stage. In fact, PRIF [23] also adopts a similar strategy to filter out outliers. Note that, advanced smoothing or interpolating techniques may be integrated to improve our framework, which is left for future exploration.

4 Experiments

Our method is evaluated on three types of public datasets: 1) the object-level synthetic Blender dataset from the original NeRF paper [47], 2) the scene-level synthetic DM-SR dataset from the recent DM-NeRF paper [73], and 3) the scene-level real-world ScanNet dataset [16].

Baselines: We carefully select the following six successful and representative implicit neural shape representations as our baselines: 1) OF [46], 2) DeepSDF [54], 3) NDF [14], 4) NeuS [77], 5) DSNeRF [19], 6) LFN [64], and 7) PRIF [23]. The OF/DeepSDF/NDF/NeuS methods are coordinatebased level-set methods, showing outstanding performance in modeling 3D structures. DS-NeRF

is a depth-supervised NeRF [47], inheriting excellent capability in rendering 2D views. LFN and PRIF are two ray-based methods with superior efficiency in generating 2D views. We note that there are many sophisticated variants of these baselines, achieving SOTA performance on various datasets. Nevertheless, we do not intend to comprehensively compare with them, basically because many of their techniques such as more advanced implementations, adding additional conditions, replacing with more powerful backbones, etc ., can be easily integrated into our framework as well. We leave these potential improvements for future exploration but only focus on our vanilla ray-surface distance field in this paper. For a fair comparison, all baselines are supervised with the same amount of depth scans as ours, carefully trained from scratch in the same scene-specific fashion. More details about the implementation of all baselines and possible minor adaptations are in Appendix A.3.1.

Metrics: For evaluation metrics of shape reconstruction, we report: 1) the per ray-surface absolute distance error (ADE) in centimeters averaged across all testing images, 2) the Chamfer distance (CD) [21] between the whole reconstructed structure and the ground truth shape for each scene. As to our method, we use the existing TSDF fusion [84] to obtain a full mesh from predicted depths at the testing views, on which we uniformly sample 30K points. Another 30K points are uniformly sampled from the ground truth 3D mesh for calculating CD. The outlier point removal is only applied to clean the reconstructed point clouds when calculating CD. No additional post-processing steps are employed when computing all ADE scores. Apparently, ADE is more accurate to evaluate the surface estimation of our method, because the CD scores may be biased due to the extra TSDF fusion. In this regard, the CD scores are just presented as a complementary metric to show the general shape quality in the whole 3D scene space. More details about how to obtain the reconstructed 3D full shape for each baseline and our method are in Appendix A.3.2. For evaluation metrics of appearance reconstruction, i . e . novel view synthesis of 2D color images, the standard PSNR , SSIM , and LPIPS scores are reported following NeRF [47]. We present Accuracy (%) and F1-Score (%) as evaluation metrics of the dual-ray visibility classifier. For qualitative results, refer to Figure 5 and Appendix A.7.

4.1 Efficiency of RayDF

Similar to the existing ray-based methods LFN and PRIF, our RayDF also has the superiority in efficiency to generate 2D images or recover the explicit surface points. To quantitatively compare the efficiency with baselines, we conduct a simple experiment to generate an 800 × 800 depth image on a computer with a single NVIDIA RTX 3090 GPU card and a CPU of AMD Ryzen 7. As shown in Table 1, it can be seen that, not surprisingly, the coordinate-based methods OF/ DeepSDF/ NDF/ NeuS/ DS-NeRF are extremely slow to render a high-resolution depth image. In particular, OF [46] needs a large number of small steps to gradually approach the surface point along a given light ray direction, while DeepSDF [54] and NDF [14] rely on expensive sphere tracing to

regress the points. NeuS [77] and DS-NeRF [19] need extra post-processing steps to calculate depth values from densities. By contrast, our RayDF and the existing ray-based methods achieve more than 1000 × faster speed to render a dense depth view.

4.2 Evaluation on Blender Dataset

The Blender dataset from NeRF [47] consists of pathtraced images of 8 synthetic objects with complicated geometry and realistic materials. Each object has 100 views for training and 200 novel views for testing. Each image has 800 × 800 pixels. Since the released dataset does not include depth images, we just use the provided Blender files to generate additional depth scans of the same resolution exactly following the original 2D view poses. Note that, the physical sizes of these models are about 2.5 meters in length/height/width, so the sphere diameter D is set as 3 meters in our method. For the baselines and our method, we conduct the following two groups of experiments.

Group 1 - 3D Shape Representation Only: Since our RayDF needs depth scans in training to learn continuous 3D shape representations, we conduct this group of experiments only on multi-view depth images. All baselines are also trained on the same number of depth views.

Group 2 - 3D Shape and Appearance Representations: Our RayDF is also flexible to add a parallel branch to output radiance field, i . e ., RGB values, so that the continuous appearance representations can be simultaneously learned. In this regard, we conduct this group of experiments on both RGB images and depth images. The baselines NeuS/ DS-NeRF/ LFN/ PRIF are also trained on the same RGBD images for comparison. The other three methods OF/ DeepSDF/ NDF are not

included here, because it is non-trivial to add RGB supervision due to their level-set formulation. Details of the parallel branch for all methods are in Appendix A.3.

Analysis: Table 2 shows the quantitative comparison. We can see that: 1) Our RayDF achieves significantly better results for explicit surface recovering on both groups of experiments, especially on the most important ADE metric, demonstrating the clear superiority over both coordinate and ray based baselines. 2) Our method also achieves comparable performance with DS-NeRF for novel view synthesis, being better than LFN and PRIF and showing the flexibility of our framework.

4.3 Evaluation on DM-SR Dataset

The DM-SR dataset from the recent DM-NeRF paper [73] consists of 8 synthetic complex 3D indoor rooms. The room types and designs follow Hypersim dataset [59], and the rendering trajectories for both training and testing images follow the Blender dataset of NeRF [47]. For each 3D scene, there are 300 views for training and 100 novel views for testing. Each view has 400 × 400 pixels. Each scene has a physical size of about 10 meters in length/height/width, so the sphere diameter D is set as 11 meters in our method. Similarly, we conduct the following two groups of experiments.

Group 1 - 3D Shape Representation Only: We conduct this group of experiments only on multi-view depth images for all methods.

Group 2 - 3D Shape and Appearance Representations: We conduct this group of experiments on both RGB images and depth images for NeuS/ DS-NeRF/ LFN/ PRIF and our method.

Analysis: From Table 3, it can be seen that our method again surpasses all baselines on the most critical ADE metric in both groups of experiments, though the rough metric CD scores are just comparable with baselines potentially due to the inaccuracy incurred by external TSDF fusion. In the meantime, our method still obtains high-quality novel view synthesis, without noticeably sacrificing shape representation when trained with both RGB images and depth scans in Group 2.

4.4 Evaluation on ScanNet Dataset

We further evaluate our method on the challenging real-world 3D dataset ScanNet [16]. We randomly select 6 scenes for evaluation. Originally, each scene has more than 2000 RGBD scans in a relatively

long sequence, whose camera poses are usually facing outwards within indoor rooms. Such long camera trajectories pose a great challenge to our method due to our sphere parameterization of input rays. To overcome this issue, we take the divide and conquer strategy to process a subset of the full video frames for each scene. In particular, for each scene, we only pick up a subset of 300 continuous RGBD scans in our experiment, where 200 are uniformly sampled for training, and 100 for testing. As to these partial 3D scenes, the physical sizes are between 4 ∼ 16 meters in length/height/width, so the sphere diameter D is set as 16 meters in our method. Similarly, we conduct the following two groups of experiments.

Group 1 - 3D Shape Representation Only: We conduct this group of experiments only on multi-view depth images for all methods.

Group 2 - 3D Shape and Appearance Representations: We conduct this group of experiments on both RGB images and depth images for NeuS/ DS-NeRF/ LFN/ PRIF and our method.

Analysis: Table 4 compared all methods on the challenging real-world scenes. Remarkably, our RayDF clearly outperforms all baselines in almost all evaluation metrics in both groups of experiments, showing a distinct advantage in representing noisy and complex real-world 3D scenes. This confirms that multi-view geometry consistency is indeed essential in learning 3D representations.

4.5 Ablation Study

Since our framework just has two MLP-based neural networks, there is few hyperparameter to tune. The key component here is the dual-ray visibility classifier. To evaluate its effectiveness, we mainly conduct the following ablations on the classifier on Blender dataset [47] only with depth images.

Removing the dual-ray visibility classifier: Without the classifier, the multi-view consistency loss function becomes the simple ℓ 1 loss: ℓ mv = | ˆ d -d | for rays in the training depth scans.

Replacing position ( x , y , z ) by ray distance d in function k () : This is to evaluate the enhancement of explicitly adding surface point position.

Removing the function k () and only using two rays: This ablation is to demonstrate the necessity of adding surface information for accurate classification.

Concatenating two rays instead of average pooling: This ablation aims to evaluate the effect of the symmetry of the input rays on our classifier.

Adding noises to the well-trained visibility classifier: Various levels of noise are introduced to the well-trained classifier to assess the reliability of our visibility classifier. In particular, we directly add a random noise drawn from a normal distribution (0, σ 2 ) to the visibility score with a clip between 0 and 1, and then use the noisy score in our multi-view consistency loss ℓ mv in Eq. 3 to optimize our ray surface distance network.

Analysis: Table 5 (1) clearly shows that, without the aid of our dual-ray visibility classifier, the main ray-surface distance field completely collapses and cannot predict reasonable distance values for novel rays in the test set. In Table 5 (2) and (3), with the input of surface point position, the classifier achieves higher accuracy and higher F1-score, thus providing the ray-surface distance network with more accurate visibility information to predict precise distance values. In addition, concatenating two input rays directly disrupts the inherent symmetry of input rays. This is shown in Table 5 (4), where the classifier trained in this way achieves high accuracy but exhibits a low F1-score. This indicates that such a classifier is significantly less robust than the one trained with symmetric input rays. Table 6 demonstrates that a less robust classifier results in the ray-surface distance network predicting inaccurate distances.

In addition to the dual-ray visibility classifier, we also perform various ablation studies on our entire pipeline on the Blender dataset.

Ablation on Multi-view Rays Sampling: We conduct an ablation study with different values of M ∈ { 10 , 20 , 40 } to determine the optimal number of multi-view rays for training a ray-surface distance network. Table 7 demonstrates that increasing the sampled multi-view rays enhances the accuracy of the ray-surface distance network. Although M = 40 achieves lower ADE and CD values, the substantial GPU memory and extended training time outweigh the limited performance improvement. Therefore, we opt to sample multi-view rays with

Table 7: Ablation study on the number of multi-view rays sampling.

M = 20 to train our ray-surface distance network in all main experiments.

Table 8: Ablation study on sparse depth supervision.

1% 5% 10% Ablation on Sparse Depth Supervision: With the advancement of existing techniques of depth estimation from RGB images, it is quite feasible to obtain sparse depth signals using existing techniques such as SfM or learningbased monocular depth estimators. In this regard, we additionally provide experimental results using sparse depth supervision. In Table 8, our method maintains satisfactory performance, even with only 1% depth values during training. We hypothesize that such robustness comes from our multi-view consistency constraint, because many depth values in the training set may be redundant thanks to our effective classifier.

Ablation of One-stage RayDF: In the paper, we adopt a two-stage training strategy for our dual-ray visibility classifier and ray-surface distance network. For a comparison, we simply train both networks simultaneously. However, not surprisingly, the performance of one-stage training drops noticeably as shown in Table 9. This drop in performance is primarily because the classifier is inaccurate at the early stage and unlikely to provide effective constraints for

Table 9: Ablation study on one/two-stage training.

the ray-surface distance network, given the similar number of training steps. Nevertheless, exploring one-stage training remains an intriguing direction for our future research efforts.

For a more comprehensive analysis, we provide extensive ablation studies and additional qualitative as well as detailed quantitative results of each scene in the Blender dataset in Appendix A.4.

5 Conclusion

In this paper, we have shown that it is truly possible to efficiently and accurately learn 3D shape representations by using a multi-view consistent ray-based framework. In contrast to the existing coordinate-based methods, we instead use a simple ray-surface distance field to represent 3D shape geometries, which is further driven by a novel dual-ray visibility classifier to be multi-view shape consistent. Extensive experiments on multiple datasets demonstrate the extremely high rendering efficiency and outstanding performance of our approach. It would be interesting to extend our framework with more advanced techniques such as faster implementation and additional regularizations.

OF

DS-NeRF

LFN

PRIF

Ours

DeepSDF

NDF

NeuS

GT

OF

LFN

DeepSDF

NDF

NeuS

GT

DS-NeRF

LFN

PRIF

Ours

GT

NeuS

Blender - Lego

OF

DS-NeRF

LFN

PRIF

Ours

DeepSDF

NDF

NeuS

GT

DS-NeRF

LFN

PRIF

Ours

GT

NeuS

OF

DS-NeRF

LFN

PRIF

Ours

DeepSDF

NDF

NeuS

GT

DM-SR - Dinning

DS-NeRF OF

LFN

PRIF

Ours

DeepSDF

NDF

NeuS

GT

ScanNet - Scene0010_00

GT

OF

DS-NeRF

LFN

PRIF

Ours

DeepSDF

NDF

NeuS

DS-NeRF

PRIF

Ours

DS-NeRF

LFN

PRIF

Ours

GT

NeuS

Distance Map

Mesh

Acknowledgement: This work was supported in part by Research Grants Council of Hong Kong under Grants 15225522 & 25207822 & 15606321, in part by National Natural Science Foundation of China under Grant 62271431, and in part by The Hong Kong Polytechnic University under Grant P0031501.

References

B. Attal, J.-B. Huang, C. Richardt, M. Zollhoefer, J. Kopf, M. O'Toole, and C. Kim. HyperReel: Highfidelity 6-DoF video with ray-conditioned sampling. CVPR , 2023.

M. Atzmon and Y. Lipman. SAL: Sign Agnostic Learning of Shapes from Raw Data. CVPR , 2020.

T. Aumentado-Armstrong, S. Tsogkas, S. Dickinson, and A. Jepson. Representing 3D Shapes with Probabilistic Directed Distance Fields. CVPR , 2022.

J. T. Barron, B. Mildenhall, M. Tancik, P. Hedman, R. M.-b. Pratul, and C. V . Mar. Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields. ICCV , 2021.

J. T. Barron, B. Mildenhall, D. Verbin, P. P. Srinivasan, and P. Hedman. Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields. CVPR , 2022.

M. Berger, A. Tagliasacchi, L. M. Seversky, P. Alliez, J. A. Levine, A. Sharf, and C. T. Silva. A Survey of Surface Reconstruction from Point Clouds. Computer Graphics Forum , 36(1):301-329, 2017.

C. Cadena, L. Carlone, H. Carrillo, Y. Latif, D. Scaramuzza, J. Neira, I. D. Reid, and J. J. Leonard. Past, Present, and Future of Simultaneous Localization and Mapping: Towards the Robust-Perception Age. TRO , 2016.

E. Camahort, A. Lerios, and D. S. Fussell. Uniformly sampled light fields. Eurographics Workshop on Rendering Techniques , 1998.

J. Cao, H. Wang, P. Chemerys, V. Shakhrai, J. Hu, Y. Fu, D. Makoviichuk, S. Tulyakov, and J. Ren. Real-Time Neural Light Field on Mobile Devices. CVPR , 2023.

A. X. Chang, T. Funkhouser, L. Guibas, P. Hanrahan, Q. Huang, Z. Li, S. Savarese, M. Savva, S. Song, H. Su, J. Xiao, L. Yi, and F. Yu. ShapeNet: An Information-Rich 3D Model Repository. arXiv:1512.03012 , 2015.

Z. Chen and H. Zhang. Learning Implicit Fields for Generative Shape Modeling. CVPR , 2019.

Z. Chen, Y. Zhang, K. Genova, S. Fanello, S. Bouaziz, C. Haene, R. Du, C. Keskin, T. Funkhouser, and D. Tang. Multiresolution Deep Implicit Functions for 3D Shape Representation. ICCV , 2021.

J. Chibane, T. Alldieck, and G. Pons-Moll. Implicit Functions in Feature Space for 3D Shape Reconstruction and Completion. CVPR , 2020.

J. Chibane, A. Mir, and G. Pons-Moll. Neural Unsigned Distance Fields for Implicit Function Learning. NeurIPS , 2020.

C. B. Choy, D. Xu, J. Gwak, K. Chen, and S. Savarese. 3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction. ECCV , 2016.

A. Dai, A. X. Chang, M. Savva, M. Halber, T. Funkhouser, and M. Nießner. ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes. CVPR , 2017.

A. Dai, D. Ritchie, M. Bokeloh, S. Reed, J. Sturm, and M. Nießner. ScanComplete: Large-Scale Scene Completion and Semantic Segmentation for 3D Scans. CVPR , 2018.

L. De Luigi, A. Cardace, R. Spezialetti, P. Z. Ramirez, S. Salti, and L. Di Stefano. Deep Learning on Implicit Neural Representations of Shapes. ICLR , 2023.

K. Deng, A. Liu, J.-Y. Zhu, and D. Ramanan. Depth-supervised NeRF: Fewer Views and Faster Training for Free. CVPR , 2022.

Y. Duan, H. Zhu, H. Wang, L. Yi, R. Nevatia, and L. J. Guibas. Curriculum DeepSDF. ECCV , 2020.

H. Fan, H. Su, and L. Guibas. A Point Set Generation Network for 3D Object Reconstruction from a Single Image. CVPR , 2017.

B. Y. Feng and A. Varshney. Signet: Efficient neural representation for light fields. ICCV , 2021.

B. Y. Feng, Y. Zhang, D. Tang, and R. Du. PRIF: Primary Ray-based Implicit Function. ECCV , 2022.

A. T. O. Fried, J. T. V. Sitzmann, S. L. K. Sunkavalli, and R. M.-b. T. S. J. Saragih. State of the Art on Neural Rendering. Computer Graphics Forum , 2020.

G. Gkioxari, J. Malik, and J. Johnson. Mesh R-CNN. ICCV , 2019.

T. Groueix, M. Fisher, V. G. Kim, B. C. Russell, and M. Aubry. A Papier-Mache Approach to Learning 3D Surface Generation. CVPR , 2018.

Guandao Yang, S. Belongie, B. Hariharan, and V. Koltun. Geometry Processing with Neural Fields. NeurIPS , 2021.

X.-f. Han, H. Laga, and M. Bennamoun. Image-based 3D Object Reconstruction: State-of-the-Art and Trends in the Deep Learning Era. TPAMI , 2019.

K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. CVPR , 2016.

P. Hedman, P. P. Srinivasan, B. Mildenhall, J. T. Barron, and P. Debevec. Baking Neural Radiance Fields for Real-Time View Synthesis. ICCV , 2021.

H. Huang, D. Yang, G. Dai, Z. Han, Y. Wang, K.-M. Lam, F. Yang, S. Huang, Y. Liu, and M. He. Agtgan: Unpaired image translation for photographic ancient character generation. ACM MM , 2022.

I. Ihm, S. Park, R. K. Lee, and M.-g. Seoul. Rendering of Spherical Light Fields. PG , 1997.

Y. Jiang, D. Ji, Z. Han, and M. Zwicker. SDFDiff: Differentiable Rendering of Signed Distance Fields for 3D Shape Optimization. CVPR , 2020.

H. Kato, Y. Ushiku, and T. Harada. Neural 3D Mesh Renderer. CVPR , 2018.

D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. ICLR , 2015.

N. Kulkarni, J. Johnson, and D. F. Fouhey. Directed Ray Distance Functions for 3D Scene Reconstruction. ECCV , 2022.

K. N. Kutulakos and S. M. Seitz. A theory of shape by space carving. IJCV , 2000.

C.-H. Lin, C. Kong, and S. Lucey. Learning Efficient Point Cloud Generation for Dense 3D Object Reconstruction. AAAI , 2018.

D. B. Lindell, J. N. P. Martel, and G. Wetzstein. AutoInt: Automatic Integration for Fast Neural Volume Rendering. CVPR , 2021.

C. Liu, Z. Li, J. Yuan, and Y. Xu. NeuLF: Efficient Novel View Synthesis with Neural 4D Light Field. Eurographics Symposium on Rendering , 2022.

L. Liu, J. Gu, K. Z. Lin, T.-S. Chua, and C. Theobalt. Neural Sparse Voxel Fields. NeurIPS , 2020.

S. Liu, S. Saito, W. Chen, and H. Li. Learning to Infer Implicit Surfaces without 3D Supervision. NeurIPS , 2019.

S. Liu, Y. Zhang, S. Peng, B. Shi, M. Pollefeys, and Z. Cui. DIST: Rendering Deep Implicit Signed Distance Function with Differentiable Sphere Tracing. CVPR , 2020.

I. Loshchilov and F. Hutter. Sgdr: Stochastic gradient descent with warm restarts. ICLR , 2017.

W. Matusik, C. Buehler, R. Raskar, S. J. Gortler, and L. McMillan. Image-based visual hulls. SIGGRAPH , 2000.

L. Mescheder, M. Oechsle, M. Niemeyer, S. Nowozin, and A. Geiger. Occupancy Networks: Learning 3D Reconstruction in Function Space. CVPR , 2019.

B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. ECCV , 2020.

T. Müller, A. Evans, C. Schied, and A. Keller. Instant Neural Graphics Primitives with a Multiresolution Hash Encoding. SIGGRAPH , 2022.

R. Mur-Artal, J. M. M. Montiel, and J. D. Tardos. ORB-SLAM: a versatile and accurate monocular SLAM system. TRO , 31(5), 2015.

T. Neff, P. Stadlbauer, M. Parger, A. Kurz, C. R. A. Chaitanya, A. Kaplanyan, and M. Steinberger. DONeRF: Towards Real-Time Rendering of Neural Radiance Fields using Depth Oracle Networks. EGSR , 2021.

M. Niemeyer, J. T. Barron, B. Mildenhall, M. S. M. Sajjadi, A. Geiger, and N. Radwan. RegNeRF: Regularizing Neural Radiance Fields for View Synthesis from Sparse Inputs. CVPR , 2022.

J. Ost, I. Laradji, A. Newell, Y . Bahat, and F. Heide. Neural Point Light Fields. CVPR , 2022.

O. Ozyesil, V. Voroninski, R. Basri, and A. Singer. A Survey of Structure from Motion. Acta Numerica , 2017.

J. J. Park, P. Florence, J. Straub, R. Newcombe, and S. Lovegrove. DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation. CVPR , 2019.

S. Peng, M. Niemeyer, L. Mescheder, M. Pollefeys, and A. Geiger. Convolutional Occupancy Networks. ECCV , 2020.

D. Rebain, W. Jiang, S. Yazdani, K. Li, K. M. Yi, and A. Tagliasacchi. DeRF: Decomposed Radiance Fields. CVPR , 2021.

C. Reiser, S. Peng, Y . Liao, and A. Geiger. KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPs. ICCV , 2021.

E. Remelli, A. Lukoianov, S. R. Richter, B. Guillard, T. Bagautdinov, P. Baque, and P. Fua. MeshSDF: Differentiable Iso-Surface Extraction. NeurIPS , 2020.

M. Roberts and N. Paczan. Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding. ICCV , 2021.

S. Saito, Z. Huang, R. Natsume, S. Morishima, A. Kanazawa, and H. Li. PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization. ICCV , 2019.

M. S. M. Sajjadi, H. Meyer, E. Pot, U. Bergmann, K. Greff, N. Radwan, S. Vora, M. Lucic, D. Duckworth, A. Dosovitskiy, J. Uszkoreit, T. Funkhouser, and A. Tagliasacchi. Scene Representation Transformer: Geometry-Free Novel View Synthesis Through Set-Latent Scene Representations. CVPR , 2022.

J. L. Schonberger and J.-M. Frahm. Structure-from-Motion Revisited. CVPR , 2016.

V. Sitzmann, J. N. P. Martel, A. W. Bergman, D. B. Lindell, and G. Wetzstein. Implicit Neural Representations with Periodic Activation Functions. NeurIPS , 2020.

V. Sitzmann, S. Rezchikov, W. T. Freeman, J. B. Tenenbaum, and F. Durand. Light Field Networks: Neural Scene Representations with Single-Evaluation Rendering. NeurIPS , 2021.

V. Sitzmann, M. Zollhöfer, and G. Wetzstein. Scene Representation Networks: Continuous 3D-StructureAware Neural Scene Representations. NeurIPS , 2019.

L. N. Smith and N. Topin. Super-convergence: Very fast training of neural networks using large learning rates. Artificial intelligence and machine learning for multi-domain operations applications , 2019.

S. Song, F. Yu, A. Zeng, A. X. Chang, M. Savva, and T. Funkhouser. Semantic Scene Completion from a Single Depth Image. CVPR , 2017.

M. Suhail, C. Esteves, L. Sigal, and A. Makadia. Light Field Neural Rendering. CVPR , 2022.

T. Takikawa, J. Litalien, K. Yin, K. Kreis, C. Loop, D. Nowrouzezahrai, A. Jacobson, M. McGuire, and S. Fidler. Neural Geometric Level of Detail: Real-time Rendering with Implicit 3D Shapes. CVPR , 2021.

J. Tang, X. Han, J. Pan, K. Jia, and X. Tong. A Skeleton-bridged Deep Learning Approach for Generating Meshes of Complex Topologies from Single RGB Images. CVPR , 2019.

M. Tatarchenko, A. Dosovitskiy, and T. Brox. Octree Generating Networks: Efficient Convolutional Architectures for High-resolution 3D Outputs. ICCV , 2017.

A. Trevithick and B. Yang. GRF: Learning a General Radiance Field for 3D Representation and Rendering. ICCV , 2021.

B. Wang, L. Chen, and B. Yang. DM-NeRF: 3D Scene Geometry Decomposition and Manipulation from 2D Images. ICLR , 2023.

B. Wang, Z. Yu, B. Yang, J. Qin, T. Breckon, L. Shao, N. Trigoni, and A. Markham. RangeUDF: Semantic Surface Reconstruction from 3D Point Clouds. arXiv:2204.09138 , 2022.

H. Wang, J. Ren, Z. Huang, K. Olszewski, M. Chai, Y. Fu, and S. Tulyakov. R2L: Distilling Neural Radiance Field to Neural Light Field for Efficient Novel View Synthesis. ECCV , 2022.

N. Wang, Y. Zhang, Z. Li, Y. Fu, W. Liu, and Y.-G. Jiang. Pixel2Mesh: Generating 3D Mesh Models from Single RGB Images. ECCV , 2018.

P. Wang, L. Liu, Y. Liu, C. Theobalt, T. Komura, and W. Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. NeurIPS , 2021.

P. Wang, Y. Liu, G. Lin, J. Gu, L. Liu, T. Komura, and W. Wang. Progressively-connected Light Field Network for Efficient View Synthesis. arXiv:2207.04465 , 2022.

Q. Xu, W. Wang, D. Ceylan, R. Mech, and U. Neumann. DISN: Deep Implicit Surface Network for High-quality Single-view 3D Reconstruction. NeurIPS , 2019.

B. Yang, S. Rosa, A. Markham, N. Trigoni, and H. Wen. Dense 3D Object Reconstruction from a Single Depth View. TPAMI , 2019.

B. Yang, S. Wang, A. Markham, and N. Trigoni. Robust Attentional Aggregation of Deep Feature Sets for Multi-view 3D Reconstruction. IJCV , 128:53-73, 2020.

A. Yu, S. Fridovich-keil, and A. Kanazawa. Plenoxels: Radiance Fields without Neural Networks. CVPR , 2022.

S. Zakharov, W. Kehl, A. Bhargava, and A. Gaidon. Autolabeling 3D Objects with Differentiable Rendering of SDF Shape Priors. CVPR , 2020.

A. Zeng, S. Song, M. Niessner, M. Fisher, J. Xiao, and T. Funkhouser. 3DMatch: Learning Local Geometric Descriptors From RGB-D Reconstructions. CVPR , 2017.

C. Zhang, Z. Cui, Y. Zhang, B. Zeng, M. Pollefeys, and S. Liu. Holistic 3D Scene Understanding from a Single Image with Implicit Representation. CVPR , 2021.

S. Zhi, T. Laidlow, S. Leutenegger, and A. J. Davison. In-Place Scene Labelling and Understanding with Implicit Scene Representation. ICCV , 2021.

C. Zou, E. Yumer, J. Yang, D. Ceylan, and D. Hoiem. 3D-PRNN: Generating Shape Primitives with Recurrent Neural Networks. ICCV , 2017.

A Appendix

A.1 Network Architecture and Training Details

We provide the details of our ray-surface distance field and the auxiliary dual-ray visibility classifier.

A.1.1 Ray-surface Distance Field

We use a 13-layer SIREN [63] with 1024 hidden units to learn a ray-surface distance field. We set the batch size as 8192 and train the network for 10 epochs with the Adam [35] optimizer and a cosine annealing strategy [44] with the learning rate initialized as 10 -5 and decayed to 10 -8 .

Radiance Branch: To learn a radiance field, we take an additional 2-layer SIREN with 1024 hidden units as the radiance branch and output RGB values with sigmoid activation. For optimization together with the distance field, we adopt the same multi-view consistency as the distance field with a mean-squared loss and set the loss weight to 1.

A.1.2 Dual-ray Visibility Classifier

We take a SIREN layer as the ray-feature encoder g () and another SIREN layer as the point-feature encoder k () , followed by a 7-layer SIREN with 512 hidden units and output the visibility with sigmoid activation. Both the ray-feature encoder and the point-feature encoder are with 512 hidden units. The batch sizes are 2048 for Blender dataset and ScanNet dataset, and 1024 for DM-SR dataset. We train the network for 5 epochs with the Adam optimizer and a cycle annealing strategy [66] with the maximum learning rate 10 -4 .

A.1.3 Training Data Generation

For a scan with extrinsic parameters [ R | t ] and intrinsic parameters K, all rays of this scan start from the camera position o = ( tx , ty , t z ) . The ray orientation of pixel ( u , v ) can be denoted as m 0 = RK -1 ( u v 1 ) , and the unit ray direction is m = m 0 / ∥ m 0 ∥ . Given an oriented ray r starting from o and a pre-defined sphere with a diameter D , we follow the two-sphere parameterization [8] to compute two intersections p in , p out and construct our input parameters r =( θ in , φ in , θ out , φ out ) .

To obtain the ray-surface distance d from the raw depth value ˙ d of pixel ( u , v ) , we have d = ˙ d √ ( u -cy ) 2 +( v -cx ) 2 + f 2 / f -d 0, where f , cy , cx are the camera intrinsic parameters ( i . e ., focal length, the center coordinates at y-axis and x-axis of image plane), and d 0 = ∥ p in -o ∥ .

For a ray r at the pixel (u, v) of a specific scan, we have the first ray-sphere intersection p in and ray direction m , then we can compute the ray-surface point p = p in + dm . To reproject this point to the remaining ( K -1 ) scans, for example, we calculate the pixel coordinate ( u k , v k ) of the k th scan using its extrinsic parameters [ R | t ] k and intrinsic parameters K k :

Then we can query the raw depth value at ( u k , v k ) from the k th scan, and obtain its ray-surface distance d k u k , v k and the ray parameters r k u k , v k (simplified as d k , r k ) following the previous steps. Besides, we calculate the multi-view distance ˜ d k between the ray-surface point p and the first intersection p in k of ray r k : ˜ d k = ∥ p -p in k ∥ . We set a closeness threshold ε = 10 millimeters to determine whether the projected k th ray is visible:

For both training and inference, the values of input parameters are normalized to be [-1, 1]. In particular, the latitude θ : = 2 θ / π -1 and the longitude φ : = φ / π . The input surface hitting point ( x 1 , y 1 , z 1 ) of our visibility classifier is also normalized to [-1, 1] by subtracting the sphere center and then dividing by the sphere radius D / 2. The ground-truth ray-surface distance d and the multi-view distance ˜ d k are scaled to be [0, 1] by dividing by the sphere diameter D .

A.2 Derivation of Surface Normal

In this section, we derive the formula of surface normal from our ray-surface distance field.

As mentioned in Section A.1.3, given a ray starting from o with direction m and a fix-sized sphere with a diameter D centered at the coordinate origin, we can construct the input parameters r = ( θ in , φ in , θ out , φ out ) from a ray-sphere intersect function F ( o , m , D ) = r .

By converting m to the spherical coordinate system, we have ∂ r ∂θ m and ∂ r ∂φ m , which is the gradient of the above intersection function F () w.r.t. the ray direction m .

As shown in Figure 6, to compute the surface normal of a predicted surface point ˆ p along the ray direction m , we define a sphere of radius R = D ˆ d + d 0 centered at the camera position o using spherical coordinates:

In general, the formula for a unit normal vector is n = ∂ Φ ∂φ m × ∂ Φ ∂θ m ∥ ∂ Φ ∂φ m × ∂ Φ ∂θ m ∥ . From Eq. 7, we have

Here, we can replace ∂ ˆ d ∂θ m = ∂ ˆ d ∂ r ∂ r ∂θ m and ∂ ˆ d ∂φ m = ∂ ˆ d ∂ r ∂ r ∂φ m , so the surface normal n can be denoted as a function Q ( ∂ ˆ d ∂ r , r , D , ˆ d ) .

A.3 Additional Implementation Details

In this section, we provide details of implementation and some minor adaptations of baselines, as well as the workflow for 3D shape reconstruction.

A.3.1 Baselines

OF/DeepSDF/NDF: Pre-processing on mesh before training is required for OF/DeepSDF/NDF. For fair comparisons, we use all depth images from training views to reconstruct the training mesh, which is then used for sampling occupancies/signed distances/unsigned distances of a 256 3 voxel grid for training. We follow the official settings of hyperparameters and network architecture to train on the three datasets. For inference, we apply

ˆ

p

d

0

D

ˆ

d

o

p

in

x

y

z

sphere tracing for DeepSDF and NDF to render depth images from testing views. The sphere tracing will stop after 100 iterations or when the predicted distance is smaller than a threshold at 0 . 005 meters. To avoid missing the intersections, we follow the damped sphere tracing [14] from NDF, i . e ., marching with α · f ( p ) with α = 0 . 6. Since sphere tracing for OF can only take a fixed step size ( e . g ., 0 . 005 meters), it requires about 1000 iterations to march the surface, which is time-consuming. We thus construct a testing mesh by querying a value grid of occupancies and use ray-mesh intersections to render depth images for evaluation.

NeuS/DS-NeRF: In general, we follow the same network architecture, training schedule, and color supervision from NeuS and DS-NeRF. The vanilla DS-NeRF adopts sparse 3D points from structurefrom-motion (SFM) and uses the reprojection error for depth supervision. For a fair comparison, we provide the same dense depth supervision from training views as ours on all three datasets for both

NeuS and DS-NeRF. For the experiments of Group 1, i . e ., 3D shape representation only, we remove the color supervision of NeuS and DS-NeRF and leave depth supervision only.

LFN/PRIF: We adopt the same network architecture and hyperparameters from the official settings of LFN/PRIF. In particular, we train LFN/PRIF for 10 epochs on each dataset, which is the same as ours. Since the proposed depth computation from LFN makes it difficult to obtain dense depths, we modify the output of the last MLP layer to predict depth value with supervision. For the experiments of Group 2, i . e ., 3D Shape and Appearance Representations, we extend a 2-layer color branch both for LFN and PRIF with appearance supervision. Note that, the LFN/PRIF results are obtained without our dual-ray visibility classifier. We additionally conduct experiments on LFN/PRIF with our own classifier for comparison. Figure 7 and Table 10 show that our method still achieves the best performance.

A.3.2 3D Shape Reconstruction for Evaluation

For a fair comparison, we initially render depth images of testing views for all baselines and ours. For OF/DeepSDF/NDF, we set the depth value to 0 for those unconverged rays ( i . e ., no intersection during sphere tracing). For PRIF and ours, we apply specific outlier removal strategies. Subsequently, we use TSDF fusion on the processed depth images to obtain complete meshes for further evaluation.

A.4 Additional Ablation Studies

In this section, we show more details of ablation studies, including ablations on dual-ray visibility classifier, ray-surface distance network, training strategies, and post-processing.

A.4.1 Ablation Studies on the Dual-ray Visibility Classifier

The key issue of a naïve ray based distance function is the lack of generalization ability across novel views during testing, fundamentally because training a single ray based network (without classifier) tends to fit all ray distance data pairs of the training set, but cannot guarantee the consistency of surface distances between seen (training) rays and unseen (testing) rays. As demonstrated in Tables 11, 12, 13 and Figure 8, the absence of the classifier significantly impacts testing performance, resulting in notably inferior results compared to those optimized with the well-trained classifier. To provide better clarity, we also present visualizations of the visibility predictions in Figure 9. Moreover, the results of our classifier with noises in Tables 14, 15, 16 and Figure 10 demonstrate that our visibility classifier is reliable for multi-view consistency.

w/o classifier

Ours

Mesh

Novel view

one-stage

w=0.979,d=-0.014m

w=0.000,d=-0.382m

w=0.872,d=-0.037m

w=0.666,a=-0.064m

LFN +

Our Visibility

Classifier

Ours

PRIF +

Our Visibility

Classifier

A.4.2 Ablation Studies on the Ray-surface Distance Network

We perform the following ablation studies on the ray-surface distance network.

Multi-view Rays Sampling: Here we report the detailed quantitative results of each scene on Blender dataset in Tables 17 and 18.

Network Architecture: We conduct additional experiments on the Blender dataset as shown in Table 19. We can see that our framework tends to favor a wider or deeper network. However, to extensively explore an optimal network architecture is non-trivial and we leave it for our future work.

Multi-view Consistency Loss ℓ mv : In our multi-view consistency loss function ℓ mv (Eq. 3), we simply change it from ℓ 1 to ℓ 2 optimization, and conduct an additional experiment on Reception of DMSR dataset. From Table 20 and Figure 12, we can see that ℓ 2 based loss function can better reconstruct thin structures but with more noisy results than ℓ 1 based loss. In this regard, it is of great interest to fully address the recovering task of thin structures and we leave it for future exploration.

Distance

Mesh

𝜎

!

= 1.0

𝜎

!

= 0.5

𝜎

!

= 0.1

Ours (

𝝈

𝟐

= 𝟎

)

Distance

Nromal

Train view

Novel view

1%

5%

10%

Ours (100%)

w/o post-processing

w/ post-processing

A.4.3 Ablation Studies on Training Strategies

Sparse Depth Supervision: We additionally provide experimental results of our method using sparse depth values in supervision in Tables 21, 22 and Figure 11.

One-stage RayDF: We present detailed quantitative and qualitative results of one-stage training of our pipeline, the performance drops noticeably as shown in Tables 23, 24 and Figure 8.

A.4.4 Ablation Studies on Ray Parameterization

Choices of Ray Parameterization: We discuss our ray parameterization in detail in Section A.1.3. In fact, our pipeline supports various ray parameterizations. To verify this, we conduct ablation studies by replacing our spherical coordinate with the one used in LFN/PRIF. As shown in Table 25 and Figure 14, PRIF parameterization can also achieve satisfactory performance, while LFN parameterization is inferior in our pipeline.

PRIF Param.

Ray-surface

Distance Network +

Visibility Classifier

Ray-surface

Distance Network +

(Our Param. +

Visibility Classifier)

LFN Param.

Ours

Ray-surface

Distance Network +

Visibility Classifier

Ray-surface

Distance Network +

(Our Param. +

Visibility Classifier)

(2) Artifacts from Spherical Coordinate: The artifacts of fixed patterns in the predicted distance maps (as shown in Figure 15 ( red )) are caused by the choice of spherical coordinate. We conduct additional experiments given two different choices of ray parameterizations. As shown in Table 25, our applied spherical coordinate can achieve the best ADE scores, and the caused artifacts can be easily fixed by using PRIF's ray parameterization as shown in Figure 15 ( green ). We leave it for our future work to fully address such an issue.

A.4.5 Ablation Studies on Post-processing

We follow the outlier removal step in Section 3.4 as postprocessing to obtain clean 3D point clouds for further mesh reconstruction and evaluation. In all experiments, we set the outlier threshold as 5 for post-processing. We conduct additional experiments w/ and w/o the outlier removal step on the Blender dataset. In Table 26, the experiment without post-processing yields higher CD errors compared to the one with post-processing. The qualitative results are shown in Figure 13.

A.5 Limitations of RayDF

Ours

Ours + PRIF Param.

Since our RayDF takes the conventional spherical parameterization for the input ray, i . e ., a sphere is predefined to bound the target 3D scene, our RayDF with such a parameterization cannot represent the 3D scene with rays starting from a position inside the sphere. However, the proposed distance field with a visibility classifier of our RayDF is flexible to different ray parameterizations. How to integrate distinct parameterizations of input rays for different types of 3D scenes with our ray-surface distance field and the dual-ray visibility classifier can be a future exploration direction.

A.6 Additional Quantitative Results

In this section, we report the detailed quantitative results of each scene on the three datasets.

Blender: In Tables 28 and 29, we show the quantitative results of the experiments in Group 1. Our RayDF achieves significantly better performance than the other baselines on most scenes, especially on complex scenes ( e . g ., Drums, Ficus, and Ship). In Tables 30, 31, 32, 33 and 34, we report the geometry and appearance metrics in Group 2. Our method achieves comparable performance with DS-NeRF for novel view synthesis while recovering the 3D shape with fewer outliers.

DM-SR: In Tables 35 and 36, we report the quantitative comparisons in Group 1 and our RayDF performs better than the other baselines on those indoor scenes with more complex objects. As shown in Tables 37, 38, 39, 40 and 41, our method still achieves high-quality results for novel view synthesis on all scenes in Group 2 experiments.

ScanNet: In Tables 42 and 43, our RayDF performs significantly better than the other baselines on the real-world scenes. To represent these real-world 3D scenes with much more noise and complex geometry, our method outperforms all baselines while obtaining photo-realistic novel view synthesis results, as shown in Tables 44, 45, 46, 47 and 48.

Moreover, we provide comparisons of optimization time with baselines in Table 27. We compare the average training time of our method and all baselines on each scene of Blender dataset, using a single NVIDIA RTX 3090 GPU card and a CPU of AMD Ryzen 7. Since we apply a two-stage training strategy, our method is not as fast as baselines during optimization. Nevertheless, once our ray-surface distance network is optimized, it can achieve superior efficiency in rendering novel views, as shown in Table 1.

A.7 Additional Qualitative Results

Time

OF [46]

0.2

DeepSDF [54]

0.6

NDF [14]

2.1

NeuS [77]

10.2

DS-NeRF [19]

22.8

LFN [64]

0.9

PRIF [23]

2.2

RayDF (Ours)

24.9

We provide additional qualitative results on the three datasets as follows, including distance map, mesh from Group 1, and radiance map, textured mesh from Group 2. We also report the visualizations of our surface normal, which is computed following Section A.2.

Blender: In Figures 16 and 17, we show the qualitative results on Materials and Drums . In particular, Materials is a scene that contains empty spaces between multiple objects. Our RayDF predicts

distance with fewer outliers near the object surfaces. Our surface normal indicates that our method recovers the 3D scene with a smoother surface while preserving more details of the complex geometry.

DM-SR: In Figures 18 and 19, we report the qualitative results on Reception and Study . Our RayDF predicts accurate distances and hence is capable of reconstructing more detailed structures.

ScanNet: In Figures 20 and 21, we show the qualitative results on Scene0005_00 and Scene0030_00 , which are challenging real-world scenes. Our method recovers the 3D real-world scenes with reasonable details while predicting realistic radiance results.

Distance

Mesh

Raidance

Textured Mesh

DS-NeRF

LFN

PRIF

GT

Ours

Normal

NeuS

OF

DeepSDF

NDF

NeuS

DS-NeRF

LFN

PRIF

Ours

GT

GT

OF

DeepSDF

NDF

NeuS

DS-NeRF

LFN

PRIF

Ours

DeepSDF

NDF

DS-NeRF

Ours

NeuS

GT

OF

DeepSDF

NDF

NeuS

GT

NeuS

LFN

PRIF

Ours

DS-NeRF

Distance

OF

DeepSDF

NDF

NeuS

GT

Mesh

DS-NeRF

LFN

PRIF

Ours

Normal

Raidance

DeepSDF

NDF

DS-NeRF

Ours

NeuS

Textured Mesh

DS-NeRF

LFN

PRIF

Ours

OF

DeepSDF

NDF

DS-NeRF

LFN

PRIF

GT

Ours

Distance

Mesh

OF

DeepSDF

NDF

DS-NeRF

LFN

PRIF

GT

Ours

DS-NeRF

LFN

PRIF

GT

Ours

DeepSDF

NDF

DS-NeRF

Ours

NeuS

NeuS

NeuS

NeuS

Normal

Raidance

Textured Mesh

OF

DeepSDF

NDF

DS-NeRF

LFN

PRIF

GT

Ours

OF

DeepSDF

NDF

DS-NeRF

LFN

PRIF

GT

Ours

DS-NeRF

LFN

PRIF

GT

Ours

DeepSDF

NDF

DS-NeRF

Ours

NeuS

NeuS

NeuS

NeuS

Distance

Mesh

Normal

Raidance

Textured Mesh

OF

DeepSDF

NDF

DS-NeRF

LFN

PRIF

GT

Ours

OF

DeepSDF

NDF

DS-NeRF

LFN

PRIF

GT

Ours

DS-NeRF

LFN

PRIF

GT

Ours

DeepSDF

NDF

DS-NeRF

Ours

NeuS

NeuS

NeuS

NeuS

Distance

Mesh

Normal

Raidance

Textured Mesh

OF

DeepSDF

NDF

DS-NeRF

LFN

PRIF

GT

Ours

OF

DeepSDF

NDF

DS-NeRF

LFN

PRIF

GT

Ours

DS-NeRF

LFN

PRIF

GT

Ours

DeepSDF

NDF

DS-NeRF

Ours

NeuS

NeuS

NeuS

NeuS

Distance

Mesh

Normal

Raidance

Textured Mesh

Peer reviews:
comments: SUMMARY:
This paper proposes MucRays, which imposes ray-based neural functions with multi-view geometry consistency. This framework contains three-parts: ray-surface distance field, dual-ray visibility classifier, and multi-view consistency optimization strategy. Quantitative results of MucRays surpass the existing coordinate- or ray- based networks for synthetic or real-world scene modeling. This ray-based method achieves 1000x faster for depth image inference.

STRENGTHS:
1.	The dual-ray visibility classifier is interesting. It is well-motivated and acts as the foundation of the multi-view consistency optimization. 
2.	The quantitative results are good. The high performance of MucRays on DAE metric proves its ability to render more accurate depth images.

WEAKNESSES:
1.	The qualitative results does not satisfy me. It seems that, MucRays has difficulty in representing thin structures. For example, in the Reception scene in Figure 4 (Appendix), the arm of the desk lamp is missing in both distance map and mesh. I would like to see a discussion about such phenomenon. 
2.	I think NeuS (NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction) is a better baseline than DeepSDF because it could use both depth and color for supervision. 
3.	I assume that this paper focused on better geometry modeling. However, it seems that the geometric result (mesh result) on Lego (Figure 5) is over smoothed compared with DS-NeRF or NDF. Is it caused by TSDF or the nature of ray distance representation itself?

QUESTIONS:
Main questions are in Weakness part. 

More detailed comments: 

The authors should use consistent abbreviation. For example, L. 80 use UDF for paper [13], while the remaining contents use NDF for [13].

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a new strategy to design ray-based neural representation of 3D shapes. Ray-based approaches towards shape representation is a recently emerging idea that bypasses extensive point-based evaluation required by conventional methods like signed distance function. A key missing component in existing ray-based neural representations is the lack of multi-view consistency. This paper proposes training a dual-ray visibility classifier and using that network's output to guide the training of the network for ray-distance prediction. The method is extensively evaluated on synthetic and real-world datasets, showing its superiority over previous ray-based methods in predicting ray hit points from novel views. Moreover, this paper also includes evaluations of color prediction of this ray-based method, showing it achieving comparable performance to previous efficient representations.\n\nSTRENGTHS:\n- Addresses a key issue of multi-view inconsistency in current ray-based scene representations.\n- Clear writing and figure presentation.\n- Extensive evaluation showing the efficacy of the proposed strategy, outperforming previous baselines without slowing down inference\n\nWEAKNESSES:\n- The proposed method operates under the setting where depth maps are available for all views, which is okay, but the paper would be more complete if it discusses how bad the results would become if depth is unknown, which is actually the setting of LFN.\n- Not totally clear how the baselines are implemented. For example, when using LFN and PRIF, the supplementary material mentions that it uses the same architecture as the official setup. However, it is not clear whether the LFN/PRIF results are obtained still with the proposed classifier. At least based on the writing, it does not eliminate the ambiguity.\n- The proposed method also adopts a different ray parametrization than LFN and PRIF. It is not clear what the reason is behind this design choice.\n- The proposed method is implemented as a 13-layer SIREN with 1024 hidden channels, which is different than previous baselines. The paper does not include any analysis on the impact of varying layer depth and hidden channels.\n- In L123, the paper says the network predictions \"must satisfy a transformation equation\", without defining what this equation is.\n- Related work misses any discussion on space carving / visual hull (e.g., [Matusik 2000], [Kutulakos 2000]), which are important early concepts behind the idea of visibility checking proposed in this work. Related work also misses some recent papers on light field / ray-based representation (e.g. HyperReel [CVPR 2023], SRT [CVPR 2022], SIGNET [ICCV 2021]).\n- L237 typo: \"RTX 30390 GPU\"\n\nQUESTIONS:\n- In Table 5, the performance drops significantly after removing the classifier, and it actually becomes much worse than the various baselines in Table 2. This is confusing and relates to the above-mentioned weakness about \"Not totally clear how the baselines are implemented\". Do the baseline methods included in Table 2, specifically LFN and PRIF, involve a classifier? That would totally change how to correctly interpret the results in Table 5.\n- If the baseline methods do not involve a classifier, and they are only just about a different ray parametrization with slightly smaller network, then the natural question becomes: how does LFN/PRIF + classifier perform?\n- In general, how does the ray parametrization affect the performance? It appears that it would have minor impact compared to the role of classifier, but the paper does not provide enough information to draw any conclusion.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposed a ray-based neural rendering method that is able to achieve good reconstruction quality from depth maps or RGBD inputs using only one network evaluation per pixel during evaluation. \nThe method requires two networks: a ray-surface distance network, and a dual-ray visibility classifier. In the first stage, the visibility classifier is trained on the multi-view depth maps. In the second stage, the ray-surface distance network can be trained with both ground truth rays from the depth maps, and random sampled novel rays. the visibilities of the random rays are determined by te visibility classifier, and only the distance of the visible rays are supervised. The proposed approach is extremely fast, and achieves competitive quality on three RGBD datasets.\n\nSTRENGTHS:\n* The proposed two-stage method is novel.\n* The proposed method is extremely fast during evaluation -- only one network evaluation is needed per ray.\n* The method achieves competitive performance on both synthetic and real datasets.\n* The paper provided useful insights on the derivation of surface normals from the proposed ray distance field.\n\nWEAKNESSES:\n* The method requires ground truth depth information for training, which is not the case for light field works such as [57]. This can significantly limit its use cases.\n* There lacked comparisons with some good performing light field methods such as [57].\n* The method relies on the visibility classifier to generalize to novel rays, which can potentially be unreliable.\n* Ther performance of the proposed method is lacking, especially according to the results in the appendix.\n* It might help understand the effect of visibility network better if there are visualizations of the visibility network predictions.\n\nQUESTIONS:\nThere appear to be some fixed-pattern noise artifacts in the demo video. I wonder what is the cause?\n\nLIMITATIONS:\nThe limitations and broader impacts of the works are adequately discussed in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}

comments: SUMMARY:
This paper proposes a framework MucRays for 3D shape representation. Specifically, the authors formulate 3D shapes as ray-based neural functions and incorporate multi-view geometry consistency to improve the performance. For the learning of multi-view geometry consistency, an auxiliary network is introduced to classify the mutual visibility of two sampled rays. Experiments are conducted on various datasets, showing good performance.

STRENGTHS:
1. The paper is easy to follow and understand.
2. The authors propose an effective framework for 3D shape representation, showing good performance in various datasets.

WEAKNESSES:
1. This paper only compares rendering time, but training/optimization time is not compared.
2. The ablation study is not convincing. The authors should compare with and without postprocessing (Section 3.2).
3. Table 5, the performance of "w/o classifier" is quite worse than the full model, and the error is very high. It would be better to discuss some possible reasons.
4. 2-stage training is not elegant. Is that possible to train these two networks simultaneously?
5. Line 57, "ray-surface distance field" representation is introduced by previous works and cannot be summarized as one contribution of this work.

QUESTIONS:
See weaknesses.

LIMITATIONS:
Negative impacts are mentioned in the paper.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-10