SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zhuoman Liu Bo Yang* Yan Luximon Ajay Kumar Jinxi Li
{zhuo-man.liu, jinxi.li}@connect.polyu.hk bo.yang@polyu.edu.hk
In this paper, we study the problem of continuous 3D shape representations. The majority of existing successful methods are coordinate-based implicit neural representations. However, they are inefficient to render novel views or recover explicit surface points. A few works start to formulate 3D shapes as ray-based neural functions, but the learned structures are inferior due to the lack of multi-view geometry consistency. To tackle these challenges, we propose a new framework called RayDF. It consists of three major components: 1) the simple ray-surface distance field, 2) the novel dual-ray visibility classifier, and 3) a multi-view consistency optimization module to drive the learned ray-surface distances to be multi-view geometry consistent. We extensively evaluate our method on three public datasets, demonstrating remarkable performance in 3D surface point reconstruction on both synthetic and challenging real-world 3D scenes, clearly surpassing existing coordinate-based and ray-based baselines. Most notably, our method achieves a  $1000 \times$  faster speed than coordinate-based methods to render an  $800 \times 800$  depth image, showing the superiority of our method for 3D shape representation. Our code and data are available at https://github.com/vLAR-group/RayDF
Learning accurate and efficient 3D shape representations is crucial for many cutting-edge applications in the fields of machine vision and robotics. Recent advances in 3D coordinate-based neural representations including occupancy fields (OF) [46, 11], un-signed distance fields (U/SDF) [54, 14], and radiance fields (NeRF) [47], have shown great potential to recover complex shapes from RGB/D images and/or point clouds. Although these methods and their variants have achieved excellent performance in a wide range of downstream tasks such as shape reconstruction [55, 58, 60], novel view synthesis [4, 72, 51, 5], and scene understanding [85, 86, 73], obtaining 3D shapes or 2D views from their trained networks is computationally expensive, due to the requirement of extensive network evaluations to regress surface points.
Very recently, a number of works start to represent 3D models as ray-based neural functions. By simply taking individual light rays as input, the methods LFN [64] and NeuLF [40] learn to directly predict the radiance (RGB) values, while PRIF [23] and DDF [3] straightly estimate the surface hitting points of input rays. Compared with coordinate-based representations, these ray-based methods are clearly more efficient to infer 3D geometry and render 2D views, as every single ray only needs to query the trained network once forward. Nevertheless, the learned 3D geometries by these methods are still lack of fidelity, primarily because they fail to explicitly take into account geometry consistency across multiple views, resulting in the network likely over-fitting individual training rays but unable to generalize to unseen rays in testing.
In this paper, we aim to address this key issue of ray-based neural representations by explicitly integrating multi-view geometry consistency into the network design. In particular, we introduce a
Figure 1: The general workflow and components of our framework.
new pipeline as shown in Figure 1. It consists of two independent neural networks together with a particular training module: 1) the main ray-surface distance network, 2) an auxiliary dual-ray visibility classifier, and 3) a multi-view consistency optimization module.
The main network simply takes a ray as input and directly infers the distance between ray origin and its hitting point on the surface. Basically, this network elegantly represents 3D shapes as ray-based implicit functions, denoted as ray-surface distance fields, keeping the unique advantage in efficiency for shape extraction and rendering. The auxiliary network takes a pair of rays as input and predicts their mutual visibility. In fact, this is a simple binary classifier, aiming at distinguishing whether any two rays hit at the same surface point or not, i.e., the mutual visibility. Having a trained auxiliary network at hand, the multi-view consistency optimization module specifies how to effectively leverage the learned dual-ray visibility to train the main network, thus driving learned ray-surface distances to be multi-view consistent from any seen or unseen viewing angles.
Since our ray-surface distance fields primarily aim at representing accurate 3D shapes, the whole pipeline is designed to be trained on depth images, although light fields (radiance) can be optionally learned in parallel if color images are also available in training. In this regard, the closest works to ours are PRIF [23] and DDF [3], neither of which explicitly considers the multi-view geometry consistency. Overall, compared with all existing coordinate-based representations, our method keeps the superiority in efficiency thanks to the ray-based formulation. Compared with the existing ray-based approaches, ours excels at learning accurate 3D geometries thanks to the multi-view consistency for ray-surface distance fields. Our method is called RayDF and our contributions are:
- We employ a straightforward ray-surface distance field for representing 3D shapes. This formulation is inherently more efficient than existing coordinate-based representations.
- We design a new dual-ray visibility classifier to learn the spatial relationships of any pair of rays, enabling the learned ray-surface distance fields to be multi-view geometry consistent.
- We demonstrate superior 3D shape reconstruction accuracy and efficiency on multiple datasets, showing significantly better results than existing coordinate-based and ray-based baselines.
Explicit 3D Shape Representations: Classic methods to recover explicit 3D geometry of objects and scenes mainly include SfM [53] and SLAM [7] systems such as Colmap [62] and ORB-SLAM [49]. Another classic method is space carving [45, 37], which involves the process of carving out the voxel grid by utilizing multiple distinct views to obtain a robust approximation within the voxel space. To model explicit 3D structures, deep learning-based methods have shown impressive progress in recovering voxel grids [15, 80, 81], point clouds [21], octree [71], polygon meshes [34, 26] and shape primitives [87]. A comprehensive survey of these methods can be found in [6, 28]. Thanks to the large-scale datasets [10] for training sophisticated neural networks [29, 31], these methods demonstrate excellent results in many downstream tasks such as shape reconstruction [76, 70], generation [38], and semantic scene perception [67, 17, 25]. However, the fidelity of these discrete shape representations is primarily limited by their spatial resolutions and memory footprint.
Coordinate-based Implicit 3D Shape Representations: To avoid the discretization issue of traditional 3D representations, there has been a growing interest in developing implicit neural 3D representations using simple MLPs to represent continuous 3D shapes. Inspired by the seminal coordinate-based methods [11, 46, 54] which have shown great success in encoding high-quality 3D shapes, a plethora of follow-up works have been developed to tackle various vision tasks in [65, 79, 2, 12, 13, 20, 33, 43, 42, 27, 83]. These coordinate-based representations can be generally classified as: 1) occupancy fields (OF) [46, 11], 2) signed distance fields (SDF) [54], 3) unsigned
Figure 2: The left block illustrates the spherical parameterization of an input ray  $\mathbf{r}$ , and the right block shows our simple MLP-based ray-surface distance field  $f_{\Theta}$ .
distance fields (NDF) [14, 74], and 4) radiance fields (NeRF) [47]. More related methods can be found in surveys [24, 18]. Among them, the OF/SDF/NDF based methods demonstrate exceptional accuracy in recovering continuous 3D structures thanks to the simple level-set formulation, while the NeRF based methods achieve an unprecedented level of fidelity in rendering 2D views thanks to the successful volume rendering equation. However, all these coordinate-based representations inevitably require dense 3D location sampling and evaluations on trained networks to regress explicit surface points, though advanced techniques [41, 57, 69, 39, 30, 56, 50, 82, 48] can mitigate this issue somewhat but at the expense of a large memory increase. In this paper, our RayDF only needs a single network evaluation to regress every surface point, while still achieving on par or better reconstruction accuracy with existing OF/SDF/NDF methods.
Ray-based Implicit 3D Shape Representations: To overcome the inefficiency of coordinate-based 3D representations, a number of works start to formulate 3D shapes as ray-based neural functions via MLPs. Ray-based methods simply take individual rays as input and directly output either radiance values (RGB), i.e. light fields, or surface points, i.e. distance fields. LFN [64] and NeuLF [40] are among the early works of light fields to encode 3D representations from observed color images. Other light field works include [22, 61, 78, 52, 68, 75, 9, 1]. PRIF [23], DDF [3] and DRDF [36] are the early ray-based distance fields to learn 3D shapes from observed depth scans. Although these methods have shown very encouraging results for novel view rendering or surface reconstruction, they are usually limited to datasets with small baselines across multi-views. Basically, this is because the multi-view geometry consistency is not taken into account in their network designs. By contrast, in our RayDF pipeline, we explicitly introduce a dual-ray visibility classifier to aid our ray-surface distance fields to be multi-view shape consistent during training.
3 RayDF
Our pipeline consists of two networks and an optimization module. As shown in Figure 2, for the main network: ray-surface distance field  $f_{\Theta}$ , it takes a single oriented ray  $r$  as input, and directly regresses the distance  $d$  between ray starting point and its surface hitting point. For the input ray  $r$ , we opt for the conventional spherical parameterization [32], as it supports querying from  $360^{\circ}$  viewing angles. In particular, a fix-sized sphere is predefined with a relatively large diameter  $D$  as a convex hull in which the target 3D scene is bounded. Each surface point  $p$  of the target scene can be regarded as a directional ray penetrating the sphere with two intersection points on the sphere. Each intersection point is parameterized by two variables specifying the angles with regard to the sphere center. Then, for each ray, we have a 4D parameterization  $r = (\theta^{in}, \phi^{in}, \theta^{out}, \phi^{out})$ . Formally, the ray-surface distance field is defined below and implementation details are in Appendix A.1.
$$
d = f _ {\Theta} (\boldsymbol {r}), \quad \text {w h e r e} \boldsymbol {r} = \left(\theta^ {i n}, \phi^ {i n}, \theta^ {o u t}, \phi^ {o u t}\right), \quad \Theta \text {a r e t r a i n a b l e p a r a m e t e r s o f M L P s} \tag {1}
$$
For the auxiliary network: dual-ray visibility classifier  $h_{\Phi}$ , it takes a pair of rays as input and simply predicts their mutual visibility, aiming at explicitly modeling the mutual spatial relationships between any pairs of rays. This network, once well-trained, will play a key role in the third component: multi-view consistency optimization. Detailed designs are discussed in Sections 3.2&3.3.
3.2 Dual-ray Visibility Classifier
The main ray-surface network alone can indeed well fit many training ray-distance pairs, but there is no mechanism driving its output distances, i.e., surface geometry, to be consistent across multiple views, especially for unseen views. For example, as illustrated in the leftmost block of Figure 3, if
Figure 3: The leftmost block illustrates the mutual visibility of a pair of rays. The remaining block shows the symmetric design of our dual-ray visibility classifier  $h_{\Phi}$ .
0: mutual invisible
1: mutual visible
two input rays  $\mathbf{r}_1$  and  $\mathbf{r}_2$  are mutually visible in the scene, both rays must hit at the same surface point, then the corresponding ray-surface distances  $d_1$  and  $d_2$  must satisfy a transformation equation  $\mathbf{r}_1^{in} + d_1\mathbf{r}_1^d = \binom{x_1}{y_1} = \mathbf{r}_2^{in} + d_2\mathbf{r}_2^d$ , where  $\mathbf{r}^d = \frac{\mathbf{r}^{out} - \mathbf{r}^{in}}{\|\mathbf{r}^{out} - \mathbf{r}^{in}\|}$  and  $\mathbf{r}^* = \begin{pmatrix} \sin \theta^* \cos \phi^* \\ \sin \theta^* \sin \phi^* \\ \cos \phi^* \end{pmatrix}$ ,  $* \in \{in, out\}$  (More details in Section A.1.3), according to both rays' angles. Similarly, if input rays  $\mathbf{r}_1$  and  $\mathbf{r}_3$  are mutually invisible, then they should never satisfy the transformation equation because both never meet at the same surface point. From this fundamental principle, we can see that the mutual visibility between any two rays is crucial to inform the ray-surface distance network to be multi-view consistent.
To this end, we design a binary classifier to discriminate the visibility of an input pair of rays. A naive idea is to simply concat two rays as input, feeding into MLPs to regress  $0/1$ , where 1 represents visible and 0 otherwise. However, such a design fails to retain the symmetry of two rays. Here symmetry means that the visibility of two rays must be invariant to the input order of two rays. With this point, as illustrated in the right block of Figure 3, our visibility classifier  $h_{\Phi}$  is formally defined as:
$$
h _ {\Phi}: \quad M L P s \left[ \frac {g \left(\theta_ {1} ^ {i n} , \phi_ {1} ^ {i n} , \theta_ {1} ^ {o u t} , \phi_ {1} ^ {o u t}\right) + g \left(\theta_ {2} ^ {i n} , \phi_ {2} ^ {i n} , \theta_ {2} ^ {o u t} , \phi_ {2} ^ {o u t}\right)}{2} \oplus k \left(x _ {1}, y _ {1}, z _ {1}\right)\right]\rightarrow 0 / 1 \tag {2}
$$
where  $(\theta_{1}^{in},\phi_{1}^{in},\theta_{1}^{out},\phi_{1}^{out})$  and  $(\theta_{2}^{in},\phi_{2}^{in},\theta_{2}^{out},\phi_{2}^{out})$  are the parameterizations of two input rays  $r_1$  and  $r_2$  respectively;  $g()$  is a shared single fully-connected layer followed by an average pooling, thus guaranteeing the symmetry of input rays. Note that, the surface hitting point of  $r_1$ , i.e.,  $(x_{1},y_{1},z_{1})$ , is also encoded via a separate single fully-connected layer  $k()$ , followed by a concatenation operation denoted by  $\oplus$ , thus enhancing the pooled ray features, as we empirically find that such an enhancement could notably improve the classifier's accuracy. Implementation details are in Appendix A.1.
3.3 Multi-view Consistency Optimization
With the designed main ray-surface distance network  $f_{\Theta}$  and the auxiliary dual-ray visibility classifier  $h_{\Phi}$  at hand, we introduce the crucial multi-view consistency optimization module to train both networks. In particular, given  $K$  posed depth images  $(H \times W)$  of a static 3D scene as the whole training data, our training module consists of two stages.
Stage 1 - Training Dual-ray Visibility Classifier
The key to training this classifier is to create correct data pairs. First of all, all raw depth values are converted to ray-surface distance values. For a specific  $i^{th}$  ray (pixel) in the  $k^{th}$  image, we project its ray-surface point back to the remaining  $(K - 1)$  scans, obtaining the corresponding  $(K - 1)$  distance values. We set 10 millimeters as the closeness threshold to determine whether the projected  $(K - 1)$  rays are visible in the  $(K - 1)$  images. In total, we generate  $K * H * W * (K - 1)$  pairs of rays together with 0/1 labels. The standard cross-entropy loss function is adopted to optimize our dual-ray visibility classifier. More details on training data generation and implementation are in Appendix A.1.3.
Note that, this classifier is trained in a scene-specific fashion. Once the network is well-trained, it basically encodes the relationships between any two rays of a specific scene into network weights.
Stage 2 - Training Ray-surface Distance Network
The ultimate goal of our whole pipeline is to optimize the main ray-surface network and drive it to be multi-view geometry consistent. However, this is non-trivial as simply fitting the network with ray-surface data points cannot generalize to unseen rays, which can be seen in our ablation study in Section 4.5. In this regard, we fully leverage the well-trained visibility classifier to aid our training of ray-surface distance network. Particularly, this stage consists of the following key steps:
- Step 1: All depth images are converted to ray-surface distances, generating  $K * H * W$  training ray-distance pairs for a specific 3D scene.
- Step 2: As illustrated in Figure 4, for a specific training ray  $(\boldsymbol{r},d)$ , called primary ray, we uniformly sample  $M$  rays  $\{\boldsymbol{r}^1\dots \boldsymbol{r}^m\dots \boldsymbol{r}^M\}$ , called multi-view rays, in a ball centering at the surface point  $\pmb{p}$ . We then calculate the distance between surface point  $\pmb{p}$  and the bounding sphere along each of  $M$  rays, obtaining multi-view distances  $\{\tilde{d}^1\dots \tilde{d}^m\dots \tilde{d}^M\}$ . This can be easily achieved according to the given distance  $d$  in the training set.  $M$  is simply set as 20 and more details are in Appendix A.4.
- Step 3: We establish  $M$  pairs of rays  $\{(r, p, r^1) \cdots (r, p, r^m) \cdots (r, p, r^M)\}$  and then feed them into the well-trained visibility classifier  $h_\Phi$ , inferring their visibility scores  $\{\nu^1 \cdots \nu^m \cdots \nu^M\}$ .
- Step 4: We feed the primary ray and all sampled  $M$  multi-view rays  $\{\pmb{r},\pmb{r}^1\dots \pmb{r}^m\dots \pmb{r}^M\}$  into the ray-surface distance network  $f_{\Theta}$ , estimating their surface distances  $\{\hat{d},\hat{d}^1\dots \hat{d}^m\dots \hat{d}^M\}$ . Since the network  $f_{\Theta}$  is randomly initialized, thus the estimated distances are inaccurate in the beginning.
- Step 5: We design the following multi-view consistency loss function to optimize the ray-surface distance network until convergence:
$$
\ell_ {m v} = \frac {1}{\sum_ {m = 1} ^ {M} v ^ {m} + 1} \left(| \hat {d} - d | + \sum_ {m = 1} ^ {M} \left(| \hat {d} ^ {m} - \tilde {d} ^ {m} | * v ^ {m}\right)\right) \tag {3}
$$
Basically, this simple loss drives the network to not only fit the primary ray-surface distance (seen rays in the training set), but also satisfy that the visible multi-view rays (unlimited unseen rays in the training set) also have accurate distance estimations.
3.4 Surface Normal Derivation and Outlier Points Removal
In the above Sections 3.1&3.2&3.3, we have two network designs and an optimization module to train them separately. Nevertheless, we empirically find that the main ray-surface distance network may predict inaccurate distance values, particularly for rays near sharp edges. Essentially, this is because the actual
Figure 4: Multi-view ray sampling.
ray-surface distances may be discontinuous at sharp edges given extreme viewing angle changes. This shape discontinuity is actually a common challenge for almost all existing implicit neural representations, because modern neural networks can only model continuous functions in theory.
Fortunately, a nice property of our ray-surface distance field is that the normal vector at every estimated 3D surface point can be easily derived in a closed-form expression using auto differentiation of the network. In particular, given an input ray  $\pmb{r} = (\theta^{in},\phi^{in},\theta^{out},\phi^{out})$ , and its estimated ray-surface distance  $\hat{d}$  from network  $f_{\Theta}$ , the corresponding normal vector  $\pmb{n}$  of that estimated surface point can be derived as a specific function shown below.
$$
\boldsymbol {n} = Q \left(\frac {\partial \hat {d}}{\partial \boldsymbol {r}}, \boldsymbol {r}, D\right), \text {d e t a i l s o f t h e f u n c t i o n} Q \text {a n d d e r i v a t i o n a r e i n A p p e n d i x A . 2 .} \tag {4}
$$
With this normal vector, we may choose to add an additional loss to regularize the estimated surface points to be as smooth as possible. Yet, we empirically find that the overall performance improvement on an entire 3D scene is rather limited, as these extremely discontinuous cases are actually sparse.
In this regard, we turn to simply removing the predicted surface points, i.e., outliers, whose normal vectors' Euclidean distances are larger than a threshold in the network inference stage. In fact, PRIF [23] also adopts a similar strategy to filter out outliers. Note that, advanced smoothing or interpolating techniques may be integrated to improve our framework, which is left for future exploration.
Our method is evaluated on three types of public datasets: 1) the object-level synthetic Blender dataset from the original NeRF paper [47], 2) the scene-level synthetic DM-SR dataset from the recent DM-NeRF paper [73], and 3) the scene-level real-world ScanNet dataset [16].
Baselines: We carefully select the following six successful and representative implicit neural shape representations as our baselines: 1) OF [46], 2) DeepSDF [54], 3) NDF [14], 4) NeuS [77], 5) DS-NeRF [19], 6) LFN [64], and 7) PRIF [23]. The OF/DeepSDF/NDF/NeuS methods are coordinate-based level-set methods, showing outstanding performance in modeling 3D structures. DS-NeRF
is a depth-supervised NeRF [47], inheriting excellent capability in rendering 2D views. LFN and PRIF are two ray-based methods with superior efficiency in generating 2D views. We note that there are many sophisticated variants of these baselines, achieving SOTA performance on various datasets. Nevertheless, we do not intend to comprehensively compare with them, basically because many of their techniques such as more advanced implementations, adding additional conditions, replacing with more powerful backbones, etc., can be easily integrated into our framework as well. We leave these potential improvements for future exploration but only focus on our vanilla ray-surface distance field in this paper. For a fair comparison, all baselines are supervised with the same amount of depth scans as ours, carefully trained from scratch in the same scene-specific fashion. More details about the implementation of all baselines and possible minor adaptations are in Appendix A.3.1.
Metrics: For evaluation metrics of shape reconstruction, we report: 1) the per ray-surface absolute distance error (ADE) in centimeters averaged across all testing images, 2) the Chamfer distance (CD) [21] between the whole reconstructed structure and the ground truth shape for each scene. As to our method, we use the existing TSDF fusion [84] to obtain a full mesh from predicted depths at the testing views, on which we uniformly sample 30K points. Another 30K points are uniformly sampled from the ground truth 3D mesh for calculating CD. The outlier point removal is only applied to clean the reconstructed point clouds when calculating CD. No additional post-processing steps are employed when computing all ADE scores. Apparently, ADE is more accurate to evaluate the surface estimation of our method, because the CD scores may be biased due to the extra TSDF fusion. In this regard, the CD scores are just presented as a complementary metric to show the general shape quality in the whole 3D scene space. More details about how to obtain the reconstructed 3D full shape for each baseline and our method are in Appendix A.3.2. For evaluation metrics of appearance reconstruction, i.e. novel view synthesis of 2D color images, the standard PSNR, SSIM, and LPIPS scores are reported following NeRF [47]. We present Accuracy (\%) and F1-Score (\%) as evaluation metrics of the dual-ray visibility classifier. For qualitative results, refer to Figure 5 and Appendix A.7.
4.1 Efficiency of RayDF