SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yixun Liang\* Hao He1,2\* Ying-Cong Chen1,2
<sup>1</sup> The Hong Kong University of Science and Technology (Guangzhou).
2 The Hong Kong University of Science and Technology.
yliang982@connect.hkust-gz.edu.cn, hheat@connect.ust.hk, yingcongchen@ust.hk
Generalizable neural surface reconstruction techniques have attracted great attention in recent years. However, they encounter limitations of low confidence depth distribution and inaccurate surface reasoning due to the oversimplified volume rendering process employed. In this paper, we present Reconstruction TRansformer (ReTR), a novel framework that leverages the transformer architecture to redesign the rendering process, enabling complex render interaction modeling. It introduces a learnable meta-ray token and utilizes the cross-attention mechanism to simulate the interaction of rendering process with sampled points and render the observed color. Meanwhile, by operating within a high-dimensional feature space rather than the color space, ReTR mitigates sensitivity to projected colors in source views. Such improvements result in accurate surface assessment with high confidence. We demonstrate the effectiveness of our approach on various datasets, showcasing how our method outperforms the current state-of-the-art approaches in terms of reconstruction quality and generalization ability. Our code is available at https://github.com/YixunLiang/ReTR.
In the realm of computer vision and graphics, extracting geometric information from multi-view images poses a significant challenge with far-reaching implications for various fields, including robotics, augmented reality, and virtual reality. As a popular approach to this problem, neural implicit reconstruction techniques [1, 2, 3, 4, 5, 6] are frequently employed, generating accurate and plausible geometry from multi-view images by utilizing volume rendering and neural implicit representations based on the Sign Distance Function (SDF) [7] and its variant. Despite their efficacy, these methods possess inherent limitations such as the lack of cross-scene generalization capabilities and the necessity for extensive computational resources for training them from scratch for each scene. Furthermore, these techniques heavily rely on a large number of input views.
Recent studies such as SparseNeuS [9] and VolRecon [8] have attempted to overcome these challenges by integrating prior image information with volume rendering methods, thereby achieving impressive cross-scene generalization capabilities while only requiring sparse views as input. Nevertheless, these solutions are still based on volume rendering, which poses some intrinsic drawbacks in surface reconstruction. Specifically, volume rendering is a simplification of the physical world and might not capture the full extent of its complexity. It models the interactions of incident photons and particles into density, which is predicted solely based on sampled point features. This oversimplified modeling fails to disentangle the contribution of the light transport effect and surface properties to the observed color, resulting in an inaccurate assessment of the actual surface. Moreover, the prediction of color
(a) source views
(c) ReTR(ours)
Figure 1: Generalizable neural surface reconstructions from three input views in (a). VolRecon [8] produces depth distribution with low kurtosis and a noisy surface as shown in (b). In contrast, our proposed ReTR successfully extracts plausible surfaces with sharper depth distribution that has high kurtosis as shown in (c).
blending heavily relies on the projected color of the source views, thereby overlooking intricate physical effects. These shortcomings can lead to less confident surface predictions, producing a depth distribution with low kurtosis, and may be accompanied by high levels of noise, thereby compromising the overall quality of the reconstruction as illustrated in Fig. 1 (b).
In this paper, we first propose a more generalized formulation for volume rendering. Then based on this formulation, we introduce the Reconstruction TRansformer (ReTR), a novel approach for generalizable neural surface reconstruction. Our method utilizes the transformer to redesign the rendering process, allowing for accurate modeling of the complex render interaction while retaining the fundamental properties that make volume rendering effective. Particularly, we propose a learnable token, the meta-ray token, that encapsulates the complex light transport effect. By leveraging this token and the "cross-attention" mechanism, we simulate the interaction of rendering with the sampled points and aggregate the features of each point to render the observed color in an end-to-end manner. Additionally, we introduce a unidirectional transformer and continuous positional encoding to simulate photon-medium interaction, effectively considering occlusion and the interval of sample points. Moreover, as our model operates in the feature space rather than the color space, to further enhance the accuracy of semantic information and enable seamless adaptation, we propose a novel hybrid extractor designed to efficiently extract 3D-aware features.
Our method provides an excellent solution for generalizable neural surface reconstruction, offering several advantages over traditional volume rendering techniques. First, the ability to generalize complicated physical effects in a data-driven way provides an efficient approach to decoupling the contribution of light transport and surface to the observed color. Second, the entire process takes place in a high-dimensional feature space, rather than the color space, reducing the model's sensitivity to projected color in source views. Moreover, the transformer employs a re-weighted softmax function of the attention map, driving the learning of a depth distribution with positive kurtosis. As a result, our method achieves a more confident distribution, leading to reduced noise and improved quality, as shown in Fig. 1 (c).
In summary, our contribution can be summarized as:
- We identify the limitation and derive a general form of volume rendering. By leveraging this form, we can effectively tailor the rendering process for task-specific requirements.
- Through the derived general form, we propose ReTR, a learning-based rendering framework utilizing transformer architecture to model light transport. ReTR incorporates continuous positional encoding and leverages the hybrid feature extractor to enhance performance in generalizable neural surface reconstruction.
- Extensive experiments conducted on DTU, BlendedMVS, ETH3D, and Tanks & Temples datasets [10, 11, 12, 13] validate the efficacy and generalization ability of ReTR.
Multi-View Stereo (MVS). Multi-view stereo methods is another branch for 3D reconstruction, which can be broadly categorized into three main branches: depth maps-based [14, 15, 16, 17, 18, 19], voxel grids-based [20, 21, 22], and point clouds-based [23, 24, 25]. Among these, depth maps-based methods are more flexible and hence more popular than the others. Depth maps-based methods typically decouple the problem into depth estimation and fusion, which has shown impressive performance with densely captured images. However, these methods exhibit limited robustness in situations with a shortage of images, thereby highlighting the problem we aim to address.
Neural Surface Reconstruction. With the advent of NeRF [26], there has been a paradigm shift towards using similar techniques for shape modeling, novel view synthesis, and multi-view 3D reconstruction. IDR [27] uses surface rendering to learn geometry from multi-view images, but it requires extra object masks. Several methods [2, 3, 1, 28, 29] have attempted to rewrite the density function in NeRF using SDF and its variants, successfully regressed plausible geometry. Among them, NeuS [2] first uses SDF value to model density in the volume rendering to learn neural implicit surface, offering a robust method for multi-view 3D reconstruction from 2D images. However, these methods require lengthy optimization to train each scene independently. Inspired by the recent success of generalizable novel view synthesis [30, 31, 32, 33, 34]. SparseNeuS [9] and VolRecon [8] achieve generalizable neural surface reconstruction using the information from the source images as the prior to neural surface reconstruction. However, these methods suffer from oversimplified modeling of light transport in traditional volume rendering and color blending, resulting in extracted geometries that fail to make a confident prediction of the surface, leading to distortion in the reconstructions. However, there are also some attempts [35, 36, 37] that adopt other rendering methods in implicit neural representation, such as ray tracing. However, these models are still based on enhanced explicit formulations; thus, their capacity to model real-world interaction is still limited. In contrast, our method introduces a learning-based rendering, providing an efficient way to overcome such limitations.
Learning Based Rendering. Unlike volume rendering, another line of work [38, 39, 40] explores deep learning techniques to simulate the rendering process. Especially Recurrent neural networks, which naturally fit the rendering process. Specifically, DeepVoxels [38] employs GRU to process voxel features along a ray, and its successor SRN [41], leverages LSTM for ray-marching. However, such methods recursively process features and demand large computation resources. The computation constraints inhibit such methods' ability to produce high-quality renderings. Unlike RNN-based methods, ReTR leverages the transformer to parallel compute each point's hitting probability. Greatly improve the efficiency of the rendering process and achieve high-quality renderings.
Transformers With Radiance Field. The attention mechanism in transformers [42] has also been widely used in the area of radiance field. In image-based rendering, IBRNet [30] proposes a ray transformer to process sampled point features and predict density. NeRFormer [43] utilizes a transformer to aggregate source views and construct feature volumes. NeuRays [44] leverages neural networks to model and address occlusions, enhancing the quality and accuracy of image-based rendering. GPBR [45] employs neural networks to transform and composite patches from source images, enabling versatile and realistic image synthesis across various scenes. However, these methods only use the transformer to enhance feature aggregation, and the sampled point features are still decoded into colors and densities and aggregated using traditional volume rendering, leading to unconfident surface prediction. Recently, GNT [46] naively replaces classical volume rendering with transformers in image-based rendering, which overlooks the absence of occlusion and positional awareness within the transformer architecture. In contrast to GNT, we improved the traditional transformer architecture in those two limitations based on an in-depth analysis of the fundamental components to make volume rendering work.
In this section, we present an analysis of the limitations of existing generalizable neural surface reconstruction approaches that adopt volume rendering from NeRF [26] and propose ReTR, a novel architecture that leverages transformer to achieve learning-based rendering. We introduce the formulations of volume rendering and revisit its limitations in generalizable neural surface reconstruction in Sec 3.1. We then depict the general form of volume rendering in Sec. 3 and present
Figure 2: Our ReTR pipeline comprises several steps: (1). Extracting features through the proposed hybrid extractor model from source views, (2). Processing features in each sample point using the feature fusion block, and (3). Using the occlusion transformer and render transformer to aggregate features along the ray and predict colors and depths.
our learning-based rendering in Sec. 3.3. To effectively extract features for our proposed rendering, we further introduce the hybrid extractor in Sec. 3.4. Our loss functions are explained in Sec. 3.5.
Generalizable neural surface reconstruction aims to recover the geometry of a scene  $S$ , which is represented by a set of  $M$  posed input views  $S = \{I_j, P_j\}_{j=1}^M$ , where  $I_j \in \mathbb{R}^{H \times W \times 3}$  and  $P_j \in \mathbb{R}^{3 \times 4}$  are the  $j$ -th view's image and camera parameters, respectively. Existing approaches [9, 8] generate the features of the radiance field from  $S$  using a neural network  $\mathcal{F}_{enc.}$ , and we formalize the process as:
$$
\mathbf {f} ^ {v}, \left\{\mathbf {f} _ {1} ^ {i m g}, \dots , \mathbf {f} _ {M} ^ {i m g} \right\} = \mathcal {F} _ {\text {e n c .}} (\boldsymbol {S}), \tag {1}
$$
where  $\mathbf{f}^v\in \mathbb{R}^{R\times R\times R\times D}$  is the volume feature with resolution  $R$  and  $\mathbf{f}_j^{img}\in \mathbb{R}^{h\times w\times D}$  is the image feature with dimension  $D$ . To decode the color of a sampled ray  $\mathbf{r} = (\mathbf{o},\mathbf{d})$  passing through the scene, where  $\mathbf{o}$  and  $\mathbf{d}$  denotes as ray original and direction, the existing approaches [9, 8] sample  $N$  points along the ray from coarse to fine sampling between near and far planes and obtain the location  $\pmb{x}\in \mathbb{R}^3$  of each sample point:
$$
\boldsymbol {x} _ {i} = \mathbf {o} + t _ {i} \mathbf {d}, \quad i = 1, \dots , N. \tag {2}
$$
The predicted SDF from features first converts to weights using the conversion function [2], denoted as  $\sigma_{i}$ . The weights are then used to accumulate the re-weighted projected colors along the ray using volume rendering. Specifically:
$$
C (\mathbf {r}) = \sum_ {i = 1} ^ {N} T _ {i} (1 - \exp (- \sigma_ {i})) \mathbf {c} _ {i}, \quad \text {w h e r e} \quad T _ {i} = \exp \left(- \sum_ {j = 1} ^ {i - 1} \sigma_ {j}\right), \tag {3}
$$
$$
\mathbf {c} _ {i} = \sum_ {j = 1} ^ {M} \mathcal {F} _ {\text {w e i g h t .}} \left(\mathbf {f} _ {i} ^ {v}, \left\{\Pi \left(\mathbf {f} _ {k} ^ {\text {i m g}}, \boldsymbol {x} _ {i}\right) \right\} _ {k = 1} ^ {M}\right) \Pi \left(\boldsymbol {I} _ {j}, \boldsymbol {x} _ {i}\right). \tag {4}
$$
Here the  $\mathcal{F}_{weight}(\cdot)$  denotes the module that predicts the weight of each projected color. The  $\mathbf{f}_i^v$  represents the volume feature at  $\pmb{x}_i$  obtained using trilinear interpolation in  $\mathbf{f}^v$ .  $\Pi(I, \pmb{x})$  denotes the operation that projects  $\pmb{x}$  onto the corresponding input grid  $\pmb{I}$  and then extracts the feature at the projected location through bilinear interpolation.
Limitation. The volume rendering, denoted by Eq. (3), greatly simplifies light transport processes, thus introducing key limitations. A complete physical model for light transport categorizes the process into absorption, emission, out-scattering, and in-scattering. Each represents a nonlinear photon-particle interaction, with incident photon properties being influenced by both their inherent nature and medium characteristics. However, Eq. (3) condenses these complexities into a single density value, predicted merely based on sampled point features, leading to an oversimplification
of incident photon modeling. Moreover, the color determination method, influenced by a weighted blend of projected colors akin to Eq. (4), over-relies on input view projected colors, overlooking intricate physical effects. As a result, the model requires a wider accumulation of projected colors from points near the exact surface, resulting in a "murky" surface appearance, as depicted in Fig. 1.
3.2 Generalized Rendering Function
To overcome the limitations we mentioned in Sec. 3.1, we propose an improved rendering equation that takes into consideration of incident photon modeling and feature-based color decoding. As we revisit Eq. (3) and determine its key components to facilitate our redesign. This differentiable equation consists of three parts: The  $T_{i}$  term accounts for the accumulated transmittance and gives the first surface a bias to contribute more to the observed color. The  $(1 - \exp (-\sigma_i))$  term denotes the alpha value of traditional alpha compositing, which represents the transparency of  $x_{i}$  and is constrained to be non-negative. The color part of Eq. (3) denotes the color of  $x_{i}$ . Based on these analyses, we summarize three key rendering properties that the system must hold:
1. Differentiable. To enable effective learning, the weight function needs to be differentiable to the training network with observed color through back-propagation.
2. Occlusion-aware. In line with the bias towards the first surface in the original equation, the weight function needs to be aware that the points close to the first exact surface should have a larger contribution to the final output color than other points.
3. Non-negative. Echoing the non-negativity constraint in the alpha value, the weight of each point also needs to be positive.
Having identified these key properties, we can now reformulate our approach to meet these requirements. Specifically, we propose a more general form of Eq. (3), which can be formalized as:
$$
C (\mathbf {r}) = \sum_ {i = 1} ^ {N} \mathcal {W} \left(\boldsymbol {F} _ {1}, \dots , \boldsymbol {F} _ {i}\right) \mathcal {C} \left(\boldsymbol {F} _ {i}\right), \tag {5}
$$
where  $\pmb{F}_i$  represents the set comprising image feature  $f^{img}$  and volume feature  $f^v$  in the  $\pmb{x}_i \in \mathbb{R}^3$ ,  $\mathcal{W}(\cdot)$  is the weight function that satisfies those three key properties we mentioned above, and and  $\mathcal{C}(\cdot)$  denotes the color function. Specifically, color  $c$  can then be interpreted as characteristic of each feature point in 5. Therefore, feature at each point can be aggregated in a manner analogous to RGB value, and enabling us to deduce the primary feature points. This can be mathematically expressed as:
$$
C (\mathbf {r}) = C \left(\sum_ {i = 1} ^ {N} \mathcal {W} \left(\boldsymbol {F} _ {1}, \dots , \boldsymbol {F} _ {i}\right) \boldsymbol {F} _ {i}\right), \tag {6}
$$
where the  $C(\cdot)$  represents the color function that maps the feature into RGB space.
3.3 Reconstruction Transformer
Note that Eq. (3) can be considered as a special form of Eq. (5). With this generalized form, we can reformulate the rendering function to overcome the oversimplification weakness of Eq. (3). Based on Eq. (5), we introduce the Reconstruction Transformer (ReTR) that preserves essential rendering properties while incorporating sufficient complexity to implicitly learn and model intricate physical processes. ReTR is composed of a Render Transformer and an Occlusion Transformer. The Render Transformer leverages a learnable "meta-ray token" to encapsulate complex render properties, enhancing surface modeling. The Occlusion Transformer utilizes an attention mask to enable the occlusion-aware property. Also, ReTR works in the high-dimensional feature space instead of the color space, and thus allows for more complex and physically accurate light interactions. Consequently, ReTR not only overcomes the restrictions of Eq. (3) but also preserves its fundamental rendering characteristics. We elaborate on the design of these components as follows.
Render Transformer. Here, we discuss the design of the render transformer. Specifically, we introduce a global learnable token, refer to as "meta-ray token" and denote as  $\mathbf{f}^{tok} \in \mathbb{R}^D$ , to capture and store the complex render properties. For each sample ray, we first use the FeatureFusion block to combine all features associated with each sample point of the ray, resulting in  $\mathbf{f}_i^f =$
FeatureFusion  $(\pmb{F}_i) \in \mathbb{R}^D$ . We then employ the cross-attention mechanism within the Render Transformer to simulate the interaction of sample points along the ray. It can be formalized as follows:
$$
C (\mathbf {r}) = \mathcal {C} \left(\sum_ {i = 1} ^ {N} \operatorname {s o f t m a x} \left(\frac {q \left(\mathbf {f} ^ {\text {t o k}}\right) k \left(\mathbf {f} _ {i} ^ {f}\right) ^ {\top}}{\sqrt {D}}\right) v \left(\mathbf {f} _ {i} ^ {f}\right)\right), \tag {7}
$$
where  $q(\cdot), k(\cdot), v(\cdot)$  denotes three linear layers and  $\mathcal{C}(\cdot)$  in this formulation is an MLP structure to directly regress observed color from the aggregated feature, and  $W(\cdot)$  translates to softmax  $\left( \frac{q(\mathbf{f}^{tok}) k(\mathbf{f}_i^f)^{\top}}{\sqrt{D}} \right)$ . Furthermore,  $C(\cdot)$  is operationalized as MLP, which serves to decode the integrated feature into its corresponding RGB value. Then, the hitting probability will be normalized by softmax function, which is Non-negative and encourages the network to learn a weight distribution with positive kurtosis. And we can extract the attention map from the Render Transformer and derive the rendered depth as:
$$
D (\mathbf {r}) = \sum_ {i = 1} ^ {N} \alpha_ {i} t _ {i}, \quad \text {w h e r e} \quad \alpha_ {i} = \operatorname {s o f t m a x} \left(\frac {q \left(\mathbf {f} ^ {\text {t o k}}\right) k \left(\mathbf {f} _ {i} ^ {f}\right) ^ {\top}}{\sqrt {D}}\right), \tag {8}
$$
where  $\alpha_{1},\ldots ,\alpha_{N}$  denotes the attention map extracted from the Eq. (7). The rendered depth map can be further used to generate mesh [47] and point cloud [16].
Occlusion Transformer. To further make our system Occlusion-aware and enable of simulation of photon-medium interaction. We introduce Occlusion Transformer. Similar to previous works [46, 48], we introduce an attention mask to achieve that the sample point interacts only with the points in front of it and the meta-ray token. Such unidirectional processes encourage the later points to respond to the preceding surface. This process can be formalized as:
$$
\mathbf {R} ^ {f} = \left\{\mathbf {f} ^ {t o k}, \mathbf {f} _ {1} ^ {f}, \mathbf {f} _ {2} ^ {f}, \dots , \mathbf {f} _ {N} ^ {f} \right\}, \tag {9}
$$
$$
\begin{array}{l} \mathbf {R} ^ {\text {o c c}} = \operatorname {O c c T r a n s} (Q, K, V = \mathbf {R} _ {f}), \\ \mathbf {P} (\mathbf {M H A} (Q, \boldsymbol {s} ^ {f}, K, V, \{\boldsymbol {s} ^ {\text {t o k}}, \boldsymbol {s} ^ {f}, \quad \boldsymbol {s} ^ {f}) \}) + \boldsymbol {s} ^ {f}). \end{array} \tag {10}
$$
$$
\text {w h e r e} \quad \mathbf {f} _ {i} ^ {\text {o c c}} = \operatorname {M L P} \left(\operatorname {M H A} \left(Q = \mathbf {f} _ {i} ^ {f}, K, V = \left\{\mathbf {f} ^ {\text {t o k}}, \mathbf {f} _ {1} ^ {f}, \dots , \mathbf {f} _ {i} ^ {f} \right\}\right) + \mathbf {f} _ {i} ^ {f}\right). \tag {10}
$$
Where MHA denotes the multi-head self-attention operation [42] and  $\mathbf{f}_i^{occ}$  denotes the refine feature of  $\pmb{x}_i$  which obtained from the render transformer. In addition,  $R^f$  is the collective set of tokens, and  $R^{occ}$  signifies the occlusion transformer that employs  $R^f$  for cross attention. Then, our Eq. (7) can be rewritten as:
$$
C (\mathbf {r}) = \mathcal {C} \left(\sum_ {i = 1} ^ {N} \operatorname {s o f t m a x} \left(\frac {q \left(\mathbf {f} ^ {\text {t o k}}\right) k \left(\mathbf {f} _ {i} ^ {\text {o c c}}\right) ^ {\top}}{\sqrt {D}}\right) v \left(\mathbf {f} _ {i} ^ {f}\right)\right). \tag {11}
$$
Continuous Positional Encoding. Following the traditional transformer design, we need to introduce a positional encoding to make the whole structure positional-aware. However, positional encoding proposed in [42] ignores the actual distance between each token, which is unsuitable for our situation. Furthermore, weighted-based resampling [26] would lead to misalignment of positional encoding when an increased number of sample points are used.
To solve this problem, we extend the traditional positional encoding formula to continuous scenarios. Specifically, it can be formulated as follows:
$$
P E _ {\left(\boldsymbol {x} _ {i}, 2 i\right)} = \sin \left(\beta t _ {i} / 1 0 0 0 0 ^ {2 i / D}\right), \tag {12}
$$
$$
P E _ {(\boldsymbol {x} _ {i}, 2 i + 1)} = \cos (\beta t _ {i} / 1 0 0 0 0 ^ {2 i / D}).
$$