Title: ReTR: Modeling Rendering Via Transformer for Generalizable Neural Surface Reconstruction
Venue: NeurIPS 2023
Accepted: yes
Keywords: 3D vision

3D reconstruction

Generalizable Neural Surface Reconstruction
Abstract: Generalizable neural surface reconstruction techniques have attracted great attention in recent years. However, they encounter limitations of low confidence depth distribution and inaccurate surface reasoning due to the oversimplified volume rendering process employed. In this paper, we present Reconstruction TRansformer (ReTR), a novel framework that leverages the transformer architecture to redesign the rendering process, enabling complex render interaction modeling. It introduces a learnable $\textit{meta-ray token}$ and utilizes the cross-attention mechanism to simulate the interaction of rendering process with sampled points and render the observed color. Meanwhile, by operating within a high-dimensional feature space rather than the color space, ReTR mitigates sensitivity to projected colors in source views. Such improvements result in accurate surface assessment with high confidence. We demonstrate the effectiveness of our approach on various datasets, showcasing how our method outperforms the current state-of-the-art approaches in terms of reconstruction quality and generalization ability. $\textit{Our code is available at }$ https://github.com/YixunLiang/ReTR.

ReTR: Modeling Rendering Via Transformer for Generalizable Neural Surface Reconstruction

Abstract

Generalizable neural surface reconstruction techniques have attracted great attention in recent years. However, they encounter limitations of low confidence depth distribution and inaccurate surface reasoning due to the oversimplified volume rendering process employed. In this paper, we present Reconstruction TRansformer (ReTR), a novel framework that leverages the transformer architecture to redesign the rendering process, enabling complex render interaction modeling. It introduces a learnable meta-ray token and utilizes the cross-attention mechanism to simulate the interaction of rendering process with sampled points and render the observed color. Meanwhile, by operating within a high-dimensional feature space rather than the color space, ReTR mitigates sensitivity to projected colors in source views. Such improvements result in accurate surface assessment with high confidence. We demonstrate the effectiveness of our approach on various datasets, showcasing how our method outperforms the current state-of-the-art approaches in terms of reconstruction quality and generalization ability. Our code is available at https://github.com/YixunLiang/ReTR .

1 Introduction

In the realm of computer vision and graphics, extracting geometric information from multi-view images poses a significant challenge with far-reaching implications for various fields, including robotics, augmented reality, and virtual reality. As a popular approach to this problem, neural implicit reconstruction techniques [1, 2, 3, 4, 5, 6] are frequently employed, generating accurate and plausible geometry from multi-view images by utilizing volume rendering and neural implicit representations based on the Sign Distance Function (SDF) [7] and its variant. Despite their efficacy, these methods possess inherent limitations such as the lack of cross-scene generalization capabilities and the necessity for extensive computational resources for training them from scratch for each scene. Furthermore, these techniques heavily rely on a large number of input views.

Recent studies such as SparseNeuS [9] and VolRecon [8] have attempted to overcome these challenges by integrating prior image information with volume rendering methods, thereby achieving impressive cross-scene generalization capabilities while only requiring sparse views as input. Nevertheless, these solutions are still based on volume rendering, which poses some intrinsic drawbacks in surface reconstruction. Specifically, volume rendering is a simplification of the physical world and might not capture the full extent of its complexity. It models the interactions of incident photons and particles into density, which is predicted solely based on sampled point features. This oversimplified modeling fails to disentangle the contribution of the light transport effect and surface properties to the observed color, resulting in an inaccurate assessment of the actual surface. Moreover, the prediction of color

(b) VolRecon

Weight

Ray distance

Weight

Ray distance

0.14

0.12

0.10

0.08

0.06

0.04

0.02

0.00

2.10

2.15

2.20

2.35

2.40

0.14

0.12

0.10

0.08

0.06

0.04

0.02

0.00

2.10

2.15

2.20

2.25

2.30

SE'Z

(a) source views

(c)

ReTR(ours)

blending heavily relies on the projected color of the source views, thereby overlooking intricate physical effects. These shortcomings can lead to less confident surface predictions, producing a depth distribution with low kurtosis, and may be accompanied by high levels of noise, thereby compromising the overall quality of the reconstruction as illustrated in Fig. 1 (b).

In this paper, we first propose a more generalized formulation for volume rendering. Then based on this formulation, we introduce the Reconstruction TRansformer (ReTR), a novel approach for generalizable neural surface reconstruction. Our method utilizes the transformer to redesign the rendering process, allowing for accurate modeling of the complex render interaction while retaining the fundamental properties that make volume rendering effective. Particularly, we propose a learnable token, the meta-ray token , that encapsulates the complex light transport effect. By leveraging this token and the "cross-attention" mechanism, we simulate the interaction of rendering with the sampled points and aggregate the features of each point to render the observed color in an end-to-end manner. Additionally, we introduce a unidirectional transformer and continuous positional encoding to simulate photon-medium interaction, effectively considering occlusion and the interval of sample points. Moreover, as our model operates in the feature space rather than the color space, to further enhance the accuracy of semantic information and enable seamless adaptation, we propose a novel hybrid extractor designed to efficiently extract 3D-aware features.

Our method provides an excellent solution for generalizable neural surface reconstruction, offering several advantages over traditional volume rendering techniques. First, the ability to generalize complicated physical effects in a data-driven way provides an efficient approach to decoupling the contribution of light transport and surface to the observed color. Second, the entire process takes place in a high-dimensional feature space, rather than the color space, reducing the model's sensitivity to projected color in source views. Moreover, the transformer employs a re-weighted softmax function of the attention map, driving the learning of a depth distribution with positive kurtosis. As a result, our method achieves a more confident distribution, leading to reduced noise and improved quality, As shown in Fig. 1 (c).

In summary, our contribution can be summarized as:

We identify the limitation and derive a general form of volume rendering. By leveraging this form, we can effectively tailor the rendering process for task-specific requirements.

Through the derived general form, we propose ReTR, a learning-based rendering framework utilizing transformer architecture to model light transport. ReTR incorporates continuous positional encoding and leverages the hybrid feature extractor to enhance performance in generalizable neural surface reconstruction.

Extensive experiments conducted on DTU, BlendedMVS, ETH3D, and Tanks & Temples datasets [10, 11, 12, 13] validate the efficacy and generalization ability of ReTR.

2 Related Works

Multi-View Stereo (MVS). Multi-view stereo methods is another branch for 3D reconstruction, which can be broadly categorized into three main branches: depth maps-based [14, 15, 16, 17, 18, 19], voxel grids-based [20, 21, 22], and point clouds-based [23, 24, 25]. Among these, depth maps-based methods are more flexible and hence more popular than the others. Depth maps-based methods typically decouple the problem into depth estimation and fusion, which has shown impressive performance with densely captured images. However, these methods exhibit limited robustness in situations with a shortage of images, thereby highlighting the problem we aim to address.

Neural Surface Reconstruction. With the advent of NeRF [26], there has been a paradigm shift towards using similar techniques for shape modeling, novel view synthesis, and multi-view 3D reconstruction. IDR [27] uses surface rendering to learn geometry from multi-view images, but it requires extra object masks. Several methods [2, 3, 1, 28, 29] have attempted to rewrite the density function in NeRF using SDF and its variants, successfully regressed plausible geometry. Among them, NeuS [2] first uses SDF value to model density in the volume rendering to learn neural implicit surface, offering a robust method for multi-view 3D reconstruction from 2D images. However, these methods require lengthy optimization to train each scene independently. Inspired by the recent success of generalizable novel view synthesis [30, 31, 32, 33, 34]. SparseNeuS [9] and VolRecon [8] achieve generalizable neural surface reconstruction using the information from the source images as the prior to neural surface reconstruction. However, these methods suffer from oversimplified modeling of light transport in traditional volume rendering and color blending, resulting in extracted geometries that fail to make a confident prediction of the surface, leading to distortion in the reconstructions. However, there are also some attempts [35, 36, 37] that adopt other rendering methods in implicit neural representation, such as ray tracing. However, these models are still based on enhanced explicit formulations; thus, their capacity to model real-world interaction is still limited. In contrast, our method introduces a learning-based rendering, providing an efficient way to overcome such limitations.

Learning Based Rendering. Unlike volume rendering, another line of work [38, 39, 40] explores deep learning techniques to simulate the rendering process. Especially Recurrent neural networks, which naturally fit the rendering process. Specifically, DeepVoxels [38] employs GRU to process voxel features along a ray, and its successor SRN [41], leverages LSTM for ray-marching. However, such methods recursively process features and demand large computation resources. The computation constraints inhibit such methods' ability to produce high-quality renderings. Unlike RNN-based methods, ReTR leverages the transformer to parallel compute each point's hitting probability. Greatly improve the efficiency of the rendering process and achieve high-quality renderings.

Transformers With Radiance Field. The attention mechanism in transformers [42] has also been widely used in the area of radiance field. In image-based rendering, IBRNet [30] proposes a ray transformer to process sampled point features and predict density. NeRFormer [43] utilizes a transformer to aggregate source views and construct feature volumes. NeuRays [44] leverages neural networks to model and address occlusions, enhancing the quality and accuracy of image-based rendering. GPBR [45] employs neural networks to transform and composite patches from source images, enabling versatile and realistic image synthesis across various scenes. However, these methods only use the transformer to enhance feature aggregation, and the sampled point features are still decoded into colors and densities and aggregated using traditional volume rendering, leading to unconfident surface prediction. Recently, GNT [46] naively replaces classical volume rendering with transformers in image-based rendering, which overlooks the absence of occlusion and positional awareness within the transformer architecture. In contrast to GNT, we improved the traditional transformer architecture in those two limitations based on an in-depth analysis of the fundamental components to make volume rendering work.

3 Methodology

In this section, we present an analysis of the limitations of existing generalizable neural surface reconstruction approaches that adopt volume rendering from NeRF [26] and propose ReTR, a novel architecture that leverages transformer to achieve learning-based rendering. We introduce the formulations of volume rendering and revisit its limitations in generalizable neural surface reconstruction in Sec 3.1. We then depict the general form of volume rendering in Sec. 3 and present

Hybrid

Extractor

R

Ray distance

Weights

Attn. map

𝛴

2

𝐷௣௥௘ௗ.

𝐷௚௧

Rendering Loss

Depth Loss

MLP

Feature

Fusion

Feature

Fusion

Feature

Fusion

source views

C

X

: Concatenate

: Multiplication

R

Trans.

C

: relative directions

MLP

𝑤ଵ

𝑤ଶ

𝑤ଷ

X

𝛴

C

: volume feature

: projected features

: fused features

Occlusion

Transformer

Render

Transformer

UUU

our learning-based rendering in Sec. 3.3. To effectively extract features for our proposed rendering, we further introduce the hybrid extractor in Sec. 3.4. Our loss functions are explained in Sec. 3.5.

3.1 Preliminary

Generalizable neural surface reconstruction aims to recover the geometry of a scene S , which is represented by a set of M posed input views S = { I j , P j } j M =1 , where I j ∈ R H × W × 3 and P j ∈ R 3 × 4 are the j -th view's image and camera parameters, respectively. Existing approaches [9, 8] generate the features of the radiance field from S using a neural network F enc. , and we formalize the process as:

where f v ∈ R R × R × R × D is the volume feature with resolution R and f img j ∈ R h × w × D is the image feature with dimension D . To decode the color of a sampled ray r = ( o , d ) passing through the scene, where o and d denotes as ray original and direction, the existing approaches [9, 8] sample N points along the ray from coarse to fine sampling between near and far planes and obtain the location x ∈ R 3 of each sample point:

The predicted SDF from features first converts to weights using the conversion function [2], denoted as σ i . The weights are then used to accumulate the re-weighted projected colors along the ray using volume rendering. Specifically:

Here the F weight. ( · ) denotes the module that predicts the weight of each projected color. The f v i represents the volume feature at x i obtained using trilinear interpolation in f v . Π( I , x ) denotes the operation that projects x onto the corresponding input grid I and then extracts the feature at the projected location through bilinear interpolation.

Limitation. The volume rendering, denoted by Eq. (3), greatly simplifies light transport processes, thus introducing key limitations. A complete physical model for light transport categorizes the process into absorption, emission, out-scattering, and in-scattering. Each represents a nonlinear photon-particle interaction, with incident photon properties being influenced by both their inherent nature and medium characteristics. However, Eq. (3) condenses these complexities into a single density value, predicted merely based on sampled point features, leading to an oversimplification

of incident photon modeling . Moreover, the color determination method, influenced by a weighted blend of projected colors akin to Eq. (4), over-relies on input view projected colors, overlooking intricate physical effect s. As a result, the model requires a wider accumulation of projected colors from points near the exact surface, resulting in a "murky" surface appearance, as depicted in Fig. 1.

3.2 Generalized Rendering Function

To overcome the limitations we mentioned in Sec. 3.1, we propose an improved rendering equation that takes into consideration of incident photon modeling and feature-based color decoding. As we revisit Eq. (3) and determine its key components to facilitate our redesign. This differentiable equation consists of three parts: The T i term accounts for the accumulated transmittance and gives the first surface a bias to contribute more to the observed color. The (1 -exp( -σ i )) term denotes the alpha value of traditional alpha compositing, which represents the transparency of x i and is constrained to be non-negative. The color part of Eq. (3) denotes the color of x i . Based on these analyses, we summarize three key rendering properties that the system must hold:

Differentiable . To enable effective learning, the weight function needs to be differentiable to the training network with observed color through back-propagation.

Occlusion-aware . In line with the bias towards the first surface in the original equation, the weight function needs to be aware that the points close to the first exact surface should have a larger contribution to the final output color than other points.

Non-negative . Echoing the non-negativity constraint in the alpha value, the weight of each point also needs to be positive.

Having identified these key properties, we can now reformulate our approach to meet these requirements. Specifically, we propose a more general form of Eq. (3), which can be formalized as:

where F i represents the set comprising image feature f img and volume feature f v in the x i ∈ R 3 , W ( · ) is the weight function that satisfies those three key properties we mentioned above, and and C ( · ) denotes the color function. Specifically, color c can then be interpreted as characteristic of each feature point in 5. Therefore, feature at each point can be aggregated in a manner analogous to RGB value, and enabling us to deduce the primary feature points. This can be mathematically expressed as:

where the C ( · ) represents the color function that maps the feature into RGB space.

3.3 Reconstruction Transformer

Note that Eq. (3) can be considered as a special form of Eq. (5). With this generalized form, we can reformulate the rendering function to overcome the oversimplification weakness of Eq. (3). Based on Eq. (5), we introduce the Reconstruction Transformer (ReTR) that preserves essential rendering properties while incorporating sufficient complexity to implicitly learn and model intricate physical processes. ReTR is composed of a Render Transformer and an Occlusion Transformer. The Render Transformer leverages a learnable "meta-ray token" to encapsulate complex render properties, enhancing surface modeling. The Occlusion Transformer utilizes an attention mask to enable the occlusion-aware property. Also, ReTR works in the high-dimensional feature space instead of the color space, and thus allows for more complex and physically accurate light interactions. Consequently, ReTR not only overcomes the restrictions of Eq. (3) but also preserves its fundamental rendering characteristics. We elaborate on the design of these components as follows.

Render Transformer. Here, we discuss the design of the render transformer. Specifically, we introduce a global learnable token, refer to as "meta-ray token" and denote as f tok ∈ R D , to capture and store the complex render properties. For each sample ray, we first use the FeatureFusion block to combine all features associated with each sample point of the ray, resulting in f f i =

FeatureFusion ( F i ) ∈ R D . We then employ the cross-attention mechanism within the Render Transformer to simulate the interaction of sample points along the ray. It can be formalized as follows:

where q ( · ) , k ( · ) , v ( · ) denotes three linear layers and C ( · ) in this formulation is an MLP structure to directly regress observed color from the aggregated feature, and W ( · ) translates to softmax ( q ( f tok ) k ( f f i ) ⊤ √ D ) . Furthermore, C ( · ) is operationalized as MLP, which serves to decode the integrated feature into its corresponding RGB value. Then, the hitting probability will be normalized by softmax function, which is Non-negative and encourages the network to learn a weight distribution with positive kurtosis. And we can extract the attention map from the Render Transformer and derive the rendered depth as:

where α 1 , . . . , α N denotes the attention map extracted from the Eq. (7). The rendered depth map can be further used to generate mesh [47] and point cloud [16].

Occlusion Transformer. To further make our system Occlusion-aware and enable of simulation of photon-medium interaction. We introduce Occlusion Transformer. Similar to previous works [46, 48], we introduce an attention mask to achieve that the sample point interacts only with the points in front of it and the meta-ray token. Such unidirectional processes encourage the later points to respond to the preceding surface. This process can be formalized as:

Where MHA denotes the multi-head self-attention operation [42] and f occ i denotes the refine feature of x i which obtained from the render transformer. In addition, R f is the collective set of tokens, and R occ signifies the occlusion transformer that employs R f for cross attention. Then, our Eq. (7) can be rewritten as:

Continuous Positional Encoding. Following the traditional transformer design, we need to introduce a positional encoding to make the whole structure positional-aware. However, positional encoding proposed in [42] ignores the actual distance between each token, which is unsuitable for our situation. Furthermore, weighted-based resampling [26] would lead to misalignment of positional encoding when an increased number of sample points are used.

To solve this problem, we extend the traditional positional encoding formula to continuous scenarios. Specifically, it can be formulated as follows:

Here, i represents the positional encoding in the i th dimension and β is a scale hyperparameter we empirically set to 100. The updated formula successfully solves the misalignment of traditional positional encoding, results are shown in Tab. 4. The specific proofs will be included in the Appendix section.

3.4 Hybrid Extractor

In our learning-based rendering, a finer level of visual feature is necessary, which is not achieved by traditional feature extractors that rely on high-level features obtained through FPN. These features

source views

CNN+FPN

feature maps

proj. feature maps

project

global feature volume

raw feature volume

3D-Unet

source views

CNN

Multi-level

feature maps

Multi-level

proj. feature maps

project

global feature volume

Multi-level

raw feature volume

3D-Decoder

Original Extractor

Hybrid Extractor

（

Ours

）

source views

source views

are highly semantic abstract and not suitable for low-level visual feature matching [49]. To overcome this limitation, inspired by NeuralRecon [50], we further propose Hybrid Extractor. Rather than relying on FPN to generate one feature map from high-level features of CNN, and using a 3D U-Net to process projected features as shown in Fig. 3, we leverage all level features from various layers to construct multi-level volume features. Then, we adopt a 3D CNN decoder to fuse and decode the multi-level volume features, producing the final global volume features.

Our approach enables us to perceive both low and high-level features, which is crucial for generalizable neural surface reconstructions that require detailed surface processing. Second, by avoiding the use of the encoder part of the 3D U-Net, we reduce the computational complexity and allow us to build a higher resolution volume feature within the same computational budget.

3.5 Loss Functions

Our overall loss function is defined as the weighted sum of two loss terms:

where L rendering constrains the observed colors to match the ground truth colors and is formulated as:

Here, S is the number of sampled rays for training, and C g ( r ) represents the ground truth color of the sample ray r . The depth loss L depth is defined as

where S 1 is the number of pixels with valid depth and D g ( r ) is the ground truth depth. In our experiments, we set α = 1 . 0 .

4 Experiments

Datasets. The DTU dataset [10] is a large-scale indoor multi-view stereo dataset consisting of 124 different scenes captured under 7 different lighting conditions. To train our frameworks, we adopted the same approach as in previous works [9, 8]. Furthermore, we evaluated our models' generalization capabilities by testing them on three additional datasets: Tanks & Templates [12], ETH3D [13], and BlendedMVS [11], where no additional training was performed on the testing datasets.

scan55

Reference

VolRecon

ReTR(ours)

scan110

scan63

Reference

VolRecon

ReTR(ours)

scan83

Baselines. To demonstrate the effectiveness of our method from various perspectives, we compared it with (1) SparseNeus [9] and VolRecon [8], the state-of-the-art generalizable neural surface reconstruction method; (2) Generalizable neural rendering methods (3) Neural implicit reconstruction [27, 2, 1, 3] which require individual training for each scene from scratch. (4) Popular multi-view stereo (MVS) [16, 51] methods. Further details on the baselines are provided in the appendix.

4.1 Sparse View Reconstruction

For comparison, we performed sparse reconstruction using only three views, following the same approach as [8, 9]. We adopted the same evaluation process and testing split as used in previous works [8, 9] to ensure a fair comparison. We use a similar approach as VolRecon [8] to generate mesh, more details can be found in Appendix. As shown in Tab. 1, our method outperforms VolRecon [8] and SparseNeuS [9] by a significant margin. Moreover, our method also outperforms popular MVS methods such as MVSNet [51]. Furthermore, we present the qualitative results of sparse view reconstruction in Fig. 4. Our reconstructed geometry exhibits smoother surfaces and less noise compared to the current SoTA methods.

4.2 Depth Map Evaluation & Full View Reconstruction

We compare our rendered depth with those generated by SparseNeuS [9], MVSNet [51] and VolRecon [8]. Following the experiment settings introduced in VolRecon [8], we also use four source views as input for depth rendering. Additionally, we evaluated the performance by fusing all depth maps into a global point cloud. As shown in Tab. 2, our method outperforms existing methods in both evaluations. Moreover, as demonstrated in Fig. 5, our method achieves a sharper boundary with less noise and fewer holes compared to the current SoTA method [8].

VolRecon

ReTR(ours)

VolRecon

ReTR(ours)

4.3 Generalization

To evaluate the generalization ability of our model without retraining, we use three datasets, namely Tank & Temples, BlendedMVS, and ETH3D [12, 11, 13]. The high-quality reconstruction of largescale scenes and small objects in different domains, as shown in Fig. 6, demonstrates the effectiveness of our method in terms of generalization capability.

5 Ablation Study

We conduct ablation studies to examine the effectiveness of each module in our design. The ablation study results are reported on sparse view reconstruction in test split following SparseNeuS [9] and VolRecon [8].

Effectiveness of Modules. We evaluate key components of our approach to generalizable neural surface reconstruction, as shown in Tab. 3. For the evaluation of the occlusion transformer, we keep the original transformer architecture while removing the special design we proposed in Sec. 3.3, to ensure the training parameter would not affect the evaluation. For the hybrid extractor part, we replace this module with the original extractor that has been used in [8]. Our results demonstrate that our approach can

better aggregate features from different levels and use them more effectively. These evaluations highlight the importance of these components in our approach.

Robustness of Different Sampling. Tab. 4 displays the effects of altering the number of sample points on the reconstruction quality of VolRecon[8] and ReTR. Our method surpasses the current SoTA, even when the number of sampling points decreases. These results suggest that existing methods that rely on sampling points of the ray struggle to provide confident predictions of the surface due to the nature of volume rendering. Our approach, which uses learning-based rendering, is more resilient to sampling strategies and can provide reliable depth estimations even with fewer samples. Meanwhile, the effectiveness of continuous P.E. in Sec. 3.3 is proved through the result.

Unsupervised Neural Surface Reconstruction. Our approach is still applicable to unsupervised neural surface reconstruction using only colors for training, which remove L depth. Meanwhile, we find

Reference

Family

(Tank & Temples)

Door

(ETH3D)

Dog

(BlendedMVS)

VolRecon

ReTR(ours)

Table 4: Number of sampling ablation. The ∗ denotes our model using traditional positional encoding.

that our method significantly outperforms the current SoTA method under unsupervised situations and is even comparable to COLMAP [16] the popular MVS technique, As shown in Tab. 5, This is further evidence that the improvement of complex rendering systems for implicit reconstruction is huge.

6 Conclusion

We have proposed Reconstruction Transformer (ReTR), a novel framework for generalizable neural surface reconstruction that uses transformers to model complex rendering processes. ReTR represents a significant advancement in the field of surface reconstruction, offering a powerful solution to the challenges faced by neural implicit reconstruction methods. Additionally, we delve into the design procedure of learning-based rendering. This exploration broadens our understanding of enhancing complex rendering systems and sets the stage for future research endeavors not only in surface reconstruction but also in other tasks relative to differentiable rendering.

References

L. Yariv, J. Gu, Y. Kasten, and Y. Lipman, 'Volume rendering of neural implicit surfaces,' Advances in Neural Information Processing Systems , vol. 34, pp. 4805-4815, 2021.

P. Wang, L. Liu, Y. Liu, C. Theobalt, T. Komura, and W. Wang, 'Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction,' arXiv preprint arXiv:2106.10689 , 2021.

M. Oechsle, S. Peng, and A. Geiger, 'Unisurf: Unifying neural implicit surfaces and radiance fields for multi-view reconstruction,' in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2021, pp. 5589-5599.

M. Niemeyer, L. Mescheder, M. Oechsle, and A. Geiger, 'Differentiable volumetric rendering: Learning implicit 3d representations without 3d supervision,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2020, pp. 3504-3515.

F. Darmon, B. Bascle, J.-C. Devaux, P. Monasse, and M. Aubry, 'Improving neural implicit surfaces geometry with patch warping,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 6260-6269.

Y. Wang, I. Skorokhodov, and P. Wonka, 'Hf-neus: Improved surface reconstruction using highfrequency details,' Advances in Neural Information Processing Systems , vol. 35, pp. 1966-1978, 2022.

J. J. Park, P. Florence, J. Straub, R. Newcombe, and S. Lovegrove, 'Deepsdf: Learning continuous signed distance functions for shape representation,' in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2019, pp. 165-174.

Y. Ren, F. Wang, T. Zhang, M. Pollefeys, and S. Süsstrunk, 'Volrecon: Volume rendering of signed ray distance functions for generalizable multi-view reconstruction,' arXiv preprint arXiv:2212.08067 , 2022.

X. Long, C. Lin, P. Wang, T. Komura, and W. Wang, 'Sparseneus: Fast generalizable neural surface reconstruction from sparse views,' in Proceedings of the European conference on computer vision (ECCV) , 2022, pp. 210-227.

H. Aanæs, R. R. Jensen, G. Vogiatzis, E. Tola, and A. B. Dahl, 'Large-scale data for multipleview stereopsis,' International Journal of Computer Vision , vol. 120, pp. 153-168, 2016.

Y. Yao, Z. Luo, S. Li, J. Zhang, Y. Ren, L. Zhou, T. Fang, and L. Quan, 'Blendedmvs: A large-scale dataset for generalized multi-view stereo networks,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2020, pp. 1790-1799.

A. Knapitsch, J. Park, Q.-Y. Zhou, and V. Koltun, 'Tanks and temples: Benchmarking large-scale scene reconstruction,' ACM Transactions on Graphics , vol. 36, no. 4, 2017.

T. Schops, J. L. Schonberger, S. Galliani, T. Sattler, K. Schindler, M. Pollefeys, and A. Geiger, 'A multi-view stereo benchmark with high-resolution images and multi-camera videos,' in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2017, pp. 3260-3269.

N. D. Campbell, G. Vogiatzis, C. Hernández, and R. Cipolla, 'Using multiple hypotheses to improve depth-maps for multi-view stereo,' in Proceedings of the European conference on computer vision (ECCV) , 2008, pp. 766-779.

D. Chang, A. Božiˇ c, T. Zhang, Q. Yan, Y. Chen, S. Süsstrunk, and M. Nießner, 'Rc-mvsnet: unsupervised multi-view stereo with neural rendering,' in Proceedings of the European conference on computer vision (ECCV) , 2022, pp. 665-680.

J. L. Schönberger, E. Zheng, J.-M. Frahm, and M. Pollefeys, 'Pixelwise view selection for unstructured multi-view stereo,' in Proceedings of the European conference on computer vision (ECCV) , 2016, pp. 501-518.

X. Gu, Z. Fan, S. Zhu, Z. Dai, F. Tan, and P. Tan, 'Cascade cost volume for high-resolution multiview stereo and stereo matching,' in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2020, pp. 2495-2504.

J. Yang, W. Mao, J. M. Alvarez, and M. Liu, 'Cost volume pyramid based depth inference for multi-view stereo,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2020, pp. 4877-4886.

K. Luo, T. Guan, L. Ju, H. Huang, and Y. Luo, 'P-mvsnet: Learning patch-wise matching confidence aggregation for multi-view stereo,' in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2019, pp. 10 452-10 461.

K. N. Kutulakos and S. M. Seitz, 'A theory of shape by space carving,' International journal of computer vision , vol. 38, pp. 199-218, 2000.

S. M. Seitz and C. R. Dyer, 'Photorealistic scene reconstruction by voxel coloring,' International journal of computer vision , vol. 35, pp. 151-173, 1999.

Y. Yao, Z. Luo, S. Li, T. Shen, T. Fang, and L. Quan, 'Recurrent mvsnet for high-resolution multi-view stereo depth inference,' in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2019, pp. 5525-5534.

Y. Furukawa and J. Ponce, 'Accurate, dense, and robust multiview stereopsis,' IEEE transactions on pattern analysis and machine intelligence , vol. 32, no. 8, pp. 1362-1376, 2009.

M. Lhuillier and L. Quan, 'A quasi-dense approach to surface reconstruction from uncalibrated images,' IEEE transactions on pattern analysis and machine intelligence , vol. 27, no. 3, pp. 418-433, 2005.

R. Chen, S. Han, J. Xu, and H. Su, 'Point-based multi-view stereo network,' in Proceedings of the IEEE/CVF international conference on computer vision , 2019, pp. 1538-1547.

B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng, 'Nerf: Representing scenes as neural radiance fields for view synthesis,' Communications of the ACM , vol. 65, no. 1, pp. 99-106, 2021.

L. Yariv, Y. Kasten, D. Moran, M. Galun, M. Atzmon, B. Ronen, and Y. Lipman, 'Multiview neural surface reconstruction by disentangling geometry and appearance,' Advances in Neural Information Processing Systems , vol. 33, pp. 2492-2502, 2020.

J. Zhang, Y. Yao, and L. Quan, 'Learning signed distance field for multi-view surface reconstruction,' in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2021, pp. 6525-6534.

W. Ge, T. Hu, H. Zhao, S. Liu, and Y.-C. Chen, 'Ref-neus: Ambiguity-reduced neural implicit surface learning for multi-view reconstruction with reflection,' arXiv preprint arXiv:2303.10840 , 2023.

Q. Wang, Z. Wang, K. Genova, P. P. Srinivasan, H. Zhou, J. T. Barron, R. Martin-Brualla, N. Snavely, and T. Funkhouser, 'Ibrnet: Learning multi-view image-based rendering,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2021, pp. 4690-4699.

A. Chen, Z. Xu, F. Zhao, X. Zhang, F. Xiang, J. Yu, and H. Su, 'Mvsnerf: Fast generalizable radiance field reconstruction from multi-view stereo,' in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2021, pp. 14 124-14 133.

A. Yu, V. Ye, M. Tancik, and A. Kanazawa, 'pixelnerf: Neural radiance fields from one or few images,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2021, pp. 4578-4587.

A. Trevithick and B. Yang, 'Grf: Learning a general radiance field for 3d representation and rendering,' in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2021, pp. 15 182-15 192.

S. Peng, M. Niemeyer, L. Mescheder, M. Pollefeys, and A. Geiger, 'Convolutional occupancy networks,' in Proceedings of the European conference on computer vision (ECCV) , 2020, pp. 523-540.

K. Zhang, F. Luan, Q. Wang, K. Bala, and N. Snavely, 'Physg: Inverse rendering with spherical gaussians for physics-based material editing and relighting,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2021, pp. 5453-5462.

X. Zhang, P. P. Srinivasan, B. Deng, P. Debevec, W. T. Freeman, and J. T. Barron, 'Nerfactor: Neural factorization of shape and reflectance under an unknown illumination,' ACM Transactions on Graphics (TOG) , vol. 40, no. 6, pp. 1-18, 2021.

J. Knodt, J. Bartusek, S.-H. Baek, and F. Heide, 'Neural ray-tracing: Learning surfaces and reflectance for relighting and view synthesis,' arXiv preprint arXiv:2104.13562 , 2021.

V. Sitzmann, J. Thies, F. Heide, M. Nießner, G. Wetzstein, and M. Zollhofer, 'Deepvoxels: Learning persistent 3d feature embeddings,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2019, pp. 2437-2446.

V. Sitzmann, S. Rezchikov, B. Freeman, J. Tenenbaum, and F. Durand, 'Light field networks: Neural scene representations with single-evaluation rendering,' Advances in Neural Information Processing Systems , vol. 34, pp. 19 313-19 325, 2021.

M. S. Sajjadi, H. Meyer, E. Pot, U. Bergmann, K. Greff, N. Radwan, S. Vora, M. Luˇ ci´ c, D. Duckworth, A. Dosovitskiy et al. , 'Scene representation transformer: Geometry-free novel view synthesis through set-latent scene representations,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 6229-6238.

V. Sitzmann, M. Zollhöfer, and G. Wetzstein, 'Scene representation networks: Continuous 3d-structure-aware neural scene representations,' Advances in Neural Information Processing Systems , vol. 32, 2019.

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, 'Attention is all you need,' Advances in neural information processing systems , vol. 30, 2017.

J. Reizenstein, R. Shapovalov, P. Henzler, L. Sbordone, P. Labatut, and D. Novotny, 'Common objects in 3d: Large-scale learning and evaluation of real-life 3d category reconstruction,' in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2021, pp. 10 901-10 911.

Y. Liu, S. Peng, L. Liu, Q. Wang, P. Wang, C. Theobalt, X. Zhou, and W. Wang, 'Neural rays for occlusion-aware image-based rendering,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 7824-7833.

M. Suhail, C. Esteves, L. Sigal, and A. Makadia, 'Generalizable patch-based neural rendering,' in Proceedings of the European conference on computer vision (ECCV) , 2022, pp. 156-174.

P. Wang, X. Chen, T. Chen, S. Venugopalan, Z. Wang et al. , 'Is attention all nerf needs?' arXiv preprint arXiv:2207.13298 , 2022.

B. Curless and M. Levoy, 'A volumetric method for building complex models from range images,' in Proceedings of the 23rd annual conference on Computer graphics and interactive techniques , 1996, pp. 303-312.

Z. J. Tang, T.-J. Cham, and H. Zhao, 'Able-nerf: Attention-based rendering with learnable embeddings for neural radiance field,' arXiv preprint arXiv:2303.13817 , 2023.

C. Cao, X. Ren, and Y. Fu, 'Mvsformer: Learning robust image representations via transformers and temperature-based depth for multi-view stereo,' arXiv preprint arXiv:2208.02541 , 2022.

J. Sun, Y. Xie, L. Chen, X. Zhou, and H. Bao, 'Neuralrecon: Real-time coherent 3d reconstruction from monocular video,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2021, pp. 15 598-15 607.

Y. Yao, Z. Luo, S. Li, T. Fang, and L. Quan, 'Mvsnet: Depth inference for unstructured multiview stereo,' in Proceedings of the European conference on computer vision (ECCV) , 2018, pp. 767-783.

K. Xu, X. Wan, H. Wang, Z. Ren, X. Liao, D. Sun, C. Zeng, and K. Chen, 'Tacc: A full-stack cloud computing infrastructure for machine learning tasks,' arXiv preprint arXiv:2110.01556 , 2021. [Online]. Available: https://arxiv.org/abs/2110.01556

Appendix

A Implementation Details

A.1 Experimental Environment

Software and hardware environment:

•

CUDA version: 11.1

•

cuDNN version: 8.0.5

PyTorch version: 1.10.1

GPU: Nvidia RTX 3090

CPU: Intel Xeon Platinum 8180 @ 2.50 GHz ×

2

A.2 Training Detail

Our model is implemented in PyTorch using the PyTorch Lightning framework. During the training stage, we resize the input image to 640 x 512 and the source views to N = 4 . To train our model, we employ the Adam optimizer on a single Nvidia 3090 GPU. Initially, the learning rate is set to 10 -4 and gradually decays to 10 -6 using a cosine learning rate scheduler. Throughout the training, we use a batch size of 2 and set the number of rays to 1024. To enhance the sampling strategy, we apply a coarse-to-fine approach with both N coarse and N fine set to 64. The N coarse points are uniformly sampled between the near and far plane, while the N fine points are sampled using importance sampling based on the coarse probability estimation. Regarding the global feature volume f v , we set its resolution to K=128. For inference on DTU, the image resolution is set to 800 x 600. For datasets such as BlendedMVS [11], ETH3D [13], and Tanks & Temples [12], we maintain the original image resolution. Training our model requires approximately 3 days on a single Nvidia 3090 GPU. Moreover, when constructing larger models such as the large and xlarge models by stacking more layers, the training time will naturally increase due to the increased model size.

A.3 Mesh and Point Cloud Generation

Following the settings employed in VolRecon [8], we generate depth maps from virtual viewpoints by shifting the original camera along its x -axis by d = 25 mm. Subsequently, we perform TSDF fusion and applied the Marching Cubes algorithm to merge all the rendered depths into a voxel grid with a resolution of 1.5 mm and extract a mesh representation. For point cloud generation, we initially generate 49 depth maps by leveraging the four nearest source views. These 49 depth maps are then fused together to form a unified point cloud.

B Technical Details and Discussion

B.1 Discussion of Hitting Probability

The attention score in ReTR can be interpreted as the probability of a ray being hit . However, when using softmax , the attention scores for each ray are forced to sum up to 1, implying that every ray should be considered a hit . To gain further insights, we examine the distribution of attention scores for rays that are not hitting . Figure 7 illustrates the results, demonstrating that the transformer intelligently employs a wider distribution to model rays that do not hit . The underlying

Render image

Hitting probability

weights

ray distance

weights

ray distance

0.4

.3

2

0.1

0.0

1.0

2.0

0.4

2

0.3

0.1

H

0.0

1.0

1.5

2.0

2.5

3.5

rationale is that the transformer treats the surrounding air as a medium that contributes to the color.

When a ray does not hit , the transformer aggregates information from the surrounding air to obtain the color from these mediums.

B.2 Hierarchical Volume Sampling through Attention Map

Given that our framework does not incorporate weights as seen in traditional frameworks like NeRF or NueS, we refine the original hierarchical sampling strategy by substituting the weights with the attention scores of each point. This approach, as discussed in the main text, is both straightforward and impactful. Additionally, we highlight that our method exhibits greater robustness in terms of the number of sampling points compared to the current state-of-the-art techniques, thereby offering an additional advantage within our framework.

B.3 Continous Positional Encoding Proof

To imbue our system with positional awareness of actual distance. we initially derive the formula for the attention score, denotes as s of features in x i and x j :

where p represents the positional encoding in x . We subsequently expand Eq. (16) as follows:

where the fourth component of Eq.(17) denotes the interaction between two locations, and W q W ⊤ k represents the trainable parameters. To ensure our MLP actual positional awareness, we need to make the function satisfy the following conditions:

where t j -t i denotes the distance between x i and x j . When we apply our positional encoding (PE), the fourth component of Eq.(17) can be simplified as:

By applying the sum-to-product identities, we obtain:

where ( t j -t i ) represents the actual distance between x i and x j . Thus, continuous positional encoding enables the attainment of actual positional awareness.

C Additional Experimental Results

Here we show additional experiment results:

C.1 Visualization Supplementary

Due to space limitations, we provide additional visual results for the experiments in this section. Specifically, we present the results for sparse view reconstruction in Fig. 9 and full view reconstruction of the point cloud in Fig. 11. Furthermore, we include the per-scene results for the number of sampling points in Tab. 6.

C.2 Error Bar of ReTR

In order to assess the reproducibility and robustness of our model, we conduct three separate training runs using different random seeds. The corresponding results are presented in Figure 8. These results demonstrate that our model exhibits consistent performance across multiple training runs, indicating good reproducibility. Moreover, the minimal variance observes in the results further underscores the robustness of our model.

C.3 Effectiveness of Stacking Transformer Blocks

To explore the potential of simulating more complex light transport effects, we extend our learnable rendering approach by stacking multiple layers of transformer blocks. Specifically, we introduce two variations: ReTR-L , where we stack two transformer blocks, and ReTR-XL , where we stack three transformer blocks. This allows us to experimentally evaluate the effectiveness of a more complex rendering system. The results of these experiments are summarized in Tab. 7. The findings indicate that by overlaying multiple layers of transformers, we can simulate complex lighting effects and achieve more powerful results. This demonstrates the potential of our approach to capture intricate light transport phenomena and enhance the overall rendering capabilities.

C.4 Novel View Synthesis

In order to assess ReTR's performance in Novel View Synthesis, a task where many multi-view stereo techniques struggle, we conduct a quantitative comparison with VolRecon [8]. The novel views are generated during the full reconstruction for fusing the point clouds as we discussed in the main paper. Our results demonstrate a significant improvement over VolRecon in terms of novel view synthesis, as shown in Tab. 8. Additionally, we provide visualizations of novel view synthesis and depth synthesis in Figure 10. It has been challenging to achieve high-quality results simultaneously in rendering-based studies and reconstruction-based studies, with few methods excelling in both aspects. However, the results achieved by our proposed framework, ReTR, are highly promising, which suggests that a learnable rendering approach based on transformers can effectively integrate both tasks, yielding impressive results on both fronts within a unified framework.

D Limitations

Our method requires approximately 30 seconds to render a depth map and image with a resolution of 600 × 800 . Similar to other rendering-based methods such as IBRNet [30], VolRecon [8], and MVSNeRF [31], our approach has limitations in terms of efficiency. While learning-based rendering offers enhanced capabilities, it does introduce additional training parameters compared to traditional volume rendering techniques. Stacking multiple layers of our model can improve performance; however, it also increases training time due to the larger model size. It is important to strike a balance between achieving higher rendering quality and maintaining reasonable computational efficiency. Further research and optimization efforts can be explored to enhance the efficiency of our method,

Error Bar of Ours (ReTR)

Scene result

Scan24

Scan37

Scan40

Scan55

Scan63

Scan65

Scan69

Scan83

Scan97

Scan105

Scan106

Scan110

Scan114

Scan118

Scan122

0

1

2

3

potentially through techniques such as model compression, parallelization, or hardware acceleration. Acknowledging these limitations, we aim to provide a comprehensive understanding of the trade-offs between rendering quality, efficiency, and model complexity within our proposed framework.

E Broader Impacts

The proposed ReTR framework not only enables accurate surface reconstruction through learnable rendering but also generates high-quality novel views. These capabilities open up possibilities for various downstream applications in fields such as virtual reality (VR), robotics, and view synthesis with geometry. While these applications offer numerous benefits, it is important to acknowledge that they also come with ethical considerations. As authors of the ReTR framework, we are committed to promoting ethical practices and responsible development. We recognize the potential for misuse, such as generating content without consent, and we prioritize fair representation and responsible usage of the technology. We strive to adhere to ethical guidelines and contribute to the development of responsible AI practices. It is crucial to ensure that technological advancements are leveraged for the betterment of society while minimizing potential negative impacts. By maintaining a focus on ethics, fairness, and responsible development, we aim to ensure that ReTR and its applications are aligned with the principles of responsible AI and contribute positively to the broader scientific community and society as a whole.

scan55

scan65

scan83

scan114

scan118

Reference

VolRecon

ReTR(ours)

scan55

scan63

scan83

scan114

scan110

Reference

VolRecon

ReTR(ours)

View Synthesis

Depth Synthesis

View Synthesis

Depth Synthesis

37

40

55

63

69

83

97

106

114

118

122

110

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a learning-based framework on the well-studied neural surface reconstruction problem. The key contribution of this paper is to take the complex photon-particle interaction into account and present a more generalized pipeline rather than relying on volume rendering. The proposed framework released the power of the transformer to achieve enhanced feature representation of sampled points along the ray. Experiments on several popular benchmarks have shown the effectiveness of the proposed approach.\n\nSTRENGTHS:\n(1) The idea of modeling the complex light transport and releasing the flexibility from regular volume rendering is novel and interesting.\n(2) Overall, the paper is well presented and easy to follow.\n(3) The paper has achieved state-of-the-art generalizable neural surface reconstruction performance across different datasets.\n\nWEAKNESSES:\n(1) (Major) The idea of using 3D feature volumes with hybrid resolution is not new and has been proposed in NeuralRecon (https://arxiv.org/pdf/2104.00681.pdf). Besides, since there are two major differences (the elimination of FPN and the construction of multi-level projected feature maps) between the proposed hybrid extraction and the original one, it is better to make separate ablation studies to further verify the effect of the two variations.\n(2) (Major) For the ablation study of the occlusion transformer, compared with directly removing this module, a better ablation way is to attend every point’s feature as the input of the key embedding in the self-attention computation. This way ensures a fair setting (roughly same architecture and complexity) and the only difference is whether the later points contribute to the former ones.\n(3) (Minor) Visual comparison on view synthesis with other methods: since one of the main motivations of this work is to model the photon-particle interaction, I assume a major outcome is more robust rendering against the variations (blur, specular..) from input views. Thus, it is better to show some visual comparison with other baselines (SparseNeus, VolRecon, NeRF…) to verify this point.\n(4) (Minor)The main diagram (Figure 2) can be displayed in a clearer and more elegant way. Basically the part of transformer details would belong to either occlusion transformer or render transformer, rather than ‘feature fusion’. Besides, there is ambiguity on what the patches with different colors stand for.\n\nQUESTIONS:\nOverall I think the idea of the paper is novel and technically sound, presenting another perspective of the mainstream methods based on volume rendering. Despite some technical concerns listed in weaknesses, now I lean toward accepting this paper. I expect authors to address my concerns by providing more comprehensive experiments to further show the effectiveness of this work.\n\nLIMITATIONS:\nAuthors have discussed their limitations on rendering speed. I think another limitation is that the SOTA neural surface reconstruction method is still only comparable with a classic MVS-based baseline (MVSNet) at this time. But MVSNet and its extensions must be much faster to get a depth map than the rendering-based ones. So there is a long way to go for this area to further release the power of implicit representation.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes ReTR, a new architecture that leverages transformer to replace the traditional volume rendering process. The insight of the paper is that: the traditional volume rendering equation is oversimplified to model photon-particle interaction. Moreover, the color compositing function highly relies on the projected input view colors, and therefore overlooking intricate physical effects. To solve these two limitations, ReTR replaces the volume rendering equation with a render transformer. The attention map can be extracted from the render transformer to synthesize geometry details. An occlusion transformer is further introduced to obtain finer features. Instead of using FPN features and ResUNet, ReTR also utilizes features from different layers to construct multi-scale features. Experiments are conducted on the DTU dataset, Tanks & Temples, ETH3D, and BlendedMVS. The method is compared with state-of-the-art generalizable NeRF methods and generalizable surface reconstruction methods and achieves the best performance among them. Ablation studies also show the effectiveness of the network architecture. Moreover, ReTR surpasses SparseNeuS and VolRecon even without depth supervision.\n\nSTRENGTHS:\nI like the insights proposed in the paper that the traditional volume rendering equation is oversimplified. The solution that utilizes transformers to replace volume rendering is straightforward yet sound and effective. Experiments are exhaustive and validated the network designation. I also like the discussion of interpreting the render transformer the hitting probability.\n\nWEAKNESSES:\nMore related work should be discussed in Section 2 for generalizable NeRF methods (NeuRays, CVPR22; Generalizable Patch-Based Neural Rendering, ECCV 2022, ...) and neural surface reconstruction methods (NeuS, ...).\nNotations in Equations (8) and (9) are not well explained, for example, what are $\\mathbf{R}^f$ and $\\mathbf{R}^{\\text{occ}}$, and $\\mathbf{f}_i^{\\text{occ}}$ did not appear before Equation (10).\nThere is also a typo at Line 232: **tanks and temples** instead of **tanks and templates**.\n\nQUESTIONS:\nThough the proposed method is effective, mathematically, I did not see how the network architecture can model the geometry better than other neural surface reconstruction methods, such as SparseNeuS and VolRecon -- Especially when the network is trained without depth supervision. I would like to see more insights and explanations from the authors in the feedback.\n\nLIMITATIONS:\nThe main concern of this paper is it requires a long training time, e.g. 3 days on a 3090. It comes with a cost when introducing transformers in the network architecture. Generally, it is not a big problem since the method is generalizable.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new framework for generalizable neural surface reconstruction, which utilizes the mechanism of transformers to model the rendering process. The authors first derive a general form of generalizable volume rendering based on existing methods and identify its limitations. They then suggest improving upon this framework by introducing a new rendering approach based on learned attention maps and performing over-ray accumulation in feature space rather than color space. Experiments are conducted on four different datasets and compared to recent baselines, demonstrating superior performance in generalizable reconstruction.\n\nSTRENGTHS:\n- The paper is well-written and exhibits a smooth flow. The authors effectively convey the motivation behind their methods, providing necessary background information and comparing against recent baselines. Additionally, the authors ensure that the reader can easily follow the logical progression of the paper.\n- Section 3.1 presents a general framework that serves as a well grounded basis for existing methods. The authors' identification of limitations within this framework offers valuable observations, contributing to the overall storyline and motivation of the paper.\n- The results presented in both the main paper and supplementary material demonstrate the superiority of the proposed method compared to the VolRecon baseline.\n- The authors provided an ablation study on the various components of the method, effectively differentiating their individual contributions and providing a solid understanding of their impact.\n\nWEAKNESSES:\n- The section describing the reconstruction transformer appears to be incomplete and confusing due to several missing details and explanations:\n\n  - It is unclear how equation 6 (and its improvement in equation 10) fit into the general framework proposed in equation 5. This confusion arises because the final MLP from feature to color does not align with the framework. Additionally, the weight function and color function are not explicitly provided.\n  - The FeatureFusion operation is not defined, leaving ambiguity in understanding its purpose and implementation.\n  - The definition of the \"meta-ray token\" $f^{tok}$ for a scene is unclear, specifically whether it pertains to per-image (per-ray/pixel) or per-scene information, and how it differs from the image features $f^{img}$.\n  - The meaning of $F_i$ in line 161 is not provided or explained.\n\n  It is crucial for the authors to address these misunderstandings and revisit the missing definitions in order to clarify the concepts.\n\n- While it is acknowledged that the general form presented in section 3.1 oversimplifies the modeling of light transport in 3D scenes, it is hard to perceive how the mechanism of cross-attention over ray samples effectively models complex photon-particle interactions. Real interactions typically occur in spatial domains, whereas the suggested approach focuses on interactions over ray samples. It is suggested that the authors either temper these claims or provide further explanation on how their framework accounts for intricate global physical effects that encompass both global and local physical effects.\n\n- The qualitative comparison is somewhat limited as it only includes a comparison to VolRecon. It is essential for the authors to provide visual comparisons with other methods as well, particularly SparseNeuS, to offer a more comprehensive evaluation.\n\n- The authors have not presented timing evaluations of their method in both training and evaluation scenarios. Given that the limitation section highlights timing as a significant drawback of generalizable methods, it is important for the authors to address this by providing timing evaluations to enhance the paper's completeness.\n\nQUESTIONS:\n- When evaluating the SparseNeuS baseline, did the authors incorporate depth supervision as well? My understanding is that both ReTR and VolRecon utilize depth supervision during training, while SparseNeuS does not. This raises concerns about the fairness of the comparison between these methods.\n- In Section 3.2, the paper suggests key rendering properties that the system should possess. However, there is no specific mention of the requirement for the weights to sum up to 1, indicating that all rays are eventually occluded. While this property is not explicitly described, it seems to be employed later in the paper using softmax. It would be helpful if the authors provided further clarification on this matter.\n\nLIMITATIONS:\nThe authors have discussed limitations in the supplementary material. However, it is necessary to present the main limitation of efficiency in the main paper, even if briefly in the conclusion section. Additionally, providing quantitative results that demonstrate the tradeoff between the number of parameters and training/rendering time would greatly benefit the presentation of the method.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis works focus on generalizable asset reconstruction: given a few posed images, predict the 3D representations using a network.\nInstead of using volume rendering to compute the transmittance, the authors propose to use transformer on the sampled points to compute the weight of each point.\nBesides, the author also  improve the CNN architecture for feature extraction.\nExtensive experiments are conducted on multiple datasets, demonstrating better performance.\n\nSTRENGTHS:\n1. Better performance compared to previous compared with previous SOTA methods\n2. Code is attached and will be release.\n3. method is well explained\n\nWEAKNESSES:\n1. Compared to previous methods SparseNeus and Volrecon, this work seems somewhat incremental. Major difference is using a hybrid CNN extractor (Fig 3) and a new transformer architecture. The \"occlusion transformer\" and \"render transformer\" seems to be two transformers with fancy names, and I don't see significant difference from the transformer in volrecon. Though Volrecon is a CVPR2023 paper, but apparently the authors use its codebase for develop , as can be seen in the code.zip in the supplementary.\n\n2. I don't agree with the \"rethink\" title, equation5 doesn't make too much sense to me. It's not explained why the equation can satisfy the occlusion-aware, and especially no guarantee of consistency across multi views.\nFor example, for a sampled point $x$, its weight may  be $1$ for the ray $r_1$, but may be $0$ for the ray $C(r_2)$ even when $x$ is the nearest point in ray $r_2$. Therefore, I'm not fully convinced with that Eq 5 is a better modelling of the  rendering, and the \"rethinking\" seems be some sort of exaggeration.\nFurthermore, given the goal of this work is reconstruction, I don't see why loosing the physics constrain in rendering can benefit the learned geometry.\n\n## Justification of rating.\n1. Pros: results are solid (multiple datasets, compared with SOTA baselines), code available.\n2. Cons: Somewhat incremental, \"rethinking\" seems a bit exaggeration.\n\nOverall I lean to a borderline accept, as no big technical flaws, but I'm not very confident.\n\nQUESTIONS:\nPlease check the weakness section. Besides:\n\n1. This may not be the weakness of this paper, but it's common in the research of this topic.  When it's called \"generalizable\", why it can not generalize to the unseen region of the object? For example, given images of the front views of the statue, why not generalize it to the backview in reconstruction. \n\n2. What's the PSNR compared to other baselines\n\nLIMITATIONS:\nDiscussion of limitation and broad impacts in the supplementary.\nNo license/asset description according to https://neurips.cc/public/guides/PaperChecklist . But I don't penalize it in the rating.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
This paper introduces a interesting solution for volume renderings in generalizable neural surface reconstruction by leverage Transformers to predict depths and colors from feature volumes. The results on sparse view reconstruction prove its useness.

STRENGTHS:
The authors identify the limitation and derive a general form of volume rendering and propose ReTR, a learning-based rendering framework utilizing transformer architecture to model light transport.
A hybrid feature extractor is also proposed for achieving better performance.

WEAKNESSES:
Can we replace the volume rendering in the optimization-based methods (e.g. NeuS / VolSDF) with the learned Transformer ? Or is the solution only works for generalizable neural surface reconstruction?

What is the performance in scene-level sparse view reconstruction, i.e., Replica/ScanNet.

Will HybridExtractor also works for volume rendering based methods (e.g. SparseNeuS / VolRecon) ?

Why Transformer? Will CNN/MLP also works for this design?

QUESTIONS:
See the weakness above.

LIMITATIONS:
See the weakness above.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-09