ReTR: Modeling Rendering Via Transformer for Generalizable Neural Surface Reconstruction
Authors: Yixun Liang
Hao He
Ying-Cong Chen
Venue: NeurIPS 2023 poster (2023)
Keywords: 3D vision
3D reconstruction
Generalizable Neural Surface Reconstruction

Abstract: Generalizable neural surface reconstruction techniques have attracted great attention in recent years. However, they encounter limitations of low confidence depth distribution and inaccurate surface reasoning due to the oversimplified volume rendering process employed. In this paper, we present Reconstruction TRansformer (ReTR), a novel framework that leverages the transformer architecture to redesign the rendering process, enabling complex render interaction modeling. It introduces a learnable $\textit{meta-ray token}$ and utilizes the cross-attention mechanism to simulate the interaction of rendering process with sampled points and render the observed color. Meanwhile, by operating within a high-dimensional feature space rather than the color space, ReTR mitigates sensitivity to projected colors in source views. Such improvements result in accurate surface assessment with high confidence. We demonstrate the effectiveness of our approach on various datasets, showcasing how our method outperforms the current state-of-the-art approaches in terms of reconstruction quality and generalization ability. $\textit{Our code is available at }$ https://github.com/YixunLiang/ReTR.

ReTR: Modeling Rendering Via Transformer for Generalizable Neural Surface Reconstruction
Yixun Liang' Hao Hel:
Ying-Cong Chen 2,
The Hong Kong University of Science and Technology (Guangzhou).
The Hong Kong University of Science Technology.
yliang9820connect. hwwsh gz..-gzz..o www.gz...uddddeee en, hheat@connect. ust.uttth .k, yingcongchenkust... .hk
Abstract
Generalizable neural surface reconstruction techniques have attracted great atten- tion in recent years. However, they encounter limitations of low confidence depth distribution and inaccurate surface reasoning due to the oversimplified volume rendering process employed. In this paper, we present Reconstruction TRans- former (ReTR), a novel framework that leverages the transformer architecture to redesign the rendering process, enabling complex render interaction modeling. It introduces a learnable meta-ray token and utilizes the cross-attention mechanism simulate the interaction of rendering process with sampled points and render the observed color. Meanwhile, by operating within high-dimensional feature space rather than the color space, ReTR mitigates sensitivity to projected colors source views. Such improvements result in accurate surface assessment with high confidence. We demonstrate the effectiveness our approach on various datasets, showcasing how our method outperforms the current tatte-of-theaatt art approaches in terms of reconstruction quality and generalization ability. Our code is available at https://githu/: com http//ixxniinnnngn ReTR
Introduction
In the realm of computer vision and graphics, extracting geometric information from multi-view images poses a significant challenge with far-reaching implications for various fields, including robotics, augmented reality, and virtual reality. As popular approach to this problem, neural implicit reconstruction techniques [ 2
6 are frequently employed, generating accurate and plausible geometry from multi-view images by utilizing volume rendering and neural implicit representations based on the Sign Distance Function (SDF) [7] and its variant. Despite their efficacy, these methods possess inherent limitations such as the lack of cross-scene generalization capabilities and the necessity for extensive computational resources for training them from scratch for each scene. Furthermore, these techniques heavily rely on a large number of input views.
Recent studies such as SparseNeuS [9] and VolRecon g have attempted to overcome these challenges by integrating prior image information with volume rendering methods, thereby achieving impressive cross-scene generalization capabilities while only requiring sparse views as input. Nevertheless, these solutions are still based on volume rendering, which poses some intrinsic drawbacks in surface reconstruction. Specifically, volume rendering is simplification of the physical world and might not capture the full extent of its complexity. It models the interactions of incident photons and particles into density, which is predicted solely based on sampled point features. This oversimplified modeling fails to disentangle the contribution of the light transport effect and surface properties the observed color, resulting in inaccurate assessment of the actual surface. Moreover, the prediction of color
"Equal contribution.
t "orresponningg author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

(s) views
(c) ReTR(ours)
Figure 1: Generalizable neural surface reconstructions from three input views in (a). VolRecon g produces depth distribution with low kurtosis and a noisy surface as shown in (b). In contrast, our proposed ReTR successfully extracts plausible surfaces with sharper depth distribution that has high kurtosis as shown in (c).
blending heavily relies on the projected color of the source views, thereby overlooking intricate physical effects. These shortcomings can lead to less confident surface predictions, producing depth distribution with low kurtosis, and may be accompanied by high levels of noise, thereby reconstruction as illustrated in Fig. (b).
compromising the overall quality
In this paper, we first propose more generalized formulation for volume rendering. Then based on this formulation, introduce the Reconstruction TRansformer (RcTR), a novel approach for generalizable neural surface reconstruction. Our method utilizes the transformer to redesign the rendering process, allowing for accurate modeling of the complex render interaction while retaining the fundamental properties that make volume rendering effective. Particularly, we propose learnable token, the meta-ray token, that encapsulates the complex light transport effect. By leveraging this token and the "cross-attention sampled points and aggregate the features each point to render the observed color in an end-to-end manner. Additionally, we introduce unidirectional transformer and continuous positional encoding to simulate photon-medium model operates in the feature space rather than the color space, further enhance the accuracy semantic information and enable seamless adaptation, we propose a novel hybrid extractor designed efficiently extract 3D-aware features.
mechanism, we simulate the interaction of rendering with the
points. Moreover, medium interaction, effectively considering occlusion and the interval of sample
Our method provides an excellent solution for generalizable neural surface reconstruction, offering several advantages over traditional volume rendering techniques. First, the ability to generalize complicated physical effects data-driven way provides an efficient approach to decoupling the contribution of light transport and surface the observed color. Second, the entire process takes place high-dimensional feature space, rather than the color space, reducing the model's sensitivity to projected color in source views. Moreover, the transformer employs re-weighted softmax function of the attention map, driving the learning of a depth distribution with positive kurtosis. As a result, method achieves more confident distribution, leading reduced noise and improved quality, As shown in Fig. (c).
In summary, contribution can be summarized as:
We identify the limitation and derive a general form of volume rendering. By leveraging this form, we can effectively tailor the rendering process for task-specific requirements.
Through the derived general form, we propose ReTR, a learning- based rendering framework utilizing transformer architecture to model light transport. ReTR incorporates continuous positional encoding and leverages the hybrid feature extractor to enhance performance in generalizable neural surface reconstruction.
Extensive experiments conducted on DTU, BlendedMVS, ETH3D, and Tanks Temples |II 12 3 validate the efficacy and generalization ability of ReTR.
datasets [00

---

2
Related Works
Multi-Vier Stereo (MVS). Multi-view stereo methods is another branch for 3D reconstruction.
which can be broadly categorized into three main branches: depth maps-based [14 15 16
17 [8] 19]
voxel grids-based [20 21 22], and point clouds-based [23 24 25). Among these, depth maps-based methods are more flexible and hence more popular than the others. Depth maps-based methods typically decouple the problem into depth estimation and fusion, which has shown impressive performance with densely captured images. However, these methods exhibit limited robustness in situations with a shortage of images, thereby highlighting the problem we aim to address.
Neural Surface Reconstruction. With the advent of NeRF [26], there has been paradigm shift towards using similar techniques for shape modeling, novel view synthesis, and multi-view 3D reconstruction. IDR (27] uses surface rendering to learn geometry from multi-view images, but requires extra object masks. Several methods 12 3 density function in NeRF using SDF and its variants, successfully regressed plausible geometry. Among them, NeuS 2 first uses SDF value to model density in the volume rendering to learn neural implicit surface, offering a robust method for multi-view 3D reconstruction from 2D images. However, these methods require lengthy optimization to train each scene independently. Inspired by the recent success of generalizable novel view synthesis 30
28 29] have attempted to rewrite the
33 34]. SparseNeuS [9] and VolRecon [ achieve generalizable neural surface reconstruction using the information from the source images as the prior neural surface reconstruction. However, these methods suffer from oversimplified modeling of light transport in traditional volume rendering and color blending, resulting in extracted geometries that fail make a confident prediction of the surface, leading to distortion in the reconstructions. However, there are also some attempts [5 36 337 that adopt other rendering methods in implicit neural representation, such as ray tracing. However, these models are still based on enhanced explicit formulations; thus, their capacity to model real-world interaction is still limited. In contrast, our method introduces learning- based rendering, providing an efficient way to overcome such limitations.
BT
32
Learning Based Rendering. Unlike volume rendering, another line of work 38
39
40
explores
deep learning techniques to simulate the rendering process. Especially Recurrent neural networks, which naturally fit the rendering process. Specifically, Deep Voxels (8 employs GRU to process voxel features along a ray, and its successor SRN [41), leverages LSTM for ray-marching. However, such methods recursively process features and demand large computation resources. The computation constraints inhibit such methods' ability to produce high-quality renderings. Unlike RNN-based methods, ReTR leverages the transformer to parallel compute each point's hitting probability. Greatly improve the efficiency of the rendering process and achieve high-qualin renderings.
Transformers With Radiance Field. The attention mechanism in transformers been widely used in the area of radiance field. In image-basee rendering, IBRNet ray transformer process sampled point features and predict density. NeRFormer a transformer to aggregate source views and construct feature volumes. NeuRays neural networks to model and address occlusions, enhancing the quality and accuracy of image-based based rendering. GPBR (45) employs neural networks to transform and composite patches from source images, enabling versatile and realistic image synthesis across various scenes. However, these methods only use the transformer to enhance feature aggregation, and the sampled point features are still decoded into colors and densities and aggregated using traditional volume rendering. leading to unconfident surface prediction. Recently, GNT [46) naively replaces classical volume rendering with transformers in image-basee based rendering, which overlooks the absence of occlusion and positional awareness within the transformer architecture. In contrast to GNT, we improved the traditional transformer architecture in those two limitations based on an in-depth analysis of the fundamental components to make volume rendering work.
42
has also
30 proposes
utilizes
44 leverages
3 Methodology
In this section, we present an analysis of the limitations of existing generalizable neural surface reconstruction approaches that adopt volume rendering from NeRF (26) and propose ReTR, a novel architecture that leverages transformer to achieve learning-based rendering. We introduce the formulations of volume rendering and revisit its limitations in generalizable neural surface reconstruction in See 3.1 We then depict the general form of volume rendering in Sec. and present
3

---

Witdlt
Transformer
Occlusion
Render Transformee
Depth
Opeed. Dgt
Hybrid Extractor
Renddriin
LJL
Trass
Multiplicriuu
Figure 2: Our ReTR pipeline comprises several steps: (I). Extracting features through the proposed hybrid extractor model from source views, (2). Processing features in each sample point using the feature fusion block, and (3). Using the occlusion transformer and render transformer aggregate features along the ray and predict colors and depths.
To effectively extract features for our proposed rendering, we further introduce the hybrid extractor in Sec. 3.4. Our loss functions are explained Sec. 3.5
our learning-base based rendering in Sec. 3.3
3.1 Preliminary
Generalizable neural surface reconstruction aims recover the geometry of scene S, which represented by a set
posed input views Py where 1 E R R x3 and Pj E 3 are the j-th view's image and camera parameters, respectively. Existing approaches 9 generate the features of the radiance field from using a neural network Fene. and we formalize the process as:
f". [fir
Fime) Fene. (S),
where Pd E BRXXXxXD is the volume feature with resolution It and 1. E hR is the image feature with dimension D. To decode the color of sampled ray r = (o. d) passing through the scene, where o and d denotes as ray original and direction, the existing approaches g sample N points along the ray from coarse to fine sampling between near and far planes and obtain the location I each sample point:
2 = o t.d, i=
The predicted SDF from features first converts to weights using the conversion funetion [2] denoted as og, The weights are then used to accumulate the re- re-weighted projected colors along the ray using volume rendering. Specifically:
C(r)
- T. (1 exp (-o,)) c, where T, exp
Σ0,
(3)
M C, F Fweight. (f,", (I[(f) a x.))
(4)
j=1
Here the Fareight (.) denotes the module that predicts the weight of each projected color. The f represents the volume feature æ; obtained using trilinear interpolation in f" II(I, x) denotes the operation that projects x onto the corresponding input grid and then extracts the feature at the projected location through bilinear interpolation.
Limitation. The volume rendering, denoted by Eq. 33), greatly simplifies light transport processes, thus introducing key limitations. A complete physical model for light transport categorizes the process into absorption, emission, out-scattering, and in-scattering. Each represents a nonlinear photon-partich particle interaction, with incident photon properties being influenced by both their inherent nature and medium characteristics. However, Eq. (3) condenses these complexities into a single density value, predicted merely based sampled point features, lcading oversimplification

---

of incident photon modeling. Moreover, the color determination method, influenced by a weighted blend of projected colors akin Eq. ( over-relies on input view projected colors, overlooking intricate physical effects. As result, the model requires wider accumulation of projected colors from points near the exact surface, resulting in a "murky" surface appearance, as depicted in Fig.
3.2 Generalized Rendering Function
To overcome the limitations we mentioned in Sec. 3.1 we propose an improved rendering equation that takes into consideration of incident photon modeling and feature-based color decoding. As we revisit Eq. (3) and determine its key components to facilitate our redesign. This differentiable equation consists of three parts: The T, term accounts for the accumulated transmittance and gives the first surface a bias to contribute more to the observed color. The (1 exp (-oz)) term denotes the alpha value of traditional alpha compositing, which represents the transparency of æ; and is
constrained to be non-negative. The color part of Eq. (3) denotes the color of æ;. Based on these analyses, we summarize three key rendering properties that the system must hold:
1. Differentiable. To enable effective learning, the weight function needs to be differentiable to the training network with observed color through pack-propagation.
2. Ocelusion-aware. In line with the bias towards the first surface in the original equation, the weight function needs to be aware that the points close to the first exact surface should have a larger contribution to the final output color than other points.
3. Non-negative. Echoing the non-negativity constraint in the alpha value, the weight of each point also needs to be positive.
Having identified these key properties, quirements. Specifically, we propose a more general form of Eq. 33, which can be formalized as:
now reformulate approach meet these re-
C(r) = (F (E)
(5)
where F; represents the set comprising image feature fimg and volume feature f" in the æ, E R. W (-)) is the weight function that satisfies those three key properties we mentioned above, and and (.) denotes the color function. Specifically, color can then be interpreted as characteristic of each feature point in Therefore, feature at each point can be aggregated in a manner analogous RGB value, and enabling us to deduce the primary feature points. This can be mathematically expressed as:
C(r) = c() Σ W(F
F)).).
1=1
where the C(-) represents the color function that maps the feature into RGB space.
3.3 Reconstruction Transformer
Note that Eq. 3 can be considered as a special form of Eq. (5). With this generalized form, we can reformulate the rendering function to overcome the oversimplification weakness of Eq. (3). Based on Eq. (5). introduce the Reconstruction Transformer (ReTR) that preserves essential rendering properties while incorporating sufficient complexity to implicitly learn and model intricate physical processes. ReTR is composed of Render Transformer and an Occlusion Transformer. The Render Transformer leverages a learnable "meta-ray token" to encapsulate complex render properties, enhancing surface modeling. The Occlusion Transformer utilizes an attention mask to enable the occlusion-aware property. Also, ReTR works in the high-dimensional feature space instead of the color space, and thus allows for more complex and physically accurate light interactions. Consequently, ReTR not only overcomes the restrictions of Eq. (3) but also preserves its fundamental rendering characteristics. We elaborate on the design of these components as follows.
Render Transformer. Here, we discuss the design of the render transformer. Specifically, we introduce a global learnable token, refer to as "meta-ray token" and denote as ftok R to capture and store the complex render properties. For each sample ray, we first use the FeatureFusion block to combine all features associated with each sample point of the ray, resulting in f
5

---

FeatureFusion(F)) RR. We then employ the cross coss-attention mechanism within the Render Transformer to simulate the interaction of sample points along the ray. It can be formalized as follows:
q(ftok (T(E) VD
c(r) =C
Σ softmax
u(f()
where g(), (,,,)(())))) denotes three linear layers and C()) in this formulation an MLP struc- ture to directly regress observed color from the aggregated feature, and W(-) translates to so fomaa g((to))((E()
Furthermore, C(-) is operationalized MLP, which serves to decode the integrated feature into its corresponding RGB value. Then, the hitting probability will be normalized by sof sollaa function, which is Non-negative and encourages the network to learn weight distribu- tion with positive kurtosis. And we can extract the attention map from the Render Transformer and derive the rendered depth as:
VD
N ( = Σ acti where ag = softmox
((fttk))((E)
(8)
VD
where Of,,..., ON denotes the attention map extracted from the Eq. be further used to generate mesh [47] and point cloud [16].
The rendered depth map can
Occlusion Transformer. To further make our system Occlasion-aware and enable of simulation of photon-medium interaction. We introduce Occlusion Transformer. Similar to previous works [46] 48) we introduce an attention mask to achieve that the sample point interacts only with the points in front of it and the meta-ray token. Such unidirectional processes encourage the later points to respond to the preceding surface. This process can be formalized as:
R = (ftok: ff. .......... f2
(9)
R occ =OecTrans (C,TTaan(QQQA0 K, (Q,K, where poCC MLP(MMAA(Q = f/, K, {ptok
Ry),
(10)
+())+(()
Where MHA denotes the multi-head self-attention operation (4) and pare denotes the refine feature of x; which obtained from the render transformer. addition, RS the collective set of tokens, and
Roce signifies the occlusion transformer that employs Rd for cross attention. Then, our Eq. 7 can be rewritten as:
(q(Ptok) )e:
C(r) C
Σ softmaz
u U(P!)
(11)
VD
Continuous Positional Encoding. Following the traditional transformer design, we need to introduce positional encoding make the whole structure positional-aware. However, positional encoding proposed in 42) ignores the aciual distance between each token, which is unsuitable for our situation. Furthermore, weighted-based resampling (26] would lead to misalignment of positional encoding when an increased number of sample points are used.
To solve this problem, extend the traditional positional encoding formula continuous scenarios. Specifically, it can be formulated as follows:
PF 24) sin(8t,/10000
(12)
PE cos(Btt/10000////
Here, i represents the positional encoding in the inh dimension and / is scale hyperparameter we empirically set to 100. The updated formula successfully solves the misalignment of traditional positional encoding, results are shown Tab. The specific proofs will be included in the Appendix section.
3.4 Hybrid Extractor
In our learning- rendering, a finer level of visual feature is necessary, which not achieved by traditional feature extractors that rely high-level features obtained through FPN. These features

---

Original Extmettr
CNN-FR SNN-EPN
3D-I
project
Mean Var
Hybrid Extractor
CNN
3D>Decoder
project
Mein, Vir
Multi-eeo
Multi leve pryj. feature mmage
Mulii leve not fasure voumme
mage
Figure 3: Comparision of the original extractor used in (top) and hybrid extractor (bottom). The original extractor primarily discerns high-level features, demonstrating less efficacy. In contrast, our hybrid extractor excels in integrating multi-level features, demonstrating superior efficacy.
are highly semantic abstract and not suitable for low-level visual feature matching 49 To overcome this limitation, inspired by NeuralRecon [50], we further propose Hybrid Extractor. Rather than relying on FPN to generate one feature map from high-level features of CNN, and using a 3D U-Net process projected features as shown in Fig. 3 leverage all level features from various layers to construct multi-level volume features. Then, we adopt 3D CNN decoder to fuse and decode the multi-level volume features, producing the final global volume features.
Our approach enables us to perceive both low and high-level features, which is crucial for generaliz- able neural surface reconstructions that require detailed surface processing. Second, by avoiding the use of the encoder part the 3D U-Net, we reduce the computational complexity and allow us to build higher resolution volume feature within the same computational budget.
3.5
Loss Functions
Our overall loss function is defined as the weighted sum of two loss terms:
C = Lrendering + a Leepth:
(13)
where Lrendering constrains the observed colors to match the ground truth colors and is formulated as:
s
Crendering
Σ [C(()) C, (r)ll2
(14)
Here, S is the number of sampled rays for training, and Cg (r) represents the ground truth color of the sample ray r. The depth loss Ldepth is defined
Ldepth
1
Σ -
(15)
S
where S is the number of pixels with valid depth and D, (r) the ground truth depth. our experiments, we set = 1.0.
4 Experiments
The DTU dataset 110 is a large-scale indoor multi-view stereo dataset consisting of 124 different scenes captured under 7 different lighting conditions. To train our frameworks, we adopted the same approach as in previous works 9 [. Furthermore, we evaluated our models' generalization capabilities by testing them on three additional datasets: Tanks & Templates [12], ETH3D [13), and BlendedMVS [Л] where additional training performed on the testing datasets.
Datasets.
7

---

Reference
VolRccon ReTR(ours)
Reference
VolRecon ReTR(ours)
scanss
Egunos
scan83
011 Tunne
Figure 4: Sparse view reconstruction testing scenes in the DTU [10] Comparison with VolRe-
con [8] (left), our proposed ReTR (right) renders a more accurate surface and preserves finer details, e.g. the window of the house (scan 24), the of smurf, ReTR produces much sharper details. Best viewed on a screen when zoomed in.
SCAN
37
40
63
65
97
105
106
114
118
COLMAP
[6
1.52
0.90
2.89
1.63
1.08
2.18
1.94
1.61
1.30
2.34
1.28
1.10
1.42
0.76
1.17
1.14
Me SD
1.22
1.05
2.52
1.71
1.04
1.45
1.52
0.88
1.29
1.38
1.05
0.91
0.66
0.61
1.08
1.16
0.91
VolSDF 1
3.41
4.03
4,21
6.12
0.91
8.24
1.73
2.74
1.82
5.14
3.09
2.08
4.81
0.60
3.51
2.18
UNISURF x
4.39
5.08
7.18
3.96
5.30
4.61
2.24
3.94
3.14
5.63
3.40
5.09
6.38
2.98
4.05
2.81
4.49
3.97
1.95
2.50
PixeINeRF
5.13
8.07
5.85
4,40
7.11
4.64
5.68
6.76
9.05
6.11
3.95
5.92
6.26
6.89
6.95
IBRNet
3D
2.32
2.29
3.70
2.66
1.83
3.02
2.83
1.77
2.28
2.73
1.96
1.87
2.13
1.58
2.05
2.09
MVSNcRF 31
2.09
1.96
3.27
2.54
1.93
2.57
2.71
1.82
1.72
2.29
1.75
1.72
1.47
1.29
2.09
2.26
SparseNeu8
9
2.17
3.29
2.74
2.69
2.42
1.58
1.35
1,50
1.45
0.98
1.86
1.87
VolRecon
8
1.38
1.20
2.59
1.56
1.08
1.43
1.92
1.11
1.48
1.42
1.05
1.19
1.38
0.74
1.23
1.27
ReTR (Ours)
1.17
1.05
2.31
1.44
0.98
1.18
1.52
0.88
1.35
1.30
0.87
1.07
0.77
0.59
1.05
1.12
Table Quantitative results sparse view reconstruction on 15 testing scenes of DTU dataset We report the chamfer distance, the lower the better, Methods are split into four categories from top to bottom: a) MVS-based methods, b) Per-scene optimization methods, c) Generalizable rendering methods, and d) Generalizable reconstruction methods. The best scores are in bold and the second best are in underlined.
[0]
Baselines. To demonstrate the effectiveness of our method from various perspectives, we com- pared with (1) SparseNeus (9] and VolRecon (8, the state-of-the-art generalizable neural surface reconstruction method; (2) Generalizable neural rendering methods (3) Neural implicit reconstruc- tion [27 2 3 which require individual training for each scene from scratch. (4) Popular multi-view stereo (MVS) [16, 51 methods. Further details on the baselines are provided in the appendix.
4.1 Sparse View Reconstruction
For comparison, we performed sparse reconstruction using only three views, following the same approach as 8 99. We adopted the same evaluation process and testing split used in previous works ensure fair comparison. We use similar approach as VolRecon [8] to generate mesh, more details can be found in Appendix. As shown in Tab. [ our method outperforms VolRecon [ and SparseNeuS [9] by a significant margin. Moreover, our method also outperforms popular MVS methods such as SVSNet [51]. Furthermore, present the qualitative results of sparse view reconstruction Fig- 4 Our reconstructed geometry exhibits smoother surfaces and less noise compared to the current SoTA methods.
4.2 Depth Map Evaluation Full View Reconstruction
We compare our rendered depth with those generated by SparseNeuS 9 MVSNet [5T] and Vol- Recon [8) Following the experiment settings introduced in VolRecon 8 we also use four source views as input for depth rendering. Additionally, we evaluated the performance by fusing all depth maps into a global point cloud. As shown in Tab. [ our method outperforms existing methods in both evaluations. Moreover, as demonstrated Fig. our method achieves sharper boundary with less noise and fewer holes compared to the current SoTA method [8].

---

VolRecon
ReT R(our))
VolRecon
Re Re
Figure 5: Full view reconstruction visualization in test set of DTU [10], comparison with VolRecon g (left), our proposed ReTR (right) reconstructs better point clouds, egg. fewer holes, the skull head top. and the house roof gives much complete representation, e.g. finer details, the house window, and the skull cheek, provides much finer details. Best viewed on a screen when zoomed in.
Metbod
4
Comp.
Chamler
<2mm
<mmm
MVSNet 51
0.55
0.59
0.57
29.95
52.82
72.33
SparseNenS
0
0.75
0.76
0.76
38.60
56.28
68.63
VolRecon
0.55
0.66
0.60
44.22
65.62
80.19
(Ours)
0.52
45.00
Table 2: Quantitative results of full view reconstruction on 15 testing scenes of DTU dataset [10] For the Accuracy (ACC), Completeness (COMP), and Chamfer Distance, the lower is the better. For depth map evaluation, threshold percentages (<1mm, <2mm, <4mm) are reported in percentage (%). The best scores are in bold.
4.3
Generalization
To evaluate the generalization ability of our model without retraining, we use three datasets, namely Tank & Temples, BlendedMVS, and ETH3D [22 The high-quality reconstruction of large- scale scenes and small objects in different domains, as shown in Fig. 6 demonstrates the effectiveness of our method in terms of generalization capability.
Ablation Study
We conduct ablation studies to examine the effectiveness of each module in design. The ablation study results are reported on sparse view reconstruction in test split following SparseNeuS 9 and VolRecon [
Effectiveness of Modules. We evaluate key components of our approach to generalizable neural surface recon- struction, as shown in Tab. 3 For the evaluation of the occlusion transformer, we keep the original transformer ar- chitecture while removing the special design we proposed in Sec. 3.3] to ensure the training parameter would not affect the evaluation. For the hybrid extractor part, we re- place this module with the original extractor that has been used in [8) Our results demonstrate that our approach can better aggregate features from different levels and use them more effectively. These evaluations highlight the importance of these components in our approach.
Reder
Occ.
Hybrid
Chamfor]
Trans.
Trans.
1.31
1.29
x
1.28
Table 3: Model component ablation. All of these parts are described in Sec. 3.3
Robustness of Different Sampling. Tab. 4 displays the effects of altering the number of sample points on the reconstruction quality of VolRecon and ReTR. Our method surpasses the current SOTA, even when the number of sampling points decreases. These results suggest that existing methods that rely on sampling points of the ray struggle to provide confident predictions of the surface due the nature of volume rendering. Our approach, which uses learning-basee rendering, more resilient to sampling strategies and can provide reliable depth estimations even with fewer samples. Meanwhile, the effectiveness of continuous P.E. in Sec. 3.3 proved through the result.
Unsupervised Neural Surface Reconstruction. Our approach still applicable to unsupervised neural surface reconstruction using only colors for training, which remove Ldepth- Mcanwhile, we find

---

Reference
VolRecon ReTR()
Dog (BlendedMVVS
(ETH3D)
Door
(Tank Temples)
Family
Figure 6: Generalize reconstruction visualization in 3 datasets BlendedMVS, ETH3D, and Tank & Temples I0 13,, comparison with VolRecon (middle), our proposed ReTR (right) generalize well to the large-scale datasets without fine-tuning. ReTR produces finer details and sharper bound- aries. Best viewed on screen when zoomed
Sample Points
ReTR (Ours")
ReTR (Ours)
Method
Training Losses
Chamfer +
(16, 16)
1.60
1.59
1.30
SparseNeuS 2]
wio Edepis
W Crrph
4.22
1.96
(32, 82)
1.45
1.42
1,20
(64,
1.45
1,26
1.22
Waeghh wio Larpis
1.38
(128, 0)
1,79
1.46
1,20
VolRecon g
2.06
(64,
1.38
W w/o Eaay
1.17
Table 4: Number of sampling ablation. The
ReTR
1.45
denotes our model using traditional positional encoding.
Table 5: Ablation study of training losses.
that method significantly outperforms the current SoTA method under unsupervised situations and is even comparable to COLMAP [16] the popular MVS technique, As shown in Tab. [ This is further evidence that the improvement of complex rendering systems for implicit reconstruction is huge.
Conclusion
We have proposed Reconstruction Transformer (RcTR), novel framework for generalizable neural surface reconstruction that uses transformers model complex rendering processes. ReTRR represents a significant advancement in the field of surface reconstruction, offering a powerful solution the challenges faced by neural implicit reconstruction methods. Additionally, we delve into the design procedure of learning- based rendering. This exploration broadens our understanding of enhancing complex rendering systems and sets the stage for future research endeavors not only in surface reconstruction but also other tasks relative to differentiable rendering.
Acknowledgements
We thank Shuai Yang and Wenhang Ge for the thoughtful review of our manuscript and valuable discussions throughout this project. Thank you to Yukang Chen, Shuhan Zhong and Jierun Chen for ideas and feedbacks our manuscript. We would also like thank the Turing Al Computing Cloud (TACC) (52) and HKUST iSING Lab for providing us with computation resources on their platform.
10

---

References
[1] L. Yariv, J. Gu, Kasten, and Y. Lipman, "Volume rendering of neural implicit surfaces," Advances in Neural Information Processing Systems, vol. 34, pp. 4805 4815, 2021.
[2] Wang, Liu, Y. Liu, Theobalt, TT Komura, and W. Wang, "Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction," arXiv preprint arXiv:2106.1068 2021.
[3] M. Oechsle, S. Peng, and A. Geiger. "Unisurf: Unifying neural implicit surfaces and radiance fields for multi-view reconstruction." " in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 5589-5599.
[4] M. Niemeyer, I.. Mescheder, M. Oechsle, and A. Geiger, "Differentiable volumetric rendering: Learning implicit 3d representations without 3d supervision." in Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 3504-3515.
[5] Darmon, B. Bascle, J.-C. Devaux, Monasse, and M. Aubry, "Improving neural implicit surfaces geometry with patch warping," in Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 6260-6269.
[6] Y. Wang, I. Skorokhodov, and P. Wonka, "Hf-neus: Improved surface reconstruction using high- frequency details," Advances Neural Information Processing Systems, vol. 35, pp. 1966-1978, 2022.
[7] J. Park, Florence, Straub, R. Newcombe, and Lovegrove, "Deepsdf: Learning continuous signed distance functions for shape representation," in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp. 165-174.
[8] Y. Ren, F. Wang. T. Zhang, M. Pollefeys, and Süsstrunk, "Volrecon: Volume rendering of signed ray distance functions for generalizable multi-view reconstruction," arXiv preprint a2
2022.
[9] X. Long, C. Lin, P. Wang, T. Komura, and W. Wang, "Sparseneus: Fast generalizable neural surface reconstruction from sparse views." in Proceedings of the European conference on computer vision (ECCV), 2022, pp. 210-227.
[10] H. Aanæs, R. R. Jensen, G. Vogiatzis. Tola, and A. B. Dahl, "Large-scale data for multiple- view stereopsis," International Journal of Computer Vision, vol. 120, pp. 153-168, 2016.
[11] Y. Yao, 7. Luo, S. Li, J. Zhang, Y. Ren, L.. Zhou, T. Fang, and L. Quan, "Blendedmvs: A large-scale dataset for generalized multi-view stereo networks." in Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition, 2020, pp. 1790-17999
[12] A. Knapitsch, Park, Q.-Y. Zhou, and Koltun, "Tanks and temples: Benchmarking large-scale scene reconstruction." ACM Transactions on Graphics, vol. 36, 4, 2017.
[13] Schops, L. Schonberger, S. Galliani, T. Sattler, K. Schindler, M. Pollefeys, and A. Geiger, "A multi-view stereo benchmark with high-resolution images and multi-camera videos." in Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, 2017, pp- 3260-3269.
[14] N. D. Campbell, G. Vogiatzis, C. Hernández, and R. Cipolla, "Using multiple hypotheses improve depth-maps for multi-view stereo," in Proceedings of the European conference on computer vision (ECCV), 2008, pp. 766-779.
[15] D. Chang, A. Božič, T. Zhang, Q. Yan, Y. Chen, S. Süsstrunk, and M. NieBner, "Re-mvsnet: un- supervised multi-view stereo with neural rendering," in Proceedings of the European conference on computer vision (ECCV), 2022, pp. 665-680.
[16] J. L. Schönberger, E. Zheng, J.-M. Frahm, and M. Pollefeys, "Pixelwise view selection for unstructured multi-view stereo," in Proceedings of the European conference on computer vision (ECCV), 2016, pp. 501-518.

---

[17] X. Gu, Z. Fan, S. Zhu, Z. Dai. F. Tan, and P. Tan, "Cascade cost volume for view stereo and stereo matching," in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 2495-2504
multi-
[18] J. Yang. W. Mao, J. M. Alvarez, and M. Liu, "Cost volume pyramid based depth inference for vulti-view stereo," " in Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, 2020, pp. 4877-4886.
[19] K. Luo, T. Guan, L. Ju, H. Huang, and Y. Luo, "P-mvsnet: Learning patch-wise matching confidence aggregation for multi-view stereo," in Proceedings the IEEE/CVF International Conference Computer Vision, 2019, pp. 10452-10461
[20] K. N. Kutulakos and S. M. Seitz, "A theory of shape by space carving." International journal of computer vision, vol. 38, pp. 199-218, 2000.
[21] S. M. Seitz and C. R. Dyer, "Photorealistic scene reconstruction by voxel coloring," International journal of computer vision, vol. 35, pp. 151-173, 1999.
[22] Y. Yao, Z. Luo, S. Li, T. Shen, Fang, and L. Quan, "Recurrent mysnet for high-resolution miewi-view stereo depth inference," in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp. 5534-5534.
[23] Y. Furukawa and J. Ponce, "Accurate, dense, and robust multiview stereopsis," " IEEE transactions on puttern analysis and machine intelligence, vol. 32, no. 8, pp. 1362-1376, 2009.
[24] M. Lhuillier and L. Quan, "A quasi-dense approach to surface reconstruction from uncalibrated images," " IEEE transactions on pattern analysis and machine intelligence, vol. 27, no. 3, pp. 418 08--433,2200 2005.
[25] R. Chen, S. Han, J. Xu, and H. Su, "Point-based multi-view stereo network," in Proceedings of the IEEE/CVF international conference on computer vision, 2019, pp. 1538-1547.
[26] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng. "Nerf: Representing scenes neural radiance fields for view synthesis," Communications of the ACM, vol. 65, no. 1, pp. 99-106, 2021.
[27] L. Yariv, Y. Kasten, D. Moran, M. Galun, M. Atzmon, B. Ronen, and Y. Lipman, "Multiview neural surface reconstruction by disentangling geometry and appearance," Advances Neural Information Processing Systems, vol. 33, pp. 2492-2502, 2020.
[28] J. Zhang, Y. Yao, and L. Quan, "Learning signed distance field for multi-view surface recon- struction," " in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 6525-6534.
[29] W. Ge, T. Hu, H. Zhao, S. Liu, and Y.-C. Chen, "Ref-neus: Ambiguity-reduced neural implicit surface learning for multi-view reconstruction with reflection," " arXiv preprint arXiv:2303. 10840, 2023.
[30] Q Wang, Z. Wang. K. Genova, P. P. Srinivasan, H. Zhou, J. T. Barron, R. Martin-Brualla, N. Snavely, and T. Funkhouser, "Ibrnet: Learning multi-view image-based rendering," in Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2021, pp. 4690-4699.
[31] A. Chen, Z. Xu, F. Zhao, X. Zhang, F. Xiang, J. Yu, and H. Su, "Mvsnerf: Fast generaliz- able radiance field reconstruction from multi-view stereo," in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 14 124-14 133.
[32] A. Yu, V. Ye, M. Tancik, and A. Kanazawa, "pixelnerf: Neural radiance fields from one or few images," " in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 4578 4587.
[33] A. Trevithick and B. Yang, "Grf: Learning a general radiance field for 3d representation and rendering," " in Proceedings of the IEEF/CVF International Conference on Computer Vision, 2021, pp. 15 182-15 192.
12

---

[34] S. Peng, M. Niemeyer, L. Mescheder, M. Pollefeys, and A. Geiger, "Convolutional occupancy networks," in Proceedings of the European conference on computer vision (ECCV), 2020, pp. 523-540.
[35] K. Zhang, F. Luan, Q. Wang, K. Bala, and N. Snavely, "Physg: Inverse rendering with spherical gaussians for physics-based material editing and relighting," Proceedings of the IEEE/CVF Conference Computer Vision and Patlern Recognition, 2021, pp. 5453-5462.
[36] X. Zhang, P. P. Srinivasan, B. Deng, P. Debevec, W. T. Freeman, and J. T. Barron, "Ner- factor: Neural factorization of shape and reflectance under an unknown illumination," ACM Transactions Graphics (TOG), vol. 40, no. 6, pp. 1-18, 2021.
[37] J. Knodt, J. Bartusek, S.-H. Back, and F. Heide, "Neural ray-tracing: Learning surfaces and reflectance for relighting and view synthesis," arXiv preprint arXiv:2 104. 13562, 2021.
[38] V. Sitzmann, J. Thies, F. Heide, M. Nießner, G. Wetzstein, and M. Zollhofer. "Deepvoxels: Learning persistent 3d feature embeddings." in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 2437-2446.
[39] V. Sitzmann, S. Rezchikov. B. Freeman, J. Tenenbaum, and F. Durand, "Light field networks: Neural scene representations with single-evaluation rendering," Advances in Neural Information Processing Systems, vol. 34, pp. 19 1133--1933 325,, 2021.
[40] M. S. Sajjadi, H. Meyer, E. Pot, U. Bergmann, K. Greff, N. Radwan, S. Vora, M. Lučić, D. Duckworth, A. Dosovitskiy el al., "Scene representation transformer: Geometry-free novel view synthesis through set-latent scene representations," in Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 2229-6238.
[41] V. Sitzmann, M. Zollhöfer, and G. Wetzstein, "Scene representation networks: Continuous 3d-structure-aware neural scene representations." Advances in Neural Information Processing Systems, vol. 32, 2019.
[42] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, "Attention all you need," Advances in neural information processing systems, vol. 30, 2017.
[43] J. Reizenstein, R. Shapovalov, P. Henzler, L. Sbordone, P. Labatut, and D. Novotny, "Common objects in 3d: Large-scale learning and evaluation of real-life 3d category reconstruction." " in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 10 10901-10911.
[44] Y. Liu, S. Peng, L. Liu, Q. Wang, Wang. C. Theobal,, X. Zhou, and W. Wang, "Neural rays for occlusion-aware image-based rendering," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 7824-7833.
[45] M. Suhail, C. Esteves, L. Sigal, and A. Makadia, "Generalizable patch- neural rendering." " in Proceedings of the European conference computer vision (ECCV), 2022, pp. 156-174.
[46] P. Wang, X. Chen, T. Chen, S. Venugopalan, Z. Wang al., "Is attention all nerf needs?" arXiv preprint arXiv:2207.13229, 2022.
[47] B. Curless and M. Levoy, "A volumetric method for building complex models from range images," " in Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, 1996, pp. 303-312.
[48] Z. J. Tang, T.-J Cham, and H. Zhao, "Able-nerf: Attention-based rendering with learnable embeddings for neural radiance field," arXiv preprint arXiv:2303. 13817, 2023.
[49] C. Cao, X. Ren, and Y. Fu, "Mvsformer: Learning robust image representations via transformers and temperature-based depth for multi-view stereo," arXiv preprint arxivv2208.02541, 2022.
[50] J. Sun, Y. Xie, I. Chen, X. Zhou, and H. Bao, "Neuralrecon: Real-time coherent 3d reconstruc- tion from monocular video," in Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition, 2021, pp. 15 1555-996690 607.
13

---

[51] Y. Yao, Z. Luo, S. Li, T. Fang, and L. Quan, "Mvsnet: Depth inference for unstructured multi- view stereo," " in Proceedings of the European conference on computer vision (ECCV), 2018, pp. 767-783.
[52] K. Xu, X. Wan, H. Wang, Z. Ren, X. Liao, D. Sun, C. Zeng, and K. Chen, "Tace: A full-stack cloud computing infrastructure for machine learning tasks," arXiv preprint arXiv:2110.01550 2021. [Online]. Available: https//arrxv.orrrab//1111011111
14

---

Appendix
implying A Implementation Details
A.1 Experimental Environment
Software and hardware environment:
CUDA version: 11.1
cuDNN version: 8.0.5
PyTorch version: 1.10.1
GPU: Nvidia RTX 3090
CPU: Intel Xeon Platinum 8180 2.50 GHz 2
A.2 Training Detail
Our model is implemented in PyTTorch using the PyTorch Lightning framework. During the training stage, we resize the input image 640 512 and the source views N 4. To train our model, we employ the Adam optimizer on a single Nvidia 3090 GPU. Initially, the learning rate is set to 10-4 and gradually decays 10 using cosine learning rate scheduler. Throughout the training, we use batch size of 2 and set the number of rays to 1024. To enhance the sampling strategy, we apply coarse-to- carss-to--fie approach with both Meourse and N fine set 64. The Neearse points are uniformly sampled between the near and far plane, while the fine points are sampled using importance sampling based on the coarse probability estimation. Regarding the global feature volume f", we set its resolution to K=128. For inference on DTU, the image resolution is set to 800 600. For datasets such as BlendedMVS [1], ETH3D [3), and Tanks Temples [2] we maintain the original image resolution. Training our model requires approximately 3 days on a single Nvidia 3090 GPU. Moreover, when constructing larger models such as the large and xlarge models by stacking more layers, the training time will naturally increase due to the increased model size.
A.3 Mesh and Point Cloud Generation
Following the settings employed VolRecon we generate depth maps from virtual viewpoints by shifting the original camera along its z-axis by d = 25 mm. Subsequently, we perform TSDF fusion and applied the Marching Cubes algorithm merge all the rendered depths into a voxel grid with resolution of 1.5 mm and extract mesh representation. For point cloud generation, we initially generate 49 depth maps by leveraging the four nearest source views. These 49 depth maps are then fused together to form unified point cloud.
B Technical Details and Discussion
B.1 Discussion of Hitting Probability
The attention score in ReTR can be interpreted as the probability of a ray being hit. However, when using soft- max, the attention scores for each ray are forced to sum up that every ray should be considered a hit. To gain further insights, exam- ine the distribution of attention scores for rays that are not hitring. Figure illustrates the results, demonstrating that the transformer intelligently em- ploys& wider distribution to model rays that do not hit. The underlying
Render image
Hitting probsbility
Figure 7: Hitting probability comparison.
rationale is that the transformer treats the surrounding air medium that contributes to the color.
15

---

Sumple Pornts
24
scan-0
$5
scan
66
97
105
106
scan
110
114
118
132
16-15
1.5M
2.49
1.31
1.00
1.6
1.04
0.37
1.35
1.45
1.05
1:21
0.77
0772
1:37
1.28
32-32
1.06
2,30
1.66
0.97
1.35
1.53
0.39
1.38
1.34
0:92
1.30
0.74
0.50
1.30
1,37
64+0
1.11
2,39
143
1.06
1.36
1,02
0.4
1.28
131
0:91
1.12
0.78
0.64
1.38
1.20
12840
1.54
1.0
Table 6: Chamfer distance of a number of different sampling points, results are shown for each scan under different settings.
When a ray does not hit, the transformer aggregates information from the surrounding air to obtain the color from these mediums.
B.2 Hierarchical Volume Sampling through Attention Map
Given that our framework does not incorporate weights as seen in traditional frameworks like NeRF or NueS, we refine the original hierarchical sampling strategy by substituting the weights with the attention scores of each point. This approach, as discussed in the main text, is both straightforward and impactful. Additionally, we highlight that our method exhibits greater robustness in terms of the number of sampling points compared to the current state-of-the-an techniques, thereby offering an additional advantage within our framework.
B.3 Continous Positional Encoding Proof
To imbue our system with positional awareness of actual distance. we initially derive the formula for the attention score, denotes as of features in æ; and æy;
8 = (E/ + P:) WT (rf +P,)",
(16)
where p represents the positional encoding in æ. We subsequently expand Eq. (6 as follows:
s = (f. W.. (T))T= + (f W. (p,)_ + (p.)W,W, (fy)T +
(P,)
(17)
where the fourth component of (q.(17) denotes the interaction between two locations, and W, W represents the trainable parameters. To ensure our MLP actual positional awareness, we need to make the function satisfy the following conditions:
(p:)(,)) f(ty t4),
(18)
where i denotes the distance between æ; and æz.. When we apply our positional encoding (PE), the fourth component of E7. can be simplified as:
(p;)
[sin( (sin(Bt/110000)//)). cos Bt. 100000
(19)
x [sin( (sim(8t/1100002//)))cooss B
100000 /0000//000)00/////100/001/1////
(p:) = sin sim(Bt//100000//))ssnnn (in 100002//D) (4000
(20)
(oo 100000///)
By applying the sum-to-product identities, we obtain:
(p:)(,,) cos( f.) 20/D)
(21)
where (t, t.) represents the actual distance between æ; and æj. Thus, continuous positional encoding enables the attainment of actual positional awareness.
Additional Experimental Results C
Here show additional experiment results:
C.1 Visualization Supplementary
Due to space limitations, we provide additional visual results for the experiments in this section. Specifically, we present the results for sparse view reconstruction in Fig. aand full view recon- struction of the point cloud in Fig. Furthermore, we include the per-scene results for the number of sampling points in Tab.
16

---

Models
Mess
scan 34
S280 57
5can.55
5ca0 63
scan 63
65
scan
scan 57
scan
105
s00
106
scan 130
114
scan 118
sco 122
ReTR-B
1.17
1,25
2.31
1.44
0.98
1.18
1.52
6.88
1.35
1.30
0.87
1.07
0.77
0.99
1,55
1.12
RoTK-
1.66
0:96
2.36
1.59
1.00
1.18
1.50
0.00
1.35
1.26
0.56
1.00
0.28
0.57
101
1,00
BeTR-XI
0:96
1:39
1.28
140
0.55
0.55
Table 7: Result of ReTR- Basc, ReTR-Large and ReTR-XLarge evaluated on DTU under 3 views
setting. We report chamfer distance, the lower the better.
Method
PSNR1
MSEJ
SSIM
LPIPS|.
MVSNcRF*
25.92
0.003
0.89
0.19
VolRccon*
23.37
0.004
0.80
0.30
ReTR-B
25.88
0,004
0.83
0.28
ReTR-L
26.03
0,003
0.84
0.27
ReTR-XL 26.33
0.003
0.84
0.27
Table 8: Novel View synthesis result on DTU,
denotes our reproduced result.
C.2 Error Bar of ReTR
In order to assess the reproducibility and robustness of our model, we conduct three separate training runs using different random seeds. The corresponding results presented in Figure These results demonstrate that our model exhibits consistent performance across multiple training runs, indicating good reproducibility. Moreover, the minimal variance observes in the results further underscores the robustness of our model.
C.3 Effectiveness of Stacking Transformer Blocks
To explore the potential of simulating more complex light transport effects, we extend our learnable rendering approach by stacking multiple layers of transformer blocks. Specifically, we introduce two variations: ReTR-L, where we stack two transformer blocks, and ReTR-XL, where we stack three transformer blocks. This allows us to experimentally evaluate the effectiveness of more complex rendering system. The results of these experiments are summarized Tab. 7 The findings indicate that by overlaying multiple layers of transformers, we simulate complex lighting effects and achieve more powerful results, This demonstrates the potential of our approach to capture intricate light transport phenomena and enhance the overall rendering capabilities.
C.4 Novel View Synthesis
In order to assess ReTR's performance in Novel View Synthesis, a task where many multi-view stereo techniques struggle, conduct quantitative comparison with VolRecon [8]. The novel views are generated during the full reconstruction for fusing the point clouds as we discussed in the main paper. Our results demonstrate significant improvement over VolRecon in terms of novel view synthesis, as shown in Tab. [
Additionally, we provide visualizations of novel view synthesis and depth synthesis in Figure 10 has been challenging achieve high-quality results simultaneously in rendering canserin-bbased studies and reconstruction- based studies, with few methods excelling in both aspects. However, the results achieved by our proposed framework, ReTR, are highly promising. which suggests that learnable rendering approach based on transformers can effectively integrate both tasks, yielding impressive results on both fronts within a unified framework.
D Limitations
Our method requires approximately 30 seconds render depth map and image with a resolution of 600 x 800. Similar to other rendering-basee based methods such as IBRNet (30), VolRecon R and MVSNeRF [1], our approach has limitations in terms of efficiency. While learning- based rendering offers enhanced capabilities, it does introduce additional training parameters compared to traditional volume rendering techniques. Stacking multiple layers of our model can improve performance: however, it also increases training time due to the larger model size. It is important to strike balance between achieving higher rendering quality and maintaining reasonable computational efficiency. Further research and optimization efforts can be explored to enhance the efficiency of our method,
17

---

Error Bar of Ours (ReTR)
Soand
Soando
Scanis
Scando
Dooe rotd
Slordo
Scrnr
Scan1 18
Ban112
Figure 8: Error bar on 3 runs of Ours (ReTR).
potentially through techniques such as model compression, parallelization, or hardware acceleration. Acknowledging these limitations, we aim to provide a comprehensive understanding of the trade-offs between rendering quality, efficiency, and model complexity within our proposed framework.
E Broader Impacts
The proposed ReTR framework not only enables accurate surface reconstruction through learnable rendering but also generates high-quality novel views. These capabilities open up possibilities for various downstream applications in fields such as virtual reality (VR), robotics, and view synthesis with geometry. While these applications offer numerous benefits, it is important to acknowledge that they also come with ethical considerations. As authors of the ReTTR framework, we are committed to promoting ethical practices and responsible development. We recognize the potential for misuse, such as generating content without consent, and we prioritize fair representation and responsible usage of the technology. We strive to adhere to ethical guidelines and contribute to the development of responsible AI practices. It is crucial to ensure that technological advancements are leveraged the betterment of society while minimizing potential negative impacts. By maintaining a focus on ethics, fairness, and responsible development, aim to ensure that ReTR and its applications are aligned with the principles of responsible Al and contribute positively to the broader scientific community and society whole.
18

---

Reference
VolRecon
ReTRs
Reference
VolRecon
ReTR(ours)
scan
Scmm5s
sean 14
HT s00e
scan 10
Figure 9: Comparison of VolRecon and ReTR in sparse view reconstruction with input views.
View Synthesis
Depth Synthesis
View Synthesis
Depth Synthesis
Figure 10: View synthesis and depth synthesis visualization of proposed ReTR.
19

---

37
40
55
63
69
83
97
106
110
114
118
122
Figure 1I: Visualization of full view generalization of point cloud of our proposed ReTR.
20


Reviews:
[{"review_id": "8nAyyjlFxn", "reviewer": "Reviewer_DVpn", "summary": "This paper introduces a interesting solution for volume renderings in generalizable neural surface reconstruction by leverage Transformers to predict depths and colors from feature volumes. The results on sparse view reconstruction prove its useness.", "questions": "See the weakness above.", "limitations": "See the weakness above.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The authors identify the limitation and derive a general form of volume rendering and propose ReTR, a learning-based rendering framework utilizing transformer architecture to model light transport.\nA hybrid feature extractor is also proposed for achieving better performance. ", "weaknesses": "Can we replace the volume rendering in the optimization-based methods (e.g. NeuS / VolSDF) with the learned Transformer ? Or is the solution only works for generalizable neural surface reconstruction?\n\nWhat is the performance in scene-level sparse view reconstruction, i.e., Replica/ScanNet.\n\nWill HybridExtractor also works for volume rendering based methods (e.g. SparseNeuS / VolRecon) ?\n\nWhy Transformer? Will CNN/MLP also works for this design?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s time and insightful evaluations. \n\n**Q1. If our learned transformer can replace volume rendering for optimization-based methods.**\n\nYes, we've conducted qualitative experiments, as shown in Figure 1 of the rebuttal PDF, using optimization-based methodologies. Rather than traditional methods, we used spatial points and applied our learning-based rendering for scene optimization. Our method notably outperforms NeuS qualitatively (high kurtosis of weights along the ray), hinting that transformers might introduce a new direction in optimization-based reconstruction, similar to NeuS and VolSDF.\n\n**Q2.  Performance in scene-level sparse view reconstruction.**\n\nIn addition to evaluating the DTU and BlendedMVS datasets, we further show the reconstruction results of our ReTR on scene-level datasets, specifically ETH3D and Tanks & Temples, following the same setting as VolRecon. The outcomes of these evaluations are visually represented in $Fig. 6$ of the main paper. More results will be included in the appendix in the next version.\n\n**Q3. Will HybridExtractor also work for volume rendering-based methods?**\n\nYes, the HybridExtractor (HE) is adept at extracting both low-level to high-level features. Additionally, it potentially diminishes computational complexity by circumventing the use of the encoder segment of the 3D U-Net. To further substantiate our proposition, we integrated our HE into the VolRecon, resulting in a notable enhancement in performance, achieving a mean cd of 1.35.\n\n**Q4. Why Transformer?**\n\nTransformers have been extensively validated across diverse tasks owing to their exceptional capability in handling sequential feature interactions. The light transport effect can analogously be construed as the interaction between a photon and a particle. As we model the light interaction as rays, the transformer becomes a nature choice as it can model the sequence of points along rays effectively. In contrast, CNNs and MLPs appear to demand more tailored designs. Thus, we opted for the transformer to implicitly model the light transport effect. Nonetheless, we deeply appreciate the reviewer for highlighting this matter, suggesting an avenue worthy of future exploration.\n\n\n"}, {"review_id": "DX0C3OYKFk", "reviewer": "Reviewer_TPgh", "summary": "This works focus on generalizable asset reconstruction: given a few posed images, predict the 3D representations using a network.\nInstead of using volume rendering to compute the transmittance, the authors propose to use transformer on the sampled points to compute the weight of each point.\nBesides, the author also  improve the CNN architecture for feature extraction.\nExtensive experiments are conducted on multiple datasets, demonstrating better performance.\n\n", "questions": "Please check the weakness section. Besides:\n\n1. This may not be the weakness of this paper, but it's common in the research of this topic.  When it's called \"generalizable\", why it can not generalize to the unseen region of the object? For example, given images of the front views of the statue, why not generalize it to the backview in reconstruction. \n\n2. What's the PSNR compared to other baselines", "limitations": "Discussion of limitation and broad impacts in the supplementary.\nNo license/asset description according to https://neurips.cc/public/guides/PaperChecklist . But I don't penalize it in the rating.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Better performance compared to previous compared with previous SOTA methods\n2. Code is attached and will be release.\n3. method is well explained\n", "weaknesses": "1. Compared to previous methods SparseNeus and Volrecon, this work seems somewhat incremental. Major difference is using a hybrid CNN extractor (Fig 3) and a new transformer architecture. The \"occlusion transformer\" and \"render transformer\" seems to be two transformers with fancy names, and I don't see significant difference from the transformer in volrecon. Though Volrecon is a CVPR2023 paper, but apparently the authors use its codebase for develop , as can be seen in the code.zip in the supplementary.\n\n2. I don't agree with the \"rethink\" title, equation5 doesn't make too much sense to me. It's not explained why the equation can satisfy the occlusion-aware, and especially no guarantee of consistency across multi views.\nFor example, for a sampled point $x$, its weight may  be $1$ for the ray $r_1$, but may be $0$ for the ray $C(r_2)$ even when $x$ is the nearest point in ray $r_2$. Therefore, I'm not fully convinced with that Eq 5 is a better modelling of the  rendering, and the \"rethinking\" seems be some sort of exaggeration.\nFurthermore, given the goal of this work is reconstruction, I don't see why loosing the physics constrain in rendering can benefit the learned geometry.\n\n## Justification of rating.\n1. Pros: results are solid (multiple datasets, compared with SOTA baselines), code available.\n2. Cons: Somewhat incremental, \"rethinking\" seems a bit exaggeration.\n\nOverall I lean to a borderline accept, as no big technical flaws, but I'm not very confident.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s time and insightful evaluations.\n\n**Q1. Incremental work of SparseNeus and Volrecon.**\n\nWe respectively disagree with this. Our main contribution is to address the intrinsic limitation of the extensively-used volume rendering (line 25-37, $Fig. 1$ in the main paper), pointing out that it is an oversimplification of incident photon modeling (line 137-138), and it over-relies on input view projected colors (line 139-140). This is largely overlooked in previous works, including SparseNeuS and Volrecon. Then, we show that these limitations can be overcome by generalizing the volume rendering to the reconstruction transformer, which allows the modeling of complicated photon properties in the feature space. In contrast, VolRecon and SparseNeuS still rely on volume rendering that condenses complicated photon-particle interaction into a single density value. We also show that our approach significantly leads to more confident and accurate surface prediction both qualitatively ($Fig. 1$, $(c)$ in the main paper) and quantitatively ($Tab. 1$ in the main paper). \n\n**Q2.1 “Rethink” seems soft of exaggeration, and why loosing physics constrain benefits learned geometry.**  \n\nWe appreciate the feedback. The term \"Rethink\" in our context reflects a process of reevaluation and reconsideration of volume rendering. We observed the oversimplification issue in the current widely used rendering pipelines and were prompted to seek an alternative approach that is more suitable for the reconstruction pipelines. \n\n**Q2.2 “Why the equation can satisfy occlusion awareness and multi-view consistency.**\n\nThe weighting term in $Eq. 5$ (line 160-161) can be further reformulated to $Eq. 10$ (line 196-197), as described in our main paper. The occlusion-aware process is done by the occlusion transformer that only allows the later points to interact with the points in front of it and the meta-ray token. Specifically, we applied an attention mask that masked out the top part above the diagonal. This is to make sure that for a given point $x$, it can only interact with the points in front of it in order to encourage the points to respond to the preceding surface. The multi-view consistency is done by conditioning the view features, similar to the previous MVS-based methods, such as VolRecon.\n\n**Q3. Why it can not generalize to the unseen region of the object?**\n\nTypically “generalizable” is defined as the ability of the model to predict an unseen scene as we follow this setting from previous works of literature (SparseNeus, Volrecon), but this is an interesting proposal. Traditionally, we approach this task from a **perceptual** standpoint. In this light, we solely generate surface geometry from three views, rigorously constraining the result with the ground truth, eliminating any element of randomness. However, reconceptualizing this task from a generative vantage point is intriguing. This would enable the network to \"imagine\" aspects previously unseen or uncharted. We believe this offers a promising avenue for exploration.\n\n**Q4. PSNR comparison.**\n\nThe goal of this work is to accurately reconstruct the surface geometry given sparse input views. Like many previous studies, we use the chamfer distance to measure the accuracy of our reconstructed meshes. Although our main focus isn't on novel view synthesis, we've included this aspect in the Appendix for a comprehensive review. Please refer to the $Tab. 3$ (line 75-76) in the Appendix.C for detailed results. It's worth noting that our method outperforms VolRecon in the novel view synthesis. For additional qualitative results, please see the rebuttal PDF $Fig. 3$. \n"}, {"review_id": "lLCS2eRBpz", "reviewer": "Reviewer_RozF", "summary": "The paper proposes a new framework for generalizable neural surface reconstruction, which utilizes the mechanism of transformers to model the rendering process. The authors first derive a general form of generalizable volume rendering based on existing methods and identify its limitations. They then suggest improving upon this framework by introducing a new rendering approach based on learned attention maps and performing over-ray accumulation in feature space rather than color space. Experiments are conducted on four different datasets and compared to recent baselines, demonstrating superior performance in generalizable reconstruction.", "questions": "- When evaluating the SparseNeuS baseline, did the authors incorporate depth supervision as well? My understanding is that both ReTR and VolRecon utilize depth supervision during training, while SparseNeuS does not. This raises concerns about the fairness of the comparison between these methods.\n- In Section 3.2, the paper suggests key rendering properties that the system should possess. However, there is no specific mention of the requirement for the weights to sum up to 1, indicating that all rays are eventually occluded. While this property is not explicitly described, it seems to be employed later in the paper using softmax. It would be helpful if the authors provided further clarification on this matter.", "limitations": "The authors have discussed limitations in the supplementary material. However, it is necessary to present the main limitation of efficiency in the main paper, even if briefly in the conclusion section. Additionally, providing quantitative results that demonstrate the tradeoff between the number of parameters and training/rendering time would greatly benefit the presentation of the method.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-written and exhibits a smooth flow. The authors effectively convey the motivation behind their methods, providing necessary background information and comparing against recent baselines. Additionally, the authors ensure that the reader can easily follow the logical progression of the paper.\n- Section 3.1 presents a general framework that serves as a well grounded basis for existing methods. The authors' identification of limitations within this framework offers valuable observations, contributing to the overall storyline and motivation of the paper.\n- The results presented in both the main paper and supplementary material demonstrate the superiority of the proposed method compared to the VolRecon baseline.\n- The authors provided an ablation study on the various components of the method, effectively differentiating their individual contributions and providing a solid understanding of their impact.", "weaknesses": "- The section describing the reconstruction transformer appears to be incomplete and confusing due to several missing details and explanations:\n\n  - It is unclear how equation 6 (and its improvement in equation 10) fit into the general framework proposed in equation 5. This confusion arises because the final MLP from feature to color does not align with the framework. Additionally, the weight function and color function are not explicitly provided.\n  - The FeatureFusion operation is not defined, leaving ambiguity in understanding its purpose and implementation.\n  - The definition of the \"meta-ray token\" $f^{tok}$ for a scene is unclear, specifically whether it pertains to per-image (per-ray/pixel) or per-scene information, and how it differs from the image features $f^{img}$.\n  - The meaning of $F_i$ in line 161 is not provided or explained.\n\n  It is crucial for the authors to address these misunderstandings and revisit the missing definitions in order to clarify the concepts.\n\n- While it is acknowledged that the general form presented in section 3.1 oversimplifies the modeling of light transport in 3D scenes, it is hard to perceive how the mechanism of cross-attention over ray samples effectively models complex photon-particle interactions. Real interactions typically occur in spatial domains, whereas the suggested approach focuses on interactions over ray samples. It is suggested that the authors either temper these claims or provide further explanation on how their framework accounts for intricate global physical effects that encompass both global and local physical effects.\n\n- The qualitative comparison is somewhat limited as it only includes a comparison to VolRecon. It is essential for the authors to provide visual comparisons with other methods as well, particularly SparseNeuS, to offer a more comprehensive evaluation.\n\n- The authors have not presented timing evaluations of their method in both training and evaluation scenarios. Given that the limitation section highlights timing as a significant drawback of generalizable methods, it is important for the authors to address this by providing timing evaluations to enhance the paper's completeness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s time and insightful evaluations.\n\n**Q1. Reconstruction’s description appears incomplete and confusing.**\n\n**1.1. how does equation 6 (and its improvement in equation 10) fit into the general framework proposed in equation 5?**\n\nThanks for the feedback. Since our rendering operates within the feature space, $Eq. 6$ naturally deviates slightly from $Eq. 5$. In the revised version, we elucidate these distinctions to ensure that readers can seamlessly navigate the transformations between the equations. Specifically, features will be aggregated using cross-attention (which can be analogized to the weight function in $Eq. 5$, with the attention map representing the weights). Subsequently, these aggregated features will be employed to predict RGB values (analogous to the color function in $Eq. 5$).\n\nColor can be interpreted as a characteristic of each feature point. In $Eq. 5$, we propose that the feature at each point can be aggregated in a manner analogous to RGB, enabling us to deduce the primary feature points. This can be mathematically expressed as:\n\n$C(\\mathbf{r})=\\mathcal{C}\\left(\\sum_{i=1}^{N}  \\mathcal{W}\\left({F}_1, \\dots,{F}_i\\right) {F}_i\\right)$,\n\nWhere the $\\mathcal{C}\\left( \\cdot \\right)$ represents the color function that maps the feature into RGB space. \n\nBuilding on this, we adapt the render transformer formulation in $Eq. 6$:\n\n$C(\\mathbf{r})=\\mathcal{C}\\left(\\sum_{i=1}^{N} softmax\\left( \\frac{q(\\mathbf{f}^{tok})k(\\mathbf{f}^{f}_i)^\\top}{\\sqrt{D}}\\right)v(\\mathbf{f}^{f}_i)\\right)$,\n\nWhere the $\\mathcal{W}\\left( \\cdot \\right)$ translates to $softmax\\left( \\frac{q(\\mathbf{f}^{tok})k(\\mathbf{f}^{f}_i)^\\top}{\\sqrt{D}}\\right)$. Furthermore, in our approach, $\\mathcal{C}\\left( \\cdot \\right)$ is operationalized as an MLP, which serves to decode the integrated feature into its corresponding RGB value.\n\nWe will incorporate this explanation into the next version.\n\n**1.2. Explanation of FeatureFusion Block.**\n\nThe FeatureFusion Block is designed to merge both the volume feature and projected features at each point into a single integrated feature. This merged feature then undergoes subsequent operations. In detail, the volume and projected features are concatenated and passed through a transformer for refinement. The refined projected features, when combined with the relative direction of each image, undergo a MLP to infer the individual weights of the projected features. The outcome—a weighted sum of the projected features—is then concatenated with the refined volume features. This resultant feature serves as the foundation for the ensuing processes.\n\n**1.3. Explanation of \"Meta-ray\" token.**\n\nThe \"meta-ray token\" acts as a universal token across the network, shared throughout scenes. At the outset of training, this token is initialized and engages in every rendering procedure during training, positioned as the primary token in the input sequence of sample points, akin to the CLS token in ViT. Distinctly, it does not derive from the point features of the input and remains separate from positional coding. As the network evolves continuously, this token gains the capability to encode specific statistical properties intrinsic to rendering.\n\n**1.4. Explanation of $f^{img}$ and $F_i$ in line 161.**\n\n$f^{img}$: This notation represents an intermediary phase of a feature from the input views, as extracted by our hybrid extractor. Later on, the conjunction of $f^{img}$ and $f^{v}$ amalgamate to create $f^f$. We recognize the need for clarity here and will address this in our upcoming version.\n\n$F_i$ in line 161: This notation represents the set comprising $f^{img}$ and $f^{v}$. \n\nWe elucidate the above-mentioned issues in our revised version to alleviate any ambiguities.\n\n**Q2. Real interaction of light occurs in spatial domains… temper the claims.**\n\nWe will tune down the claim the claims about particle interactions.\nOur primary contribution is the identification and rectification of inherent limitations in the widely used volume rendering approach, specifically its oversimplification of incident photon modeling. This issue has been largely unaddressed in prior works. However, modeling real interaction between photon particles face many challenges, such as the high cost of computation.\nWe approximate this problem as interaction over ray samples, as we show in $Sec. 3$. In addition, our approach is much more cost-effective. We hope such observation can pave a new research direction for the community. \n\n**Q3. Qualitative comparison with SparseNeus.**\n\nPlease kindly refer to the rebuttal PDF $Fig. 2$ for a comparative analysis between our proposed ReTR and SparseNeus. It's important to highlight that SparseNeus requires fine-tuning on specific scenes to attain the reported outcomes, whereas our approach involves direct inference on previously unseen scenes. We present results for both the direct inference and the fine-tuned scenarios for SparseNeus. In both instances, ReTR consistently demonstrates superior performance over SparseNeus.\n\n**Q4. Time evaluation.**\n\nNeus: RTX 2080ti, 16 hours per scene training.\n\nSparseNues: Two RTX 2080ti; pretraining takes 3 days and requires 20 mins per scene fine-tuning. \n\nVolRecon: single A100, pretraining takes 3 days; no further finetuning is needed.\n\nReTR (ours): single RTX3090, pretraining takes 3 days; no further finetuning is needed. \nInference takes about 30 seconds to render one image (DTU). \n\n**Q5. Depth supervision.**\n\nIn $Tab. 5$ of the main manuscript, we compare ReTR against various baselines, considering both scenarios: with and without depth supervision. Notably, ReTR consistently surpasses these baselines in both settings.\n\n**Q6. Weight of the ray sum to 1.**\n\nFor a more in-depth exploration of this issue, please refer to Appendix $B.1$.\n\n"}, {"review_id": "cSvf2rYqU7", "reviewer": "Reviewer_feWL", "summary": "This paper proposes ReTR, a new architecture that leverages transformer to replace the traditional volume rendering process. The insight of the paper is that: the traditional volume rendering equation is oversimplified to model photon-particle interaction. Moreover, the color compositing function highly relies on the projected input view colors, and therefore overlooking intricate physical effects. To solve these two limitations, ReTR replaces the volume rendering equation with a render transformer. The attention map can be extracted from the render transformer to synthesize geometry details. An occlusion transformer is further introduced to obtain finer features. Instead of using FPN features and ResUNet, ReTR also utilizes features from different layers to construct multi-scale features. Experiments are conducted on the DTU dataset, Tanks & Temples, ETH3D, and BlendedMVS. The method is compared with state-of-the-art generalizable NeRF methods and generalizable surface reconstruction methods and achieves the best performance among them. Ablation studies also show the effectiveness of the network architecture. Moreover, ReTR surpasses SparseNeuS and VolRecon even without depth supervision.", "questions": "Though the proposed method is effective, mathematically, I did not see how the network architecture can model the geometry better than other neural surface reconstruction methods, such as SparseNeuS and VolRecon -- Especially when the network is trained without depth supervision. I would like to see more insights and explanations from the authors in the feedback.", "limitations": "The main concern of this paper is it requires a long training time, e.g. 3 days on a 3090. It comes with a cost when introducing transformers in the network architecture. Generally, it is not a big problem since the method is generalizable.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "I like the insights proposed in the paper that the traditional volume rendering equation is oversimplified. The solution that utilizes transformers to replace volume rendering is straightforward yet sound and effective. Experiments are exhaustive and validated the network designation. I also like the discussion of interpreting the render transformer the hitting probability.", "weaknesses": "More related work should be discussed in Section 2 for generalizable NeRF methods (NeuRays, CVPR22; Generalizable Patch-Based Neural Rendering, ECCV 2022, ...) and neural surface reconstruction methods (NeuS, ...).\nNotations in Equations (8) and (9) are not well explained, for example, what are $\\mathbf{R}^f$ and $\\mathbf{R}^{\\text{occ}}$, and $\\mathbf{f}_i^{\\text{occ}}$ did not appear before Equation (10).\nThere is also a typo at Line 232: **tanks and temples** instead of **tanks and templates**.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s time and insightful evaluations.\n\n**Q1. More related work should be discussed in Section 2.**\n\nThanks for the feedback. NeuRays(CVPR22) leverages neural networks to model and address occlusions, enhancing the quality and accuracy of image-based rendering. GPBR(ECCV22) employs neural networks to transform and composite patches from source images, enabling versatile and realistic image synthesis across various scenes. NeuS(NeurIPS21) uses SDF value to model density in the volume rendering to learn neural implicit surfaces, offering a robust method for multi-view 3D reconstruction from 2D images.  In the next version, we will discuss more relative works and incorporate all the suggested discussions into Section 2.\n\n**Q2. What are $R^f$ and $R^{occ}$, and $f^{occ}_{i}$ did not appear before Equation (10).**\n\n$R^f$ denotes the collective set of tokens. Specifically, $f^{tok}$ represents the \"meta-ray token\", employed to capture the global representation (as delineated in line 175). The remaining components, represented by $f_N$, describe the point features distributed along the ray.\n\n$R^{occ}$, on the other hand, signifies the occlusion transformer, which employs $R^f$ for cross attention.\n\n$f^{occ}_{i}$ denotes the refined features of $x_i$ which obtained from the render transformer. \n\nWe recognize the importance of clarity and precision and will ensure that these notations are further elucidated and any typographical errors rectified in our forthcoming version.\n\n**Q3. How the network architecture can model the geometry better ... Especially when the network is trained without depth supervision.**\n\nTraditional volume rendering determines each point's hit probability based on its inherent features, with weights for each point then calculated via a set Cumulative Distribution Function (CDF). In this methodology, the impact of all prior points on the current point's weight is merely reflected in their cumulative probability.\n\nIn contrast, our ReTR approach introduces a \"meta-ray token\" that serves as a global token, assimilating all features within a ray through cross-attention. This is similar to the **CLS_TOKEN** in **ViT**, wherein the global token will learn the statistical properties of rendering. The process then becomes about more than just individual point features; it's also influenced by the overarching information within the \"meta-ray token,\" making for a richer information pool.\n\nFurthermore, our model trains using a rendering loss and is even effective **without** depth supervision. The softmax operation within our framework encourages the model to learn a specific peak – typically the surface – making it the primary influence on the rendered color. We conducted an experiment substituting Neus's rendering with ours (refer to the rebuttal PDF $Fig. 1$). The result indicated that our rendering approach guides the network towards a superior weight distribution (high kurtosis) along the ray, even when applying to per-scene optimization methods that do not require depth supervision, further showing the effectiveness of our proposed rendering approach. Traditional volume rendering can, on the other hand, produce flawed outcomes by identifying areas of low hit probability. Such outcomes are detrimental to accurate surface reconstruction, as we've illustrated in $Fig. 1$ of our main paper.\n\n\n"}, {"review_id": "EzS13n9gKO", "reviewer": "Reviewer_1Qeo", "summary": "The paper presents a learning-based framework on the well-studied neural surface reconstruction problem. The key contribution of this paper is to take the complex photon-particle interaction into account and present a more generalized pipeline rather than relying on volume rendering. The proposed framework released the power of the transformer to achieve enhanced feature representation of sampled points along the ray. Experiments on several popular benchmarks have shown the effectiveness of the proposed approach. ", "questions": "Overall I think the idea of the paper is novel and technically sound, presenting another perspective of the mainstream methods based on volume rendering. Despite some technical concerns listed in weaknesses, now I lean toward accepting this paper. I expect authors to address my concerns by providing more comprehensive experiments to further show the effectiveness of this work.", "limitations": "Authors have discussed their limitations on rendering speed. I think another limitation is that the SOTA neural surface reconstruction method is still only comparable with a classic MVS-based baseline (MVSNet) at this time. But MVSNet and its extensions must be much faster to get a depth map than the rendering-based ones. So there is a long way to go for this area to further release the power of implicit representation.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "(1) The idea of modeling the complex light transport and releasing the flexibility from regular volume rendering is novel and interesting.\n(2) Overall, the paper is well presented and easy to follow.\n(3) The paper has achieved state-of-the-art generalizable neural surface reconstruction performance across different datasets.\n", "weaknesses": "(1) (Major) The idea of using 3D feature volumes with hybrid resolution is not new and has been proposed in NeuralRecon (https://arxiv.org/pdf/2104.00681.pdf). Besides, since there are two major differences (the elimination of FPN and the construction of multi-level projected feature maps) between the proposed hybrid extraction and the original one, it is better to make separate ablation studies to further verify the effect of the two variations.\n(2) (Major) For the ablation study of the occlusion transformer, compared with directly removing this module, a better ablation way is to attend every point’s feature as the input of the key embedding in the self-attention computation. This way ensures a fair setting (roughly same architecture and complexity) and the only difference is whether the later points contribute to the former ones.\n(3) (Minor) Visual comparison on view synthesis with other methods: since one of the main motivations of this work is to model the photon-particle interaction, I assume a major outcome is more robust rendering against the variations (blur, specular..) from input views. Thus, it is better to show some visual comparison with other baselines (SparseNeus, VolRecon, NeRF…) to verify this point.\n(4) (Minor)The main diagram (Figure 2) can be displayed in a clearer and more elegant way. Basically the part of transformer details would belong to either occlusion transformer or render transformer, rather than ‘feature fusion’. Besides, there is ambiguity on what the patches with different colors stand for.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s time and insightful evaluations.\n\n**Q1. Hybrid resolution is not new, and justification of the proposed hybrid extractor with the original one.**\n\nOur main contribution is to rectify the oversimplification in traditional volume rendering by introducing a generalized form that better models incident photons. Yet HybridExtractor (HE) is a technical design that is advantageous in terms of efficiency and performance. We appreciate the reviewer highlighting the multi-level feature maps in NeuralRecon, and we will ensure to cite this paper and integrate discussion about this paper in the 3.4 section.\n\nTo address the reviewer's concerns and showcase the effectiveness of HE in comparison with the baselines, we conducted additional experiments where HE was incorporated into the VolRecon framework. This was in response to the reviewer's observation regarding the \"elimination of FPN and the construction of multi-level projected feature maps\" as potential factors enhancing VolRecon's performance. Our experiments specifically evaluated the efficiency of both multi-level and single-level projected feature maps, resulting in a mean Chamfer distance of 1.35, marginally better than VolRecon's 1.38.\n \nAt the core of HE is the ability to construct multi-level projected feature maps. In an attempt to address concerns about the exclusion of FPN, we developed a HybridExtractor variant that employs FPN for aggregating the features from the last two layers, resulting in a last-level projected feature. This model achieved a mean Chamfer distance of 1.34, indicating that the difference when not using FPN is minimal. However, it's worth noting that introducing FPN not only increases the parameter count but also enlarges the resolution of the last-level projected features, equating it with the second-level features. Our findings suggest that by constructing multi-level projected feature maps, we've essentially mirrored the multi-level feature aggregation function of the FPN, but in a more efficient manner.\n\n**Q2. Ablation study of Occlusion Transformer.**\n\nThank you for the valuable suggestion. To clarify, the experiment was indeed conducted in a similar manner as suggested. Our occlusion transformer utilizes a mask, characterized by a diagonal matrix with its upper triangle masked out, ensuring that the subsequent point can only perceive points ahead of it. This design encourages the latter point to contribute to the object's surface formation. In the ablation experiment highlighted in $Tab. 3$ of the main document, we omitted this mask to observe any shifts in accuracy using chamfer distance as a metric. We acknowledge the confusion arising from the descriptions in $Tab. 3$ and lines 267-268 and will address this in our subsequent edition.\n\n **Q3. Rendering against variations.**\n\nThe goal of this work is to accurately reconstruct the surface geometry given sparse input views. However, ReTR still demonstrates strong performance in view synthesis. Here we show rendering results in rebuttal PDF Figure 3; for the “Train” scene, we can see ReTR is robust to the lighting changes: the background (red region in $Fig. 3$)  and the pavement (yellow region in $Fig. 3$) in front of the train, where VolRecon struggles to predict on such area. In addition, for “Scan 24” in DTU under different lighting, VolRecon again gives incorrect predictions about the front and back of the roof. Such a result shows ReTR is more robust under different settings. \n\n**Q4. To display Fig.2 in a more elegant way.**\n\nWe will incorporate all the suggestions and revise them in the next version.\n"}]

Author rebuttal:
We deeply appreciate the reviewers for their thoughtful feedback and time invested in evaluating our work. We're heartened by Reviewer DVpn's commendation of our solution as "interesting in generalizable neural surface reconstruction" and by Reviewer TPgh's acknowledgment that our "results are solid."  We are pleased that Reviewer Rozf found our paper “well-written” and “presents a general framework serves as basis for existing methods”. Moreover, we are encouraged by Reviewer feWL characterization of our work as "sound and effective", noting the "insights proposed in the paper". And by Reviewer 1Qeo's praise for our “novel and interesting” idea, complimenting that our paper is “well presented and easy to follow”. We will address each of the reviewers' additional comments in our subsequent responses. Thank you again for your invaluable feedback.

Meta-review: The paper proposes reconstruction with a learned image formation model based on Transformers. Previous models used physics-inspired equations instead.

The paper received lukewarm-to-supportive reviews. These were rebutted in detail, and for some resulted in a more positive angle. No agreement was achieved wether or not this new formulation is really beneficial or what the deeper inner logic is. The rebuttal did not really sway the mind of those questioning the idea. Authors also responded to the ACs questions in a plausible way that indicates they are able to change the paper.

Ultimately, the AC thought that the panel agrees that the paper's proposals of a learned image formation model for 3D reconstruction is 1) novel 2) demonstrated to improve the state of play. Most issues we see here come from presentation and language issues (no "photons" here, sorry) that could be fixed.
Decision: Accept (poster)