TransformerFusion: Monocular RGB Scene Reconstruction using Transformers
Authors: Aljaz Bozic
Pablo Palafox
Justus Thies
Angela Dai
Matthias Niessner
Venue: NeurIPS 2021 Poster (2021)
Keywords: 3d reconstruction
3d vision
3d scenes
3d from x

Abstract: We introduce TransformerFusion, a transformer-based 3D scene reconstruction approach. From an input monocular RGB video, the video frames are processed by a transformer network that fuses the observations into a volumetric feature grid representing the scene; this feature grid is then decoded into an implicit 3D scene representation. Key to our approach is the transformer architecture that enables the network to learn to attend to the most relevant image frames for each 3D location in the scene, supervised only by the scene reconstruction task. Features are fused in a coarse-to-fine fashion, storing fine-level features only where needed, requiring lower memory storage and enabling fusion at interactive rates. The feature grid is then decoded to a higher-resolution scene reconstruction, using an MLP-based surface occupancy prediction from interpolated coarse-to-fine 3D features. Our approach results in an accurate surface reconstruction, outperforming state-of-the-art multi-view stereo depth estimation methods, fully-convolutional 3D reconstruction approaches, and approaches using LSTM- or GRU-based recurrent networks for video sequence fusion.

TransformerFusion: Monocular RGB Scene Reconstruction using Transformers
Aljaž Božič
Pablo Palafox
Justus Thies 1.2
Angela Dai
Matthias Nießner
Technical University Munich
Max Planck Institute for Intelligent Systems, Tübingen, Germany al aljazbozic. thub. io/ ttt.rrraarrrrffuuiii/
Abstract
We introduce TransformerFusion, a transformer-based 3D scene reconstruction approach. From input monocular RGB video, the video frames are processed by a transformer network that fuses the observations into a volumetric feature grid representing the scene; this feature grid is then decoded into an implicit 3D scene representation. Key to our approach is the transformer architecture that enables the network to learn attend to the most relevant image frames for each 3D location in the scene, supervised only by the scene reconstruction task. Features are fused coarse-to- to-rse-to-fine fashion, storing fine-level features only where needed, requiring lower memory storage and enabling fusion at interactive rates. The feature grid is then decoded to a higher-resolution scene reconstruction, using an MLP-based surface occupancy prediction from interpolated coarse-to-fine 3D features. Our approach results in an accurate surface reconstruction, outperforming statc-of-the-an multi-view stereo depth estimation methods, fully-convolutional 3D reconstruction approaches, and approaches using LSTM- or GRU-based recurrent networks for video sequence fusion.
Introduction
Monocular 3D reconstruction is core task in 3D computer vision, aiming to reconstruct a complete and accurate 3D geometry of object or environment from only 2D observations captured by an RGB camera. A geometric understanding is key to applications such as robotic or autonomous vehicle navigation interaction, as well model creation and scene editing for augmented and virtual reality. In addition, geometric scene reconstructions form the basis for 3D scene understanding, supporting tasks such 3D object detection, semantic, and instance segmentation [34 35 36 29
43 [5 16)
While state-of-the-art SLAM systems 3 achieve robust and scale-accurate camera tracking leveraging both visual and inertial measurements, dense and complete 3D reconstruction of large- scale environments from monocular video remains a very challenging problem particularly for interactive settings. Simultaneously. notable progress has been made multi- vieti-wiewwae depth estimation, estimating depth from pairs of images by averaging features extracted from the images in a feature Unfortunately, averaging features across full video sequence can lead to eqial-weight treatment of each individual frame, despite some frames possibly containing less information in various regions (e.g., from motion blur. rolling shutter artifacts, very glancing or partial views of objects), making high-fidelity scene reconstruction challenging.
cost volume [42 19 38
Inspired by the recent advances in natural language processing (NLP) that leverage transformer-base we propose a transformer- method that fuses sequence of RGB input frames into 3D representation of scene at interactive rates. Key to
models for sequence to sequence modelling 40
11 2
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

TIME
VIEW ONLINE INPUT ATTENTION RECONSTRUCTTO
frame
framee k-I
frame
frame k+3
frame k+4
frame A+5
frame A+8
frame 6+7
Figure 1: TransformerFusion is an online scene reconstruction method that takes monocular RGB video as input. The features extracted from each observed image are fused incrementally with transformer architecture. This fusion approach learns to attend to the most relevant image frames for each 3D location (see view attention color maps of the most relevant frame) achieving state-of-the-art reconstruction results.
approach is a learned feature fusion of the video frames using transformer-based architecture, which learns to attend to the most informative image features to reconstruct local 3D region of the scene. A new observed RGB frame is encoded into 2D feature map, and unprojected into 3D volume, where our transformer learns a fused 3D feature for each location the 3D volume from the image view features. This enables extraction of the most informative view features for each location in the 3D scene. The 3D features are fused in coarse-to-fine fashion, providing both improved reconstruction performance well as interactive runtime. These features are then decoded into high-resolution scene geometry with an MLPP-base surface occupancy prediction.
In summary, our main contributions to achieve robust and accurate scene reconstructions are:
Learned multi-view feature fusion the temporal domain using a transformer network that attends only the most informative features of the image views for reconstructing each location in a scene.
A coarse-to-fine hierarchy of our transformer-basee feature fusion that enables an online reconstruction approach running at interactive frame-rates.
2 Related Work
Multi-view depth estimation. Estimating depth from multi-view image observations has been long-studied computer vision. COLMAP introduced patch matching based approach which achieves impressive accuracy and remains established one of the most popular methods for multi- view stereo. While COLMAP offers robust depth estimation for distinctive features in images, the patch matching struggles to densely reconstruct areas without many distinctive color features, such as floor and walls. Recently, learning- approaches that build data-driven priors from large-scale datasets have improved depth estimation in these challenging scenarios. Some proposed methods rely only on a 2D network with multiple images concatenated as input 42). Several recent approaches instead build shared 3D feature cost volume in reference camera space using feature averaging [13] [1] 19 25 26]. These approaches estimate the reference frame's 's depth within a local window of frames, but some also propagate information from previously estimated depth maps by using probabilistic filtering (25). a Gaussian process 17], or an L.TTM bottleneck layer [13] Such multi-view depth estimation approaches predict single-view depth maps, which must be fused together to construct geometric 3D representation of the observed scene.
3D reconstruction from monocular RGB input. Multi-view depth estimation approaches can be combined with depth fusion approaches, such volumetric fusion (6], obtain volumetric

---

reconstruction of the observed scene. MonoFusion [3 is one of the first methods using depth estimate from a real-time variant of PatchMatch stereo T However, fusing noisy depth estimates causes artifacts in the 3D) reconstruction, which lead to the development of recent approaches that directly predict the 3D surface reconstruction instead of per-frame depth estimates. One of the first approaches to predict 3D surface occupancy from two input RGB images is SurfaceNet [20]. which converts volumetrically averaged colors into 3D surface occupancies using a 3D convolutional network. Atlas [28) extends this approach to multi-view setting, while also leveraging learned features instead of colors. Recently NeuralRecon 39] proposed real-time 3D reconstruction framework, adding GRU units distributed in 3D to fuse reconstructions from different local windows of frames. Our approach also fuses together learned features from RGB frame input in an online fashion, but our transformer- based multi-view feature fusion enables relying only on the most informative features from the observed frames for a particular spatial location in the reconstructed scene, producing
accurate 3D reconstructions.
Transformers in computer vision. The transformer architecture 40 has achieved profound im- pact in many computer vision tasks in addition to its natural language processing origins. For detailed survey, we refer the reader to [22]. In computer vision, transformers have been leveraged successfully for tasks such object detection 44) video classification [4), image classification [12). image generation [0], and human reconstruction 45 In this work, we propose transformer-basee feature fusion for 3D scene reconstruction from a monocular video. Given a sequence of observed RGB frames, our approach learns to attend to the most informative features from each image to predict a dense occupancy field.
3 End-to-end 3D Reconstruction using Transformers
Given set of N RGB images I; IR Wx33 of scene with corresponding camera intrinsic parameters K, e 33x3 and extrinsic poses P, E IR R4x4 our method reconstructs the scene geometry by predicting occupancy values E [0, 1J for every 3D point in the scene. Fig. 1igg2ishoo shows overview of our approach. Each input image I; is processed by a 2D convolutional encoder O extracting coarse and fine image features ($f and Φ respectivelly:
+: T. B (!)
From these 2D image features, we construct 3D feature grid in world space. To this end, regularly sample grid points in 3D at coarse resolution every v, 30 cm and a fine resolution of Vy 10 cm. For these coarse and fine sample points, we query corresponding 2D features in all N images and predict fused coarse v and fine 3D features s using transformer networks [40]:
To (
45 + (vb,ww) w.)
T ( 1..... 1 (vid w
Note that we also store the intermediate attention weights w and wf of the first transformer layers for efficient view selection, which is explained in Sec. 3.4
To further improve the features the 3D spatial domain, we apply 3D convolutional networks C, and Cy. at the coarse and fine level, respectively:
C. ( [W + Wxxxx
C 1(65, (xxxFF 15/ |xFFXF
Finally, to predict the scene geometry occupancy for a point p E R-3 the coarse vhe and fine features J are trilinearly interpolated and multi-layer perceptron S maps these features to occupancies:
S (Wte, 3 ~ o E [0,
This extraction of surface occupancies is inspired by convolutional occupancy networks 332 and IFNets [5]. From this occupancy field we extract surface mesh with Marching cubes 27 Note that addition surface occupancy, we also predict occupancy masks for near-surface locations the coarse and fine levels. These masks used for coarse-to fonee surface filtering (see Sec. 3.2 which

---

improves reconstruction performance with focus on the surface geometry prediction and enables interactive runtime.
We train our approach in end-to-end fashion by supervising the surface occupancy predictions using the following loss:
+ f + C..
where Le and f denote binary cross-entropy (BCE) losses on occupancy mask predictions for near-surface locations at the coarse and fine levels, respectively (see Sec. 3.2 and L. denotes BCE loss for surface occupancy prediction (see Sec. 3.3
Occupancy
Coarse Feature Refinement
Coarse Feature Fusion
Prediction
A40 Wore
350 Novm
Fead Forwort
Amanionn
Fonwert
Transformer (8x)
Fine
Fine Fusion
Feature
Refinement
Midh- Head
Feed Forward
3D CNN
Transformer (8x)
Input Images
Image Encoder
Method overview: given multiple input images, we compute coarse and fine level features. Using transformer architecture, we separately fuse these coarse and fine features in a voxel grid. To improve the spatial features, From these feature grids, we extract an occupancy field using a lightweight MLP.
Figure 2:
refinement network for both the coarse and the fine features.
3.1 Learning Temporal Feature Fusion via Transformers
For spatial location p E R3 the scene reconstruction, we learn to fuse coarse off and fine level features g from the N coarse and fine feature images (#f and Φ. respectively). which are extracted train instances of transformer model, one for fusing coarse-level features zi and one for fusing fine-level features & Both transformers Te and Ty share the architecture. Thus, for simplicity, omit the coarse and fine notation in the following.
by the 2D encoder θ. Specifically,
Our transformer model T independently applied to each sample point in world space. For a point P. the transformer network takes a series of 2D features o; as input that are bilinearly sampled from the feature maps 4, at the corresponding projective image location. The projective image location (K,
is computed via full-perspective projection 11, intrinsics K; and extrinsics P. = (R,, tz). To inform the transformer about invalid features (i.c., sample point is projected outside image), input. In addition to these 2D features Φ., we concatenate the projected depth d; = (R.p + t,).. and the viewing ray r (p- c;)/Ilp- c;ll2 22 the input (c, E RD denoting the camera center of view i). These input features are converted to an embedding vector Og E RR' using a linear layer 0: FCN(6, di, Vi, ri), before feeding into the transformer network that then predicts fused feature v E D
t,)). assuming known camera
also provide the pixel validity v {0, 1) as
T (0, ........ 0x) - (1p, w))
As described above, denotes the attention values of the initial attention layer, which are used for view selection to speed-up fusion (see Sec. 3.4
Transformer architecture. We followed [2] when designing the transformer architecture T. It consists of 8 modules of feed-forward and attention layers, using multi-head attention with 4 attention heads and embedding dimension = 256. Feed-forward layers process the temporal inputs independently, and contain ReLU activation, linear layers with residual connection, and layer norm.

---

The model returns both fused feature & E R. and attention weights w E R' over all temporal inputs from the initial attention layer that are later used for selecting which views maintain over longer sequences of input image views.
3.2 Spatial Feature Refinement
While the transformer network fuses 2D observations in the temporal domain, we additionally imbue explicit spatial reasoning by applying a 3D CNN to spatially refine the fused features {5* Jxxxx and { Fx FXFXXXX FXF that are computed by the transformers Te and T, on the coarse and fine grid. respectively. The coarse features [cx))xxee are refined by 3D CNN C. consisting of 3 residual blocks that maintain the same spatial resolution and produce refined features xxxxx... These features are upsampled to a fine grid resolution using nearest- nearest-neighbor upsampling, and concatenated with fused features fine level { PxFFF. A fine-level 3D CNN Cj is then applied to the concatenated features, resulting in refined fine features {./FFFFF. Both, coarse v and fine features & are used for surface occupancy prediction.
Coarse-to-fine surface filtering. The refined features are also used to predict occupancy masks for near-surface locations at both coarse and fine levels, thus, filtering out free-space regions and sparsifying the volume, such that the higher-resolution and computationally expensive fine-scale surface extraction is performed only in regions close the surface. To achieve this, additional 3D CNN layers M. and My are applied to the refined features, outputting a near-surface mask m,, m E [0, 1] for every grid point:
Mo: ( Joxexo + fmxxe
My ( ( [m/]FXFXF
Only spatial regions where both ma and me are larger than 0.5, i.e., close to the surface, are processed further to compute the final surface reconstruction; other regions are determined to be free space. This improves the overall reconstruction performance by focusing the capacity of the surface prediction network to cuss-to-the-suuface regions and enables a significant runtime speed-up.
Intermediate supervision of near-surface masks m and m employed using masks m.. and m generated from the ground truth scene reconstruction, denoting the grid point as near-surface if there exists ground truth surface in the radius of ve or vy from the point. Binary cross entropy losses
Le BCE(m", m and L = BCE(mL . are applied.
3.3 Surface Occupancy Prediction
The final surface reconstruction is predicted by decoding the coarse and fine feature grids to occupancy values E [0, 1), with values 0.5 representing occupied points and values < 0.5 represent- ing free-space points. For point p 3R3 we compute its feature representation by trilinearly interpolating coarse and fine grid features:
vp Trilinear (p, (v Ixxxxc)
o/f Trilincar (p, f (FXFxF)
We concatenate the interpolated features and predict the point's occupancy o S(1b vl.). where is a multi-layer perceptron (MLP) with 3 modules feed-forward layers, containing ReLU activation, linear layer with residual connection, and layer norm.
Surface occupancy supervision. We train on 1.5 1.5 x 1.5 m volumetric chunks of scenes for training efficiency. To supervise the surface occupancy loss, 1k points are sampled inside the chunk, with 80% of samples drawn from a truncation region at most 10 em from the surface, and 20% sampled uniformly inside the chunk. Ground truth occupancy values Ogt are computed using the ScanNet RGB-D reconstructions [8] For uniform samples it straightforward to generate unoccupied point samples by sampling points in free space in front of the visible surface, but it is unknown whether point sample is occupied when lies behind seen surfaces. In order to prevent artifacts behind walls, we follow the data processing applied in 28 and additionally label point samples as occupied, if they are sampled in areas where an entire vertical column of voxels occluded in the scene. A binary cross entropy loss C. BCE(o, ogt) is then applied to the occupancy predictions
5

---

3.4 View Selection for Online Scene Reconstruction
We aim to consider all N frames as input to our transformer for each 3D location in a scene; however, this becomes extremely computationally expensive with long videos large-scale scenes, which prohibits online scene reconstruction. Instead, we proceed with the reconstruction incrementally, pro- cessing every video frame one-by-one, while keeping only a small number K 16 of measurements for every 3D point. We visualize this online approach in Fig. [
During training, for efficiieny, we use only K, random images for each training volume. At test time, we leverage the attention weights w and w of the initial transformer layers to determine which views to keep in the set of K measurements. Specifically, for new RGB frame, extract its 2D features, and run feature fusion for every coarse and fine grid point inside the camera frustum. This returns the fused feature and also the attention weights over all currently accumulated input measurements. Whenever the maximum number of K measurements is reached, a selection is made by dropping out measurement with lowest attention weight before adding new measurements in the latest frame. This guarantees a low number of input measurements, speeding up fusion processing times considerably. Furthermore, by using coarse-to-fine filtering, described in Sec. 3.2 we can further accelerate fusion by only considering higher resolution points in the area near the estimated surface. Together with incremental processing that results in high performance benefits, our approach performs per-frame feature fusion at about 7 FPS despite an unoptimized implementation.
3.5 Training Scheme
Our approach has been implemented using the PyTorch library 31 The architecture details of the used networks are specified in the supplemental document. To train our approach we use SeanNet dataset [8], an RGB-D dataset of indoor apartments. We follow the established train-val-test split. randomly sample 1.5 x 1.5 1.5 volume chunks of the train scenes, sampling less chunks in free space and more samples in areas with non-structural objects, i.e. not only consisting of floor or walls. This results in & 165k training chunks. For each chunk, we randomly sample K, 8 RGB images among all frames that include the chunk in their camera frustums.
For training,
The 2D convolutional encoder è for image feature extraction is implemented as ResNet- 18 [14 network, pre-trained on ImageNet (24] During training. batch size of 4 chunks used with an Adam 23 optimizer with 31 0.9, 82 = 0.999, 10-8 and weight regularization of 10-1, We use learning rate of 10-4 with 5k warm-up steps at initialization, and square root learning rate decay afterwards. When computing the losses of coarse and fine surface filtering predictions, a higher weight of 2.0 is applied to near-surface voxels, to increase recall and improve overall robustness. Training takes about 30 hours using an Intel Xeon 6242R Processor and an Nvidia RTX 3090 GPU,
4 Experiments
Metrics. To evaluate our monocular scene reconstruction, we several measures of reconstruction performance. We evaluate geometric accuracy and completion, with accuracy measuring the average point-to-poin error from predicted to ground truth vertices, completion measuring the error in the opposite direction, and chamfer as the average of accuracy and completion (in cm). To account for possibly different mesh resolutions among methods, we uniformly sample 200k points over mesh faces of every reconstructed mesh. Additionally, we threshold these point-to-point errors and compute precision and recall by computing the ratio of point-to-point matches within distance & 5 em. Since it is easy to maximize either precision (by predicting only a few but accurate points) or recall (by over-completing reconstructions with noisy surface), we found the most reliable metric be F-score, determined by both precision and recall.
Our ground truth reconstructions are obtained by automated 3D reconstruction 9 from RGB-D videos of real-world environments and, thus, they are often incomplete due to unobserved and occluded regions in the scene. To avoid penalizing methods for reconstructing more complete scene w.r.t. the available ground truth, we apply an additional occlusion mask at evaluation.
As most state of the art, particularly for depth estimation, rely a pre-sampled set of keyframes (based on sufficient translation or rotation difference between camera poses), we evaluate all ap- proaches based sequences of sampled keyframes, using the keyframe selection of [13].

---

Table 1: Quantitative comparison with basclines and ablations on test set of Scannet dataset 8
Method
Ace +
Compl y
Chamfer
Prec 1
Recall
F F-score 1
RevisitingSI 18
14.29
16.19
15.24
0.346
0.293
0.314
MVDepthNett 42
12.94
8.34
10.64
0.443
0.487
0.460
GPMVS 17
12.90
8.02
10.46
0.453
0.510
0.477
ESTDepth [26
12.71
7.54
10.12
0.456
0.542
0.491
DPSNet [19
11.94
7.58
9.77
0.474
0.519
0.492
DELTAS [8
11.95
7.46
9.71
0.478
0.533
0.501
DeepVidcoMMS 13
10.68
6.90
8.79
0.541
0.592
0.563
COLMAP 37
10.22
11.88
11.05
0.509
0.474
0.489
NeuralRecon (39
5.09
9.13
7.11
0.630
0.612
0.619
Atlas 28
7.16
7.61
7.38
0.675
0.605
0.636
Ours: w/o TRSF, avg
7.23
9.74
8.48
0.635
0.501
0.557
Ours: w/o TRSF, weight
6.11
11.12
8.61
0.686
0.512
0.583
Ours: w/o TRSF, conv
6.56
9.84
8.20
0.661
0.524
0.582
Ours: w/o spatial ref.
10.46
16.91
13.68
0.479
0.295
0.361
Ours: w/o C2F filter
6.57
7.69
7.13
0.678
0.592
0.631
Ours: w/o proj. depth
8.06
10.02
9.04
0.594
0.475
0.525
Ours: wo viewing ray
5.71
8.59
7.15
0.706
0.559
0.621
Ours: 30 cm voxel size
7.92
17.33
12.63
0.491
0.258
0.335
Ours: 15 cm voxel size
5.79
9.62
7.71
0.686
0.520
0.589
Ours: 4 images, RND
8.01
10.28
9.15
0.587
0.445
0.502
Ours: 4 images
6.80
8.40
7.60
0.661
0.524
0.581
Ours: 8 images, RND
6.74
8.55
7.64
0.665
0.544
0.596
Ours: 8 images
6.17
7.69
6.93
0.704
0.584
0.636
Ours: 16 images, RND
5.80
8.56
7.18
0.711
0.584
0.638
Ours
5.52
8.27
6.89
0.728
0.600
0.655
4.1
Comparison with State of the Art
In Tab. we compare our approach with state-of-the-art methods. All methods are trained on the ScanNet dataset [8), using the official train/val/test split. We use the pre-trained models provided by the authors for MVDepthNet (42). GPMVS [7] and DPSNet [19] which are fine-tuned on ScanNet. For baselines that predict depth in reference camera frame instead of directly reconstructing 3D surface, volumetric fusion method (6] is used to fuse different depth maps into a 3D truncated signed distance field. The single-view depth prediction method RevisitingSI [8] suffers from the more challenging task formulation without the use of multiple views, leading to noisier depth predictions and inconsistencies between frames. Multi-view depth estimation methods leverage the additional view information for improved performance, with the LSTM-basse approach of VieeMidMM [3]] achieving the best performance among these approaches. Reconstruction quality further improves with methods that directly predict the 3D surface geometry, such NeuralRecon (39 and Atlas (28). Our transformer-bact based feature fusion approach enables more robust reconstruction and outperforms all existing methods in both chamfer distance and F-score. The performance improvement also be clearly seen in the qualitative comparisons in Fig.
4.2 Ablations
To demonstrate the effectiveness of our design choices, we conducted quantitative ablation study which is shown in Tab. Tab.[and and discussed in the following.
What is the impact of learning to fuse features from different views with transformers? We evaluate the effect of our learned feature fusion by replacing the transformer blocks with a multi-layer perceptron (MIP) that processes input image observations independently. The per-view outputs of this MLP are fused using average (w/o TRSF, avg) using weighted average with weights

---

DeepVideoMVS [8]
NeuralRecon [32]
Atlas [13]
Ours
Ground truth
Figure 3: Qualitative comparison of scene reconstructions test set of ScanNet dataset [8]; note that only RGB input used by each method while the ground truth is reconstructed using the input depth.

---

predicted by the MLP (w/o TRSF, weight). Additionally, we implemented convolutional feature fusion, using a I-dimensional CNN that processes features in temporal domain and predicts fused features (w/o TRSF, conv). We find that our wannformer-basee view fusion effectively learns to attend to the most informative views for a specific location, resulting in significantly improved performance over these feature fusion alternatives.
Does spatial feature refinement help reconstruction performance? Spatial feature refinement is indeed very important for reconstruction quality. It enables the model to aggregate feature information in spatial domain and produce more spatially consistent and complete reconstructions, without it (w/o spatial ref.) the geometry completion (and recall metric) are considerably worse
How important is coarse-to-fine filtering? Predicting the coarse and fine near-surface masks provides an additional performance improvement compared to the model without it (w/o C2F filter,, as it allows more focus on surface geometry. Furthermore, this enables a speed-up of the fusion runtime by a factor of approximately 3.5, resulting in processing times of FPS (instead of 2 FPS).
Are additional inputs to the transformer networks needed? Existing reconstruction approaches 39 [28] aggregate 2D features using a simple average operation. In comparison, our approach uses transformer to learn the feature fusion. That makes it possible to use additional inputs that don't 't support straight-forwan average operation, but could be very informative for the task of multi-view surface reconstruction, such as projected depth and viewing ray. In Tab. [ we conducted additional quantitative ablation study w.r.t. the input to the transformer networks. Both the projected depth as well as the view ray help the transformer better fuse the features for the task of 3D reconstruction.
How does voxel size of feature grids influence reconstruction performance? We compared the reconstruction performance when using different voxel sizes for the feature grid. We only varied fine feature grid resolution, voxel size of coarse grid was always 30 cm. More specifically, we replaced the voxel size of 10 the fine grid level with 30 and 15 cm. In both cases, the performance decreased considerably; i.e., the higher the resolution, the better the results. That is reflected also in qualitative comparison in the supplemental document.
How many views should used for feature fusion? In our experiments, we use a limited number of K = 16 frame observations to inform the feature for every 3D grid location. We find that these views all contribute, with performance degrading somewhat with sparser sets of observations (K = or K = 4). The number of frames is limited because of execution time and memory consumption for bigger scenes.
How effective is frame selection using attention weights? The K frames for each 3D grid feature are selected based on the computed attention weights and are updated during scanning. To evaluate this frame selection, we compare against a frame selection scheme that randomly selects frames that observe the 3D location (RND), which results in a noticeable drop performance for both chamfer and F-score. The performance difference is even larger when using less views for fusion (K 8 or K = 4), where view selection becomes even more important. In Fig. [ we visualize the most important view for locations in the scene, selected by the highest attention weight. Relatively smooth transitions between selected views among neighboring 3D locations suggest that view selection is spatially consistent. To illustrate the frame selection, also visualize all selected frames with corresponding attention weights for specific 3D locations in the supplemental document.
4.3
Limitations
Under severe occlusions and partial observation of the scene, our method can struggle to reconstruct details of certain objects, such as chair legs, monitor stands, books on the shelves. Furthermore, transparent objects, such as glass windows without frames, are often inaccurately reconstructed as empty space. We show qualitative examples of these failure cases in Fig. 4 These challenging scenarios are often not properly reconstructed even when using ground truth RGB-D data, and we believe that using self-supervised losses [00] for monocular scene reconstruction could be an interesting future research direction. Additionally, higher resolution geometric fidelity could potentially be achieved by sparse operations in 3D or learning local geometric priors on detailed synthetic data (21)

---

Ours
Ground truth
Figure Limitations of our approach are the lack detail at partially observed and occluded objects, and inaccurate reconstruction of transparent surfaces, such as glass windows.
5 Conclusion
We introduced TransformerFusion for monocular 3D scene reconstruction, leveraging a new transformer-based approach for online feature fusion from RGB input views. A coarse-to-fine formulation of our transformer-based feature fusion improves the effective reconstruction perfor- mance as well as the runtime. Our feature fusion learns exploit the most informative image view features for geometric reconstruction, achieving state-of-the-arr reconstruction performance. We believe that our interactive scanning approach provides exciting avenues for future research, and enables new possibilities in learning multi-view perception and 3D scene understanding.
Broader Impact
Our work proposes a novel monocular scene reconstruction approach that can be used applications in the field of augmented and virtual reality, and also serves basis for 3D) scene understanding from monocular RGB input, enabling navigation of autonomous agents unknown environments. Being building block for these applications, impacts of some applications, such as the improper use of autonomous robots in military, or labor market disruptions RGB-D data as supervision, we also need to be aware of related privacy concerns when capturing new datasets for 3D reconstruction.
need to be aware of the potential negative societal
consequence of job automation. Since approach is data-driven, using
Acknowledgments
This project is funded by the Bavarian State Ministry of Science and the Arts and coordinated by the Bavarian Research Institute for Digital Transfornation (bidt), TUM-IAS Rudolf Mößbauer Fellowship. the ERC Starting Grant Scan2CAD (804724), and the German Research Foundation (DFG) Grant Making Machine Learning on Static and Dynamic 3D Data Practical
10

---

References
[1] M. Bleyer. C. Rhemann, and C. Rother. Patchmatch stereo-stereo matching with slanted support windows.
In Bmve, volume 11, pages 1-11, 2011.
[2] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, al. Language models are few-shot learners. arXiv preprint arXiv:2005. 2020.
[3] C. Campos, R. Elvira, J. J. G. Rodríguez. J. M. Montiel, and J. D. Tardós. Orb-slam3: An accurate open-source library for visual, visual-inertial and multi-map slam. arXiv preprint arXiv:2007. 1898, 2020.
[4] N. Carion, F. Massa, G. Syunaeve. N. Usunier, A. Kirillov, and S. Zagoruyko. End-to-end object detection with transformers. In European Conference on Computer Vision, pages 213-229. Springer, 2020.
[5] Chibane, Alldieck, and G. Pons-Moll. Implicit functions feature space for shape reconstruction and completion. In JEEE Conference on Computer Vision and Puttern Recognition (CVPR). IEEE, jun 2020.
[6] B. Curless and M. Levoy. volumetric method building complex models from range images. In Proceedings of the 23rd anmual conference on Computer graphics and interactive techniques, pages 303-312. 1996.
[7] A. Dai and M. Nicßner. 3dmv: Joint 3d-m ad-multi-view prediction for 3d semantic scene segmentation. In Proceedings of the European Conference on Computer Vision (ECCV), pages 452-468, 2018.
[8] A. Dai, A. X. Chang. M. Savva, Halber, T. Funkhouser, and M. NieBner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5828-5839, 2017.
[9] A. Dai, NieBner, M. Zollhöfer, S. Izadi, and C. Theobalt. Bundlefusion: Real-time globally consistent reconstruction using on-the-fly surface reintegration. ACM Transactions on Graphics (ToG), 36(4):1, 2017.
[10] A. Dai, C. Diller, and M. NieBuer. Sg-nn: Sparse generative neural networks for sull-supervised scene completion of rgb-d scans. In Proceedings the Recognition, pages 449-858, 2020.
the IEEE/CVF Conference on Computer Vision and Pattern
[11] Devlin, M.-W. Chang. Lee, and Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
[12] A. Dosovitskiy, I.. Beyer, A. Kolesnikov, D. Weissenbor,, X. Zhai, T. Unterthiner, M. Dehghani, M. Min- derer, Heigold, Gelly, et al. An image worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010..1 19999, 2020.
[13] A. Düzzęker, Galliani, C. Vogel, P. Speciale, M. Dusmanu, and M. Pollcfeys. Deepvideomvs: Multi-view stereo on video with recurrent spatio-temporal fusion, 2020.
[14] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEFE conference on computer vision and pattern recognition, pages 770-778, 2016.
[15] J. Hou, A. Dai, and M. Nießner. 3d-sis: 3d semantic instance segmentation of rgb-d scans. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4421-4430. 2019.
[16] J. Hou, A. Dai, and M. Nießner. Revealnet: Seeing behind objects in rgb-d scans. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pages 2098-2107, 2020.
[17] Y. Hou, J. Kannala, and A. Solin. Multi-view stereo by temporal nonparametric fusion. Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 2651 2651-2660, 2019.
[18] Hu, Ozay, Zhang, and T. Okatani. Revisiting single image depth estimation: Toward higher resolution maps with accurate object boundarics. 2019.
[19] S. Im, H.-G. Jeon, S. Lin, and I. S. Kweon. Dpsnet: End-to-end deep plane sweep stereo. arXiv preprint arXiv: Nww:1905,00533 2019.
[20] M. Ji, J. Gall, H. Zheng, Y. Liu, and L. Fang. Surfacenet: An end-to-end 3d neural network for multiview stereopsis. In Proceedings of the IEEE International Conference on Computer Vision, pages 2307-2315, 2017.
[21] C. Jiang, A. Sud, A. Makadia, J. Huang, M. Nießner, T. Funkhouser, al. Local implicit grid representations for 3d scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 6001-6010, 2020.
[22] S. Khan, M. Nascer, M. Hayat, S. W. Zamir, F. S. Khan, and M. Shah. Transformers in vision: A survey. arXiv preprint arXiv:2 169, 202.1
[23] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.6980. 2014.
11

---

[24] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25:1097-1105, 2012.
[25] C. Liu, J. Gu, K. Kim, S. G. Narasimhan, and J. Kautz. Neural rgb (r) sensing: Depth and uncertainty from a video camera. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11986-109955 2019.
[26] X. Long, L. Liu, W. Li, C. Theobalt, and W. Wang. Multi-view depth estimation using epipolar spatio- temporal network. CVPR, 2021.
[27] W. E. Lorensen and H. E. Cline. Marching cubes: A high resolution 3d surface construction algorithm.
ACM siggraph computer graphics, 21(():163-11) 169, 1987.
[28] Z. Murez. I. van As, J. Bartolozzi, A. Sinha, V. Badrinarayanan, and A. Rabinovich. Atlas: End-to n 3d scene reconstruction from posed images. In ECCV, 2020. URI.btttps://arxii org/ aba/2003 10432
[29] Y. Nic, J. Hou, X. Han, and M. Nießner. Rfd-net: Point scene understanding by semantic instance reconstruction. arXiv preprint arXiv:2011, 1..-24441,,444, 2020.
[30] N. Parmar, A. Vaswani, Uszkoreit, L.. Kaiser, N. Shazeer, A. Ku, and D. Tran. Image transformer. In International Conference on Machine Learning, pages 4055-4064. PMLR, 2018.
[31] A. Paszke, S. Gross, F. Massa, Lerer, J. Bradhury, G. Chaman, T. Killeen, 7. Lin, N. Gimelshein,
L. Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. arXiv preprint 2019.
[32] S. Peng, M. Niemeyer, L. Mescheder, M. Pollefeys, and A. Geiger. Convolutional occupancy networks. In European Conference on Computer Vision (ECCV), Cham, Aug. 2020. Springer International Publishing.
[33] V. Pradeep. C. Rhemann, S. Izadi, C. Zach, M. Bleyer, and S. Bathiche. Monofusion: Real-time 3d reconstruction of small scenes with single web camera. In 2013 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). pages 83- 88. JEEE, 2013.
[34] C. R. Qi, H. Su, K. Mo, and L. J. Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 652-660, 2017.
[35] C. R. Qi, L. Yi, H. Su, and L. J. Guibas. Pointnet+4: Deep hierarchical feature learning on point sets in a metric space. arXiv preprint (06.vv/70060221111 2017.
[36] C. Qi, O. Litany, K. Hc, and I.. J. Guibas. Deep hough voting for object detection in point clouds, In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9277-9286, 2019.
[37] J. L. Schönberger, E. Zheng. M. Pollefeys, and J.-M. Frahm. Pixelwise View Selection for Unstructured Multi-View Stereo. European Conference on Computer Vision (ECCV), 2016.
[38] A. Sinha, Z. Murez, J. Bartolozzi, V. Badrinarayanan, and A. Rabinovich. Deltas: Depth estimation by learning triangulation and densification of sparse points. In ECCV, 2020. (tRLsttts::/// org/ abs/2003. 08933
[39] J. Sun. Y. Xie, L. Chen, X. Zhou, and H. Bao. NeuralRecon: Real-time coherent 3D reconstruction from monocular vidco. CVPR, 2021.
[40] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need, arXiv preprint arXiv:1706. 03762, 2017.
[41] L. von Stumberg, V. Usenko, and D. Cremers. Direct sparse visual-inertial odometry using dynamic marginalization. In International Conference on Robotics and Automation (ICRA), May 2018.
[42] K. Wang and S. Shen. Mydepthnet: Real-time multiview depth estimation neural network. In 2018 International conference on 3d vision (3DV), pages 248-257 257. IEEE, 2018.
[43] W. Wang, R. Yu, Q. Huang, and U. Neumann. Sgpn: Similarity group proposal network for 3d point cloud instance segmentation. In Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition, pages 2569-2578, 2018.
[44] X. Wang, R. Girshick, A. Gupta, and K. He. Non-local neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7794-7803, 2018.
[45] P. Zins, Xu, Boyer, S. Wuhrer, and T. Tung. Learning implicit 3d representations of dressed humans from sparse views. arXiv preprint arXiv: 100.:1004.00.3,, 2021.
12


Reviews:
[{"review_id": "uHiPLgD7B0_", "reviewer": "Reviewer_4TZ2", "summary": "This paper proposes a method to reconstruct dense surface geometry of a static scene from a monocular RGB video. \n\nThe method combines implicit functions and transformers for achieving the final reconstruction. Specifically, images in the video are encoded by a 2D image encoder. After that, for each voxel in a 3D feature volume, the feature was calculated by: first bilinearly sampling per-view features by projecting the voxel on the 2D feature maps in different views, and then, merge the per-view features from different views together using a multi-head transformer. After calculating the 3D feature volume, 3D convolutions are used to directly refine the features in it. Finally, for each query point in 3D space, an MLP-based occupancy decoder was introduced to decode the occupancy value given the corresponding feature of the query point (which is sampled in the refined 3D feature volume). \nTo further improve the reconstruction efficiency, the whole reconstruction pipeline was conducted in a coarse-to-fine manner. \n\nThe main contribution of the paper lies in the transformer-based feature merging stage. By utilizing the strong self-attention performance in the transformers for aggregating features from different views, the proposed method achieves plausible and more complete surface reconstruction results than previous methods. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Reconstructing dense surface geometry using a monocular RGB camera is a very challenging task. Accurate and complete surface reconstruction will benefit a lot of areas like AR/VR, semantic understanding, etc. \n\nThe paper writing is clear and easy to follow. The results are plausible, and the experiments are thorough. \n\nMy major concern is the technical novelty of the proposed method. Since there already have a lot of works that use transformers for feature aggregation (as mentioned in the related work section), simply transferring this technique to static scene reconstruction is not that a breakthrough, despite the fact that the way of using a transformer is relatively straightforward in the paper. \nMoreover, as shown in Fig.1 (1st row) of the supplementary material: the blue color views share similar image quality and viewing direction with the red color views. However, they contribute quite differently to the final results according to the output of the transformer, which is confusing. I would like to suggest the authors explain the reason for this phenomenon in detail. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": " ", "review_text": "", "extra_scores": {"ethical_concerns": " ", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "jFpINqjUBLq", "reviewer": "Reviewer_AGwq", "summary": "This paper presents an RGB 3D reconstruction algorithm with monocular RGB input, using transformer network. The transformer architecture is utilized for temporal feature fusion from multi-view images. Coarse-to-fine hierarchical scheme was designed for efficient surface detection and real-time reconstruction. Also, attention weights from the transformer is used in order to select the most dominant (feature-abundant) frames so that more interactive scene reconstruction can be available. By doing so, the proposed algorithm shows comparable performance on the benchmark experiment, compared to CNN-based previous methods.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper introduced a way of successfully utilizing transformer architecture inspired from NLP to the 3D scene reconstruction problem in computer vision. It is well-written and easy to follow. Also, the paper ablates well on their design choices.\n\nThe followings are my concerns and questions.\n1. While I do think that utilizing transformer architecture in the 3D reconstruction field alone can be a meaningful contribution, the overall pipeline looks very similar to NeuralRecon [39], except for the substitution of GRU Fusion unit with the ViT-based transformer module. Please elaborate on the additional difference or stand-outs that the proposed algorithm has over [39]. It will be really helpful to reconsider my rating towards acceptance.\n\n2. In figure 2, it seems like the coarse features are extracted from the 4th layer of the image encoder, and the fine features are from the 1st. Assuming the basic architecture of the ResNet, the former will have lower spatial resolution but the features will be more deeply embedded, and vice-versa. I wonder if the channel size of the encoded feature can also be a factor in choosing whether it is more appropriate for coarse/fine pipeline. What would happen if the 1st layer feature is utilized in T_c(Coarse Transformer) and the 4th layer feature is used for T_f(Fine Transformer)?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I appreciate the authors for mentioning the limitations of their algorithm on the lack of details in scene reconstruction results.\n\nAdditionally, considering the capture rate of the conventional RGB cameras, I do not think 7 FPS satisfies as ‘online’ or ‘real-time’. \n", "review_text": "", "extra_scores": {"ethical_concerns": ".", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10"}, "extra_text": {}}, {"review_id": "gg9kgJdMe62", "reviewer": "Reviewer_mh3Q", "summary": "This paper proposes to use vision Transformer in a coarse-to-fine manner as a new learned multi-view feature fusion scheme in monocular reconstruction task. Additional blocks such as spatial refinement and C2F filtering help the method generate coherent geometry and improve computational efficiency.\n\nCompared to existing approaches, the full TransformerFusion system shows competitive results on 3D reconstruction quality using ScanNet benchmark with detailed ablation study.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "There are many hanging fruits of applying Transformer in vision tasks. In this paper, the idea of using Transformer to adpatively select which frames to contribute more is well motivated and technically sound. As vision transformer naturally fits the task of temporal fusion of sequential video image data. Both quantitative and qualitative results shows the effectiveness of proposed system.\n\nPros:\n - Paper is well written and the ablation experiments address several concerns of architecture design.\n - The introduction of transformer in multi-view feature fusion is well motivated.\n - The carefully designed pipeline reach competitive performance with reasonable computational cost and experiments are  well-designed.\n\n\n\nCons:\n- Incremental contribution.\n\n    - Given the general pipeline of TransformerFusion is similar to existing approaches such as NeuralRecon, the claimed two unique contributions of this paper come from the adoption of Transformer and C2F refinement to make Transformer practically applicable. In my understanding, these two contributions is in principle a single contribution, i.e., bringing Transformer into monocular reconstruction and make it applicable. The contribution is a bit weak and incremental from this point of view. \n \n    - As mentioned above, the designed modules look very tailored to Transformer-based fusion. To strengthen the contribution and its generalisation, though not the main focus of this paper, it would be good to see if proposed modules like C2F surface filtering is beneficial to other exisiting systems as well.\n\n- Data Generalisation\n\n    - Current evaluation is tested using ScanNet alone and a common concern remains on the generalisations of trained TransformerFusion to other unseen scenes like 7-scenes or real-world exemplar scenes. \n\n- Numerical Comparison\n\n    - From Table 1 and Figure 2 in supplementary material, the refinement modules shows the strongest influence to reconstruction performance, even larger than the contribution from Transformer itself. Given the noisy qualitative result without refinement, is it caused by the lack of inductive biases in MLP-based Transformer fusion?\n\n    - Based on previous point, in Table 1 author compares the ablation experiments without Transformer fusion but a plain MLP with learned or equal weights during fusion. For me it is more interesting and fair to see how convolutional-like fusion structure performs here. In addition. given inductive biases in convolutional structure, will the benefits from refinement module become less pronounced?\n\n    - More explanation of these factors would be appreciated and help better understand what is the pros and cons of using Transformer here. \n\nSome detailed comments and questions:\n\n(1) How to select frames in video at test time? Are all frame-rate frames fed into the network or a pre-sampled subset is input due to efficiency consideration?\n\n(2) An implicit MLP decoder is used to finally predict the occupancy values. However, as mentioned in the limitation,  \nfine structures are prone to missing which is a slightly against the advantages of using continuous MLP. Is this result related to the pre-defined voxel resolution in coarse and fine scale or due to the limited resolution of fused feature? \n\n(3) In section 3.1, a 3D point in world coordinate is projected to multiple camera to query 2D features, is depth information of each view/point used here during projection? Counterpart approaches such as Altas or NeuralRecon emit a ray traversing the whole 3D voxel as no depth information is given. Could the author clarify this?\n\n(4) How about using more than 16 frames as only fewer frames are tested. Is there a trend to saturate the performance given more views and redundancy or the performance consistently improves?\n\nOverall, I think this is a well-motivated paper and is well written, therefore tend to accpet this paper (between 6 and 7). The quality of this paper is technically sound, though the contribution is a bit incremental and weak in my point of view, considering that applying Transformer to this task is the key uniqueness. Transformer shows its merits in multi-view fusion but there also comes with other related issues such as lack of inductive bias, computational cost due to point-wise query. More in-depth analysis of the pros and cons of using transformer in mono recontruction is appreciated, especially compared with other learning-based fusion approach using convolutional structure.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have well addressed the potential negative societal impact of their work in the main paper.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6 hours"}, "extra_text": {}}, {"review_id": "dtT9_4XLLo", "reviewer": "Reviewer_Ert2", "summary": "The paper presents a fully end-to-end trainable deep 3D reconstruction pipeline taking monocular RGB video sequences as input. The proposed pipeline works in an online fashion and at interactive frame rates.\nIn the first stage, the method extracts 2D features from input RGB images. In a next stage, 3D features are generated by projecting a 3D location into the images, pooling the corresponding 2D features and fusing them through a transformer network. After refining the 3D features with 3D convolutions, the geometry is predicted through a neural implicit decoder and finally extracted through marching cubes.\nNotably, the method uses a two-level hierarchy of features for improved quality and runtime performance. Furthermore, the self-attention weights are reused to only keep the most important observations for each 3D point at any given time.\nFinally, the authors compare their pipeline to several state-of-the-art methods and perform an ablation study to validate the individual components.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "#Originality: \nThe idea of integrating self-attention into online 3D reconstruction pipelines and using the attention weights for view selection is novel and interesting, although not ground-breaking.\nUsing feature hierarchies or coarse-to-fine level approaches in order to reduce run-time or improve quality by incorporating larger perception fields is a well tested concept (e.g. Chibane et al. [5]; Saito et al. 2020 PIFuHD; NeuralRecon [39]) and not really new in the field of 3D reconstruction.\nAs such, this paper can be considered more a work which combines well-known techniques (Self-attention, Image Feature Extraction with 2D convolutions, using Feature Hierarchies, 3D Convolutions for “Refining”/Propagating 3D Features and using neural implicit functions to predict geometry) rather than a theoretical contribution but is nonetheless valuable and interesting.\nIn contrast to previous work the presented method is online and fast enough to run at interactive frame-rates. There exist other deep learning based online 3D reconstruction approaches (e.g. RoutedFusion, Weder et al.) but they work on depth inputs which is an easier task than RGB-only input. \n\n#Quality:\nFrom a qualitative perspective, the work feels well thought through and complete and the paper makes sense from a technical standpoint. The performed experiments and evaluations are extensive and one can see that the authors put a lot of effort into establishing a fair comparison with multiple SotA methods and baselines. The ablation study is very informative and valuable and proves that every proposed component is useful. Specifically, during the read-through, I was asking myself how much influence the viewing ray input had and was glad to find the corresponding ablation study in the supplementary material.\n\n#Clarity:\nThe paper is well written, is clear and well organized. The most relevant details are present and the pipeline should be reproducible with help of the supplementary material. The most important related work has been adequately cited.\n\n#Significance: \nQuantitatively, the paper only slightly outperforms previous work (Atlas) and is roughly on par with SotA (NeuralRecon). Qualitatively, they seem to yield better results, even better than NeuralRecon. Therefore, the results are interesting and show that at least some level of improvements can be expected when using self-attention. I believe that others will build upon the idea to incorporate self-attention into deep learning architectures for 3D reconstruction pipelines and therefore think that exploring self-attention for 3D reconstruction pipelines is valuable. Besides the quantitative results being not overly impressive, I believe that the results provided are valuable to the community.\n\n#Post-rebuttal:\nI agree with the findings of reviewer mh3Q that the novelty of the paper is of an incremental nature. Nevertheless, I believe that the improved results and the technical contributions offset this and justify accepting the paper.\nNeuralRecon can be viewed as parallel work, given that it has been published at CVPR 2021 (after NeurIPS deadline) and therefore I consider coincidental similarities in the pipelines not as a valid reason for rejecting the paper (although in my opinion, there are enough technical differences in the two pipelines, e.g., TransformerFusion projects 3D points into views, while NeuronRecon uses raycasting).\n\nThe observations of reviewer 4TZ2 regarding Fig. 1 of the supplementary were concerning at first but the answer of the authors makes sense, i.e. that out of several similar views only one is assigned the highest weight in order to remove redundancy. However, I suggest, the authors include their answer in the paper and/or the supplementary.\n\nFor the above mentioned reasons, my initial rating persists.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The potential negative societal impact has been addressed adequately.\nLimitations have been addressed somewhat. For me it is not clear yet where the lack of details in qualitative results stems from. Is it only due to occlusions? Could it be that the refinement step (3D convolution) smooths out a lot of details? Could this be addressed with another self-attention mechanism in the refinement step?\n", "review_text": "", "extra_scores": {"ethical_concerns": "No ethical concerns.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper has rather positive reviews (7,6,6,5). In general, the reviewers appreciated the quality of the writing and experiments, as well as the (moderate) novelty the paper offers. Reviewer AGwq in particular was convinced by the rebuttal, especially on novelty vs the recent NeuralRecon paper, and increased their score from 5 to 6. Overall, the AC agrees with the reviewers and recommends acceptance.

 
Decision: Accept (Poster)