Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes to use vision Transformer in a coarse-to-fine manner as a new learned multi-view feature fusion scheme in monocular reconstruction task. Additional blocks such as spatial refinement and C2F filtering help the method generate coherent geometry and improve computational efficiency.

Compared to existing approaches, the full TransformerFusion system shows competitive results on 3D reconstruction quality using ScanNet benchmark with detailed ablation study.
main_review: There are many hanging fruits of applying Transformer in vision tasks. In this paper, the idea of using Transformer to adpatively select which frames to contribute more is well motivated and technically sound. As vision transformer naturally fits the task of temporal fusion of sequential video image data. Both quantitative and qualitative results shows the effectiveness of proposed system.

Pros:
 - Paper is well written and the ablation experiments address several concerns of architecture design.
 - The introduction of transformer in multi-view feature fusion is well motivated.
 - The carefully designed pipeline reach competitive performance with reasonable computational cost and experiments are  well-designed.



Cons:
- Incremental contribution.

    - Given the general pipeline of TransformerFusion is similar to existing approaches such as NeuralRecon, the claimed two unique contributions of this paper come from the adoption of Transformer and C2F refinement to make Transformer practically applicable. In my understanding, these two contributions is in principle a single contribution, i.e., bringing Transformer into monocular reconstruction and make it applicable. The contribution is a bit weak and incremental from this point of view. 
 
    - As mentioned above, the designed modules look very tailored to Transformer-based fusion. To strengthen the contribution and its generalisation, though not the main focus of this paper, it would be good to see if proposed modules like C2F surface filtering is beneficial to other exisiting systems as well.

- Data Generalisation

    - Current evaluation is tested using ScanNet alone and a common concern remains on the generalisations of trained TransformerFusion to other unseen scenes like 7-scenes or real-world exemplar scenes. 

- Numerical Comparison

    - From Table 1 and Figure 2 in supplementary material, the refinement modules shows the strongest influence to reconstruction performance, even larger than the contribution from Transformer itself. Given the noisy qualitative result without refinement, is it caused by the lack of inductive biases in MLP-based Transformer fusion?

    - Based on previous point, in Table 1 author compares the ablation experiments without Transformer fusion but a plain MLP with learned or equal weights during fusion. For me it is more interesting and fair to see how convolutional-like fusion structure performs here. In addition. given inductive biases in convolutional structure, will the benefits from refinement module become less pronounced?

    - More explanation of these factors would be appreciated and help better understand what is the pros and cons of using Transformer here. 

Some detailed comments and questions:

(1) How to select frames in video at test time? Are all frame-rate frames fed into the network or a pre-sampled subset is input due to efficiency consideration?

(2) An implicit MLP decoder is used to finally predict the occupancy values. However, as mentioned in the limitation,  
fine structures are prone to missing which is a slightly against the advantages of using continuous MLP. Is this result related to the pre-defined voxel resolution in coarse and fine scale or due to the limited resolution of fused feature? 

(3) In section 3.1, a 3D point in world coordinate is projected to multiple camera to query 2D features, is depth information of each view/point used here during projection? Counterpart approaches such as Altas or NeuralRecon emit a ray traversing the whole 3D voxel as no depth information is given. Could the author clarify this?

(4) How about using more than 16 frames as only fewer frames are tested. Is there a trend to saturate the performance given more views and redundancy or the performance consistently improves?

Overall, I think this is a well-motivated paper and is well written, therefore tend to accpet this paper (between 6 and 7). The quality of this paper is technically sound, though the contribution is a bit incremental and weak in my point of view, considering that applying Transformer to this task is the key uniqueness. Transformer shows its merits in multi-view fusion but there also comes with other related issues such as lack of inductive bias, computational cost due to point-wise query. More in-depth analysis of the pros and cons of using transformer in mono recontruction is appreciated, especially compared with other learning-based fusion approach using convolutional structure.


limitations_and_societal_impact: The authors have well addressed the potential negative societal impact of their work in the main paper.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper presents a fully end-to-end trainable deep 3D reconstruction pipeline taking monocular RGB video sequences as input. The proposed pipeline works in an online fashion and at interactive frame rates.
In the first stage, the method extracts 2D features from input RGB images. In a next stage, 3D features are generated by projecting a 3D location into the images, pooling the corresponding 2D features and fusing them through a transformer network. After refining the 3D features with 3D convolutions, the geometry is predicted through a neural implicit decoder and finally extracted through marching cubes.
Notably, the method uses a two-level hierarchy of features for improved quality and runtime performance. Furthermore, the self-attention weights are reused to only keep the most important observations for each 3D point at any given time.
Finally, the authors compare their pipeline to several state-of-the-art methods and perform an ablation study to validate the individual components.

main_review: #Originality: 
The idea of integrating self-attention into online 3D reconstruction pipelines and using the attention weights for view selection is novel and interesting, although not ground-breaking.
Using feature hierarchies or coarse-to-fine level approaches in order to reduce run-time or improve quality by incorporating larger perception fields is a well tested concept (e.g. Chibane et al. [5]; Saito et al. 2020 PIFuHD; NeuralRecon [39]) and not really new in the field of 3D reconstruction.
As such, this paper can be considered more a work which combines well-known techniques (Self-attention, Image Feature Extraction with 2D convolutions, using Feature Hierarchies, 3D Convolutions for “Refining”/Propagating 3D Features and using neural implicit functions to predict geometry) rather than a theoretical contribution but is nonetheless valuable and interesting.
In contrast to previous work the presented method is online and fast enough to run at interactive frame-rates. There exist other deep learning based online 3D reconstruction approaches (e.g. RoutedFusion, Weder et al.) but they work on depth inputs which is an easier task than RGB-only input. 

#Quality:
From a qualitative perspective, the work feels well thought through and complete and the paper makes sense from a technical standpoint. The performed experiments and evaluations are extensive and one can see that the authors put a lot of effort into establishing a fair comparison with multiple SotA methods and baselines. The ablation study is very informative and valuable and proves that every proposed component is useful. Specifically, during the read-through, I was asking myself how much influence the viewing ray input had and was glad to find the corresponding ablation study in the supplementary material.

#Clarity:
The paper is well written, is clear and well organized. The most relevant details are present and the pipeline should be reproducible with help of the supplementary material. The most important related work has been adequately cited.

#Significance: 
Quantitatively, the paper only slightly outperforms previous work (Atlas) and is roughly on par with SotA (NeuralRecon). Qualitatively, they seem to yield better results, even better than NeuralRecon. Therefore, the results are interesting and show that at least some level of improvements can be expected when using self-attention. I believe that others will build upon the idea to incorporate self-attention into deep learning architectures for 3D reconstruction pipelines and therefore think that exploring self-attention for 3D reconstruction pipelines is valuable. Besides the quantitative results being not overly impressive, I believe that the results provided are valuable to the community.

#Post-rebuttal:
I agree with the findings of reviewer mh3Q that the novelty of the paper is of an incremental nature. Nevertheless, I believe that the improved results and the technical contributions offset this and justify accepting the paper.
NeuralRecon can be viewed as parallel work, given that it has been published at CVPR 2021 (after NeurIPS deadline) and therefore I consider coincidental similarities in the pipelines not as a valid reason for rejecting the paper (although in my opinion, there are enough technical differences in the two pipelines, e.g., TransformerFusion projects 3D points into views, while NeuronRecon uses raycasting).

The observations of reviewer 4TZ2 regarding Fig. 1 of the supplementary were concerning at first but the answer of the authors makes sense, i.e. that out of several similar views only one is assigned the highest weight in order to remove redundancy. However, I suggest, the authors include their answer in the paper and/or the supplementary.

For the above mentioned reasons, my initial rating persists.
limitations_and_societal_impact: The potential negative societal impact has been addressed adequately.
Limitations have been addressed somewhat. For me it is not clear yet where the lack of details in qualitative results stems from. Is it only due to occlusions? Could it be that the refinement step (3D convolution) smooths out a lot of details? Could this be addressed with another self-attention mechanism in the refinement step?
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a method to reconstruct dense surface geometry of a static scene from a monocular RGB video. 

The method combines implicit functions and transformers for achieving the final reconstruction. Specifically, images in the video are encoded by a 2D image encoder. After that, for each voxel in a 3D feature volume, the feature was calculated by: first bilinearly sampling per-view features by projecting the voxel on the 2D feature maps in different views, and then, merge the per-view features from different views together using a multi-head transformer. After calculating the 3D feature volume, 3D convolutions are used to directly refine the features in it. Finally, for each query point in 3D space, an MLP-based occupancy decoder was introduced to decode the occupancy value given the corresponding feature of the query point (which is sampled in the refined 3D feature volume). 
To further improve the reconstruction efficiency, the whole reconstruction pipeline was conducted in a coarse-to-fine manner. 

The main contribution of the paper lies in the transformer-based feature merging stage. By utilizing the strong self-attention performance in the transformers for aggregating features from different views, the proposed method achieves plausible and more complete surface reconstruction results than previous methods. 
main_review: Reconstructing dense surface geometry using a monocular RGB camera is a very challenging task. Accurate and complete surface reconstruction will benefit a lot of areas like AR/VR, semantic understanding, etc. 

The paper writing is clear and easy to follow. The results are plausible, and the experiments are thorough. 

My major concern is the technical novelty of the proposed method. Since there already have a lot of works that use transformers for feature aggregation (as mentioned in the related work section), simply transferring this technique to static scene reconstruction is not that a breakthrough, despite the fact that the way of using a transformer is relatively straightforward in the paper. 
Moreover, as shown in Fig.1 (1st row) of the supplementary material: the blue color views share similar image quality and viewing direction with the red color views. However, they contribute quite differently to the final results according to the output of the transformer, which is confusing. I would like to suggest the authors explain the reason for this phenomenon in detail. 
limitations_and_societal_impact:
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper presents an RGB 3D reconstruction algorithm with monocular RGB input, using transformer network. The transformer architecture is utilized for temporal feature fusion from multi-view images. Coarse-to-fine hierarchical scheme was designed for efficient surface detection and real-time reconstruction. Also, attention weights from the transformer is used in order to select the most dominant (feature-abundant) frames so that more interactive scene reconstruction can be available. By doing so, the proposed algorithm shows comparable performance on the benchmark experiment, compared to CNN-based previous methods.
main_review: This paper introduced a way of successfully utilizing transformer architecture inspired from NLP to the 3D scene reconstruction problem in computer vision. It is well-written and easy to follow. Also, the paper ablates well on their design choices.

The followings are my concerns and questions.
1. While I do think that utilizing transformer architecture in the 3D reconstruction field alone can be a meaningful contribution, the overall pipeline looks very similar to NeuralRecon [39], except for the substitution of GRU Fusion unit with the ViT-based transformer module. Please elaborate on the additional difference or stand-outs that the proposed algorithm has over [39]. It will be really helpful to reconsider my rating towards acceptance.

2. In figure 2, it seems like the coarse features are extracted from the 4th layer of the image encoder, and the fine features are from the 1st. Assuming the basic architecture of the ResNet, the former will have lower spatial resolution but the features will be more deeply embedded, and vice-versa. I wonder if the channel size of the encoded feature can also be a factor in choosing whether it is more appropriate for coarse/fine pipeline. What would happen if the 1st layer feature is utilized in T_c(Coarse Transformer) and the 4th layer feature is used for T_f(Fine Transformer)?

limitations_and_societal_impact: I appreciate the authors for mentioning the limitations of their algorithm on the lack of details in scene reconstruction results.

Additionally, considering the capture rate of the conventional RGB cameras, I do not think 7 FPS satisfies as ‘online’ or ‘real-time’.
review_title: null

Meta-review: This paper has rather positive reviews (7,6,6,5). In general, the reviewers appreciated the quality of the writing and experiments, as well as the (moderate) novelty the paper offers. Reviewer AGwq in particular was convinced by the rebuttal, especially on novelty vs the recent NeuralRecon paper, and increased their score from 5 to 6. Overall, the AC agrees with the reviewers and recommends acceptance.

 
Decision: Accept (Poster)