Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes an SDF transformer network to improve monocular scene reconstruction. Firstly, the 3D transformer is introduced to aggregate the 3D features at different levels in a coarse-to-fine pattern. Secondly, a sparse window multi-head attention module is adopted to save computation costs. Thirdly, the dilate-attention structure, the global attention module, and the global context encoding module are designed to further improve the performance. The results of the ScanNet dataset demonstrate the effectiveness of the proposed method. 
strength_and_weaknesses: Strength: 
+ The approach is reasonable. 
+ Paper is well organized and easy to understand.
+ Ablation study is extensive. And the results validate the proposed modules.


Weaknesses:
- The contributions are slightly incremental. Most approaches are introduced from existing works. The sparse window module is a standard technique in volume-based methods. Global attention and context encoding are commonly-used in 2D tasks. 
- The effect of dilation-attention is doubtful. Intuitively, the dilation and the downsampling may lose more geometry information, while the paper explains that the geometry structure benefits from dilation.
clarity,_quality,_novelty_and_reproducibility: This paper is generally well-written, and I believe it's not difficult to implement.

Questions:
Is this method heavily relied on the accuracy of pose estimation? Is it robust to reconstruct the scene when the poses of several frames are miscalculated?
summary_of_the_review: The paper aims to improve monocular scene reconstruction with 3D SDF transformers. A series of techniques are adopted to enhance the features. Experimental results validate the effectiveness of all the proposed modules. Considering the high quality of paper and incremental contributions, my current recommendation is marginally above acceptance.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper describes a method to fuse coarse-to-fine Truncated Signed Distance Fields (TSDF) predictions over a number of time frames in an image sequence captured by monocular cameras. The TSDF are directly regressed from images with known poses at different scales, similarly to previous works such as NeuralRecon. The main contribution here is to introduce a transformer attention (e.g., instead of a GRU) module to fuse the coarse to fine TSDF volumes using sparse attention in the so-called "top-down-bottom-up” approach. The experiments shows ablations and comparisons where the proposed approach performs well.
strength_and_weaknesses: Strengths:

The problem statement is well explained and has a nicely written related work section.

The illustrations and explanations show efforts and consideration. The simplicity of the figures draw the reader’s attention to the key aspects of the paper.

The proposed architecture looks reasonable and ablation experiments have been provided for the empirical justification.

The qualitative results show where the proposed approach have highest gains clearly.

Weaknesses:

I have reservations on the way the paper has been posed as an independently explored approach for SDF fusion in relation to architectures such as NeuralRecon.

Discussions on what does not work when directly integrating the transformers instead of GRU would help the reader more. Some details in the experiments are missing

Some of the writing contains rather “loosely written” phrases. Experiments writing can be improved. Minor problems exist in citations. E.g., Schonberger 2016 is not SLAM. NeuralRecon citation is missing in the experiments section.

What is the sequence length used for the experiments at a time? Are the images input one by one or it requires a certain number, say 5 or 10 images for a single iteration of training. If so what is this length?

How is the inference performed? More details should be explicitly provided on inference.

Figure 5 gt has texture which makes the comparison difficult. It is better to either have both or only the texture-less rendering for visualization.
clarity,_quality,_novelty_and_reproducibility: As mentioned in the weaknesses, the experiment part lacks some details and explanations. Refer to the weaknesses.

Analysis on limitations is a must as it helps readers and engineers identify possible problems more easily. A responsible reviewer will always rate the paper higher for having it and the authors should consider that.

Although the novelty maybe sufficient, particularly as engineering sparse attention transformers is non-trivial, it is advisable to pose the paper more clearly with respect to related works.
summary_of_the_review: My biggest concern is that the experiments and inference are not as well communicated. Despite this I am rating the paper above the acceptance threshold hoping that the authors will answer the questions/concerns.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper aims for 3D reconstruction in a coarse-to-fine manner, and the key ideas include: 1) variance based 3D volume feature fusion; 2) sparse window multi-head attention; 3) dilate attention; and 4) SDF transformer backbone for the 3D volume.
strength_and_weaknesses: Strengths:
1. The approach is novel. Although 1), 2) and 4) are not ideas that are hard to figure out individually, it is good to put them together to build the 3D reconstruction system, and investigate their effects to the reconstruction quality. Moreover, I think idea 3) is something novel which might inspire or followed by other 3D sparse volume UNet style work if it is effective as described in the paper.
2. The experiment is thorough. The comparison with other methods has a good coverage, and it shows a fair amount of improvement comparing to the previous arts. Also the ablation study clearly shows the benefits of each proposed component.

Weaknesses:
1. This work might not work very well with very small details due to the volume size (note that this is discussed in the appendix). However, it is actually not slow for offline reconstruction, and it would be interesting to see the results with smaller volume sizes and the analysis of the running time with regard to the volume sizes. Remember sparsity should help a lot for small volume sizes.
clarity,_quality,_novelty_and_reproducibility: About Novelty please see the above section.

Clarifications:
1. It is good to show some qualitative results when dilated attention is not applied.
2. How is V^2 (the sparse volume) generated in the coarsest level (see Figure 1)? What is the density of the coarsest level volume?
summary_of_the_review: This work has good technique contributions, and the experiment is well done.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The manuscript proposes to to use transformer layers applied to 3D voxel tokens at different levels of detail in order to improve the reconstruction quality of monocular 3D reconstruction (given frame poses) over related work that typically relies on 3D CNNs for multi-view aggregation and fusion.

The core contribution of the work is adapting the transformer paradigm to 3D reconstruction. This involves overcoming the cubic scaling of 3D space via sparse and windowed attention modules, and making sure the geometry does not degrade while downsampling via dilated attention. 
strength_and_weaknesses: The paper is overall well written with some problems in the model section. The illustrations are well done and help communicate the system and properties of the components. Figure 2 was important for me to understand the dilated attention (dilated can have different meanings as well!) On that note I would consider changing the name to something else. Dilation in CNNs is usually a step size between pixels/features used at the input into the kernel. But here we are growing the set of 3d occupied 3d voxels. Its a bit different.

One of the main weak points to me is the explanation of the SDF-Transformer model. 
- In Fig3 there are some arrows from the left to the right side of the multi-scale transformer system at the level of the 3D S-W-Attn blocks. As far as I can tell there is no explanation of what that connection does? Does it mean we are using the feature volume on the down-sample path as the queries in the upsample path? 
- Also the connections from the feature volumes after Fusion (V^i in Fig 1) are not shown in Fig3. I assume they are also arrows into the 3D S-W-Attn module on the downsample-side (left) of the diagram? 
- Also I am assuming 3D S-W-Attn stands for Sparse Window attention module? Please clarify in the text. 

I am also not 100% sure about the dilated attention. Specifically I am confused about this sentence:
> Then we calculate the sparse window attention of the dilated voxels and join them into the downsampled volume

How is the volume downsampled? Local averaging?  What does it mean to join them into the downsampled volume? Does it mean we are running some kind of attention mechanism? If so what are keys and values for this one? Or is the attention map inside the self attention module modified?  Or does this just mean that for the purposes of the S-W-Attn module we just append the list of dilated voxels to the list of occupied voxels?  If we dilate the attention map to extend over non-occupied voxels how do we have features in those voxels? Or is this not using a sparse feature volume? 

In the feature volume construction Eq 1 is called variance but it looks more like a squared deviation from the mean? Usually the sample variance would be computed as 1/(N-1) \sum_i (V_i - V_mean)^2 ? So more like the "total variance" in Eq 3. Just wanting to double check on the notations here. 


 The quantitative evaluation is well done. Comparison with major related work on the key datasets.  The ablation studies are important and clearly support the use of the 3D Transformers over 3D CNNs. Since this is the core contribution these ablations are key. 

Lots of qualitative visualizations illustrate what to expect from the approach.  However, in the qualitative comparisons the GT mesh should be shown without color so we can better compare the geometric accuracy.  While I do see that the gains from larger window size seem to saturate from 8->10 I would still have liked to see some higher window sizes as well.  There is an opportunity to show the run-time tradeoff in this ablation. I.e. how much slower does the model get when we go from 8->10 or 10->12 window size? I think that would make it pretty clear why we want to stick with 10?
clarity,_quality,_novelty_and_reproducibility: 
I think the paper is clear except for the model description (unfortunately) as noted in the weaknesses.  This is unfortunate since the paper otherwise is of high quality (related work, intro, visuals, evaluation). 

Because of the lack of clarity in the model description, it will be hard to reproduce the results based on the paper alone. However, the authors promise the release of the code which should make it fairly straight forward to reproduce.
summary_of_the_review: 
My main concern with the paper is the clarity of the description of the 3D Transformer network that I could not completely follow in full detail.  Otherwise the paper is is solid and makes a worthwhile contribution to the community.  If the clarity problems can be resolved, I think this is a good paper that should be accepted. Until then I am leaning marginally towards reject.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper tackles the task of dense 3D reconstruction from monocular sequences, and introduces a transformer-based architecture for inference. While the overall approach follows prior methods, the key contributions relate (efficiently) incorporating 3D transformers for the multi-view aggregation. The reviewers are all positive about the contributions and empirical results, and while there were some clarity concerns expressed by two of the reviewers, these have been adequately addressed by the revisions.  Beyond the task studied here, the AC also feels that the proposed Transformer design could also be informative for other 3D prediction/reconstruction tasks.

justification_for_why_not_higher_score: This paper, while obtaining impressive results based on the proposed 3D Transformer, does closely follow the overall outline of prior methods. 
justification_for_why_not_lower_score: The work demonstrates impressive results/improvements on a core 3D task, and the memory-efficient 3D transformer may also be of intrinsic interest.
Decision: Accept: poster