Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper considers the problem of Multi-View Stereo reconstruction from images. It proposes a transformer-based approach, where specialized transformers are introduced for epipolar-guided stereo matching (in a region around the epipolar line to account for inaccuracies in camera calibration and poses), for aggregating costs, and for regularization in the cost volume. The proposed approach represents the current state-of-the-art on both the intermediate and hard scenes of the Tanks & Temples dataset.
strengths_and_weaknesses: Strength:
S1) The proposed approach achieves state-of-the-art results on the challenging Tanks & Temples dataset (both for the intermediate and the hard categories). To me, the results on Tanks & Temples are one of the main strengths of the paper.

S2) The proposed pipeline is technically sound and well-engineered. It is clearly described.

Weaknesses:
W1) The paper states that "In order to introduce epipolar constraints into attention-based feature matching while maintaining robustness to camera pose and calibration inaccuracies, we develop a Window-based Epipolar Transformer (WET), which matches reference pixels and source windows near the epipolar lines." It claims that it introduces "a window-based epipolar Transformer (WET) for enhancing patch-to-patch matching between the reference feature and corresponding windows near epipolar lines in source features". To me, taking a window around the epipolar line into account seems like an approximation to estimating the uncertainty region around the epipolar lines caused by inaccuracies in calibration and camera pose and then searching within this region (see [Förstner & Wrobel, Photogrammetric Computer Vision, Springer 2016] for a detailed derivation of how to estimate uncertainties). Is it really valid to claim this part of the proposed approach as novel?

W2) I am not sure how significant the results on the DTU dataset are: 
a) The difference with respect to the best performing methods is less than 0.1 mm (see Tab. 1). Is the ground truth sufficiently accurate enough that such a small difference is actually noticeable / measurable or is the difference due to noise or randomness in the training process?
b) Similarly, there is little difference between the results reported for the ablation study in Tab. 4. Does the claim "It can be seen from the table that our proposed modules improve in both accuracy and completeness" really hold?
Why not use another dataset for the ablation study, e.g., the training set of Tanks & Temples or ETH3D?

W3) I am not sure what is novel about the "novel geometric consistency loss (Geo Loss)". Looking at Eq. 10, it seems to simply combine a standard reprojection error in an image with a loss on the depth difference.  I don't see how Eq. 10 provides a combination of both losses.

W4) While the paper discusses prior work in Sec. 2, there is mostly no mentioning on how the paper under review is related to these existing works. In my opinion, a related work section should explain the relation of prior work to the proposed approach. This is missing.

W5) There are multiple parts in the paper that are unclear to me:
a) What is C in line 106? The term does not seem to be introduced.
b) How are the hyperparameters in Sec. 4.1 chosen? Is their choice critical?
c) Why not include UniMVSNet in Fig. 5, given that UniMVSNet also claims to generate denser point clouds (as does the paper under review)?
d) Why use only N=5 images for DTU and not all available ones?
e) Why is Eq. 9 a reprojection error? Eq. 9 measures the depth difference as a scalar and no projection into the image is involved. I don't see how any projection is involved in this loss.

Overall, I think this is a solid paper that presents a well-engineered pipeline that represents the current state-of-the-art on a challenging benchmark. While I raised multiple concerns, most of them should be easy to address. E.g., I don't think that removing the novelty claim from W1 would make the paper weaker. The main exception is the ablation study, where I believe that the DTU dataset is too easy to provide meaningful comparisons (the relatively small differences might be explained by randomness in the training process. 

The following minor comments did not affect my recommendation:
* References are missing for Pytorch and the Adam optimizer.

**Post-rebuttal comments**

Thank you for the detailed answers. Here are my comments to the last reply:

>  Q: Relationship to prior work.

Thank you very much, this addresses my concern.

>  A: Fig. 5 is not used to claim our method achieves the best performance among all the methods in terms of completeness, it actually indicates that our proposed method could help reconstruct complete results while keeping high accuracy (Tab. 1) compared with our baseline network [7] and the most relevant method [3]. In that context, we not only consider the quality of completeness but also the relevance to our method to perform comparison in Fig. 5.

As I understand lines 228-236 in the paper, in particular "The quantitative results of DTU evaluation set are summarized in Tab. 1, where Accuracy and Completeness are a pair of official evaluation metrics. Accuracy is the percentage of generated point clouds matched in the ground truth point clouds, while Completeness measures the opposite. Overall is the mean of Accuracy and Completeness. Compared with the other methods, our proposed method shows its capability for generating denser and more complete point clouds on textureless regions, which is visualized in Fig. 5.", the paper seems to claim that the proposed method generates denser point clouds. Maybe this could be clarified?

>  A: As a) nearly all the learning-based MVS methods (including ours) take the DTU as an important dataset for evaluation, b) the GT of DTU is approximately the most accurate GT we can obtain (compared with other datasets), c) the final results are the average across 22 test scans, we think that fewer errors could indicate better performance. However, your point about the accuracy of DTU GT is enlightening, and we think it's valuable future work.

This still does not address my concern. My question is whether the ground truth is accurate enough that we can be sure that the small differences between the different components really comes from improvements provided by adding components. In this context, stating that "the GT of DTU is approximately the most accurate GT we can obtain (compared with other datasets)" does not answer this question as, even though DTU has the most accurate GT, it might not be accurate enough to measure differences at this level of accuracy (0.05 mm difference). If the GT is not accurate enough to differentiate in the 0.05 mm range, then averaging over different test scans will not really help. That "nearly all the learning-based MVS methods (including ours) take the DTU as an important dataset for evaluation" does also not address this question. Since the paper claims improvements when using the different components and uses the results to validate the components, I do not think that answering the question whether the ground truth is accurate enough to make these claims in future work is really an option. I think it would be better to run the ablation study on a dataset where improvements can be measured more clearly.

**Final rating**

I am inclined to keep my original rating ("6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations."). I still like the good results on the Tanks & Temples dataset and believe that the proposed approach is technically sound. However, I do not find the authors' rebuttals particularly convincing and thus do not want to increase my rating. In particular, I still have concerns about the ablation study as I am not sure whether the ground truth of the DTU dataset is accurate enough that it makes sense to claim improvements if the difference is 0.05 mm or smaller. Since this only impacts the ablation study, it is also not a reason to decrease my rating.


questions: I will consider raising my rating if the weaknesses W1-W5 are successfully addressed in a rebuttal. Addressing W2 is particularly important.
limitations: Limitations are reasonably addressed.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes a new learned MVS method based on MVSNet pipeline.
The main contribution includes the introduction of window-based transformers to exchange the information among different views and a Geo Loss to consider the supervision signal of multiple views.
The experiements show the proposed method achieves a new state-of-the-art and ranks 1st on the tanks and temple dataset.
strengths_and_weaknesses: Strengths
1. The paper is generally well written. All the figures and equations are clear.
2. It is novel to introduce transformer in the learned MVS method to exchange the information among different views before differentiable warping.
3. The Geo Loss is novel and reasonable to condiser the supervision signal of multiple views.
4. The overal scores rank 1st on the tanks and temple dataset.

Weaknesses
1. Although the framework of MVSNet and NeRF based methods are different, they have the same inputs and can output put the 3D representation of the objects. However, the authors do not mention NeRF based methods in the whole paper. 
2. The memory cost of the transformer is large, so the proposed method cannot deal with high resolution inputs.
questions: The differentiable warping part in MVSNet actually implicitly tries to find the best matches among epipolar lines, so the window-based epipolar inter-attention module seems do the same things. How do the authors think about this? 
limitations: The authors discuss the limitation briefly and the work does not have potential negative societal impact.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a transformer-based architecture for multi-view stereo. Main contributions include inter-frame and inter-frame attention mechanisms designed to leverage epipolar constraints. The method is simple and performs very well on standard benchmarks. Ablations show that the proposed components are the contributing factors for strong performance.
strengths_and_weaknesses: Strengths:
* The network architecture is simple - especially compared to other top performing neural network architectures for multi-view stereo. 
* The epipolar-window transformer is a novel and interesting contribution
* Results on the DTU and Tanks & Temples dataset are very good.
* The visualization of the probability volume is interesting and not something that has been typically shown with prior MVS nets.


Weaknesses:
* Some components of the network are not properly explained. For example, nowhere in the paper is it stated how the cost volume V is computed (I assume it is constructed using the same method as CasMVSNet but this is not explicitly stated). There is also no mention of filtering or post-processing in the paper. This is an important step of MVS systems which can significantly impact performance.
* This is not a weakness of this work in particular, but many deep MVS papers in general. There are no results presented on the ETH MVS dataset (either low-res or high-res). Deep MVS networks perform very well on DTU and Tanks & Temples but much less so on ETH. I think it would be helpful to show results on datasets where deep learning has been somewhat less successful compared to classical MVS approaches in order to better assess progress, rather than just showing results on datasets where deep learning is successful. I would not consider non-SOTA results on this benchmark to be a weakness of the paper.
* Memory and timing are only mentioned in the supplemental. Basic time / memory requirements should be given in the main body of the paper.

Not major weaknesses, but some related work that I think should be discussed in more detail in the relevant sections
*  The CT Architecture is very closely related to the Dense Prediction Transformer [20] (main difference being 2D vs 3D processing). I think DPT should be referenced in Sec. 3.3.
* Many ideas presented in this work are closely related to GMFlow (Xu et al, CVPR 2022) - such as inter / intra frame attention for obtaining a better cost volume. 
questions: How is depth filtering and post-processing performed when submitting to evaluation benchmarks?
How is the cost volume computed?
How is the geometric consistency loss able to properly handle outliers caused by occlusions?

limitations: Yes limitations are adequately addressed.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper presents WT-MVSNet, a deep learning based multi-view stereo method. In particular, it makes uses of Swin Transformer to aggregate global context in inter- and intra-feature attentions. For inter-attention, the authors propose Window-based Epipolar Transformer (WET) to reduce matching redundancy. In the cost volume regularization phase, the authors propose to replace prior 3D U-Net based architecture with attentions in 3D, named Cost Transformer. Cross entroy and the proposed geometric loss is used as the training objectives. The experiments show that WT-MVSNet achieved state-of-the-art results and the ablation studies prove the effectivness of different proposed components.
strengths_and_weaknesses: ## Strengths
- The paper is well written and one can follow quite smoothly.
- The way of combining (Swin) Transformer makes a lot of sense. In particular, the authors not only use Transformer to enhance 2D image feature extraction and matching, but also improve the 3D cost volume regularization with 3d attention machanism.
- The authors provide extensive evaluation results on different benchmark with state-of-the-art precision as well as detailed ablation studies.

## Weaknesses
- Overall the paper is lack of significant novelty. The work in general embeds Swin Transformer into the previously proposed Cascade MVSNet pipeline.
- For the readers who are not familiar with Swin Transformer and Cascade MVSNet, it may be a bit difficult to understand details. It would be better if the authors can provide more context on how W-MSA, SW-MSA, and Cascade MVSNet work.
- Obviously, WT-MVSNet takes so much meomry (8 x V100 with batchsize = 1) to train, which is way more than, if not all of, most of the prior works.
questions: 1. The authors claim that the proposed WET is robust to inaccurate pose and camera calibrations. Any experiments to prove the claim?
2. Why cross entropy loss is preferable than L1/L2 regression loss?
3. What is the runtime and memory usage for inference?
limitations: The authors have discussed about the limitations in the paper. The most notable limitation of the work is the huge memory consumption during training.
review_title: null

Meta-review: metareview: The paper is concerned with multi-View Stereo reconstruction from images with several transformers for specific subtasks. SOTA performance is attained. Reviewers acknowledge a technically sound pipeline. The writing is also clear and limitations are addressed. All reviewers recommend the paper for acceptance and so do I.
Nevertheless, the authors must include the feedback provided by reviewers e.g. on possibly limited significance of 0.05 mm better results on DTU is significant or not given accuracy limitations of the ground truth. Also connections to pointed out related work must be discussed.
confidence: Certain
Decision: Accept