WT-MVSNet: Window-based Transformers for Multi-view Stereo
Authors: Jinli Liao
Yikang Ding
Yoli Shavit
Dihe Huang
Shihao Ren
Jia Guo
Wensen Feng
Kai Zhang
Venue: NeurIPS 2022 Accept (2022)

Abstract: Recently, Transformers have been shown to enhance the performance of multi-view stereo by enabling long-range feature interaction. In this work, we propose Window-based Transformers (WT) for local feature matching and global feature aggregation in multi-view stereo. We introduce a Window-based Epipolar Transformer (WET) which reduces matching redundancy by using epipolar constraints. Since point-to-line matching is sensitive to erroneous camera pose and calibration, we match windows near the epipolar lines. A second Shifted WT is employed for aggregating global information within cost volume. We present a novel Cost Transformer (CT) to replace 3D convolutions for cost volume regularization. In order to better constrain the estimated depth maps from multiple views, we further design a novel geometric consistency loss (Geo Loss) which punishes unreliable areas where multi-view consistency is not satisfied. Our WT multi-view stereo method (WT-MVSNet) achieves state-of-the-art performance across multiple datasets and ranks $1^{st}$ on Tanks and Temples benchmark. Code will be available upon acceptance.

WT-MVSNet: Window-based Transformers for Multi-view Stereo
Jinli Liao
1.2
Yikang Ding
Yoli Shavit
Dihe Huang
Shihao Ren
Jia Guo
Wensen Feng 2
Kai Zhang 1.49
Tsinghua University
Huawei Technologies
Bar-Illn University
Research Institute of Tsinghua, Pearl River Delta
Abstract
Recently, Transformers have been shown to enhance the performance of multi-view stereo by enabling long-range feature interaction. In this work, propose Window- based Transformers (WT) for local feature matching and global feature aggregation in multi-view stereo. We introduce Window- Epipolar Transformer (WET) which reduces matching redundancy by using epipolar constraints. Since point- to-line matching is sensitive erroneous camera pose and calibration, we match windows near the epipolar lines. A second Shifted WT is employed for aggregating global information within cost volume. We present novel Cost Transformer (CT) to replace 3D convolutions for cost volume regularization. In order to better constrain the estimated depth maps from multiple views, further design novel geometric consistency loss (Geo Loss) which punishes unreliable areas where multi-view consistency not satisfied. Our WT multi-view stereo method (WT- MVSNet) achieves state-of-the-ae performance across multiple datasets and ranks ss on Tanks and Temples benchmark.
Introduction
Multi-view stereo (MVS) estimates depth maps of multi-view calibrated images in order to perform dense 3D reconstruction. MVS solutions aim to find correspondences between pixels in reference image and epipolar lines in source images in order to estimate consistent depth values. Recently, Transformers were proposed for enabling long-range feature matching between reference and source images [3], achieving impressive reconstruction quality. However, matching each pixel in reference and source images without epipolar geometry constraints incurs matching redundancy. A recent effort to perform attention breentio--based matching along the epipolar lines of source images [32], suffers instead from sensitivity inaccurate camera pose and calibration, which can in turn results to erroncous matching. Another key step in contemporary learned MVS methods is the regularization of cost volume, generated by stacking
maps associated with respective depth hypotheses. Typically, 3D CNNs are employed along the feature channels to estimate a probability function over different depth values [7, 33]. Recently, shifted window Transformers
shown to enable local feature aggregation while maintaining long-range cross interaction, surpassing CNNs across different vision tasks [4, 11, 12, 14, 15]. Interestingly, while learned MVS methods aim estimate the likelihood of depth hypotheses from multi-view feature consistency, they calculate the absolute error between ground truth and predicted depth expectation without geometrical consistency supervision [2. 7, 33].
In this work, we propose Window- based Transformers (WT) to address both local feature matching and global feature aggregation. In order to introduce epipolar constraints into attention- based feature
"Equal contribution.
t Corresponding author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Completenoss Error (mnn)
Advanced F-score
0.34
35
TranlCVSNG
0.30
Onn
33
31
Accuracy Ervor (mm)
Intermediate F-score
(a) DTU
(b) Tanks and Temples
Figure I: Comparison of performance with state-of-the-art learning- baased MVS methods [3, 7, 16, 17, 19, 26, 27, 32, 37] on DTU [ataset[S] (lower is better) and Tanks and Temples benchmark (higher is better).
matching while maintaining robustness to camera pose and calibration inaccuracies, develop Window-basee Epipolar Transformer (WET), which matches reference pixels and source windows near the epipolar lines. Motivated by the success of Transformers in visual feature aggregation, we further employ window-based Cost Transformer (CT) for cost volume regularization. The CT is able aggregate the global information within the whole cost volume and produces much smoother and more complete probability volume. Finally, we extend the commonly used cross entropy loss (CE Loss) with our Geo Loss, which penalizes depth values that are not geometrically consistent across multiple views. Our WT-based method (WT-MVSNet) is evaluated on multiple MVS datasets. We show that achieves significant and consistent improvement terms of reconstruction accuracy and completeness and note that it ranks 1 on the intermediate and advanced sets of Tanks and Temples benchmark [9].
In summary, our main contributions are as follows:
We introduce a Window-based Epipolar Transformer (WET) for enhancing patch-to-pates to-patch matching between the reference feature and corresponding windows near epipolar lines in source features.
We propose window-based Cost Transformer (CT) to better aggregate global information within the cost volume and improve smoothness.
We design a novel geometric consistency loss (Geo Loss) to supervise the estimated depth map with geometric multi-view consistency.
Extensive experiments show that method achieves state-of-the-arr performance multiple datasets. ranks 1s on the online Tanks and Temples benchmark.
Related works
2.1
Learning-based MVS
Over the past few years, learning-basse based MVS methods [7, 26, 34] have made significant progress in terms of accuracy, completeness, runtime and memory. standard learning-base based MVS pipeline typically consists of three modules: feature extraction, cost volume construction and cost volume regularization. MVSNet [33], which initially suggested this pipeline, computes a cost map for each depth hypothesis by warping the extracted features from several overlapping views using differentiable homography. It then builds cost volume from stacked cost maps and regularizes it with a 3D UNet. This regularization process outputs a probability distribution over the different depth hypotheses. The depth map is estimated with the expectation or winner-take-an strategy of depth hypotheses, only with the supervision of ground truth depth map in reference view.
While being the first learning- based MVS method to achieve comparable results with traditional methods, the use of a 3D UNet for regularization carried high memory and runtime costs. Different methods were soon to follow, proposing alternative regularization approaches for mitigating this problem. Specifically, two main MVS categories have emerged: cascade- 3D CNNs methods [2, 3,,,, 36, 37] and RNN-based methods [25, 27, 28, 30, 34]. Cascade-based methods leverage multi-stage strategy to regularize the cost volume with gradually narrower depth dimension. These

---

methods achieve significant improvement in memory, runtime and reconstruction quality compared to MVSNet, but struggle when trying to scale to high-resolution images. In order address this limitation, RNN-based methods were proposed for performing recurrent regularization of cost maps. Such methods scale better in terms of memory but are much slower (due to the sequential nature of their regularization). However, neither of these kinds of methods utilize global receptive field to perform regularization. Unlike these methods, our proposed CT first uses 3D window-based transformers for aggregating global information in cost regularization, making depth maps smoother and more complete.
2.2 Transformers and Attention for Feature Matching
Transformer [24], which was initially designed for natural language processing (NLP), has sparked the interest of the computer vision community due to its superior performance vision challenges U, 14, 18, 20J. The ideology of Transformer has been used the process of feature matching because of its inherent advantage in capturing global context information with an attention mechanism.
With alternating self-attention and cross- atssettttnnion SuperGlue [21] matches the sparse features via spatial relationships and the visual appearance of the keypoints. LoFTR [23] utilizes Transformer in a cascade structure learn densely arranged and globally consented matching priors in ground-truth matches with interleaving self self-attention and cross-attention for dense features matching. STTR [10] captures long-range global context information interaction between different features, leveraging alternating self-attention and cross-attention along epipolar lines.
TransMVSNet [3] first introduces Transformers into MVS task, which matches each reference pixel with the whole source images without using the epipolar constraints and achieves impressive results. MVS2D [32] uses the attention mechanism to perform point-to- matching while ignoring the inaccurate camera calibrations. To solve such problems, we propose the WET that utilizes the window-based transformers for enhancing patch-to-patch matching, taking into account the epipolar constraint and the inaccurate camera calibration. Compared with TVansMVSNet, our WET uses much smaller matching space and costs less computation; compared with MVS2D, WET is more robust to the imperfect camera calibrations and poses.
Method
This section introduces the main contributions of our paper in detail. Firstly, we review the overall architecture of WT-MVSNet in Sec. 3.1, and then describe the three main contributions: Window- based Epipolar Transformer (WET) for global feature interaction in Sec. 3.2, Cost Transformer (CT) for cost regularization in Sec. 3.3 and geometric consistency loss (Geo Loss) in Sec. 3.4.
3.1 Network Overview
The overall architecture of WT-MVSNet shown in Fig. Given a reference image I E Hx several source images (I.) N-1 their corresponding camera extrinsic matrixs (T, (N- and intrinsic matrixs N-1 well as depth ranges (dma), WT-MVSNet aims to estimate the reference depth map D, to reconstruct a dense 3D point cloud. Based on CasMVSNet [7], the first step is to extract the multi-scale features
[F.]. N-1 of all input images via Feature Pyramid Network (FPN) [13] 1/4, 1/2 and full image resolutions. To strengthen the global feature interaction within and across multi-view images, we propose a Window- intra-attention and inter-attention alternately on the extracted features. Then we warp the transformed source features into reference view for building 3D cost volume V of H x W x C' x D. where C. and D denote the number of feature channels and the number of depth candidates. After that, we use the proposed Cost Transformer (CT) to regularize V and produce the probability volume P of H W x D. which aggregates the global cost information and can be used to generate the estimated depth. In the end, we utilize cross entropy loss (CE Loss) to supervise the probability volume and propose geometric consistency loss (Geo Loss) to impose punishment in the area where the geometric consistency is not satisfied.
Epipolar Transformer (WET) to perform
3

---

Transformer
Selft
Window-haved Epipolar
CE Loss
Trarsfoone Bleck
Decour
Block
Goo
Trrsforme: Block
Bleck
One-hot
Fo
(Fd
[1d N-1
Depth Map
Window-bwsed Fiiipolar Intor-mteetion
Differentable Warping
Probability Volume
Ground Truth Binnsy Oecupancy
Figure 2: Overview of WT-MVSNet. Given multi-view images [T and the corresponding camera extrinsic matrixs (T, N-1 (T and intrinsic matrixs {Ke L--1 WT-MVSNet firstly extracts multi-scale features {F.) of all input images. Then it uses Window-basee Epipolar Transformer (WET) to strengthen the global feature interaction within and across {F, JF By performing global cost regularization using Cost Transformer (CT) on 3D cost volume V, WT-MVVNet is able to produce probability volume, which will be used to compute the estimated depth map by using winner-take-all strategy.
3.2
Window-based Epipolar Transformer
Most existing learning-basse based MVS methods that build cost volume directly via warping extracted features, result lacking global context information. To address this problem, we introduce Window-based Epipolar Transformer (WET) which reduces matching redundancy by using epipolar constraints. Since point-to-line matching is sensitive to erroneous camera calibration, we match windows near the epipolar lines. Sec. 3.2. introduces the preliminaries including Swin Transformer and intra-attention. Sec. 3.2.2 introduces the proposed Window- Epipolar Inter-altention, which is the core of WET. Sec. 3.2.3 describes the overall architecture of WET.
3.2.1 Preliminaries
Attention mechanism Swin Transformer [1 [14] proposes hierarchical feature representation with only linear computational complexity, achieving excellent performance variety of computer vision tasks. A Swin Transformer block contains window-based multi-head self attention (W-MSA) and shifted window-based multi-head self attention (SW-MSA), which can be formulated as:
z W-MSA(ENN(2-1) + +
(1)
z' MLL((L +21,
2 441 SW- WWW-MSALNN +2',
(3)
(LP(LN(( +
where IN and MLP denote the LayerNorm and Multilayer Perception. & and z' are the outputs of (5) W-MSA and MLP the 1th block. Swin Transformer divides the feature into non-overlapping windows and groups query Q, key K and value V. The feature-wise similarity is extracted by the dot product of Q and K corresponding to each V. The formula for attention can be defined as:
Attention Q. K, V) SoftMar (QK" //d//++/ V.
(5)
where d presents the dimension of the query and key features. And B is the relative position bias.
intra-attention and Inter-attention When Q and K are taken from the same feature map, the attention layers will capture relevant information within the given feature map. On the contrary, when Q and K are taken from different feature maps, the attention layers will enhance context interaction between different views.

---

Intra- Intra-attention module
Inter-attention module
SW MSA
W M-1
(W
SW-MSA
FF EF
Intru-attention module
Figure 3: Illustration of WET. Given 1 WET divides F; into non-overlapping windows and feed the flattened windows in Intra-attention module and compute W-MSA and SW-MSA sequentially, M-1 and warps the
Then WET divides the reference feature into non-overlapping windows {W:) oij=0 corresponding center points to partition the corresponding windows on source features. By performing Window- epipolar anter-attentin, WET able to enhance the feature interaction across views and improve the feature matching quality.
3.2.2 Window-based Epipolar Inter-attention
We perform inter-attention between Fo and each F;, while only updates F; by following TransMVS- Net [3]. Specifically, we perform inter-attention between the pixels in Fo and corresponding patches along the epipolar lines each F, separately. The first step to divide Fo into M non-overlapping M-1 with the same size hwin wwin and warp the center points P, of w into F; windows (W:) 1
via differentiable homography warping. The warped center points of 7th source view P are:
p K R (K, pdd)) +t]
where Rt and t denote the rotation and translation between reference view and source view. d is the estimated depth value the very first iteration of the coarsest stage. To perform inter-attention, we partition a window w around each p with the same size hwin X Wwins in which the epipolar line of P in source feature goes through. Therefore, the inter-attention is able to strengthen long-range global context information interaction between the window of reference feature and the windows near the epipolar lines on source features.
3.2.3 WET Architecture
The architecture of the proposed WET is represented in Fig. WET is mainly composed of intra-attention module and inter-attention module. In intra-attention module, the extracted features (F.) N-= are divided into non-overlapping windows. We flatten each window and feed it to W-MSA and SW-MSA sequentially. With only performing intra-attention within each partitioned window, the W-MSA cannot capture global context of the whole input feature. To solve this problem, we utilize SW-MSA and the shifted window partitioning strategy to enhance information interaction between different windows and obtain global context. order to reduce matching redundancy and avoid erroneous camera pose and calibration, we perform Window-basee Epipolar Inter-attention between reference and source views. inter-attention module, we divide F, into non-overlapping windows and warp each center point partition the corresponding windows in source features. After flattening the partitioned windows, we compute the inter-attention between each window in Fo and the corresponding window in each F; to transform and update F; only.
3.3 Cost Transformer
In this section, we further explore the effects of different regularizations and find that the global receptive field has significant impact on final performance. In practice, propose novel window- based Cost Transformer (CT) to aggregate the global information within cost volume. As shown in Fig. 4-(b), with the expansion of receptive field, the voxels of probability volume with the highest
5

---

Liar Babiki
1056 bay
Bai
[CHMMSNNN)
CNN [CHAMVSNN))
(a) The architecture of CT
(b) Visualization comparison of probability volume
Figure 4: (a) The architecture of CT composes of encoder, decoder and skip connections. CT takes as input initial cost volume and performs regularization with aggregating the global cost information. (b) Visualization of the voxel distribution of probability volume with the highest probability in depth dimension. Compared with 3D CNNs and no-regularization, CT is able to produce smoother and more complete probability volume, as well as higher confidence (yellow regions indicate higher probability, which is equivalent to higher confidence).
probability in depth dimension become smoother and more complete, as well as higher confidence. In comparison 3D CNNs and no-regularization, proposed CT produces probability volume with higher quality.
3D attention As mentioned in Sec. 3.2.1, given 2D feature F, W-MSA and SW-MSA are able to capture the global context within F. In the task of multi-view stereo, we are naturally faced with handling 3D features (cost volume V). To leverage global receptive field in cost regularization, we extend the W-MSA and SW-MSA to 3D version. To do so, we generally flatten the 3D volume in spatial and depth dimensions. The following operations are similar 2D attentions.
CT Architecture The overall architecture of CT is shown in Fig. 4-(a), which consists of encoder, decoder and skip connections. Given an input cost volume V, the encoder firstly divides V into non-overlapping 3D blocks, each 3D block will then be flattened from 2 x 4 x 4 x C' to 32C. Besides, linear embedding layer is applied produces the embedded cost volume V". After that, we further divide V' into non-overlapping 3D windows, and flatten each 3D window from duii x hwin X Wain x C' to dwwin Wwin X C'. The flattened windows are then fed into N 3D attention blocks, and each block consists of 3DW-Intra-Alt, 3DSW-Intra-AN Att and patch merging layer. The patch merging layer is responsible for spatially down- sampling and increasing channel dimension. In the decoder, leverage deconvolutions restore the resolution. To decrease the loss of spatial information generated by the patch merging layer, we concatenate the shallow features and the deep features together with skip connections, which fuses the multi-scale features from the encoder with the decoder. Followed by linear embedding layer and patch expansion layer, the transformed V remains the same as the V dimensions. In the end, 3D convolution with x 1 x kernel is applied to produce the final probability volume P.
project the channel dimension 32C into C' and
3.4 Loss Function
Geometric consistency loss In general, the estimated depth maps are only supervised in reference view without using the multi-view consistency, which is usually used to filter out the outliers during the inference phase. We try to utilize such multi-view consistency in training phase and propose a novel geometric consistency loss (Geo Loss) to impose punishment in the area where geometric consistency is not satisfied. Firstly, warp each pixel p in the estimated depth map Do of reference
view obtain the corresponding pixel P; in th neighboring source view by P; To K, Du (p)p. where Do(p) denotes the depth value of pixel p. In turn, we back-project p; into 3D space and then reproject to the reference view as p
( (.TT
p

---

CasMVSNer
TransMVNNet
Ours
Ground Truth
Figure 5: Visualization comparison with state-of-the-art methods on DTU evaluation dataset [8] (scan15 and scan32). Compared with our baseline [7] and TransMVSNet [3], our WT-MVSNet obtains a more complete point cloud shown the red bounding boxes.
Table Quantitative results on DTU evaluation set [8]. The best results are in Bold and the second best figures are in underlined. Compared with existing methods, our method achieves superior performance in terms of overall error (lower better).
Method
(cc.(mam
Comp.(mm)
Overall(mmm)
Gipuma [6]
0.283
0.873
0.578
COLMAP [22]
0.400
0.664
0.532
R-MVSNet [34]
0.385
0.459
0.422
AA-RMVSNet [27]
0.376
0.339
0.357
CasMVSNet [7]
0.325
0.385
0.355
EPP-MVSNet [16]
0.413
0.296
0,355
PatchmatchNet [26]
0.427
0.277
0,352
MVS2D [32]
0.394
0.290
0.342
UniMV [19]
0.352
0.278
0,315
TransMVSNet [3]
0.321
0.289
0.305
GBi-Net [17]
0.327
0.268
0,298
WT-MVSNet
0.309
0.281
0.295
where D denotes the ground truth depth value of pj. We define two reprojection
Ep = |||p p" l'
(8)
1
DolD') "o(d"))D((p))
Ea Do(p)
Thus the final Geo Loss LGeo can be written by:
LGeo
Σ
(10)
((())
where & denotes the Sigmoid function normalize the combined reprojection errors with hyper- parameters y, {P.] represents a set of valid spatial coordinates obtained by the valid mask map and {Pa) is of all pixels whose reprojection errors within the given thresholds, e.g. &p Ti and Ea where Tj and T2 is hyperparameters which decrease with the increase of the stage.
Total loss In summary, the loss function consists of cross entropy loss (CE Loss) and Geo Loss:
McCE + AgLGeo
(1I)
where dj and X2 are the weights and M is the number of the stages.
7

---

F-score 28.11
E-scare - 34,7
Fascoe 36A2
Faccree 3615
Parerr 46.26
Eiscore $3,11
Esscure 56,94
Farere 57.76
Zr
3r
CasMVSNet
UniMVSNet
MVSNer
Ours
Figure 6: Comparison of reconstructed results with our baseline [7] and top-2 published models
[19, 3] on Tanks and Temples benchmark [9]. The first row shows Recall on the advanced scene of Temple (r 15mm); the second row shows Recall on the intermediate scene of Horse (T 33mm).
4
Experiments
4.1
Implementation Details
We implement WT-MVSNet based on Pytorch, which is trained on DTU training set. Similar to CasMVSNet [7] with 3 stages of 1/4, 1/2 and full image resolutions, the corresponding depth interval decays by 0.25 and 0.5 from stage to 3; the number of depth hypotheses is 48, 32 and 8 for each stage. When trained on DTU, the number of images is to N 5 and the image resolution is set to 512 x 640. We train our model using Adam for 16 epochs at a learning rate of 0,001, which decays by factor of 0.5 after
12 epochs, respectively. We combination coefficient 7 100.0, the loss weights Å = 2.0 and A2 = 1.0, the reprojection errors thresholds to 3.0, 2.0, 1.0 and 72 to 0.1, 0.05, 0.01 3 resolutions. We train our model with the batch size being set 1 on Tesla V100 GPUs. WT-MVSNet typically takes 15 hours and occupies 13GB of each GPU's RAM. Besides, the runtime and memory for inference are 0.786s and 5221MB respectively.
4.2 Datasets
There several commonly used training and evaluation datasets for MVS follows: (a) DTU is an indoor dataset that contains 128 scans with 49 views of different lighting conditions under well- controlled laboratory conditions with fixed camera trajectory. Following the setting of MVSNet [33], DTU dataset is split into 79 training scans, 18 validation scans, and 22 evaluation scans. (b) Tanks and Temples is large-scale benchmark captured in realistic indoor and outdoor scenarios which contains an intermediate subset of 8 scenes and an advanced subset of 6 scenes. (c) BlendedMVS a large-scale synthetic dataset for MVS training that contains cities, sculptures and small objects, which is split into 106 training scans and validation seans.
4.3 Experimental Performance
Evaluation on DTU We train our model on DTU training set and test our model on DTU evaluation dataset [8] towards point clouds with the number of images N = 5 and the image resolution as 864 x 1152. The quantitative results of DTU evaluation set are summarized in Tab. 1, where Accuracy and Completeness are pair of official evaluation metrics. Accuracy is the percentage of generated point clouds matched the ground truth point clouds, while Completeness measures the opposite. Overall is the mean of Accuracy and Completeness. Compared with the other methods, our proposed method shows its capability for generating denser and more complete point clouds on textureless regions, which is visualized in Fig. 5. All point clouds DTU evaluation set available from the Supplementary Material.
Evaluation on Tanks and Temples Before evaluating on Tanks and Temples benchmark [9] to validate the generalization of our model, we finetune WT-MVSNet BlendedMVS [35] training
8

---

Table 2: Benchmarking results F-score (higher is better) on the intermediate leaderboards of Tanks and Temples 19]. The best results are in Bold and the second best figures are in underlined. WT-MVSNet ranks 1s among all the submitted methods on the intermediate leaderboards of Tanks and Temples benchmark (May. 19, 2022).
Method
Int.Mcan Family Francis Horse L.H. M60
Panther
P.G. Train
COLMAP [35]
42.14
50.41
22.25
26.63 56.43 44.83
46.97
48.53 42.04
R-MVSNet [34]
50.55
73.01
54.46
43.42 43.88 46.80
46.69
50.87 45.25
Patchmatch [26]
53.15
66.99
52.64
43.24
54.87 52.87
49.54
54.21 50.81
Ces [7]
56.84
76.37
58.45
46.26
55.81 56.11
54.06
58.18
49.51
ACMM [29]
57.27
69.24
51.45
46.97
63.20 55.07
57.64
60.08 54.48
GBi-Net [17]
61.42
79.77
67.69
51.81
61.25 60.37
55.87
60.67 53.89
AA-RMVSNet [27]
61.51
77.77
59.53
51.53
64.02 64.05
59.47
60.85 54.90
EPP-MVSNet [16]
61.68
77.86
60.54
52.96
62.33 61.69
60.34
62.44 55.30
TransMVVSNtt [3]
63.52
80.92
65.83
56,94
62.54 63.06
60.00
60.20 58.67
UniMVSNet [19]
64.36
81.20
66.43
53.11
63.46
66.09
64.84
62.23 57.53
WT-MVSNet
65.34
81.87
67.33
57.76 64.77
65.68
64.61
62.35 58.38
Table 3: Benchmarking results of F-score (higher is better) on the advanced leaderboards of Tanks and Temples [9]. Compared with all the published methods, WT-MVSNet achieves saate-of-the-ase
performance on the advanced leaderboards of Tanks and Temples benchmark (May. 19, 2022).
Method
Adv.Mean Auditorium Ballroom Courtroom Museum Palace Temple
COLMAP [35]
27.24
16.02
25.23
34.70
41.51
18.05
27.94
R-MVSNet [34]
29.55
19.49
31.45
29.99
42.31
22.94
31.10
CasMVSNet [7]
31.12
19.81
38.46
29.10
43.87
27.36
28.11
PatchmatchNet [26]
32.31
23.69
37.73
30.04
41.80
28.31
32.29
AA-RMVSNet [27]
33.53
20.96
40.15
32.05
46.01
29.28
32.71
ACMM [29]
34.02
23.41
32.91
41.17
48.13
23.87
34.60
EPP-MVSNet [16]
35.72
21.28
39.74
35.34
49.21
30.00
38.75
TransMVSNet [3]
37.00
24.84
44.59
34.77
46.49
34.69
36.62
GBi-Net [17]
37.32
29.77
42.12
36.30
47.69
31.11
36.93
UniMVSNet [19]
38.96
28.33
44.36
39.74
52.89
33.80
34.63
WT-MVSNet
39.91
29.20
44.48
39.55
53.49
34.57
38.15
dataset to improve performance in real-world scenes using the original image resolution 576 x 768 and the number images N 7. Compared with the other methods, the performance of our model introduced in Tab. 2 and Tab. 3. where F-score is defined completeness. Besides, Fig. shows qualitative results on the scene Courtroom of advanced set and Horse of intermediate set. WT-MVSNet surpasses all existing learning- bbased MVS methods on intermediate set and outputs all submitted methods on advanced set, confirming efficiency and generalizability. More point cloud results may be found in the Supplementary Material.
harmonic mean of accuracy and
4.4 Ablation Study
As aforementioned, we adopt a new loss function and Transformer applied in feature matching and regularization. To demonstrate the effectiveness of our proposed modules, we conduct a quantitative ablation study. Our method basically based on CasMVSNet [7], which applies feature correlation similar to PatchmatchNet [26] and L-1 regression loss to train. All experiments are conducted with same parameters. With the quantitative performance shown in Tab. 4, we add CE Loss, Geo Loss, WET and CT based baseline gradually. It can be seen from the table that our proposed modules improve both accuracy and completeness. More extensive ablation experiments of our design choices are analyzed in the Supplementary Material.

---

Table 4: Ablation results (lower is better) with different components on DTU evaluation dataset [8].
module Settings
Mean Distance
CE Loss Geo Loss WET CT (mc.
Comp. Overal((mm)
(a)
0.351
0.311
0.331
(b)
0,342
0.307
0.325
(C)
0.336
0.302
0.319
(d)
/
0.328
0.288
0.308
(c)
0.309
0.281
0.295
Discussion
5.1 Limitations
In inter-attention module, the warped points are fixedly selected from reference feature, without considering the importance of the central point. Besides, the introduction of Transformer inevitably incurs high memory costs during the training phase and slows down the speed of inference.
5.2 Conclusion
In this paper, we propose WT-MVSNet, an end-to-end learning-based MVS network, for both local feature matching and global feature aggregation. We introduce Window-based Epipolar Transformer (WET) to match windows near the epipolar lines which reduces matching redundancy and avoids erroneous camera pose and calibration. Besides, we present a novel Cost Transformer (CT) to replace 3D convolutions for aggregating global information within the cost volume. To better constrain the estimated depth maps from multiple views, we design a geometric consistency loss (Geo Loss) to impose punishment in the unreliable areas where multi-view consistency is not satisfied. Our WT- MVSNet achieves state-of-the of-thc-rrttt performance across multiple datasets and ranks 1st on challenging Tanks and Temples benchmark.
6 Acknowledgments
This work is supported by the Key-Area Research and Development Program of Guangdong Province (No. 200000000000000003, the Science and Technology Innovation Project of Shenzhen (ISGG20210802254800770) and the National Natural Science Foundation of China under Grants 61902415.
10

---

References
[1] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usumier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European Conference on Computer Vision, pages 213-229. Springer, 2020.
[2] Shuo Cheng, Zexiang Xu, Shilin Zhu, Zhuwen Li, Li Erran Li, Ravi Ramamoorthi, and Hao Su. Deep stereo using adaptive thin volume representation with uncertainty awareness. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2524-255 2534, 2020. 1.2
[3] Yikang Ding, Wentao Yuan, Qingtian Zhu, Haotian Zhang, Xiangyue I Liu, Yuanjiang Wang, and Xiao
Liu. Transmvsnet: Global context-aware multi-view stereo network with transformers. arXiv preprint
arXiv:2111.46000 2021. 1, 3. 5, 7, 8,
[4] Xiaoyi Dong, Jianmin Bao, Dongdong Chen, Weiming Zhang, Nenghai Yu, Lu Yuan, Dong Chen, and Baining Guo. Cswin transformer: A general vision transformer backbone with cross-shaped windows,
2021.
[5] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn. Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold. Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021. 2,3
[6] Silvano Galliani. Katrin Lasinger, and Konrad Schindler. Massively parallel multiview stereopsis by surface normal diffusion. In Proceedings the IEEE International Conference on Computer Vision, pages
873 1, 2015.
[7] Xiaodong Gu, Zhiwen Fan, Siyu Zhu, Zuozhuo Dai, Feitong Tan, and Ping Tan. Cascade cost volume for high-resolution multi-view stereo and stereo matching. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 2495-2504, 2020. 1, 2, 3, 7, 8, 9
[8] Rasmus Jensen, Anders Dahl, George Vogiatzis. Engin Tola, and Henrik Aanæs. Large scale multi-view stereopsis evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition,
pages 406-413 016-413,20 2014. 7, 8, 10
[9] Arno Knapitsch, Jaesik Park, Qian-Yi Yi Zhou, and Vladlen Koltun. Tanks and temples: Benchmarking
large-scale scene reconstruction. ACM Transactions Graphics (ToG), 36(4))1-13, 2017. 2, 8,
[10] Zhaoshuo Li, Xingtong Liu, Nathan Drenkow, Andy Ding, Francis X Creighton, Russell H Taylor, and Mathias Unherath. Revisiting stereo depth estimation from sequence-to-sequeece perspective with transformers. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pages 6197-6206, 2021. 3
[11] Jingyun Liang, Jiezhang Cao, Guolci Sun, Kai Zhang. Lue Van Gool, and Radu Timofte. Swinir: Image restoration using swin transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, pages 1833- 1844, October 2021. 1
[12] Ai-Jun Lin, Bingzhi Chen, Jiayu Xu, Zheng Zhang. and Guangming Lu. Ds-transunet: Dual swin transformer u-net for medical image segmentation. ArXiv, abs/2 106.06766, 2021. 1
[13] Tsung- Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongic. Feature pyramid networks for object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 21 2117-2125, 2017. 3
[14] Ze Liu, Yutong Lin, Yue Cao. Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hicrarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF
International Conference on Computer Vision (ICCV), 2021. 3,
[15] 7e Liu, Jia Ning. Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu. Video swin transformer.
arXiv preprint arXiv:2106. 13..33.0... 2021. 1
[16] Xinjun Ma, Yue Gong, Qirui Wang, Jingwei Huang, Lei Chen, and Fan Yu. Epp-mvsnet: Epipolar- assembling based depth prediction for multi-view sterco. In Proceedings of the IEEE/CVF International
Conference on Computer Vision. pages 5732-5744 2021. 2, 7,9
[17] Zhenxing Mi, Chang Di, and Dan Xu. Generalized binary search network for highly-efficient multi-view stereo. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
2. 7.9
[18] Xuran Pan, Zhuofan Xia, Shiji Song, Li Erran Li, and Gao Huang, 3d object detection with pointformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7463-7472,
2021. 3
[19] Rui Peng, Rongjie Wang, Zhenyu Wang, Yawen Lai, and Ronggang Wang. Rethinking depth estimation for multi-view stereo: A unified representation. In Proceedings of the IEEE Conference Computer
Vision and Pottern Recognition (CVPR), 2022. 2, 7, 8,
[20] René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision. pages 12179-12181 2021.
[21] Paul- auul-Edoarrd Sarlin, Daniel DeTone, Tomasz Malisicwicz, and Andrew Rabimovich. Supergluc: Learning feature matching with graph neural networks. In Proceedings of the IEEE/CVF conference on computer
vision and puttern recognition, pages 4938-4947, 2020. 3
[22] Johannes Lutz Schönberger, Enliang Zheng, Marc Pollefeys, and Jan-Michael Frahm. Pixelwise View
Selection for Unstructured Multi- View Stereo. In European Conference on Computer Vision (ECCV),
11

---

[23]
Jiaming Sun, Zehong Shen, Yuang Wang, Hujun Bao, and Xiaowei Zhou. Loftr: Detector- free local feature matching with transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern
Recognition, pages 8922-8931, 2021.
[24] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances neural information processing systems, pages 5998-6008, 2017. 3
[25] Fangjinhua Wang. Silvano Galliani, Christoph Vogel, and Mare Pollefeys. Itermss: Iterative probability
estimation for efficient multi-view stereo, 2022. 2
[26] Fangjinhua Wang. Silvano Galliani, Christoph Vogel, Pablo Speciale, and Mare Pollefeys. Patchmatchnet: Learned multi-view patchmatch stereo. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14194-14203, 2021. 2, 7,
[27] Zizhuang Wei, Qingtian Zhu, Chen Min, Yisong Chen, and Guoping Wang. Aa-rmvsnet: Adaptive aggregation recurrent multi-view stereo network. In Proceedings of the IEEF/CVF International Conference
on Computer Vision, pages 6187-6196, 2021. 2.7.9
[28] Qingshan Xu, Martin R Oswald, Wenbing Tao, Mare Pollefeys. and Zhaopeng Cui. Non-local recurrent regularization networks for multi-view stereo. arXiv preprint arXiv:2 2tt::2//00000 10.06436, 2021.
[29] Qingshan Xu and Wenbing Tao. Multi-scale geometrie consistency guided multi-view stereo. Proceed- ings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5483-5492, 2019.
[30] Jianfeng Yan, Zizhuang Wei, Hongwei Yi. Mingyu Ding, Runze Zhang, Yisong Chen, Guoping Wang, and Yu-Wing Tai. Dense hybrid recurrent multi-view stereo net with dynamic consistency checking. In ECCV,
2020. 2
[31] Jiayu Yang. Wei Mao, Jose M. Alvarez, and Miaomiao Liu. Cost volume pyramid based depth inference for multi-view stereo. In The IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR).
June 2020. 2
[32] Zhenpei Yang, Zhile Ren, Qi Shan, and Qixing Huang. Mvs2d: Efficient multi-view stereo via attention-
driven 2d convolutions. arXiv preprint arXiv:21044 /3325, 2021. 1, 7
[33] Yao Yao, Zixin Luo, Shiwei Li. Tian Fang. and Long Quan. Mvsnet: Depth inference for unstructured
multi-view stereo. European Conference on Computer Vision (ECCV), 2018. 1, 8
[34] Yao Yao, Zixin Luo, Shiwei Li, Tianwei Shen, Tian Fang, and Long Quan. Recurrent mvsnet for high- resolution multi-view stereo depth inference. Computer Vision and Puttern Recognition (CVPR), 2019. 2,
7.9
[35] Yao Yao, Zixin Luo, Shiwei Li, Jingyang Zhang, Yufan Ren, Lei Zhou, Tian Fang. and Long Quan. Blendedmvs: A large-scale dataset for generalized multi-view stereo networks. Computer Vision and Pattern Recognition (CVPR), 2020. 8,
[36] Hongwei Yi, Zizhuang Wei, Mingyu Ding, Runze Zhang. Yisong Chen, Guoping Wang, and Yu-Wing Tai. Pyramid multi-view stereo net with self- self-adaptive view aggregation. In ECCV, 2020. 2
[37] Jingyang Zhang, Yao Yao, Shiwei Li, Zixin Luo, and Tian Fang. Visibility-aware multi-view stereo
network. British Machine Vision Conference (BMVC), 2020. 2
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes] See Section 1.
(b) Did you describe the limitations of your work? [Yes] See Section 5.
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethies review guidelines and ensured that your paper conforms to them? [Yes] Totally conform.
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main ex- perimental results (either in the supplemental material or as URL)? [Yes] In the supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] In Section 4.
(c) Did you report error bars (e-g., with respect to the random seed after running experi- ments multiple times)? [No]
12

---

(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] In Section 4.
4. If you using existing assets (e.g., code, data, models) curating/releasing assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [Yes]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "ztkO_MJeiXU", "reviewer": "Reviewer_h2KQ", "summary": "The paper proposes a new learned MVS method based on MVSNet pipeline.\nThe main contribution includes the introduction of window-based transformers to exchange the information among different views and a Geo Loss to consider the supervision signal of multiple views.\nThe experiements show the proposed method achieves a new state-of-the-art and ranks 1st on the tanks and temple dataset.", "questions": "The differentiable warping part in MVSNet actually implicitly tries to find the best matches among epipolar lines, so the window-based epipolar inter-attention module seems do the same things. How do the authors think about this? ", "limitations": "The authors discuss the limitation briefly and the work does not have potential negative societal impact.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n1. The paper is generally well written. All the figures and equations are clear.\n2. It is novel to introduce transformer in the learned MVS method to exchange the information among different views before differentiable warping.\n3. The Geo Loss is novel and reasonable to condiser the supervision signal of multiple views.\n4. The overal scores rank 1st on the tanks and temple dataset.\n\nWeaknesses\n1. Although the framework of MVSNet and NeRF based methods are different, they have the same inputs and can output put the 3D representation of the objects. However, the authors do not mention NeRF based methods in the whole paper. \n2. The memory cost of the transformer is large, so the proposed method cannot deal with high resolution inputs.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "PzffvWKx7h", "reviewer": "Reviewer_cEDu", "summary": "This paper presents WT-MVSNet, a deep learning based multi-view stereo method. In particular, it makes uses of Swin Transformer to aggregate global context in inter- and intra-feature attentions. For inter-attention, the authors propose Window-based Epipolar Transformer (WET) to reduce matching redundancy. In the cost volume regularization phase, the authors propose to replace prior 3D U-Net based architecture with attentions in 3D, named Cost Transformer. Cross entroy and the proposed geometric loss is used as the training objectives. The experiments show that WT-MVSNet achieved state-of-the-art results and the ablation studies prove the effectivness of different proposed components.", "questions": "1. The authors claim that the proposed WET is robust to inaccurate pose and camera calibrations. Any experiments to prove the claim?\n2. Why cross entropy loss is preferable than L1/L2 regression loss?\n3. What is the runtime and memory usage for inference?", "limitations": "The authors have discussed about the limitations in the paper. The most notable limitation of the work is the huge memory consumption during training.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n- The paper is well written and one can follow quite smoothly.\n- The way of combining (Swin) Transformer makes a lot of sense. In particular, the authors not only use Transformer to enhance 2D image feature extraction and matching, but also improve the 3D cost volume regularization with 3d attention machanism.\n- The authors provide extensive evaluation results on different benchmark with state-of-the-art precision as well as detailed ablation studies.\n\n## Weaknesses\n- Overall the paper is lack of significant novelty. The work in general embeds Swin Transformer into the previously proposed Cascade MVSNet pipeline.\n- For the readers who are not familiar with Swin Transformer and Cascade MVSNet, it may be a bit difficult to understand details. It would be better if the authors can provide more context on how W-MSA, SW-MSA, and Cascade MVSNet work.\n- Obviously, WT-MVSNet takes so much meomry (8 x V100 with batchsize = 1) to train, which is way more than, if not all of, most of the prior works.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "KQd-eejn7J", "reviewer": "Reviewer_nW8B", "summary": "The paper considers the problem of Multi-View Stereo reconstruction from images. It proposes a transformer-based approach, where specialized transformers are introduced for epipolar-guided stereo matching (in a region around the epipolar line to account for inaccuracies in camera calibration and poses), for aggregating costs, and for regularization in the cost volume. The proposed approach represents the current state-of-the-art on both the intermediate and hard scenes of the Tanks & Temples dataset.", "questions": "I will consider raising my rating if the weaknesses W1-W5 are successfully addressed in a rebuttal. Addressing W2 is particularly important.", "limitations": "Limitations are reasonably addressed.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\nS1) The proposed approach achieves state-of-the-art results on the challenging Tanks & Temples dataset (both for the intermediate and the hard categories). To me, the results on Tanks & Temples are one of the main strengths of the paper.\n\nS2) The proposed pipeline is technically sound and well-engineered. It is clearly described.\n\nWeaknesses:\nW1) The paper states that \"In order to introduce epipolar constraints into attention-based feature matching while maintaining robustness to camera pose and calibration inaccuracies, we develop a Window-based Epipolar Transformer (WET), which matches reference pixels and source windows near the epipolar lines.\" It claims that it introduces \"a window-based epipolar Transformer (WET) for enhancing patch-to-patch matching between the reference feature and corresponding windows near epipolar lines in source features\". To me, taking a window around the epipolar line into account seems like an approximation to estimating the uncertainty region around the epipolar lines caused by inaccuracies in calibration and camera pose and then searching within this region (see [Förstner & Wrobel, Photogrammetric Computer Vision, Springer 2016] for a detailed derivation of how to estimate uncertainties). Is it really valid to claim this part of the proposed approach as novel?\n\nW2) I am not sure how significant the results on the DTU dataset are: \na) The difference with respect to the best performing methods is less than 0.1 mm (see Tab. 1). Is the ground truth sufficiently accurate enough that such a small difference is actually noticeable / measurable or is the difference due to noise or randomness in the training process?\nb) Similarly, there is little difference between the results reported for the ablation study in Tab. 4. Does the claim \"It can be seen from the table that our proposed modules improve in both accuracy and completeness\" really hold?\nWhy not use another dataset for the ablation study, e.g., the training set of Tanks & Temples or ETH3D?\n\nW3) I am not sure what is novel about the \"novel geometric consistency loss (Geo Loss)\". Looking at Eq. 10, it seems to simply combine a standard reprojection error in an image with a loss on the depth difference.  I don't see how Eq. 10 provides a combination of both losses.\n\nW4) While the paper discusses prior work in Sec. 2, there is mostly no mentioning on how the paper under review is related to these existing works. In my opinion, a related work section should explain the relation of prior work to the proposed approach. This is missing.\n\nW5) There are multiple parts in the paper that are unclear to me:\na) What is C in line 106? The term does not seem to be introduced.\nb) How are the hyperparameters in Sec. 4.1 chosen? Is their choice critical?\nc) Why not include UniMVSNet in Fig. 5, given that UniMVSNet also claims to generate denser point clouds (as does the paper under review)?\nd) Why use only N=5 images for DTU and not all available ones?\ne) Why is Eq. 9 a reprojection error? Eq. 9 measures the depth difference as a scalar and no projection into the image is involved. I don't see how any projection is involved in this loss.\n\nOverall, I think this is a solid paper that presents a well-engineered pipeline that represents the current state-of-the-art on a challenging benchmark. While I raised multiple concerns, most of them should be easy to address. E.g., I don't think that removing the novelty claim from W1 would make the paper weaker. The main exception is the ablation study, where I believe that the DTU dataset is too easy to provide meaningful comparisons (the relatively small differences might be explained by randomness in the training process. \n\nThe following minor comments did not affect my recommendation:\n* References are missing for Pytorch and the Adam optimizer.\n\n**Post-rebuttal comments**\n\nThank you for the detailed answers. Here are my comments to the last reply:\n\n>  Q: Relationship to prior work.\n\nThank you very much, this addresses my concern.\n\n>  A: Fig. 5 is not used to claim our method achieves the best performance among all the methods in terms of completeness, it actually indicates that our proposed method could help reconstruct complete results while keeping high accuracy (Tab. 1) compared with our baseline network [7] and the most relevant method [3]. In that context, we not only consider the quality of completeness but also the relevance to our method to perform comparison in Fig. 5.\n\nAs I understand lines 228-236 in the paper, in particular \"The quantitative results of DTU evaluation set are summarized in Tab. 1, where Accuracy and Completeness are a pair of official evaluation metrics. Accuracy is the percentage of generated point clouds matched in the ground truth point clouds, while Completeness measures the opposite. Overall is the mean of Accuracy and Completeness. Compared with the other methods, our proposed method shows its capability for generating denser and more complete point clouds on textureless regions, which is visualized in Fig. 5.\", the paper seems to claim that the proposed method generates denser point clouds. Maybe this could be clarified?\n\n>  A: As a) nearly all the learning-based MVS methods (including ours) take the DTU as an important dataset for evaluation, b) the GT of DTU is approximately the most accurate GT we can obtain (compared with other datasets), c) the final results are the average across 22 test scans, we think that fewer errors could indicate better performance. However, your point about the accuracy of DTU GT is enlightening, and we think it's valuable future work.\n\nThis still does not address my concern. My question is whether the ground truth is accurate enough that we can be sure that the small differences between the different components really comes from improvements provided by adding components. In this context, stating that \"the GT of DTU is approximately the most accurate GT we can obtain (compared with other datasets)\" does not answer this question as, even though DTU has the most accurate GT, it might not be accurate enough to measure differences at this level of accuracy (0.05 mm difference). If the GT is not accurate enough to differentiate in the 0.05 mm range, then averaging over different test scans will not really help. That \"nearly all the learning-based MVS methods (including ours) take the DTU as an important dataset for evaluation\" does also not address this question. Since the paper claims improvements when using the different components and uses the results to validate the components, I do not think that answering the question whether the ground truth is accurate enough to make these claims in future work is really an option. I think it would be better to run the ablation study on a dataset where improvements can be measured more clearly.\n\n**Final rating**\n\nI am inclined to keep my original rating (\"6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.\"). I still like the good results on the Tanks & Temples dataset and believe that the proposed approach is technically sound. However, I do not find the authors' rebuttals particularly convincing and thus do not want to increase my rating. In particular, I still have concerns about the ablation study as I am not sure whether the ground truth of the DTU dataset is accurate enough that it makes sense to claim improvements if the difference is 0.05 mm or smaller. Since this only impacts the ablation study, it is also not a reason to decrease my rating.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "ETcg1SgJk7u", "reviewer": "Reviewer_rrwL", "summary": "This paper proposes a transformer-based architecture for multi-view stereo. Main contributions include inter-frame and inter-frame attention mechanisms designed to leverage epipolar constraints. The method is simple and performs very well on standard benchmarks. Ablations show that the proposed components are the contributing factors for strong performance.", "questions": "How is depth filtering and post-processing performed when submitting to evaluation benchmarks?\nHow is the cost volume computed?\nHow is the geometric consistency loss able to properly handle outliers caused by occlusions?\n", "limitations": "Yes limitations are adequately addressed.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* The network architecture is simple - especially compared to other top performing neural network architectures for multi-view stereo. \n* The epipolar-window transformer is a novel and interesting contribution\n* Results on the DTU and Tanks & Temples dataset are very good.\n* The visualization of the probability volume is interesting and not something that has been typically shown with prior MVS nets.\n\n\nWeaknesses:\n* Some components of the network are not properly explained. For example, nowhere in the paper is it stated how the cost volume V is computed (I assume it is constructed using the same method as CasMVSNet but this is not explicitly stated). There is also no mention of filtering or post-processing in the paper. This is an important step of MVS systems which can significantly impact performance.\n* This is not a weakness of this work in particular, but many deep MVS papers in general. There are no results presented on the ETH MVS dataset (either low-res or high-res). Deep MVS networks perform very well on DTU and Tanks & Temples but much less so on ETH. I think it would be helpful to show results on datasets where deep learning has been somewhat less successful compared to classical MVS approaches in order to better assess progress, rather than just showing results on datasets where deep learning is successful. I would not consider non-SOTA results on this benchmark to be a weakness of the paper.\n* Memory and timing are only mentioned in the supplemental. Basic time / memory requirements should be given in the main body of the paper.\n\nNot major weaknesses, but some related work that I think should be discussed in more detail in the relevant sections\n*  The CT Architecture is very closely related to the Dense Prediction Transformer [20] (main difference being 2D vs 3D processing). I think DPT should be referenced in Sec. 3.3.\n* Many ideas presented in this work are closely related to GMFlow (Xu et al, CVPR 2022) - such as inter / intra frame attention for obtaining a better cost volume. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept