A Global Depth-Range-Free Multi-View Stereo Transformer Network with Pose Embedding
Authors: Yitong Dong
Yijin Li
Zhaoyang Huang
Weikang Bian
Jingbo Liu
Hujun Bao
Zhaopeng Cui
Hongsheng Li
Guofeng Zhang
Venue: NeurIPS 2024 poster (2024)
Keywords: Multi-view stereo
Attention
Pose embedding
Primary area: machine_vision

Abstract: In this paper, we propose a novel multi-view stereo (MVS) framework that gets rid of the depth range prior. Unlike recent prior-free MVS methods that work in a pair-wise manner, our method simultaneously considers all the source images. Specifically, we introduce a Multi-view Disparity Attention (MDA) module to aggregate long-range context information within and across multi-view images.  Considering the asymmetry of the epipolar disparity flow, the key to our method lies in accurately modeling multi-view geometric constraints. We integrate pose embedding to encapsulate information such as multi-view camera poses, providing implicit geometric constraints for multi-view disparity feature fusion dominated by attention. Additionally, we construct corresponding hidden states for each source image due to significant differences in the observation quality of the same pixel in the reference frame across multiple source frames. We explicitly estimate the quality of the current pixel corresponding to sampled points on the epipolar line of the source image and dynamically update hidden states through the uncertainty estimation module. Extensive results on the DTU dataset and Tanks\&Temple benchmark demonstrate the effectiveness of our method.

A Global Depth-Range-Free Multi-View Stereo Transformer Network with Pose Embedding
Yitong Dong!" Yijin Li Zhaoyang Huang? Hujun Bao' Zhaopeng Cui'
Weikang Bian?
Jingbo Liu'
Hongsheng Li Guofeng Zhang'1
State Key Lab of CAD&CG, Zhejiang University
CCUH MMLab
Abstract
In this paper, we propose novel multi-view stereo (MVS) framework that gets rid of the depth range prior. Unlike recent prior-free MVS methods that work pair-wise manner, our method simultaneously considers all the source images. Specifically, we introduce a Multi-view Disparity Attention (MDA) module to aggregate long-range context information within and across multi-vin view images. Considering the asymmetry of the epipolar disparity flow, the key to our method lies in accurately modeling multi-view geometric constraints. We integrate pose embedding to encapsulate information such as multi-view camera poses, providing implicit geometric constraints for multi-view disparity feature fusion dominated by attention. Additionally, we construct corresponding hidden states for each source image due significant differences in the observation quality of the same pixel in the reference frame across multiple source frames. We explicitly estimate the quality of the current pixel corresponding sampled points on the epipolar line of the source image and dynamically update hidden states through the uncertainty estimation module. Extensive results on the DTU dataset and Temple benchmark demonstrate the effectiveness of our method. The code is available at our project page: https://zjuddv http:////i...iiiuuuoo///i----- ioo.i--osGDMSSS.....
Temple
Introduction
Multi-view stereo matching (MVS) is crucial technique 3D reconstruction, which aims to recover robust and reliable 3D representations from multiple RGB images [ 2 3]. Traditional methods 6 rely on hand-crafted similarity metries and regularizations to compute dense correspondences between the input images. These methods are prone to degradation in challenging scenarios. such as varying illumination, textureless regions, and occlusion regions. Recently, learning- baased methods TO 11 directly learn discriminative features from the input images through neural networks such CNN and Transformers. By sampling some possible depth hypothesis within given depth range, they warp the features from the source images to the reference view (i.e., the plane sweep algorithm (12) and compute the cost volume, which is then regularized also through the neural network to obtain the final depth maps. However, obtaining a suitable depth range is non-trivial when applied in real-world scenarios while these methods are generally sensitive to the depth range, which limits their application.
7, 9
To get rid of the dependence on depth range, some methods [3
transform the regression problem in the given depth space into a matching problem on the epipolar lines. Similar to optical flow [16] and feature matching [87 [9] 20),, these methods also adopt a pair-wise manner. For example, DispMVS [14] computes the depth map of the source image multiple times through pairs
15
"qqual contribution Cooresppondig author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

that contain different source images and computes the final depth map by weight of sum. However, the pair-wise manner neglects the inter-image correspondence between the source images and could lead to sub-optimal solutions. Meanwhile, although DispMVS mitigates the influence of depth priors on constructing the 3D cost volume, initialization based on depth range can still lead to significant performance degradation when the depth range error is too large, as shown in Fig.
We argue that these methods need consider all the source images the same time. Our ideas are inspired by the recent methods 21 [2] of optical flow which concurrently estimate optical flows for multiple frames by sufficiently exploiting temporal cues. However, we find these frameworks cannot be trivially applied in the task of multi-view stereo. The reasons are twofold. First, a strong cue in the multi-frame optical flow estimation is that the flow originating from the same pixel belongs to a continuous trajectory in the temporal dimension. Additionally, the frames are sequentially aligned along this temporal dimension. Such inductive bias makes it easy to learn. But in the context of multi-view stereo, the source images may be captured in no particular order, lacking similar constraint of continuity. Unlike optical flow, the input images in multi-view stereo are unordered. These distinctions pose significant challenge when attempting to adapt the multi-frame optical flow framework for use in multi-view stereo. Second, the arbitrary positions and viewing angles of the source images, coupled with potentially large temporal gaps between captures, exacerbate issues such as varying illumination, significant viewport differences, and occlusions which call for new designs.
Based on the above observations, in this paper, we propose novel framework that gets rid of the depth range assumption. Unlike some recent methods [3 14 [5] that work in a pair-wise manner, the proposed method estimates the depth maps of a reference image by simultaneously considering all the source images. To address the first issue, we design careful injection of geometric information into disparity features using 3D pose embedding, followed by multi-frame information interaction through an attention module. Subsequently, we encode multi-view relative pose information and geometric relationships between specific sampled points into 3D pose embedding, which is subsequently transferred to the Multi-view Disparity Attention (MDA) module. This method efficiently incorporates the relationship between depth and pixels within the network, facilitating improved information integration across multiple frames. Second, to mitigate the challenge of fluctuating image quality stemming from occlusion and other factors, we maintain and update the disparity hidden features to reflect the depth uncertainty of the current sampling point for each iteration. We design the disparity feature encoding module learn disparity features along the epipolar lines of multi-view frames. This approach enables us to explicitly characterize occlusion scenarios for each pixel across diverse source images and dynamically adapt them during epipolar disparity flow updates. Consequently, the auxiliary information is furnished for subsequent information fusion within the module. Furthermore, we designed a novel initialization method to further eliminate the influence of the depth range compared to DispMVS 14
In summary, our contributions can be highlighted as follows: (1) A multi-view disparity transformer network, which facilitates the fusion of information across multi-view frames, (2) A specially designed 3D pose embedding which is utilized to implicitly construct relationships of the epipolar disparity flow among multi-view frames, and (3) An uncertainty estimation module and dynamically updated hidden states representing the quality of source images during iterations. We evaluate method against other MVS methods on the DTUU dataset (3 and Tanks& Tanks&Temple dataset (24), and demonstrate its generalization in Fig. [
2
Related Work
2.1 Traditional MVS
Multi-View Stereo has been developed for many years and has many downstream or related ap- plications such as simultaneous localization and mapping (SLAM)
225 visual localization [26].
3D reconstruction 27 28). 3D generation [2] and scene understanding [30. Traditional methods for Multi- View Stereo (MVS) can generally be categorized into three classes: volumetric, point
cloud-based, and 2D depth map-based methods. Volumetric methods [1 typically partition the 3D space into voxels and annotate them as either interior or exterior to the object surface. Point cloud-based methods [3 34 directly optimize the point cloud coordinates of objects in 3D space.
Depth map-based methods 2
4 35
6) first estimate 2D depth corresponding to images and then

---

ducin 552.5mm
desin 425mm
(dwin 14 141.6m
Omix 85mm
mmaz 807.5 55 mum
dweax 93.mmm
Awus 2805mm
4 Amax 4675mm
Ours
DisMVS
TtrMVS
Figure 1: The robustness testing on the depth range. Under identical training configurations, our method exhibits superior robustness to variations in depth range compared with two statc-of-the-art The red markings denote the actual depth range used during training.
methods 13 14
fuse the 2D depths of the same scene to obtain a 3D point cloud. However, these traditional methods remain constrained by manually crafted image features and similarity matrices.
2.2 Deep learning based MVS
CNN-based MVS methods generally leverage convolutional neural networks to construct and refine
44 45] 46 477. For instance, (38] uses isotropic and anisotropic 3D convolution based learning networks to estimate the depth map. [40] introduces pixel-wise network to obtain visibility. [ applies a multi-stage CNN framework to enable reconstruction. [44] and [45] build kind of pyramid to realize 3D cost volume. Similarly, [46]
3D cost volume [36] 37 38
39 40
42
43
proposes sparse-to-dense CNN framework when constructing the 3D cost volume.
RNN-based methods mainly exploit recurrent network structures 48 49] to regularize 3D cost [3
53 [54). For example, (50) utilizes recurrent encoder-decoder structure and 2D CNN framework to solve large-scale MVS reconstruction. [5T] introduces a scalable RNN-based MVS framework. IterMVS [3] uses a GRU-based estimator encode the probability distribution of depth. Compared with 3D CNN, RNN highly reduces the memory requirement, which makes it more
volume (50 52 9
suitable for large-scale MVS reconstruction [55]
and first introduced into the field of MVS reconstruction by 110 due to its ability capture global context information. Transformer is incorporated into feature encoding
Transformer is popular in 3D vision tasks 56
57
58
61 to capture features within and between input images. The succeeding work (62) implements a transformer to assign weights different pixels in the aggregating process. 663 employs an Epipolar Transformer to perform non-local feature augmentation. However, these deep learning-basse MVS methods commonly exhibit sensitivity to the depth range, thereby restricting their broad applicability.
10
60,
Scale-agnostic MVS methods infer the depth information from the movement along epipolar lines to reduce the heavy dependence of depth range priors. Several methods [4] [1]] perform 2D sampling between frames and iteratively update flows to find the matching points. Specifically, DispMVS [4] is randomly initialized within the depth range and performs depth fusion by utilizing weighted sum. R.M.eeph (15) selects a random source image in each iteration. However, both methods fail to fully exploit multi-frame constraints during the flow updates due to the mismatch of 3D information sampling points. In this paper, we enhance the epipolar matching process by simultaneously considering multi-frame information.
3
Method
Given a reference image Io E Hxxx3 and multi-view source images (1)) input, the task of MVS is to calculate the depth map of the reference image. We treat MVS as matching problem: for pixel point pr in the reference image, we identify the corresponding point p. in the source image, then we can get depth by triangulation. Given the initial matching point po obtained by the initial depth, we adopt an iterative update strategy. Since the matching point lies on the epipolar line of the source image, the wff.ddrreoooo-freoooo epipolar disparity flow is used
#xxx3 as
3

---

E HXXCC and context feature extraction module to Disgarils Festure Encolig
Mullborees Biggariy Atration
Updating
Dagarity
Dapardy Hiddon State of
Prie Enbodding
Attertion
Fa
Festure of
Updatiag
Pornts
56 36
Self Cross Aftontinn
Curceat Puint
Fased Eplaskar Dagarils Flee
0g
Pr
Warping
Sampling
Upunping
Festure Stuvlarity
llenatooss
Updating Variable
Figure 2: Overview of our method. We introduce the disparity feature encoding module to encode viewpoint quality differences, and the Multi-view Disparity Attention (MDA) module to facilitate information interaction between multi-view images. The MDA module is depicted in Fig. Starting from an initial depth map Do, the epipolar disparity flows iteratively updated and fused the depth of the next stage.
to represent the network's ' iterative updates. The epipolar disparity flow e E Hxxl is 1-d flow along the epipolar line on the source image during each iteration:
Edir (p= p2),
where edir is the normalized direction vector of the epipolar line. is the dot product of vectors, and is the iteration time. Different from previous methods [14, [3]. we fully eliminate the dependence on depth range during initialization and achieve synchronous updating of the epipolar disparity flow across multi-view images. This is done by our design of disparity information interaction.
The overall pipeline of our method is illustrated in Fig. 2 The proposed method starts from a feature extraction module to extract multi-scale features (Sec. B.I. Then, we discuss how to initialize the depth map without depth range (Sec. 3.2) and perform feature encoding (Sec. 3.3). To facilitate information fusion across multi-view source images, we introduce Multi-view Disparity Attention (MDA) module (Sec. 3.4). enhanced with Pose Embedding. Finally, the features enhanced by the MDA module are fed into a GRU module to update the epipolar disparity flow, as described (Sec. which is then fused to generate the depth map.
3.5
3.1 Feature Extraction
we employ convolutional neural networks (CNNs) for image feature extraction. Moreover, we adopt coarse-to-fine toneine strategy to extract multi-scale image features. Specifically, we utilize two share- weighted feature extraction modules extract image features F/ E #xxCX and {F!) N-1 extract context features.
Following previous methods
3.2 Initialization
Differing from DispMVS 14, we design novel initialization method without depth range to further mitigate the influence of depth priors. Specifically, we select an initial position along the epipolar line and then convert it into the depth map. First, we derive the correspondence between depth and position along the epipolar line. Given pixel pr the reference image Lo, the geometric constrain between it and the warped pixel of the source image T; can be written as:
p.d, = R. k-dp
+ +T,
(2)
where d, denotes the depth in reference view, de denotes the depth in source view. R and t denote the rotation and translation between the reference and the source view, K-1 and K-- denote the intrinsic matrices of the reference and the source view. Let T = (tx. t²) K.-p, (Paz. Pay: Paz) PT and R. K pr (pre, Pry Prz) TT we can associate d, and d, with pixel coordinates:

---

Far-zs(Pr) 2 I yr vyrsys(pr)
(PenPr (taPsz tzPsa)/ PsmPrr PszPrø);
d,
(3)
(tyPaz t=Pay) / (DxyPr= Ds=Pry),
otherwise
F (pr) (p
(LePrz t=Pra)/ PsuPr= PszPre),
d,
(4)
(byPre - txPry) (PoyPrs PssPry),
otherwise
where 7 is a 2D flow vector along the epipolar line that provides flow in the dimension 2r-+zs (Pr) and y dimension gp (Pr). To obtain an appropriate initial position, we first determine the geometrically valid range along the epipolar line, which has not been considered in other works (53] [4]. If a point is observable in the current view, it must have physical significance, meaning must lie in front of the camera. Therefore, we identify the search range along the epipolar line on the source image that satisfies the condition d, > O, dg O. We obtain the initial position pa by selecting the mid-point in search range along epipolar line.
3.3 Disparity Hidden State Based Feature Encoding
Due to occlusion, moving objects, blurring, or other factors violating the multi-view geometry assumptions, the quality of sampling points from different source images varies, which limits the network's 's performance in depth estimation. To address this issue, we extract uncertainty information from the sampling point feature and encode it with cost volume as epipolar disparity feature Ey As shown in Fig. 2 we design the disparity hidden state H maintain the sampling information the current source image and update it during iterations by incorporating new uncertainty information.
Cost Volume Construction. For each source image, after determining the position p¹ for the current iteration, we uniformly sample M points around p¹, along the epipolar line at each scale with a distance of one pixel. By constructing 4-layer pyramid feature using average pooling, uniform pixel sampling at different levels allows for a larger receptive field. The sampling interval in 2D is fixed. Given image features F/ and { (F we obtain the features of M sampled points in the source image through interpolation and calculate the visual similarity. The cost volume V R xM is constructed by computing the dot product between pairs of image feature vectors:
N-1
Vi(p,) = (Fir (pr) FI, (p- +r)),
(5)
rEn
where R represents the set of sampling points uniformly sampled along the epipolar line the source image, and M denotes the number of sample points.
Disparity Feature Encoding with Uncertainty. When estimating the epipolar disparity flow from multi-view frames, it is essential to encode the differences between source images caused by variations in occlusion situations and image quality. Motivated by this, we conduct disparity hidden state H E R xxxxxC to explicitly represent the situation of point relative to the source image. Motivated by this, we introduce a disparity hidden state H E R
ff x x to explicitly represent the condition of points relative to the multi-view source images. H is randomly initialized and consecutively updated throughout the iterative process. We introduce a variance-base based uncertainty estimation module to encode the correlation features, which is formulated as follows:
U. =
Σ (V, (V)))
(6)
where V; denotes the cost volume of source image, V. denotes the average value of V,, and (.)) is the sigmoid function. Then, the uncertainty U,, the disparity hidden state of the previous iteration, the correlation features and the epipolar disparity flows are fed into the convolutional layers to generate epipolar disparity feature Fdd and update the disparity hidden state H1
3.4 Multi-view Stereo Transformer
DispMVS estimates the epipolar disparity flow from each two-frame image pair (To, 1, which overlooks the abundant multi-view information. Inspired by VideoFlow (11. estimate the epipolar
5

---

3D Pose Embedding
x
3D Pose Fmbedding
Disparity Feature F?
Normalized positional encoding
Disparity Feature F?
Figure 3: Illustration of MDA module. After concatenating features with 3D pose embedding and 2D normalized positional encoding, we achieve intra-image and inter-image information interaction through self-attention and cross-attention. As shown in the right figure, 3D pose embedding encodes relative pose and pixel geometric information into the features to enhance the learning capability of the attention mechanism.
flow of multi-view images simultaneously. However, since multiple source images are not sequentially arranged and points uniformly 2D sampled across source images can not establish robust 3D spatial correspondences, directly learning the continuity between flows, 21] does not work.
Therefore, unlike
etc., we design some special structures for information aggregation among multi-view images, Although the depths of sampled points along epipolar lines do not correspond, we observe that there is a regular pattern in the direction of depths along epipolar lines. As shown in Fig. 3 we design Multi-view Disparity Attention to learn the global information and utilize pose embedding to implicitly model the correspondence between pixel coordinates and depth on multiple source images, enabling the network to learn the direction and scale relationship of corresponding flows across different source images.
Multi-view Disparity Attention. To effectively capture extensive global information across epipolar disparity features from different views, we leverage the Multi-view Disparity Attention (MDA) module to further enhance the disparity features. We utilize an attention module to globally interact with disparity features of multi-view source images, thereby achieving multi-view feature fusion.
Given epipolar disparity features {FT)) N-1 Ji=i , we first use self-attention to achieve intra-image infor- mation interaction. We concatenate epipolar disparity features Fdd E Hx and set H x W the as sequence length L, generating pd e: IR (BNN-I)((HXW)xC
Then we use cross-attention achieve inter-frame information interaction and learn the relations among multi-view. We concatenate epipolar disparity features F and set the number source images N 1 the as sequence length L, generating Fd (PMMW)x(NNI)))
To reduce computation cost, for the self-attention we use linear transformer to compute attention, which replaces the original kernel function with:
Attention (Q, KV) = + (Q) ( ()))
where ((-) = relu(-) + and relu(-) represents the activation function of exponential linear units.
Pose Embedding. Due to the depths of sampling points varying for different source images, utilize pose embedding to construct implicit disparity relationships among multi-view frames. To effectively convey useful information to the attention module, categorize the features of pose embedding into two types: multi-view relative pose information and geometric information between specific sampled points. Fig. 3 iillustrtes the variables used to construct the pose embedding.
On one hand, the multi-view relative pose information between cameras contains crucial information about disparity features. By explicitly injecting relative poses into the attention module, the network can learn image-level geometric constraints. We represent the angle 00.i between rays as embedding. Inspired by (65), encode the rotation matrix and translation matrix between the reference and the source view into the relative pose distance pO.:
2 | + tr (I RO.5),
po.t
(8)
On the other hand, we encode the geometric information between specific sampled points. Due to our incorporation of pixel-level attention in addition to inter-frame attention, it is necessary to encode not

---

only image-level camera poses but also the pixel-level information corresponding to sampled points. It is important note that for each pixel in the reference image and its corresponding sampled point in the source image, we can obtain the corresponding 3D point coordinates P through triangulation based on stereo geometry. Accordingly, we encode the 2DD coordinates ps of the source image, the depth d, from the perspective of the reference image, and the depth d, from the perspective of the source image, thereby transforming the 3D information into corresponding relationships on the 2D plane. Moreover, we encode the normalized direction To, to the 3D location of point.
3.5 Iterative Updates
In the GRU updating process, we iteratively update the epipolar disparity flow e e + Ae, obtained from the MDA module for each source image. In each iteration, the input to the update operator includes 1) the hidden state; 2) the disparity feature output from the MDA module; 3) the current epipolar flow; and 4) the context feature of the reference image. The output of the update operator includes 1) new hidden state; 2) an increment to the disparity flow; and 3) the weight of disparity flow for multi-view images. We derive the depth from the disparity flow and employ a weighted sum to integrate the depth across multi-view source images. After fusion, the depth is converted back to disparity flow to perform the next iteration.
3.6 Loss Function
Similar to [14, we output depth after each iteration and construct the loss function accordingly. We construct the depth L1 loss. The loss function is represented in Eq. 9
loss Σ
norm (gt+) norm (d²)
(9)
j=testy D<=k<j
where te. t are iterations at the coarse and fine stage, y is a hyper- haaeeeep which is set to 0.9.
Experiments
In this section, we first introduce the datasets (Sec. 4.1. followed by the implementation details of the experiment (Sec. 4.2).. Subsequently, we delineate the experimental performance (Sec. 4.3) and
conduct ablation experiments validate the efficacy of each proposed module (Sec. 4.4
4.1 Datasets
DTU dataset 23 is an indoor multi-view stereo dataset captured in well-controleed laboratory conditions, which contains 128 different scenes with 49 views under 7 different lighting conditions. Following MVSNet [), we partitioned the DTU dataset into 79 training sets, 18 validation sets, and 22 evaluation sets. BlendedMVS dataset [66] is large-scale outdoor multi-view stereo dataset that contains a diverse array of objects and scenes, with 106 training scenes and validation scenes. Tanks and Temples 24 is public multi-view stereo benchmark captured under outdoor real-world conditions. It contains an intermediate subset of 8 scenes and an advanced subset of 6 scenes.
4.2 Implementation Details
Implemented by PyTorch 667, two models are trained on the DTU dataset and large-scale Blended- MVS dataset, respectively. On the DTU dataset, we set the image resolution as 640 x 512 and the number of input images
for the training phase. On the BlendedMVS dataset, set the image resolution as 768 x 576 and the number input images as 5 for the training phase. For all models, we use the AdamW optimizer with an initial learning rate of 0,0002 that halves every four epochs for 16 epochs. The training procedure is finished on two A with f = 8, ty = 2. For depth filtering and fusion, process 2D depth maps generate point clouds and compare them with ground truth.
4.3 Experimental Performance
In this section, we compare our method with other state-of-the-art methods and scale-agnostic methods. Existing methods are categorized into traditional methods (2, 35), 3D cost-e
7

---

Figure 4: Some qualitative results of the proposed method on DTU and Tanks and Temples datasets.
Table 1: The Quantitative point cloud evaluation results on DTU evaluation set. The lower the Accuracy (Acc), Completeness (Comp), Overall, the better. We split methods into four categories and highlight the best in bold for each.
Method
Comp-(mm)) ACC.(
Overall(mm)
Gipuma
0.578
COLMAP
35
0,400
0.664
0.532
MVSNet
8
0.396
0.527
0.462
RA-RMV
0.357
PatchmatchNet
53:
0.427
0.277
0.352
UniMVSNet
0,352
0.315
TransMVSNer 10
0.321
0,289
0.305
MVSTER 6X
0,340
0.266
0.303
GooMVS (
0.295
GBiNet 47
0.315
0,262
0.289
MVSForert+
0.281
BeeMVS 1
0.373
0.354
0.363
CER-MVS 70
0.359
0.305
0,332
RAMDepth 1B]
0.447
0.362
DispMVS 44
0.354
0.324
0.339
Ours
0.338
0.331
0.335
methods 8 53 9 68
47
I0:
69} 61,, RNN-based methods 13 70 and scale-agnostic methods
11 [4. Methods that leverage scene depth range have an advantage as they can utilize accurate and robust information, thereby mitigating outliers, especially textureless regions.
Evaluation on DTU. We evaluate the proposed method on the evaluation set of DTU dataset. We the image resolution as 1600 x 1152 and the number of input images as As shown in Table [ our method has the best overall performance among depth-range-fin -ree methods. CER-MVS [70] and MVSFormer++ [6]] demonstrate superior performance; however, they are heavily dependent on the accuracy of the depth range. Our approach outperforms when compared with depth fane--rree methods like DispMVS 14] and RAMDepth [L5), which demonstrates the effectiveness of our method in exploiting correlations among multi-view frames.
Evaluation on Tanks and Temples. Since the Tanks and Temples dataset does not provide training samples, we use a model pre-trained on the BlendedMVSS dataset for testing. We set the image resolution as 1920 1024 and the number of input images as for the evaluation phase. Table presents the comparison between our method and other oofte-of-theaar methods. Our method achieves the best performance among scale-agnen aggestii methods 14. Since RAMDepth [15] has not provided results on the Tanks and Temples dataset and source code, we are unable to make a comparison. Although our method exhibits certain gap when compared state-of-the-art methods [70] 61 based on precise depth priors, it demonstrates superior robustness across a broader depth range.
We visualize point clouds generated on DTU and Tanks and Temples dataset in Fig. which demonstrates that our method is capable of constructing a comprehensive and precise point cloud.
4.4 Ablation Study
In this subsection, we conduct ablation studies of our model trained on DTU 23 datasets to discuss the effectiveness of core parts of our method. The implemented baseline basically based on DispMVS [4]. All the experiments are performed with the same hyperparameters.

---

Table 2: The Quantitative point cloud evaluation results Tanks and Temples benchmark. The metric is F-score and "Mean" refers to the average F-score of all scenes (higher is better). We split methods into four categories and highlight the best in bold for each.
advanced
intermediate
Method
Mean Aud. Bal.
Cou.
Mus.
Pul.
Tem. Mean Far..
Fru.
Hor.
Lig.
M60
Pan.
Pla.
Tra.
COL COLMAP
15
27.24 16.02
25.23
34.70
41.51
18.05
27.94
42.14
50.41
22.25
26.63
56.43
44
46.97
48.53
42.04
MVSNet g
43.48
55,
55.99
28.55
25.07
50.79
53,
50,86
47.90
34,69
PatchmatchNet
32.31
23.09
37.73
30.04
40.80
28.31
32.29
53.15
66.99
52, 54.64
43.
54.87
87
52
49.
54.21
50.81
AA-RMVSNet
9
33.53
20.96 40.15 52.05
46.01
29.28
32.71
61.51
77.77
59.53
51.53
64
64.02.0
64.05
59.47
60.85 54,90
MVSTER (68)
37,53
26.68 42.14
65
49.37
32.16
39.19
60.92
80.21 63.51 S2.
61.38
61
47
58.16
58.98 51.38
GBi-Net
37.32
29.77 42.12
36.30
36.93
79.77
67, 67.69
81
25
60
85
60.67 53,89
IransMVSNet
10
37,00
24.84 44.59
34.77
46.49
34,69
36.62
63.52
80.92 65 83 56.
94
62.54
63.06
60.00
60.20 58.67
GeoMVSNet [9
41.52
39.98
65.49
MVSForme++
41.70 6T
30.39
45.85
39.35
53.62
35.34
45.66
67.03
82.87
68.90
64.21
68.65
65
66.43
60.07 60.12
IterVV 0
33.24 22.95 38.74 30.64
43, 44
28.39
35,27
94
76.12
55,
50.55 53 56.05
57, 68 52, 62 55,70 50.99
CER-MVS 70
40.19
39.65
51.75
42.97
64.21
63.99
DispMVS
L4
34.90
26.09
38.01
33.19
44,90 28.49 38.75
59.07
74.73
60.67
54.13
59.58
58. $3.39 58.63 53.42
Ours
35.95 26.75 40.22 33.87 45.78 29.58 39,50 59.27 75.05 61.63 53.15 60.24 58.44 53.34 58.79 53.54
Table 3: Quantitative evaluation the effectiveness of each component of the model was conducted on the DTU dataset. The lower the Accuracy (Acc), Completeness (Comp), Overall, the better.
Pose Embedding
Uncertainty
Disparity Hidden State
Acc.fmm))
Comp.(mm),
Overall(mm)]
0.363
0.382
0,372
0.356
0.384
0.370
0.354
0,362
0,336
0.378
0.357
0.338
0.331
0,335
Pose Embedding. We conducted ablation experiments validate the effectiveness of the pose embedding. Specifically, within the multi-view attention module, we remove 3D pose embedding and retain only the original 2D) position encoding of attention. As shown in Table 3. after applying pose embedding, the overall performance improves by 9.46%. The result indicates that the current task heavily relies on the relative pose and geometric information contained in the pose embedding. Without incorporating geometric constraints across multi-view source images, typically achieved through pose embedding, the performance of Transformer in this task may degrade significantly.
Disparity Feature Encoding coupling with Uncertainty. Following [4], we further attempt to remove uncertainty estimation and disparity hidden state and directly perform feature encoding on disparity flow and cost volume. As shown in Table 3] with the disparity feature encoding coupling with uncertainty, the overall performance improves by 9.95%. The result validates the effectiveness of the module, demonstrating that explicitly estimating the quality of sampled points on the epipolar line of source images and updating the disparity hidden state in the network is effective. Additionally, we designed two separate ablation experiments, removing the uncertainty and disparity feature hidden states, to further evaluate the impact of these two modules on the network. The uncertainty and disparity feature hidden states improved the overall performance by 6.16% and 7.46%, respectively. Compared to the performance without disparity feature encoding coupling with uncertainty, this demonstrates the effectiveness of the uncertainty and disparity feature hidden state updating modules.
4.5
Depth Range
In this section, we compare the generalization of different networks to depth range. We don't 't compare with RAMDepth [5] in the ablation studies due to the lack of its source code. We select several
state-of-the -ht----artttt methods (GeoMVS [69], MVSFormer++ [61), RNN-base methods (IterMVS [3] CER-MVS 70)) and depth--ange--free method (DispMVS (14) to conduct experiments to evaluate the generalization of depth range. Our main comparison is with depth-range-free methods [14), which reduce dependence on depth priors through network design. All methods are trained on DTU dataset with the same depth range and subsequently inference under different depth ranges.

---

Table 4: Quantitative evaluation of the sensitivity of methods to depth range. The results for Accu- racy (Acc), Completeness (Comp), and Overall presented in millimeters (mm).
Table 5: The influence of the depth range ob- tained from COLMAP of methods. The results for Accuracy (Acc), Completeness (Comp), and Overall are presented in millimeters (mm).
Depih Range Method
Acc.J.
Comp.
Overall
Depih Range Method
Acc.j.
Comp-
Overall
lierMVS
13
0.373
0.354
0.363
lierMVS
13
0.373
0.354
0.363
0.354
0.324
0.339
0.354
0.324
0.339
(425mm.
CER CER-MVS 70
0.359
0.305
0.332
GT
CER-MV
70
0.359
0.305
0.332
935mmm)
(69
0.331
0.259
0.295
Ge
0.331
0.259
0.295
MVSFormen++
7
0.309
0.252
0.281
MVSFormer+
2
0.309
0.252
0.281
Ours
0,338
0.331
0.335
Ours
0.338
0.331
0.335
13r
0.532
1,471
1.002
leeMM 13
0.454
1.486
0.970
Dis
0.348
0.404
0.376
Disp
0.339
0.372
0.356
(212.5mm,
CER TTR.MVSPD 70
1.805
1.161
1,483
COLMAP
CER MVS 20
0,816
0.326
0.571
1870mm)
Ge
0.435
0.619
0.527
GeoMVS (9
0.374
0.415
0,394
MVSFormer++
0:478
0.359
MVSFormer++
0.319
0.340
Ours
0,338
0.331
0.335
Ours
0.338
0.331
0.335
TieMMS 13
0.935
6,985
3,960
DispMVS 14
0.314
0.671
0.493
(141 66mm.
CER-MVS 20
11:464
12.683
12.073
2805mm)
(e6
0.602
1.663
1.133
MVSFormer++
0.739
0.820
0.780
Ours
0.338
0.331
0.335
61 that rely on depth range prior for depth sampling, whether based on RNN Transformer, they may exhibit better performance with accurate depth priors. However, as shown in Table there is a marked decline in performance for these methods with larger depth range. Although DispMVS (14) showed insensitivity depth range, its performance still exhibited certain degree of decline with larger depth ranges. In contrast, our method, which is independent of depth range, maintained consistent performance regardless of changes in depth range.
For methods 13 70
69
crucial to emphasize that the depth range provided by the dataset is exceptionally accurate. For instance, the ground truth for the Tanks- aemellss dataset is captured using industrial laser scanner. However, in practical applications, while Stracture-from-Mootie depth ranges from sparse feature points, the resulting depth estimates are often prone inaccuracies. These inaccuracies arise from the inherent sparsity of feature points, as well as challenges such as occlusion and suboptimal viewpoint selection. To verify the robustness of the MVS models in practical applications, we use the depth range obtained from COLMAP to replace the depth range ground truth (GT). As shown in Table 5] there is a significant decline in performance for GeoMVS [69]. MVSFormer++ [61, IterMVS [3] and CER-MVS 77 when we use the depth range obtained from COLMAP. DispMVS 14 also exhibits certain degree of decline. In contrast, our method maintained consistent performance. This result further demonstrates the necessity of eliminating the depth range.
(SfM) can derive
Conclusion
We propose prior-free multi-view stereo framework that simultaneously considers all the source images. To fully fuse the information from disordered and arbitrarily posed source images, we propose t---oo---mbeddding aided and sivertaint--rivver transformer- based network. Extensive experiments show that our methods achieve state-of-the-arr performances among the prior-free methods and exhibit greater robustness the depth range prior. Limitations: The proposed method cannot run in real-time (i.e., 30 FPS), which could limit its application in mobile devices or other time-sensitive scenarios. Besides, our method shows a performance gap compared SOTA cost- volume-based methods on the mainstream benchmark, despite these methods relying on highly precise depth range priors. In the future work, hope close the gap.
Acknowledgement
This work was partially supported by NSF of China (No. 61932003).
10

---

References
[1] Robust Multiview Stereopsis. Accurate, dense, and robust multiview stereopsis. JEEE TRANS- ACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, 32(8), 2010.
[2] Silvano Galliani, Katrin Lasinger, and Konrad Schindler. Massively parallel multiview stereopsis by surface normal diffusion. In Proceedings of the IEEE International Conference on Computer Vision, pages 873-881, 2015.
[3] Johannes L Schönberger, Enliang Zheng, Jan-Michael Frahm, and Mare Pollefeys. Pixelwise view selection for unstructured multi-view stereo. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 1 11-14, 2016, Proceedings, Part III 14, pages 501-518. Springer, 2016.
[4] Qingshan Xu and Wenbing Tao. Multi-scale geometric consistency guided multi-vied stereo. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5483 5492, 2019.
[5] Andrea Romanoni and Matteo Matteucci. Tapa-mvs: Textureless-awar patchmatch multi-view stereo. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10413-10422, 2019.
[6] Qingshan Xu and Wenbing Tao. Planar prior assisted patchmatch multi-view stereo. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 12516-12523, 2020.
[7] Xiaodong Gu, Zhiwen Fan, Siyu Zhu, Zuozhuo Dai, Feitong Tan, and Ping Tan. Cascade cost volume for high-resolution multi view stereo and stereo matching. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 2495-2504, 2020.
[8] Yao Yao, Zixin Luo, Shiwei Li, Tian Fang, and Long Quan. Mvsnet: Depth inference for unstructured multi-view stereo. In Proceedings of the European Conference on Computer Vision (ECCV), September 2018.
[9] Zizhuang Wei, Qingtian Zhu, Chen Min, Yisong Chen, and Guoping Wang. Aa-rmvsnet: Adaptive aggregation recurrent multi-view stereo network. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pages 6187-6196, October 2021.
[10] Yikang Ding, Wentao Yuan, Qingtian Zhu, Haotian Zhang, Xiangyue Liu, Yuanjiang Wang, and Xiao Liu. Transmvsnet: Global context-aware multi-view stereo network with transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8585-8594, June 2022.
[11] Shuo Cheng, Zexiang Xu, Shilin Zhu, Zhuwen Li, Li Erran Li, Ravi Ramamoorthi, and Hao Su. Deep stereo using adaptive thin volume representation with uncertainty awareness. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
[12] Robert T Collins. A space-sweep approach to true multi-image matching. In Proceedings CVPR IEEE computer sociery conference on computer vision and pattern recognition, pages 358-363. leee, 1996.
[13] Fangjinhua Wang, Silvano Galliani, Christoph Vogel, and Mare Pollefeys. Itermvs: Itera- tive probability estimation for efficient multi-view stereo. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8615, June 2022.
[14] Qingsong Yan, Qiang Wang, Kaiyong Zhao, Bo Li, Xiaowen Chu, and Fei Deng. Rethinking disparity: a depth range free multi-view stereo based on disparity. Proceedings of the AAAI Conference Artificial Intelligence, volume 37, pages 3091-3099, 2023.
[15] Andrea Conti, Matteo Poggi, Valerio Cambareri, and Stefano Mattoccia. Range-agnostic multi-view depth estimation with keyframe selection. arXiv preprint 2xXiv:2401.1400// 2024.
11

---

[16] Zhaoyang Huang, Xiaoyu Shi, Chao Zhang, Qiang Wang, Yijin Li, Hongwei Qin, Jifeng Dai, Xiaogang Wang, and Hongsheng Li. Flowformer: transformer architecture and its masked cost volume autoencoding for optical flow. arXiv preprint arXiv:2300. 05442, 2023.
[17] Junjie Ni, Yijin Li, Zhaoyang Huang, Hongsheng Li, Hujun Bao, Zhaopeng Cui, and Guofeng Zhang. Pats: Patch area transportation with subdivision for local feature matching. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17776-17786, 2023.
[18] Weikang Bian, Zhaoyang Huang. Xiaoyu Shi, Yitong Dong, Yijin Li, and Hongsheng Li. Context-pips: Persistent independent particles demands context features. Advances in Neural Information Processing Systems, 36, 2024.
[19] Hongjia Zhai, Xiyu Zhang, Zhao Boming, Hai Li, Yijia He, Zhaopeng Cui, Hujun Bao, and Guofeng Zhang. Splatloe: 3d gaussian splatting- -hased visual localization for augmented reality. arXiv preprint arXiv:2409.14067, 2024.
[20] Junjie Ni, Guofeng Zhang, Guanglin Li, Yijin Li, Xinyang I. Liu, Zhaoyang Huang, and Hujun Bao. Eto: Efficient transformer-based local feature matching by organizing multiple homography hypotheses. arXiv preprint
2024.
[21] Xiaoyu Shi, Zhaoyang Huang, Weikang Bian, Dasong Li, Manyuan Zhang, Ka Chun Cheung, Simon See, Hongwei Qin, Jifeng Dai, and Hongsheng Li. Videoflow: Exploiting temporal cues for multi-frame optical flow estimation. arXiv preprint 2xXiv:2303.08340 2023.
[22] Michal Neoral, Jan Sochman, and Jifí Matas. Continual occlusion and optical flow estimation. In Asian Conference on Computer Vision, pages 159-174. Springer, 2018
[23] Henrik Aanæs, Rasmus Ramsbøl Jensen, George Vogiatzis, Engin Tola, and Anders Bjorholm Dahl. Large-scale data for multiple-view stereopsis. International Journal of Computer Vision, 120:153-168 168, 2016.
[24] Arno Knapitsch, Jaesik Park, Qian- Yi Zhou, and Vladlen Koltun. Tanks and temples: Bench- marking large-scale scene reconstruction. ACM Transactions on Graphics (ToG), 36(4):1-13, 2017.
[25] Xinyang Liu, Yijin Li, Yanbin Teng, Hujun Bao, Guofeng Zhang, Yinda Zhang, and Zhaopeng Cui. Multi-modal neural radiance field for monocular dense slam with a light-weight tof sensor. In Proceedings of the ieee/cvf international conference on computer vision, pages 1-11, 2023.
[26] Zhaoyang Huang, Han Zhou, Yijin Li, Bangbang Yang, Yan Xu, Xiaowei Zhou, Hujun Bao, Guofeng Zhang, and Hongsheng Li. Vs-net: Voting with segmentation for visual localization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6101-6111, 11, 2021.
[27] Shuo Chen, Mao Peng, Yijin Li, Bing-Feng Ju, Hujun Bao, Yuan-Liu Chen, and Guofeng Zhang. Multi-view neural 3d reconstruction of micro-and nanostructures with atomic foree microscopy. Communications Engineering, 3(1):131, 2024.
[28] Zhichao Ye, Chong Bao, Xin Zhou, Haomin Liu, Hujun Bao, and Guofeng Zhang. Ec-sfm: Efficient covisibility-base structure-from www.--rooooonnono motion for both sequential and unordered images. IEEE Transactions on Circuits and Systems for Video Technology, 34(1): 1et:1100123331 123, 2023.
[29] Xiaoliang Ju, Zhaoyang Huang, Yijin Li, Guofeng Zhang, Yu Qiao, and Hongsheng Li. Diffindscene: Diffusion-based high-quality 3d indoor scene generation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 4526-4535,
2024.
[30] Bangbang Yang, Yinda Zhang, Yijin Li, Zhaopeng Cui, Sean Fanello, Hujun Bao, and Guofeng Zhang. Neural rendering in a room: amodal 3d understanding and free-viewpoint rendering for the closed scene composed of pre-captured objects. ACM Transactions on Graphics (TOG). 41 (4):1-10, 2022.
12

---

[31] Steven M Seitz and Charles R Dyer. Photorealistic scene reconstruction by voxel coloring. International Journal of Computer Vision, 35:151-173, 1999.
[32] Ilya Kostrikov, Esther Horbert, and Bastian Leibe. Probabilistic labeling cost for high-accuracy multi-view reconstruction. In Proceedings of the ieee conference on computer vision and pattern recognition, pages 1534-1541, 2014.
[33] Maxime Lhuillier and Long Quan. A quasi-dense approach to surface reconstruction from uncalibrated images. JEEE transactions on pattern analysis and machine intelligence, 27(3): 418 433, 2005.
[34] Yasutaka Furukawa and Jean Ponce. Accurate, dense, and robust multiview stereopsis. IEEE transactions on pattern analysis and machine intelligence, 3322:133 2009.
[35] Johannes L Schonberger and Jan-Michael Frahm. Structure- froene--frommmee motioon mrotmmoooo revisited. In Proceed- ings of the IEEE conference on computer vision and pattern recognition, pages 4104 4113, 2016.
[36] Keyang Luo, Tao Guan, Lili Ju, Yuesong Wang. Zhuo Chen, and Yawei Luo. Attention-awars multi-view stereo. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
[37] Hongwei Yi, Zizhuang Wei, Mingyu Ding, Runze Zhang, Yisong Chen, Guoping Wang, and Wing Tai. Pyramid multi-view stereo net with self-adaptive view aggregation. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part IX 16, pages 766-782. Springer, 2020.
[38] Keyang Luo, Tao Guan, Lili Ju, Haipeng Huang, and Yawei Luo. P-mvsnet: Learning patch- wise matching confidence aggregation for view stereo. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
[39] Jingyang Zhang. Yao Yao, Shiwei Li, Zixin Luo, and Tian Fang. Visibility- multi-view stereo network, 2020.
[40] Qingshan Xu and Wenbing Tao. Pysnet: Pixelwise visibility aware multi-view stereo network, 2020.
[41] Rui Peng, Rongjie Wang, Zhenyu Wang, Yawen Lai, and Ronggang Wang. Rethinking depth estimation for multi-view stereo: A unified representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8645-8654, June 2022.
[42] Youze Xue, Jiansheng Chen, Weitao Wan, Yiqing Huang, Cheng Yu, Tianpeng Li, and Jiayu Bao, Mvserf: Learning multi-view stereo with conditional random fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
[43] Xinjun Ma, Yue Gong, Qirui Wang, Jingwei Huang, Lei Chen, and Fan Yu. Epp-mvsnet: Fpipolar-assembling based depth prediction for multi-view stereo. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 5732-5740, October 2021.
[44] Jiayu Yang, Wei Mao, Jose M. Alvarez, and Miaomiao Liu. Cost volume pyramid based depth inference for multi-view stereo. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
[45] Xiaodong Gu, Zhiwen Fan, Siyu Zhu, Zuozhuo Dai, Feitong Tan, and Ping Tan. Cascade cost volume for high-resolution multi-view stereo and stereo matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
[46] Zehao Yu and Shenghua Gao. Fast-mvsnet: Sparse-to-dense multi-view stereo with learned propagation and nauss-newton refinement. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
[47] Zhenxing Mi, Chang Di, and Dan Xu. Generalized binary search network for highly-efficient milti-view stereo. Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pages 12991-13000 June 2022.
13

---

[48] Xingjian Shi, Zhourong Chen, Hao Wang, Yan Yeung, Wii-Kin Wong, and Wang-chun Woo. Convolutional Istm network: A machine learning approach for precipitation noweasting. Advances in neural information processing systems, 28, 2015.
[49] Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder- decoder for statistical machine translation, 2014.
[50] Jin Liu and Shunping Ji. A novel recurrent encoder-decoder structure for large-scale multi-view stereo reconstruction from an open aerial dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
[51] Yao Yao, Zixin Luo, Shiwei Li, Tianwei Shen, Tian Fang, and Long Quan. Recurrent mvsnet for high-resolution multi-view stereo depth inference. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
[52] Jianfeng Yan, Zizhuang Wei, Hongwei Yi, Mingyu Ding, Runze Zhang, Yisong Chen, Guoping Wang, and Yu-Wing Tai. Dense hybrid recurrent multí-view stereo net with dynamic consistency checking. In European conference on computer vision, pages 674-689. Springer, 2020.
[53] Fangjinhua Wang, Silvano Galliani, Christoph Vogel, Pablo Speciale, and Marc Pollefeys. Patchmatchnet: Learned multi-view patchmatch stereo. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14194 18194-14205 14203, June 2021.
[54] Michael Bleyer, Christoph Rhemann, and Carsten Rother. Patchmatch stereo-stereo matching with slanted support windows. In Bmve, volume 11, pages 1-11, 2011.
[55] Qingtian Zhu, Chen Min, Zizhuang Wei, Yisong Chen, and Guoping Wang. Deep learning for multi-view stereo via plane sweep: A survey. arXiv preprint arXiv:2106.15. 2021.
[56] Kai Zhang, Sai Bi, Hao Tan, Yuanbo Xiangli, Nanxuan Zhao, Kalyan Sunkavalli, and Zex- iang Xu. Gs-Irm: Large reconstruction model for 3d gaussian splatting. arXiv preprint arXiv:2404. 19702, 2024.
[57] Yijin Li. Zhaoyang Huang, Shuo Chen, Xiaoyu Shi, Hongsheng Li. Hujun Bao, Zhaopeng Cui, and Guofeng Zhang. Blinkflow: A dataset to push the limits of event-basee optical flow estimation. In 2023 IEEE/RSJ International Conference on Intellient Robots and Systems (IROS), pages 3881 1888.. IEEE, 2023.
[58) Yijin Li, Yichen Shen, Zhaoyang Huang, Shuo Chen, Weikang Bian, Xiaoyu Shi, Fu-Yun Wang, Keqiang Sun, Hujun Bao, Zhaopeng Cui, Guofeng Zhang, and Hongsheng Li. Blinkvision: A benchmark for optical flow, scene flow and point tracking estimation using rgb frames and events. In European conference on computer vision. Springer, 2024.
[59] Yijin Li, Xinyang Liu, Wenqi Dong, Han Zhou, Hujun Bao, Guofeng Zhang, Yinda Zhang, and Zhaopeng Cui. Deltar: Depth estimation from a light- tof sensor and rgb image. In European conference on computer vision, pages 619-636. Springer, 2022.
[60] Chenjic Cao, Xinlin Ren, and Yanwei Fu. Mvsformer: Multi-view stereo by learning robust image features and temperature-based depth. arXiv preprint arXiv:2208.09 02541, 2022.
[61] Chenjie Cao, Xinlin Ren, and Yanwei Fu, Mysformer++: Revealing the devil in transformer's details for multi-view stereo. arXiv preprint arXiv:2401.1167. 2024.
[62] Jie Zhu, Bo Peng, Wanqing Li, Haifeng Shen, Zhe Zhang, and Jianjun Lei. Multi-view stereo with transformer. arXiv preprini 1rXiv:2112.003366 2021.
[63] Tianqi Liu, Xinyi Ye, Weiyue Zhao, Zhiyu Pan, Min Shi, and Zhiguo Cao. When epipolar constraint meets non-local operators in multi-view stereo. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 18088-18097 2023.
[64] Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In Computer Vision-ECCV 2020: 16th Furopean Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part II 16, pages 402-419. Springer, 2020.
14

---

[65] Arda Duzceker, Silvano Galliani, Christoph Vogel, Pablo Speciale. Mihai Dusmanu, and Mare Pollefeys. Deepvideomvs: Multi-view stereo on video with recurrent Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15324-15333, 2021.
fusion. In
[66] Yao Yao, Zixin Luo, Shiwei Li, Jingyang Zhang, Yufan Ren, Lei Zhou, Tian Fang, and Long Quan. Blendedmvs: large-scale dataset for generalized multi-view stereo networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR),
June 2020.
[67] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
[68] Xiaofeng Wang. Zheng Zhu, Guan Huang, Fangbo Qin, Yun Ye, Yijia He, Xu Chi, and Xingang Wang. Mvster: Epipolar transformer for efficient multi-view stereo. In European Conference on Computer Vision, pages 573-591. Springer, 2022.
[69] Zhe Zhang, Rui Peng, Yuxi Hu, and Ronggang Wang. Geomvsnet: Learning multi-view stereo with geometry perception. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 21508 21518, 2023.
[70] Zeyu Ma, Zachary Teed, and Jia Deng. Multiview stereo with cascaded epipolar raft. In European Conference on Computer Vision, pages 734-750. Springer, 2022.
[71] Chenjie Cao, Xinlin Ren, and Yanwei Fu, Mysformer++: Revealing the devil in transformer's details for multi-view stereo. arXiv preprint arXiv:2401.11673, 2024.
15

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We claim the contributions and scope of this paper in the abstract and introduc- tion, specifically in line 4 and 22.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
In Section Conclusion, we discuss the limitations of this work.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
16

---

We present the theoretical derivation of the depth range in Section 3.2 and the Appendix.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
The paper fully disclose all the experimental information in Sec.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
17

---

Answer:
[No]
Justification:
We plan to release the code and detailed results later.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm
ceriiinnolccyc ic for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ ccpppp.ccppulli//uuuii wwwtc//ggue////eeeeeeoocc CoodeSubeiiieeee ssi wsionnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper specify all the training and test details in Sec. 4.2
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification:
The experiments in this paper do not require error bars to test the validity.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
18

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The paper provide sufficient information on the computer resources in Sec.
422
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http:///wwwrppppppccicc//ee/eeae
Answer: [Yes]
Justification: The research conducted in the paper conforms with the NeurIPS Code of Ethics
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., . if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no societal impact of the work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
19

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA].
Justification: The paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Yes, the creators or original owners the assets (e.g., code, data, models) used in the paper properly credited. The citations for these assets can be found in Sec. and Sec.
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version the asset used and, possible, include URI.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
20

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [No]
Justification:
We currently organizing assets and will release them together later.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moddl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
21


Reviews:
[{"review_id": "01LmHpQpHB", "reviewer": "Reviewer_RLMs", "summary": "The paper describes an MVS approach that does not depend on given depth ranges, which MVS algorithms typically require when building 3D cost volumes. The solution, previously proposed in DispMVS, is to perform searching and iterative updates in disparity space. Compared to DispMVS, the authors propose several novel designs to further eliminate the dependency on depth range during initialization, as well as to allow synchronous updates from all source views.", "questions": "I look forward to answers to the questions above regarding clarity, generalization and performance.", "limitations": "The authors recognize speed as a limitation. The performance gap behind cost-volume counterparts is also worth mentioning.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "**Significance**: The complete elimination of the dependency on depth ranges brings convenience and avoids performance degradation due to improper ranges. These benefits may enable wider adoption of MVS in real-world applications. \n\n**Originality**: Although the paper largely follows the foundation built in DispMVS, the several improvements are novel and nontrivial. The combination of 3D pose embedding and cross-attention as a solution for synchronous updates among all source views is quite interesting and proven effective. \n\n**Quality**: The results show consistent improvements over other depth-range-free options. The ablation studies and the analysis of impact from varying depth ranges help make the overall design more convincing.", "weaknesses": "**Clarity**\nThe paper has some rather noteworthy issues regarding writing. The method section needs to have the right level of detail. The approach is built on top of DispMVS and shares many concepts and details, and the authors' choices to keep and omit details seem somewhat arbitrary, resulting in a non-self-contained method section. For example, there are also some concepts used without introduction, e.g. \"sampling range\" on L.168, \"t_c, t_f\" in Eq.11. The GRU updating process isn't defined. The relationship between epipolar disparity flow and depth is a critical concept that the paper does not explain. \n\nThe resulting poor readability is potentially a significant issue, but unfortunately, it is hard to address in a rebuttal. I'd like to see how other reviewers weigh its impact. \n\n**Performance**\nWhile beating other depth-range-free solutions, including DispMVS, on the two benchmarks, results still need to catch up on cost-volume approaches like GBi-Net, often by a large margin. It'll be quite valuable to understand what causes such discrepancy. Is it due to the limited receptive field, search gratuity, or else? Can improving on any of these help close the gap? \n\n**Generalization**\nThere is no discussion of generalization capability in the paper. Conceptually, it seems a depth-range-free approach would be more generalizable due to being invariant to object/scene scales. The ability to train a single model that operates in various use cases (indoor, outdoor, etc.) is arguably the most desirable feature of such approaches.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> \"sampling range\" on L.168.\n\nThanks for your reminder. Following DispMVS, the \"sampling range\" on L.168 refers to the set of sampling points uniformly sampled along the epipolar line.\n\n> \"$t_c$, $t_f$\" in Eq.11.\n\non L.163, \"$t_c$, $t_f$\" are iterations at the coarse and fine stage.\n\n> The GRU updating process isn't defined.\n\nFor GRU updating process, the epipolar flow is iteratively updated for each source image. In each iteration, the input to the update operator includes the hidden state, the disparity feature output from MDA module, the current epipolar flow, and the context feature of the reference image. The output of the update operator includes a new hidden state, an increment to the disparity flow, and the weight. We get the depth from disparity flow and utilize a weighted sum to the depth from multi-view situation. After fusion, the depth is converted back to disparity flow to perform the next iteration.\n\n> The relationship between epipolar disparity flow and depth is a critical concept that the paper does not explain.\n\nFor the relationship between epipolar disparity flow $e_i$ and depth $d_0$, we can obtain the position $p_i$ in the source image by adding epipolar disparity flow $e_i$ to initial position $p_i^0$, then we can get $d_0$ by triangulation as Eq.3. Similarly, after obtaining $d_0$, we get the position $p_i$ by Eq.2, and subtract the initial position $p_i^0$ to obtain $e_i$. We will clarify this relationship more clearly in paper.\n\n> Results still need to catch up on cost-volume approaches like GBi-Net, often by a large margin. Is it due to the limited receptive field, search gratuity, or else? Can improving on any of these help close the gap?\n\nCompared to other methods that uniformly sample depth based on a depth range, our network requires a more powerful retrieval capability to regress the correct depth due to the lack of depth range priors and outperform known depth-free methods. There are two main reasons for the discrepancy between our method and cost-volume approaches.\n\nOne reason is the search gratuity. Although our method addresses the receptive field problem to a large extent by sampling points along the epipolar line at features with different scales, iterating over the depth range of $(0,\\infty)$ significantly increases the search gratuity compared to uniformly sampling within a predefined depth range.\n\nThe second reason is the development potential of datasets for deep learning network. Existing MVS datasets, such as DTU, have a relatively uniform depth distribution, mostly around a mean depth of 600. This allows methods directly based on a narrow predefined depth range to achieve precise convergence, limiting the advantage of our method. However, in real-world scenarios, there are many scenes with a wide depth distribution, such as near and far objects, where the background cannot be crudely masked out like in the DTU dataset. In such cases, depth cannot be recovered with a narrow depth prior, necessitating an expanded depth search range, which increases the difficulty of convergence inevitably. I think enhancing accuracy over a large depth range is a crucial problem that MVS must address in the future. Currently, our ongoing work focuses on optimizing the acquisition of initial values and constructing more diverse datasets to endow the network with stronger learning capabilities.\n\n> There is no discussion of generalization capability in the paper.\n\nTo evaluate the generalization of our method, we used an iPad to capture data and add inference experiments in multiple scenes. We collected images from various real-world environments, with intrinsic and extrinsic parameters obtained by running COLMAP. As shown in Fig. 5, our model is capable of generating dense point cloud reconstructions for both indoor and outdoor environments. However, the accuracy of these reconstructions is somewhat lacking, which is affected by the following three factors:\n\n\nInaccurate Camera Pose: The pose estimation from COLMAP is far from satisfying. In contrast, the evaluation benchmark provides more accurate camera pose. For example, DTU uses a structured light scanner and MATLAB calibration toolbox for camera pose estimation. The inaccurate camera pose can lead to large error during the MVS process.\n\nImage Quality Issues: As illustrated in Fig. 6, issues such as overexposed, inadequate lighting and blurriness affect the reconstruction quality. These deficiencies in image quality contribute to the observed inaccuracies in the point cloud.\n\n\nTraining Data Limitations: Our model was trained on the DTU dataset, which is relatively small and features a narrow range of scenes. While our model can effectively mitigate depth range effects in various environments, it still struggles with fine detail accuracy. The limited diversity of the DTU dataset constrains the model's ability to capture detailed features accurately. Constructing an MVS dataset with diverse scenes is a promising approach to enhance the robustness and accuracy of point cloud reconstructions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lS30XUB6ul", "reviewer": "Reviewer_heJr", "summary": "The paper proposes a depth-rage free method for MVS. It leverages the transformers for designing a global-aware model, with using pose positional embedding to guide the model and also predict the uncertainty at the same time. The methods demonstrates good performance on diverse datasets and is also robust to different depth range.", "questions": "Please see the weakness section.", "limitations": "The scale of the experiment is not big enough. It would be great if the authors could perform larger-scale experiments, ideally Dust3r-level scales, to understand the scaling capability of the proposed method. I can understand it is not easy to acquire more resources for training, but it would be great if the authors could have more discussions on this.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The idea of building global-aware models using transformers makes sense.\n- I like the idea of injecting inductive bias using pose positional embedding and the uncertainty method.\n- The proposed method demonstrates satisfying experimental results and robustness to different depth range.", "weaknesses": "- Related work. Using transformers for building global-aware model is very common for 3D reconstruction now [1,2,3]. I would be good if the authors could discuss these related works.\n\n[1] Wang, Peng et al. “PF-LRM: Pose-Free Large Reconstruction Model for Joint Pose and Shape Prediction.” ICLR 2024. [2] Jiang, Hanwen et al. “LEAP: Liberate Sparse-view 3D Modeling from Camera Poses.” ICLR 2024. [3] Zhang, Kai et al. “GS-LRM: Large Reconstruction Model for 3D Gaussian Splatting.” ArXiv 2024.\n\n- Dense view inputs. The experiments are performed in the sparse-view setting where the computation of transformers is not a big problem. Is the method also efficient in the dense view setting? If not, I hope the authors could have more discussions in the paper.\n\n- I would say that the gains in the ablation experiments are not that significant.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Using transformers for building global-aware model is very common for 3D reconstruction now [1,2,3]. I would be good if the authors could discuss these related works.\n\nThank you for your suggestion. We will discuss the corresponding references in the related work section.\n\n[1] Wang, Peng et al. “PF-LRM: Pose-Free Large Reconstruction Model for Joint Pose and Shape Prediction.” ICLR 2024.\n\n[2] Jiang, Hanwen et al. “LEAP: Liberate Sparse-view 3D Modeling from Camera Poses.” ICLR 2024.\n\n[3] Zhang, Kai et al. “GS-LRM: Large Reconstruction Model for 3D Gaussian Splatting.” ArXiv 2024.\n\n> Is the method also efficient in the dense view setting?\n\nFor images with a resolution of 512x640, our method can handle up to 77 views in V100 during testing phase, and up to 5 views simultaneously during training phase. Using the model trained on the DTU dataset with 5 views, we verify the impact of the number of views on system accuracy. Due to DTU providing co-visible relationships for only 10 frames, we selected 10 frames as the upper limit. The results are shown in the following table, which uses 2D metric. It can be observed that the error decreases as the number of used views increases and gradually coverage. Increasing the view during training may further increase the performance after convergence.\n\n|             | 3 views | 4 views | 6 views | 8 views |\n| :---------- | :-----: | :-----: | ------- | ------- |\n| Depth Error |  5.375  |  5.108  | 4.964   | 4.967   |\n\n\n> The gains in the ablation experiments are not that significant.\n\nWe respectfully argue that the gain of adding the proposed components is large enough. For example, the performance gain of adding uncertainty is 9.95\\% and that of adding pose embedding is 9.46\\%. Besides, we also test the gain in the 2D metric (absolute depth error), the gain of adding uncertainty and pose embeddings is 1.46\\% and 35.62\\%, respectively.\n\n> Perform larger-scale experiments, ideally Dust3r-level scales, to understand the scaling capability of the proposed method.\n\nWe collected some real-world data to test our method in more large-scale environment and also test its generalization. The camera intrinsic and camera pose are both obtained by running COLMAP. As shown in Fig. 5, our model is capable of generating dense point cloud reconstructions for the collected data, which shows basic generalization ability. However, the accuracy of these reconstructions is somewhat lacking. We guess one of the primary factors is the inaccurate camera pose from the COLMAP. It is a promising direction to explore the joint optimization of the camera pose for the MVS in the future. Besides, the limited training data also hinders the performance of our method. Compared with Dust3r which is trained with a mixture of eight datasets, covering millions of image data, our method and other MVS methods are only trained on DTU or BlendedMVS. How to utilize these large datasets for training MVS is also one of our future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DmrlDLNWeU", "reviewer": "Reviewer_fCJ7", "summary": "- The author proposes a depth-range-free multi-view stereo framework that simultaneously takes into account all the source images.\n- The author has specially designed a 3D pose embedding to better encode specific geometric information.\n\n- The Multi-View Stereo method proposed in the paper achieves more robust depth estimation by cascading Disparity Feature Encoding, Multi-view Disparity Attention, and GRU Updating, and has demonstrated good performance across multiple datasets.", "questions": "- Are the training results of the epipolar disparity flows truly meaningful? Are there any visual results to demonstrate this?\n\n- How is $D_0$ constructed? During construction, were all source images initialized with the reference image? Was $D_0$ calculated as an average of initial values obtained from each source view? The explanation could be clearer.\n\n- How is the depth updated through the GRU? The section 3.5 is not clearly explained.\n\n- The ablation study is not sufficient:\n  - What are the different initializations of depth, such as random or all zeros?\n  - How is the convergence?\n\n- In Figure 2, what is the relationship between the $e_i$ output from the GRU and the $e_i$ after fusion? How is $D_i$ obtained from $e_i$, and how is $U_i$ combined?\n\n- As a method that also uses GRU for updates, there is a lack of reference to and discussion of \"Multiview Stereo with Cascaded Epipolar RAFT.\"", "limitations": "The author has already addressed the limitations of the method.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The depth-range has always been an issue affecting the robustness of Multi-View Stereo algorithms. Although previous methods have attempted to address it, they are still imperfect. The method proposed by the author exhibits technical novelty.\n\n- The iterative updating approach proposed by the author can simultaneously take into account the information of all source images.\n\n- The method presented in the article has shown good performance on the dataset's benchmark.", "weaknesses": "- The visual results of the method proposed in the article exhibit many floater artifacts.\n\n- The overall description of the network architecture needs further refinement. The writing need to be improved. The current introduction is not well-balanced in detail, and there are several typographical errors with symbols.\n\n- The use of symbols like $F^d_i$ is inconsistent in terms of superscripts and subscripts, making it difficult to understand and appearing unprofessional in typesetting. This is not the only symbol with usage issues; multiple symbols have similar problems. It is recommended to provide a comprehensive explanation of the symbols in the supplementary material and to revise the symbols used in the figures and text.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> The visual results of the method proposed in the article exhibit many floater artifacts.\n\nThanks for your reminder. Artifacts are generated during the depth fusion step. For point cloud fusion, we directly sampled the pcd method from DispMVS. This method selects multiple relevant depth views for each image to perform back-projection. After threshold filtering, the back-projected depth is weighted and combined with the current depth, which can lead to the generation of artifacts.\n\n> The overall description of the network architecture needs further refinement. The writing need to be improved. The current introduction is not well-balanced in detail, and there are several typographical errors with symbols.\n\nWe will thoroughly review the entire manuscript to complete the details of the network design and correct any inconsistencies between the formulas and symbols in the figures.\n\n> The use of symbols like $F_i^d$ is inconsistent in terms of superscripts and subscripts. It is recommended to provide a comprehensive explanation of the symbols in the supplementary material and to revise the symbols used in the figures and text.\n\nThanks for your  reminder. I will standardize the notation for epipolar disparity features as $Fd_i$. I will carefully align the symbols in the formulas and images throughout the entire manuscript.\n\n> Are the training results of the epipolar disparity flows truly meaningful? Are there any visual results to demonstrate this?\n\nTo consider the information interaction of multiple source images during 2D sampling, it is necessary to train the flow. As shown in Fig. 1, after the MDA and GRU modules, the flow obtained from different source images is consistent in terms of details. By incorporating geometric information, although the flow magnitudes on different source images vary, the representation of edges and other details is unified.\n\n> How is $D_0$ constructed? During construction, were all source images initialized with the reference image?\n\nOn L.151, we describe that $D_0$ is obtained by selecting the midpoint of the sampling range. On L.153, we first initialize $D_0$ for all source images and then take the average as the initial depth obtained from each source view.\nThank you for your suggestion, we will make this description clearer.\n\n> How is the depth updated through the GRU?\n\nWe follow the structure of DispMVS, iteratively updating the epipolar flow for each source image. We obtain the depth from the disparity flow and utilize a weighted sum for the depth in a multi-view situation. After fusion, the depth is converted back to disparity flow to perform the next iteration.\n\n> The use of symbols like $F_i^d$ is inconsistent in terms of superscripts and subscripts, symbols' problems.\n\nThanks for your  reminder. We will standardize the notation for cost volume as $V_i(p0)$, epipolar disparity features as $Fd_i$, and disparity hidden State as $Hd_i$ to eliminate the gap between the formulas and the figures.\nWe will recheck the entire document for correct notation usage, including matrix and vector parts, and align the symbols in the formulas and images.\n\n> What are the different initializations of depth, such as random or all zeros? How is the convergence?\n\nWe add the comparison with different initializations of depth. However, due to the unknown depth range, it is not feasible to design a random sampling range for 3D sampling, which is impractical. Additionally, when the initial depth is set to 0, significant noise can occur during feature warping. Therefore, we designed three sets of ablation experiments: randomly initializing within the epipolar search range, and initializing fixed at the left endpoint of the epipolar line.\n\n|                    |   Acc.(mm)↓   |   Comp.(mm)↓   |  Overall(mm)↓  |\n| :----------------- | :-------------: | :-------------: | :-------------: |\n| The left endpoint  |      0.389      |      4.373      |      2.381      |\n| randomly 2D sample |      2.597      |      20211      |      2.404      |\n| Ours               | **0.338** | **0.331** | **0.335** |\n\nFor each case, as the GRU iteratively updates, the error gradually decreases, and the depth gradually converges. Fig. 4. shows the convergence behavior when the initial point is set to the left endpoint.\n\n> In Figure 2, what is the relationship between the $e_i$ output from the GRU and the $e_i$ after fusion? How is $D_i$ obtained from $e_i$, and how is $U_i$ combined?\n\nIn each iteration, the GRU update operator output the current epipolar flow $e_i$ and weight $U_i$. We obtain the depth from the disparity flow by Eq.3 and utilize a weighted sum for the depth in a multi-view situation. After fusion, the depth is converted back to disparity flow $e_i$ to perform the next iteration.\n\n> As a method that also uses GRU for updates, there is a lack of reference to and discussion of \"Multiview Stereo with Cascaded Epipolar RAFT.\n\nPlease read the detailed explanation in the Author Rebuttal section for this part of the response.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HdIoLZgwaI", "reviewer": "Reviewer_Q3xc", "summary": "This paper proposes a depth-range-free Multi-View Stereo (MVS) method, which iteratively updates the depth using a GRU-based approach. To eliminate the dependency on depth priors, the paper improves the depth initialization method of DispMVS. To fully utilize multi-frame information, the paper encodes the observation information between multiple source frames and the reference frame into features and proposes a Multi-view Disparity Attention module for multi-frame information fusion. To enhance the preservation and utilization of geometric information, the paper introduces 3D pose embedding, uncertainty estimation, and disparity hidden states. The paper has been tested on the DTU and T&T datasets, and the experimental results show that the method exhibits robust depth estimation results in the absence of depth range priors.", "questions": "-The number of Fd_i encoded by the authors is related to the number of source frames. From Fig. 3, it seems that during the GRU update process, they are directly concatenated, which will cause the channel number to be related to the number of source frames. So, if the model uses 5 frames in training and then the number of frames cannot be changed during testing, is that correct?\n\n-From the numerical results and method description, the paper has completely abandoned the use of depth priors now. But if the depth prior is known and the prior is more compact than the depth range calculated in Sec 3.2, can using the depth prior improve the performance of this method?\n\n-As a GRU-type method, how does the accuracy of the estimated results change with the number of iterations?", "limitations": "-Limited performance: From the experimental results, the method in this paper is not outstanding on the benchmark, and the gap with the state-of-the-art methods is quite obvious, even only compared with GRU-based methods (such as CER-MVS).\n\n-Limited novelty: The method in this paper is more like a combination of existing modules.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "-The motivation for the model design is clear, and the experimental results to some extent reflect the effectiveness of the model.\n\n-The experimental results demonstrate that the method can achieve good predictive results without relying on depth priors.", "weaknesses": "-Poor writing quality: The consistency of the symbols in the paper is poor, such as the left side of Eq. 5 should be V_i(p0); The Fd_i in Fig. 2, Eq. 8, and F^d_i in Sec 3.4, the H_i in Fig. 2 and  Hd_i in Eq. 8; the use of symbols is not standardized, such as the representation of matrices and vectors, the authors are advised to carefully check the entire text; In Eq. 8, there is an extra dot multiplication symbol before phi_4.\n\n-Missing method description: For the final GRU update part, the paper lacks a complete description. My current guess is that the authors use Fd_i and context features as input information to update the hidden state not mentioned in the paper, and decode the residual from this hidden state.\n\n-Incorrect key citation: There is an error in the citation of one of the main comparative methods, DispMVS[5], and the authors are advised to carefully check the reference list. \"Rethinking the multi-view stereo from the perspective of rendering-based augmentation.\" -> \"Rethinking Disparity: A Depth Range Free Multi-View Stereo Based on Disparity\"\n\n-The application of cross-attention is strange: The cross-attention in this paper is only calculated at the same pixel positions in different frames, and the paper emphasizes that there are few continuous smooth trajectories between different frames in MVS, making this design very illogical. The authors are expected to provide a detailed explanation.\n\n-Missing comparisons: CER-MVS [Ⅰ], as an early GRU-based MVS work, should be included in the comparison. Other works such as MVSFormer series[Ⅱ, Ⅲ], GeoMVSNet [Ⅳ] should not be ignored either.  \n\n-Incomplete ablation experiments: There is a lack of separate ablation experiments on uncertainty and disparity feature hidden states, and the existing ablation experiments couple the two together. There is a lack of ablation experiments on the MDA module.\n\n[Ⅰ] Ma, Zeyu, Zachary Teed, and Jia Deng. \"Multiview stereo with cascaded epipolar raft.\" ECCV 2022.\n\n[Ⅱ] Cao, Chenjie, Xinlin Ren, and Yanwei Fu. \"MVSFormer: Multi-view stereo by learning robust image features and temperature-based depth.\" arXiv preprint arXiv:2208.02541 (2022).\n\n[Ⅲ] Cao, Chenjie, Xinlin Ren, and Yanwei Fu. \"MVSFormer++: Revealing the Devil in Transformer's Details for Multi-View Stereo.\" ICLR 2024.\n\n[Ⅳ] Zhang, et al. \"Geomvsnet: Learning multi-view stereo with geometry perception.\" CVPR 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> the left side of Eq. 5 should be $V_i(p0)$; The $Fd_i$ in Fig. 2, Eq. 8, and $F^d_i$ in Sec 3.4, the $H_i$ in Fig. 2 and $Hd_i$ in Eq. 8.\n\nThanks for your  reminder. I will standardize the notation for cost volume as  $V_i(p0)$, epipolar disparity features as $Fd_i$, and disparity hidden State as $Hd_i$ to eliminate the gap between the formulas and the figures.\n\n> the use of symbols is not standardized, such as the representation of matrices and vectors.\n\nWe will recheck the entire document for correct notation usage, including matrix and vector parts, and make the necessary corrections.\n\n> In Eq. 8, there is an extra dot multiplication symbol before $\\varphi_4$.\n\nAdditionally, we will remove the redundant dot multiplication symbol before $\\varphi_4$ in Eq. 8.\n\n> For the final GRU update part, the paper lacks a complete description.\n\nWe follow the structure of DispMVS, iteratively updating the epipolar flow for each source image. In each iteration, the input to the update operator includes the hidden state, the disparity feature $Fd_i$ output from MDA module, the current epipolar flow, and the context feature of the reference image. The output of the update operator includes a new hidden state, an increment to the disparity flow, and the weight.\nWe obtain the depth from the disparity flow and utilize a weighted sum for the depth in a multi-view situation. After fusion, the depth is converted back to disparity flow to perform the next iteration.\n\n> There is an error in the citation of DispMVS[5]\n\nThank you very much for your reminder. We will correct the citation accordingly.\n\n> Cross-attention is only calculated at the same pixel positions in different frames, making this design very illogical.\n\nIt was misunderstood that the attention is not on the sample pixels' image features in different frames but on the disparity features encoded from the cost volume. This cost volume is obtained by back-projecting the sampled depths into the source images. Therefore, the features among multiple frames are associated through the sampled points' depths. Additionally, we enhance the implicit disparity relationships among multi-view frames by encoding pose embedding, which introduces multi-view relative pose information and geometric information between specific sampled points.\n\n> Missing comparisons: CER-MVS, MVSFormer series, GeoMVSNet. \\& Limited performance\n\nThank you for your suggestion. We include CER-MVS, GeoMVSNet, and MVSFormer++ in our comparative experiments. However, these methods all require the depth prior, whereas our algorithm is designed to operate in a depth-range-free setting.\nIn paper, we mainly focus on experiments related to Depth Range in Sec. 4.5. And our main comparison is with depth-range-free methods like DispMVS, which reduce dependence on depth priors through network design.\nFor methods that rely on depth range priors, whether based on GRU or Transformer, they may exhibit better performance with accurate depth priors. However, their performance significantly degrades when there is an error in the depth prior.\nDue to the length limitations, the detailed explanation and experiments for this part are placed in the Author Rebuttal.\n\n\n> There is a lack of separate ablation experiments on uncertainty and disparity feature hidden states.\n\nDue to the length limitations of the rebuttal, the detailed explanation and experiments for this part are placed in the Author Rebuttal.\n\n> During the GRU update process, the channel number to be related to the number of source frames, and the number of frames cannot be changed during testing?\n\nThis is not the case. In Fig. 3, \"concat\" refers to concatenating the disparity features output from self-attention and cross-attention. After the MDA module, similar to DispMVS, the disparity feature corresponding to each source image is fed to GRU updating module individually. However, since we have already performed extensive global information interaction, the epipolar flows obtained by multi-view situation will be interconnected.\n\n> If the depth prior is known and the prior is more compact than the depth range calculated in Sec 3.2, can improve the performance?\n\nDue to the length limitations of the rebuttal, the detailed explanation and experiments for this part are placed in the Author Rebuttal.\n\n> As a GRU-type method, how does the accuracy of the estimated results change with the number of iterations?\n\nAs shown in Fig. 3, the depth error decreases progressively with each iteration.\nThe vertical axis represents the depth error, and the horizontal axis represents the number of iterations. Iterations 0-7 correspond to the coarse stage, while iterations 8-9 correspond to the fine stage.\nFig. 3  shows the depth maps on DTU, in which we can see that the depth map recovers from coarse to fine.\n\n> Limited novelty: The method in this paper is more like a combination of existing modules.\n\nDriven by practical application needs, we creatively proposed applying transformer operations to the disparity features constructed after 2D sampling to remove the dependency on depth range. Additionally, we were the first to address the issue of depth mismatches among different source images during 2D sampling. By designing uncertainty and pose embedding, we endowed the features with geometric relationships, making multi-frame consistent estimation more efficient and accurate.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IYaVLZSYwR", "reviewer": "Reviewer_s1Ui", "summary": "This paper proposes a depth-range free MVS network that considers the information within the MVS framework. The initial depth is derived from the epipolar geometry of the reference image and the source images. In addition, the disparity features are enhanced by an uncertainty encoding module and a multi-view disparity attention module. The proposed method is assessed on several benchmarks compared to existing works. It shows promising results regarding the robustness to different depth ranges and the overall performance compared to RNN-based methods.", "questions": "1. In Figure 1, when the depth range is clipped within the actual depth range, e.g., the first column, how can you still get the depth for the foreground (425mm -- 552.5mm) and background (807.5mm -- 935mm) points? It seems that the proposed method yields the same results for all configurations as the four figures look the same.\n\n2. When sampling M points to construct the cost volume, what is the interval between the points and how it is decided? Is it adaptive during the iterations?", "limitations": "1. It is suggested that the authors refine the writing of the methodology section and make clarifications regarding W1 and Q2.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The evaluation of the proposed method shows better results than the existing RNN-based methods on several public benchmarks and it is more robust to different depth ranges.", "weaknesses": "1. The methodology lacks some clarity. For example, a) no explanation for $\\vec{f}$ in equations 3 and 4, b) no explanation for $0$ and $1$ in $p^{0}$ and $p^{1}$ for the search range, c) the dot before $\\varphi4$ in equation 8, d) how $H_{di}$ is initialized in equation 8, e) in equation 10, is $R^{0,i}$ the relative rotation between two images, or generated by the intersection angle $\\theta^{0,1}$, f) no explanation for $t_{c}$ and $t_{f}$ in equation 11. Readers in the same research field might get the actual meaning without indication, but the broader audiences need to be considered. \n\n2. There are problems with writing. For example, a) wrong reference for DispMVS, b) table 2 is indexed before tables 3 and 4 but it is mentioned after them in the main text, 3) missing reference of a section in the last paragraph in section 1.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> no explanation for $\\overrightarrow{f}$ in equations 3 and 4.\n\nSimilar to DispMVS, $\\overrightarrow{f}$ is a 2D flow vector along the epipolar line that provides flow in the x dimension $\\overrightarrow{f} _{xr\\to xs }(p_r)$ and y dimension $\\overrightarrow{f} _{yr\\to ys }(p_r)$.\n\n> no explanation for 0 and 1 in $p_0$ and $p_1$ for the search range.\n\nIn Sec 3.2 we define the search range as $[p_{d_0,d_i>0}^0, p_{d_0,d_i>0}^1 ]$, where 0 and 1 represent the left end and right end of the range.\n\n> the dot before $\\varphi_4$ in equation 8.\n\nWe will remove the redundant dot before $\\varphi_4$ in Eq. 8.\n\n> how $Hd_i$ is initialized in equation 8.\n\n$Hd_i$ is randomly initialized and learned via training.\n\n> in equation 10, is $R^{0,i}$ the relative rotation between two images, or generated by the intersection angle $\\theta ^{0,i}$.\n\nAs mentioned in paper, the calculation of relative pose distance is exactly the same as in [12], where $R^{0,i}$ is the relative rotation between two images.\n\n> no explanation for $t_c$ and $t_f$ in equation 11.\n\nSimilar to DispMVS, $t_c$ represents the iterations at the coarse stage, and $t_f$ represents the iterations at the fine stage.\n\n> wrong reference for DispMVS\n\nWe will correct the citation accordingly.\n\n> table 2 is indexed before tables 3 and 4 but it is mentioned after them in the main text.\n\nThis issue is due to the LaTeX formatting, and we will make the necessary modifications.\n\n> missing reference of a section in the last paragraph in section 1.\n\nThank you for your suggestion. We will make the necessary modifications.\n\n> In Figure 1, when the depth range is clipped within the actual depth range, how can you still get the depth for the foreground (425mm -- 552.5mm) and background (807.5mm -- 935mm) points?\n\nSince we have completely removed the depth range prior, the output results are not affected by any errors in the depth range.\nIn contrast, IterMVS samples based on the depth range, which prevents it from obtaining the correct depth for foreground (425mm - 552.5mm) and background (807.5mm - 935mm) points.\nDispMVS's initialization also rely on the depth range. During each iteration, it uses the depth range to apply a depth\nnormalization and filter out outliers for stability, so an underestimated depth range can significantly affect its performance.\n\n> What is the interval between the sampling points and how it is decided? Is it adaptive during the iterations?\n\nSimilar to DispMVS, we perform 2D sampling, where we sample\n$M$ points around current position $p_i$ along the epipolar line at each scale with a distance of one pixel. As noted on L.169, by constructing a 4-layer pyramid feature using average pooling, uniform pixel sampling at different levels allows for a larger receptive field. The sampling interval in 2D is fixed.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank all reviewers for valuable feedback and positive comments like "novel and nontrivial"(Reviewer RLMs), "satisfying experimental results"(Reviewer heJr), "exhibits technical novelty"(Reviewer fCJ7), "motivation for the model design is clear"(Reviewer Q3xc), "promising results ... to different depth ranges"(Reviewer s1Ui). We will correct all typos in the final version. We will release the code to facilitate more practical research on depth-range-free methods upon acceptance.

The following are some detailed answers.

>Add comparisons: CER-MVS, MVSFormer++, GeoMVSNet

We include CER-MVS, GeoMVSNet, and MVSFormer++ in our comparative experiments. However, these methods all require the depth range prior, whereas our algorithm is designed to operate in a depth-range-free setting.

In paper, we mainly focus on experiments related to Depth Range in Sec. 4.5. And our main comparison is with depth-range-free methods like DispMVS, which reduce dependence on depth priors through network design. For methods that rely on depth range priors, whether based on GRU or Transformer, they may exhibit better performance with accurate depth priors.
However, their performance significantly degrades when there is an error in the depth prior.

|   Depth Range   |   Method   |   Acc.(mm)↓   |   Comp.(mm)↓   |  Overall(mm)↓  |
| :-------------: | :---------: | :-------------: | :-------------: | :-------------: |
|  （425，935）  | MVSFORMER++[Ⅱ] | **0.309** | **0.252** | **0.281** |
|  （425，935）  |   GeoMVS[Ⅲ]   |      0.331      |      0.259      |      0.295      |
|  （425，935）  |    Ours    |      0.338      |      0.331      |      0.335      |
| （212.5，1870） | MVSFORMER++[Ⅱ] |      0.361      | **0.319** |      0.340      |
| （212.5，1870） |   GeoMVS[Ⅲ]   |      0.374      |      0.415      |      0.394      |
| （212.5，1870） |    Ours    | **0.338** |      0.331      | **0.335** |
| （141.6，2805） | MVSFORMER++[Ⅱ] |      0.739      |      0.820      |      0.780      |
| （141.6，2805） |   GeoMVS[Ⅲ]   |      0.602      |      1.663      |      1.133      |
| （141.6，2805） |    Ours    | **0.338** |      **0.331**      |      **0.335**      |

CER-MVS does not directly sample within the depth range but requires the use of depth ground truth (depth gt) to scale the entire scene to a range with a mean of 600. When we introduce a certain amount of noise to the depth gt, thereby altering the scale, the performance of CER-MVS declines sharply.

| Depth GT Noise | Method |   Acc.(mm)↓   |   Comp.(mm)↓   |  Overall(mm)↓  |
| :------------: | :-----: | :-------------: | :-------------: | :-------------: |
|       0       | CER-MVS[Ⅰ] | 0.359 | **0.305** | **0.332** |
|      20\%      | CER-MVS[Ⅰ] |      9.230          |       10.236         |       9.858         |
|      30 \%     | CER-MVS[Ⅰ] |      9.385      |     10.098     |      9.741      |
|                |  Ours  | **0.338** |      **0.331**      |      **0.335**      |

We also add CER-MVS, GeoMVSNet, and MVSFormer++ to depth range experiment in Appendix B.4. Experiments demonstrate that for methods dependent on depth range prior, even if a rough depth range can be obtained from COLMAP, their performance still degrades.


| Depth Range |   Method   | Acc.(mm)↓ | Comp.(mm)↓ | Overall(mm)↓ |
| :---------: | :---------: | :--------: | :---------: | :-----------: |
|     GT     | MVSFORMER++[Ⅱ] | **0.309** | **0.252** |  **0.281**  |
|     GT     |   GeoMVS[Ⅲ]   |   0.331   |    0.259    |     0.295     |
|     GT     |    Ours    |   0.338   |    0.331    |     0.335     |
|   COLMAP   | MVSFORMER++[Ⅱ] |   0.361   | **0.319** |     0.340     |
|   COLMAP   |   GeoMVS[Ⅲ]  |   0.374   |    0.415    |     0.394     |
|   COLMAP   |    Ours    | **0.338** |    0.331    |  **0.335**  |


> If the depth prior is known and the prior is more compact than the depth range calculated in Sec 3.2, can improve the performance?

To validate the effectiveness of the depth prior, we designed the experiment. The initial depth is reverted to random initialization based on the depth prior, similar to DispMVS. The results are shown as follows. It can be observed that adding depth prior can improve performance to some extent, but the difference compared to our depth-range-free method is not significant. This indicates that the current initial point selection strategy and the design of the Transformer enable the network to regress to the correct range, resulting in accurate depth estimation.

|                                               |   Acc.(mm)↓   |   Comp.(mm)↓   |  Overall(mm)↓  |
| :-------------------------------------------- | :-------------: | :-------------: | :-------------: |
| Random Depth Initialization among Depth Range |      **0.331**      |      0.335      |      **0.333**      |
| Ours                                          | 0.338 | **0.331** | 0.335 |




[Ⅰ] Ma, Zeyu, Zachary Teed, and Jia Deng. "Multiview stereo with cascaded epipolar raft." ECCV 2022.

[Ⅱ] Cao, Chenjie, Xinlin Ren, and Yanwei Fu. "MVSFormer++: Revealing the Devil in Transformer's Details for Multi-View Stereo." ICLR 2024.

[Ⅲ] Zhang, et al. "Geomvsnet: Learning multi-view stereo with geometry perception." CVPR 2023.

Meta-review: Depth-range agnostic multi-view stereo is an interesting recent research trend, and this paper tackles it by proposing a framework that exploits all source images simultaneously in contrast to competitors in this specific context. On the one hand, the reviewers acknowledged an innovative contribution and good performance, especially compared to depth-range agnostic methods. On the other hand, they highlighted significant issues, mainly regarding clarity, notation and experimental evaluation. The rebuttal/discussion allowed for the clarification of most of these concerns, and finally, all reviewers leaned toward acceptance. Nonetheless, the paper requires a significant revision to increase readability and clarity and integrate all the additional information/experiments that emerged during the discussion with the authors. Additionally, the authors of RAM-Depth provide pre-trained models on Blended (and TartanAir) at https://github.com/andreaconti/ramdepth. Thus, an exhaustive comparison (when training on Blended) with this direct competitor is feasible and required.
Decision: Accept (poster)