A Global Depth-Range-Free Multi-View Stereo Transformer Network with Pose Embedding

Authors: yitong  dong
yijin  li
zhaoyang  huang
weikang  bian
jingbo  liu
hujun  bao
zhaopeng  cui
hongsheng  li
guofeng  zhang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 18:11:26

Abstract: In this paper, we propose a novel multi-view stereo (MVS) framework that gets rid of the depth range prior. Unlike recent prior-free MVS methods that work in a pair-wise manner, our method simultaneously considers all the source images. Specifically, we introduce a Multi-view Disparity Attention (MDA) module to aggregate long-range context information within and across multi-view images.  Considering the asymmetry of the epipolar disparity flow, the key to our method lies in accurately modeling multi-view geometric constraints. We integrate pose embedding to encapsulate information such as multi-view camera poses, providing implicit geometric constraints for multi-view disparity feature fusion dominated by attention. Additionally, we construct corresponding hidden states for each source image due to significant differences in the observation quality of the same pixel in the reference frame across multiple source frames. We explicitly estimate the quality of the current pixel corresponding to sampled points on the epipolar line of the source image and dynamically update hidden states through the uncertainty estimation module. Extensive results on the DTU dataset and Tanks\&Temple benchmark demonstrate the effectiveness of our method.

Main paper: 
 introduction: 
 Multi-view stereo matching (MVS) is a crucial technique in 3D reconstruction, which aims to recover robust and reliable 3D representations from multiple RGB images [1; 2; 3] . Traditional methods [4; 5; 6] rely on hand-crafted similarity metrics and regularizations to compute dense correspondences between the input images. These methods are prone to degradation in challenging scenarios, such as varying illumination, textureless regions, and occlusion regions. Recently, learning-based methods [7; 8; 9; 10; 11] directly learn discriminative features from the input images through neural networks such as CNN and Transformers. By sampling some possible depth hypothesis within a given depth range, they warp the features from the source images to the reference view (i.e., the plane sweep algorithm [12] ) and compute the cost volume, which is then regularized also through the neural network to obtain the final depth maps. However, obtaining a suitable depth range is non-trivial when applied in real-world scenarios while these methods are generally sensitive to the depth range, which limits their application.To get rid of the dependence on depth range, some methods [13; 14; 15] transform the regression problem in the given depth space into a matching problem on the epipolar lines. Similar to optical flow [16] and feature matching [17; 18; 19; 20] , these methods also adopt a pair-wise manner. For example, DispMVS [14] computes the depth map of the source image multiple times through pairs that contain different source images and computes the final depth map by weight of sum. However, the pair-wise manner neglects the inter-image correspondence between the source images and could lead to sub-optimal solutions. Meanwhile, although DispMVS mitigates the influence of depth priors on constructing the 3D cost volume, its initialization based on depth range can still lead to significant performance degradation when the depth range error is too large, as shown in Fig. 1 .We argue that these methods need to consider all the source images at the same time. Our ideas are inspired by the recent methods [21; 22] of optical flow which concurrently estimate optical flows for multiple frames by sufficiently exploiting temporal cues. However, we find these frameworks cannot be trivially applied in the task of multi-view stereo. The reasons are twofold. First, a strong cue in the multi-frame optical flow estimation is that the flow originating from the same pixel belongs to a continuous trajectory in the temporal dimension. Additionally, the frames are sequentially aligned along this temporal dimension. Such inductive bias makes it easy to learn. But in the context of multi-view stereo, the source images may be captured in no particular order, lacking a similar constraint of continuity. Unlike optical flow, the input images in multi-view stereo are unordered. These distinctions pose a significant challenge when attempting to adapt the multi-frame optical flow framework for use in multi-view stereo. Second, the arbitrary positions and viewing angles of the source images, coupled with potentially large temporal gaps between captures, exacerbate issues such as varying illumination, significant viewport differences, and occlusions which call for new designs.Based on the above observations, in this paper, we propose a novel framework that gets rid of the depth range assumption. Unlike some recent methods [13; 14; 15] that work in a pair-wise manner, the proposed method estimates the depth maps of a reference image by simultaneously considering all the source images. To address the first issue, we design careful injection of geometric information into disparity features using 3D pose embedding, followed by multi-frame information interaction through an attention module. Subsequently, we encode multi-view relative pose information and geometric relationships between specific sampled points into 3D pose embedding, which is subsequently transferred to the Multi-view Disparity Attention (MDA) module. This method efficiently incorporates the relationship between depth and pixels within the network, facilitating improved information integration across multiple frames. Second, to mitigate the challenge of fluctuating image quality stemming from occlusion and other factors, we maintain and update the disparity hidden features to reflect the depth uncertainty of the current sampling point for each iteration. We design the disparity feature encoding module to learn disparity features along the epipolar lines of multi-view frames. This approach enables us to explicitly characterize occlusion scenarios for each pixel across diverse source images and dynamically adapt them during epipolar disparity flow updates. Consequently, the auxiliary information is furnished for subsequent information fusion within the module. Furthermore, we designed a novel initialization method to further eliminate the influence of the depth range compared to DispMVS [14] .In summary, our contributions can be highlighted as follows: (1) A multi-view disparity transformer network, which facilitates the fusion of information across multi-view frames, (2) A specially designed 3D pose embedding which is utilized to implicitly construct relationships of the epipolar disparity flow among multi-view frames, and (3) An uncertainty estimation module and dynamically updated hidden states representing the quality of source images during iterations. We evaluate our method against other MVS methods on the DTU dataset [23] and Tanks&Temple dataset [24] , and demonstrate its generalization in Fig. 1 . 
traditional mvs: 
 Multi-View Stereo has been developed for many years and has many downstream or related applications such as simultaneous localization and mapping (SLAM) [25] , visual localization [26] , 3D reconstruction [27; 28] , 3D generation [29] and scene understanding [30] . Traditional methods for Multi-View Stereo (MVS) can generally be categorized into three classes: volumetric, point cloud-based, and 2D depth map-based methods. Volumetric methods [31; 32] typically partition the 3D space into voxels and annotate them as either interior or exterior to the object surface. Point cloud-based methods [33; 34] directly optimize the point cloud coordinates of objects in 3D space. Depth map-based methods [2; 4; 35; 6] first estimate 2D depth corresponding to images and then fuse the 2D depths of the same scene to obtain a 3D point cloud. However, these traditional methods remain constrained by manually crafted image features and similarity matrices.CNN-based MVS methods generally leverage convolutional neural networks to construct and refine 3D cost volume [36; 37; 11; 38; 39; 40; 41; 42; 43; 44; 45; 46; 47] . For instance, [38] uses isotropic and anisotropic 3D convolution-based learning networks to estimate the depth map. [40] introduces a pixel-wise network to obtain visibility. [11] applies a multi-stage CNN framework to enable reconstruction. [44] and [45] build a kind of pyramid to realize 3D cost volume. Similarly, [46] proposes a sparse-to-dense CNN framework when constructing the 3D cost volume.RNN-based methods mainly exploit recurrent network structures [48; 49] to regularize 3D cost volume [50; 51; 52; 9; 13; 53; 54] . For example, [50] utilizes recurrent encoder-decoder structure and 2D CNN framework to solve large-scale MVS reconstruction. [51] introduces a scalable RNN-based MVS framework. IterMVS [13] uses a GRU-based estimator to encode the probability distribution of depth. Compared with 3D CNN, RNN highly reduces the memory requirement, which makes it more suitable for large-scale MVS reconstruction [55] .Transformer is popular in 3D vision tasks [56; 57; 58; 59] , and first introduced into the field of MVS reconstruction by [10] due to its ability to capture global context information. Transformer is incorporated into feature encoding [10; 60; 61] to capture features within and between input images. The succeeding work [62] implements a transformer to assign weights to different pixels in the aggregating process. [63] employs an Epipolar Transformer to perform non-local feature augmentation. However, these deep learning-based MVS methods commonly exhibit sensitivity to the depth range, thereby restricting their broad applicability.Scale-agnostic MVS methods infer the depth information from the movement along epipolar lines to reduce the heavy dependence of depth range priors. Several methods [14; 15] perform 2D sampling between two frames and iteratively update flows to find the matching points. Specifically, DispMVS [14] is randomly initialized within the depth range and performs depth fusion by utilizing a weighted sum. RAMDepth [15] selects a random source image in each iteration. However, both methods fail to fully exploit multi-frame constraints during the flow updates due to the mismatch of 3D information at sampling points. In this paper, we enhance the epipolar matching process by simultaneously considering multi-frame information. 
method: 
 Given a reference image I 0 ∈ R H×W ×3 and multi-view source images {I i } N -1 i=1 ∈ R H×W ×3 as input, the task of MVS is to calculate the depth map of the reference image. We treat MVS as a matching problem: for a pixel point p r in the reference image, we identify the corresponding point p s in the source image, then we can get depth by triangulation. Given the initial matching point p 0 s obtained by the initial depth, we adopt an iterative update strategy. Since the matching point lies on the epipolar line of the source image, the one-degree-of-freedom epipolar disparity flow is used to represent the network's iterative updates. The epipolar disparity flow e k s ∈ R H×W ×1 is 1-d flow along the epipolar line on the source image during each iteration:EQUATIONwhere → e dir is the normalized direction vector of the epipolar line, • is the dot product of vectors, and k is the iteration time. Different from previous methods [14 ; 13] , we fully eliminate the dependence on depth range during initialization and achieve synchronous updating of the epipolar disparity flow across multi-view images. This is done by our design of disparity information interaction.The overall pipeline of our method is illustrated in Fig. 2 . The proposed method starts from a feature extraction module to extract multi-scale features (Sec. 3.1). Then, we discuss how to initialize the depth map without depth range (Sec. 3.2) and perform feature encoding (Sec. 3.3). To facilitate information fusion across multi-view source images, we introduce the Multi-view Disparity Attention (MDA) module (Sec. 3.4), enhanced with Pose Embedding. Finally, the features enhanced by the MDA module are fed into a GRU module to update the epipolar disparity flow, as described (Sec. 3.5), which is then fused to generate the depth map.Following previous methods [10; 13; 60; 64] , we employ convolutional neural networks (CNNs) for image feature extraction. Moreover, we adopt a coarse-to-fine strategy to extract multi-scale image features. Specifically, we utilize two share-weighted feature extraction modules to extract image featuresF l 0 ∈ R H×W ×C and {F l i } N -1i=1 ∈ R H×W ×C and a context feature extraction module to extract context features.Differing from DispMVS [14] , we design a novel initialization method without depth range to further mitigate the influence of depth priors. Specifically, we select an initial position along the epipolar line and then convert it into the depth map. First, we derive the correspondence between depth and position along the epipolar line. Given a pixel p r of the reference image I 0 , the geometric constrain between it and the warped pixel p s of the source image I i can be written as:EQUATIONwhere d r denotes the depth in reference view, d s denotes the depth in source view. R and t denote the rotation and translation between the reference and the source view, K -1 0 and K -1 i denote the intrinsic matrices of the reference and the source view. Let T = (t x , t y , t z ) T , K -1 i p s = (p sx , p sy , p sz ) T and R • K -1 0 p r = (p rx , p ry , p rz ) T , we can associate d r and d s with pixel coordinates:EQUATIONEQUATIONwhere -→ f is a 2D flow vector along the epipolar line that provides flow in the x dimension -→ f xr→xs (p r ) and y dimension -→ f yr→ys (p r ). To obtain an appropriate initial position, we first determine the geometrically valid range along the epipolar line, which has not been considered in other works [53; 14] . If a point is observable in the current view, it must have physical significance, meaning it must lie in front of the camera. Therefore, we identify the search range along the epipolar line on the source image that satisfies the condition d r > 0, d s > 0. We obtain the initial position p 0 s by selecting the mid-point in search range along epipolar line.Due to occlusion, moving objects, blurring, or other factors violating the multi-view geometry assumptions, the quality of sampling points from different source images varies, which limits the network's performance in depth estimation. To address this issue, we extract uncertainty information from the sampling point feature and encode it with cost volume as epipolar disparity feature F d i . As shown in Fig. 2 , we design the disparity hidden state H d i to maintain the sampling information of the current source image and update it during iterations by incorporating new uncertainty information.Cost Volume Construction. For each source image, after determining the position p t s for the current iteration, we uniformly sample M points around p t s along the epipolar line at each scale with a distance of one pixel. By constructing a 4-layer pyramid feature using average pooling, uniform pixel sampling at different levels allows for a larger receptive field. The sampling interval in 2D is fixed. Given image features F l 0 and {F l i }N -1 i=1 , we obtain the features of M sampled points in the source image through interpolation and calculate the visual similarity. The cost volume V ∈ R H×W ×M is constructed by computing the dot product between pairs of image feature vectors:EQUATIONwhere R represents the set of sampling points uniformly sampled along the epipolar line in the source image, and M denotes the number of sample points.Disparity Feature Encoding with Uncertainty. When estimating the epipolar disparity flow from multi-view frames, it is essential to encode the differences between source images caused by variations in occlusion situations and image quality. Motivated by this, we conduct disparity hidden state H d i ∈ R H×W ×C to explicitly represent the situation of point p r relative to the source image. Motivated by this, we introduce a disparity hidden state H d i ∈ R H×W ×C to explicitly represent the condition of points relative to the multi-view source images. H d i is randomly initialized and consecutively updated throughout the iterative process. We introduce a variance-based uncertainty estimation module to encode the correlation features, which is formulated as follows:EQUATIONwhere V i denotes the cost volume of source image, V i denotes the average value of V i , and σ(•) is the sigmoid function. Then, the uncertainty U i , the disparity hidden state of the previous iteration, the correlation features and the epipolar disparity flows are fed into the convolutional layers to generate epipolar disparity feature F d i and update the disparity hidden state H d i .DispMVS estimates the epipolar disparity flow from each two-frame image pair {I 0 , I i } N -1 i=1 , which overlooks the abundant multi-view information. Inspired by VideoFlow [21] , we estimate the epipolar flow of multi-view images simultaneously. However, since multiple source images are not sequentially arranged and points uniformly 2D sampled across source images can not establish robust 3D spatial correspondences, directly learning the continuity between flows, as [21] , does not work.Therefore, unlike [13; 10] , etc., we design some special structures for information aggregation among multi-view images. Although the depths of sampled points along epipolar lines do not correspond, we observe that there is a regular pattern in the direction of depths along epipolar lines. As shown in Fig. 3 , We design Multi-view Disparity Attention to learn the global information and utilize pose embedding to implicitly model the correspondence between pixel coordinates and depth on multiple source images, enabling the network to learn the direction and scale relationship of corresponding flows across different source images.To effectively capture extensive global information across epipolar disparity features from different views, we leverage the Multi-view Disparity Attention (MDA) module to further enhance the disparity features. We utilize an attention module to globally interact with disparity features of multi-view source images, thereby achieving multi-view feature fusion.d i } N -1i=1 , we first use self-attention to achieve intra-image information interaction. We concatenate epipolar disparity features F d i ∈ R H×W ×C and set H × W the as sequence length L, generating F d ∈ R (N -1)×(H×W )×C .Then we use cross-attention to achieve inter-frame information interaction and learn the relations among multi-view. We concatenate epipolar disparity features F d i and set the number of source images N -1 the as sequence length L, generating F d ∈ R (H×W )×(N -1)×C .To reduce computation cost, for the self-attention we use a linear transformer to compute attention, which replaces the original kernel function with:EQUATIONwhere Φ(•) = relu(•) + 1 and relu(•) represents the activation function of exponential linear units.Pose Embedding. Due to the depths of sampling points varying for different source images, we utilize pose embedding to construct implicit disparity relationships among multi-view frames. To effectively convey useful information to the attention module, we categorize the features of pose embedding into two types: multi-view relative pose information and geometric information between specific sampled points. Fig. 3 illustrates the variables used to construct the pose embedding.On one hand, the multi-view relative pose information between cameras contains crucial information about disparity features. By explicitly injecting relative poses into the attention module, the network can learn image-level geometric constraints. We represent the angle θ 0,i between rays as embedding.Inspired by [65] , we encode the rotation matrix and translation matrix between the reference and the source view into the relative pose distance P 0,i :EQUATIONOn the other hand, we encode the geometric information between specific sampled points. Due to our incorporation of pixel-level attention in addition to inter-frame attention, it is necessary to encode not only image-level camera poses but also the pixel-level information corresponding to sampled points.It is important to note that for each pixel in the reference image and its corresponding sampled point in the source image, we can obtain the corresponding 3D point coordinates P through triangulation based on stereo geometry. Accordingly, we encode the 2D coordinates p s of the source image, the depth d r from the perspective of the reference image, and the depth d s from the perspective of the source image, thereby transforming the 3D information into corresponding relationships on the 2D plane. Moreover, we encode the normalized direction r 0 , r i to the 3D location of a point.In the GRU updating process, we iteratively update the epipolar disparity flow e k+1 s = e k s + ∆e s obtained from the MDA module for each source image. In each iteration, the input to the update operator includes 1) the hidden state; 2) the disparity feature output from the MDA module; 3) the current epipolar flow; and 4) the context feature of the reference image. The output of the update operator includes 1) a new hidden state; 2) an increment to the disparity flow; and 3) the weight of disparity flow for multi-view images. We derive the depth from the disparity flow and employ a weighted sum to integrate the depth across multi-view source images. After fusion, the depth is converted back to disparity flow to perform the next iteration.Similar to [14] , we output depth after each iteration and construct the loss function accordingly. We construct the depth L1 loss. The loss function is represented in Eq. 9:EQUATIONwhere t c , t f are iterations at the coarse and fine stage, γ is a hyper-parameter which is set to 0.9. 
experiments: 
 In this section, we first introduce the datasets (Sec. 4.1), followed by the implementation details of the experiment (Sec. 4.2). Subsequently, we delineate the experimental performance (Sec. 4.3) and conduct ablation experiments to validate the efficacy of each proposed module (Sec. 4.4).DTU dataset [23] is an indoor multi-view stereo dataset captured in well-controlled laboratory conditions, which contains 128 different scenes with 49 views under 7 different lighting conditions. Following MVSNet [8] , we partitioned the DTU dataset into 79 training sets, 18 validation sets, and 22 evaluation sets. BlendedMVS dataset [66] is a large-scale outdoor multi-view stereo dataset that contains a diverse array of objects and scenes, with 106 training scenes and 7 validation scenes. Tanks and Temples [24] is a public multi-view stereo benchmark captured under outdoor real-world conditions. It contains an intermediate subset of 8 scenes and an advanced subset of 6 scenes.Implemented by PyTorch [67] , two models are trained on the DTU dataset and large-scale Blended-MVS dataset, respectively. On the DTU dataset, we set the image resolution as 512 × 640 and the number of input images as 5 for the training phase. On the BlendedMVS dataset, we set the image resolution as 768 × 576 and the number of input images as 5 for the training phase. For all models, we use the AdamW optimizer with an initial learning rate of 0.0002 that halves every four epochs for 16 epochs. The training procedure is finished on two A100 with t c = 8, t f = 2. For depth filtering and fusion, we process 2D depth maps to generate point clouds and compare them with ground truth.In this section, we compare our method with other state-of-the-art methods and scale-agnostic methods. Existing methods are categorized into traditional methods [2; 35], 3D cost-volume Evaluation on DTU. We evaluate the proposed method on the evaluation set of DTU dataset. We set the image resolution as 864 × 1152 and the number of input images as 5. As shown in Table 1 , our method has the best overall performance among depth-range-free methods. CER-MVS [70] and MVSFormer++ [61] demonstrate superior performance; however, they are heavily dependent on the accuracy of the depth range. Our approach outperforms when compared with depth range-free methods like DispMVS [14] and RAMDepth [15] , which demonstrates the effectiveness of our method in exploiting correlations among multi-view frames.Evaluation on Tanks and Temples. Since the Tanks and Temples dataset does not provide training samples, we use a model pre-trained on the BlendedMVS dataset for testing. We set the image resolution as 864 × 1152 and the number of input images as 7 for the evaluation phase. Table 2 presents the comparison between our method and other state-of-the-art methods. Our method achieves the best performance among scale-agnostic methods [14] . Since RAMDepth [15] has not provided results on the Tanks and Temples dataset and source code, we are unable to make a comparison. Although our method exhibits a certain gap when compared to state-of-the-art methods [70; 61] based on precise depth priors, it demonstrates superior robustness across a broader depth range.We visualize point clouds generated on DTU and Tanks and Temples dataset in Fig. 4 , which demonstrates that our method is capable of constructing a comprehensive and precise point cloud.In this subsection, we conduct ablation studies of our model trained on DTU [23] datasets to discuss the effectiveness of core parts of our method. The implemented baseline is basically based on DispMVS [14] . All the experiments are performed with the same hyperparameters. Pose Embedding. We conducted ablation experiments to validate the effectiveness of the pose embedding. Specifically, within the multi-view attention module, we remove 3D pose embedding and retain only the original 2D position encoding of attention. As shown in Table 3 , after applying pose embedding, the overall performance improves by 9.46%. The result indicates that the current task heavily relies on the relative pose and geometric information contained in the pose embedding. Without incorporating geometric constraints across multi-view source images, typically achieved through pose embedding, the performance of Transformer in this task may degrade significantly.Disparity Feature Encoding coupling with Uncertainty. Following [14] , we further attempt to remove uncertainty estimation and disparity hidden state and directly perform feature encoding on disparity flow and cost volume. As shown in Table 3 , with the disparity feature encoding coupling with uncertainty, the overall performance improves by 9.95%. The result validates the effectiveness of the module, demonstrating that explicitly estimating the quality of sampled points on the epipolar line of source images and updating the disparity hidden state in the network is effective. Additionally, we designed two separate ablation experiments, removing the uncertainty and disparity feature hidden states, to further evaluate the impact of these two modules on the network. The uncertainty and disparity feature hidden states improved the overall performance by 6.16% and 7.46%, respectively. Compared to the performance without disparity feature encoding coupling with uncertainty, this demonstrates the effectiveness of the uncertainty and disparity feature hidden state updating modules.In this section, we compare the generalization of different networks to depth range. We don't compare with RAMDepth [15] in the ablation studies due to the lack of its source code. We select several state-of-the-art methods (GeoMVS [69] , MVSFormer++ [61] ), RNN-based methods (IterMVS [13] , CER-MVS [70] ) and depth-range-free method (DispMVS [14] ) to conduct experiments to evaluate the generalization of depth range. Our main comparison is with depth-range-free methods [14] , which reduce dependence on depth priors through network design. All methods are trained on DTU dataset with the same depth range and subsequently inference under different depth ranges. For methods [13; 70; 69; 61] that rely on depth range prior for depth sampling, whether based on RNN or Transformer, they may exhibit better performance with accurate depth priors. However, as shown in Table 4 , there is a marked decline in performance for these methods with larger depth range.Although DispMVS [14] showed insensitivity to depth range, its performance still exhibited a certain degree of decline with larger depth ranges. In contrast, our method, which is independent of depth range, maintained consistent performance regardless of changes in depth range.It is crucial to emphasize that the depth range provided by the dataset is exceptionally accurate. For instance, the ground truth for the Tanks-and-Temples dataset is captured using an industrial laser scanner. However, in practical applications, while Structure-from-Motion (SfM) can derive depth ranges from sparse feature points, the resulting depth estimates are often prone to inaccuracies. These inaccuracies arise from the inherent sparsity of feature points, as well as challenges such as occlusion and suboptimal viewpoint selection. To verify the robustness of the MVS models in practical applications, we use the depth range obtained from COLMAP to replace the depth range ground truth (GT). As shown in Table 5 , there is a significant decline in performance for GeoMVS [69] , MVSFormer++ [61] , IterMVS [13] and CER-MVS [70] when we use the depth range obtained from COLMAP. DispMVS [14] also exhibits a certain degree of decline. In contrast, our method maintained consistent performance. This result further demonstrates the necessity of eliminating the depth range. 
conclusion: 
 We propose a prior-free multi-view stereo framework that simultaneously considers all the source images. To fully fuse the information from disordered and arbitrarily posed source images, we propose a 3D-pose-embedding-aided and uncertainty-driven transformer-based network. Extensive experiments show that our methods achieve state-of-the-art performances among the prior-free methods and exhibit greater robustness to the depth range prior. Limitations: The proposed method cannot run in real-time (i.e., 30 FPS), which could limit its application in mobile devices or other time-sensitive scenarios. Besides, our method shows a performance gap compared to SOTA costvolume-based methods on the mainstream benchmark, despite these methods relying on highly precise depth range priors. In the future work, we hope to close the gap.Answer Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] .Justification: The paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: Yes, the creators or original owners of the assets (e.g., code, data, models) used in the paper are properly credited. The citations for these assets can be found in Sec. 3 and Sec. 4.1.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. 
 
 Appendix: Justification: There is no societal impact of the work performed.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 


Decision: Depth-range agnostic multi-view stereo is an interesting recent research trend, and this paper tackles it by proposing a framework that exploits all source images simultaneously in contrast to competitors in this specific context. On the one hand, the reviewers acknowledged an innovative contribution and good performance, especially compared to depth-range agnostic methods. On the other hand, they highlighted significant issues, mainly regarding clarity, notation and experimental evaluation. The rebuttal/discussion allowed for the clarification of most of these concerns, and finally, all reviewers leaned toward acceptance. Nonetheless, the paper requires a significant revision to increase readability and clarity and integrate all the additional information/experiments that emerged during the discussion with the authors. Additionally, the authors of RAM-Depth provide pre-trained models on Blended (and TartanAir) at https://github.com/andreaconti/ramdepth. Thus, an exhaustive comparison (when training on Blended) with this direct competitor is feasible and required.

Reviews: review: paper_summary: paper_summary: The paper describes an MVS approach that does not depend on given depth ranges, which MVS algorithms typically require when building 3D cost volumes. The solution, previously proposed in DispMVS, is to perform searching and iterative updates in disparity space. Compared to DispMVS, the authors propose several novel designs to further eliminate the dependency on depth range during initialization, as well as to allow synchronous updates from all source views.
strength_weakness: strength_weakness: **Clarity**
The paper has some rather noteworthy issues regarding writing. The method section needs to have the right level of detail. The approach is built on top of DispMVS and shares many concepts and details, and the authors' choices to keep and omit details seem somewhat arbitrary, resulting in a non-self-contained method section. For example, there are also some concepts used without introduction, e.g. "sampling range" on L.168, "t_c, t_f" in Eq.11. The GRU updating process isn't defined. The relationship between epipolar disparity flow and depth is a critical concept that the paper does not explain. 

The resulting poor readability is potentially a significant issue, but unfortunately, it is hard to address in a rebuttal. I'd like to see how other reviewers weigh its impact. 

**Performance**
While beating other depth-range-free solutions, including DispMVS, on the two benchmarks, results still need to catch up on cost-volume approaches like GBi-Net, often by a large margin. It'll be quite valuable to understand what causes such discrepancy. Is it due to the limited receptive field, search gratuity, or else? Can improving on any of these help close the gap? 

**Generalization**
There is no discussion of generalization capability in the paper. Conceptually, it seems a depth-range-free approach would be more generalizable due to being invariant to object/scene scales. The ability to train a single model that operates in various use cases (indoor, outdoor, etc.) is arguably the most desirable feature of such approaches.
questions: questions: I look forward to answers to the questions above regarding clarity, generalization and performance.
limitations: limitations: The authors recognize speed as a limitation. The performance gap behind cost-volume counterparts is also worth mentioning.
paper_summary: paper_summary: The paper proposes a depth-rage free method for MVS. It leverages the transformers for designing a global-aware model, with using pose positional embedding to guide the model and also predict the uncertainty at the same time. The methods demonstrates good performance on diverse datasets and is also robust to different depth range.
strength_weakness: strength_weakness: - Related work. Using transformers for building global-aware model is very common for 3D reconstruction now [1,2,3]. I would be good if the authors could discuss these related works.

[1] Wang, Peng et al. “PF-LRM: Pose-Free Large Reconstruction Model for Joint Pose and Shape Prediction.” ICLR 2024. [2] Jiang, Hanwen et al. “LEAP: Liberate Sparse-view 3D Modeling from Camera Poses.” ICLR 2024. [3] Zhang, Kai et al. “GS-LRM: Large Reconstruction Model for 3D Gaussian Splatting.” ArXiv 2024.

- Dense view inputs. The experiments are performed in the sparse-view setting where the computation of transformers is not a big problem. Is the method also efficient in the dense view setting? If not, I hope the authors could have more discussions in the paper.

- I would say that the gains in the ablation experiments are not that significant.
questions: questions: Please see the weakness section.
limitations: limitations: The scale of the experiment is not big enough. It would be great if the authors could perform larger-scale experiments, ideally Dust3r-level scales, to understand the scaling capability of the proposed method. I can understand it is not easy to acquire more resources for training, but it would be great if the authors could have more discussions on this.
paper_summary: paper_summary: - The author proposes a depth-range-free multi-view stereo framework that simultaneously takes into account all the source images.
- The author has specially designed a 3D pose embedding to better encode specific geometric information.

- The Multi-View Stereo method proposed in the paper achieves more robust depth estimation by cascading Disparity Feature Encoding, Multi-view Disparity Attention, and GRU Updating, and has demonstrated good performance across multiple datasets.
strength_weakness: strength_weakness: - The visual results of the method proposed in the article exhibit many floater artifacts.

- The overall description of the network architecture needs further refinement. The writing need to be improved. The current introduction is not well-balanced in detail, and there are several typographical errors with symbols.

- The use of symbols like $F^d_i$ is inconsistent in terms of superscripts and subscripts, making it difficult to understand and appearing unprofessional in typesetting. This is not the only symbol with usage issues; multiple symbols have similar problems. It is recommended to provide a comprehensive explanation of the symbols in the supplementary material and to revise the symbols used in the figures and text.
questions: questions: - Are the training results of the epipolar disparity flows truly meaningful? Are there any visual results to demonstrate this?

- How is $D_0$ constructed? During construction, were all source images initialized with the reference image? Was $D_0$ calculated as an average of initial values obtained from each source view? The explanation could be clearer.

- How is the depth updated through the GRU? The section 3.5 is not clearly explained.

- The ablation study is not sufficient:
  - What are the different initializations of depth, such as random or all zeros?
  - How is the convergence?

- In Figure 2, what is the relationship between the $e_i$ output from the GRU and the $e_i$ after fusion? How is $D_i$ obtained from $e_i$, and how is $U_i$ combined?

- As a method that also uses GRU for updates, there is a lack of reference to and discussion of "Multiview Stereo with Cascaded Epipolar RAFT."
limitations: limitations: The author has already addressed the limitations of the method.
paper_summary: paper_summary: This paper proposes a depth-range-free Multi-View Stereo (MVS) method, which iteratively updates the depth using a GRU-based approach. To eliminate the dependency on depth priors, the paper improves the depth initialization method of DispMVS. To fully utilize multi-frame information, the paper encodes the observation information between multiple source frames and the reference frame into features and proposes a Multi-view Disparity Attention module for multi-frame information fusion. To enhance the preservation and utilization of geometric information, the paper introduces 3D pose embedding, uncertainty estimation, and disparity hidden states. The paper has been tested on the DTU and T&T datasets, and the experimental results show that the method exhibits robust depth estimation results in the absence of depth range priors.
strength_weakness: strength_weakness: -Poor writing quality: The consistency of the symbols in the paper is poor, such as the left side of Eq. 5 should be V_i(p0); The Fd_i in Fig. 2, Eq. 8, and F^d_i in Sec 3.4, the H_i in Fig. 2 and  Hd_i in Eq. 8; the use of symbols is not standardized, such as the representation of matrices and vectors, the authors are advised to carefully check the entire text; In Eq. 8, there is an extra dot multiplication symbol before phi_4.

-Missing method description: For the final GRU update part, the paper lacks a complete description. My current guess is that the authors use Fd_i and context features as input information to update the hidden state not mentioned in the paper, and decode the residual from this hidden state.

-Incorrect key citation: There is an error in the citation of one of the main comparative methods, DispMVS[5], and the authors are advised to carefully check the reference list. "Rethinking the multi-view stereo from the perspective of rendering-based augmentation." -> "Rethinking Disparity: A Depth Range Free Multi-View Stereo Based on Disparity"

-The application of cross-attention is strange: The cross-attention in this paper is only calculated at the same pixel positions in different frames, and the paper emphasizes that there are few continuous smooth trajectories between different frames in MVS, making this design very illogical. The authors are expected to provide a detailed explanation.

-Missing comparisons: CER-MVS [Ⅰ], as an early GRU-based MVS work, should be included in the comparison. Other works such as MVSFormer series[Ⅱ, Ⅲ], GeoMVSNet [Ⅳ] should not be ignored either.  

-Incomplete ablation experiments: There is a lack of separate ablation experiments on uncertainty and disparity feature hidden states, and the existing ablation experiments couple the two together. There is a lack of ablation experiments on the MDA module.

[Ⅰ] Ma, Zeyu, Zachary Teed, and Jia Deng. "Multiview stereo with cascaded epipolar raft." ECCV 2022.

[Ⅱ] Cao, Chenjie, Xinlin Ren, and Yanwei Fu. "MVSFormer: Multi-view stereo by learning robust image features and temperature-based depth." arXiv preprint arXiv:2208.02541 (2022).

[Ⅲ] Cao, Chenjie, Xinlin Ren, and Yanwei Fu. "MVSFormer++: Revealing the Devil in Transformer's Details for Multi-View Stereo." ICLR 2024.

[Ⅳ] Zhang, et al. "Geomvsnet: Learning multi-view stereo with geometry perception." CVPR 2023.
questions: questions: -The number of Fd_i encoded by the authors is related to the number of source frames. From Fig. 3, it seems that during the GRU update process, they are directly concatenated, which will cause the channel number to be related to the number of source frames. So, if the model uses 5 frames in training and then the number of frames cannot be changed during testing, is that correct?

-From the numerical results and method description, the paper has completely abandoned the use of depth priors now. But if the depth prior is known and the prior is more compact than the depth range calculated in Sec 3.2, can using the depth prior improve the performance of this method?

-As a GRU-type method, how does the accuracy of the estimated results change with the number of iterations?
limitations: limitations: -Limited performance: From the experimental results, the method in this paper is not outstanding on the benchmark, and the gap with the state-of-the-art methods is quite obvious, even only compared with GRU-based methods (such as CER-MVS).

-Limited novelty: The method in this paper is more like a combination of existing modules.
paper_summary: paper_summary: This paper proposes a depth-range free MVS network that considers the information within the MVS framework. The initial depth is derived from the epipolar geometry of the reference image and the source images. In addition, the disparity features are enhanced by an uncertainty encoding module and a multi-view disparity attention module. The proposed method is assessed on several benchmarks compared to existing works. It shows promising results regarding the robustness to different depth ranges and the overall performance compared to RNN-based methods.
strength_weakness: strength_weakness: 1. The methodology lacks some clarity. For example, a) no explanation for $\vec{f}$ in equations 3 and 4, b) no explanation for $0$ and $1$ in $p^{0}$ and $p^{1}$ for the search range, c) the dot before $\varphi4$ in equation 8, d) how $H_{di}$ is initialized in equation 8, e) in equation 10, is $R^{0,i}$ the relative rotation between two images, or generated by the intersection angle $\theta^{0,1}$, f) no explanation for $t_{c}$ and $t_{f}$ in equation 11. Readers in the same research field might get the actual meaning without indication, but the broader audiences need to be considered. 

2. There are problems with writing. For example, a) wrong reference for DispMVS, b) table 2 is indexed before tables 3 and 4 but it is mentioned after them in the main text, 3) missing reference of a section in the last paragraph in section 1.
questions: questions: 1. In Figure 1, when the depth range is clipped within the actual depth range, e.g., the first column, how can you still get the depth for the foreground (425mm -- 552.5mm) and background (807.5mm -- 935mm) points? It seems that the proposed method yields the same results for all configurations as the four figures look the same.

2. When sampling M points to construct the cost volume, what is the interval between the points and how it is decided? Is it adaptive during the iterations?
limitations: limitations: 1. It is suggested that the authors refine the writing of the methodology section and make clarifications regarding W1 and Q2.

Comments: title: Response to Reviewer
Response to Reviewer
Response to Reviewer
Response to concerns
About the model performance
Comparison with CER-MVS
About the limited novelty
The claim and comparison with CER-MVS is uncrediable
Rebuttal
Separate ablation experiments
comment: Thanks for your positive feedback on our work. We promise to address the corresponding points in the final version.
Thanks very much for your recognition of our work.
Dear Reviewer:

We are pleased that our response addressed your questions. We notice that your rating is still borderline reject, and we sincerely want to know if there are any remaining concerns about this work.
Thank you for the response and clarification. I believe the current further response has addressed some of my concerns. I will give Borderline Accept. I believe that if the revised version of the paper addresses these points and incorporates updates, its overall quality would range between Borderline Accept and Weak Accept.
Thanks for the response, this address my quetions.
Thanks for your reply and positive feedback. In response to your concerns, we provide the following answers:

> I am unable to understand why there are many floater artifacts in the DTU data.

Thanks for your reminder. As shown in Fig. 2 of PDF in Author Rebuttal, we compare our method with MVSFORMER++(SOTA) and find that point clouds inevitably exhibit artifacts in current MVS methods including the state-of-the-art methods. Artifacts are generated during the depth fusion step. For point cloud fusion, the pcd method selects multiple relevant depth views for each image to perform back-projection. After threshold filtering, the back-projected depth is weighted and combined with the current depth, which can lead to the generation of floater artifacts. This is due to 2D depth errors and inconsistencies across multi-view frames. Adjusting the threshold can mitigate this issue but may affect the overall quality of the 3D point cloud.

> The use of symbols by the authors seems confusing.

Thanks for your suggestion, we will make the adjustments accordingly.

> more comprehensive discussion comparing their method with CER-MVS.

CER-MVS uses depth gt priors to calculate the scale of scene and the way varies across different datasets. CER-MVS primarily uses three datasets: BlendedMVS dataset, Tanks-and-Temples dataset, and DTU dataset. BlendedMVS dataloader (https://github.com/princeton-vl/CER-MVS/blob/main/datasets/blended.py) provides two scaling methods: the default method (self.scaling == "median") uses depth gt to scale the scene to a median of 600 mm on Line 72, while the alternative method scales the scene using the depth range prior provided by the dataset (labeled as 'depth range gt') to achieve a minimum of 400 mm on Line 75. Tanks-and-Temples dataloader (https://github.com/princeton-vl/CER-MVS/blob/main/datasets/tnt.py) uses depth range prior to scale the scene to achieve a minimum of 400 mm. In the code, these depth range gt priors are loaded and represented by the 'scale_info' variable on Line 74 and 75. For DTU dataloader (https://github.com/princeton-vl/CER-MVS/blob/main/datasets/dtu.py), the depth range gt is not directly loaded in dataloader because DTU dataset already has a depth median of 600 mm and a minimum depth of 400 mm. The scene scale meets the network's requirements. 

CER-MVS performs uniform sampling on inverse depth, fixing the depth sample range (https://github.com/princeton-vl/CER-MVS/blob/main/core/corr.py). In the code (https://github.com/princeton-vl/CER-MVS/blob/main/core/raft.py), the maximum disparity $d_{max}$ is set to 0.0025, and the disparity increments of stage1 and stage2 are set to $d_{max}/64$ and $d_{max}/320$ on Line 81. By scaling the dataset, CER-MVS can obtain more accurate depth increments and maintain updates within the inverse depth sampling interval.

To validate the robustness of CER-MVS to depth gt priors, we have designed experiments by altering the median or minimum depth values of the scene. Specifically, we have introduced noise perturbations to change the median of DTU dataset during rebuttal.

When there is no per-pixel depth gt, CER-MVS uses the depth range to scale the scene depth to a minimum value of 400 mm. It is worth noting that the depth range provided by the dataset is very accurate. For instance, the ground-truth data for Tanks-and-Temples dataset is captured using an industrial laser scanner. However, in practical applications, the depth range obtained through COLMAP are often inaccurate due to the sparsity of feature points and issues such as occlusion and suboptimal viewpoint selection. To verify the robustness of depth range priors, we use the depth range obtained from COLMAP to replace the depth range gt.  

| Depth Range | Method |   Acc.(mm)↓   |   Comp.(mm)↓   |  Overall(mm)↓  |
| :---------: | :-----: | :-------------: | :-------------: | :-------------: |
|     GT     | CER-MVS |      0.359      | **0.305** | **0.332** |
|     GT     |  Ours  | **0.338** |      0.331      |      0.335      |
|   COLMAP   | CER-MVS |      0.816      | **0.326** |      0.571      |
|   COLMAP   |  Ours  | **0.338** |      0.331      | **0.335** |

From the table, it can be seen that CER-MVS exhibites a certain degree of decline due to the noise in the depth range caused by COLMAP. In contrast, our method, which is independent of depth range, maintained consistent performance regardless of changes in depth range. This further demonstrates the necessity of eliminating the depth priors.
Thanks for your feedback. Regarding the performance comparison test of CER-MVS with respect to depth gt priors, we have included this in the Author Rebuttal. We hope this response can address your concerns.
We respectfully argue that there are some misunderstandings in your interpretation of the scaling operations in the code. CER-MVS uses depth gt priors to calculate the scale of scene and the way varies across different datasets. 

CER-MVS primarily uses three datasets: BlendedMVS dataset, Tanks-and-Temples dataset, and DTU dataset. BlendedMVS dataloader (https://github.com/princeton-vl/CER-MVS/blob/main/datasets/blended.py) provides two scaling methods: the default method (self.scaling == "median") uses depth gt to scale the scene to a median of 600 mm on Line 72, while the alternative method scales the scene using the depth range prior provided by the dataset (labeled as 'depth range gt') to achieve a minimum of 400 mm on Line 75. Tanks-and-Temples dataloader (https://github.com/princeton-vl/CER-MVS/blob/main/datasets/tnt.py) uses depth range gt prior to scale the scene to achieve a minimum of 400 mm. In the code, these depth range gt priors are loaded and represented by the 'scale_info' variable on Line 74 and 75. For DTU dataloader (https://github.com/princeton-vl/CER-MVS/blob/main/datasets/dtu.py), the depth range gt is not directly loaded in dataloader because DTU dataset already has a depth median of 600 mm and a minimum depth of 400 mm. The scene scale meets the network's requirements. 

CER-MVS performs uniform sampling on inverse depth, fixing the depth sample range (https://github.com/princeton-vl/CER-MVS/blob/main/core/corr.py). In the code (https://github.com/princeton-vl/CER-MVS/blob/main/core/raft.py), the maximum disparity $d_{max}$ is set to 0.0025, and the disparity increments of stage1 and stage2 are set to $d_{max}/64$ and $d_{max}/320$ on Line 81. By scaling the dataset, CER-MVS can obtain more accurate depth increments and maintain updates within the inverse depth sampling interval.

To validate the robustness of CER-MVS to depth gt priors, we have designed experiments by altering the median or minimum depth values of the scene. Specifically, we have introduced noise perturbations to change the median of DTU dataset during rebuttal.

When there is no per-pixel depth gt, CER-MVS uses the depth range to scale the scene depth to a minimum value of 400 mm. It is worth noting that the depth range provided by the dataset is very accurate. For instance, the ground-truth data for Tanks-and-Temples dataset is captured using an industrial laser scanner. However, in practical applications, the depth range obtained through COLMAP are often inaccurate due to the sparsity of feature points and issues such as occlusion and suboptimal viewpoint selection. To verify the robustness of depth range priors, we use the depth range obtained from COLMAP to replace the depth range gt.  

| Depth Range | Method |   Acc.(mm)↓   |   Comp.(mm)↓   |  Overall(mm)↓  |
| :---------: | :-----: | :-------------: | :-------------: | :-------------: |
|     GT     | CER-MVS |      0.359      | **0.305** | **0.332** |
|     GT     |  Ours  | **0.338** |      0.331      |      0.335      |
|   COLMAP   | CER-MVS |      0.816      | **0.326** |      0.571      |
|   COLMAP   |  Ours  | **0.338** |      0.331      | **0.335** |

From the table, it can be seen that CER-MVS exhibites a certain degree of decline due to the noise in the depth range caused by COLMAP. In contrast, our method, which is independent of depth range, maintained consistent performance regardless of changes in depth range. This further demonstrates the necessity of eliminating the depth priors.
I appreciate the authors' positive response and their clarification of my questions, as well as their commitment to revising symbols and expressions. However, I still have a few concerns.

Firstly, I am unable to understand why there are many floater artifacts in the DTU data. The authors' response was somewhat vague, although I believe these issues could be addressed by carefully tuning the depth fusion parameters.

Additionally, the use of symbols by the authors seems confusing. For example, using multiple letters to represent a single variable, such as $Fd_i$ instead of $F^d_i$, appears unconventional.

Moreover, I think the authors should include a more comprehensive discussion comparing their method with CER-MVS, especially regarding the use of depth range, as pointed out by Reviewer Q3xc, which currently seems inaccurate.

Overall, I am inclined to adjust the score to Borderline Accept.
I appreciate the detailed responses. The clarifications are useful, though it remains to be seen how well they will be integrated into a final draft. The comments regarding generalization and gaps in performance are reasonable; however, they did not directly address the concerns. Given the significant practical advantages afforded by a range-free model, I'm still keeping a borderline recommendation.
I appreciate for the authors' response. Now, I think I understand this work. It is heavily based on DispMVS, but there is a main improvement.

1. Replace the vanilla feature correlation (named SIMI) in DispMVS with Disparity Feature Encoding combining Multi-view Disparity Attention

Besides, there are several less important improvement

1. The variance-based uncertainty, which replaces the sum weight in original DispMVS and is also used in disparity feature encoding.
2. The initialization method for each flow map

It shows effectiveness of the proposed modules, although being with more compuration costs. 

I still have concerns about the model performance (especially when being compared with CERMVS, which is also a GRU based method), and therefore I decide to keep my rating
The authors claim that
>CER-MVS does not directly sample within the depth range but requires the use of depth ground truth (depth gt) to scale the entire scene to a range with a mean of 600. When we introduce a certain amount of noise to the depth gt, thereby altering the scale, the performance of CER-MVS declines sharply.

However, in CERMVS paper, it is claimed that

>To pair neighbor views with reference views, we use the same method as MVSNet. In BlendedMVS, **which is used for training only**, the scenes have large variations in the range of depth values, we scale each reference view, along with its neighbor views, so that its ground-truth depth has a median value **600 mm**. When we evaluate on Tanks-and-Temples, due to lack of ground-truth and noisy background, **we scale each reference view, along with its neighbor views**, so that its minimum depth of a set of reliable feature points (computed by COLMAP as in MVSNet) is 400 mm. To stitch the predicted depth maps from multiple reference views, we simply scale back each depth map to its original scale.

For evaluation and test, CERMVS do not rely on the ground truth. When I check the source code of CERMVS again, there is even no depth ground truth loaded in the test dataloader. Why dose the author claim CERMVS rely on depth gt during inference?
Dear Reviewers,

please provide feedback about the authors' rebuttal; the deadline is approaching.
 
Thnak you
Thanks for the responses and clarification.
We sincerely apologize for not noticing the failure in transferring the ablation experiments due to a system refresh.

> There is a lack of separate ablation experiments on uncertainty and disparity feature hidden states. 

We add the separate ablation experiments. Compared to the performance without the MDA module, this demonstrates the effectiveness of the uncertainty and disparity feature hidden state update modules.

|                                    |   Acc.(mm)↓   |   Comp.(mm)↓   |  Overall(mm)↓  |
| :--------------------------------- | :-------------: | :-------------: | :-------------: |
| No Disparity Feature Hidden States |      0.370      |      0.354      |      0.362      |
| No Uncertainty                     |      **0.336**      |      0.378      |      0.357      |
| Ours                               | 0.338 | **0.331** | **0.335** |