SM4Depth: Seamless Monocular Metric Depth Estimation across Multiple Cameras and Scenes by One Mode

Authors: yihao  liu
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-08 04:13:17

Abstract: In the last year, universal monocular metric depth estimation (universal MMDE) has gained considerable attention, serving as the foundation model for various multimedia tasks, such as video and image editing. Nonetheless, current approaches face challenges in maintaining consistent accuracy across diverse scenes without scene-specific parameters and pre-training, hindering the practicality of MMDE. Furthermore, these methods rely on extensive datasets comprising millions, if not tens of millions, of data for training, leading to significant time and hardware expenses. This paper presents SM4Depth, a model that seamlessly works for both indoor and outdoor scenes, without needing extensive training data and GPU clusters. Firstly, to obtain consistent depth across diverse scenes, we propose a novel metric scale modeling, i.e., variationbased unnormalized depth bins. It reduces the ambiguity of the conventional metric bins and enables better adaptation to large depth-gaps of scenes during training. Secondly, we propose a “divide and conquer" solution to reduce reliance on massive training data. Instead of estimating directly from the vast solution space, the metric bins are estimated from multiple solution sub-spaces to reduce complexity. Additionally, we introduce an uncut depth dataset, Campus Depth, to evaluate the depth accuracy and consistency across various indoor and outdoor scenes. Trained on a consumer-grade GPU using just 150K RGB-D pairs, SM4Depth achieves outstanding performance on the most never-before-seen datasets, especially maintaining consistent accuracy across indoors and outdoors. The code can be found in the supplementary material.

Main paper: 
 introduction: 
 Monocular depth estimation is a fundamental visual task with wideranging applications in the field of multimedia, such as video editing [7] image editing [17, 69, 71] , 3D generation/synthesis [45, 47, 67] , 3D reconstruction [55, 56] , and human pose estimation [36, 50] . In this community, early research focused on MMDE [44, 49, 51, 58, 60, 70, 74] , which were trained and tested only on specific datasets. However, they suffered from poor generalization when applied to unseen datasets, which limited their applications in the real world. To solve this issue, much research shifted their focus to monocular relative depth estimation (MRDE) [29, 38, 66] while disregarding the metric scale. Leveraging diverse and easily accessible relative depth data, these studies have achieved impressive performance, enabling their application in scale-free tasks, such as image editing [13, 34] and image stylization [22, 32] , but not scale-sensitive applications, e.g. virtual reality [1, 33, 75] , 3D reconstruction [52] , and even robot navigation [8, 31, [61] [62] [63] .Beyond these approaches, universal MMDE has recently gained prominence for its generalization capabilities, marked by Depth Anything [64] , ZoeDepth [5] , and Metric3D [65] . However, these methods still face challenges in the two aspects of MMDE:(1) Inconsistent accuracy across scenes: The real world varies widely in depth, ranging from [1𝑚, 2𝑚] (close-up scenes) to [0.5𝑚, 80𝑚] (street scenes), making models tend to focus on specific scenes and causing inconsistent accuracy across scenes.(2) Heavy reliance on data amount: The reliance on massive training data (e.g. 8M metric depth data for Metric3D) remains due to the high complexity of determining a unique metric scale from a vast solution space of the natural scene.Aiming to address these issues, we propose a Seamless Model for MMDE across Multiple cameras and scenes (SM 4 Depth for short). First, based on explicit modeling of metric scale, we propose novel variation-based unnormalized depth bins which adaptively activate some parts rather than all of the fixed-length bins vector to describe the metric scale. This reduces the bin ambiguity inherent in previous width-based bins, and promotes the learning of widely different depth ranges in multiple scenes. Regarding the second issue, we propose a domain-aware bin estimation mechanism based on the "divide and conquer" idea, which estimates metric bins from various solution sub-spaces, not the entire one, for reducing complexity. Divide: we divide the common depth range into several range domains (RDs) offline and generate independent metric bins for each RD online. Conquer: we predict the RD that the input image belongs to and weightedly fuse all bins into a single one. To verify the accuracy consistency across diverse scenes, we propose BUPT Depth, a seamless RGB-D dataset, that consists of various indoor and outdoor scenes. Owing to the design of SM 4 Depth, it performs superiously on the consistency of accuracy across diverse scenes, which can be seen in Fig. 1 . Notably, it also achieves comparative performance compared with the state-of-the-art zero-shot MMDE methods but uses far fewer training samples (only 150K RGB-D pairs) and an affordable GPU (only single RTX 3090).Our primary contributions are as follows:(1) SM  
related work: 
 Monocular metric depth estimation is a classic visual task, in which determining metric scales is a crucial point, and there are two paradigms. Mainstream MMDE methods [11, 16, 24, 27, 28, 57, 72, 76] directly model this task as a pixel-wise regression problem (predicting continuous depth values in the real metric space), where metric scales are implicitly encoded. In contrast, since [19] , several methods [3, 4, 46] have defined this task as a classification problem.Adabins [3] explicitly encoded the metric scale into image-level depth bins. We follow the latter paradigm as this paper focuses on recovering the metric scale. However, the same bin on two images with a large gap in depth range represents drastically different depths, causing misleading back-propagation during training. In this paper, we introduce variation-based bins to overcome this issue. Zero-shot generation has become a new trend in monocular depth estimation in recent years. Early works [29, 39, 40] mainly achieved this goal by training with more accessible relative depth data. Initially, Li et al. [29] developed an MRDE pipeline on large-scale relative depth data. Ranftl et al. [40] trained an MRDE model on five datasets and re-applied the training strategy to [39] . For high practicality, the universal MMDE was first proposed in [5] which combined relative depth and metric depth to achieve generalization. Yin et al. [65] trained the model on 8M metric depth data for generalization. Yang et al. [64] proposed DepthAnything trained on 1M depth data and over 60M unlabeled data. Piccinelli et al. [37] designed a camera-adaptive model, UniDepth, and trained it on 300M metric depth images. This reliance on numerous training data is due to the complexity of determining correct metric scales from diverse scenes. Our approach aims to reduce this reliance. 
problem analysis and countermeasures: 
 In this section, we delve deep into the two issues of MMDE at the scene and data levels, and provide specific solutions for each issue. Generally, real-world images exhibit vastly different depth ranges, e.g. [1𝑚, 2𝑚] for indoor close-up and [0.5𝑚, 80𝑚] for street scenes. Such a large gap causes the model to overly concentrate on specific scenes instead of all scenes, leading to inconsistent accuracy across different scenes. In this section, we solve this issue by novel variance-based depth bins that bridge the gap of metric scale representation across scenes. Before that, we briefly review the conventional depth bin [3] and outline its weakness. Reviewing width-based depth bin and its weakness. Given the input image 𝐼 ∈ R ℎ×𝑤×3 , Adabins [3] generates an 𝑁 -channel probability map 𝑃 ∈ R ℎ×𝑤×𝑁 and a vector 𝑐 ∈ R 𝑁 ×1 representing the centers of 𝑁 depth bins discreted from the depth interval, which are linearly combined to obtain a metric depth map 𝐷 ∈ R ℎ×𝑤 :EQUATIONwhere 𝐷 (𝑖) is the 𝑖 th pixel's predicted depth, and 𝑃 𝑛 (𝑖) denotes the probability for pixel 𝑖 that its depth is equal to the 𝑛 th bin center 𝑐 𝑛 . In Eq. ( 1), the bin center 𝑐 𝑛 is calculated by accumulating the width of each bin 𝑏 ∈ R 𝑁 ×1 :EQUATIONwhere 𝑏 𝑛 = (𝑏 ′ 𝑛 + 𝜖)/ 𝑁 𝑖=1 (𝑏 ′ 𝑖 + 𝜖) denotes the normalized width of the 𝑛 th depth bin, with 𝜖 = 10 -3 and 𝑏 ′ 𝑛 ∈ [0, +∞) being the unnormalized width predicted through a feedforward neural network (FFN) with the ReLU activation function. During training, the bi-directional Chamfer loss [18] is employed to enforce the small width 𝑏 ′ within the interesting depth interval in the ground truth depth map D:L 𝑏𝑖𝑛 (𝑐, D) = ∑︁ d∈D min 𝑐 𝑛 ∈𝑐 ||d -𝑐 𝑛 || 2 + ∑︁ 𝑐 𝑛 ∈𝑐 min d∈D ||d -𝑐 𝑛 || 2 (3)where d is the pixel's correct depth.In natural scenes, the depth range of images varies significantly, yet all images must represent metric scales using the 𝑁 bins. This leads to great variations in the metric depth represented by a single bin, a phenomenon we call "bin ambiguity". Taking Fig. 2 as an example, a 3𝑚 pixel would be classified in the 208 th bin for an image with a range of [0, 3.5𝑚] (Fig. 2(b) ), but in the 72 𝑡ℎ bin for an image with a range of [0, 30𝑚] (Fig. 2(a) ). Such an excessive gap would confuse the metric meaning of the probability map's channels and lead to the back-propagation of misleading signals during training. Depth variation-based bins for consistent accuracy. The intuitive idea is to use the front portion of bins for short-range images and the entire bins for long-range images. To achieve this, we propose the variation-based unnormalized depth bins. Unlike the conventional bin 𝑏 ′ 𝑛 , we use only an FFN without ReLU activation. In this way, the FFN outputs variations that allow negative values, denoted as b′ ∈ R 𝑁 ×1 . Then, the bin center 𝑐 in Eq. ( 2) is re-formulated to an unnormalized bin center ĉ, which is no longer limited by the depth range of specific datasets (e.g., [0𝑚, 10𝑚] for NYUDv2 and [0𝑚, 80𝑚] for KITTI):EQUATIONAnalysis: Since the depth variations b′ are allowed to be negative, the bin center can reach the maximum depth on some intermediate bin ĉn (n < 𝑁 ) in short-range images, not necessarily the last bin center ĉ𝑁 . Thus, all pixels can be fully expressed by the front bins { ĉ𝑛 |𝑛 ∈ [1, n]}, and do not have to involve the latter bins { ĉ𝑛 |𝑛 ∈ (n, 𝑁 ]}. As indicated by the red lines in Fig. 2 , for the close-up image, the bin center reaches the maximum depth at the 110 𝑡ℎ bin and then continues to decrease. While the later bins { ĉ𝑛 |𝑛 ∈ (110, 𝑁 ]} correspond to some probability channels close to zero {𝑃 𝑛 |𝑛 ∈ (110, 𝑁 ]}, which can be observed through the visualization of part channels in Fig. 2 (c ). This illustrates that these later bins are not actually used.Fig. 3 presents additional statistics information, i.e., the frequency of depth values occurring in each depth bin. For the width-based depth bin (𝑐, 𝑏, 𝑏 ′ ), depths below 4𝑚 occur most frequently across all bins. Conversely, variation-based depth bin ( ĉ, b′ ) exhibits larger depths in the latter bins. This means that the depth values represented by each bin center are pulled apart on the level of the entire dataset, suppressing the bin ambiguity.Reason behind the reliance. Practical applications differ from specific datasets in that images are taken from various camera angles and innumerable scenes. Due to the diverse nature of appearance, mapping from visual cues to a wide range of depth values becomes highly intricate and cannot be exhaustively presented. Consequently, determining metric depth bins entails exploring a vast solution space, which necessitates greater attention to reducing its complexity. However, previous works have overlooked this crucial aspect by directly making predictions (e.g., [5] solves for metric bins and [65] predicts metric depth) from the entire solution space, inevitably requiring massive training data. To address this issue, we first divide the whole solution space into several sub-spaces. Then, a "divide and conquer" method is proposed to generate metric bins in each sub-space and predict the best metric bins for the input.Stage 1: Online depth range domain generation. To divide the solution space into sub-spaces, the previous approaches group all images according to semantic categories [5, 30] . However, a large gap in depth range may exist within one scene. Differently, we group all training images according to the depth range that better constrains the perspective and scene from which the image is taken.According to [19] , the amount of information for depth estimation decreases as the depth value increases. Thus, we employ a space-increasing strategy to gain more image groups (named range domain, RD) when the depth value is smaller. Assuming that the depth range is [𝑍 min , 𝑍 max ] and there are 𝐾 RDs, the 𝑘 th RD can be formulated as:EQUATIONWe further visualize the RDs in the supplementary material here.Stage 2: Online domain-aware bin estimation design. We design a domain-aware bin estimation mechanism that generates metric bins for each RD and finds the best-matching metric bins, following the "divide and conquer" idea in two steps.The "Divide" step aims to discretize each depth interval 𝑅𝐷 𝑘 into 𝑁 bins. Specifically, given the deep feature of the input image, we leverage a transformer encoder to learn the relationship between the deep feature and 𝐾 preset learnable 1-D embeddings (called bin queries). The output embeddings of these queries are fed into an FFN The "Conquer" step aims to estimate the correct RD for the input image and determine the best-matching metric bins. Specifically, we preset an additional 1-D embedding (called domain query) alongside the bin queries. Its corresponding output is then fed into a classification head (CLS) to generate the probability that the input image belongs to each RD, denoted as{𝑦 𝑘 ∈ [0, 1]|𝑘 ∈ [1, 𝐾]}.Subsequently, considering the possibility of images being positioned near the decision boundary of RD classification, we do not select the top-scoring metric bin but instead combine all bin center vectors ĉ [𝑘 ] to a single one by using the RD probabilities {𝑦 𝑘 |𝑘 ∈ [1, 𝐾]} as weights:EQUATION)where c is the final bin center vector.4 ARCHITECTURE OF SM 4 DEPTH 4.1 Pipeline.Fig. 4 illustrates the structure of our network. Given an RGB image I ∈ R h×w×3 , we first pre-process I to obtain a new image 𝐼 ∈ R ℎ×𝑤×3 with a unified field of view (FOV) (see Sec.4.2). Then, we extract the deep feature from 𝐼 by an encoder. Next, a pyramid scene transformer (PST) [46] is positioned between the encoder and decoder. It consists of three parallel transformer encoders with inputs of different patch sizes, respectively. We employ the transformer encoder with the smallest patch size to process all queries. Based on the mechanism in Sec.3.2, we obtain the bin center vector c of image 𝐼 . Finally, we design a decoder with hierarchical scale constraints (HSC-Decoder) to anchor the metric scale in multiple resolutions and output the metric depth map 𝐷 (see Sec.4.3). 
fov alignment pre-processing.: 
 According to [65] , eliminating "metric ambiguity" is the key to achieving universal MMDE. Therefore, we pre-process all input images to align their field of view (FOV). Given an input image EQUATIONNext, we crop this region 𝐼 ′ from the original image I, and fill the pixels beyond I with 255. Finally, the region 𝐼 ′ is resized to the target resolution (ℎ, 𝑤) to generate a new image 𝐼 as input of the network. Note that this FOV alignment is similar to the CSTM_image [65] in result, but without maintaining a canonical camera space. Thus it is more straightforward.Our decoder draws inspiration from the refinement decoder structures [9, 25] , but the divergence lies in scale constraints on the metric depth at each stage. As shown in Fig. 4 , taking the PST's output, denoted as 𝐹 1 , as input, we employ the DPT's decoder [39] to gradually recover the resolution of features, denoted as 𝐹 𝑠 with a size ℎ 2 (6-𝑠 ) × 𝑤 2 (6-𝑠 ) , where 𝑠 ∈ {1, 2, 3, 4, 5} is the stage number. In the first stage, 𝐹 1 is compressed into 𝑁 -channel and then multiplied pixel-wisely with c by Eq. ( 1), generating a low-resolution depth map𝐷 ′ 1 ∈ R ℎ 32 × 𝑤 32 .In the following 𝑠 th stage, the depth map of the former stage 𝐷 ′ 𝑠 -1 is upsampled and fused with feature 𝐹 𝑠 by a residual convolution block [9] . Then we linearly combine the fused feature and the bin centers c to generate the depth map𝐷 ′ 𝑠 ∈ R ℎ 2 (6-𝑠 ) × 𝑤 2 (6-𝑠 ) .In this way, the depth map of the last stage𝐷 ′ 5 ∈ R ℎ 2 × 𝑤2 is obtained. Compared to the previous refinement decoder [9] , the HSC-Decoder incorporates the metric bins into each stage to progressively refine the depth range, thus performing better in recovering the depth range. The loss functions are further described in the supplementary material here. 
uncut rgbd dataset: bupt depth: 
 BUPT Depth (see in Fig. 5 ) is proposed to evaluate consistency in accuracy across indoor and outdoor scenes, including streets, canteen, classroom, and lounges, etc. This dataset shows a variety of lighting, scenes, and viewing angles, making depth estimation challenging. It consists of 14,932 continuous RGB-D frames captured in BUPT by ZED2. In addition to the outputs of ZED2, we provide the re-generated depth maps from CREStereo [26] and the sky segmentation from ViT-Adapter [10] . The color and depth streams are captured with intrinsics of 1091.517 and a baseline of 120.034mm. 
experiments 6.1 experimental setting: 
 Datasets: For training, we randomly sample RGB-Depth pairs from various datasets. Specifically, we sample 24K pairs from ScanNet [15] , 15K pairs from Hypersim [41] , 51K pairs from DIML [12] , 36K pairs from UASOL [2] , 14K pairs from ApolloScape [21] , and 11K pairs from CityScapes [14] . During training, NYUD [35] and KITTI [53] are used for validation. In addition, we apply the same preprocessing steps to the training data as [5, 40] , elaborated in the supplementary material here. During testing, we employ eight datasets not seen during training: SUN RGB-D [48] , iBims-1 [23] , ETH3D Indoor/Outdoor [43] , DIODE Indoor/Outdoor [54] , nuScenes-val [6] , and DDAD [20] . Note that we remove the test set of NYUD from SUN RGB-D for a fair comparison. Metrics: We employ four metrics [3] for evaluation: the accuracy under threshold (𝛿 𝑘 < 1.25 𝑘 , 𝑘 = 1, 2, 3), the absolute relative error (REL), and the root mean squared error (RMSE). In addition, we use the relative improvement across datasets (mRI 𝜂 ) and metrics (mRI 𝜃 ) in [5] . During the evaluation, the final output is obtained by averaging the predictions for an image and its mirror image. In addition, the final output is upsampled to match the original image size, and all metrics are computed within the same FOV. Implementation Details: SM 4 Depth employs the Swin Transformer Base as the backbone, and runs on a single NVIDIA RTX 3090 GPU. The network is trained by the Adam optimizer with parameters (𝛽 1 , 𝛽 2 ) = (0.9, 0.999). The training runs for 20 epochs with a batch size of 10. The initial learning rate is set to 2 × 10 -5 and gradually reduced to 2 × 10 -6 . Note that, an over-large fixed FOV would cause too large invalid area in the small FOV dataset, making the network underfitting. We empirically set the fixed FOV to(𝜔 ′ x , 𝜔 ′ y ) = (58 • , 45 •) and the fixed resolution to (𝑤, ℎ) = (564, 424).Table 1 presents a quantitative comparison between SM 4 Depth and state-of-the-art methods, categorized based on different ground truth. SM 4 Depth achieves superior performance in most metrics when utilizing ground truth from either ZED2 or CREStereo [26] . Notably, SM 4 Depth is trained on a smaller dataset of 0.015M pairs (only 0.02%∼5% of the data used by previous methods), with FLOPs and parameter count typically half that of prior methods.Fig. 6 provides a more intuitive comparison between SM 4 Depth and state-of-the-art methods from a time-series perspective, visualizing the RMSE of each frame using two line graphs. SM 4 or outdoor scenes on both two ground truth. In contrast, other methods fail to achieve high accuracy simultaneously indoors and outdoors. Specifically, ZoeD-NK, Metric3D, and UniDepth show fluctuating accuracy in outdoor scenes, with some even exceeding over 50 at times. And the accuracy of SM 4 Depth varies more smoothly. In the first two examples we presented in Fig. 6 , both SM 4 Depth and UniDepth obtain a more accurate metric scale than others, but SM 4 Depth provides sharper outputs, even enabling the wooden fence in frame 9957 to be discernible. In the indoor scene, SM 4 Depth produces clearer outlines than other methods.Comparison to the state of the art 6.3.1 Quantitative Result. We employ two classical MMDE methods, i.e., BTS [25] and AdaBins [3] , as well as two more advanced MMDE approaches, i.e., NeWCRFs [68] and MIM [73] , for comparison. Moreover, we also employ several universal MMDE methods, i.e., Metric3D [65] , ZoeDepth [5] , DepthAnything [64] , and UniDepth [37] , for comparison (N indicates NYUD fine-tuning and K for KITTI fine-tuning; they are also applicable to SM 4 Depth).In Table 2 , the upper part shows the zero-shot performance on four indoor datasets, and the lower part shows that on four outdoor datasets. SM 4 Depth not only achieves accuracy consistency across indoor and outdoor but also outperforms most MMDE methods on public datasets and competes with top-performing algorithms like UniDepth and DepthAnything. Compared to the earliest universal MMDE algorithm, ZoeDepth, SM 4 Depth demonstrates superior accuracy across all datasets, leading in both absolute metrics (𝛿 1 , RMSE) and relative metrics (REL). This indicates SM 4 Depth's ability to learn more accurate relative depth from metric depth data. Compared to Metric3D, SM 4 Depth performs better on most datasets, (i.e., SUN RGB-D, ETH3D, DIODE, and DDAD) and similar on iBims-1, but is only trained 150K images, which proves the effectiveness of SM 4 Depth. Especially, SM 4 Depth outperforms Metric3D by +58.08% and +8.10% mRI 𝜃 on ETH3D Outdoor and DDAD. In addition, SM 4 Depth outperforms Metric3D on nuScenes-val by -1.285 of RMSE, but falls behind on 𝛿 1 and REL, as Metric3D is trained on much more self-driving datasets, which endows it an advantage in such scenes. Notably, SM 4 Depth outperforms Metric3D by +58.08% and +8.10% mRI 𝜃 on ETH3D Outdoor and DDAD, respectively. Although not surpassing UniDepth in accuracy overall, SM 4 Depth closely approaches or even exceeds it on some datasets (iBims-1, ETH3D Indoor/Outdoor).Table 3 displays the results on NYUD and KITTI. With the zeroshot setting, our method obtains lower 𝛿 1 and higher RMSE than Metric3D on NYUD and KITTI. However, after being fine-tuned on NYUD and KITTI, SM 4 Depth achieves competitive accuracy with the state-of-the-art methods, while avoiding a significant degradation in accuracy on zero-shot datasets (see in Table 2 ). 6.3.2 Qualitative Result. Fig. 7 visualizes several methods' predictions and depth distributions. The 1 st -3 rd columns show close-up scenes challenging depth range determination. Previous methods obtain incorrect depth distributions, while Metric3D tends to push the background farther when the foreground boundary is clearly strategy [19] when using the best 𝐾. Fig. 8 shows all variants' performance on the mixing test sets. As 𝐾 increases, RMSE decreases slowly. RMSE suddenly drops below 3.4 when 𝐾 = 4 and increases again at 5 or 6. We argue that this phenomenon occurs because RDs better describe images with different appearance when 𝐾 = 4 and prevent excessive similarity between RDs due to redundant division. In addition, using the uniform partition strategy leads to a notable decrease in 𝛿 1 and RMSE.Categories Method SUN RGB-D iBims-1 Benchmark ETH3D Indoor DIODE Indoor 𝛿1 ↑ REL ↓ RMSE ↓ mRI 𝜃 ↑ 𝛿1 ↑ REL ↓ RMSE ↓ mRI 𝜃 ↑ 𝛿1 ↑ REL ↓ RMSE ↓ mRI 𝜃 ↑ 𝛿1 ↑ REL ↓ RMSE ↓ mRI 𝜃 ↑ Full-shot BTS [6.4.2 Ablation study. We conduct the ablation study by gradually removing our designs and comparing all variants on the mixing test sets. In Table 4 , the baseline (last row) consists of only an encoder-decoder structure and a Pyramid Scene Transformer [46] . Observably, the RMSEs increase overall as the proposed modules and innovations are gradually removed. The depth-variation based bins make the greatest contribution (+4.89% mRI 𝜂 ), indicating its effectiveness in learning large depth range gaps. The entire domainaware bin estimation increases mRI 𝜂 by 2.36%, with the weighted fusion scheme contributing 1.5% of this. In addition, the HSC-decoder improves mRI 𝜂 by 3.67%.  
conclusion: 
 This paper proposes a seamless MMDE algorithm, SM 4 Depth, to solve the problems of inconsistent accuracy across diverse scenes and reliance on massive training data. Firstly, we discuss the inherent issue of the bin-based methods when learning depth range with large gap, that is the large inconsistency of the same bin in different images. To address this issue, we propose the variation-based depth bins that allow the network to effectively learn scenes with different depth ranges. Next, to reduce the complexity of estimating correct metric bins from a vast solution space, this paper designs a "divide and conquer" method to determine metric bins from multiple solution sub-spaces, thereby reducing the network's reliance on massive training data. Finally, we propose an uncut depth dataset, BUPT Depth, to verify the accuracy consistency across scenes. Our method obtains outstanding performance with only 150K RGB-D pairs for training and achieves accuracy consistency. 
 
 Appendix:  


Decision: This paper introduces SM4Depth, a model designed for consistent monocular depth estimation across diverse environments. The model is well motivated and it achieves decent performance, with limited training data. Extensive experimentation demonstrates the effectiveness of the proposed method. Several concerns were raised by the reviewers on lack of consistency in performance across datasets, reliability of the ground truth in the proposed dataset. Some missing analysis and results were also reported. The authors addressed most of these concerns in the rebuttal, however,  reviewers found that some aspects like lack of consistency were not properly addressed. I recommend the authors to add the same in the camera ready version and also include the relevant points from the authors response. Overall, looking at the consensus, I am recommending for acceptance of this paper.

Reviews: review: paper_summary: paper_summary: This paper introduces SM4Depth, a novel algorithm for universal monocular metric depth estimation (MMDE) that addresses the challenges of maintaining consistent accuracy across diverse scenes and reducing reliance on massive training data. The paper introduces an uncut depth dataset, Campus Depth, to evaluate the accuracy consistency across various indoor and outdoor scenes. This dataset allows for the validation of depth accuracy across diverse environments.The proposed method outperforms existing approaches in terms of accuracy and efficiency, demonstrating superior performance with a smaller training dataset.
strength_weakness: strength_weakness: 1.	The paper introduces several novel contributions to the field of monocular metric depth estimation (MMDE). The variation-based depth bins proposed in this work offer a fresh perspective on modeling metric scale, addressing the challenges associated with scenes containing large depth gaps.
2.	The proposed SM4Depth algorithm has wide-ranging applications across various multimedia tasks, including video and image editing.
3.	The paper demonstrates a strong motivation to address practical challenges in the MMDE domain, such as inconsistent accuracy across diverse scenes and the reliance on massive training data.
limitations: limitations: 1. The paper suggests that by dividing the solution space into sub-spaces and estimating metric bins from various sub-spaces, complexity can be reduced. However, it's unclear how the division of the solution space into sub-spaces is performed and how these sub-spaces are defined.
2. Domain-Aware Bin Estimation  sounds promising in theory, it raises questions about computational efficiency and model scalability.
3. The paper introduces a "divide and conquer" approach to reduce reliance on massive training data by estimating metric bins from various solution sub-spaces. However, the rationale behind this approach is not thoroughly justified.
4. The paper lacks clarity on how incorporating metric bins into each stage of the decoder facilitates better depth range recovery and how it compares to existing refinement decoder structures.
5. It's unclear how representative the selected datasets are of real-world scenarios. For instance, are there specific challenges or biases in the datasets that might affect the performance evaluation?
6. The paper does not discuss potential failure cases or scenarios where the method might underperform.
paper_summary: paper_summary: This paper introduces SM4Depth, a model for consistent monocular depth estimation in various environments. It uses variation-based depth bins and domain-aware bin estimation to improve accuracy and reduce the need for extensive training data. The model's effectiveness is demonstrated by the experiments across multiple datasets. Additionally, the authors introduce a new dataset, Campus Depth, for robust evaluation of depth estimation models.
strength_weakness: strength_weakness: S1：SM4Depth achieves high performance with significantly fewer training data and only requires a single consumer-grade GPU for training.

S2：The well-motivated focus on the issue of depth consistency enhances the model's reliability and applicability.

S3：Extensive experimentation with various results supports the effectiveness of the proposed method.
limitations: limitations: W1: The authors claim their method has advantages in terms of training sample size, GPU usage, computational complexity, and parameter count. However, the experimental section seems to lack comparisons of training time and inference speed, which hinders a comprehensive assessment of the model's efficiency.

W2: The authors highlight the consistent depth accuracy of their method as one of the major advantages, providing impressive results in Figure 6. This figure shows their method significantly outperforms all others in outdoor scenes (indicated by the gray background). However, the model fails to achieve state-of-the-art performance on outdoor datasets like KITTI, DIODE Outdoor and ETH3D Outdoor (shown in Table 2 and Table 3). The conflicting results need further explanation from the authors.

W3: In the proposed Campus Depth dataset, the use of CREStereo's predictions as ground truth is questionable. It is unclear whether CREStereo itself suffers from inconsistent accuracy across scenes, raising concerns about the consistency and reliability of its outputs as ground truth.
paper_summary: paper_summary: This paper introduces a novel metric scale modeling method that reduces the graph channel ambiguity caused by width-based depth bins under large depth gaps in training data. Additionally, a divide-and-conquer strategy is employed to achieve training on consumer-grade GPUs. Furthermore, the authors propose a new dataset.
strength_weakness: strength_weakness: 1) This method appears to be highly lightweight, with significant reductions in parameter count and computational cost compared to other approaches.

2) The use of variation-based depth bins to address bin ambiguity issues sounds very interesting.
limitations: limitations: 1) The authors should provide more detailed explanations in the paper regarding the differences between this dataset and other existing datasets.

2) Why is there such a significant different performance between SM4Depth and Metric3D/Unidepth on Campus Depth and other datasets?

3) The Depth variation based bins proposed in the paper may have certain limitations. It appears that they may not be as effective for short-range depth estimation. It would be beneficial if the authors could provide additional evidence of its effectiveness from various perspectives.
paper_summary: paper_summary: This paper has proposed a novel metric scale modeling method for universal depth estimation with a single model. A new dataset is collected by using the ZED stereo camera. Extensive experiments are conducted which verify that the proposed SM4Depth model achieves competitive accuracy and generalization capacity on unseen data, while trained on 150K RGB-D image pairs. The source code has also been made available to foster future research.
strength_weakness: strength_weakness: 1. A large-scale dataset is collected using the ZED stereo dataset. 
2. The paper is overall well written and presents extensive experimental analyses, which verifies the effectiveness of the proposed method for universal depth estimation using a single camera.
3. The generalization capacity of the proposed model is verified in diverse scenes.
4. The proposed method succeeds in achieving state-of-the-art accuracy using only 150K RGB-D training pairs.
limitations: limitations: 1. While the generalization capacity in diverse scenes has been verified, it would be nice to show some results in adverse scenes for analysis, which is an important aspect for assessing generalizability in real-world applications. For example, results in adverse weather conditions and specular surfaces could be discussed.
2. As the proposed dataset leverages the ZED camera for data collection, how to ensure that the depth estimation ground truth is reliable?
3. In Table 1, the Depth Anything model could be compared for analysis.
4. Some failure cases and challenging cases of such universal depth estimation models could be presented.
5. While this work follows a setting of a fixed FoV, it would be nice to discuss whether the method can scale to other cameras like active stereo cameras and panoramic cameras with wide FoV.
6. In the main paper, the description of the dataset should be greatly enriched as this is one of the important contributions of this paper. The distribution of different scenes and the diversity of the conditions covered in the dataset could be presented.