Metric from Human: Zero-shot Monocular Metric Depth Estimation via Test-time Adaptation
Authors: Yizhou Zhao
Hengwei Bian
Kaihua Chen
Pengliang Ji
Liao Qu
Shao-yu Lin
Weichen Yu
Haoran Li
Hao Chen
Jun Shen
Bhiksha Raj
Min Xu
Venue: NeurIPS 2024 poster (2024)
Keywords: Metric Depth Estimation
Human Mesh Recovery
Test-time Adaptation
Primary area: machine_vision

Abstract: Monocular depth estimation (MDE) is fundamental for deriving 3D scene structures from 2D images. While state-of-the-art monocular relative depth estimation (MRDE) excels in estimating relative depths for in-the-wild images, current monocular metric depth estimation (MMDE) approaches still face challenges in handling unseen scenes. Since MMDE can be viewed as the composition of MRDE and metric scale recovery, we attribute this difficulty to scene dependency, where MMDE models rely on scenes observed during supervised training for predicting scene scales during inference. To address this issue, we propose to use humans as landmarks for distilling scene-independent metric scale priors from generative painting models. Our approach, Metric from Human (MfH), bridges from generalizable MRDE to zero-shot MMDE in a generate-and-estimate manner. Specifically, MfH generates humans on the input image with generative painting and estimates human dimensions with an off-the-shelf human mesh recovery (HMR) model. Based on MRDE predictions, it propagates the metric information from painted humans to the contexts, resulting in metric depth estimations for the original input. Through this annotation-free test-time adaptation, MfH achieves superior zero-shot performance in MMDE, demonstrating its strong generalization ability.

Metric from Human: Zero-shot Monocular Metric Depth Estimation via Test-time Adaptation
Yizhou Zhao', Hengwei Bian', Kaihua Chen', Pengliang Ji', Liao Qu', Shao-yu Lin', Weichen Yu', Haoran Li Hao Chen Jun Shen², Bhiksha Raj', Min Xu
Carnneie Mellon University. Pittsburgh University of Wollongong, Wollongong https://githn: http://tuub.ccooSaaaaaaa//////// com/Skaldak/MfH
Abstract
Monocular depth estimation (MDE) is fundamental for deriving 3D scene struc- tures from 2D images. While state-of-the-art monocular relative depth estimation (MRDE) excels in estimating relative depths for in-the-wild images, current monoc- ular metric depth estimation (MMDE) approaches still face challenges handling unseen scenes. Since MMDE can be viewed the composition of MRDE and metric scale recovery, we attribute this difficulty to scene dependency, where MMDE models rely scenes observed during supervised training for predicting scene scales during inference. address this issue, we propose to use humans landmarks for distilling indee-indeeendent metric scale priors from generative painting models. Our approach, Metric from Human (MfH), bridges from general- izable MRDE zero-shot MMDE in memeerte-and-sstinee
manner. Specifically, MfH generates humans on the input image with generative painting and estimates human dimensions with off-the-shed human mesh recovery (HMR) model. Based on MRDE predictions, it propagates the metric information from painted humans to the contexts, resulting in metric depth estimations for the original input. Through this annotation-fiee test-time adaptation, MiH achieves superior zero-shot performance in MMDE, demonstrating its strong generalization ability.
M Monocular Metric Depth Estimation
Monocular Relative Depth Estimation
In-the wildd Image
Pstimated M Metric Depths (e) Fully Supervised Monocular Metric Depti Estimetion (MMDE)
In whldwild Image (b) Fully Sugervised Mococular Relative Depth Estimaiion (MRDE)
Peb satmmttdd Relative Depth
Generation with Generative Painting
Opimiizatinn
5,5 argmin Σ
Estimation with MRDE & Human Mesh Recovery (HMR)
la-the- wild Image
Astimated Metric Depth
6G f++tx(
Metric Human (MfH):
MRDE
Test-time dapiaation Zaro-shot
Figure Illustration of our motivation. (a) Fully supervised MMDE cannot generalize well on unseen data as (b) MRDE, with its reliance on training scenes for predicting metric scales during test time. (c) Hence, we develop MfH to distill metric scale priors from generative models in generate-and-estimall manner, bridging the gap from generalizable MRDE to zero-shot MMDE. We use grayscale to represent normalized depths MRDE predictions, while colormap mapping metric depth from meters to RGB values in MMDE results. In @, z(-) denotes rasterized metric depths.
"Corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

iilmme DIML Outdorr
DIODE Indoar DIODE Oundooe
RiperRin DDAD
SUN Virual KITTI
ZoeDepth
UniDepth
MetricSD
(74k)
ZepoDe
Marigold
Depti Auything
RITTI AbaRel (%K
80-
Desth Any thing (1.53)
MensicaD (SM))
60-
8
40-
5
ZoeDegch (2 7M;
20-
ZernDepth (LYM)
Uniteepth (3M))
11
0-1
0.3
0.4 0.5 0.6
NYU+2- - AbeRel (5)
Maximam Coeine Sauilarit with Teaining Somples
Comparison of state-of-the-art MRDE and MMDE methods in terms of AbsRel and the number of training samples. Marigold [ and Depth Anything [2] are de- signed for MRDE, while the rest are for MMDE. We observe MMDE approaches require notably more data to achieve similar AbsRel as MRDE.
Figure 2:
Figure 3: MMDE dN versus the maximum co- sine similarity between each test sample and all metric-annotated training samples. "x/o"" from indoor/outdoor datasets. We see that the scale-related performance of a test sample pos- itively correlates with its similarity to training samples. Details can be found in Appendix A.
Introduction
Monocular depth estimation (MDE) is essential in understanding the 3D structure of scenes from 2D images and has many applications in robotics 3 autonomous driving (5] 6). and virtual reality 7 g. It requires recovering depth information from single image without relying on additional sensors or stereo cameras, thereby being inherently ill-posed.
Recent literature mainly explores MDE in two branches, namely, monocular relative depth estimation (MRDE) 2 D 9 and monocular metric depth estimation (MMDE) [TO]17]. MRDE estimates normalized depths or disparities by factoring out the scale. Its scale-invariant nature enables large- scale training on diverse datasets with distinct camera parameters, while the cost of bringing in scale ambiguity. In contrast, MMDE predicts absolute depths in meters. Due to the unbounded output range and the intertwined relationship between depths and focal lengths, early works of this line often cannot perform well on test data with arbitrary scene scales or camera intrinsics. To compensate for this, recent progress resorts to injecting scene information [12] or camera information [10] [1] [4]
into the model. The former attempt learns scene-specific scale priors, modeled with metric heads for indoor or outdoor scenes, and uses either to transform relative depths into metric depths in a heuristic manner. The latter aims to disambiguate scale prediction with extra camera inputs. However, as
shown in Fig. 2 both lines of work require notably larger amounts of labeled training data to achieve similar mean absolute relative errors (AbsRe) their MRDE counterparts.
What causes the data hunger of MMDE, and what makes MMDE harder to generalize? Given that MMDE can be viewed as the composition of MRDE and metric scale recovery, we posit the latter might be the primary factor. MMDE models might face challenges in inferring scene scales without sufficient exposure to similar annotated scenes during training. which is not a problem for scale- invariant MRDE. To validate our assumption, we evaluate a scale-related metric (61) of an MMDE model, ZepDepth 12), on randomly sampled test images. Meanwhile, we calculate the maximum cosine similarity between each test sample and all training samples with metric annotations using DINOv2 [1]. Our findings from Fig. 33 indicate a clear trend: higher similarity to training samples positively correlates with better performance, and vice versa. This reflects scene dependency of MMDE models, likely arising from their supervised training paradigm. In other words, they tend to learn an implicit mapping between training scenes and metric scales from <image, metric annotation pairs, As result, adapting to novel scenes may require extra domain-specifi fine-tuning.

---

To address this dependency for better generalization capabilities, we propose to avoid scene-dependent supervised learning, while leveraging ceene-independent metric scale prior. Our insights are two- fold. First, we observe generative painting models can paint objects of proper sizes based on partial contexts, indicating an underlying sense of scales. Additionally, humans can be potentially utilized as relatively universal landmarks, since humans exhibit sizes that are generally more comparable to each other than other common in-the-wild objects, e.g, tables, trees, and cars. To explicitly derive a metric scale prior from generative painting models, we notice state-of-the-ar human mesh recovery (HMR) approaches [1921 can robustly estimate human dimensions for in-the-wild images. Also, they typically output SMPL 22 23 representations with shape space defined in meters. While the input image does not guarantee to include humans, we speculate an off- shellf image painting model can paint proportionate humans in the scene, which provides opportunity to retrieve metric-scale information for the original input by measuring painted humans. Hence, we introduce a test-time adaptation pipeline, Metric from Human (MfH), illustrated in Fig. [ Concretely, it 1) paints humans with partial contexts of the input image, 2) estimates human dimensions from the painted image, and 3) propagates the metric-scale information from humans to the contexts for MMDE. In this way, we can distill the metric scale prior hidden inside the generative painting model, unleashing its power to comprehend diverse scenes. As result, our MIH mitigates the scene dependency issue in fully supervised MMDE, thereby being potentially more generalizable to unseen scenes.
Our contributions can be summed up follows:
1. We discuss that the current obstacle for generalizable MMDE lies in scene dependency and propose to use a scene-independent metric scale prior as solution. Further, we find possible to establish such prior by distilling from generative painting models.
2. To extract the metric scale prior from generative painting models for zero-shot MMDE, we design test-time adaptation framework, Metric from Human (MfH). Using humans as landmarks, we bridge from MRDE to MMDE by generate-anee andtttiitt pipeline.
3. Through qualitative and quantitative experiments, we demonstrate the superiority and gener- alization ability of MfH in zero-shot MMDE, needless of any metric depth annotations.
2 Related Work
Monocular Depth Estimation (MDE) has garnered significant interest in recent years. Early approaches focused on supervised methods that predict either monocular metric depth estimation (MMDE) [16] 24 [5] or monocular relative depth estimation (MRDE) (27-29 9. Despite remarkable progress in network architectures [30 16
[3] 34 [5], existing MMDE methods often confine their training and testing to specific domains, leading to performance degradation under minor domain shifts and poor generalization to unseen environments. In contrast, relative depth models have demonstrated better generalization by leveraging scale-invarial losses [9 35 36 on diverse datasets. However, these models cannot recover metric scales, which are crucial for downstream applications. Recent works explored generalizable MMDE models [12 I1 10] for diverse domains, leveraging or normalization based on camera properties 38 17 14]. They often require fine-tuning to adjust to specific domains [1] 10].
camera awareness through explicit incorporation of intrinsics 37
Several recent studies explore zero-shot MMDE, using language as a prior to ground predictions to metric scale [39/4]] However, their hand-crafted depth captions to connect the language and metric worlds are often too coarse to capture accurate depths. Our MIH instead distills metric scale priors from generative painting models, enhancing both the generalization capability and pixel-wise precision of zero-shot MMDE models without relying on metric depth annotations.
Human Mesh Recovery (HMR) aims to reconstruct 3D human bodies from visual inputs. Optimization-based HMR relies on iterative optimization techniques fit parametric body models such as SMPL to detect image features. Examples include SMPLify [42] and its variants 23 43 which iteratively minimize an objective function to align the model with 2D key points and silhouettes. In contrast, feed-forward methods 44 directly regress the body shape and pose parameters from a single image using deep learning techniques. Among them, HMR 2.0 20 is fully transformer-bant based approach for recovering 3D human meshes from single images. We adopt it as our HMR model for estimating in-the-wild human structures and poses. With HMR, we derive metric scale priors from generative painting models, thereby bridging generalizable MRDE to zero-shot MMDE.
3

---

Tod-lime Training
Edentity Negation Hadamard Prod.
Frozen
Human Mesh
Torrable Data Flow
Recovery
Supervasion
Human Moshes (V..)
Human (D,x)
Gencrative Painting
Binary Erosdon
Image
Paimed Images (l,)
Masis (Max)
(DG)
(h man" woman") Text Prompts (Pt
Dg
Monocular
Relative Depth
Metric Hrad
Estimation
Musk Prompts (prtast
Relative Degths (Dx"))
Metric Depths (62)
Inference
feesE
+
Relative Depth
Metric Head
Estimation
Fst. Relative Depth Drd
Est. Metric Depti D
Figure 4: The framework of Metric from Human (MfH). Our pipeline comprises two phases. The test-time training phase learns metric head that transforms relative depths into metric depths based on images randomly painted upon the input image and the corresponding pseudo ground truths. After training the metric head, the inference phase estimates metric depths for the original input.
Method
Taking an RGB image I R H with its camera intrinsic K E RR333 as input, we aim to estimate its pixel-wise metric depths Dm E IR #xxx1, Unlike existing MMDE methods [12 14 [0] that typically train on images with metric annotations to unseen inputs, we instead consider test-time adaptation scenario. That is, to estimate metric depths on certain image without training on domain-specific metric annotations. To achieve this, we propose a framework to learn metric head for each input as depicted in Fig. 4 The metric head is concatenated after an off- off-the-shelf MRDE model to transform relative depths into metric depths. While the inference pipeline is simplistic, landmarks during test-time training. Since humans are not guaranteed to exist in in-the-wild images, introduce a generate- generate-and-cn and--adddiieee and-estimate method in Sec. 3.2 to extract metric scale information from the given image. This extracted information together with the estimated relative depths allows us to approach annotation moooeetion-free zero-shot metric depth estimation, outlined in Sec. 3.3
expect the trained model to generalize
key insight is to use humans metric
3.1 Preliminaries
3.1.1 Monocular Depth Estimation (MDE)
Assuming a pinhole camera model, we have the following relation
b b-f D rel(D")
D - ((D))
D
(1)
dm
s(Dm)
where and are the camera baseline and focal length, 1 and d are metric depths and disparities, and s(-) and t(-) are scalar functions, denoting a scale and translation to normalize input. use superseript m and rel to refer metric and relative values, accordingly. Due to the correlation between camera parameters and the scale of depth. MMDE cannot generalize well the camera intrinsic is unknown the scene scale is hard to predict, e.g., when the scene is unseen during training. In contrast, MRDE enjoys better generalization ability for its affine-invarian formulation.
3.1.2 Human Mesh Recovery (HMR)
We adopt state-of-the-ant HMR model, HMR 2.0 (20, for reconstructing camera-frame human meshes from an image i, H with K people. Starting with human segmentation masks from Mask R-CNN [50), HMR 2.0 predicts SMPL [22] parameters for each
(MA Hxw ]==

---

human as Pnn, nuk Buk. Fuk Tnk]k= k= These parameters represent global orientation Pnk E R body pose Ank E R22x3xx3 shape Bak E RR,, and root translation nk E R Then the human body meshes with vertices nk E p3x6890 can be recovered with the SMPL model
nk SMPL ( nkk P Ban) + Taks where h = ..2,...,K. K.
(2)
Since the shape space of SMPL is defined in meters, the generated vertices (V are also in meters. As result, all HMR regressors, such as HMR 2.0 we use, inherit such data prior.
3.2 Generating Humans as Metric Landmarks
To start off, we randomly place people on the input image with generative image painting, with text prompts (P fn= sampled from {"a man", "a woman") and mask prompts (Pmask _
Jn=1
===1 xij sampled from rectangles smaller than the whole image. We write P.. (Plen Pmask) as shorthand. Once a time, we randomly generate N painted images with humans
i paint (I)Px. where n =
and (In E |'XWx3J Jn=1 are the painted images, We observe the generative image painting model can paint people of proper sizes that are compatible with the unmasked background. This allows us to use the painted people as landmarks to inform our model of the metric scale. To this end, we fed these painted images into HMR 2.0 to predict human instance segmentation masks (M.a ) N,K I==1k= and meshes [v J == where the subscripts denote the k-th person in the nith image. We obtain the pseudo metric ground truths D with rasterization
D min (erode (Mns O Sakk) @ Dnn),
where S.k. Dn - P(K, V n*k)) the rasterized silhouettes and depths, respectively, and @ stands for the Hadamard product. We erode the intersection of the instance segmentation mask M.k and the rasterized silhouette S., avoid overlapping and take the minimum of k depth maps that the depth values from closer humans can occlude the farther ones. Using these pseudo ground truths, we supervise the learning of the metric head with the scale-invariant log (Slog) loss [5]]
2
Lsho (D D
HW Σ 2
Σ
(5)
(HW)2
where = log Dm log D with DT being the estimated metric depth map, subscript. denotes the index of each pixel, and À E [o, 1]. Since the rasterized depths D are meters and the first term in Lsing is pixel-wise 12, this loss provides crucial metric scale information to the metric head.
3.3 Transforming Relative Depths into Metric Depths
During training, we estimate relative depth map D for each painted image I, with pre-trained MRDE model, and learn metric head to transform Diel into estimated metric depth map Dm Similarly, we obtain Deel and D from the original input I during inference. According to Eq. {T). we can learn simple linear layer as our metric head, i.c., D 3. D + f. In addition, we need account for the difference between the relative depths predicted from the original image Drel and those from the painted images (D on the unpainted regions. Considering the iffine-invariant nature of MRDE, we further decompose metric depth predictions with
m = s- (s, Dee + f.)) + = D +t, where D = = Diel tt.
(6)
and {sn}. (th. s R are optimizable parameters. Then we align Deel and (Dre) with
LMSE(D,)) D = (1(1-1
- maskk) pmink) @ (D
D
This objective considers the pixel-wise alignment of unpainted regions with affine transform specific to each painted image. Finally, we formulate our complete objective function as
min
Eshog (D. D + Σ min LMSF (Drel )
(8)
Susta
5

---

Table 1: Performance comparisons of our MfH and state-of-the- methods the NYU-Depth V2 and KITTI (55] datasets.
22
'LORN uses 200 images and 2,500 partial images for training.
NYUv2
KITTI
Method
Supervision
81 1
AbsRel
Slao
RMSE
51 f
AbsRel
Shis
RMSE
ZeroDepth
many-shot
90.1
10.0
0.380
89.2
10.2
4.38
Metric3D 14
many-shot
92.6
9.38
9.13
0.337
97.5
5.33
7.28
2.26
UniDepth- C
10
shon
97.2
6,26
6.41
0.232
97.9
4.69
6.71
2.00
UniDepth-V 10
many-shol
98.4
5.78
5.27
0.201
98.6
4.21
5.84
1.75
LORN [58]
few-shot
70.3
101
9.452
Hu et al. 40
one-shot
42.8
34.7
1.049
31.2
38.4
12.29
Depth
59
zero-shot
39.4
38.8
1.167
28.1
47.3
12.96
MIH (Ours)
zeto-shot
83.2
13.7
9.78
0.487
81.2
13.3
10.5
4.21
Such formulation propagates metric scale information from human pixels to background pixels, thereby enabling the metric head to predict for the non-human context. After optimization, it is then possible to infer metric depths for the original input image. While one can deploy fancier metric head and apply up-to-affine consistency constraints between aligned relative depth predictions D and metric depth predictions DT for better robustness, we observe simple affine transform capable of providing good predictions, leaving further parameterization for future works.
Experiments
4.1
Experimental Setting
Under our test-time adaptation setting, we do not train on any datasets but only test on each input image directly after image-specific optimizations. Specifically, evaluate the zero-shot MMDE capability of MfH on NYU-Depth V2 [52]. IBims-1 [53], ETH-3D [54] with the split
Datasets.
from 133 and official masks, and KITTI [55] with the corrected Eigen-split from (5I). Following
prior works 112 15 T6]. we apply the Eigen evaluation mask [5]] on NYU-Depth V2 and IBims-1
while the Garg evaluation mask (56) on KITTI.
Evaluation Metries. We employ several common metrics to assess the performance of baseline methods and our model. The 8, ZA Den max
1.25 metric evaluates the fraction of predicted depth values that are within threshold factor of their corresponding true values; the Mean Absolute Relative Error, AbsRel HW ΣNY [De 1
Dani Dz. Dg
Dg measures the average absolute difference between the predicted and true depth values, normalized by the true depth; the Scale Invariant Logarithmic Error, Shog = 100 /Var(log D Dpred - log Dr), quantifies the error in a logarithmic scale that is invariant to the absolute scale of the scene; the Root Mean Squared
Error, RMSE V HW Σ (Dgr D,rea)?, focuses the square root of the mean of squared differences between the predicted and actual depth values, emphasizing larger errors.
Implementation Details. We adopt Depth Anything 2 without fintuning on metric annotations as MRDE model, Stable Diffusion v2 [57] for generative painting, and HMR 2.0 20 for human mesh recovery. In Lsting we follow ZoeDepth [12 to set the X = 0.15. For optimizing the alignment parameters {8,), ).,), we leverage linear regression to obtain a close-formed solution. As for optimizing the metric head parameters, t. we use the L-BFGS optimizer with fixed learning rate of for 50 steps. Unless otherwise specified, we randomly paint 32 images for our comparison one NVIDIA A100 GPU.
experiments and 4 for ablation studies. All experiments
4.2 Comparison Results
We first evaluate the MMDE results on NYU-Depth V2 522 and KITTI (55), common benchmarks with indoor and outdoor scenes, respectively. In Tab. J we show that our zero-shot MfH consistently outperforms other approaches trained with few/one/zero-shot supervision. This indicates that genera- tive painting models are capable of capturing metric scale information, which is potentially more accurate than that embedded language models 44 [9]. With our generate-and-estimal pipeline, the metric scale prior hidden inside generative painting models can be leveraged for zero-shot MMDE.

---

Table 2: Performance comparisons of our MfH and many-shot methods the DIODE (Indoor) 60 iBims-1 1 [61, and ETH3D [54] datasets. *-(N, K, NK): fine-tuned on NYUv2 [52], KITTI [55 or the union of them. We re-evaluate all results with consistent pipeline for metric completeness.
DJODE (Indoor) Slos
iBims
ETHED
Method
Absltel
RMSE
6,
AbsRel
Slag
IMSE
AbsReel
Slaz
RMSE -
ZreDepti-NK 12
38.8
33.0
13.3
1.598
61.0
18.7
8.98
0.778
33.5
47.3
14.0
2.094
Depth Anything-N
29.7
32.7
12.5
1.4%6
71.3
15.0
7.58
0.594
25.2
38.7
10.2
2.327
Depth Anything
2
11.1
231
15.5
5.199
2,88
217
17.2
5,385
16.9
136
17.1
4.202
ZeroDepth [1]
30.0
1.392
32.6
19966
Metric3D [4]
26.8
14499
14.4
0.646
34.2
2.965
UniDepth-C 10
62.8
23.8
11.5
0.968
$1.
14.3
8.30
0.536
43.3
35.5
10.3
1.532
UniDepth-V - [0]
79.8
18.1
10.4
0.760
23,4
35.7
6.87
1,063
27.2
43.1
8.93
1.950
(Ours)
34.5
1.363
9.73
Table 3: Ablation study for MRDE models and optimization targets on the NYUv2 dataset. True depth/disparity represents the performance with oracle depths/disparities as optimization targets.
Table 4: Ablation study for optimization param- eters and optimization targets on NYUv2. We optimize the predictions in the same space as optimization targets, i.e., the depth space for depth targets and the inverted depth space for
Medel
Optin: Torgrt
4
AbaRel 1
Sl
RMSF
disparity targets. The same applies to Tab.
mue depth
12.4
24.5
0.692
ZoeDopth
peintal deph
mu disparity
29.5
30.1
35.5
23.2
50.6
0.658
Optim. Paran
Optin.. Torget
6
AbwRel 1
Shy
RMSSE
peinted disparit
32.3
25.1
1.158
tree disparity
tue depth
5,88
paintel disgurity
28.0
21.2
1,079
Mangood
cue disparity
painted depth
painted diparity
21.4
31.1
15.1
20.9
31.3
19.5
1,086
(n)).
true disparity
paintal disgurity
74.4
43.5
36.6
Depth Anything
paintel
me disparity
eue depth
peintal disparity
(a))
(A.).aa).
tree disparity
paintea disgurity
tree disparity
paintel disgurity
98.9
4.57
21.9
113
In Tab. 2] we further provide comparison between MfH and state-of-the-ar many- methods, which are typically trained upon large-scale datasets with dense metric depth annotations. Our model achieves performance on par with, and sometimes superior to, these approaches, especially on ETH3D (54), which contains both indoor and outdoor scenes. It is noteworthy that these prior arts can do well on certain datasets while failing on others. For instance, ZeroDepth [Л]] performs well on iBims-I but struggles to estimate depths on DIODE (Indoor) and ETH3D accurately. Similarly, UniDepth-V [0] shows promising results on DIODE (Indoor) while underperforming on the other two benchmarks. These findings signifyy the scene-dependent nature of existing fully supervised methods, which may result in degraded performance on unseen scenes. In contrast, our model demonstrates robust zero-shot generalization capabilities across diverse scenes. We further highlight the comparison among our MiH and Depth Anything (2] fine-tuned on NYUv2 or KITTI (2nd 3rd rows). These methods adopt common Depth Anything MRDE backbone while deploying different strategies for MMDE. The results demonstrate that our test-time adaptation strategy generally works better than domain-specift fine-tuning, without the need for training on metric depth annotations.
4.3 Ablation Study
Impact of MRDE models. In Tab. B we investigate the performance of MíH with different MRDE models and various optimization targets. For ZoeDepth [12] and Depth Anything [2, we only adopt their pre-trained MRDE backbone as our MRDE model. Note that we use ground truth depths or disparities as optimization targets show approximate upper bound of performances, while only accessing painted depths or disparities as pseudo ground truths during real test-time adaptation. We also ensure consistency by optimizing predictions in the same space the target, i.c., the depth space for depth targets and the inverted depth space for disparity targets. Overall, we observe using Depth Anything as our MRDE model and painted disparities as our optimization target shows the best performance. Comparing the first two rows and the last two rows for each MRDE model, we see that optimizations in the disparity space yield superior results for Zoe 12 and Depth Anything (2), whereas optimizations in the depth space prove more effective for Marigold M. A similar scenario can also be found the last two rows of each MRDE model. This variation performance may stem from the difference in their output spaces. To be concrete, ZeeDepth and Depth Anything produce inverted depths, while Marigold outputs depths. Optimizing in the original output space can provide better numerical stability, leading to better optimization results.
7

---

Table Ablation study for loss functions on NYUv2.
Figure 5: Ablation study for the number of painted images on
NYUv2. Horizontal axis: N E {4, 8, 16, 32] inpainted images.
Loss
161
Abwltel
Shy
RMSE
Tue (n80)
24.2
201
26.9
4.645
RABE
62.3
16.7
0.975
MSF
67.5
10
(a) Inpat & Output
(b) Painted Images & Recovered Human Meshes with Aligned Point Clouds
Figure 6: An alternative perspective on MfH. (a) Using camera intrinsics, the estimated metric depths can be transformed into a metric point cloud. (b) Our -nateet-a process can be seen as aligning the estimated point cloud of each painted human to its corresponding human mesh.
Impact of optimization parameters. In Tab. we verify the benefits of aligning relative depth estimations of the original input and painted images with {s,,.. } {tn] and parameterizing the metric head with s, f. We view the results from bottom to top. First, optimizing all parameters (7th rows) yields the lowest error. When we fix the scaling factor to while keeping the other parameters optimizable (5th rows), the model has the highest error and cannot be aligned with the ground truth. Instead, when using optimizable scales 4thh4th rows) in the metric head, the model can better capture depths, which indicates an accurate scene scale is crucial in MMDE. Removing the alignment between the input image MRDE and painted image MRDEs 21 rows) results in sub-optimal predictions since the same contents on two different images might result in distinct MRDE predictions. The performance difference of using different optimization parameters while the true disparity as the target (3rd vs. 5th vs. 7th row) shows it is possible to apply an affine transformation upon MRDE to achieve good MMDE predictions, if with accurate scale and translation recovered.
Impact of loss functions. In Tab. 5 we ablate the effect of using various loss functions for test-time training the metric head. Notably, employing an l loss (1st row) yields inferior performance compared to losses incorporating an la term (2nh rows). This is probably because our generate-and -nd-ettimn---e process can introduce noises to a certain degree. Since the 1 loss treats all errors equally, regardless of their magnitude, it can be more sensitive to small perturbations. An la term that focuses more on large errors thus provides better robustness. Furthermore, a comparison between the 2nd and the last two rows shows optimizing in the log space brings better performance, which is expected since logarithmic transformation tends to mitigate the impact of outliers. This also accords with the general experience in training depth models using ground truth annotations, suggesting that the depths of generated humans might also be normally distributed in the log space, akin to real-world scenarios. From the last two rows, we see optimizing with the MSEn loss or the Shog loss is not discrepant by much. We opt for the Shog loss in our optimization process due to its relatively superior AbsRel.
Impact of painting numbers N. In Fig. 5 we analyze the effect of increasing the number of painted images with humans. Specifically, we paint 4, 8, 16, and 32 images with humans, plotting curves as well as error bars for various metrics against the painting numbers. To draw reliable conclusions, we conduct experiments across five consistent random seeds for each painted image quantity. Our analysis reveals a clear linear association between the per-sample runtime and the number of painted images, as depicted in the time plot. The 8, and RMSE plots show an upward trend in MMDE performance with the increment in painting numbers, albeit sublinearly. By further examining the error bars, we see a larger number of painted images results in better robustness of predictions, which is demonstrated by smaller, gradually converging standard deviation.
8

---

0.5
Diode (Indoo)
0.15
0.10
0.05
0.00
1Bims-
0.30
5.00
0.30
0.25
(a) Input& GT (b) ZoeDepth-NK
(c) ZeroDepth
(d) UniDepth-C
MíH (Ours) Meters
bsRel
Figure 7: Zero-shot qualitative results. Each pair consecutive rows corresponds to one test sample. Each odd row shows input RGB image alongside the absolute relative error map, while each even row shows the ground truth metric depth and predicted metric depths.
4.4 Qualitative Analysis
We present an alternative view of our MfH in Fig. 6 With camera intrinsics, the estimated metric depths for both the painted images (D.) and the original input D correspond to metric point clouds. Our MfH stretches the point clouds for (Dm) along the z-axis by setting human landmarks in 3D, revealing the 3D structure of the unpainted background in meters. With random painting, we progressively capture the 3D structure of the entire original input, thus bridging MRDE to MMDE.
demonstrate metric depth predictions and pixel-wise A AbsRel in Fig. 7 highlighting the strong zero-shot generalization ability of our MfH. Additionally, we show MMDE results for in-the-wil wild samples captured by DSLR cameras and smartphones in Fig. 8 Besides the robust performance of MIH. we observe that fully supervised MMDE methods like UniDepth [10] often provide bounded metric depths, inheriting from the limited range of sensors used in their training ground truths. In contrast, MfH can provide more flexible results.
Case studies, user studies, and more qualitative analysis can be found in Appendix
5 Conclusion
We present MfH, method that infers metric depths from in-the- images without the need for training on metric depth annotations. Utilizing humans landmarks extract metric scale priors from generative painting models, our approach addresses the challenge of scene dependency inherent MMDE trained with metric depth supervision. Through test-time adaptation pipeline, MfH effectively captures metric scale information from images by generating and estimating humans, which then leveraged for zero-shot MMDE. Our experiments demonstrate that MfH achieves superior performance and better generalization ability compared to existing methods.

---

20
(e) Input.
(b) UriDepth-C
(c) (Oun()
Meters
(60 Ingaut.
(e) UniDepth
(1 MIH (Ouns)
Mehes
Figure 8: wn-the-wild qualitative results. Each group of rows (a)-(c) (d)-(f) corresponds to one in-the-wild test sample captured by DSLR camera or a smartphone.
Limitation discussion. Our MfH works based on the assumption that humans can exist in the scene that it is possible to paint human upon the input image. While this holds for most usages of MMDE, it might not be ideal for some cases, e.g., close-up scenes. This opens new challenges, such as incorporating objects other than humans into the generate-and-estimaal pipeline as metric landmarks. Another assumption is the MRDE predictions align with true depths up to affine. Despite the training objectives of MRDE being linearly transformed true depths, the MRDE predictions can contain noises, making the linear metric head hard to capture accurate metric depths. Whether other parameterizations of the metric head can tackle this remains an open question.
Broader impacts. Our MiH decreases the demand for metric depth annotation which commonly requires depth sensors stereo systems, making MMDE models more environmentally friendly. However, its usage of human-related models can perpetuate biases present in the training data, leading unfair discriminatory outcomes.
Acknowledgments and Disclosure of Funding
This study was partially funded by U.S. NIH grants ROIGMI34020 and P41GM103712. NSF grants DBI-1949629, DBI-2238093, IIS-2007595. IIS 21--2215555 1597, and MCB-2205148. Additionally, received support from Oracle Cloud credits and resources provided by Oracle for Research, as well as computational resources from the AMD HPC Fund.
10

---

References
[1] Bingxin Ke, Anton Ohukhov, Shengyu Huang. Nando Metzger, Rodrigo Caye Daudt, and Konrad Schindler. Repurposing diffusion- image generators for monocular depth estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), 2024.
[2] Lihe Yang. Bingyi Kang, Zilong Huang. Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything: Unleashing the power of large-scale unlabeled data. In IEEE Conference on Computer Vision and Pattern Recognittin (CVPR), 2024.
[3] Xingshuai Dong, Matthew A Garratt, Sreenatha G Anavatti, and Hussein A Abbass. Towards real-time monocular depth estimation for robotics: A survey. IEEE Transactions on Intelligent Transportation Systems, 23( 11)::16940-16961 2022.
[4] Diana Wofk, Fangchang Ma, Tien-Ju Yang, Sertac Karaman, and Vivienne Sze. Fastdepth: Fast monocular depth estimation embedded systems. In 2019 International Conference Robotics and Automation (ICRA), pages 6101 6108. IEEE, 2019.
[5] Dennis Park, Rares Ambrus, Vitor Guizilini, Li, and Adrien Gaidon. pseudo-lidar needed for monocular 3d object detection? In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3142-3153 2021.
[6] Wanli Peng. Hao Pan, He Liu, and Yi Sun. Ida-3d: Instance-depth-aware 3d object detection from stereo vision for autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13015-13024, 2020.
[7] Adam Jones, Edward Swan, Gurjot Singh, Fric Kolstad, and Stephen Ellis. The effects of virtual reality, augmented reality, and motion parallax on egocentric depth perception. In Proceedings of the Sth symposium on Applied perception in graphics and visualizution, pages 9-14, 2008
[8] Fatima El Jamiy and Ronald Marsh. Survey depth perception in head mounted displays: distance estimation in virtual reality, augmented reality, and mixed reality, IET Image Processing, 13(5):707-712 2019.
[9] René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. IEEE transactions on pattern analysis and machine intelligence, 44(3):1623-16377 2020.
[10] Luigi Piccinelli, Yung-Hsu Yang, Christos Sakaridis, Mattia Segu, Siyuan Li, Luc Van Gool, and Fisher Yu. Unidepth: Universal monocular metric depth estimation. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[11] Vitor Guizilini, Igor Vasiljevic, Dian Chen, Rares Ambrus, and Adrien Gaidon. Towards zero-shot scale-aware monocular depth estimation. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pages 9233-9243. 2023.
[12] Shariq Farooq Bhat, Reiner Birkl, Diana Wofk, Peter Wonka, and Matthias Müller. Zoedepth: Zero-shot transfer by combining relative and metric depth. arXiv preprint 1iXXx::3322.122888 2023.
[13] Luigi Piccinelli, Christos Sakaridis, and Fisher Yu. idise: Internal discretization for monocular depth estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21477-21487, 2023.
[14] Wei Yin, Chi Zhang, Hao Chen, Zhipeng Cai, Gang Yu, Kaixuan Wang, Xiaozhi Chen, and Chunhua Shen. Metric3d: Towards zero-shot metric 3d prediction from single image. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9043 9053, 2023.
[15] Weihao Yuan, Xiaodong Gu, Zuozhuo Dai, Siyu Zhu, and Ping Tan. Neural window fully-connected crís for monocular depth estimation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 3916-3925, 2022.
[16] Shariq Farooq Bhat, Ibraheem Alhashim, and Peter Wonka. Adabins: Depth estimation using adaptive bins. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4009-4018, 2021.
[17] Jin Han Lee, Myung-Ky Han, Dong Wook Ko, and 1l Hong Suh. From big to small: Multi-scale local planar guidance for monocular depth estimation. arXiv preprint arXiv:1907.10326 2019.
11

---

[18] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaekdin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07192 2023.
[19] Fabien Baradel, Matthieu Armando, Salma Galaaoui, Romain Brégier. Philippe Weinzaepfel. Grégory Rogez, and Thomas Lucas. Multi-hmr: Multi perti- whole-body human mesh recovery in single shot. arXiv preprint 44X4422402.1664. 2024.
[20] Shubham Goel, Georgios Pavlakos, Jathushan Rajasegaran, Angjoo Kanazawa, and Jitendra Malik. Hu- mans in 4d: Reconstructing and tracking humans with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 17794, 2023.
[21] Kevin Lin, Lijuan Wang, and Zicheng Liu. End-to-end human pose and mesh reconstruction with transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1954 1963,1993,2200 2021.
[22] Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael Black. Smpl: A skinned multi person lincar model. In Seminal Graphics Papers: Pushing the Boundaries, Volume 2, pages 851 16-8666 2023.
[23] Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani. Timo Bolkart, Ahmed AA Osman. Dimitrios Tzionas, and Michacl J Black. Expressive body capture: 3d hands, face, and body from single image. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10975-10985 10985, 2019.
[24] Jinyoung Jun, Jac-Han Lee, Chul Lee, and Chang-Su Su Kim. Depth map decomposition for monocular depth estimation. In European Conference on Computer Vision, pages 18-34, Springer, 2022.
[25] Shariq Farooq Bhat. Ibraheem Alhashim, and Peter Wonka. Localbins: Improving depth estimation by lcarning local distributions. In European Conference Computer Vision, pages 480-496. Springer, 2022.
[26] Zhenyu Li, Xuyang Wang. Xianming Liu, and Junjun Jiang. Binsformer: Revisiting adaptive bins for monocular depth estimation. IEEE Transactions on Image Processing, 2024.
[27] Alican Mertan, Damien Jade Duff, and Gozde Unal. Single image depth estimation: An overview. Digital Signal Processing, 123: 10::4,441, 2022.
[28] Jae-Han Lee and Chang-Su Kim. Monocular depth estimation using relative depth maps. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9729-9738. 2019.
[29] René Ranftl, Alexey Bochkovskiy. and Vladlen Koltun. Vision transformers for dense prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12188812888, 2021.
[30] Huan Fu, Mingming Gong, Chaohui Wang, Kayhan Batmanghelich, and Dacheng Tao. Deep ordinal regression network for monocular depth estimation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pages 2002-200 2002-0011.20 2018
[31] Iro Laina, Christian Rupprecht, Vasileios Belagiannis, Federico Tombari, and Nassir Navab. Deeper depth prediction with fully convolutional residual networks. In 2016 Fourth international conference on 3D vision (3DV), pages 239 248. IEEE, 2016.
[32] Fayao Liu, Chunhua Shen, Guosheng Lin, and lan Reid. Learning depth from single monocular images using deep convolutional neural fields. IEEE transactions pattern analysis and machine intelligence, 38 (10):2024 2039, 2015,
[33] Vaishakh Patil, Christos Sakaridis. Alexander Liniger, and Lue Van Gool. P3depth: Monocular depth estimation with piecewise planarity prior. In Proceedings of the IEEESCVF Conference Computer Vision and Pattern Recognition, pages 1610-1621, 2022.
[34] Guanglei Yang, Hao Tang, Mingli Ding, Nicu Sebe, and Elisa Ricci. Transforerrbased attention networks for continuous pixel-wise prediction. In Proceedings of the IEEE/CVF International Conference on Computer vision, pages 16269-162777 2021.
[35] Ainaz Eftekhar, Alexander Sax, Jitendra Malik, and Amir Zamir. Omnidata: A scalable pipeline for making multi-task mid-level vision datasets from 3d scans. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10786-10796, 2021.
[36] Wei Yin, Jianming Zhang, Oliver Wang, Simon Niklaus, Long Mai, Simon Chen, and Chunhua Shen. Learning to recover 3d scene shape from single image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 204-213, 2021.
12

---

[37] Jose M Facil, Benjamin Ummenhofer, Huizhong Zhou. Luis Montesano. Thomas Brox, and Javier Civera. Cam-convs: Camera-aware multi-scale convolutions for single-view depth. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 18826-18835, 2019.
[38] Manuel López Antequera. Pau Gargallo, Markus Hofinger, Samuel Rota Buld, Yubin Kuang, and Peter Kontschieder. Mapillary planet-scale depth dataset. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23 28.28, 2020, Proceedings, Part II 16, pages 589-604 Springer, 2020.
[39] Renrui Zhang, Ziyao Zeng, Ziyu Guo, and Yafeng Li, Can language understand depth? In Proceedings of the 30th ACM International Conference on Multimedia, pages 6868-6874, 2022.
[40] Xueting Hu, Ce Zhang, Yi Zhang, Bowen Hai, Ke Yu, and Zhihai He. Learning to adapt clip for few-shot monocular depth estimation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 5594-5603, 2024.
[41] Ziyao Zeng, Daniel Wang. Fengyu Yang. Hyoungseob Park, Stefano Soatto, Dong Lao, and Alex Wong. Wordepth: Variational language prior for monocular depth estimation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 9708 9919,,99999999999 2024.
[42] Federica Bogo, Angjoo Kanazawa, Christoph Lassner, Peter Gehler, Javier Romero, and Michael J Black. Keep it smpl: Automatic estimation of 3d human pose and shape from a single image. In Computer Vision ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part V 14, pages 561 578. Springer, 2016.
[43] Garvita Tiwari, Dimitrije Antić, Jan Eric Lenssen, Nikolaos Sarafianos, Tony Tung, and Gerard Pons-Moll. Pose-ndf: Modeling human pose manifolds with neural distance fields. In European Conference on Computer Vision, pages 572-589. Springer, 2022.
[44] Angjoo Kanazawa, Michael Black, David W Jacobs, and Jitendra Malik. End-to-end recovery of human shape and pose. In Proceedingy of the IEEE conference on Computer Vision and Puttern Recognition, pages 7122-7131, 2018.
[45] Nikos Kolotouros, Georgios Pavlakos, Michael J Black, and Kostas Daniilidis. Learning to reconstruct 3d human pose and shape via model-fitting in the loop. In Proceedings of the IEEE/CVF international conference compuder vision, pages
2019.
[46] Hongwen Zhang, Yating Tian, Xinchi Zhou, Wanli Ouyang. Yebin Liu, Limin Wang, and Zhenan Sun. Pymaf: 3d human pose and shape regression with pyramidal mesh alignment feedback loop. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 1 11446-1 1456,, 2021.
[47] Muhammed Kocabas, Chun-Hao P Huang, Otmar Hilliges, and Michael Black. Pare: Part attention regressor for 3d human body estimation. In Proceedings of the IEFF/CVF international conference on computer vision, pages 11 1272111137
[48] Riza Alp Guler and Iasonas Kokkinos. Holopose: Holistic 3d human reconstruction in-the-wild. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10884-10894, 2019.
[49] Yizhou Zhao, Tuanfeng Yang Wang, Bhiksha Raj. Min Xu, Jimei Yang, and Chun-Hao Paul Huang. Synergistic global-spac camera and human reconstruction from videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2216-1226 2024.
[50] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE International Conference on Computer Vision, pages 2961-2969, 2017.
[51] David Bigen, Christian Puhrsch, and Rob Fergus. Depth map prediction from a single image using a multi-scale deep network. Advances in neural information processing systems, 27, 2014.
[52] Nathan Silherman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and support inference from rghd images. In Computer Vision-ECCC 2012: 12th European Conference on Computer Vision, Florence, Italy, October 7-13, 2012, Proceedings, Part V /2, pages 746-760. Springer, 2012.
[53] Tobias Koch, Lukas Lichel, Marco Körner, and Friedrich Fraundorfer. Comparison of monocular depth estimation methods using geometrically relevant metries on the ibims-1 dataset. Computer Vision and Image Understanding. 191:102877, 2020.
[54] Thomas Schops, Johannes Schonberger, Silvano Galliani, Torsten Sattler, Konrad Schindler, Mare Pollefeys, and Andreas Geiger. A multi-vew stereo benchmark with high- high-resolution images and multi- camera videos. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3260-3269, 2017.
13

---

[55] Andreas Geiger, Philip Leaz. and Raquel Urtasun. Are ready for autonomous driving? the kitti vision benchmark suite. In 2012 IEEE conference computer vision and pattern recognition, pages 3354-3361.
IEEE, 2012.
[56] Ravi Garg. Vijay Kumar Bg, Gustavo Cameeio, and Ian Reid. Unsupervised for single view depth estimation: Geometry to the rescue. Computer Vision ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings. Part VIII 14, pages 740-756. Springer. 2016.
[57] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pages 10684-10699 2022.
[58] Shuai Li, Jiaying Shi, Wenfeng Song, Aimin Hao, and Hong Qin. Few-shot learning for monocular depth estimation based local object relationship. In 2019 JEEE 31 International Conference Tools with Artificial Intelligence (ICTAI), pages 1221 1228. IEEE, 2019.
[59] Renrui Zhang, Ziyao Zeng, Ziyu Guo, and Yafeng Li. Can language understand depth? In Proceedings of the 30th ACM International Conference on Multimedia, pages 6868-6874, 2022.
[60] Igor Vasiljevic, Nick Kolkin, Shanyi Zhang, Ruotian Luo, Haochen Wang, Falcon Z Dai, Andrea F Danicle, Mohammadreza Mostajabi, Steven Basart, Matthew Walter, et Diode: A dense indoor outdoor depth dataset. arXiv preprint arXiv: /908,00463, 2019.
[61] Tobias Koch, Lukas Liebel, Friedrich Fraundorfer, and Marco Korner. Evaluation of enn-based single- image depth estimation methods. In Proceedings of the European Conference on Computer Vision (ECCV) Workshops, pages 0-0, 2018.
[62] Jathushan Rajasegaran, Georgios Pavlakos, Angjoo Kanazawa, and Jitendra Malik. Tracking people by predicting 3d appearance, location and pose. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2740-2749, 2022.
[63] Sai Kumar Dwivedi, Yu Sun, Priyanka Patel, Yao Feng, and Michael J Black. Tokenhmr: Advancing human mesh recovery with tokenized pose representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1323-1333, 2024.
[64] Mu Hu, Wei Yin. Chi Zhang, Zhipeng Cai, Xiaoxiao Long, Hao Chen, Kaixuan Wang, Gang Yu, Chunhua Shen, and Shaojie Shen. Metric3d v2: A versatile monocular geometric foundation model for zero-shot metric depth and surface normal estimation. arXiv preprint arXiv:2404. 15506, 2024.
14

---

A Experiments on Scene Dependency
A.1 Similarity with Training Samples
We provide more details of plotting MMDE 81 against the maximum cosine similarity between each test sample and training samples in Fig. 3 Specifically, we run ZepDepth [12 on each sample of its test sets and calculate their 61 metrics. We also calculate the cosine similarity of each test sample against each training sample with metric annotations with DINOv2 [8] and find the similarity value for the nearest neighbor. Since ZeeDepth first pre-trains its backbone for MRDE and then finetunes on NYUv2 [52] and KITTI (55] for MMDE, we only include the two datasets used for MMDE training during similarity calculation. For the scatter plot, we randomly take 10 different samples each time and plot their average 5j and maximum cosine similarity with training samples as one point on the figure. In this way, we plot 100 points for each testing dataset.
A.2 MRDE Performances of MMDE Models
Table Performance comparisons of our MMDE methods terms of MMDE and MRDE settings on the DIODE (Indoor) [60], iBims-1 [6]] and ETH3D [54] datasets. *-IN, K, NK): fine-tuned on NYUv2 [52], KITTI [55], or the union them. We re-evaluate all results with fair and consistent pipeline for metric completeness.
DIODE (Indoor)
iBims-I AbsRel
ETH3D AbsRel
Method
Setting
81
AbsRel
Slogg
81 t
Shog
51
Shog
ZoeDepth-NK
P2
38.8
33.0
13.3
61.0
18.7
8.98
33.5
47.3
14.0
Depth Anything-
29.7
32.7
12.5
71.3
15.0
7.58
25.2
38.7
10.2
Depth Anything-K
231
15.5
2.88
217
16.9
136
17.1
ZeroDepth [11
MMDE
43.2
30.0
13.2
74.6
16.4
10.6
31.2
32.6
13.4
Metric31) 14
26.8
14.4
34.2
UniDepth-C 100
62.8
23.8
11.5
81.1
14.8
8.30
43.3
35,5
10.3
UniDepth-V 10
79.8
18.1
10.4
23.4
35.7
6,87
27.2
43.1
8.93
ZoeDeptl-NK
12
91.6
12.2
11.9
97.3
5.61
7.74
94.9
8.15
9.85
Depth Anything-N
2
94.4
10.2
10.6
98.4
4.45
6.18
93.8
8.03
9,95
Depth Anything-K
12.0
97.7
ZeroDepth [1]
MRDE
91.8
12.1
12.1
94.8
6.51
9.25
87.6
11.6
12.1
UniDepth-C 10
94.2
10.2
10.7
97.6
4.68
7.24
96.6
6.96
8.76
UniDepth-
95.6
8.89
98.6
3.41
5.69
97.4
5.79
7.53
To use an MMDE model for MRDE, we align the predictions with ground truths by solving optimal scales and translations. As demonstrated in Tab. 6 under the MRDE setting. MMDE models all perform well on wn-the-wild data. In contrast, under the MMDE setting, they degrade to various degrees. These results show that, with fully supervised training, depth models are generally more generalizable under the MRDE setting than the MMDE setting. Since the MMDE task can be understood combination of MRDE and metric scale recovery, believe the latter can be more difficult to conduct in the wild. Either implicitly or explicitly, conventional MMDE models predict metric scales in discriminative manner, conditioned the input image. With the fully supervised training scheme, they may tend to rely on the relation between training scenes and the testing scene infer the metric scale. If the testing scene is unseen during training, can be difficult infer scene scale. The MRDE task is relatively easier since it can potentially utilize more local visual clues for lower-level predictions. While establishing global correspondence between training scenes and the testing scene might be challenging, the model can still identify local correspondences to predict relationships within different parts of scene.
B More Ablation Study and Analysis
Impact mask prompts. For generative painting, we randomly create mask prompts with heights h equal to the input image height H and widths w E Ja min(H, W). min(H, W)J, where a < 3 < are two hyperparameters.
Tab. we ablate choices them by painting N images with humans for each input. The results demonstrate that different ar, 3 values will affect the model performance, while to a small degree. By comparing among the last three rows, we see model performs better with smaller mask prompts within reasonable range. When mask prompts
15

---

Table Ablation study for mask sizes on the NYUv2 dataset.
Min Ratio a
Max Ratio 3
81 f
AbsRel
Shog
RMSE 4
0.1
0.7
67.1
23.3
15.4
0.812
0.2
0.8
66.8
21.9
15.2
0.792
0.3
0,9
66.0
23.5
15.0
0,834
0.2
0.6
66.0
22.4
14.7
0.827
0.3
0.7
67.5
22.9
14.9
0.806
0.4
0.8
54.3
27.3
16.9
1.031
constrained to larger sizes, as seen in the last row, model performance degrades. Moreover, when comparing rows (1st vs. 4th 2nd vs. 5th 3td vs. 6th with common centers (ax + 3)/2 while different ranges (B a),, observe larger ranges yields lower RMSE. This might be due mask sizes providing an opportunity to paint humans of more flexible sizes. As we can imagine, a close human will occupy larger areas in the painted image, and vice versa. This diversity leads to better numerical stabilities during solving the scale s and the translation & in Eq. TT. Therefore, we 0.2, 0.8 in our main experiments for optimal A bsRel results.
larger range of
Table 8: Ablation study for different generative paint- ing models on the NYUv2 dataset.
Table 9: Ablation study for different HMR models the NYUv2 dataset.
Model
&
Absltel
Shog
RMSE
Model
7
Absltel
Slag
RMSE +
Stable Diffusion
53
v1.5
74.0
16.8
11.5
0.642
HMAR 0
82.0
14.2
9.83
0.489
Stable Diffusion
XL
78.5
11.3
0.533
TokenHMR
63
80.4
14.9
9.55
0.495
Stable Diffusion
50
v2
83.2
13.7
9.78
0.487
HMR 2.0 20
83.2
13.7
9.78
0.487
Impact of generative painting models. In Tab. 8 we show the effect of using different generative painting models by generating N 32 images for each input. The results indicate that current generative painting models generally work well with MfH in MMDE. MfH combined with Stable Diffusion v2 produces the best outcomes, likely due to its superior ability to generate realistic paintings. possible that if a generation model can better capture the real-world 2D image distributions, it has better sense of scale, serving more effective source of metric scale priors. Hence, we anticipate further performance gain of MíH with more advanced generative painting models.
Impact HMR models. In Tab. we evaluate the impact of using different HMR models within MfH. Here we use N 32 painted images for each input. Our findings indicate that the performance of our approach remains stable regardless of the HMR model used, suggesting that humans serve as effective universal landmarks for deriving metric scales from images. Furthermore, current HMR models reliably contribute to extracting metric scales for MMDE within the MiH framework.
40.00
60,00
50,00
42.48
33.58
46.24
36.22
ZoeDepth NK
34.88
UniDepth-C
33.68
43.96
37.19
MfH (Ours)
47.52
35.80
50.74
27.91
AbaRa
30.00
25.19
22.94
22.57
24.26
24.01
24.02
20.00
10.00
0.00
low
level
high
short (close-up)
medium distance of shot
long (wide-view)
angle of view
Figure AbsRel (4) comparisons for different types of shots on the ETH3D dataset.
Impact of input shot types. To analyze the contribution of metric information from humans, we look into the MMDE results on ETH3D (54), which includes both indoor and outdoor scenes with diverse shot types. Specifically, we annotate ETH3D images with two shot-related attributes and plot the AbsRel comparisons in Fig. 9 They confirm that our MfH can robustly recover metric depths, as it consistently achieves low errors across various types of shots. We also identify that the metric information from humans helps the most for level-angle inputs. This is likely because MRDE models
16

---

tend to interpret similar semanties, such different parts of human body, as having similar depths. This interpretation aligns well with standing humans, which are typically generated in level-angle images. Moreover, we do not observe significant degradation with varying the distance of shots. This indicates that MfH can effectively handle general close-up and wide-view shots.
C Difference with Previous Methods
Discriminative vs. Generative Discriminative. Traditional MMDE models metric depth prediction in discriminative manner. That is, given an input image I, they model a conditional probability of metric depths D with neural network 0, i.e., Pe(D ( (). Further, the distribution of metric depths can be considered as a joint distribution of relative depths D and metric scales S. According to the chain rule, we have
Po Po( 1)= Po(D ( (s( 1) PO(
(9)
Since the relative depth scale-invariant, D is independent S,
Po(D" 1) 1z(D))1)) + Pa(D") Pa(D")l)) Pa(S)
(10)
10 2 parameterize the two terms po (D I). (S) jointly with single discriminative 8. In contrast, we consider introducing generative and discriminative priors in modeling P(S(Z). With the law of total probability, we have
Most prior arts 12 11 14
P(SIT) Σ
P(S) 7 T) P((7rain) T),
(11)
Irsint
where spaint is the random variable for the painted image. In this equation, P(Zpaint) I) can be captured by the generative painting model conditioned on the input, and P(S) reaint I) is estimated by HMR and our metric head, which are discriminative. The summation over /raint corresponds to our global optimization upon random generative painting. Our MíH can thus approximate P(S[Z) through Monte Carlo sampling, bridging the gap between MRDE, i.e., P(D [T), and MMDE, i.e., P(D" /).
Training vs. Pre-training + Fine-tuning vs. Test-time Adaptation Most traditional MMDE approaches [17 16 [5] IT [4 [0] follow a fully supervised training paradigm, for which we discuss can cause dependency to training scenes during test time. For them, expensive metric depth annotations diverse scenes are necessary for zero-shot abilities. Some recent works seek to reduce the cost of data labeling by introducing pretraining over relative depth [12] or unlabeled image data [2]. While this mitigates the data hunger to some degree, these works also require downstream fine-tuning with metric depth annotations for MMDE. We instead consider a test-time adaptation scenario having no access to metric annotations. Under this setting, our model is required predict the metric depth merely dependent on one input image.
D
More Qualitative Analysis
D.1
Case Study
Since the performance of MfH relies on the quality of the pseudo ground truths (D;), illustrate both successful and failed cases generated during this process in Fig. TO Typical failure cases, shown in the first three rows, include 1) the generative painting model producing non- non-human objects with human features (1st row), 2) the generative painting model incorrectly capturing the scene scale and producing out-of-proportion humans (2 row), and 3) the human mesh recovery model predicting meshes that penetrate each other (3 row). In contrast, success cases in the last three rows demonstrate accurate space and scale relationships between human figures and scenes, leading to effective pseudo ground truths. These visualizations partially explain why more painted images help to improve the MMDE performance. The reason that a larger number of painted images dilutes the influence of outliers, facilitating more robust optimization. We hence speculate prompt engineering, as well as better sampling and filtering strategies in human painting, can further improve the performance of MIH. We leave the exploration of these to future work.
17

---

Fail Caeses
Succeess Caeres
f14)
(a) Original Inputs
(b) Painted Images
(c) Human Meshes
(d) Human Masks
(e) Pseudo GT Depths
Figure 10: Success cases and failure cases of MIH during the process of pseudo ground truth
D generation. The first three rows show failure cases, while the last three rows show success ones.
D.2 User Study
We further conduct user study for in-the- wild inputs where ground truths are unavailable. This study includes MMDE results from DepthAnything- (N, K} (2), Metric3D-vl [14. Metric3D-v2 (64]. UniDeppth-E (C, V] I0) ZeroDepth [T. ZoeDepth-NK 12], and our proposed MfH. As shown in Fig. participants are presented with input images, corresponding MMDE results from the listed methods, along with a color bar mapping metric depth values to colors. They were instructed to select the most reasonable MMDE result for each sample, with the following guidance:
Please choose the most reasonable metric depth estimation for each question given the input image and the meter bar. Different colors represent different metric depth values. Note that depth values farther than the maximum value nearer than the minimum value on the meter bar are truncated.
To analyze the results, we take each input image as a separate sample and add one count to the corresponding method if its MMDE result is selected the most reasonable MMDE given the corresponding input image and the meter bar. We then calculate the selection rate for each method, representing the proportion of selected results for this method out of the total number of selections. We received 50 responses with the results Tab. I0 According to the meter bar attached, we roughly break down the selection rate for short, medium, and long depth ranges.
These results indicate that our MfH method achieves the highest selection rate across all depth ranges, demonstrating its robustness. Metric3D-v2 also performs well, securing the second-highed selection
18

---

Table 10: Selection rate as the most reasonable MMDE result across different ranges. The ranges indicate the maximum value of the meter bar related to each input sample.
Range
Max Depch
DA-K
DA-N
M3Dxv1
[4]
M3D-22 64
UD-C [0
UD-V [10
11]
20-NK [2]
MIH (Ours)
Shoet-range
Madium-nnge
2,09
Long-range
80m
13.59
2.0%
11.0%
21.0%
6.0%
2.5%
0.5%
3,5%
40.0%
Overall
10m s0m
12.4%
6.4%
3.0%
18.4%
6.0%
5.8%
0.8%
4.1%
42.6%
Ixppt
MiH
Figure 11: In-the-wild qualitative results for DSLR camera smartphone captured images.
rate. In contrast, other methods show variability in performance across different depth ranges. For example, DepthAnything--N has a high selection rate for short-range inputs but is not selected for inputs with larger maximum depths. This is probably due to its scene dependency. Since it is trained on NYUv2, an indoor scene dataset, MMDE ability focuses more on short-range scenes.
D.3 Zero-shot Qualitative Results
We demonstrate more metric depth predictions and pixel-wise AbsRel in Figs. 12 to 16
19

---

0.16
0.15
0.32
0.10
0.05
0.63
0.00
0.00
0.15
0.32
0.10
0.c8
0.00
0.20
0.32
0.10
0.00
0.00
0.20
0.18
0.30
0.00
0.00
0.20
0.38
0.15
0.30
0.68
0.05
0.18
6.15
0.32
0.08
6.55
0.03
(a) Input & GT
(b) ZoeDepth-NK (c) Zexooppth
(d) UniDepth-C
(0) MIH (Ours)
Meters AbsRel
Figure 12: Zero-shot qualitative results NYU Depth v2.
20

---

0.07
0.01
0.00
0.03
0.02
0.c1
0.00
0.05
0.04
0.C3
0.02
0.C1
0.c0
0.05
0.04
0.68
0.02
0.11
0.00
0.65
0.04
(a) Input & GT
(b) ZoeDepth-NK (c) ZeroDepth
(d) UniDepth-C
(c) MiH (Ours)
Meters AbsRe
Figure 13: Zero-shot qualitative results KITTI.
21

---

0.15
0.10
0.05
0.00
0.25
0.20
0.15
0.30
0.05
0.00
0.30
0.25
0.20
0.35
0.30
0.65
0.00
0.10
0.25
0.20
0.15
0.10
0.05
0.00
0.30
0.25
0:20
0.35
(a) Input & GT
(b) ZoeDepth-NK (c) ZexoDepth
(d) UniDepth-C
(0) MiH (Ours)
Meters
AbsRe
Figure 14: Zero-shot qualitative results on Diode (Indoor).
22

---

0.20
0.00
Boue io أد
س سننم
مااما
0.30
0.00
i عياد
0.30
0.00
0.50
0.40
0.30
0.20
0.30
0.00
0.50
0.40
0.30
0.20
0.50
0.00
0.50
0.40
0.20
(a) Input & GT
(b) ZoeDepth-NK (c) Zeroopth
(d) UniDepth-C
(c) MIH (Ours)
Meters
AbsRel
Figure 15: Zero-shot qualitative results on iBims-1.
23

---

0.30
0.05
0.00
0.10
0.25
0.20
0.15
0.30
0.05
0.00
0.30
0.25
0.20
0.35
0.30
0.65
0.00
0.30
0.25
0.20
0.15
0.10
0.05
0.00
0.30
0.25
0:20
0.35
(a) Input & GT
(b) ZoeDepth-NK (c) Zexooppth
(d) UniDepth-C
(c) MIH (Ours)
Meters
AbsRe
Figure 16: Zero-shot qualitative results on ETH3D.

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification:
We discuss the paper's contributions and scope in Sec.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations in Sec. 5
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
25

---

Justification:
There are no theoretical results in our paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide detailed experimental settings for our main experiment in Sec. and details of our pilot experiments in Appendix A
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
26

---

Answer: [Yes]
Justification:
Our code is available at
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
Cooeeeoodemsssseeeeee / casssiinnccccyyc for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: We provide detailed experimental settings for our main experiment in Sec. 4.1 and details of our pilot experiments in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
We plot the results with error bars in Fig. 5
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
27

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide information on the computer resources in Sec. 4.1
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics httppp////errrr.....oo/////////a
Answer: [Yes]
Justification:
The paper conforms to the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss both positive societal impacts and negative societal impacts in Sec.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
28

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cite all used public datasets and pre-trained models in Sec. 4.1
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
29

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not introduce new assets in the paper.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
30


Reviews:
[{"review_id": "bm6mZxd0Gl", "reviewer": "Reviewer_4QJ3", "summary": "This paper presents target zero-shot monocular metric depth estimation in the wild. They propose to use humans as landmarks to achieve metric scale and without any other information, such as focal length used in metric3D and zerodepth.  The key ideas are creative and well-motivated, addressing an important challenge in the field. While there are some limitations and areas for further exploration, the method shows clear improvements over existing approaches and opens up interesting directions for future work. However, it lacks more in-the-wild evaluation. From the comparisons, the metric accuracy is not convincing. I cannot know if the model can recover metrics in the wild.", "questions": "1.\tYou did not include comparisons with the state-of-the-art methods such as Metric3D and UniDepth in Table 1. These works include test results on the NYUv2 and KITTI datasets.\n2.\tYou should include comparative experiments with other generative painting and human mesh recovery methods because the method relies heavily on the performance of the generative painting and human mesh recovery models, which could introduce errors or biases.\n3.\tHave you encountered cases where SD v2 and HMR2.0 failed to generate satisfactory human body and human mesh recovery? How did you handle situations where they couldn't produce good results? \n4.\tProvide more details on the generative painting model and human mesh recovery model used and how it affects the results.\n5.\tI noticed that in Figure 6, the human body in the Recovered Human Meshes and Painted Images point clouds do not overlap well. Please explain this phenomenon.\n6.\tMore discussion and analysis of failure cases of the approach.", "limitations": "The paper lacks enough discussion of limitations. More details are provided in Questions.", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "Novelty: The paper presents an innovative approach to zero-shot monocular metric depth estimation by leveraging generative painting models and human mesh recovery. This is a creative solution to the challenge of generalizing metric depth estimation to unseen scenes. \n\nProblem Formulation: The authors clearly articulate the limitations of current MMDE approaches, particularly their scene dependency and data hunger. The motivation for their method is well-explained and supported by empirical evidence (Fig. 2 and 3). \n\nMethod: The proposed Metric from Human (MfH) framework is well-designed and clearly explained. The use of humans as metric landmarks and the generate-and-estimate pipeline are interesting ideas. \n\nPotential Impact: If successful, this approach could significantly advance the field of monocular metric depth estimation, enabling better generalization to unseen scenes without requiring large amounts of metric-annotated training data.", "weaknesses": "Lack of Detailed Results: The paper does not present any quantitative results or comparisons with existing methods in Table 1, such as Metric3D or UniDepth. This makes it difficult to assess the actual performance and advantages of the proposed approach. \n\nLimited Discussion of Limitations: While the method is promising, there's little discussion of its potential limitations or failure cases. The method relies heavily on the performance of the generative painting and human mesh recovery models, which could introduce errors or biases.  For instance, how does it perform when the generative painting model produces unrealistic or poorly scaled humans?\n\nComputational Complexity: Given that it involves generative painting and human mesh recovery at test time, it may be significantly slower than existing approaches. The computational cost and inference time of the test-time adaptation process are not addressed", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the time and effort to review our paper. Below please find our specific answers to the questions.\n\n1. **Comparisons with state-of-the-art methods.**\n    \n    We will update Table 1 in our revised manuscript as Table R1 in the attached PDF. Our previous Table 1 aims to provide a fair comparison based on the availability of metric depth annotations (the number of shots). So we focus on zero/one/few-shot methods, excluding many-shot methods like Metric3D and UniDepth.\n    \n2. **Comparative experiments with different generative painting methods.**\n    \n    In Table A, we ablate the effect of using different generative painting models in MfH. The results indicate that current generative painting models generally work well with MfH in MMDE. MfH combined with SD v2 produces the best outcomes, likely due to its superior ability to generate realistic paintings. It is possible that if a generation model can better capture the real-world 2D image distributions, it has a better sense of scale, serving as a more effective source of metric scale priors. Hence, we anticipate further performance gain of MfH with more advanced generative painting models.\n    \n    | Model | $\\delta_1$ $\\uparrow$ | AbsRel $\\downarrow$ | SI$_{\\log}$ $\\downarrow$ | RMSE $\\downarrow$ |\n    | --- | --- | --- | --- | --- |\n    | SD v1.5 | 74.0 | 16.8 | 11.5 | 0.642 |\n    | SD-XL | 78.5 | 15.9 | 11.3 | 0.533 |\n    | SD v2 | 83.2 | 13.7 | 9.78 | 0.487 |\n    \n    Table A. Comparative experiments with different generative painting models on NYUv2.\n    \n3. **Comparative experiments with different HMR methods.**\n    \n    In Table B, we examine the effect of different HMR models in MfH. We find that our approach is not sensitive to such changes. This suggests that humans can serve as relatively universal landmarks for deriving metric scales from images. Also, current HMR models can robustly help extract metric scales for MMDE with our MfH framework.\n    \n    | Model | $\\delta_1$ $\\uparrow$ | AbsRel $\\downarrow$ | SI$_{\\log}$ $\\downarrow$ | RMSE $\\downarrow$ |\n    | --- | --- | --- | --- | --- |\n    | HMAR [R1] | 82.0 | 14.2 | 9.83 | 0.489 |\n    | TokenHMR [R2] | 80.4 | 14.9 | 9.55 | 0.495 |\n    | HMR 2.0 [20] | 83.2 | 13.7 | 9.78 | 0.487 |\n    \n    Table B. Comparative experiments with different HMR models on NYUv2.\n    \n    [R1] Tracking People by Predicting 3D Appearance, Location & Pose, CVPR 2022\n    \n    [R2] TokenHMR: Hybrid Analytical-Neural Inverse Kinematics for Whole-body Mesh Recovery\n    \n4. **Discussion and analysis of failure cases.**\n    \n    We show three typical failure cases in Figure 8 of our appendix in the main PDF. They include 1) the generative painting model producing non-human objects with human-like appearances, 2) the generative painting model incorrectly capturing the scene scale and producing out-of-proportion humans, and 3) the HMR model predicting meshes that penetrate each other. Since the generative painting model can paint plausible humans in most cases, a few failures will not significantly impact the overall result. Our random generate-and-estimate process with sufficient painted images makes MfH robust to outliers. Further, we speculate prompt engineering, as well as better sampling and filtering strategies in human painting, can improve the performance of MfH.\n    \n5. **Discussion of limitations.**\n    \n    We discuss the limitations of MfH in Section 5, pointing out two main assumptions MfH based on:\n    \n    1. We assume humans can exist in the scene so that generative painting is possible to paint humans on the input image. While this holds for most usages of MMDE, it might not be ideal for some cases, e.g., close-up scenes. To this end, one future direction is incorporating objects other than humans into the generate-and-estimate pipeline as metric landmarks.\n    2. We assume the MRDE predictions align with true metric depths up to affine. Since the MRDE predictions can contain non-linear noises, a simple linear metric head as in MfH might not be optimal. Exploring alternative parameterizations of the metric head remains an open question.\n6. **Computational complexity.**\n    \n    We acknowledge that MfH is not currently efficient. The runtime shown in Figure 5 is based on a sequential generate-and-estimate process, where painted images are processed one after another. In Table C, we provide a breakdown of runtime with 32 images to paint when paralleled, revealing that the majority of the time is consumed by generative painting with diffusion models. Given the rapid advancements in diffusion sampling [R3, R4], we anticipate further improvements in MfH’s inference speed in the near future.\n    \n    | HMR | Generative Painting | MRDE | Optimization | Total |\n    | --- | --- | --- | --- | --- |\n    | 2.4s | 5.5s | 0.1s | 0.3s | 8.3s |\n    \n    Table C. Runtime breakdown for an input image.\n    \n    [R3] Consistency Models, ICML 2023\n    \n    [R4] One-step Diffusion with Distribution Matching Distillation, CVPR 2024\n    \n7. **Human bodies not completely overlapping with human point clouds.**\n    \n    Since we only optimize a scale and a translation to convert MRDE to MMDE, each point cloud is stretched with the same set of scale and translation. Hence, we do not expect all human bodies to overlap perfectly with their corresponding point clouds but try to seek a “mode” of MRDE-to-MMDE transformation. This simplistic parameterization also provides a regularization against outliers during the generate-and-estimate process.\n    \n8. **In-the-wild evaluation.**\n    \n    We present qualitative results for DSLR and smartphone captured images in Figure R2 of the attached PDF. These results demonstrate our MfH can effectively handle in-the-wild inputs. Also, we observe fully supervised MMDE methods like UniDepth often provide bounded metric depths, inheriting from the limited range of sensors used in their training ground truths. In contrast, our MfH can provide more flexible results.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ug7brfWaRF", "reviewer": "Reviewer_bPK1", "summary": "This paper enables monocular depth estimation to output metric-scale depth maps from only single images. To do this, the key idea of this paper is to leverage painting human 3D models into the input images in test-time adaptation, whose motivation is that human painter depicts subjects in consideration of scene configurations. With the painted human, the authors use it as landmarks to account for scenes in metric scale. After making an initial metric scale scene depths using the painted human, the proposed framework propagates the metric scale cues onto whole scenes using optimizations. The proposed method shows impressive performance on the zero/few-shot depth prediction over state-of-the-art methods.", "questions": "I have two requests as below:\n\n1. Can you show me several results on images taken by the authors using DSLR and smartphone. To demonstrate the generality of this work, the results will be very helpful.\n\n2. Following the first request, it would be interesting for the authors to do experiments on a case with an existence of radial distortions in images. Nowadays, commercial cameras, especially smartphone cameras, is not based on pinhole camera models. That means the output from Metric Head is sometimes inaccurate according to the camera used. In this consideration, the authors need to discuss the practical issue based on real-world experiments.", "limitations": "This paper has well described their limitations. During reading this paper, I though that these two limitations, and knew that the authors think about them.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The most strength point of this work is to propose a new paradigm for monocular depth estimation with metric scale. In real-world scenarios, the roughly estimated metric scale depth is enough unless precise accuracy is required. Since this work is not targeting depth estimation with precise accuracy, there is no technical issue. \n\nIn addition, the idea using both human painting and its observation is very interesting. In monocular depth estimation, to obtain metric scale, finding reasonable metric scale cue in a scene is important and not easy. This work successfully utilizes it and shows the robust performance over relevant works. \n\nLastly, this paper is well-written and easy to understand for readers. I hope the authors to release their source codes and pre-trained weights for the same field researchers.", "weaknesses": "I do not have any weakness of this paper. Please check the Questions because I have some requests for better technical descriptions.\n\nMerely, the authors fix the references. For example, do not refer paper as Arxiv like [41] which is published in CVPR. Plus, please check the details of reference papers like [60] (0 to 0 pages).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the time and effort to review our paper. Below please find our specific answers to the questions.\n\n1. **Results from DSLR and smartphone.**\n    \n    We demonstrate qualitative results for DSLR and smartphone captured images in Figure R2 of the attached PDF. Depth predictions are truncated at different maximum values and displayed in color maps. These results show that our MfH can handle in-the-wild MMDE well, even for inputs with distortions, e.g., the first and last rows of the right column. Also, we observe fully supervised MMDE methods like UniDepth often provide bounded metric depths, inheriting from the limited range of sensors used in their training ground truths. In contrast, our MfH can provide more flexible results.\n    \n2. **Practical issue when the camera deviates from pinhole.**\n    \n    We do observe distortions in some in-the-wild images, where using a linear metric head might not be ideal since some pixels might appear closer than the others. For general slight distortions, we expect the MRDE model, as one component of MfH, to handle these, as it leverages abundant training data which might contain distortions. To further compensate for stronger distortions, we can use a radius-related metric head to transform relative depths to metric depths, formally, the scale $s = s(r)$ and the translation $t = t(r)$. Then we believe our MfH can still work since our generate-and-estimate strategy with random painting allows local optimization of scales and translations.\n    \n3. **Reference errors.**\n    \n    We appreciate your suggestion and have checked and fixed the reference errors accordingly in our revised manuscript.\n    \n4. **Source code and pre-trained weights.**\n    \n    We appreciate your suggestion and will release our source code and pre-trained weights of support models upon acceptance.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NWKqEiOusx", "reviewer": "Reviewer_2dzs", "summary": "This paper introduces Metric-from-Human (MfH), a method to infer metric depths from images without needing metric depth annotations. Using humans as landmarks, MfH extracts scene-independent metric scale priors from generative painting models, overcoming the challenge of scene dependency in Monocular Metric Depth Estimation (MMDE). They propose a test-time adaptation framework that bridges Monocular Relative Depth Estimation (MRDE) to MMDE via a generate-and-estimate pipeline. Experiments show MfH's superior performance and generalization ability in zero-shot MMDE. The paper also addresses limitations, broader impacts, ethical considerations, and provides experimental settings and statistical significance of the results.", "questions": "Questions are included in the weakness.", "limitations": "As mentioned in the weaknesses, the overall metric performance is suboptimal. The method appears to be highly dependent on the performance of human recovery models and generative painting models. It is likely that in certain difficult scenes, the performance will not be maintained effectively.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper provides a thorough analysis of the advantages and disadvantages of recently researched MMDE and MRDE models, highlighting their differences. It introduces a novel method for obtaining metric-depth in a scalable manner.\n\nThe innovative use of generative painting and Human Mesh Recovery (HMR) techniques to leverage the strong prior of human figures is a significant advantage, which has potentials to reduce the reliance on expensive metric-depth annotations.\n\nFrom an architectural perspective, the protocol for incorporating metrics into MRDE appears reasonable. Additionally, the paper reasonably discusses the limitations and broader impacts of the research.", "weaknesses": "The experimental results presented seem to be quite poor. For example, [1] also achieves zero-shot performance on NYU or KITTI datasets, showing significantly better results than this paper. Of course, the proposed method is focused on converting MRDE to metric depth without using any metric annotations, which is a disadvantage. Therefore, I am curious whether \"Metric from Human\" method would also benefit MMDE models like [1].\n\n\nSimilarly, the metric performance seems less than ideal, potentially due to the complexity of the scenes or inaccuracies in the human prior. Therefore, a detailed and fine-grained analysis is needed to determine the accuracy of the metric information provided by humans.\n\n\n##### [1] UniDepth: Universal Monocular Metric Depth Estimation", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the time and effort to review our paper. Below please find our specific answers to the questions.\n\n1. **Experimental results.**\n    \n    We acknowledge that currently our zero-shot MfH does not always outperform state-of-the-art many-shot methods. However, we would like to highlight our main contribution as pointing out the scene dependency problem of fully supervised many-shot MMDE and offering a potential solution. Our comparisons demonstrate MfH’s strong zero-shot MMDE performance across diverse scenarios, while fully supervised many-shot methods may degrade on unseen scenes. Additionally, we view MfH as a general framework for zero-shot MMDE using test-time adaptation, with performance that can be further enhanced with improved MRDE, HMR, and generative painting models.\n    \n2. **MfH can benefit MMDE models like UniDepth.**\n    \n    We replace the current MRDE model with UniDepth in MfH and present the MMDE results in Tables A, B, and C below. The results show that MfH with UniDepth achieves better results on iBims-1 and ETH3D, but worse on DIODE (Indoor), than original UniDepth. This confirms strong metric scale priors extracted by MfH can enhance MMDE models on unseen scenes. The degradation on DIODE (Indoor) is because it contains extreme close-up scenes, where painting humans can meet difficulties, as we acknowledge in Section 5. For such scenarios, we see potential in incorporating objects other than humans into the generate-and-estimate pipeline as metric landmarks.\n    \n    | Model | $\\delta_1$ $\\uparrow$ | AbsRel $\\downarrow$ | SI$_{\\log}$ $\\downarrow$ | RMSE $\\downarrow$ |\n    | --- | --- | --- | --- | --- |\n    | UniDepth-V | 79.8 | 18.1 | 10.4 | 0.760 |\n    | MfH (Ours) w/ Depth Anything | 42.2 | 34.5 | 13.2 | 1.363 |\n    | MfH (Ours) w/ UniDepth-V | 43.5 | 32.6 | 11.9 | 1.390 |\n    \n    Table A. Performance comparisons of our MfH with different MRDE methods and UniDepth on DIODE (Indoor).\n    \n    | Model | $\\delta_1$ $\\uparrow$ | AbsRel $\\downarrow$ | SI$_{\\log}$ $\\downarrow$ | RMSE $\\downarrow$ |\n    | --- | --- | --- | --- | --- |\n    | UniDepth-V | 23.4 | 35.7 | 6.87 | 1.063 |\n    | MfH (Ours) w/ Depth Anything | 67.7 | 23.3 | 9.73 | 0.738 |\n    | MfH (Ours) w/ UniDepth-V | 69.8 | 20.0 | 8.99 | 0.664 |\n    \n    Table B. Performance comparisons of our MfH with different MRDE methods and UniDepth on iBims-1.\n    \n    | Model | $\\delta_1$ $\\uparrow$ | AbsRel $\\downarrow$ | SI$_{\\log}$ $\\downarrow$ | RMSE $\\downarrow$ |\n    | --- | --- | --- | --- | --- |\n    | UniDepth-V | 27.2 | 43.1 | 8.93 | 1.950 |\n    | MfH (Ours) w/ Depth Anything | 47.1 | 24.0 | 8.16 | 1.366 |\n    | MfH (Ours) w/ UniDepth-V | 51.2 | 25.5 | 9.17 | 1.489 |\n    \n    Table C. Performance comparisons of our MfH with different MRDE methods and UniDepth on ETH3D.\n    \n3. **Detailed analysis of human scale priors with respect to scenes.**\n    \n    To analyze the contribution of metric information from humans, we look into the MMDE results on ETH3D, which includes both indoor and outdoor scenes with diverse type of shots. Specifically, we annotate ETH3D images with two shot-related attributes and plot the AbsRel comparisons in Figure R1 of the attached PDF. They confirm that MfH can robustly recover metric depths, as it consistently achieves low errors across various type of shots. We also identify that the metric information from humans helps the most for level-angle inputs. This is likely because MRDE models tend to interpret similar semantics, such as different parts of a human body, as having similar depths. This interpretation aligns well with standing humans, which are typically generated in level-angle images. Moreover, we do not observe significant degradation with varying the distance of shots. This indicates MfH can effectively handle general close-up and wide-view shots. We will include these analysis in our revised manuscript.\n    \n4. **Dependency on human recovery models and generative painting models.**\n    \n    We acknowledge MfH depends on HMR and generative painting models. To assess their impacts, we conduct ablation study in Tables D and E. Table D demonstrates that MfH combined with stronger generative painting models yields better performance, likely due to its superior ability to generate realistic paintings. It is possible that if a generation model can better capture the real-world 2D image distributions, it has a better sense of scale, serving as a more effective source of metric scale priors. Table E indicates that different HMR models do not significantly affect MfH’s performance, probably because current HMR models can robustly assist in extracting metric information for MMDE within our MfH framework. Overall, we anticipate better MMDE results with more advanced support models, which can be integrated in a plug-and-play manner.\n    \n    | Model | $\\delta_1$ $\\uparrow$ | AbsRel $\\downarrow$ | SI$_{\\log}$ $\\downarrow$ | RMSE $\\downarrow$ |\n    | --- | --- | --- | --- | --- |\n    | SD v1.5 | 74.0 | 16.8 | 11.5 | 0.642 |\n    | SD-XL | 78.5 | 15.9 | 11.3 | 0.533 |\n    | SD v2 | 83.2 | 13.7 | 9.78 | 0.487 |\n    \n    Table D. Ablation study for different generative painting models on NYUv2.\n    \n    | Model | $\\delta_1$ $\\uparrow$ | AbsRel $\\downarrow$ | SI$_{\\log}$ $\\downarrow$ | RMSE $\\downarrow$ |\n    | --- | --- | --- | --- | --- |\n    | HMAR [R1] | 82.0 | 14.2 | 9.83 | 0.489 |\n    | TokenHMR [R2] | 80.4 | 14.9 | 9.55 | 0.495 |\n    | HMR 2.0 [20] | 83.2 | 13.7 | 9.78 | 0.487 |\n    \n    Table E. Ablation study for different HMR models on NYUv2.\n    \n    [R1] Tracking People by Predicting 3D Appearance, Location & Pose, CVPR 2022\n    \n    [R2] TokenHMR: Advancing Human Mesh Recovery with a Tokenized Pose Representation, CVPR 2024", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MG5poIJdjT", "reviewer": "Reviewer_xSFg", "summary": "This work proposes a test time training technique to turn a monocular relative depth estimation model into a metric monocular one. The core insight of the work is to rely on the prior of a text-to-image model (Stable Diffusion v2) to generate humans in the scene for which we are interested in knowing metric depth. Given the prior of the model, the humans should be generated in a scale aware manner and can therefore be used as hints to recover the metric scale of the scene. Using off-the-shelf methods (HMR-2) it is then possible to fit a SMPL model to the images of the humans and recover estimated metric depth for the humans. This information is finally used to train a simple linear layer that will transform a relative depth predicted by an independent model into a metric one for that specific scene. This whole process is repeated for each test image (therefore is very very slow) but does not require any metric training data and achieve decent results.", "questions": "a. What technique is it used for inpainting the image? (this should be specified in the paper)\nb. Is Eq. 5 optimized only over the pixels with humans?", "limitations": "The paper discuss limitations of the current method", "rating": 3, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "+ Original idea. While it is quite know in the literature that objects provides clues to metric monocular depth models to recover the scale of a scene (and for this reason they can be easily fooled by optical illusions) I haven’t seen before a proposal to use generative models to paint these objects (e.g., humans) when they do not occur naturally in the scene.\n\n+ Clear motivation. I think Fig. 2 and Fig. 3 together with the introduction do a good job in explaining why the authors think that this problem is relevant and what are limitations of the existing solutions proposed in the literature.", "weaknesses": "1. Practicality. As reported in Fig. 5 the proposed pipeline requires up to 5 minutes per image, to ground the discussion a forward pass of the biggest depth anything model requires tens of milliseconds. Besides the latency, this work requires relying on three different models (a relative depth model, an image inpainting model and  a human mesh prediction model) to achieve metric depth prediction on a single image. I’m not convinced that these settings are realistic for any practical use case. I would suggest to the authors to explore the same idea but in an offline settings, i.e., as a way of generating pseudo metric ground truth for a dataset that does not have it. \n\n2. Inherently limited by the support models. The proposal basically shifts the problem of recovering the scale of the scene from the depth estimation model to the image generation and human mesh estimation models. In particular the inpainting model is the one that does the heavy lifting in this work since it is tasked, provided with an arbitrary mask, to generate plausible humans. If any of the two models fail in their task the pipeline has no way of recovering as highlighted also by the authors in the failure case section of the work. If this intuition is correct, an approach that, like Marigold, starts from a pre-trained Stable diffusion and fine-tunes it for metric depth seems to be more promising with respect to the proposed solution. This solution was not explored by the authors since they didn’t want to rely on any metric depth data.\n\n3. Works only under certain assumptions. Besides the assumption of having good behaving support model discussed in weakness (2), the proposed solution also assumes that:\n   1. The definition of the area of the image to inpaint (which is a heuristic in the current implementation) picks often an area big enough and where a human can be generated. This in turns completely excludes entire categories of scenes (e.g. close ups or extremely wide views)\n   2. The depth estimation models are not affected by the artifacts introduced in the scene by the inpainting methods (e.g., Fig. 6  column 5 and 6 where the inpainted images do not make much sense from a semantic point of view)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the time and effort to review our paper. Below please find our specific answers to the questions.\n\n1. **Practicality.**\n    \n    We acknowledge that MfH is not currently efficient. The runtime shown in Figure 5 is based on a sequential generate-and-estimate process, where painted images are processed one after another, illustrating a linear correlation between computational cost and the number of painted images. In Table A, we provide a breakdown of runtime with 32 images to paint when paralleled, revealing that the majority of the time is consumed by generative painting with diffusion models. Given the rapid advancements in diffusion sampling [R1, R2], we anticipate further improvements in MfH’s inference speed in the near future.\n    \n    | HMR | Generative Painting | MRDE | Optimization | Total |\n    | --- | --- | --- | --- | --- |\n    | 2.4s | 5.5s | 0.1s | 0.3s | 8.3s |\n    \n    Table A. Runtime breakdown for an input image.\n    \n    [R1] Consistency Models, ICML 2023\n    \n    [R2] One-step Diffusion with Distribution Matching Distillation, CVPR 2024\n    \n2. **Offline settings.**\n    \n    We appreciate your suggestion to explore offline settings, which is a direction we find promising. Both solutions have their advantages. The offline approach requires training with abundant in-the-wild images and pseudo ground truths. While being expensive to train and prone to scene dependency, it offers fast inference. In contrast, our MfH provides a more cost-effective solution that is training-free and directly benefits from advancements in support models.\n    \n3. **Inherently limited by the support models.**\n    \n    We agree that the performance of MfH is related to the support models. However, our experimental results demonstrate that MfH, using current HMR and generative painting models, can predict metric depths satisfactorily. We further conduct ablation study in Tables B and C to show the impacts of different generative painting models and HMR models. They further indicates the potential for improved MMDE results with more advanced support models.\n    \n    | Model | $\\delta_1$ $\\uparrow$ | AbsRel $\\downarrow$ | SI$_{\\log}$ $\\downarrow$ | RMSE $\\downarrow$ |\n    | --- | --- | --- | --- | --- |\n    | SD v1.5 | 74.0 | 16.8 | 11.5 | 0.642 |\n    | SD-XL | 78.5 | 15.9 | 11.3 | 0.533 |\n    | SD v2 | 83.2 | 13.7 | 9.78 | 0.487 |\n    \n    Table B. Ablation study for different generative painting models on NYUv2.\n    \n    | Model | $\\delta_1$ $\\uparrow$ | AbsRel $\\downarrow$ | SI$_{\\log}$ $\\downarrow$ | RMSE $\\downarrow$ |\n    | --- | --- | --- | --- | --- |\n    | HMAR [R3] | 82.0 | 14.2 | 9.83 | 0.489 |\n    | TokenHMR [R4] | 80.4 | 14.9 | 9.55 | 0.495 |\n    | HMR 2.0 [20] | 83.2 | 13.7 | 9.78 | 0.487 |\n    \n    Table C. Ablation study for different HMR models on NYUv2.\n    \n    [R3] Tracking People by Predicting 3D Appearance, Location & Pose, CVPR 2022\n    \n    [R4] TokenHMR: Advancing Human Mesh Recovery with a Tokenized Pose Representation, CVPR 2024\n    \n4. **Will failure cases lead to failure result?**\n    \n    Our random generate-and-estimate process provides tolerance for failures in each component. Since the generative painting model can paint plausible humans in most cases, a few failures will not significantly impact the overall result. A sufficiently large number of painted images dilutes the influence of these failures, providing reasonable predictions. However, the idea of fine-tuning pre-trained Stable Diffusion for metric depth estimation is interesting, especially without metric depth annotations. We are excited to explore this dedicated problem in the future.\n    \n5. **Works only under certain assumptions.**\n    \n    Thank you for pointing out the potential assumptions of our method. We address each concern below:\n    \n    1. To assess whether MfH performs well with close-up or wide-view shots, we examine the MMDE results on the ETH3D dataset, which includes both indoor and outdoor scenes with various shot types. We annotate ETH3D images by differentiating their shot distances and plot the AbsRel comparisons in Figure R1 of the attached PDF. The results indicate no significant performance degradation when handling close-up or wide-view inputs, demonstrating that MfH can effectively handle general close-up and wide-view shots. Similar conclusion can also be drawn from the in-the-wild qualitative results in Figure R2. However, as we acknowledged in Section 5, MfH may not perform well in extreme cases where humans cannot be present in the scene.\n    2. Since MfH only aligns the MRDE of unpainted areas ($\\mathbf{D}^\\text{rel}$ vs. $\\{\\hat{\\mathbf{D}}^\\text{rel}_n\\}$), and MMDE of human areas ($\\mathbf{D}^*_n$ vs. $\\{\\hat{\\mathbf{D}}^\\text{m}_n\\}$), the potential non-human artifacts are not taken into account and will not affect the optimization process. Although artifacts can be semantic meaningless, we do not observe them significantly impacting the aligned depths of other contents, as evidenced by the point clouds in Figure 6 column 5 and 6.\n6. **Image inpainting technique.**\n    \n    We adopt Stable Diffusion v2 for generative painting, which is based on Conditional Latent Diffusion [56]. Starting from a text-conditioned diffusion model checkpoint, the denoising UNet is finetuned with additional input channels for VAE-encoded masked image to result in a diffusion-based generative painting model. We will clarify this in our revised manuscript.\n    \n7. **Optimization range of Eq. 5.**\n    \n    Your understanding is correct. Eq. 5 is optimized over the pixels of humans.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank all reviewers for their insightful feedback. They acknowledge the results as showing clear improvements (4QJ3), impressive (bPK1), superior (2dzs), and decent (xSFg). Reviewer 4QJ3 further finds our key idea creative and well-motivated while reviewer bPK1 sees our method establishing a new paradigm.

We include Table R1 and Figures R1 and R2 in the PDF file attached, showing more complete comparisons with state-of-the-art models, performance comparisons with respect to different types of shots, and more in-the-wild qualitative results respectively. Below we separately address concerns raised in the reviews. We hope our responses could clarify your confusion, and are more than happy to provide further explanations if needed.

Meta-review: The paper explores interesting observations that generative models can usually hallucinate humans suitable for a context, both indoor and outdoor, and we have strong structural priors for the shapes of human bodies. Those two observations can potentially lead to new ways of monocular metric depth estimation. This paper explores the insights by utilizing the external generative models and human 3D pose models. The results are quite promising. Although Reviewer xSFg clearly pointed out valid concerns of using the proposed approach in practical scenarios, the AC believes that this exploration can be interesting for the machine learning community and potentially inspire future works to generate practical methods with the ideas.
Decision: Accept (poster)