RSA: Resolving Scale Ambiguities in Monocular Depth Estimators through Language Descriptions
Authors: Ziyao Zeng
Yangchao Wu
Hyoungseob Park
Daniel Wang
Fengyu Yang
Stefano Soatto
Dong Lao
Byung-Woo Hong
Alex Wong
Venue: NeurIPS 2024 poster (2024)
Keywords: Monocular Depth Estimation
Vision-Language Model
Multimodal Learning
Primary area: machine_vision
TL;DR: We use language descriptions to transform relative depth to metric depth.

Abstract: We propose a method for metric-scale monocular depth estimation. Inferring depth from a single image is an ill-posed problem due to the loss of scale from perspective projection during the image formation process. Any scale chosen is a bias, typically stemming from training on a dataset; hence, existing works have instead opted to use relative (normalized, inverse) depth. Our goal is to recover metric-scaled depth maps through a linear transformation. The crux of our method lies in the observation that certain objects (e.g., cars, trees, street signs) are typically found or associated with certain types of scenes (e.g., outdoor). We explore whether language descriptions can be used to transform relative depth predictions to those in metric scale. Our method, RSA , takes as input a text caption describing objects present in an image and outputs the parameters of a linear transformation which can be applied globally to a relative depth map to yield metric-scaled depth predictions. We demonstrate our method on recent general-purpose monocular depth models on indoors (NYUv2, VOID) and outdoors (KITTI). When trained on multiple datasets, RSA can serve as a general alignment module in zero-shot settings. Our method improves over common practices in aligning relative to metric depth and results in predictions that are comparable to an upper bound of fitting relative depth to ground truth via a linear transformation. Code is available at: https://github.com/Adonis-galaxy/RSA.

RSA: Resolving Scale Ambiguities in Monocular Depth Estimators through Language Descriptions
Ziyao Zeng'
Yangchao Wu²
Hyoungseob Park'
Daniel Wang!
Fengyu Yang'
Stefano Soatto²
Dong Lao² Byung-Woo Hong'
Alex Wong'
Yale University University of California, Los Angeles *Chung-Ang University
(ziyao. zeng, hyoungseob. park, daniel wang, dhw33 d 10ylle.yalll.e
fengyu.yane alex. wone..oogil... @yale edu wuyangchaol 9978g wag..ucl..e... edu (ssoatoo, las) ucla.edu "hon@@cau.ac.k
Whate the noala
How to describe acenes?
area with seap. towel mirror
"Clessroon room decks, chains, organtzed
"Narrow cobiientone street with parked cara and residential el buildings"
Large
pedestrians distanca"
Figure 1: Can infer the scale of 3D scenes from their descriptions? Consider the description above, may observe that scale the scene closely related to objects (and their typical sizes) populating it.
Abstract
We propose method for metric-scale monocular depth estimation. Inferring depth from single image is an ill-posed problem due to the loss of scale from perspective projection during the image formation process. Any scale chosen a bias, typically stemming from training on dataset; hence, existing works have instead opted to use relative (normalized, inverse) depth. Our goal is to recover metric-scaled depth maps through a linear transformation. The crux of our method lies in the observation that certain objects (e.g., cars, trees, street signs) are typically found associated with certain types of scenes (e.g., outdoor). We explore whether language descriptions can used to transform relative depth predictions those in metric scale. Our method, RSA, takes input text caption describing objects present in an image and outputs the parameters of a linear transformation which can be applied globally relative depth map yield metric-scaled depth predictions. We demonstrate our method on recent general-purpose purpose monocular depth models on indoors (NYUv2, VOID) and outdoors (KITTI). When trained on multiple datasets, RSA can serve as general alignment module in zero-shot settings. Our method improves over common practices ali aligning relative metric depth and results in predictions that are comparable to an upper bound of fitting relative depth ground truth via a linear transformation. Code is available https: : / ///gittt github..... com/ Adonis eo--galaxyy /RSA
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Introduction
3-dimensional (3D) reconstruction from images is an ill- pol-posed problem due to the loss of a dimension through perspective projection during the image formation process: Any point along the ray of projection can yield the same image coordinate. This extra degree of freedom is often addressed by using multiple images of the same scene, such as stereo video. While an additional image (assuming co-visible sufficiently exciting textures across both images) allows one to triangulate unique points in space, scale ambiguity exists with the absence of camera calibration, measurements from an additional sensor (e.g., range, initial), or a strong prior (e.g. a supervised training set). One may argue that, such additional information should already be available during data collection. Still, modern large-scale training [44, 67] often utilizes data from diverse sources with drastically diverse setups. making resolving the scale anbiguity issue crucial.
When comes to monocular depth estimation, which predicts a dense depth map from a single image, the problem is also ill- ill-posed in that one cannot measure the distance from the camera from single view. Hence, to make inference possible, one must rely on the existence of a training set. While one option is to "bake in" additional bias of scale by training on a number of different datasets (indoors and outdoors) and attributing depth to pixel intensities, these dataset-specific biases come at the cost of generalization, limiting model transfer from one domain (indoor) to another (outdoor), let alone mixing multiple data sources. Existing monocular depth methods resort to predicting relative (normalized, inverse) depth to factor out the scale biases, but leave behind practical utility, as a trade-off, in downstream applications in spatial tasks such as manipulation, planning, and navigation.
We consider whether an additional modality can be used to resolve the scale ambiguity in linle-image 3D reconstruction, i.e., transforming scaleless relative depth to metric depth. One might observe that natural (including man-made) scenes do not occur by chance, but rather by design, with the regularity of object-scene co-occurrences [18, 28]: Certain scenes (e.g., outdoors) are composed of certain categories of objects (e.g., cars, trees, buildings) and associated with a certain order of magnitude in scale (e.g. tens of meters). Hence, we hypothesize that language, in the form of text captions or descriptions, can be used to infer the scale of the 3D scene and to transform relative depth to metric depth. The choice of language also has practical value in that it does not require costly data acquisition with an additional synchronized and calibrated sensor (e.g., lidar, time-of-flight With publicly available pre-trained panoptic segmentation or object detection models, image captioners, and lision-language models, one can automate the data acquisition, training, and inference process. Nonetheless, they are not a necessary part of the work, but may facilitate ease of use, to simulate the language description provided by human users in practical scenarios.
To test the feasibility of our hypothesis, we consider monocular depth estimation, where a strong prior is necessary for inference; this prior may come from an image, an independent modality such as language. Specifically, we consider monocular depth models belonging to the ceneral-purposs relative depth estimation paradigm to control for side effects such as the scale learned along with shapes in existing works that focus on predicting metric depth for a specific dataset. To this end, we assume access to a pre-trained scaleless monocular depth estimation model; aside from a set of images, we also assume text captions describing them. As the standard procedure to factor out scale to normalize through linear transformation, we propose to learn a parameterized function that predicts the parameters of a linear transformation based on the text description. Applying them to a relative depth map and inverting its values will transform it to a metric-scales scaled depth map to resolve the scale ambiguity. We term our method RSA, as an acronym for "Resolving Scale Ambiguities".
In one mode, like existing works that also transform relative to metric depth, albeit with images and domain-specific scaling [3], we train specific models of RSA for specific datasets (e.g., NYUv2 [46], KITTI [15], VOID [S8]. In another mode, when trained on multiple datasets across different domains, RSA generalizes well and can not only handle images sampled from the datasets it was trained on, but also those from novel datasets in zero-shot manner. The use of language, which is invariant to illumination, object orientation, occlusion, scene layout, etc., many of the nuisance variables that vision algorithms are sensitive to, demonstrates a promising avenue for general- -urross relative to metric scale recovery to complement the growing works in monocular depth estimation. We evaluate our method on indoor and outdoor benchmarks, where we improve over common practices in aligning relative depth to metric scale. We show that using RSA is comparable to matching relative depth via a linear transformation to the ground truth, or scaling using the median value of the ground truth, both of which are considered oracle relative metric recovery methods.

---

Our contributions are as follows: (i) We proposed a novel formulation of the relative to metric depth transfer. (ii) We demonstrate the feasibility of inferring scale from language and as general alignment module. (iii) We performed extensive experiments to validate performance on indoor and outdoor domains, sensitivity to text. caption, and zero-shot generalization. To the best of our knowledge, we are the first to use language as a means of relative to metric depth alignment.
Related Work
Monocular depth estimation using metric depth. Metric depth models learn to infer pixel-wise depth in metric scale (i.e. meters) by minimizing loss between depth predictions and ground-truth depth maps [2. 12, 26, 35, 55, 71, 79]. Each model typically applies to only one data domain in which it is trained, with similar camera parameters and object scales. Specifically, DORN [12] discretization technique. AdaBins [2] partitions depth ranges into
leverages a adaptive bins. NeW CRFs [71] uses neural window fully-connected CRFs compute energy. When ground-truth depth is not available, self-supervvsed approaches [4, 11, 23, 27, 30, 38, 41, 51, 52, 53, 59, 61, 70, 74, 80, 81, 84, 86] rely on geometric constraints, where scale is attributed through lidar [10, 37, 40, 57, 56, 58, 63, 68], radar [47]. binocular images [14, 17, 16]. or inertials [20, 54]. However, models that predict metric depth are limited specific datasets or scenes, and sensors [60]; thus, they do not generalize well. RSA aims to serve as a general alignment module that can predict metric depth based on relative depth across different domains.
Monocular depth estimation using relative depth. Trained across different domains, self-supervised depth estimators trained with multi-view photometric objects produce up-to-scale predictions that are linearly correlated with their absolute depth values across the domain [8, 5, 20, 21, 54, 62, 64], thus requiring scaling of their depth prediction. However, due the scale-ambiguity of multi-view photometric objective, such models are normally evaluated by aligning predictions to ground-trut at test time (typically median-scaling [16, 17, 86)), at the expense of practicality since ground-truth might not be feasible during real-world application. To enable generalization across different scenes, some (semi-) supervised depth models trained with single images adopt image-level normalization techniques to generate affine-invarian depth representations (i.e. relative depth) [24, 43, 44, 67. 73]. HND [73] hicrarchically normalizes the depth representations with spatial information and depth distributions. Depth Anything [67] learns from large-scale automatically annotated data. DPT [43] leverages vision transformers using a scale- and shift-invariant trimmed loss. MiDas [44] mixes multiple datasets with training objectives invariant to depth range and scale. Marigold [24] associates fine-tuning protocol with diffusion model. However, by definition, relative depth is scaleless, which limits their applications that require metric scaled reconstruction. Our proposed RSA addresses this limitation by grounding relative depth into a metric scale, enabling metric scale 3D reconstruction.
Relative to metric depth transfer. To transform the predicted relative depth into metric depth for evaluation and real-world application, ZoeDepth [3] fine-tunes a metric bins module on each metric depth dataset to output metric depth. Depth Anything [67] follows ZoeDepth [3] using a decoder where metric bins module computes per-pixel depth bin centers that are linearly combined to output the metric depth. MiDas [44] and Marigold [24] use linear fit to align predictions and ground truth in scale and shift for each image in inverse-depth space based on the least-square criterion before measuring errors. DPT [43] fine-tunes a global scale and shift on metric depth datasets. DistDepth [62] conducts transfer by leveraging left-right stereo consistency to integrate metric scale into scale-agnostic depth network. ZeroDepth [21] achieves transformation using input-level geometric embedding to learn an indoor scale prior over objects via a variational latent representation. However, metric decoders are limited to specific datasets, and aligning scale and shift of predictions requires ground truth during test time. RSA produces scale using text to transfer relative depth to metric depth across domains and does not require ground truth during test time.
Language model for monocular depth estimation. Vision- Language models [6. 32, 33, 39, 42, 49] acquire comprehensive understanding of languages and images through pre-training under diverse datasets, thus forming an effective baseline for downstream tasks [34, 65, 69, 76, 78, 79, 88, 66, 50]. Typically, CLIP [42] conducts contrastive learning between text-image pairs, empowering various tasks like few-shot image classification [13, 75, 77, 85], image segmentation [45, 83], object detection [45, 87], and 3D perception [22, 76, 79, 88]. In light of their emerging ability, some works [1, 22, 79, 82] have tried to apply vision-language models for monocular depth estimation. WorDepth [72] learned the distribution 3D scenes from text captions. DepthCLIP [79] leverages the
3

---

Metric depth
with wooden cabinens,
Scale:
sheel appllontes, includin
gramil countar lopa, slarnisa
Model
Shift:
dalnvshhr with lowal hargh ging
Inverse relative depth
1 = 1/(ā y
y
Depih
Ground Truth
Model
Figure 2: Overview, We infer scale and shift from the language description of an image transform the inverse relative depth from the depth model into metric depth (absolute depth in meters) prediction.
semantic depth response of CLIP [42] with a depth projection scheme to conduct zero-shot monocular depth estimation. Hu al. [22] extends DepthC1.IP with learnable prompts and depth codebook to narrow the depth domain gap. Auty et al. [1] modifies DepthCI.IP [79] using continuous learnable tokens in place of discrete human-language words. In contrast, RSA uses language to directly predict
scale, which serves as an explicit scaling constraint by transforming relative depth into metric scale.
3 Method
We consider dataset D = (T(m), {(n), , In=] with N samples of synchronized RGB image, text descriptions, depth maps, where 7 Ω C + R denotes image, y" the ground-truth metric depth map, a text description of the image, and Q the image space. We assume access a pretrained monocular depth estimation model he for the purpose of learning the parameters to predict the transformation between relative and metric depth. Given an RGB image, monocular depth estimation model predicts inverse relative depth y Ω C R R2 using parameterized function h realized as neural network, i... y == h- To recover metric-scale from (scaleless) inverse relative depth, we consider global linear transformation through the use of language description pertaining to the image of the 3D scene. Given the text description of an image as input, our method, RSA, predicts a pair of scalars denoting the scale and shift parameters of the transformation: (a, 3) = Sw: (t) RR2. The metric depth prediction is obtained by ý = 1/(á y + 3).
[n]]]
R R.
RSA. To infer the parameters of the linear transformation to align relative depth to metric scale, we employ the existing pretrained CLIP text encoder [42] feature extractor. Having been trained on large scale dataset, CLIP offers an latent space suitable to preprocess object-centric text descriptions. We note that CLIP text encoder is frozen within RSA. Given descriptions t (t1, :2..... we first encode them into text embeddings and feed them into a 5-layer shared multi-layer perceptron (MLP) project them into
256 hidden dimensions followed by two separate sets of 5- MLPs, one serves as the output head vaa I2 + R for scale parameter á and the other as the output head d/ F + R4 for shift 3 parameter. Scale and shift are assumed to be positive in favor of optimization. For ease of notation, we refer to d the parameters for the shared MLP well as the output heads, where (á. () = gv
Optimizing RSA involves minimizing a supervised loss with respect to th, which requires a forward pass of a given training image f(n) through the monocular depth model to yield y(m) = hy (I()). To ensure that the monocular depth model does not drift and update its parameters during the training of RSA, we freeze while optimizing for tb. Hence, training entails minimizing an L1 loss:
N
6² = -arg min
Σ M (z (z)lg())) JEQ
(m)
(I)
M
where ((m)) = 1/(a(m), (m) + A(a)) denotes the predicted metric-scale depth aligned from relative depth (M), x E denotes an image coordinate, and M Ω + {0. 1} denotes a binary mask indicating valid coordinates in the ground truth depth y" with values greater than zero.
Text prompt design. To test our hypothesis, we require text descriptions be paired with images. As standard benchmarks do not provide the text description of each image, we extend existing datasets by associating each image with several text descriptions. achieve this, we propose to use off-the-shelf models to generate different kinds of text. First, considered structured text, which adheres to

---

Models
Scaling
Dataset
< 1.25
8 1.33
6 1,25" 1
Abs Rel
log,0
RMSE
ZeeDepth
Image
NYUv2
0.951
0.994
0.999
0,077
0.033
0.282
DistDepth
NYUv2
0.706
0,934
1.077
DisiDepth
Median
NYUV2
0.791
0.942
0.985
0.158
0,548
ZerolSept
0.961
0.350
ZeroDepth
Median
0.926
0.986
0.081
0.338
Median
NYUV2
0.736
0.919
0.981
0.181
0.073
0.912
Lineur Fit
NYUv2
0.926
0.991
0,999
0,094
0.040
0.332
Global
NYUV2
0.904
0.988
0.958
0.109
0.045
0,357
Image
NYUv2
0.914
0.990
0.998
0.097
0,042
0.350
DPT
Image
NYUN2,KITTI
0.911
0,989
0.998
0.098
0,043
0,355
Image
NYLV2,KITTL.VOD
0.903
0.985
0.997
0.100
0,045
0.367
RSA (Ours)
NYUV2
0.916
0.990
0.998
0.097
0,042
0.347
RSA (Ours)
NYU-2.KITTI
0.913
0.988
0.998
0.099
0,042
0.352
RSA (Ours)
NYUVZ.KITTL.OID
0,912
0.989
0.998
0,099
0.043
0,355
Median
NYUV2
0.499
0.694
0,850
0.411
0.151
2,010
Linear Fit
NYUv2
0.780
0.970
0.995
0.151
0.069
0.433
Global
NYUv2
0.689
0.949
0.992
0,183
0.078
0.600
Image
NYUv2
0.729
0.958
0.994
0.175
0.072
0.563
MiDas
Image
NYUV2.IITTI
0.724
0.952
0,9 9.2
0,173
0.074
0.579
Image
NYLV2,KITTI,VOID
0.712
0.948
0.988
0.181
0.075
0.583
RSA (Ours)
NYUv2
0.955
0,993
0,171
0.072
0.569
RSA (Ouns)
NYUV2,KITTI
0.737
0.959
0.993
0.168
0.071
0.561
RSA (Ours)
NYUV2.KITTLVVII
0,709
0.944
0,989
0.173
0.076
0,580
Modian
NYUv2
0.480
0.734
0,886
0.353
0.135
1.743
NYUv2
Global
NYUV2
0.630
0.926
0.987
0.199
0.087
0.646
Image
NYUv2
DepthAnything
Image
NYUV2,KITTI
0,710
0.947
0,992
0.181
0.075
0,574
Image
NYUV2.KIUHL.OI
RSA (Ours)
NYUV2
0.775
0.975
0,997
0.147
0.065
0.484
(Ours)
NYUV2.KITTI
0.49%
RSA (Our)
NYUV2.KITTLVON
0.752
0.964
0.992
0.156
0.071
0.528
Table 1: Quantitative results on NYUv2. RSA (yellow), especially when trained with multiple datasets, generalizes better than using images predict the transformation parameters. Global refers optimizing single scale and shift for the entire dataset (same scale and shift for every sample). Image denotes predicting scales and shifts using images. Red denotes scaling that uses ground truth. Median indicates scaling using the ratio between median of depth prediction and ground truth. Linear fit denotes optimizing scale and shift to fit to ground truth for each image. DA refers domain adaptation. ZoeDepth performs per- per-pixel refinement.
certain template. We use a panoptic segmentation model MaskDINO [31] to extract the significant objects and background in the image. For an input image T, the segmentation model returns set of B object and background instances (n(), J
Jez where el denotes the class of the object or
background, and n denotes the number of instances of the object or background. Using the set of
instances, structured caption for an image can be obtained: An image with (1) c(1), (2) c22),
(B) (BB) We will shuffle the order of instances to produce 5 different structured captions for each image. Then we consider the natural text, where the text doesn't adhere certain templates and is closer to human descriptions, we use two visual
-anteirnnnn models LLa v1.6 Vicuna and LLaVA v1.6 Mistral [36]. For each model, we prompt it with the input image and a prompt, asking the model to describe the image. For each model, we provide 5 different prompts to produce different natural captions. During training, in each iteration, for given image, we randomly select one caption from those 15 captions to predict scale and shift.
4 Experiments
Datasets. We present our main result on three datasets: NYUv2 [46] and VOID [58] for indoor scenes, and KITTI [15] for outdoor scenes. NYUv2 contains images with a resolution of 480 640 where depth values from 1 10-3 3 to 10 meters. We follow [29, 35, 79] for the dataset partition, which contains 24,231 train images and 654 test images. VOID contains images with resolution of 480 where depth values from 0.2 to 5 meters. contains 48.248 train images and 800 test images following the official splits [58]. KITTI contains images with resolution of 352 1216 where depth values from 1 x 10-3 80 meters. We adopt the Eigen Split [9] consisting of 23,488 training images and 697 testing images. Following [2, 71]. we remove samples without valid ground truth, leaving 652 valid images for testing. We also report zero-shot generalization results on SUN-RGBD [48], which contains 5050 testing images, and DDAD [19], which contains 3950 validation images.
5

---

Models
Lincar lit 8 < 1.25
Scaling
Dutaset
8 1.25'
dx 1.25'
Abs Rel
RMSEE
RMSE 4
ZoeDepth
Image
0.971
0.996
0.999
0,054
2.281
Monodepth2
Median
KITTI
0.877
0.959
0.981
0.115
0.193
4.863
Zero
DA
0.892
0.961
0.977
0.102
0.196
4.378
Zepo
DA, Median
0.886
0.965
0.984
0,108
0.178
4. 4.194
Modian
KITTI
0.950
0.994
0.999
0.069
0.100
3.365
Lineur fit
KITTI
0.974
0.997
0.999
0,052
0,080
2.198
Global
0.062
2.575
Image
KITTI
0.961
0.995
0.999
0.064
0.092
2.379
DPT
Image
WVIV2,KIITI
0.956
0.989
0.993
0.066
0.098
2.477
Image
NYUVL.IITTLVIDD
0.952
0.987
0.993
0,068
0.098
2.568
RSA (Ours)
KITTI
0.963
0.995
0.999
0.061
0.090
2.354
RSA (Ours)
NYUV2KITTTI
0.962
0.994
0.998
0.060
0.089
2.342
RSA (Ours)
NYUW2KITTLVOI
0.961
0.994
0.999
0.064
0.091
2.335
Modian
KITTI
0.856
0.959
0.988
0.138
0.204
6.372
KITTI
0.824
0.952
0.989
0.154
0.174
3.833
Global
0.729
0.978
0.192
Image
KITTI
0.749
0.949
0.982
0.164
0.199
4.254
MiDas
Image
NYUV2KITTI
0.718
0.943
0.979
0.171
0.211
4,456
Image
NYUVIKITTTLVOID
0.683
0.931
0.972
0.165
0.232
4.862
RSA (Ours)
KITTI
0.798
0.948
0.981
0.163
0.185
4.082
RSA (Ours)
NYUVIKITTI
0.782
0.946
0.980
0,100
0.194
4.232
RSA (Ours)
NYTN2KITTLLOID
0.794
0.960
0.992
0,155
0.179
3.989
Modian
KITTI
0.925
0.986
0.996
0.091
0.929
3.648
Linear fit
KITTI
0.824
0.896
0.922
0,149
0.224
3.595
Global
KITTI
0.663
5.273
Image
KITTI
0.768
0.951
0.983
0,162
0.195
4.483
DepihAnything
Image
KITT2,KITTI
0.697
0.933
0.977
0.181
0.218
4.824
Image
NYUVL.IITTLVOLD
0.678
0.924
0.974
0.186
0.243
5.021
RSA (Ours)
KITTI
0.780
0.958
0.988
0.160
0.189
4,437
RSA (Ours)
NYUv2.
0.756
0.987
0.158
0.191
4.457
RSA (Ours)
NYUV2
0.786
0.967
0.995
0,147
0.179
4.143
Table 2: Quantitative results on KITTI Eigen Split. RSA (yellow), especially when trained with multiple datasets, generalizes better than using images predict the transformation parameters. Please refer to Table 1 for more details about notations.
Depth models. For DPT [43]. we use DPT-Hybrid fine-tuned for NYUv2 and KITTI respectiveyy with 123M parameters. We used the one fine-tuned NYUv2 for VOID. For MiDas [44], use MiDaS 3.1 Swin2 large-384 with 213M parameters. For AnythAny [67]. we use Depth- Anything-Small with 24.8M parameters. Different from our setting, DepthAnnthing evaluates using ZoeDepth [3]'s depth decoder that is separately fine-tuned on NYUv2 and KITTI to produce a pixel-wise scale, and MiDas evaluates by aligning prediction with ground truth in scales and shifts. We re-implement several baselines aligning with the setting of predicting a global scale and shift for scaling relative depth, provided in Table and 2.
Hyperparameters. We use the Adam [25] optimizer without weight decay. The learning rate is reduced from 3 10 to x 10 by cosine learning rate scheduler. The model is trained for 50 epochs under this scheduler. We run our experiment on GeForce RTX 3090 GPUs, with 24GB memory. For reference, if using single GPU, the training time for RSA with DepthAnything jointly NYUv2, KITTI, and VOID for 50 epochs takes 57 hours..
Evaluation metrics. We follow [7, 35, 79] to evaluate using mean absolute relative error (Abs Rel), root mean square error (RMSE), absolute error in log space (log logarithmic root mean square enror (RMSE1ng) and threshold accuracy (8:).
Quantitative results. We show results NYUv2 in Table KITTI in Table 2, and VOID in Table 3, where we improve over baselines across all evaluation metrics and approach the performance of using ground truth for scaling. Following DPT, optimize the scale and shift for the "Global" scaling baseline over the training set and use it for evaluation (i.c., same scale and shift for all test samples). We obtain the "Image" baseline by substituting CLIP text features with CLIP image features. Following [44], we perform linear regression to find the scale and shift that minimizes the eqast-square error between the ground truth metric depth and the predicted metric depth. Additionally, we also test median scaling [17]. a common practice for evaluation, which uses the ratio between median of depth prediction and ground truth as the scaling factor. Both linear fitting and median scaling are shown to demonstrate what is achievable if one were to directly fit to ground truth. We train separate RSA models for each dataset, as well as unified RSA model combining both KITTI and NYUv2, or all KITTI, NYUv2, and VOID.

---

Models
Sculing
Datuset
< 1.25 1
< 1:25"
8 < 1.255
Abs Rel
logio 4
RMSE 4
Median
VOID
0.782
0.962
0,990
0.064
0.340
Global
NYUv2 (2cre-sbot)
0.456
0,743
0.912
0.312
0.136
0.896
DPT
Image
Image
NYUV2.KITTI (zero-shot)
NYUVZ.KITTI.OLI
0.516
0.534
0.827
0.812
0.936
0,941
0.289
0.266
0.112
0.108
0.545
0.634
RSA (Ours)
NYUV2KITTI (2cro-shoo)
0.601
0.886
0,970
0.254
0.09%
0.444
(Ours)
NYUV2,KITTI.VV
0.598
0.877
0.956
0.100
0.475
Median
VOID NYUv2 (zero-shot)
0.500
0
0.899
0.347
0,130
0.829
Global
0.268
0.597
0.735
0.512
0.193
1.346
MiDas
Image
Image
NYUV2KITTII (zem-shoo)
XYUV2.KITTI.VIID
0.304
0.389
0.626
0.812
0.911
0.487
0,159
0.652
0.913
RSA (Ours)
NYUx2KITTI (acro-shoo)
0.392
0.696
0,892
0.448
0.148
0.660
RSA (Ours)
NYUv2, KITTI,
0.535
0.829
0.945
0.528
Median
VOID
0.249
01465
0.643
0.682
0.254
1.251
Global
NYUv2 shart-shot)
0.084
0,194
0.376
1.674
0,389
2,046
DepthAnything
Image
NYUv2KITTI (2cro (zero-shot)
0.093
0.215
0.412
1.497
0.345
1.963
Image
KYUV2,KITTI,
0.323
0.612
0.768
0.589
0.196
0.956
RSA (Ours)
KYUV2KITTII (2cro-shot)
0.104
0,262
0,450
1.287
0.323
1.216
RSA (Ours)
NYU-2.KITII.VOID
0,374
0.673
0.837
0.477
0.168
0.792
Table 3: Quantitative results VOID. In zero-shot generalization and multi-datase training (including the target dataset), RSA outperforms image scaling due the robustness of text, which supports better generalization. Please refer to Table for more details about notations.
13.
Figure Left: Predicted inverse scale w.r.f. median depth ground truth. Larger scenes tend to have larger median ground truth depth. For RSA trained on com- bined KITTI and NYUv2 with Depth Anything model, we fit an inverse proportional function for the predicted inverse scale in the test set (each point is an image), to verify that the scale is proportional to the median depth, that larger scenes predicted with larger scales.
23.
5.0 12.0
12.5
Although RSA trained on a single dataset may achieve slightly better performance than the unified model, the difference is minimal. In some metrics, the unified model even outperforms, demonstrating the generalizability of using language as input, given the narrow domain gap for language. Addition- ally, RSA models consistently outperform "Image" baselines in both in-domain and cross-domain scenarios and are comparable with existing methods under various settings. Our method significantly narrows the performance gap to that of using ground truth for scaling, validating the effectiveness of using language instead of the input image to predict scale.
To examine our scale predictions in more detail, Figure shows curve fitting plotted for our predicted (inverse) scale against the median of the ground truth. The trend line shows that the scale inferred from text descriptions matches well with median scaling, which a robust estimator.
Qualitative comparisons. We present representative visual examples comparing RSA with the baseline method on the NYUv2 and KITTI datasets in Figure 4 and Figure 5, respectively, to highlight the benefits of RSA. The error maps illustrate the absolute relative error. Unlike the original DPT, which uses fixed scale and shift, RSA enhances accuracy uniformly across the image without altering the structure or fine details of the depth map. This improvement is evidenced by the darker areas in the error maps, indicating better scaling and reduced errors.
Zero-shot Generalization. Considering the smaller domain gap in language descriptions across various scenes, we perform zero-shot transfer experiment to demonstrate RSA's generalization ability. We evaluate the models on the Sun-RGBD [48] and DDAD [19] without fine-tuning. As shown in Table 4 and Table 5, RSA achieves superior results compared to baselines, existing methods, and ground truth scaling. This suggests that language descriptions offer a viable option for relative to metric transfer when generalizing across diverse data domains. Note that single global scale and shift are ineffective for both indoor and outdoor settings. Therefore, for the "Global" model, we only to NYUv2 to obtain reasonable global scale and shift for the zero-shot Sun Sun-RGBD evaluation, and fit it to KITTI for DDAD evaluation.
Prompt design for input text. In Table 6, we investigate different designs of RSA text prompts in training and how they affect the performance. Here, to make the experiment more controllable, we use only structured text and only produce one caption for each image to train each model.
7

---

Image
Ours
Ours Error
DPT
DPT
Ground Truth
Depth
Velua
Reb
0.590
0.75
0.90
Figure 4: Visualization of depth estimations NYUv2. Building upon DPT, while better scale factor does not change structure of the depth prediction, leading to visually similar depth maps, it significantly reduces the overall error (darker in the error map). Note: Zeros in ground truth indicate the absence of valid depth values.
Models
Scaling
Dataset
1.25
8< 1.25"
1.25"
Abs Rel
log.u
RMSE
Adabins
NYUv2
0.771
0.944 0.970
0,983 0.993
0.159
0.068
0.476
DepthFormer
NYUv2
0.815
0.137
0.059
0.408
Zepth
Image
NYUv2
Deetephh-111
Image
NYUv2
0.864
0.1
0.346
ZeeDepth
Image
0.356
Innar PM
SUN-RGED
0.12
0.967
0.993
0,139
0.059
0.412
NYUv2
0.773
0.945
0.984
0.154
0,482
DPT
Image
NYUV2,KITTI
0.778
0.953
0.984
0.153
0.068
0.478
RSA (Ours)
NYUV2.KITTI
0.781
0.953
0.152
0.066
0.463
RSA (Ours)
NYUV2
0.788
0.953
0.986
0.150
0.065
0.458
Linear Fát
SUN RGBD
0.632
0.912
0.97]
0.241
0.102
1.132
Global
NYLv2
0.572
0.889
0.956
0.297
0,132
1.464
MiDas
Image
NYUS2.KITTI
0.594
0.895
0.962
0.275
0.125
1.374
RSA (Ours)
NYUV2.KITTI
0.612
0,903
0.964
0.268
0.122
1.302
RSA (Ours)
NYUVLKITTL
0.623
0.908
0.968
0.253
0.116
1,223
Linear Fi
SUN-KGRD
0.378
0.979
0.995
0.113
0.054
0.332
Global
NYUv2
0,534
0.872
0.951
0.313
0.138
1 1.692
DepthAnything
KITTI.KITTIVVOI
0.588
0.892
0.963
0.279
0.126
1.392
RSA (Ours)
NYUV2KITTI
0.621
0.915
0.970
0.238
0.099
1.024
RSA
NYUV2.KITTI.VOM
0.645
0.927
0.978
0.203
0.095
1.137
Table 4: Zero-shot generalization to SUN-RGBD. With training datasets for scale prediction, RSA model generalizes better due the robustness text, but predicting scale using images suffers from domain gaps among training images. The models are tested on the Sun-RGBD without any fine-tuning. For ZoeDepth. X indicates pre-training, and MI2 indicates 12 datasets for pre-training. ZoeDepth results were taken from their original manuscripts, using depth decoder for scaling. Please refer to Table for detailed notations.
In the 1st and 2nd rows of Table 6, we use an object detector Detie [87] produce only foreground objects in images and form input text using only foreground objects. We observe that if the input text only specifies the types of objects and their numbers, RSA can still accurately predict the scale for indoor scenes, these spaces are typically filled with various pieces of furniture. However, this approach performs poorly outdoor scenes, which tend be more open and sparse. For instance, parking lots of different sizes may contain varying numbers of cars: small lot may be crowded, while a large lot may appear empty. In the 3rd and 4th rows of Table 6, we use text formed using segmentation results; we observe that after including background classes, the model works better

---

Models
Scaling
Dutaset
8 1.25
6 1,25'
1.25.
AbsRel.
RMSEE
RMSE 1
0.154
8,560
NeWCRPs
KITTI
0.874
0.1
6.183
ZaeDepth X
Image
KITTI
0.790
0.137
7.734
MoeDepu-M12
Image
KITTI
0.835
0.129
7.108
ZocDepth-M12
Image
KITTI
0.824
0.138
7.225
Lineur Fit
DDAD
0.802
0.954
0.990
0.163
0.254
10.342
Global
Image
NYUV2KITTI
0.763
0.931
0.978
0.179
0.308
14,408
DPT
Image
NYUW2.KFTTLVOD
0.731
0.910
0.622
0.191
0.324
16.132
RSA (Ouns)
NYUV2.KITTI
0.777
0.938
0.981
0.171
0.284
13.539
RSA (Ours)
NYUV2.KITTLVOID
0.768
0,942
0.983
0.165
0.276
12.437
Lineur Fii
DDAD
0.664
0312
0.733
0,209
0.301
18.34]
Global
KITTI
0.603
0.864
0.925
0.253
0.336
20.594
MiDas
Image
Image
NYUN2KITTLVOID
NYUV2.KITTI
0.564
0.616
0,883
0.862
0.34
0.925
0.231
0,243
0.352
0.331
22,689
20 034
RSA (Ours)
NYUv2 KITTI
0.631
0.903
0.966
0.223
19.342
RSA (Ours)
NYUV2KIITTLVODD
0.642
0.908
0.966
0,218
0.331
18,293
Lincar Fit
DDAD
0.673
0.932
0.833
0.182
0.286
18.423
Global
KITTI
0.612
0.883
0.963
0.221
0.323
21.345
DepihAAnyhhing
Image
NYUT2KITTI
0.623
0.890
0.968
0.217
0.316
20.834
Image
NYUVZKITTLVOID
0.586
0.874
0.956
0,243
0.348
22.351
RSA (Ours)
NYUV2KITTI
0.642
0.903
0.976
0.207
0.303
19.715
NYUVZKKITTLVOID
Table 5: Zero-shot generalization to DDAD. With more training datasets for scale prediction, RSA model achieves a better generalization due to the robustness of text, but predicting scale using images suffers from domain gaps among training images, Models tested on the DDAD without any fine-tuning. Please refer to Table and Table for more details about notations.
Image
Ours
Ours Error
Emor
Abba Reil
0.00
DPT
0.125
DPT Error
0.25
0.000
0005
Ground Truth
0000
Figure 5: Visualization of depth estimations on KITTI. Building upon DPT, while better scale factor does not change the structure of the depth prediction, it significantly reduces the overall error (darker the error map). Note: Zeros in ground truth depth indicate the absence of valid depth values.
especially for outdoor scenes, since different backgrounds (like wall, sidewalk, highway, road, sky, house, building) can reflect different scales.
Sensitivity different text input. To demonstrate the impact of various text inputs on scale prediction, we utilize a trained RSA model based on the DepthAnything model. By altering the text input, we observe changes scale and shift predictions. We provide the experiment results in Table From top to bottom, we provide text descriptions of scenes from small to large, and our model can properly predict the corresponding scale and shift. This shows promise that our method able to manipulate the scale of 3D scenes with case.

---

Prompt
NYUv2
KITTI
"An image with e()), el"), el.) ... with Object Detection Results
0.106
0.070
"An image with (1)) (()), K(3) e(a), with Object Detection Results
0.101
0.068
"An image with e()), el3) e()) with Panoptic Segmentation Results
0,102
0.063
"An image with K0) e()) K el") with Panoptic Segmentation Results
0.100
0.061
Table 6: Different prompt design for RSA. Absolute relative errors (Abs Rel) reported. RSA models are trained
using cross-datasets with the DPT model. For one given image, clt) is the class of detected or segmented instance, k( is the numher of all instances helonging to c! By using segmentation results, the text includes background, which improves scale predication, especially for outdoors.
Input Text
Iny scale
Inv shift
A room with a refrigerator, table, and shelf.
0.0387
0.2286
A black oftice chair bedroom, next white door and clothes rack.
0.0354
0.2437
The image shows store with variety of items for sale.
0,0276
0,1812
The image shows classroom with desks and chairs, hulletin board, and clock.
0.0254
0.1633
A group of people walking down a city street.
0,0102
0.0063
A bustling city street with white driving down it.
0.0096
0.0053
A busy highway filled with cars, with blue and white sign on the right side.
0.0067
0.0045
Table 7: Sensitivity study to different text input. We show the inverse scale and shift here; smaller value indicates a larger scene. From top to bottom, we describe scenes from small to large scale. Results show that we could control the scale of a scene by providing different text descriptions, to better manipulate a 3D scene.
5 Discussion
Conclusion. We present the first study exploring whether language, as an additional input modality, resolve the scale ambiguity in monocular depth estimation, issue particularly relevant in the context of the recent trend towards large-scale mixed dataset training. We propose a framework, RSA, which learns to convert scaleless (relative) depth map metric depth using language descriptions as input. RSA utilizes the pre-trained CLIP encoder, and maps a language description of the image to scale and shift factors that transform the relative depth metric depth. RSA is validated through extensive experiments on three benchmark datasets and three pre-trained relative depth models, demonstrating significant promise by drastically closing the gap between depth estimation accuracy and its upper bound (that relies on ground truth), validating the hypothesis that language carries valuable scale information that could be used to enhance depth estimation. Moreover, demonstrate that a unified model trained on both indoor and outdoor datasets with diverse scene compositions generalizes across both scenarios, highlighting the robustness of language information in inferring scale. Finally, zero-shot transfer experiment shows that the minimal domain gap of language description across scenes further generalizes to unseen data domains, without additional training. The generalizability of RSA stems of our choice of modality, language, which is invariant to nuisance variability that are present in images from lighting conditions and occlusions to specular reflectance and object deformations. Our results demonstrate that RSA is a viable choice to support relative to metric scale alignment for general- purrosse monocular depth estimators.
Limitations and future work. We assume that the estimated 3D scene is up to an unknown scale. Although simple global scale has proven effective, may not always be sufficiently expressive for converting relative depth to absolute depth, especially when the relative depth is inaccurate. In such cases, global scaling may not adequately recover high-fidelity metric-scaled depth map due to the presence of outliers. To address this, one may need to refine the relative depth outputted by general-purpose monocular depth estimators. Future research may include extending RSA to handle finer adjustments to also refine depth estimates and investigate the potential of inferring region- wise or even pixel-wise scales using language input. Lastly, while language boasts high ease of use, RSA is also vulnerable to malicious users who may choose captions to steer predictions incorrectly.
10

---

Acknowledgements. This work was supported by NSF 2112562 Athena Al Institute, IITP-2021--- 01341, and SRRF-RS-2023-02251366
References
[1] Dylan Auty and Krystian Mikolajczyk. Learning to prompt clip for monocular depth estimation: Exploring the limits of human language. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2039-2047, 2023.
[2] Shariq Farooq Bhat, Ibraheem Alhashim, and Peter Wonka. Adabins: Depth estimation using adaptive bins. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4009-4015 4018, 2021.
[3] Shariq Farooq Bhat, Reiner Birkl, Diana Wofk, Peter Wonka, and Matthias Müller Zoedepth: Zero-shot transfer by combining relative and metric depth. arXiv preprint vrXiv:2302.12286 2023.
[4] Jia- Ha-Wang Bian, Huangying Zhan, Naiyan Wang, Zhichao Li, Le Zhang, Chunhua Shen, Ming- Ming Cheng, and Ian Reid. Unsupervised scale-consistent depth learning from video. Interna- lional Journal of Computer Vision, (29)))2248-2564, 2021.
[5] Jiawang Bian, Zhichao Li, Naiyan Wang. Huangying Zhan, Chunhua Shen, Ming-Ming Cheng, and Ian Reid. Unsupervised scale-consistent depth and ego-motion learning from monocular video. Advances in neural information processing systems, 32, 2019.
[6] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski. and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the JEEF/CVF international conference on computer vision, pages 9650-9660, 2021.
[7] Wenjie Chang, Yueyi Zhang, and Zhiwei Xiong. Transformer-based monocular depth estimation with attention supervision. In 32nd British Machine Vision Conference (BMVC 2021), 2021.
[8] Alexandra Dana, Nadav Carmel, Amit Shomer, Ofer Manela, and Tomer Peleg. One scalar is all you need-absolute depth estimation using monocular self-superrision. arXiv preprint arXiv:2303. 07662, 2023.
[9] David Eigen, Christian Puhrsch, and Rob Fergus. Depth map prediction from a single image using multi-scale deep network. Advances in neural information processing systems, 27, 2014.
[10] Vadim Ezhov. Hyoungseob Park. Zhaoyang Zhang. Rishi Upadhyay, Howard Zhang, Chethan Chinder Chandrappa. Achuta Kadambi, Yunhao Ba, Julie Dorsey, and Alex Wong. All-day depth completion. In 2023 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2024.
[11] Xiaohan Fei, Alex Wong, and Stefano Soatto. Geo-supervised visual depth prediction. IEEE Robotics and Automation Letters, 4(2):1661-1668, 2019.
[12] Huan Fu, Mingming Gong, Chaohui Wang, Kayhan Batmanghelich, and Dacheng Tao. Deep ordinal regression network for monocular depth estimation. In Proceedings of the IEEE conference on computer vision and patiern recognition, pages 2002-2011, 2018.
[13] Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter: Better vision-language models with feature adapters. arXiv preprint arXiv:21 10.0454, 2021.
[14] Ravi Garg, Vijay Kumar Bg, Gustavo Cameerr, and Ian Reid. Unsupervised enn for single view depth estimation: Geometry to the rescue. In Computer Vision 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part VIII 14, pages 740-756. Springer, 2016.
[15] Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In 2012 IEEE conference on computer vision and pattern recognition, pages 1364-3361. IEEE, 2012.
[16] Clément Godard, Oisin Mac Aodha, and Gabriel J Brostow. Unsupervised monocular depth estimation with left-right consistency. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 270-279, 2017.

---

[17] Clément Godard, Oisin Mac Aodha, Michael Firman, and Gabriel J Brostow. Digging into self-supervised monocular depth estimation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 3828-3838, 3838, 2019.
[18] Michelle R Greene. Statistics of high-level scene context. Frontiers in psychology, 4:54269, 2013.
[19] Vitor Guizilini, Rares Ambrus, Sudeep Pillai, Allan Raventos, and Adrien Gaidon. 3d packing for self-supervised monocular depth estimation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
[20] Vitor Guizilini, Igor Vasiljevie, Rares Ambrus, Greg Shakhnarovich, and Adrien Gaidon. Full surround monodepth from multiple cameras. IEEE Robotics and Automation Letters, 7(2):5397- 5404, 2022.
[21] Vitor Guizilini, Igor Vasiljevic, Dian Chen, Rares Ambrus, and Adrien Gaidon. Towards zero- shot scale-aware monocular depth estimation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9233-9243, 2023.
[22] Xueting Hu, Ce Zhang, Yi Zhang, Bowen Hai, Ke Yu, and Zhihai He. Learning to adapt clip for few-shot monocular depth estimation. arXiv preprint arXiv:2311.0103 2023.
[23] Pan Ji, Runze Li, Bir Bhanu, and Yi Xu. Monoindoor: Towards good practice of self superviid monocular depth estimation for indoor environments. In Proceedings of the IEEE/CVE Interna- tional Conference on Computer Vision, pages 12787-12796, 2021.
[24] Bingxin Ke, Anton Obukhov, Shengyu Huang, Nando Metzger, Rodrigo Caye Daudt, and Kon- rad Schindler. Repurposing diffusion- based image generators for monocular depth estimation. arXiv preprint arXiv:23 02/2 2023.
[25] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 141iv:1412.69800 2014.
[26] Dong Lao, Yangchao Wu, Tian Yu Liu, Alex Wong, and Stefano Soatto. Sub-token vit embedding via stochastic resonance transformers. In International Conference Machine Learning. PMLR, 2024.
[27] Dong Lao, Fengyu Yang. Daniel Wang, Hyoungseob Park, Samuel Lu, Alex Wong, and Stefano Soatto. On the viability of monocular depth pre-training for semantic segmentation. In European Conference on Computer Vision. Springer, 2024.
[28] Tim Lauer, Tim HW Cornelissen, Dejan Draschkow, Verena Willenbockel, and Melissa L-H Vö. The role of scene summary statistics in object recognition. Scientific reports, 8(1):14666, 2018.
[29] Jin Han Lee, Myung-Kyu Han, Dong Wook Ko, and II Hong Suh. From big small: Multi-scale local planar guidance for monocular depth estimation. arXiv preprint arXiv::907. 10326, 2019.
[30] Boying Li, Yuan Huang, Zeyu Liu, Danping Zou, and Wenxian Yu. Structdepth: Leveraging the structural regularities for self-supervised indoor depth estimation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12663-12673, 2021.
[31] Feng Li, Hao Zhang, Huaizhe Xu, Shilong Liu, Lei Zhang, Lionel M Ni, and Heung- Yeung Shum. Mask dino: Towards a unified transformer-basee framework for object detection and segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3041-3050, 2023.
[32] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301. 12597, 2023.
[33] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language- image tre-training for unified vision-languaa language understanding and generation. In International Conference on Muchine Leurning, pages 12888-12900. PMLR, 2022.
[34] Panfeng Li, Qikai Yang, Xieming Geng, Wenjing Zhou, Zhicheng Ding, and Yi Nian. Exploring diverse methods in visual question answering. In 2024 5th International Conference on Electronic Communication and Artificial Intelligence (ICECAI), pages 681-685. IEFF, 2024.
[35] Ce Liu, Suryansh Kumar, Shuhang Gu, Radu Timofte, and Lue Van Gool. Va-depthnet: A variational approach to single image depth prediction. arXiv preprint 230.::2300.0656666 2023.
12

---

[36] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, 2023.
[37] Tian Yu Liu, Parth Agrawal, Allison Chen, Byung- Woo Hong, and Alex Wong. Monitored distillation for positive congruent depth completion. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part II, pages 35-53. Springer, 2022.
[38] Reza Mahjourian, Martin Wicke, and Anelia Angelova. Unsupervised learning of depth and ego-motion from monocular video using 3d geometric constraints. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5667-5675, 2018.
[39] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023.
[40] Hyoungseob Park, Anjali Gupta, and Alex Wong. Test-time adaptation for depth completion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20519-20522 2024.
[41] Rui Peng, Ronggang Wang, Yawen Lai, Luyang Tang, and Yangang Cai. Excavating the potential capacity of self-supervised monocular depth estimation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15560-15569, 2021.
[42] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning,
pages 8748-8788788 8763. PMLR, 2021.
[43] René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction. In Proceedings of the IEEE/CVF international conference on computer vision, pages 12179-
12188, 2021.
[44] René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. IEEE transactions on pattern analysis and machine intelligence, 4233):163311377, 2020.
[45] Yongming Rao, Wenliang Zhao, Guangyi Chen, Yansong Tang. Zheng Zhu, Guan Huang. Jie Zhou, and Jiwen Lu. Denseclip: Language-guide dense prediction with context-aware prompting. arXiv preprint arXiv:21 12.0::2212..051/ 12.01518, 2021.
[46] Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and support inference from rgbd images. In Computer Vision-ECCV 2012: 12th European Conference on Computer Vision, Florence, Italy, October 7-13, 2012, Proceedings, Part V 12, pages 746-760. Springer, 2012.
[47] Akash Deep Singh, Yunhao Ba, Ankur Sarker, Howard Zhang, Achuta Kadambi, Stefano Soatto, Mani Srivastava, and Alex Wong. Depth estimation from camera image and mmwave radar point cloud. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9275-9285, 2023.
[48] Shuran Song, Samuel P Lichtenberg, and Jianxiong Xiao. Sun rgb-d: A rgb-d scene understand- ing benchmark suite. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 567-576, 2015.
[49] Yiyi Tao, Zhuoyue Wang, Hang Zhang, and Lun Wang. Nevlp: Noise-robust framework for efficient vision-language pre-training. arXiv preprint arXiv:2409.09555 2024.
[50] Jiahang Tu, Hao Fu, Fengyu Yang, Hanbin Zhao, Chao Zhang, and Hui Qian. Texttoucher: Fine-grained text-to-touch generation. arXiv preprint arXiv:2409.0542) 2024.
[51] Rishi Upadhyay, Howard Zhang, Yunhao Ba, Ethan Yang, Blake Gella, Sicheng Jiang, Alex Wong, and Achuta Kadambi. Enhancing diffusion models with 3d perspective geometry constraints. ACM Transactions on Graphics (TOG), 42(6):1-15, 2023.
[52] Ruoyu Wang, Zehao Yu, and Shenghua Gao. Planedepth: Self-supervise depth estimation via orthogonal planes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21425-21434 2023.
13

---

[53] Youhong Wang, Yunji Liang, Hao Xu, Shaohui Jiao, and Hongkai Yu. Sqldepth: Generalizable self-supervised fine-structured monocular depth estimation. arXiv preprint arXiv:2309.005526 2023.
[54] Yi Wei, Linqing Zhao, Wenzhao Zheng, Zheng Zhu, Yongming Rao, Guan Huang, Jiwen Lu, and Jie Zhou. Surrounddepth: Entangling surrounding views for supervieee multi-camera depth estimation. In Conference on Robot Learning, pages 539-549. PMLR, 2023.
[55] Alex Wong. Safa Cicek, and Stefano Soatto. Targeted adversarial perturbations for monocular depth prediction. Advances in neural information processing systems, 11:8486-8499) 2020.
[56] Alex Wong, Safa Cicek, and Stefano Soatto. Learning topology from synthetic data for unsupervised depth completion. IEEE Robotics and Automation Letters, 6(2):1495-1502, 2021.
[57] Alex Wong. Xiaohan Fei, Byung-War Woo Hong. and Stefano Soatto. An adaptive framework for learning unsupervised depth completion. IEEE Robotics and Automation Letters, 6(2):3120- 3127, 2021.
[58] Alex Wong, Xiaohan Fei, Stephanie Tsuei, and Stefano Soatto. Unsupervised depth completion from visual inertial odometry. IEEE Robotics and Automation Letters, 5(2):1899-1906 2020.
[59] Alex Wong and Stefano Soatto. Bilateral cyclic constraint and adaptive regularization for unsupervised monocular depth prediction. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 5644-5656 2019.
[60] Alex Wong and Stefano Soatto. Unsupervised depth completion with calibrated backprojection layers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12747-12756, 2021.
[61] Cho-Ying Wu, Jialiang Wang, Michael Hall, Ulrich Neumann, and Shuochen Su. Toward practical monocular indoor depth estimation. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 381 3814-3824, 2022.
[62] Ying Wu, Jialiang Wang, Michael Hall, UIrich Neumann, and Shuochen Su. Toward practical monocular indoor depth estimation. In Proceedings of the IEEE/CVF conference on computer vision and patlern recognition, pages 3814-3824, 2022.
[63] Yangchao Wu, Tian Yu Liu, Hyoungseob Park, Stefano Soatto, Dong Lao, and Alex Wong. Augundo: Scaling up augmentations for monocular depth completion and estimation. In European Conference on Computer Vision. Springer, 2024.
[64] Feng Xue, Guirong Zhuo, Ziyuan Huang, Wufei Fu, Zhuoyue Wu, and Marcelo H Ang. Toward hierarchical self-supervised monocular absolute depth estimation for autonomous driving applications. In 2020 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pages 2330-2337. IEEE, 2020.
[65] Fengyu Yang, Chao Feng. Ziyang Chen, Hyoungseob Park, Daniel Wang, Yiming Dou, Ziyao Zeng, Xien Chen, Rit Gangopadhyay, Andrew Owens, and Alex Wong. Binding touch to every- thing: Learning unified multimodal tactile representations. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2024.
[66] Fengyu Yang, Chao Feng, Daniel Wang. Tianye Wang. Ziyao Zeng, Zhiyang Xu, Hyoungseob Park, Pengliang Ji, Hanbin Zhao, Yuanning Li, et al. Neurobind: Towards unified multimodal representations for neural signals. arXiv preprint arXiv:2407.14022 2024.
[67] Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything: Unleashing the power of large-scale unlabeled data. arXiv preprint Stf.:22401.18911, 2024.
[68] Yanchao Yang, Alex Wong, and Stefano Soatto. Dense depth posterior (ddp) from single image and sparse range. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern 2019.
Recognition, pages
[69] Chenyu You, Yifei Mint, Weicheng Dai, Jasjeet S Sekhon, Lawrence Staib, and James S Duncan. Calibrating multi-modal representations: A pursuit of group robustness without annotations. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 26140-26150. IEEE, 2024.
[70] Zehao Yu, Lei Jin, and Shenghua Gao. P net: Patch-match and plane-regularizztton for unsupervised indoor depth estimation. In European Conference on Computer Vision, pages 206-222. Springer, 2020.
14

---

[71] Weihao Yuan, Xiaodong Gu, Zuozhuo Dai, Siyu Zhu, and Ping Tan. Neural window fully- connected crfs for monocular depth estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 39 3916-3925, 2022.
[72] Ziyao Zeng, Daniel Wang, Fengyu Yang, Hyoungseob Park, Stefano Soatto, Dong Lao, and Alex Wong. Wordepth: Variational language prior for monocular depth estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9708-9719 19, 2024.
[73] Chi Zhang, Wei Yin, Billzb Wang, Gang Yu, Bin Fu, and Chunhua Shen. Hierarchical normal- ization for robust monocular depth estimation. Advances in Neural Information Processing Systems, 35:111288-411131 2022.
[74] Mingliang Zhang, Xinchen Ye, Xin Fan, and Wei Zhong. Unsupervised depth estimation from monocular videos with hybrid geometrie-refined loss and contextual attention. Neurocomputiigg 379:250-261, 2020.
[75] Renrui Zhang, Rongyao Fang, Peng Gao, Wei Zhang, Kunchang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li. Tip-adapter: Training-free free clip-adapter for better lasgun-anguuag modeling. 2021.
arXiv preprint arXiv:2111.033
[76] Renrui Zhang, Ziyu Guo, Wei Zhang, Kunchang Li, Xupeng Miao, Bin Cui, Yu Qiao, Peng
Gao, and Hongsheng Li. Pointclip: Point cloud understanding by clip. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8552-8562, 2022.
[77] Renrui Zhang, Longtian Qiu, Wei Zhang, and Ziyao Zeng. Vt-clip: Enhancing vision-language
models with visual-guided texts. arXiv preprint arXiv:21 2021.
[78] Renrui Zhang, Ziyao Zeng, Ziyu Guo, Xinben Gao, Kexue Fu, and Jianbo Shi. Dspoint: Dual- scale point cloud recognition with frequfrry fusion. arXiv preprint arXiv:2111.10332. 2021.
[79] Renrui Zhang, Ziyao Zeng, Ziyu Guo, and Yafeng Li. Can language understand depth? In Proceedings of the 30th ACM International Conference on Multimedia, pages 6868-6874, 2022.
[80] Chaoqiang Zhao, Youmin Zhang, Matteo Poggi, Fabio Tosi, Xianda Guo, Zheng Zhu, Guan Huang. Yang Tang, and Stefano Mattoccia. Monovit: Self-supervised monocular depth estima- tion with vision transformer. In 2022 International Conference on 3D Vision (3DV), pages 668-678. IEEE, 2022.
[81] Wang Zhao, Shaohui Liu, Yezhi Shu, and Yong-Jin Liu, Towards better generalization: Joint dopth-pose learning without posenet. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 9151-9161, 2020.
[82] Wenliang Zhao, Yongming Rao, Zuyan Liu, Benlin Liu, Jie Zhou, and Jiwen Lu. Unleashing text-to-image image diffusion models for visual perception. arXiv preprint arXiv:2303.02153 2023.
[83] Chong Zhou, Chen Change Loy, and Bo Dai. Denseclip: Extract free dense labels from clip.
arXiv preprini arXiv:21 2021.
[84] Junsheng Zhou, Yuwang Wang, Kaihuai Qin, and Wenjun Zeng. Moving indoor: Unsuper- vised video depth learning in challenging environments. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8618-8627, 2019.
[85] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. arXiv preprint arXiv:2 109.0/ 134, 2021.
[86] Tinghui Zhou, Matthew Brown, Noah Snavely, and David G Lowe. Unsupervised learning of depth and ego-motion from video. In Proceedings of the IEFE conference on computer vision and pattern recognition, pages 1851-1858, 2017.
[87] Xingyi Zhou, Rohit Girdhar, Armand Joulin, Philipp Krähenbühl, and Ishan Misra. Detecting twenty-thousand classes using image-level supervision. In European Conference on Computer Vision, pages 350-368. Springer, 2022.
[88] Xiangyang Zhu, Renrui Zhang, Bowei He, Ziyu Guo, Ziyao Zeng Zipeng Qin, Shanghang Zhang, and Peng Gao. Pointelip v2: Prompting clip and gpt for powerful 3d open-world learning, In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2639-2650, 2023.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We claim in the abstract and introduction that we proposed novel formulation of the relative to metric depth transfer, and verify the feasibility of inferring scale from language and as a general alignment module.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: We create a separate section discuss the potential limitations, including lack of exploring human- menaa-ggeerrtedd descriptions, lack of certain factors in current descriptions, and may not be expressive enough especially when the relative depth inaccurate.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
16

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: We don't have theoretical result in this paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the suppiemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Answer: [Yes]
Justification: We have provided enough details about model architectures, datasets, depth model setup, evaluation metrics, and hyperparameters to reproduce all necessary experi- ments.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
17

---

reproducible, they 5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Question:
Answer: [Yes]
Justification: We release our code and checkpoints, including training and evaluation scripts. All datasets used in the paper open open-sourced.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines
cc/
public/ guides Code s ssionpoiccy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: /nips... cc/ cc/public///ccccccccccc l/gggddess Codee Submi ss onooninninnooo for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
We have provided enough details about training and testing, including data splits, hyperparameters, type of optimizer, ablation study, and analysis to understand the results.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Error bars are not reported because it would be too computationally expensive.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
18

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We have stated sufficient information for computer resources in the hyperpa- rameter section, including the type of compute worker, memory, and time of execution.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: https://nnuuriae /twp//euupp...// wttpp//wlc//tttccceeeeeeeeeeeeee
Answer: [Yes]
Justification:
We have carefully reviewed the NeurIPS Code of Ethics and will obey it.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws regulations in their jurisdiction)
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discussed that method could potentially enhance safety in autonomous systems and benefit augmented reality and virtual reality, but might lead to potential job losses in sectors like transportation.
Guidelines:
19

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: As model for Monocular Depth Estimation, our model doesn't 't have a high risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We have properly cited all existing assets used in the paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
20

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, papers thoode com/ dat a seets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We release and document our code well.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our paper doesn't 't involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Our paper doesn't 't involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
21

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
22


Reviews:
[{"review_id": "t7Rjp8XA6X", "reviewer": "Reviewer_KH1n", "summary": "This paper tackles the challenge of scale ambiguity in monocular scenes. The idea is to leverage the priors in LLMs to estimate the scales of an image. Evaluation is performed on NYU and KITTI datasets in comparison to baseline heuristics.", "questions": "see above", "limitations": "see above.\n\n- Only few datasets\n- Code not yet available\n- Insignificant improvements\n- Soundness of solution is not convincing", "rating": 6, "confidence": 4, "soundness": 1, "presentation": 3, "contribution": 2, "strengths": "- The value of LLMs is shown in a novel domain\n- code will be made available", "weaknesses": "- The proposed solution seems unreasonable, at least how it is presented in the paper. Specifically, the LLM does not utilize the image but only text alone. Admittedly, the text is generated from image input, but is limited to existing object detection and only tested in 2 domains/datasets. Since the image data is not used (or the relative depth prediction) the LLM prediction seems to learn dataset biases rather than from real image data.\n- This concern is also shown in the evaluation, were only a limited evaluation on 2 datasets is performed. Results are insignificantly improved. The comparison is limited so simple baselines, while work that explicitly uses object scales to refine relative depth are not shown.\n- The value of extracting text from images to then compute scales from text seems limited in real applications. A comparison to finetuned GPT4v (or equivalent) with image data alone to directly estimate scales is suggested.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Review: The proposed solution seems unreasonable... LLM does not utilize the image but only text alone.\n\nResponse: There may be a confusion. We do not use an LLM, nor do we mention LLMs in the main paper. The proposed study investigates the hypothesis of whether language, in the form of captions or descriptions, can infer the scale of 3D scenes. This bears both scientific and practical merit. Scientifically (see L20-38), this addresses the long-standing problem of scale ambiguity in monocular depth or structure-from-motion problems, where its ill-posedness necessitates a prior, or an inductive bias. We posit that objects observed in the 3D scene provide sufficient cues to infer the scale (e.g., order of millimeters, meters, 10s of meters, etc.) of the reconstruction. To the best of our knowledge, we are the first to use language to infer scale for monocular depth estimation (L76-77). Practically (see L46-50), text is arguably cheaper to obtain than range measurements (e.g., from lidar, radar) and is invariant to nuisances (e.g., illumination, occlusion, viewpoint) present in images. This allows one to generalize across domains, as evidenced by our zero-shot evaluation in Tables 1, 2 attached and Table 3 in the main paper. Our work also opens avenues to forgo additional sensors.\n\n\nReview: This concern is also shown in the evaluation, were only a limited evaluation on 2 datasets is performed. Results are insignificantly improved.\n\nResponse: We clarify that we tested our method on three datasets: NYUv2, KITTI, and SUN3D. Nonetheless, per the reviewer’s suggestion, we evaluate our method on two additional datasets: VOID (indoor) and DDAD (outdoor) (see Tables 1, 2 attached). Our conclusions from the main paper still hold.\n\n\nReview: The comparison is limited so simple baselines, while work that explicitly uses object scales to refine relative depth are not shown. (Only few datasets, Insignificant improvements)\n\nResponse: As stated in L76-77, to the best of our knowledge, we are the first to consider using descriptions of objects to infer scale. The closest work is ZeroDepth [16] (excerpt from [16], “geometric embeddings ... to learn a scale prior over objects”), which we compare with in Tables 1, 2 in the main paper. If the reviewer has specific works in mind, we are happy to make comparisons. Nonetheless, in Tables 3, 4 attached, we provide a baseline of scaling the reconstruction based on 3D object sizes and their projections onto the image by directly solving for scale specific to each image. Our method improves over explicitly using known object sizes to scale the reconstruction for both KITTI and NYUv2.\n\n\nReview: …limited to existing object detection... The value of extracting text from images to then compute scales from text seems limited in real applications.\n\nResponse: As stated in L48-50 of the main paper, an object detector is only used for ease of experiments; there is nothing in our method that ties us down to it. Tables 5, 6 in the attached show that segmentation models can also be used to construct captions (see rows marked with ``Objects + Background''). In practice, assuming spatial continuity, one only needs to describe the 3D scene once and use the inferred transformation parameters for the rest of the scene. This facilitates metric-scale depth estimates in sensor heavy applications such as autonomous driving and augmented, mixed, and virtual reality.\n\nReview: A comparison to finetuned GPT4v (or equivalent) with image data alone to directly estimate scales is suggested.\n\nResponse: We do provide this in Tables 1, 2, 3 of the main text. Rows marked with \"Image\" in the \"Scaling\" column refer to using CLIP image features alone to directly estimate scale.\n\n\nReview: Code not yet available\n\nResponse: We will release our code upon acceptance.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VLqnWVzA9L", "reviewer": "Reviewer_ZAYV", "summary": "The paper proposes a method for metric-scale monocular depth estimation called RSA. RSA aims to recover metric-scaled depth maps through a linear transformation, leveraging language descriptions of objects in scenes. The proposed method is validated on multiple standard datasets.", "questions": "How does the method perform in real-world scenarios where text captions may not be readily available or may be noisy?", "limitations": "N/A", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The use of language descriptions to convert relative depth predictions into metric scale depth maps;\n\nThe proposed technique outperforms previous methods;", "weaknesses": "The paper could benefit from a deeper theoretical analysis of why the proposed method works and its limitations;\n\nAdditional experiments on a wider range of datasets and scenes would strengthen the evidence for the method’s generalizability and robustness;\n\nThe description of the linear transformation process and the role of text captions need more detail to fully understand the approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Review: The paper could benefit from a deeper theoretical analysis of why the proposed method works \n\nResponse: Our method predicts scale from language description to ground relative depth estimation to metric scale. Theoretically speaking, our hypothesis is that certain objects (e.g., cars, trees, street signs) with certain observed sizes are frequently associated with specific scene types and sizes  (see L40-44), leading to clusters of these object features in the latent space. The centers of these clusters can then be mapped to a 3D scene by attributing them to depth. By assuming that scene features can be mapped to a scale and shift, our models aim to learn the mapping from object semantics to the corresponding scale and shift through the context of scenes.\n\nReview: …and its limitations;\n\nResponse: We discussed limitations in L277 - 287 in the main paper. At present, the text inputs we evaluate are generated from detection results, but this is to facilitate experiments. Future research could consider more varied language inputs, including text contains background description (see Tables 5, 6 attached) and those generated by humans. Also, factors such as occlusions, rotations, and varying object sizes, which are only partially captured in current language descriptions, can impact RSA's accuracy in predicting scale information. Moreover, while a simple global scale has been effective, it may not always be sufficiently precise for converting relative depth to absolute depth, especially when the predicted relative depth is inaccurate. In such situations, global transformation parameters might fail to adequately scale the depth map as the assumption that reconstruction is up to an unknown scale is violated. Future studies could expand RSA to accommodate finer adjustments and explore the possibility of inferring region-specific or pixel-level scales using language inputs. Finally, although language inputs are highly user-friendly, RSA remains susceptible to manipulation by malicious users who might provide misleading captions to distort predictions.\n\nReview: Additional experiments on a wider range of datasets and scenes would strengthen the evidence for the method’s generalizability and robustness;\n\nResponse: Thank you for the suggestion. We have provided additional experiments on DDAD and VOID in Tables 1, 2 attached. Additionally, in Tables 3, 4, 5, 6 attached, we provide experiments on using 3D object sizes, background together with objects, background only, detector features, coordinates and configuration of objects, camera intrinsics, as well as different architecture design choices. Our conclusions from the main paper still hold.\n\nReview: The description of the linear transformation process and the role of text captions need more detail to fully understand the approach.\n\nResponse: For each input image, RSA predicts two scalars: scale and shift, given the text description. Then, as shown in Figure 2 in the paper, for a relative depth predicted by a frozen depth model, we take its inverse, multiply it with the predicted scale, and add the predicted shift, to obtain the transformed metric depth prediction (see Sec. 3). The role of text captions serves as input for the model to predict scale. The problem we would like to investigate is whether language can be used as input to infer scale. In this way, we could ground relative depth prediction from SOTA Monocular Depth Estimation models, like Midas or Depth Anything, to metric scale. In practical usage, it would be a human-computer interaction scenario, where humans provide short descriptions of the scene to ground model predictions. For experiments, we use derived text from images to simulate description that humans would provide in real-world scenarios.\n\nReview: How does the method perform in real-world scenarios where text captions may not be readily available or may be noisy?\n\nResponse: We do test for robustness to noisy descriptions in Table 5 of the main paper. We note that RSA is trained with augmentations including random swapping and deletion of objects, so our model is robust to noisy descriptions (missing or change in order). We note that our paper is to investigate the use of text captions to infer scale, so forgoing them completely would leave us with no input. Instead, in Tables 5 and 6 attached (see rows marked with \"Background only\"), we present results for the more likely case that no objects are detected and the model is presented only with the background.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ffzn17YLjO", "reviewer": "Reviewer_Nu8S", "summary": "The paper proposes a method for learning the scale of single-view observation of scenes from embeddings of text descriptions of the scenes. The method (referred to as RSA model) is of a simple design of MLP layers and is trained on text embeddings extracted from single images. Evaluation is done on various datasets and combination of in-dataset generalization and few-to-zero-shot settings. The paper is able to demonstrate the superior capabilities of the method when compared with baseline methods in various settings both quantitively and quantitatively, as well as to include ablation study on design choices and statistics of the outputs.", "questions": "Please see above comments for questions raised.", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "[1] The method is simple and novel. The paper illustrates a simple yet effective finding that the scale of a single-view scene can be learned from the text description of the image. The idea is intuitive and not limited to specific domains and is proven to generalize well to unseen domains given the text description is a high-level abstraction of the scene appearance. The findings may be inspiring to the community on the potential of using text prompts as an image representation which is useful to yield high-level scene properties.\n\n[2] Extensive evaluation. Based on the assumption, the paper is able to include relative extensive experiments on abundant datasets to demonstrate the method is applicable to a wide range of modern depth estimation methods to recover the scale, and to outperform methods predicting absolute depth. The paper also includes ablation study that answers to several questions including important design choices of the text description, and analysis of the results across domains.\n\n[3] The paper is well-written and easy to follow. The results are well presented.", "weaknesses": "[1] Robustness of the method. Despite various design choices and sample images are discussed in the paper, it is not clear how well the method generalizes to difficult images, where there is less context or objects present (e.g. an empty corner of a room, or empty section of a street), or when the off-the-shelf object detectors make mistakes. In this case, methods which rely on intermediate representations including RSA may not be as robust as methods that directly operate on the image appearance.\n\n[2] Limited evaluation. Despite evaluation is done on various dataset on different settings, it still feel the scale of evaluation is limited, given the claim that the proposed method is not domain specific. In this case, there should ideally be more datasets involved to represent domains in the wild (e.g. indoor/outdoor/driving, real/synthetic), yet the paper only picks a few to evaluate on.\n\n[3] Simple model design and prompt design. The paper picks a very simple MLP-based model without discussing on alternative choices. Moreover, the prompt design can be further elaborated, to include e.g. coordinates of objects, intrinsics of the camera, configuration of the objects in the scenes, as those properties may add additional information on the scale of the scene and to help further disambiguate the issue (for instance, existing methods for geometric estimation are known to perform worse when the FoV of the camera changes).\n\n[4] Minor questions and suggestions.\n\n(1) What is the method termed RSA? (2) L199, how does the model take CLIP image and text features respectively? What are the dimension of features and layers of the model in both cases?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Review [1]: Robustness of the method. Despite various design choices and sample images are discussed in the paper, it is not clear how well the method generalizes to difficult images, where there is less context or objects present (e.g. an empty corner of a room, or empty section of a street), or when the off-the-shelf object detectors make mistakes. In this case, methods which rely on intermediate representations including RSA may not be as robust as methods that directly operate on the image appearance.\n\nResponse [1]: Indeed, we consider this in Table 5 of the main paper, where we deleted, replaced, and introduced erroneous descriptions. By design of augmentations, our method is robust to typical noise and errors in the text, such as mis-detections, deletions, and swappings. Rows marked with \"Background only\" also show that our model is able to operate only background and without any objects detected. However, as discussed in the limitations (L277-287), by offering flexibility in use through captions, our method also is susceptible to malicious users who may explicitly choose completely incorrect descriptions to alter the scale of the 3D scene. This is also shown in row 8 of Table 5, where one may completely disregard indoor objects present in the image, e.g., coffee table, chair, etc., and provide outdoor objects, e.g., car, traffic light, tree, etc. Our model naturally scales the reconstruction larger, offering user controllability of the scale of the 3D scene relevant to 3D artists, but does not score well on the evaluation protocols under this scenario. In light of the review, we experimented with including background (obtained from panoptic segmentation using MaskDINO) as part of the descriptions and found that it improves results even more (see rows marked with \"Object + background\" in Table 5, 6 attached).\n\nReview [2]: Limited evaluation. Despite evaluation is done on various dataset on different settings, it still feel the scale of evaluation is limited, given the claim that the proposed method is not domain specific. In this case, there should ideally be more datasets involved to represent domains in the wild (e.g. indoor/outdoor/driving, real/synthetic), yet the paper only picks a few to evaluate on.\n\n\nResponse [2]: Thanks for the advice. Apart from those three datasets (NYUv2, KITTI, and SUN3D) we evaluated in the main paper, we now have provided additional zero-shot generalization experiments on DDAD (outdoor) and VOID (indoor) in Tables 1, 2 attached. Our conclusions from the main paper remain valid.\n\nReview [3]: Simple model design and prompt design. The paper picks a very simple MLP-based model without discussing on alternative choices. Moreover, the prompt design can be further elaborated, to include e.g. coordinates of objects, intrinsics of the camera, configuration of the objects in the scenes, as those properties may add additional information on the scale of the scene and to help further disambiguate the issue (for instance, existing methods for geometric estimation are known to perform worse when the FoV of the camera changes).\n\nResponse [3.1]: The RSA model needs to map text features (typically of dimension 1024) to two scalars (scale and shift). This mapping is relatively simple, and using an overly complex model like Transformers might lead to overfitting. Our comparison of different model designs, as shown in Tables 5, 6 attached, indicates that a simple 3-layer MLP is more robust and generalizable when jointly trained on both indoor and outdoor datasets, effectively avoiding overfitting. For fair comparison, We use Transformer or MLP to map input features (dim=1024) to hidden representations (dim=256), then use two separate 3-layer MLPs to map hidden representations to two scalars (scale and shift). For Transformer, we set the number of attention heads to 4.\n\nResponse [3.2]: Regarding prompt design, we conducted experiments with various text prompts, as shown in Tables 5, 6 attached. We incorporated object coordinates, camera intrinsics, and scene configurations into the text descriptions. For object coordinates, we used the center point, e.g., \"a bed located at (124,256) occupying 13.42\\% of the image.\" For object configurations, we divided the image into 9 grids (top left, top, top right; left, center, right; bottom left, bottom, bottom right) and selected the grid where most of the object falls, e.g., \"a bed located in the bottom left occupying 13.42\\% of the image.\" Our findings indicate that overly complex prompts can introduce noise and misguide the model. Additionally, in real-world scenarios, humans typically do not provide highly complex text descriptions. Therefore, we limited the complexity of text descriptions to include only the observed size and type of objects.\n\nReview [4.1]: What is the method termed RSA? \n\nResponse [4.1]: It stands for \"Resolving Scale Ambiguities\" in the paper title.\n\n\nReview [4.2]: L199, how does the model take CLIP image and text features respectively? What are the dimension of features and layers of the model in both cases?\n\nResponse [4.2]: We use CLIP ResNet-50 model, and extract image features and text features using its visual encoder and text encoder respectively. We use the final feature after global pooling, whose dimension is 1024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IwvzFKWqFO", "reviewer": "Reviewer_Sd3q", "summary": "This paper proposed a novel approach to leverage the language inputs to retrieval the scaling factor between relative depth prediction and ground truth metric depth. The task and the approach is novel and interesting.", "questions": "(1), It's better to have more comparison with other works that are trying to retrieve metric depth scaling factor.\n\n(2), As mentioned above, in my understanding, this approach is leveraging prior knowledges from the pre-trained detector and CLIP model. The text generation and encoding is more like a human-designed bootstrapping of \"how to extract prior knowledge\". \nIt would be interesting and greatly helpful if the author could have comparison to direct fusion of detector/CLIP feature into the RSA model to prove the effectiveness of this bootstrapping.", "limitations": "The text description of images could be hard to obtain in autonomous driving or AR scenarios, limits the potential application of this method.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(1), To the best of my knowledge this is the first work to use language (text) as input to retrieve the metric depth scale.\n\n(2), The presentation of this paper is clear and easy to follow.\n\n(3), Authors conducted extensive analysis of the text input design and sensitivities.", "weaknesses": "(1), My primary concern is the practical applicability of this problem. Is there a significant use case for depth prediction using text descriptions as input? For autonomous driving or AR applications, the user might not be always speaking and describing the environment.\n\n(2), For the case of automatically generating text from a detection model, essentially, this is leveraging prior knowledges from the pre-trained detector and CLIP model. What if you just extract the detector/CLIP features as input to the RSA model?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Review (W1): My primary concern is the practical applicability of this problem. Is there a significant use case for depth prediction using text descriptions as input? For autonomous driving or AR applications, the user might not be always speaking and describing the environment.\n\nResponse (W1): Our method predicts the scale for an entire 3D scene based on a text description, as the scale is closely related to the objects and their typical sizes within the scene. In practical use, a human only needs to provide a description once upon entering a new 3D scene. As long as the surroundings do not change dramatically within that scene, there is no need for repeated descriptions.\n\nReview (W2): For the case of automatically generating text from a detection model, essentially, this is leveraging prior knowledges from the pre-trained detector and CLIP model. What if you just extract the detector/CLIP features as input to the RSA model?\n\nResponse (W2): The comparisons are presented in the paper. In Tables 1, 2, 3 in the main paper, the \"Image\" entry under the \"Scaling\" column represents the use of CLIP image features as input for the RSA model. The findings show that predicting scale using images is less robust, especially when trained on both indoor and outdoor datasets simultaneously and when evaluated by zero-shot transfer to other unseen datasets. This is due to the larger domain gap for images compared to text description across different scenes.\n\nReview (Q1): It's better to have more comparison with other works that are trying to retrieve metric depth scaling factor.\n\nResponse (Q1): To the best of our knowledge, we have compared all works or settings that retrieve metric depth scaling factors. If the reviewer has specific works in mind, we are happy to make comparisons. Works producing relative depth like Depth Anything and Midas conduct evaluation by doing linear fit to fit the optimal scale and shift for relative depth and ground truth, or using the median value of depth ground truth to calculate scale to conduct median scaling. We treat both of these settings as oracles of our model, since they use depth ground truth to calculate scale and shift. Also, we compare with a domain adaptation method ZeroDepth[16] in Table 1, 2, 3 in the main paper, which retrieves metric depth scaling factor to adapt a pre-trained depth model to a new domain. Additionally, in Tables 3, 4 attached, we provide a baseline of scaling the reconstruction based on 3D object sizes and their projections onto the image by directly solving for scale specific to each image. We achieve better performance across various base relative depth models, metrics, and datasets compared with ZeroDepth and the scaling baseline.\n\nReview (Q2): As mentioned above, in my understanding, this approach is leveraging prior knowledges from the pre-trained detector and CLIP model. The text generation and encoding is more like a human-designed bootstrapping of \"how to extract prior knowledge\". It would be interesting and greatly helpful if the author could have comparison to direct fusion of detector/CLIP feature into the RSA model to prove the effectiveness of this bootstrapping.}\n\nResponse (Q2): To analyze the effect of prior knowledge from the detector (Detic[48]) we use, we present the experiment results labeled \"Detector feature\" in Tables 5, 6 attached. To fuse all proposal features, for each image processed by the detector, we pass each proposal feature through a 3-layer MLP, then perform average pooling on all proposal features to obtain a global detection feature of the image. This detection feature is then used as input to train our RSA model to predict scale. The results indicate that prior knowledge from the detector does not improve scale prediction and performs worse than using text descriptions as input. When jointly training and evaluating on both indoor and outdoor datasets, the domain gap between indoor and outdoor detection features is larger than that of text descriptions, weakening its generalization ability across different scenes.\n\nReview (L1): The text description of images could be hard to obtain in autonomous driving or AR scenarios, limits the potential application of this method.\n\nResponse (L1): Since our method estimates the scale of an entire 3D scene from a text description. In practical scenarios, a human only needs to describe the scene once upon entering a new 3D environment. This approach is user-friendly and does not require extensive text descriptions. Provided that the surroundings remain relatively unchanged, there is no need for repeated descriptions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to express our sincere gratitude to each reviewer for their thoughtful and constructive feedback on our manuscript. 

We sincerely thank each reviewer for highlighting the strengths of our paper on “novel (R-KH1n)”, “outperforms previous methods (R-ZAYV)”, “simple, novel, generalizable, extensive evaluated and well-written (R-Nu8S)”, “first work to use language to retrieve the metric depth scale, clear with extensive analysis (R-Sd3q)”. 

We have carefully considered and addressed each of your comments and suggestions in our individual responses accordingly.

Meta-review: The paper obtained  2 Weak-Accept, 1 Border-line Accept, and Borderline-Reject. After reading the reviews, rebuttal, and discussions I think the paper addresses an interesting problem and intriguing solution. For these reasons, I think the community can find this work as interesting and thus I recommend acceptance. Nevertheless, please use the given feedback provided by the reviewers to improve the narrative of the paper.
Decision: Accept (poster)