RSA: Resolving Scale Ambiguities in Monocular Depth Estimators through Language Descriptions

Authors: ziyao  zeng
yangchao  wu
hyoungseob  park
daniel  wang
fengyu  yang
stefano  soatto
dong  lao
byung-woo  hong
alex  wong
Venue: NeurIPS_cc_2024_Conference
Published: 2024-04-23 17:22:51

Abstract: We propose a method for metric-scale monocular depth estimation. Inferring depth from a single image is an ill-posed problem due to the loss of scale from perspective projection during the image formation process. Any scale chosen is a bias, typically stemming from training on a dataset; hence, existing works have instead opted to use relative (normalized, inverse) depth. Our goal is to recover metric-scaled depth maps through a linear transformation. The crux of our method lies in the observation that certain objects (e.g., cars, trees, street signs) are typically found or associated with certain types of scenes (e.g., outdoor). We explore whether language descriptions can be used to transform relative depth predictions to those in metric scale. Our method, RSA , takes as input a text caption describing objects present in an image and outputs the parameters of a linear transformation which can be applied globally to a relative depth map to yield metric-scaled depth predictions. We demonstrate our method on recent general-purpose monocular depth models on indoors (NYUv2, VOID) and outdoors (KITTI). When trained on multiple datasets, RSA can serve as a general alignment module in zero-shot settings. Our method improves over common practices in aligning relative to metric depth and results in predictions that are comparable to an upper bound of fitting relative depth to ground truth via a linear transformation. Code is available at: https://github.com/Adonis-galaxy/RSA.

Main paper: 
 claims: 
 Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes]Justification: We claim in the abstract and introduction that we proposed a novel formulation of the relative to metric depth transfer, and verify the feasibility of inferring scale from language and as a general alignment module.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
related work: 
 Monocular depth estimation using metric depth. Metric depth models learn to infer pixel-wise depth in metric scale (i.e. meters) by minimizing loss between depth predictions and ground-truth depth maps [2, 12, 26, 35, 55, 71, 79] . Each model typically applies to only one data domain in which it is trained, with similar camera parameters and object scales. Specifically, DORN [12] leverages a spacing-increasing discretization technique. AdaBins [2] partitions depth ranges into adaptive bins. NeWCRFs [71] uses neural window fully-connected CRFs to compute energy. When ground-truth depth is not available, self-supervised approaches [4, 11, 23, 27, 30, 38, 41, 51, 52, 53, 59, 61, 70, 74, 80, 81, 84, 86] rely on geometric constraints, where scale is attributed through lidar [10, 37, 40, 57, 56, 58, 63, 68] , radar [47] , binocular images [14, 17, 16] , or inertials [20, 54] . However, models that predict metric depth are limited to specific datasets or scenes, and sensors [60] ; thus, they do not generalize well. RSA aims to serve as a general alignment module that can predict metric depth based on relative depth across different domains.Monocular depth estimation using relative depth. Trained across different domains, self-supervised depth estimators trained with multi-view photometric objects produce up-to-scale predictions that are linearly correlated with their absolute depth values across the domain [8, 5, 20, 21, 54, 62, 64] , thus requiring scaling of their depth prediction. However, due to the scale-ambiguity of multi-view photometric objective, such models are normally evaluated by aligning predictions to ground-truth at test time (typically median-scaling [16, 17, 86] ), at the expense of practicality since ground-truth might not be feasible during real-world application. To enable generalization across different scenes, some (semi-) supervised depth models trained with single images adopt image-level normalization techniques to generate affine-invariant depth representations (i.e. relative depth) [24, 43, 44, 67, 73] . HND [73] hierarchically normalizes the depth representations with spatial information and depth distributions. Depth Anything [67] learns from large-scale automatically annotated data. DPT [43] leverages vision transformers using a scale-and shift-invariant trimmed loss. MiDas [44] mixes multiple datasets with training objectives invariant to depth range and scale. Marigold [24] associates fine-tuning protocol with a diffusion model. However, by definition, relative depth is scaleless, which limits their applications that require metric scaled reconstruction. Our proposed RSA addresses this limitation by grounding relative depth into a metric scale, enabling metric scale 3D reconstruction.Relative to metric depth transfer. To transform the predicted relative depth into metric depth for evaluation and real-world application, ZoeDepth [3] fine-tunes a metric bins module on each metric depth dataset to output metric depth. Depth Anything [67] follows ZoeDepth [3] using a decoder where a metric bins module computes per-pixel depth bin centers that are linearly combined to output the metric depth. MiDas [44] and Marigold [24] use linear fit to align predictions and ground truth in scale and shift for each image in inverse-depth space based on the least-square criterion before measuring errors. DPT [43] fine-tunes a global scale and shift on metric depth datasets. DistDepth [62] conducts transfer by leveraging left-right stereo consistency to integrate metric scale into a scale-agnostic depth network. ZeroDepth [21] achieves transformation using input-level geometric embedding to learn an indoor scale prior over objects via a variational latent representation. However, metric decoders are limited to specific datasets, and aligning scale and shift of predictions requires ground truth during test time. RSA produces scale using text to transfer relative depth to metric depth across domains and does not require ground truth during test time.Language model for monocular depth estimation. Vision-Language models [6, 32, 33, 39, 42, 49] acquire a comprehensive understanding of languages and images through pre-training under diverse datasets, thus forming an effective baseline for downstream tasks [34, 65, 69, 76, 78, 79, 88, 66, 50] . Typically, CLIP [42] conducts contrastive learning between text-image pairs, empowering various tasks like few-shot image classification [13, 75, 77, 85] , image segmentation [45, 83] , object detection [45, 87] , and 3D perception [22, 76, 79, 88] . In light of their emerging ability, some works [1, 22, 79, 82] have tried to apply vision-language models for monocular depth estimation. WorDepth [72] learned the distribution 3D scenes from text captions. DepthCLIP [79] leverages the semantic depth response of CLIP [42] with a depth projection scheme to conduct zero-shot monocular depth estimation. Hu et al. [22] extends DepthCLIP with learnable prompts and depth codebook to narrow the depth domain gap. Auty et al. [1] modifies DepthCLIP [79] using continuous learnable tokens in place of discrete human-language words. In contrast, RSA uses language to directly predict scale, which serves as an explicit scaling constraint by transforming relative depth into metric scale.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: We create a separate section to discuss the potential limitations, including lack of exploring human-generated descriptions, lack of certain factors in current descriptions, and may not be expressive enough especially when the relative depth is inaccurate.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
method: 
 We consider a dataset D = {I (n) , t (n) , y * (n) } N n=1 with N samples of synchronized RGB image, text descriptions, depth maps, where I : Ω ⊂ R 2 → R 3 denotes an image, y * : Ω ⊂ R 2 → R + the ground-truth metric depth map, t a text description of the image, and Ω the image space. We assume access to a pretrained monocular depth estimation model h θ for the purpose of learning the parameters to predict the transformation between relative and metric depth. Given an RGB image, a monocular depth estimation model predicts inverse relative depth y : Ω ⊂ R 2 → R + using a parameterized function h realized as a neural network, i.e., y := h θ (•). To recover metric-scale from (scaleless) inverse relative depth, we consider a global linear transformation through the use of a language description pertaining to the image of the 3D scene. Given the text description t of an image as input, our method, RSA, predicts a pair of scalars denoting the scale and shift parameters of the transformation: (α, β) = g ψ (t) ∈ R 2 . The metric depth prediction is obtained by ŷ = 1/(α • y + β).To infer the parameters of the linear transformation to align relative depth to metric scale, we employ the existing pretrained CLIP text encoder [42] as a feature extractor. Having been trained on a large scale dataset, CLIP offers an latent space suitable to preprocess object-centric text descriptions. We note that CLIP text encoder is frozen within RSA. Given text descriptions t = {t 1 , t 2 , ...}, we first encode them into text embeddings and feed them into a 5-layer shared multi-layer perceptron (MLP) to project them into k = 256 hidden dimensions followed by two separate sets of 5-layer MLPs, one serves as the output head ψ α : R k → R + for scale parameter α and the other as the output head ψ β : R k → R + for shift β parameter. Scale and shift are assumed to be positive in favor of optimization. For ease of notation, we refer to ψ as the parameters for the shared MLP as well as the output heads, where (α, β) = g ψ (t).Optimizing RSA involves minimizing a supervised loss with respect to ψ, which requires a forward pass of a given training image I (n) through the monocular depth model to yield y (n) = h θ (I (n) ). To ensure that the monocular depth model does not drift and update its parameters during the training of RSA, we freeze θ while optimizing for ψ. Hence, training entails minimizing an L1 loss:EQUATIONwhereŷ(n) = 1/(α (n) • y (n) + β(n) )denotes the predicted metric-scale depth aligned from relative depth y (n) , x ∈ Ω denotes an image coordinate, and M : Ω → {0, 1} denotes a binary mask indicating valid coordinates in the ground truth depth y * with values greater than zero.Text prompt design. To test our hypothesis, we require text descriptions to be paired with images. As standard benchmarks do not provide the text description of each image, we extend existing datasets by associating each image with several text descriptions. To achieve this, we propose to use off-the-shelf models to generate different kinds of text. First, we considered structured text, which adheres to a 1 : Quantitative results on NYUv2. RSA (yellow), especially when trained with multiple datasets, generalizes better than using images to predict the transformation parameters. Global refers to optimizing a single scale and shift for the entire dataset (same scale and shift for every sample). Image denotes predicting scales and shifts using images. Red denotes scaling that uses ground truth. Median indicates scaling using the ratio between median of depth prediction and ground truth. Linear fit denotes optimizing scale and shift to fit to ground truth for each image. DA refers to domain adaptation. ZoeDepth performs per-pixel refinement.certain template. We use a panoptic segmentation model MaskDINO [31] to extract the significant objects and background in the image. For an input image I, the segmentation model returns a set of B object and background instances {n (i) , c (i) } B i=1 , where c (i) denotes the class of the object or background, and n (i) denotes the number of instances of the object or background. Using the set of instances, a structured caption for an image I can be obtained: " An image with n (1) c (1) , n (2) c (2) , ..., n (B) c (B) . " We will shuffle the order of instances to produce 5 different structured captions for each image. Then we consider the natural text, where the text doesn't adhere to certain templates and is closer to human descriptions, we use two visual question-answering models LLaVA v1.6 Vicuna and LLaVA v1.6 Mistral [36] . For each model, we prompt it with the input image and a prompt, asking the model to describe the image. For each model, we provide 5 different prompts to produce different natural captions. During training, in each iteration, for a given image, we randomly select one caption from those 15 captions to predict scale and shift.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments: 
 Datasets. We present our main result on three datasets: NYUv2 [46] and VOID [58] for indoor scenes, and KITTI [15] for outdoor scenes. NYUv2 contains images with a resolution of 480×640 where depth values from 1 × 10 -3 to 10 meters. We follow [29, 35, 79] for the dataset partition, which contains 24,231 train images and 654 test images. VOID contains images with a resolution of 480×640 where depth values from 0.2 to 5 meters. It contains 48,248 train images and 800 test images following the official splits [58] . KITTI contains images with a resolution of 352×1216 where depth values from 1 × 10 -3 to 80 meters. We adopt the Eigen Split [9] consisting of 23,488 training images and 697 testing images. Following [2, 71] , we remove samples without valid ground truth, leaving 652 valid images for testing. We also report zero-shot generalization results on SUN-RGBD [48] , which contains 5050 testing images, and DDAD [19] , which contains 3950 validation images. Table 2 : Quantitative results on KITTI Eigen Split. RSA (yellow), especially when trained with multiple datasets, generalizes better than using images to predict the transformation parameters. Please refer to Table 1 for more details about notations.Depth models. For DPT [43] , we use DPT-Hybrid fine-tuned for NYUv2 and KITTI respectively, with 123M parameters. We used the one fine-tuned on NYUv2 for VOID. For MiDas [44] , we use MiDaS 3.1 Swin2_large-384 with 213M parameters. For DepthAnything [67] , we use Depth-Anything-Small with 24.8M parameters. Different from our setting, DepthAnything evaluates using ZoeDepth [3] 's depth decoder that is separately fine-tuned on NYUv2 and KITTI to produce a pixel-wise scale, and MiDas evaluates by aligning prediction with ground truth in scales and shifts. We re-implement several baselines aligning with the setting of predicting a global scale and shift for scaling relative depth, provided in Table 1 and 2 .Hyperparameters. We use the Adam [25] optimizer without weight decay. The learning rate is reduced from 3 × 10 -5 to 1 × 10 -5 by a cosine learning rate scheduler. The model is trained for 50 epochs under this scheduler. We run our experiment on GeForce RTX 3090 GPUs, with 24GB memory. For reference, if using a single GPU, the training time for RSA with DepthAnything on jointly NYUv2, KITTI, and VOID for 50 epochs takes 57 hours..We follow [7, 35, 79] to evaluate using mean absolute relative error (Abs Rel), root mean square error (RMSE), absolute error in log space (log 10 ), logarithmic root mean square error (RMSE log ) and threshold accuracy (δ i ).Quantitative results. We show results on NYUv2 in Table 1 , KITTI in Table 2 , and VOID in Table 3 , where we improve over baselines across all evaluation metrics and approach the performance of using ground truth for scaling. Following DPT, we optimize the scale and shift for the "Global" scaling baseline over the training set and use it for evaluation (i.e., same scale and shift for all test samples). We obtain the "Image" baseline by substituting CLIP text features with CLIP image features. Following [44] , we perform a linear regression to find the scale and shift that minimizes the least-square error between the ground truth metric depth and the predicted metric depth. Additionally, we also test median scaling [17] , a common practice for evaluation, which uses the ratio between the median of depth prediction and ground truth as the scaling factor. Although RSA trained on a single dataset may achieve slightly better performance than the unified model, the difference is minimal. In some metrics, the unified model even outperforms, demonstrating the generalizability of using language as input, given the narrow domain gap for language. Additionally, RSA models consistently outperform "Image" baselines in both in-domain and cross-domain scenarios and are comparable with existing methods under various settings. Our method significantly narrows the performance gap to that of using ground truth for scaling, validating the effectiveness of using language instead of the input image to predict scale.To examine our scale predictions in more detail, Figure 3 shows a curve fitting plotted for our predicted (inverse) scale against the median of the ground truth. The trend line shows that the scale inferred from text descriptions matches well with median scaling, which is a robust estimator.Qualitative comparisons. We present representative visual examples comparing RSA with the baseline method on the NYUv2 and KITTI datasets in Figure 4 and Figure 5 , respectively, to highlight the benefits of RSA. The error maps illustrate the absolute relative error. Unlike the original DPT, which uses a fixed scale and shift, RSA enhances accuracy uniformly across the image without altering the structure or fine details of the depth map. This improvement is evidenced by the darker areas in the error maps, indicating better scaling and reduced errors.Zero-shot Generalization. Considering the smaller domain gap in language descriptions across various scenes, we perform a zero-shot transfer experiment to demonstrate RSA's generalization ability. We evaluate the models on the Sun-RGBD [48] and DDAD [19] without fine-tuning. As shown in Table 4 and Table 5 , RSA achieves superior results compared to baselines, existing methods, and ground truth scaling. This suggests that language descriptions offer a viable option for relative to metric transfer when generalizing across diverse data domains. Note that a single global scale and shift are ineffective for both indoor and outdoor settings. Therefore, for the "Global" model, we fit it only to NYUv2 to obtain a reasonable global scale and shift for the zero-shot Sun-RGBD evaluation, and fit it to KITTI for DDAD evaluation.Prompt design for input text. In Table 6 , we investigate different designs of RSA text prompts in training and how they affect the performance. Here, to make the experiment more controllable, we use only structured text and only produce one caption for each image to train each model. In the 1st and 2nd rows of Table 6 , we use an object detector Detic [87] to produce only foreground objects in images and form input text using only foreground objects. We observe that if the input text only specifies the types of objects and their numbers, RSA can still accurately predict the scale for indoor scenes, as these spaces are typically filled with various pieces of furniture. However, this approach performs poorly for outdoor scenes, which tend to be more open and sparse. For instance, parking lots of different sizes may contain varying numbers of cars: a small lot may be crowded, while a large lot may appear empty. In the 3rd and 4th rows of Table 6 , we use text formed using segmentation results; we observe that after including background classes, the model works better especially for outdoor scenes, since different backgrounds (like wall, sidewalk, highway, road, sky, house, building) can reflect different scales.Sensitivity to different text input. To demonstrate the impact of various text inputs on scale prediction, we utilize a trained RSA model based on the DepthAnything model. By altering the text input, we observe changes in scale and shift predictions. We provide the experiment results in Table 7 . From top to bottom, we provide text descriptions of scenes from small to large, and our model can properly predict the corresponding scale and shift. This shows promise that our method is able to manipulate the scale of 3D scenes with ease. Prompt NYUv2 KITTI "An image with c (1) , c (2) , c (3) , ... " with Object Detection Results 0.106 0.070 "An image with K (1) c (1) , K (2) c (2) , ... " with Object Detection Results 0.101 0.068 "An image with c (1) , c (2) , c (3) , ... " with Panoptic Segmentation Results 0.102 0.063 "An image with K (1) c (1) , K (2) c (2) , ... " with Panoptic Segmentation Results 0.100 0.061Table 6 : Different prompt design for RSA. Absolute relative errors (Abs Rel) reported. RSA models are trained using cross-datasets with the DPT model. For one given image, c (i) is the class of a detected or segmented instance, K (i) is the number of all instances belonging to c (i) . By using segmentation results, the text includes background, which improves scale predication, especially for outdoors.A room with a refrigerator, a A busy highway filled with cars, with a blue and white sign on the right side. 0.0067 0.0045Table 7 : Sensitivity study to different text input. We show the inverse scale and shift here; a smaller value indicates a larger scene. From top to bottom, we describe scenes from small to large scale. Results show that we could control the scale of a scene by providing different text descriptions, to better manipulate a 3D scene. 
discussion: 
 Conclusion. We present the first study exploring whether language, as an additional input modality, can resolve the scale ambiguity in monocular depth estimation, an issue particularly relevant in the context of the recent trend towards large-scale mixed dataset training. We propose a framework, RSA, which learns to convert a scaleless (relative) depth map to metric depth using language descriptions as input. RSA utilizes the pre-trained CLIP encoder, and maps a language description of the image to scale and shift factors that transform the relative depth to metric depth. RSA is validated through extensive experiments on three benchmark datasets and three pre-trained relative depth models, demonstrating significant promise by drastically closing the gap between depth estimation accuracy and its upper bound (that relies on ground truth), validating the hypothesis that language carries valuable scale information that could be used to enhance depth estimation. Moreover, we demonstrate that a unified model trained on both indoor and outdoor datasets with diverse scene compositions generalizes across both scenarios, highlighting the robustness of language information in inferring scale. Finally, a zero-shot transfer experiment shows that the minimal domain gap of language description across scenes further generalizes to unseen data domains, without additional training. The generalizability of RSA stems of our choice of modality, language, which is invariant to nuisance variability that are present in images from lighting conditions and occlusions to specular reflectance and object deformations. Our results demonstrate that RSA is a viable choice to support relative to metric scale alignment for general-purpose monocular depth estimators.Limitations and future work. We assume that the estimated 3D scene is up to an unknown scale. Although a simple global scale has proven effective, it may not always be sufficiently expressive for converting relative depth to absolute depth, especially when the relative depth is inaccurate. In such cases, global scaling may not adequately recover a high-fidelity metric-scaled depth map due to the presence of outliers. To address this, one may need to refine the relative depth outputted by general-purpose monocular depth estimators. Future research may include extending RSA to handle finer adjustments to also refine depth estimates and investigate the potential of inferring region-wise or even pixel-wise scales using language input. Lastly, while language boasts high ease of use, RSA is also vulnerable to malicious users who may choose captions to steer predictions incorrectly.NeurIPS Paper Checklist 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes]Justification: We have stated sufficient information for computer resources in the hyperparameter section, including the type of compute worker, memory, and time of execution.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Justification: We have carefully reviewed the NeurIPS Code of Ethics and will obey it.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: We discussed that our method could potentially enhance safety in autonomous systems and benefit augmented reality and virtual reality, but might lead to potential job losses in sectors like transportation.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: As a model for Monocular Depth Estimation, our model doesn't have a high risk for misuse.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: We have properly cited all existing assets used in the paper.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [Yes]Justification: We release and document our code well.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: Our paper doesn't involve crowdsourcing or research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: Our paper doesn't involve crowdsourcing or research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
: 
 1 Introduction 3-dimensional (3D) reconstruction from images is an ill-posed problem due to the loss of a dimension through perspective projection during the image formation process: Any point along the ray of projection can yield the same image coordinate. This extra degree of freedom is often addressed by using multiple images of the same scene, such as stereo or video. While an additional image (assuming co-visible sufficiently exciting textures across both images) allows one to triangulate unique points in space, a scale ambiguity exists with the absence of camera calibration, measurements from an additional sensor (e.g., range, initial), or a strong prior (e.g. a supervised training set). One may argue that, such additional information should already be available during data collection. Still, modern large-scale training [44, 67] often utilizes data from diverse sources with drastically diverse setups, making resolving the scale ambiguity issue crucial.When it comes to monocular depth estimation, which predicts a dense depth map from a single image, the problem is also ill-posed in that one cannot measure the distance from the camera from a single view. Hence, to make inference possible, one must rely on the existence of a training set. While one option is to "bake in" an additional bias of scale by training on a number of different datasets (indoors and outdoors) and attributing depth to pixel intensities, these dataset-specific biases come at the cost of generalization, limiting model transfer from one domain (indoor) to another (outdoor), let alone mixing multiple data sources. Existing monocular depth methods resort to predicting relative (normalized, inverse) depth to factor out the scale biases, but leave behind practical utility, as a trade-off, in downstream applications in spatial tasks such as manipulation, planning, and navigation.We consider whether an additional modality can be used to resolve the scale ambiguity in single-image 3D reconstruction, i.e., transforming scaleless relative depth to metric depth. One might observe that natural (including man-made) scenes do not occur by chance, but rather by design, with the regularity of object-scene co-occurrences [18, 28] : Certain scenes (e.g., outdoors) are composed of certain categories of objects (e.g., cars, trees, buildings) and associated with a certain order of magnitude in scale (e.g. tens of meters). Hence, we hypothesize that language, in the form of text captions or descriptions, can be used to infer the scale of the 3D scene and to transform relative depth to metric depth. The choice of language also has practical value in that it does not require costly data acquisition with an additional synchronized and calibrated sensor (e.g., lidar, time-of-flight). With publicly available pre-trained panoptic segmentation or object detection models, image captioners, and vision-language models, one can automate the data acquisition, training, and inference process. Nonetheless, they are not a necessary part of the work, but may facilitate ease of use, to simulate the language description provided by human users in practical scenarios.To test the feasibility of our hypothesis, we consider monocular depth estimation, where a strong prior is necessary for inference; this prior may come from an image, or an independent modality such as language. Specifically, we consider monocular depth models belonging to the general-purpose, relative depth estimation paradigm to control for side effects such as the scale learned along with shapes in existing works that focus on predicting metric depth for a specific dataset. To this end, we assume access to a pre-trained scaleless monocular depth estimation model; aside from a set of images, we also assume text captions describing them. As the standard procedure to factor out scale is to normalize through a linear transformation, we propose to learn a parameterized function that predicts the parameters of a linear transformation based on the text description. Applying them to a relative depth map and inverting its values will transform it to a metric-scaled depth map to resolve the scale ambiguity. We term our method RSA, as an acronym for "Resolving Scale Ambiguities".In one mode, like existing works that also transform relative to metric depth, albeit with images and domain-specific scaling [3] , we train specific models of RSA for specific datasets (e.g., NYUv2 [46] , KITTI [15] , VOID [58] ). In another mode, when trained on multiple datasets across different domains, RSA generalizes well and can not only handle images sampled from the datasets it was trained on, but also those from novel datasets in a zero-shot manner. The use of language, which is invariant to illumination, object orientation, occlusion, scene layout, etc., many of the nuisance variables that vision algorithms are sensitive to, demonstrates a promising avenue for general-purpose relative to metric scale recovery to complement the growing works in monocular depth estimation. We evaluate our method on indoor and outdoor benchmarks, where we improve over common practices in aligning relative depth to metric scale. We show that using RSA is comparable to matching relative depth via a linear transformation to the ground truth, or scaling using the median value of the ground truth, both of which are considered oracle relative to metric recovery methods.Our contributions are as follows: (i) We proposed a novel formulation of the relative to metric depth transfer. (ii) We demonstrate the feasibility of inferring scale from language and as a general alignment module. (iii) We performed extensive experiments to validate performance on indoor and outdoor domains, sensitivity to text caption, and zero-shot generalization. To the best of our knowledge, we are the first to use language as a means of relative to metric depth alignment. 
 
 Appendix:  


Decision: The paper obtained  2 Weak-Accept, 1 Border-line Accept, and Borderline-Reject. After reading the reviews, rebuttal, and discussions I think the paper addresses an interesting problem and intriguing solution. For these reasons, I think the community can find this work as interesting and thus I recommend acceptance. Nevertheless, please use the given feedback provided by the reviewers to improve the narrative of the paper.

Reviews: review: paper_summary: paper_summary: This paper tackles the challenge of scale ambiguity in monocular scenes. The idea is to leverage the priors in LLMs to estimate the scales of an image. Evaluation is performed on NYU and KITTI datasets in comparison to baseline heuristics.
strength_weakness: strength_weakness: - The proposed solution seems unreasonable, at least how it is presented in the paper. Specifically, the LLM does not utilize the image but only text alone. Admittedly, the text is generated from image input, but is limited to existing object detection and only tested in 2 domains/datasets. Since the image data is not used (or the relative depth prediction) the LLM prediction seems to learn dataset biases rather than from real image data.
- This concern is also shown in the evaluation, were only a limited evaluation on 2 datasets is performed. Results are insignificantly improved. The comparison is limited so simple baselines, while work that explicitly uses object scales to refine relative depth are not shown.
- The value of extracting text from images to then compute scales from text seems limited in real applications. A comparison to finetuned GPT4v (or equivalent) with image data alone to directly estimate scales is suggested.
questions: questions: see above
limitations: limitations: see above.

- Only few datasets
- Code not yet available
- Insignificant improvements
- Soundness of solution is not convincing
paper_summary: paper_summary: The paper proposes a method for metric-scale monocular depth estimation called RSA. RSA aims to recover metric-scaled depth maps through a linear transformation, leveraging language descriptions of objects in scenes. The proposed method is validated on multiple standard datasets.
strength_weakness: strength_weakness: The paper could benefit from a deeper theoretical analysis of why the proposed method works and its limitations;

Additional experiments on a wider range of datasets and scenes would strengthen the evidence for the method’s generalizability and robustness;

The description of the linear transformation process and the role of text captions need more detail to fully understand the approach.
questions: questions: How does the method perform in real-world scenarios where text captions may not be readily available or may be noisy?
limitations: limitations: N/A
paper_summary: paper_summary: The paper proposes a method for learning the scale of single-view observation of scenes from embeddings of text descriptions of the scenes. The method (referred to as RSA model) is of a simple design of MLP layers and is trained on text embeddings extracted from single images. Evaluation is done on various datasets and combination of in-dataset generalization and few-to-zero-shot settings. The paper is able to demonstrate the superior capabilities of the method when compared with baseline methods in various settings both quantitively and quantitatively, as well as to include ablation study on design choices and statistics of the outputs.
strength_weakness: strength_weakness: [1] Robustness of the method. Despite various design choices and sample images are discussed in the paper, it is not clear how well the method generalizes to difficult images, where there is less context or objects present (e.g. an empty corner of a room, or empty section of a street), or when the off-the-shelf object detectors make mistakes. In this case, methods which rely on intermediate representations including RSA may not be as robust as methods that directly operate on the image appearance.

[2] Limited evaluation. Despite evaluation is done on various dataset on different settings, it still feel the scale of evaluation is limited, given the claim that the proposed method is not domain specific. In this case, there should ideally be more datasets involved to represent domains in the wild (e.g. indoor/outdoor/driving, real/synthetic), yet the paper only picks a few to evaluate on.

[3] Simple model design and prompt design. The paper picks a very simple MLP-based model without discussing on alternative choices. Moreover, the prompt design can be further elaborated, to include e.g. coordinates of objects, intrinsics of the camera, configuration of the objects in the scenes, as those properties may add additional information on the scale of the scene and to help further disambiguate the issue (for instance, existing methods for geometric estimation are known to perform worse when the FoV of the camera changes).

[4] Minor questions and suggestions.

(1) What is the method termed RSA? (2) L199, how does the model take CLIP image and text features respectively? What are the dimension of features and layers of the model in both cases?
questions: questions: Please see above comments for questions raised.
limitations: limitations: N/A
paper_summary: paper_summary: This paper proposed a novel approach to leverage the language inputs to retrieval the scaling factor between relative depth prediction and ground truth metric depth. The task and the approach is novel and interesting.
strength_weakness: strength_weakness: (1), My primary concern is the practical applicability of this problem. Is there a significant use case for depth prediction using text descriptions as input? For autonomous driving or AR applications, the user might not be always speaking and describing the environment.

(2), For the case of automatically generating text from a detection model, essentially, this is leveraging prior knowledges from the pre-trained detector and CLIP model. What if you just extract the detector/CLIP features as input to the RSA model?
questions: questions: (1), It's better to have more comparison with other works that are trying to retrieve metric depth scaling factor.

(2), As mentioned above, in my understanding, this approach is leveraging prior knowledges from the pre-trained detector and CLIP model. The text generation and encoding is more like a human-designed bootstrapping of "how to extract prior knowledge". 
It would be interesting and greatly helpful if the author could have comparison to direct fusion of detector/CLIP feature into the RSA model to prove the effectiveness of this bootstrapping.
limitations: limitations: The text description of images could be hard to obtain in autonomous driving or AR scenarios, limits the potential application of this method.

Comments: title: Thank the reviewer for recommending acceptance~
Thank the reviewer for recommending acceptance~
Thanks authors for the response
Thanks Reviewer Sd3q for further question~
Thanks authors for the response
comment: Thank you very much for your thoughful response. We are grateful that you acknowledge our paper is "a useful investigation". Here are our response to your concerns:


**Question: "Image is also important (Why not using image to predict scale?)"**

As the reviewer rightfully highlights, images are indeed important. However, predicting depth from a single image is an ill-posed problem, primarily due to scale ambiguity -- meaning that a single image does not afford scale as it is lost during the projection from 3D scene to 2D image. This issue has long been recognized in classic vision research and is evident in modern methods, which often focus on estimating relative depth rather than absolute metric depth. In our study, we explore the potential of using language as an input modality to address this scale ambiguity. This approach not only has practical applications but also provides scientific insights into the challenges of monocular depth estimation.

To maintain control over experimental variables, we have decomposed the process of estimating metric depth into two steps: first, estimating relative depth using state-of-the-art pre-trained image models, and second, determining the corresponding metric-scale transformation using descriptions of objects, which tend to exhibit similar metric-scale sizes for a given object category, within the 3D scene.

We discuss in detail in L30-38 why directly using images to predict scale tends to yield suboptimal results. The key issue is that image-based depth predictions can hinder generalization across different datasets. In contrast, as demonstrated in our experiments, language-based scale predictions generalize more effectively across datasets. There is solid motivation behind this attempt: certain scenes are composed of certain categories of objects and associated with a certain scale, making language a robust modality of input for scale prediction, as explained in L39-50.



**Question: "Human/naturally generated captions are not investigated."**

Response: Per the reviewer's request, we ask several humans to describe each image in the test set of NYUv2, without the requirement of structural text templates. Each human is asked to "describe this scene in one sentence." Then we evaluate our RSA model on this human-described test set without further finetuning. The results show that our model trained with automatically generated language description can generalize well to natural human description, and achieve a comparable performance, since attributes that are essential for scale prediction (objects' types and numbers) are usually properly reflected within natural human description.

Examples of human description:

"A small bathroom sink area with a towel, soap, and toiletries neatly arranged around the countertop."

"A home office setup with a laptop, printer, and office supplies on a wooden desk, with a map of the United States hanging on the wall."

“A colorful classroom with small tables and chairs set up for children's activities, toys on shelves, and an alphabet rug on the floor.”


**Comparision with human described NYUv2 test set.**

|Test set|Depth Model|Scaling|Training Dataset|$\delta<1.25 \uparrow$|$\delta<1.25^{2} \uparrow$|$\delta<1.25^{3} \uparrow$|Abs Rel $\downarrow$|$\log _{10} \downarrow$|RMSE $\downarrow$|
|-|-|-|-|-|-|-|-|-|-|
|Human described NYUv2 test set|DepthAnything|RSA|NYUv2, KITTI|0.748|0.970|**0.996**|0.150|0.068|**0.487**|
|Original NYUv2 test set|DepthAnything|RSA|NYUv2, KITTI|**0.776**| **0.974**|**0.996**|**0.148**| **0.065**| 0.498 |


**Question: "It seems more natural to estimate scale directly from image using GPT4 / VLM, rather than from just text."**

Response: Per the reviewer's request, we finetune the visual encoder from the LLaVA-1.6 7B model on NYUv2 and KITTI, with Depth Anything, to predict the scale and shift given monocular images as input. The results are provided in the table below, where RSA models surpass predict scale using images from a foundation model, owing to the robustness of language which is invariant to nuisance (e.g., object orientation, scene layout) that vision algorithms are sensitive to.

**Evaluation on NYUv2.**

|Scaling|Depth Model|Training Dataset|$\delta<1.25 \uparrow$|$\delta<1.25^{2} \uparrow$|$\delta<1.25^{3} \uparrow$|Abs Rel $\downarrow$|$\log _{10} \downarrow$|RMSE $\downarrow$|
|-|-|-|-|-|-|-|-|-|
|LLaVA-1.6|DepthAnything|NYUv2, KITTI |0.702|0.938|0.990|0.186|0.078|0.589|
|**RSA**|DepthAnything|NYUv2, KITTI|**0.776**| **0.974**|**0.996**|**0.148**| **0.065**| **0.498** |

**Evaluation on KITTI Eigen Split.**

|Scaling|Depth Model|Training Dataset|$\delta<1.25 \uparrow$|$\delta<1.25^{2} \uparrow$|$\delta<1.25^{3} \uparrow$|Abs Rel $\downarrow$|$\text{RMSE}_{\log} \downarrow$|RMSE $\downarrow$|
|-|-|-|-|-|-|-|-|-|
|LLaVA-1.6|DepthAnything| NYUv2,KITTI|0.685|0.930|0.978|0.183|0.215|4.878|
|**RSA**|DepthAnything|NYUv2,KITTI|**0.756**|**0.956**|**0.987**|**0.158**|**0.191**|**4.457**|
We are grateful to the reviewer for recommending acceptance. We value the constructive discussion and will update the final version of our manuscript accordingly.
We thank the reviewer for recommending acceptance. We appreciate the constructive discussion and will incorporate it into our final version.
Dear Authors:

Thank you for the prompt response!

The overall idea of this paper looks interesting, while my major concern is still the improvement of user-friendly text input is marginal compared to CLIP features. I would like to remain my rating as boarderline accept.
I would like to thank the authors for addressing all my concerns and solving most of them. Given the additional results provided in the rebuttal on more datasets, the generalization of the method is more convincing, and I particularly like the overall idea of the paper and the effectiveness of it indicated by the experiments. I would keep my original rating of Accept in acknowledgement of the rebuttal and discussions.
Thank you very much for your question!

**Detector Knowledge.**

 For the detector knowledge, please see Tables 5 and 6 in the PDF file attached to the "Author Rebuttal by Authors”. We have also put the results in the tables below for your convenience.

**User-friendly Text.** 

While there exists a trade-off between user friendliness and performance, user-friendly prompts still improve when using CLIP image features. Due to space limitations, we only showed the AbsRel metric for DPT in Table 4 in the main paper. Below, we provide experiments for all depth models tested, where User-friendly-1 denotes input with "An image with $\mathbf{c^{(1)}}$, $\mathbf{c^{(2)}}$, ..." and User-friendly-2 denotes "An image with $\mathbf{K^{(1)}}$ $\mathbf{c^{(1)}}$, $\mathbf{K^{(2)}}$ $\mathbf{c^{(2)}}$, ...", and RSA (Ours) denotes "An image with $\mathbf{c^{(1)}}$ occupied $\mathbf{p^{(1)}}$ of image, ...". For one given image, $\mathbf{c^{(i)}}$ is the class of a detected instance, $\mathbf{K^{(i)}}$ is the number of all instances belonging to $\mathbf{c^{(i)}}$, $\mathbf{p^{(i)}}$ is the bounding box area of the instance with respect to the image.

Naturally, with a more descriptive text caption (e.g., including the percentage of occupied space in the image), one can improve performance, but at the expense of ease of use. While we reported the overall best as our method [RSA (Ours)], we note that user-friendly prompts are comparable when considering all metrics. All improve over CLIP image features thanks to the text being invariant to nuisances  (e.g., illumination, occlusion, viewpoint) present in images.

**Table 5: Additional experiments on NYU-Depth-v2.**
| **Models**| **Scaling**| **Dataset**| $\delta<1.25 \uparrow$|$\delta<1.25^{2} \uparrow$|$\delta<1.25^{3} \uparrow$|Abs Rel $\downarrow$|$\log _{10} \downarrow$|RMSE $\downarrow$ |
|---|---|---|---|---|---|---|---|---|
| **DPT**| |||||| ||
|| CLIP image feature| NYUv2, KITTI|0.911|0.989|0.998|**0.098**| 0.043| 0.355 |
|| Detector feature| NYUv2, KITTI|0.903|0.989|0.998|0.101| 0.045| 0.366|
|| User-friendly-1| NYUv2, KITTI |0.912|0.988|0.998|0.099|0.043|0.353|
|| User-friendly-2| NYUv2, KITTI |**0.915**|**0.990**|**0.999**|**0.098**|0.043|0.354|
|| **RSA (Ours)**| NYUv2, KITTI|0.913| 0.988| 0.998|0.099| **0.042**| **0.352**|
| **MiDas**||||||| ||
|| CLIP image feature| NYUv2, KITTI|0.724| 0.952| 0.992|0.173| 0.074| 0.579 |
|| Detector feature| NYUv2, KITTI|0.714|0.963| 0.995| 0.181| 0.073| 0.554|
|| User-friendly-1| NYUv2, KITTI |0.727|0.954|0.994|0.170|0.072|0.557|
|| User-friendly-2| NYUv2, KITTI |0.731|**0.964**|**0.996**|0.171|0.073|**0.552**|
|| **RSA (Ours)**| NYUv2, KITTI|**0.737**|0.959| 0.993| **0.168**| **0.071**| 0.561|
| **DepthAnything**||||||| ||
|| CLIP image feature| NYUv2, KITTI|0.710|0.947|0.992|0.181|0.075|0.574 |
|| Detector feature| NYUv2, KITTI|0.705|0.961| 0.995| 0.161| 0.074| 0.533|
|| User-friendly-1| NYUv2, KITTI |0.753|0.963|0.995|0.152|0.069|0.512|
|| User-friendly-2| NYUv2, KITTI |0.765|0.970|**0.996**|0.150|0.067|0.503|
|| **RSA (Ours)**|NYUv2, KITTI|**0.776**| **0.974**|**0.996**|**0.148**| **0.065**| **0.498** |

**Table 6: Additional experiments on KITTI Eigen Split.**
| **Models**| **Scaling**| **Dataset**| $\delta<1.25 \uparrow$|$\delta<1.25^{2} \uparrow$|$\delta<1.25^{3} \uparrow$|Abs Rel $\downarrow$|$\text{RMSE}_{\log} \downarrow$|RMSE $\downarrow$ |
|---|---|---|---|---|---|---|---|---|
| **DPT**| ||||||||
|| CLIP image feature| NYUv2, KITTI|0.956|0.989|0.993|0.066|0.098|2.477 |
|| Detector feature| NYUv2, KITTI|0.918|0.989| 0.998| 0.108| 0.135| 2.997|
|| User-friendly-1| NYUv2, KITTI |0.958|0.992|0.998|0.065|0.092|2.387|
|| User-friendly-2| NYUv2, KITTI |0.960|0.992|**0.999**|0.066|0.090|2.356|
|| **RSA (Ours)**| NYUv2, KITTI |**0.962**|**0.994**|0.998| **0.060**| **0.089**| **2.342** |
| **MiDas**|||||||||
|| CLIP image feature| NYUv2, KITTI|0.718|0.943|0.979|0.171|0.211|4.456|
|| Detector feature| NYUv2, KITTI|0.735|0.934| 0.985| 0.168| 0.202| 4.468|
|| User-friendly-1| NYUv2, KITTI |0.763|0.944|0.986|0.163|0.199|4.313|
|| User-friendly-2| NYUv2, KITTI |0.775|0.945|**0.988**|0.162|0.198|4.268|
|| **RSA (Ours)**| NYUv2, KITTI |**0.782**| **0.946**|0.980| **0.160**| **0.194**| **4.232**|
| **DepthAnything**|||||||||
|| CLIP image feature| NYUv2, KITTI|0.697|0.933|0.977|0.181|0.218|4.824 |
|| Detector feature| NYUv2, KITTI|0.708|0.934| 0.989| 0.173| 0.210| 4.660|
|| User-friendly-1| NYUv2, KITTI |0.721|0.951|0.986|0.167|0.201|4.512|
|| User-friendly-2| NYUv2, KITTI |0.743|0.953|**0.990**|0.161|0.198|**4.347**|
|| **RSA (Ours)**|NYUv2, KITTI |**0.756**|**0.956**| 0.987|**0.158**|**0.191**| 4.457 |
Thank you for your reply. However, I still see only limited value in the current implementation of the work. 

"The proposed study investigates the hypothesis of whether language, in the form of captions or descriptions, can infer the scale of 3D scenes."
- I agree that this is a useful investigation, however, the current study is limited: 1) figure 2 shows that scale is extracted only from text. This seems unreasonable in general settings with natural captions. As image is also important (a distant object is not different to an occluded object in your captions) 2) Currently you use generated and very structural text that is also challenging for human to produce. While this works somewhat (limited significant in your experiments) it is still directly based on image data. However, human/naturally generated captions are not investigated. Table 4 is not really covering natural captions. 

"An object detector is only used for ease of experiments"
- I think the use of object detector also places the experiments into an unnatural setting. They are hard to produce as human. 

"CLIP image features"
- CLIP features are trained in objects. They are not the same as other more advanced Foundation Models, like GPT4. It seems more natural to estimate scale directly from image using GPT4 / VLM, rather than from just text. 

To conclude, I think the fundamental experiment with natural captions is missing. The evaluation is therefore not convincing. Currently the experiments show that the scale estimation can be derived from object/segmentation rather than from natural language.
Dear authors:

Thank you for the prompt response!

For the user input text, it might be hard for the user to say "XXX object occupies YYY% of the image". 
For some user-friendly text design like “An image with c(1) , c(2) , c(3) , ... ", seems it works worse than directly using CLIP features.

For the detector knowledge, I can not find the "tables 5 and 6".

Looking forward to the further discussion!