DepthLM: Metric Depth from Vision Language Models
Authors: Zhipeng Cai
Ching-Feng Yeh
Hu Xu
Zhuang Liu
Gregory P. Meyer
Xinjie Lei
Changsheng Zhao
Shang-Wen Li
Vikas Chandra
Yangyang Shi
Venue: ICLR 2026 Oral (2026)
Keywords: Metric depth
Vision language model
Spatial reasoning
Primary area: foundation or frontier models, including LLMs
TL;DR: The first proof that VLMs can have expert model level depth estimation accuracy without architecture or loss change

Abstract: Vision language models (VLMs) can flexibly address various vision tasks through text interactions. Although successful in semantic understanding, state-of-the-art VLMs including GPT-5 still struggle in understanding 3D from 2D inputs. On the other hand, expert pure vision models achieve super-human accuracy in metric depth estimation, a key 3D understanding task. However, they require task-specific architectures and losses. Such difference motivates us to ask: Can VLMs reach expert-level accuracy without architecture or loss change? We take per-pixel metric depth estimation as the representative task and show that the answer is yes! Surprisingly, comprehensive analysis shows that text-based supervised-finetuning with sparse labels is sufficient for VLMs to unlock strong 3D understanding, no dense prediction head or complex regression/regularization loss is needed. The bottleneck lies in pixel reference and cross-dataset camera ambiguity, which we address through visual prompting and intrinsic-conditioned augmentation. With much smaller models, our method DepthLM surpasses the accuracy of most advanced VLMs by over 2x, making VLMs for the first time comparable with pure vision models. The simplicity of DepthLM also enables a single VLM to cover various 3D tasks beyond metric depth. Code and model are available at https://github.com/facebookresearch/DepthLM_Official.

Published as a conference paper at ICLR 2026
DEPTHLM: METRIC DEPTH FROM VISION LANGUAGE MODELS
Zhipeng Cai'*, Ching-Feng Yeh', Xu Hu', Zhuang Liu", Gregory P. Meyer', Xinjie Lei', Changsheng Zhao', Shang- Wen Li,, Vikas Chandra', Yangyang Shi! Meta, Princeton University
Project lead & corresponding author Soptc2h@ggmil.com
Question: How far is this point the camera?
Metric models
ZoeDepth
Metric3D
Ground Truth: 15.45m Beed1.55-LL. 5.38m
Unidepth
0.533
GPT-5: 8.5m
Depth Pro
0.864
Ours: 14.37m
MetricDDV2
0.809
UnidepthV2
0.908
(a) Outputs of advanced VLMs and DepthLM.
Qwen2.5-VL (72B)
0.228
0.263
Query each
VLMs
Gemini2.5-Pro
pixel independently
960m
GPT-5
0.388
Ours (3B)
0.859
0.2
0.8
Input
Ours
(c) Accuracy 8, (1) over datasets (Nuscenes, ETH3D, sunRGBD, ibims
(b) Point clouds generated by DepthLM.
Figure 1: We propose DepthLM, a simple and effective method that turns VLMs into strong pixel-level metric depth estimators. The latest VLMs, including GPT-5, still struggle in under- standing 3D from 2D inputs. Our model is the first VLM that has comparable accuracy to advanced metric depth models, and can generate point clouds with accurate metric scales.
ABSTRACT
Vision language models (VLMs) can flexibly address various vision tasks through interactions. Although successful in semantic understanding, state-of-the-art VLMs including GPT-5 still struggle in understanding 3D from 2D inputs. On the other hand, expert pure vision models achieve super-human accuracy met- ric depth estimation, key 3D understanding task. However, they require task- specific architectures and losses. Such difference motivates us to ask: Can VLMs reach expert-level accuracy without architecture loss change? We take per- pixel metric depth estimation as the representative task and show that the answer is yes! Surprisingly, comprehensive analysis shows that text-based supervised- finetuning with sparse labels is sufficient for VLMs to unlock strong 3D under- standing, dense prediction head complex regression/reguarriztton loss is needed. The bottleneck lies in pixel reference and cross-datase camera ambigu- conditioned augmen-
ity. which we address through visual prompting and tation. With much smaller models, our method DepthI surpasses the accuracy of most advanced VL.Ms by over 2x, making VLMSs for the first time comparable with pure vision models. The simplicity of DepthLM also enables single VLM to cover various 3D tasks beyond metric depth. Code and model are available

---

Published as a conference paper at ICLR 2026
INTRODUCTION
Understanding 3D from 2D inputs lies at the core of many applications, such as self-driving and robotics. Pure vision models (Hu et al. 2024; Bochkovskii et al. 2024) achieve super-human accu- racy through task- musk-specific architectures and complex training losses, which often require drastically different design for different tasks, Vision language models (VLMs) (Liu et al., 2023) connect visual inputs with language models, making it possible for a unified model to fiexibly handle diverse vision tasks through language interactions. However, Fig. | shows that even the most advanced VLMs still struggle with basic 3D understanding, auder-performing pure vision models by large margin.
In this work, we use the classic 3D understanding task. pixel-level metric depth estimation, as the representative and conduct comprehensive analysis on different aspects of VLMs. Surprisingly, we show that the poor 3D understanding of VLMs is not due to the lack of dense prediction head or complex regression/regularizaiin
losses. We propose DepthLM,, simple yet effective method that can turn VLMs into strong metric depth estimators without loss or architecture change.
At the core of DepthLM are 1) visual prompting, where render markers images rather than putting coordinates in text prompts, so that VLMs can accurately understand pixel locations, 2) intrinsic-conddtionedd augmentation, where unify the focal length of different images to resolve camera ambiguity and enable zero-shot generalization, 3) sparsely labeled images with text-based supervised fine-tuning (SFT) (Wei et al. 2021), where we can use labeled pixel per training image to enable strong 3D understanding. We also investigate reinforcement learning (RL) (Kaelbling al.] 1996) for model training. Interestingly, both SFT and RL can learn 3D understanding, while SFT is more efficient. We curate public datasets into a VLM benchmark suite, Depthh.MBench, to train VL.Ms and directly compare with pure vision models 3D understanding.
With 3B model, DepthLM outperforms most advanced and much larger VLMs including GPT- 5 (Singh et al. 2025), achieving over 2x accuracy (61 (Ranfil et al., 2020)) improvement across indoor and outdoor datasets (Fig. Ic DepthLM also outperforms DepthPro (Bochkovskii et al. 2024) and Metric3Dv2 (Hu et al. 2024). demonstrating for the first time that VLMs can match the accuracy of advanced pure vision models, and generate a high quality point cloud of an image with an accurate metric scale (Fig. [b). Importantly, all these results are achieved with standard VLMs and text interactions, no dense prediction head or extra module is needed. The simplicity of DepthI_M . also makes it more flexible and scalable than expert pure vision models. With the same framework, DepthLM naturally allows us to train a unified VLM, achieving high accuracy on diverse and complex 3D tasks involving reasoning, multi-point and multi-image understanding.
2 RELATED WORK
Metric Depth Estimation. Metric depth estimation is a classic 3D understanding task that predicts the depth of a pixel in meters. ZoeDepth (Bhat et al.) 2003) and Metric3D series (Yin et al., 2023] Hu al. 2024) pioneered the efforts of zero-shot metric depth estimation. ZoeDepth trained two prediction heads for indoor and outdoor scenes respectively, and used a router to automatically decide which head to use given the input. Metric3D leveraged the camera intrinsics information to normalize either the input images or the output labels so that the model could effectively learn a unified metric scale of the world even when mixing with data from different types of cameras. To remove the need of camera intrinsics, recent works (Bochkovski et al. 2024 Piccinelli et al. [2024]
2025 Wang et al. 2025 Zeng al., 2024) designed specific architectures or leverage language priors to predict the intrinsics of the input directly. This work shows that handling camera ambiguity also essential in VLMs. Through comprehensive analysis, we find that augmenting inputs is more effective for VLMS than other approaches.
VLMS for 3D understanding. SpatialVIM (Chen et al. 2024) pioneered the work of 3D under- standing with VLMs. It turned the output of pure vision models into text prompts for training. SpTt (Cheng et al., 2024) extended this idea where besides using templates to generate the prompts, it leveraged text- LLMs to generate complex reasoning questions. However, knowl- edge distillation from a complex pipeline with various pure vision models will accumulate errors in training data. Focusing object-level tasks also makes these VLMs hard to compare with pure vision models, making unclear on how far existing VLMs are from advanced pure vision models. SpatialBot (Cai al. 2024) leverages the depth map derived from pure vision models (Bhat al.)
2

---

Published as a conference paper at ICLR 2026
2023) to enable pixel-level 3D understanding. Such setting not only requires extra modules in the architecture, but also limits the tasks it can handle. Without involving more expert vision models, it struggles in more complex multi-image tasks such camera pose estimation. Recent works (Xu et a 2025 Guo et al. 2025) also studied pixel-level metric depth estimation with VLMMs. However, they did not release models investigate the cause of the performance gap between VLMSs and pure vision models. As result, their accuracy was still distant from pure vision models.
3 DEPTHLM
To understand why VLMs are behind pure vision models on 3D understanding, we first conduct comprehensive analysis on major VLM components except the architecture that we do not change. They include prompt design (Sec. 3.I. training losses (Sec. (.2) and
training (Sec. 3.2)
Then we propose our final method inspired by the analysis findings (Sec. 3.4).
DepthLMBench.. We curate a mixture of public datasets widely used in pure vision models into DepthLMMenchh, a new benehmark suite to enable the training of VLMs for 3D understanding and directly compare with pure vision models during evaluation. For both training and evaluation data, we convert the numerical 3D labels into text with template based approaches in Sec. 3.1
For training, we mix 7 widely used datasets in pure vision models with indoor and outdoor scenes. For outdoor scenes, we use Argoverse2 (Wilson et al. 2023), Waymo (Mei et al.| 2022), and NuScenes (Caesar et al. 2020). For indoor scenes, we use ScanNet++ (Yeshwanth et al. 2023). Taskonomy (Zamir et al.
2018 HM3d (Ramakrishnan et al. (02I) and Matterport3d (Chang et al.
2017). This results in around 16M training images (see Appendix A.2 details). Pure vision mod- els (Bochkovskii et al. 2024 Piccinelli et al. 2025) often mix > 20 datasets for training, including large portion of synthetic data. Empirically, low quality and synthetic data do not benefit VLM training without cleaning, hence we only use the high quality datasets above for simplicity.
For evaluation, we mix datasets including both indoor and outdoor scenes non-overlapping with the training data, which allows us to thoroughly evaluate the performance of VLMMs on metric depth estimation. There are 3 outdoor datasets: Argoverse2, DDAD (Guizilini et al. 2020). NuScenes; 4 indoor datasets: ScanNet++, sunRGBD (Song et al. 2015). iBimsl (Koch et al. 2018) and NYUv2 (Silberman et al. 2012); and dataset ETH3D (Schops et al. 2017) with both indoor and outdoor scenes. Since existing evaluation datasets with accurate labels are limited, we still include data from Argoverse2, Nuscenes and Scannet++ to ensure scene diversity, and reserve around 10 scenes per dataset that are non-overlapping with the training data for evaluation.
Analysis Setup. Unless otherwise stated, we finetune DepthI.M on the pre-trained 3B model of (Bai et al. 2025) using text-hased supervised fine-tuning (SFT) for analysis. See Sec. for implemen- tation details. We follow the standard in pure vision models (Bochkovski et al. 2024) and use d to evaluate the prediction accuracy, i.e., ratio of the outputs that are within 25% difference to the ground-truth (GT). We evaluate on each dataset for 8, random samples, which provides stable statistics, i,e, increasing this number introduces negligible difference.
3.1 PROMPT DESIGN
We start our analysis from the simplest setup where the training and evaluation data are from the same dataset to remove issues from mix-data training.
Text or marker-based pixel reference? VLMs have no pre-defined grid-shaped output domain as pure vision models. How to prompt them to understand fine-grained pixel locations is an important question, since slight error could make the model output depth on completely different object, leading to severe errors. Existing VLMs (Cai et al. 2024, Guo et al.] 2025) often refer to a pixel using its coordinate (X, Y) in text. However, our initial experiments show that VLMs struggle to map text-based coordinates to the pixel locations, even when trained with them. We find that visual prompting is an effective solution to this issue.
For visual prompting, we render markers pointing to the query pixel directly on the input image, and ask: "How many meters is this point from the camera?". See Appendix A.II for the rendered markers. During training, we set the answer to "The point is around X meters away from the cam- era.". where X is treated normal text and is rounded two decimal places, which prevents long
3

---

Published as a conference paper at ICLR 2026
floating point values and is sufficient for typical 3D understanding tasks. Empirically, VLMs not sensitive to the prompt we use. We do not ask for the principal herward-backward axis distance as in pure vision models (Bhat al. 2023) since euclidean distance is more common in language interactions and the two values can convert to each other using camera intrinsics. Sec. 3.4 shows the prompt that enable VLMS to prediet principal axis distance.
As shown in Fig. 2 text-based pixel reference (similar to (Guo al. 2025), J see Appendix[A.1. degrades the model accuracy on both indoor and outdoor data. The gap is especially large (0.15) on indoor scenes (ScanNett++), we conjecture that this is because more objects and occlusions are presented in indoor scenes, leading to more boundary regions. Meanwhile, the ac- curacy remains similar with different marker shapes, showing the robustness of visual prompting- Interestingly, Sec. [howw that even for VLMs trained with text-based coordinates, apply- ing visual prompting still benefits the accuracy. In summary
ScanNett+
Argoverse2
Marker
0.733
Marker (squanc)
0.678
0.7443
Marker (smow)
0.685
0.744
0.55
0.65 Acturacy 4,(1)
0.75
Figure 2: Pixel reference.
Finding 1. VLMs understand marker-based pixel reference much better than text- bas-baaedd one.
3.2 Loss
SFT or RL. Given the advancement of reasoning. reinforcement learning (RI) especially GRPO (Shao et al., 2024) has become popular alternative of SFT. To scale up training, it is important to understand which method is more suitable for 3D understanding. GRPO leverages format-based rewards for LLM training. Since metric depth estimation, as many 3D understanding tasks, returns numbers as the answer, we can use regression- based metrics as the reward. We apply the GRPO style prompt and ask: "How far is this point from the camera? Output the thinking pro- cess in <think> </think> and final answer (the meter number only, without the unit) in <answer> <Janswer> tags.". We extract X from templates
compute the reward. Empirically, the model shares similar reasoning traces for all inputs after GRPO training. An example answer is: "<think> The point located about X meters from the camera. </think>, <answer>> X </answer>".
As shown in Fig. 3al GRPO is not very sensitive to the reward function, we experiment with 1) 81. 2) negative Abs i.e., the negative value of absolute relative difference (Ranfdl et al. 2020). 3) the negative Ly loss, and 4) the negative L1 loss. All metrics have reasonable accuracy. Overall, the negative L1 loss is simple and effective reward. With this reward, we tuned other GRPO hyper- parameters (see Appendix (A.4), and compare GRPO with SFT. As shown in Fig. 3b] given the same number of training samples, SFT and GRPO perform similarly well. However, GRPO requires much heavier per-sample compute than SFT (8-16 times slower in practice). Therefore, conclude that
Finding 2. Though both SFT and RL can achieve reasonable accuracy, SFT is more efficient to scale up VLM training for 3D understanding.
0.8
GRPO
SFT
81
0.38
Accuracy (1t)
GRPO reward
NegAbsRel
0.6
0.47
0.53
0.4
NegL2
0.44
0.28
0.23
NegL1
0.53
0.4
0.6
800
8000
80000
Accuracy ò1 (t)
Number of training samples
(a) Effect of GRPO rewards (8K training samples).
(b) Accuracy training data size.
Figure 3: SFT vs GRPO. (a) Negative L, (NegI.1) the best GRPO reward while most common metrics have reasonable accuracy. (b) While SFT and GRPO achieve similar accuracy given the same train dataset size, SFT has much higher per-sample efficiency. We use Argoverse2 dataset for experiments, see Appendix A.3 for cross dataset evaluation, which shows the same trend.
4

---

Published as a conference paper at ICLR 2026
Acauney 6,th)
0.75
0.65
0.694
0.686
0.7
0.691
0.694
Acourasy fx(1)
0.85
0.804
0.811
0.823
0.824
0.56
0.75
0.45
0.686
0.35
0.358
0.302
0.328
0.7
0.894
0.26
Direct sraning
Intrinsics
Prodict
0.675
0.671
0671
0.65
0.8
1.a0sted 3kml muining Image
2+ beriibeke trakiing imagge
1000
7.0
14,0
21.0
28.0 30.0
drection
Focal Length
Number training samples (millom)
(a) Accuracy of different camera (b) Increasing funi benefits the per- (c) Accuracy with different number of training samples.
ambiguity handling strategies. formance until 1000 pixels.
Figure& 4: Mix data training analysis. For (a) and (b), we train on 500K samples on the mixed datasets of DepthLMBench, and report the average accuracy across all evaluation datasets.
3.3 MIX-DATA TRAINING
How to handle camera ambiguity in VLMs? An important problem in metric depth mix-data training is camera ambiguity. where different images can be captured by different cameras. Simi- larly looking images from different cameras have drastically different scale. Directly mix such data for training is sufficient for relative depth (Ranftl et al.] 2020). However, it makes the model struggle to learn generalizable world scale, failing on metric depth estimation (Yin et al. 2023) Different strategies have been proposed to handle camera ambiguity in pure vision models (Bhat al.) 2023 Yin al. 2023] [Bochkovski et al.) 2024). For VLMs, See (Guo et al. 2025) put camera intrinsics into text prompts. However, what is the most effective approach for VLMs remains unclear.
To answer this question, we compare four popular approaches: 1) Direct training with images of varied focal lengths, hoping VLMs can implicitly learn to distinguish different cameras. Adding camera intrinsics explicitly into the text prompt as in Seed1 VL. 3) Predicting camera intrinsics by letting the model output the camera ray direction before metric depth, similar to (Bochkovskij et al. 2024 Piccinelli et al. (2055). 4) Unifying the focal length through intrinsic-conditioned image augmentation, similar to (Yin et al. 2023) (See Sec. [3.4] for details). See Appendix (A.5) for the prompts of 2) and 3). As shown in Fig. 4a unifying the local length doubles the accuracy of other approaches. Moreover, adding intrinsics explicitly in text or predicting camera intrinsics do not have higher accuracy than direct mix data training. This result shows that
Finding 3. Without architecture change, VLMs struggle to distinguish different cameras, unifying the focal length by intrinsic-conditionee augmentations can effectively resolve camera ambiguity.
Best unified focal length. In intrinsic-conditionee augmentation, we have to decide the value of the unified focal length. As shown in Fig. 4bb increasing the unified focal length, which increases image sizes, benefits the accuracy until 1,000 pixels. Meanwhile, the accuracy changes <3% across a wide range of values, showing the stability of augmentation--aase approach.
How many labeled pixels we need per image? Pure vision models are trained on millions of densely labeled images, where each image typically involve tens of thousands of labeled pixels. Each VLM training sample only has sparsely labeled pixels by design. How many labeled pixels do VLMS need to see per image in order to catch up with pure vision models is an interesting and unexplored question. Fig. [4c shows the accuracy of DepthLM with different number of training samples. Before the dashed line (16M images), the model only sees labeled pixel per training image, which already achieves over 0.8 6, and as shown later in Sec. already matches the accuracy of pure vision models. Appendix ?? provides further analysis showing that increasing the number of images is more effective than increasing the number of labeled points for DepthLM. This somewhat surprising result shows that
Finding 4. VLMs can learn 3D understanding from as sparse as I labeled pixel per training image. Image diversity is more important than label density for VLM training.
Increasing the number of images vs increasing label density. As mentioned in Finding 4 DepthLM benefits more from having more images than having denser labels. To further demon- strate this point, we conduct an experiment using the Argoverse2 dataset by training the model with
5

---

Published as a conference paper at ICLR 2026
0.8
0.775
0.75
Accuracy
0.7
0.642
10
100
Labeled points per image (80K samples in total)
Figure 5: Increase number of images increase label density. Given the same training dataset size (80K samples), increasing label density while proportionally decreasing the number of images hurts the performance of DepthLM..
DepthLM
Vision Beeeaa Mldder
The pixel is 15.32 meters away from the camera.
Input
Marked Image
Input with unified focal length
How this pixel from camera?
Intrinsics
Marker
Figure 6: DepthLM. DepthLM first augment the input image to have unified focal length. Then, it renders visual markers on the image for pixel reference and uses text to interact with VLMs directly.
the same number of samples, i.e., question-answe answwrr pairs, but vary the label density. Specifically, we train the model on 80K argoverse samples, with label density levels: 1) point per image, i.e., 80K different images with 1 labeled point per image seen during training. 2) 10 points per image, i.e., 8K different images with 10 labeled points per image seen during training. 3) 100 points per image, i.e. 800 different images with 100 labeled points per image seen during training. The evaluation is done on our Argoverse2 evaluation split.
As shown in Fig. 5 increasing the label density, which reduces the number of images, significantly reduces the accuracy. The accuracy difference was larger on 10 - 100 points change than on 10 points change.
3.4 FINAL METHOD
The findings from analysis inspire us to propose the complete method of DepthLM. Fig. shows the method overview.
Pixel reference. Based on Finding T. instead of putting pixel coordinates into the text prompt as previous VLMs (Cai al., 22024] Guo al. (2025), render visual markers, i.c. small arrows pointing to the pixels, directly on the image to enable accurate pixel location understanding for VLMs. After rendering the visual marker, we can use text prompts enable various tasks.
Architecture and training. We finetune our models from pretrained VLMs without architecture change. Based on Finding [ we apply standard SFT with the next token prediction paradigm and the cross-entropy loss on the text tokens. As suggested by Finding no regression regularization loss used in pure vision models needed.
Resolve camera ambiguity. Based on Finding 3 we resize the input image before rendering mark- ers unify its focal length. For simplicity, we assume the input image is undistorted and fol- lows the pinhole camera model (Hartley & Zisserman 2003). We resize the original input image I BR xx3 into I' E IRW W x/''f
where W Jum W and H' Lut H. fa and fy are the focal

---

Published as a conference paper at ICLR 2026
length parameters (in pixels) of I, and funi 1000 is the pre-defined unified focal length. Pure vision models Yin et al. 2023 Hu et al. 2024) also crop I' to a fixed size to have unified principle points. As discussed in [Bochkovskii al. 2024), such strategy makes the model sensitive to the image size, requiring different croping sizes for different evaluation datasets to achieve the reported numbers. In DepthI.M, we found that long the focal length is unified, image size using random crop so that during evaluation no cropping is needed.
vary the training
Beyond metric depth single VLM for flexible 3D understanding. We use metric depth esti- mation to verify that VLMs can achieve comparable accuracy as pure vision models. However, the value of DepthLM beyond metric depth. Since no task-specific architecture or loss is needed, we can apply the same DepthLM framework to train a unified model on various 3D understanding tasks.
As concept proof, we expand DepthLM five other representative tasks. They involve 1) prin- cipal axis distance, which estimates the distance in the forward-backwarr direction rather than the euclidean distance, and the metric depth that pure vision models actually predict, 2) speed, which estimates the speed needed to reach point in a limited time, 3) time, which estimates the time needed to reach point given speed, 4) two point distance, and 5) metric scale camera pose, which estimates the distance that the camera has moved between two images. Fig. visualizes task exam- ples and the outputs of GPT-5 and DepthLM. These tasks cover single image single point questions, reasoning questions that require simultancous 3D and math understanding, single image multi mullii-point questions, and multi-image questions.
RESULTS
Implementation. To simplify experiments and avoid the need of cropping after unifying the focal length, we use VLMs that can accept images without resizing to a fixed shape. We choose the 3B and 7B models of (Bai al., 2025) as the default VLM architecture. To demonstrate the general applicability of DepthI.M, we also experiment with the 12B model of (Agrawal et al. 2024). which has a significantly different architecture. We train models with PyTorch on 23M 6MM60M samples from DepthLMBenchh, i.e., 2-4 labeled pixels per image, with 128 H100 GPUs for about 2-4 days depending on the model. We sample equally from each dataset during training except that the weight of Matterport3d is reduced to 0.1 of other datasets since the number of scenes is smaller. During training, we randomly crop the images after normalizing the focal length, with width sampled from 1,000 to 1,400 pixels, and height from 700 to 1.200 pixels. During evaluation, we do not cropping. See appendix A.6 for detailed hyper-parameters individual models.
Main result. Table and 2 compare DepthLM with existing VLMMs and pure vision models spectively. Most VL.Ms perform not much better than the naive baseline with constant outputs, especially on the indoor data. Even GPT-5 and Gemini-2.5-Pro (Comaniei al. 2025) have be- low 0.4 81. which is much worse than the worst performing pure vision model as shown in Fig. [C
Spatial VLLMs like SpaceLLaVA (github contributors 2024) (third-party implementation of (Chen et al. 2024)) and Spatial-RGP7 (Cheng et al. (024) perform not better than generalist VLMs, though they have been trained for object level 3D understanding. This not only because they lack
Speed How maters second should we move in order to nech this point esacty acconds?
many reach with the spend of Orisi?
Two point distance: How for are these 2 soints from each other?
this forward backward directio??
Metric Scale Camera
Pose: How many meters
Ground Truth: 1.78m's OPT.S 250mle The point around 7.23 mebers away. Henoe, the speed should be around 7.29 1.81mts
Ground Truth: 7.75s OPTS 2.5
the camera moved betwean these images?
Ours:
Ours: The point is around 48.28 metere mway. Hence, need around 48.28 6.0 8.05s
Ground Truth: 2 2.rbm GPT.S: 8.87mn Ours: 2.68m
Ground Trulh: 4-40m GPT.5: 2.7m Ours: 4.30m
Ground Teutr: 5 5.94m
GFT.S om Ours: 5.62m
Single Image Single Point
(Single Image Single Point) Reesoning
Multi-eint
Mult-Image
Figure 7: Scaling DepthLM to complex 3D tasks.

---

Published as a conference paper at ICLR 2026
Table 1: VLM result. We tune the prompt for VLMs that are not trained directly on our task to maximize their performance. State-of-the-an VLMs including GPT-5 have only below 0.4 ø1. MepthLM, though orders of magnitudes smaller, achieves an over 2x improvement.
(1) of different methods
Our+ ETH3D
Argoverse2
DDAD
NuSeenes
ScanNet++
sunROBD
iBimsl
NYUv2
Average
Naiiv Prodiction with Constont Answerrs
ALWAYS
0.006
0.010
0.010
0.106
0.383
0.186
VLMs
OWEN2.5-VL (3B)
0.133
0.083
0.090
0.087
0.120
0.134
0.080
0.128
0.106
W.L. (7B)
0.077
0.120
0.070
0.126
0.089
0.168
0.118
OWEN2.5-VI. (728)
0.119
0.140
0.186
0,220
0.272
0,276
0.212
0.324
0.219
MOLMO (7B-D)
0.200
0.132
0.126
0.225
0.203
PIXTRAL (126)
0.157
0.132
0.118
0,141
0.318
0.308
0.270
0.145
0.199
GEMINI-2.5 2.5-PRO
0.280
0.252
0.365
0.328
0.380
0.270
0.466
0.394
0.342
GPT-03
0,208
0.283
0.309
0,305
0.375
0.426
0.375
0.470
0,344
GPT-5
0.218
0.302
0.382
0.313
0.428
0,471
0.307
0.540
0.370
Spatial VL.Ms
SPACELLAVA (13B)
0,100
0.067
0.083
0,090
0.269
0.233
0.208
0.178
0.154
RGPT (8B)
0.055
0.046
0.100
0,220
0.346
0.369
0.240
0.265
0.205
VLMs Trained Metrie Depth Ertimationn
SEBDI,S-VL (OPICIAL SETUP)
0.009
0.012
0.013
0,219
0.495
0.321
0.459
0.412
0.243
SEEDI.5-VL. (OUR PROMPT)
0.040
0.074
0.309
0.593
0.841
0.400
OURS (3B)
0.745
0.868
0.824
OURS (7B)
0.833
0,747
0,865
0,718
0,850
0.859
0.920
0,915
0,838
OURS PIXTRAL (128)
0.734
0.670
0.819
0.653
0.834
0.786
0.799
0,771
Table 2: Comparison with pure vision models. For pure vision models, we use the numbers reported in (Piccinelli et al., 2025), and (Bochkovskii et al. 2024) if some numbers do not exist in (Piccinelli et al. (0255. means result reported in previous papers. The last column reports the relative accuracy improvement of pure vision models over our model, i.c., (8 80uns) 5/0purs Ours. Our model is the first VLM that has comparable accuracy pure vision models.
(.(t) different methods
Out
Out+In ETH3D
Jn ihims
DDAD
Nuscenes
sunRGBD
vs Ours (1)
ZOEDEPTH
0.272
0.283
0.350
0.867
0.580
-42.8%
DEPTHANY YYHHIGG
0.354
0.093
0,850
0.714
-40.3%
SAPTHAKYTHINGT
0.171
0.363
0.724
-48.5
METRIC3D
0.723
UNIDEPTH
0.858
0.846
0.185
0.943
0.157
27.3%
DEPTH PRO
0.299
0.566
0.397
METRIC3DV2
0.841
0.900
0.812
0.684
-3.8%
UNIDEPTIV2
0.882
0.870
0.852
0.964
+9.2%
OURS (?B)
0.747
0.865
0.718
0.859
0.920
pixel-level training data, but also because they do not consider camera ambiguity. Seed1.5 VL is the only model that has been trained on the same task DepthLM, however, its accuracy especially on outdoor data, is still much worse than pure vision models.
Interestingly, instead of using the official setup from its open-source implementation, Seed1.5-VI. performs much better when we replace their text-based pixel coordinates with our marker-based pixel reference, i.c., Seed1.5-V even with our strategy, It still has only 0.4 6j. and has very low accuracy on outdoor data. In contrast, DepthLM has over 0.83 6 which surpasses the accuracy of Depth Pro and Metric3Dv2. This is the first time that a VLM can match the accuracy of expert metric depth models on both indoor and outdoor data. More importantly, DepthLM can be flexibly extended to handle various tasks beyond metric depth estimation, with the same architecture and training framework.
(our prompt), supporting the generalization of our findings. But
In terms of the model size, our 3B model is orders of magnitudes smaller than baseline VLMs like Qwen2.5-VL. (72B) and Seed1.5-VL. (20B Active MoE Parameters), but still improves their 61 by over 2x. If comparing to the base Qwen2.5- LL 3B model, the improvement is about 8x. Our 7B model has slightly higher accuracy than the 3B model, reaching over 0.9 6 on ibims I and NYUv2. This result shows that larger model sizes help, but they are not necessary for VLMs to have accurate 3D understanding. Meanwhile, has reasonable accuracy, though slightly lower than using the default architecture. This shows the cross-architecture applicability of MM and the base model performance matters for DepthI.M.
12B model finetuned from (Agrawal et al. 2024) also
8

---

Published as a conference paper at ICLR 2026
Input
Metric3D
2.226m
Unidapth
1.444m
DapthPro
O.EZAM
4.47m
MetricaDv2
0.7c0m
Unidepthv2
0.8ATI
3.208
74:m
Ours
0.1
2.960m
(a)
(b)
(c)
(d)
(e)
(f)
Figure 8: Visualization. The scale of DepthLM is close to GT from small indoor scenes to large out- door scenes. Pure vision models produce smooth points, which is advantageous on non- boundary re- gions. However, they also have over-smoothing effect on the (marked) boundary regions, leading to flying points between two distinct objects. Interestingly, DepthI naturally avoids over-smoothing without enforcement during training any post-processing.
Visualization. Interestingly, though only trained for single point prediction, DepthLM can already generate high quality point clouds simply by querying each pixel independently, without dense pre- diction head. Fig. [8 visualizes the point clouds generated by pure vision models and DepthLM (7B). We query 10K pixels uniformly spanned
each image. The point cloud scale of DepthLM is stable and reasonably close to GT across small indoor scenes and large outdoor scenes. Some pure vision models have severe errors in the scene scale, such Depth Pro (b, e, f), Unidepth (c), and Metric3D on (a, b. d, f).
Another interesting observation that pure vision models and DepthLM have different detail pat- terns. Specifically, pure vision models generate point clouds with smooth surfaces, however, they also over-smooth the boundary regions as marked in each image, which causes flying points and thin objects merged into other objects (the road lights in (e) and (f)). Interestingly, without any post-procsssing or loss for regularization, DepthLM naturally avoids flying points, showing clear boundaries between different objects, with slightly higher noise in the smooth regions. Such prop- erty beneficial for tasks that require accurate boundary separation, e.g., judging whether a robot can safely move between 2 objects. It also shows that visual prompting effectively enables accurate pixel location understanding in VLMs, since otherwise the boundaries would not be clear.
Other Tasks. To demonstrate the multi-task flexibility of DepthLM, we finetune unified model on the tasks proposed in Sec. Sec.B.4 together with the metric depth task using our 7b model. The training done by equally sampling from each task. For simplicity, only show the average number of each task across multiple datasets. Please refer to Appendix [A.7 for results on each dataset. The hyper-parameters the same in the main experiments, except that train for 40M samples.

---

Published as a conference paper at ICLR 2026
Table 3: Multi-task result. Since not all datasets have pose labels, we train the pose task on
Argoverse2 and evaluate on Argoverse2 and Nuscenes.
61(1) on different tasks
Single image single point
Reasoming
Multi-point Two point distance
Malti-image Prsse
Distance
Principal axis distance
Speed
Time
Average
ALWAYS OUTPUT 2.0
0.186
0.172
0.087
0.094
0.119
0.189
0.141
QWEN2.5-VL (7B)
0,118
0.085
0.136
0.087
0.066
0,048
0.09
SPACELLAVA (13B)
0,154
0.163
0.116
0.122
0.157
0.047
0.127
SPXTIALRGPT (8B)
0.205
0.132
0.167
0.122
0,143
0.195
0.161
SEEDI.S-VL prompt)
0,400
0.174
0.223
0.119
0.101
0.000
0,170
Gemini-2.5-Pro
0.342
0,209
0.213
0,209
0.140
0.025
0.189
GPT-5
0.241
0.199
0.181
0.150
OuRS (7B)
0.828
0.831
0.817
0.816
0.657
0.876
0.804
As shown in Table without our training, the Qwen2.5-VL (7B) model fails on all tasks, having <0.1 81 on average. Advanced proprietary models including GPT-5 also struggle on more complex 3D tasks. As shown in Fig. 7 GPT-5 can return Om in camera pose estimation when the camera actually moves over 5m. Such catastrophic failure is within expectation since these baselines cannot understand the metric scale even in the basic depth estimation task. DepthLM achieves more than 0.8 o1. outperforming the baselines by over 3.8x. Note that the gap between baselines and our model is much larger than in Table [ showing that VLMs have a larger room to be improved on complex 3D tasks, and the effectiveness of DepthLM. This result also demonstrates the flexibility of D over expert models, where it can cover diverse tasks with unified architecture and training framework.
5 CONCLUSION
We propose DepthLM, simple and effective framework that can make VLMs strong pixel-level metric depth estimators. Through comprehensive analysis, we show that VLMs under- mnder-perform pure vision models in 3D understanding or complex training losses. The key problem lies in pixel reference and camera ambiguity, which augmentation
because they lack extra modules like dense prediction heads
we effectively address with the proposed visual prompting and approaches. These strategies allow for the first time to use text-based SFT with standard VLMs to match the accuracy of expert pure vision models, and expand the same framework to cover various tasks with unified model. In terms of limitations, we focus on the simplest and most important design of VLMs. We believe more ane-grained strategies can be investigated in the future to even make VLMs surpass the accuracy of pure vision models. For example, design data filtering pipelines to add more datasets. And training with diverse and complementary tasks to enhance generalization.
REFERENCES
Pravesh Agrawal, Szymon Antoniak, Emma Bou Hanna, Baptiste Bout, Devendra Chaplot, Jes- sica Chudnovsky, Diogo Costa, Baudouin De Monicault, Saurabh Garg, Theophile Gervet, et al. Pixtral 12b. arXiv preprint ar arXiv:2410.0700 2024.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502. 13923, 2025.
Shariq Farooq Bhat, Reiner Birkl, Diana Wolk, Peter Wonka, and Matthias Müller. Zoedepth: Zero- shot transfer by combining relative and metric depth. arXiv preprint arXiv:2302. 2023.
Aleksei Bochkovskii, AmaĀGi Delaunoy, Hugo Germain, Marcel Santos, Yichao Zhou, Stephan R Richter, and Vladlen Koltun. Depth pro: Sharp monocular metric depth in less than a second. arXiv preprint ar arXiv:2410.02073. 2024.
Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. Nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1162-116631 2020.
10

---

Published as a conference paper at ICLR 2026
Wenxiao Cai, laroslav Ponomarenko, Jianhao Yuan, Xiaoqi Li, Wankou Yang, Hao Dong, and Bo Zhao. Spatialbot: Precise spatial understanding with vision language models. arXiv preprint arXiv:2406.13642, 2024.
Angel Chang, Angela Dai. Thomas Funkhouser, Maciej Halber, Matthias Niessner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. Matterportåd: Learning from rgb-d data in indoor environments. arXiv preprint arXiv: 709.0009966600 06158, 2017.
Boyuan Chen, Zhuo Xu, Sean Kirmani, Brain Ichter, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. Spatialvim: Endowing vision-language models with spatial reasoning capabilities. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14455-14465, 2024.
An-Chieh Cheng, Hongxu Yin, Yang Fu, Qiushan Guo, Ruihan Yang, Jan Kautz, Xiaolong Wang, and Sifei Liu. Spatialrgpt: Grounded spatial reasoning in vision-language models. Advances in Neurul Information Processing Systems, 37:135062-335093, 2024.
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capa- bilities. arXiv preprint arXiv:2507.0626. 2025.
SpaceLLaVA github contributors.
Spacellava. 2024. URL https://uuggingface
SSaaceLLaaa
Vitor Guizilini, Rares Ambrus, Sudeep Pillai, Allan Raventos, and Adrien Gaidon. 3d packing for self-supervised monocular depth estimation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2485-2494, 2020.
Dong Guo, Faming Wu, Feida Zhu, Fuxing Leng. Guang Shi, Haobin Chen, Haoqi Fan, Jian Wang. Jianyu Jiang, Jiawei Wang, al. Seedl. 5-v1 technical report. arXiv preprint arXiv:2505.07062. 2025.
Richard Hartley and Andrew Zisserman. Multiple view geometry in computer vision. Cambridge university press, 2003.
Mu Hu, Wei Yin, Chi Zhang, Zhipeng Cai, Xiaoxiao Long, Hao Chen, Kaixuan Wang, Gang Yu, Chunhua Shen, and Shaojie Shen. Metric3d v2: A versatile monocular geometric foundation model for zero-shot metric depth and surface normal estimation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
Leslie Pack Kaelbling, Michael Littman, and Andrew W Moore. Reinforcement learning: A survey. Journal of artificial intelligence research, 4:237-285, 1996.
Tobias Koch, Lukas Liebel, Friedrich Fraundorfer, and Marco Korner. Evaluation of enn-based single-image depth estimation methods. In Proceedings of the European Conference on Computer Vision (ECCV) Workshops, pp. 0-0, 2018.
Shuaipeng Li, Penghao Zhao, Hailin Zhang, Xingwu Sun, Hao Wu, Dian Jiao, Weiyan Wang, Chengjun Liu, Zheng Fang, Jinbao Xue, et al. Surge phenomenon in optimal learning rate and batch size scaling. Advances Neural Information Processing Systems, 37:132722-132746, 2024.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892-34916, 2023.
Jieru Mei, Alex Zihao Zhu, Xinchen Yan, Hang Yan, Siyuan Qiao, Liang-Chich Chich Chen, and Hen- rik Kretzschmar. Waymo open dataset: Panoramic video panoptic segmentation. In European Conference on Computer Vision, pp. 53-72. Springer, 2022.
Luigi Piccinelli, Yung-Hsu Hsu Yang, Christos Sakaridis, Mattia Segu, Siyuan Li, Lue Van Gool, and Fisher Yu. Unidepth: Universal monocular metric depth estimation. In Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition, pp. 10106-101 10116. 2024.

---

Published as a conference paper at ICLR 2026
Luigi Piccinelli, Christos Sakaridis, Yung-Hsu Yang, Mattia Segu, Siyuan Li, Wim Abbeloos, and Lue Van Gool. Unidepthy2: Universal monocular metric depth estimation made simpler. arXiv preprint ar 2rX1v:2502.20110, 2025.
Santhosh K Ramakrishnan, Aaron Gokaslan, Erik Wijmans, Oleksandr Maksymets, Alex Clegg, John Turner, Eric Undersander, Wojciech Galuba, Andrew Westbury, Angel X Chang, et al. Habitat- matterport 3d dataset (hm3d): 1000 large-scale 3d environments for embodied ai. arXiv preprint ar arXiv:2109.00222 2021.
René Ranftl, Katrin Lasinger. David Hafner, Konrad Schindler, and Vladlen Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. IEEE transac- tions on pattern analysis and machine intelligence, 44(3):1623-16337, 2020.
Thomas Schops, Johannes I. Schonberger, Silvano Galliani, Torsten Sattler, Konrad Schindler, Mare Pollefeys, and Andreas Geiger. A multi-view stereo benchmark with high-resolution images and multi-camera videos. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3260-3269, 2017.
Zhihong Shao, Peiyi Wang. Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemati- cal reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.
Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and sup- port inference from rgbd images. In European conference on computer vision, pp. 746-760. Springer, 2012.
Aaditya Singh, Adam Fry, Adam Perelman, Adam Tart, Adi Ganesh, Ahmed El-Kishky, Aidan McLaughlin, Aiden Low, AJ Ostrow, Akhila Ananthram, Akshay Nathan, Alan Luo, Alec Helyar, Aleksander Madry, Aleksandr Efremov, Aleksandra Spyra, Alex Baker Alex Beutel, Alex Karpenko, Alex Makelov, et al. Gpt-5 blog post. httpps://opena: com/index:
introducing- 57 2025.
Shuran Song, Samuel P Lichtenberg, and Jianxiong Xiao. Sun rgb-d: A rgb-d scene understand- ing benchmark suite. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 567-576, 2015.
Ruicheng Wang, Sicheng Xu, Yue Dong, Yu Deng, Jianfeng Xiang, Zelong Lv, Guangzhong Sun, Xin Tong, and Jiaolong Yang. Moge-2: Accurate monocular geometry with metric scale and sharp details. arXiv preprint arXiv:2507.025444 2025.
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc Le. Finetuned languaag models are zero-shot learners. arXiv preprint 10X.::2009.01652. 2021.
Benjamin Wilson, William Qi, Tanmay Agarwal, John Lambert, Jagjeet Singh, Siddhesh Khan- delwal, Bowen Pan, Ratnesh Kumar, Andrew Hartnett, Jhony Kaesemodel Pontes, et al. Ar- goverse 2: Next generation datasets for self-driving perception and forecasting. arXiv preprint NXXv:23011.0493 2023.
Runsen Xu, Weiyao Wang, Hao Tang, Xingyu Chen, Xiaodong Wang, Fu-Jen Chu, Dahua Lin, Matt Feiszli, and Kevin Liang. Multi-spatialmllnn Multi-frame spatial understanding with multi- modal large language models. arXiv preprint arXiv:2505.17015, 2025.
Chandan Yeshwanth, Yueh-Cheng Liu, Matthias Nießner, and Angela Dai. Scannet++: A high- fidelity dataset of 3d indoor scenes. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 12-22, 2023.
Wei Yin, Chi Chi Hao Chen, Zhipeng Cai, Gang Yu, Kaixuan Wang, Xiaozi Chen, and Chunhua Shen. Metric3d: Towards zero-shot metric 3d prediction from single image. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 9043-9053, 2023.
Amir R Zamir, Alexander Sax, William Shen, Leonidas J Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning. In Proceedings of the IEEE con- ference computer vision and pattern recognition, pp. 3712-3722, 2018
12

---

Published as a conference paper at ICLR 2026
Ziyao Zeng, Daniel Wang. Fengyu Yang. Hyoungseob Park, Stefano Soatto, Dong Lao, and Alex Wong. Wordepth: Variational language prior for monocular depth estimation. In Proceedings of the IEEE/CVF Conference Computer Viston and Pattern Recognition, pp. 2024.
13

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 EXAMPLES OF RENDERED MARKERS AND TEXT-BASED PIXEL REFERENCE
+
arTow
Square
CrOSS
Figure 9: Actual markers rendered by our method. We show here different types of markers used in the experiment of Fig.
Fig. 9 shows the actual markers rendered in the experiment of Sec. 3.1 Empirically, a marker of roughly 5 pixels wide is enough for VLMs to recognize it. We also tried different markers and sizes when evaluating baseline VL.Ms to maximize their accuracy.
For text-based pixel reference experimented in Sec. 3.1 we simply ask: "Given this image of size (width W, height H), how far is the pixel at (X, Y) from the camera?"
A.2 STATISTICS OF DEPTHLMBENCH
Table 4: Statistics of different training datasets. We report <images available in the dataset>
<images used for our training>.
Dataset
Argoverse2
Waymo
Nuscenes
ScanNet++
Taskonomy
HM3d
Matterport3d
Number images
3.5M/IM
IM/700K
200K/200K
10M/IM
4M/M
9M/9M
190K/190K
Table shows the number of images for each training dataset in DepthLMBench. Datasets like
Argoverse2 and ScanNet++ contain highly similar video frames, which do not help to improve the performance. Hence, we only use subset of the frames for training.
A.3 CROSS- EVALUATION FOR SFT VS GRPO EXPERIMENTS
0.6
GRPO
SFT
Accuracy (t)
0.41
0.45
0.47 0.48
0.4
0.2
0.15
800
8000
80000
Number training samples
Figure 10: SFT vs GRPO with cross dataset evaluation. We show the result when we train Argoverse2 and evaluate on NuScenes. The trend is similar as in Fig. 3b
As mentioned in Sec. 3.2. SFT and GRPO have similar accuracy when trained on the same number of samples. Here, we show the same experiment with cross dataset evaluation to verify whether GRPO bring any benefit in the zero-shot scenario. We train the model on Argoverse2 and evaluate NuScenes. As shown in Fig. I0, the trend cross-dataset evaluation is similar to the evaluation on the same dataset.
14

---

Published as a conference paper at ICLR 2026
A.4 GRPO
Table 5: Hyper-parameters of different experiments.
Experiment
36 SFT
7b SFT
12b SFT
3b GRPO
Number training samples
35M
60M
23M
Leaming rate
5.2e:5
1.0c4
5.6c
1.0e-5
Learning rale schedule
Cosine with linear warmup
Cosine with lincar warmup
Cosine with linear warmup
Constant
Warmup ratio
0.1
0.1
Batch size
1280
1280
384
32
FSDP
Yes
Yes
No
Gradient clipping
0.1
0.5
1.0
1.0
Grodient checkpointing
Yes
Yes
Yes
Yes
Bíloat16
Yes
Yes
Yes
Flash Attention 2
Yes
Yes
Yes
Yes
Unified Focal Length
1000 pixels
1000 pixels
750 pixels
1000 pixels
For GRPO experiments, we use the group size of 8 with format reward, and set 3 during training, where B is the weight in the GRPO loss terms control the similarity between the outputs of the trained and the original models. Empirically changing the group size and learning rate schedules do affect the performance significantly, but having non-zero tends to make the model underfit the metric depth task, leading to much lower accuracy and long but unnecessary reasoning traces. We apply format reward in GRPO, which did not improve accuracy but make the model follow the output template. Other standard make SFT hyper-parameters as close as possible GRPO, we reduce the batch size 32 during the experiment of SFT vs GRPO and reduce the learning rate correspondingly following the square root scaling rule (Li et 2024).
are shown in the last column of Table 5 To
A.5 PROMPTS FOR CAMERA AMBIGUITY ANALYSIS
As mentioned in Sec. 3.3 we compare different approaches for handling camera ambiguity study what the best approach is for VLMs. To add camera intrinsics information explicitly into the text prompt, follow the similar format of Sced1.5-VL. and ask: "Given this image of size (width W, height = H). where the camera intrinsics are (fx = A. fy = B, cx = C, = D) and the images are without distortions, how many meters is this point away from the camera?". To let the model predict camera intrinsics explicitly, we add in the answer of each sample the camera ray direction before the actual metric depth, so that the model can condition on the predicted intrinsics to decide the metric depth more accurately. Specifically, we set the answer to "The point is around X degrees to the righu/left, degrees above/below and Z meters away from the camera." " We do not predict the camera intrinsics (fx. fy, ex, cy) directly since adding the exact values of them into the input prompt do not help.
A.6 HYPER-PARAMETER FOR DEPTHLM TRAINING
See Table for the hyper-parameter of the models trained by SFT. For the 12B model from AAgrwall et al. 2024). we set the unified focal length to 750 pixels due to memory limits.
A.7 DEPTHLM SULTI-TASE PENCORMANCE ON EACH DATASET
We report in Table 6 the multi-task performance DepthLM on individual datasets.
Table 6: MultiTask performance on individual datasets.
Method
Out
Out+JIn ETH3D
In
Argo
DDAD
Nuscenes
ScanNet++
sunRGBD
ibims
NYLv2
Avg (t)
DISTANCE
0.809
0.728
0.850
0.708
0.856
0.843
0.950
0.876
0.828
PRINCIPAL AXIS DISTANCE
0.809
0.683
0.850
0.744
0.846
0.856
0.960
0.896
0.831
0.722
0.844
0.854
TIME
0,799
0.722
0.843
0.722
0.842
0.817
0.936
0.846
0.816
Two POINT DISTANCE
0.651
0.423
0.670
0.479
0.784
0.727
0.760
0.759
0.657
POSE
0.989
0.762
0.876
15


Reviews:
[{"review_id": "4LT0tckmS2", "reviewer": "Reviewer_5nYc", "summary": "This paper introduces DepthLM, a vision-language model designed for single-image depth estimation. The authors explore four key aspects: (1) how to prompt a VLM with a query point, (2) whether SFT or RL is more effective, (3) how to handle focal-length generalization, and (4) how much supervision per image is required. The method achieves promising results across experiments.", "questions": "see weakness", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Clear and focused presentation. The paper is well-structured and easy to follow. Each experimental question is clearly motivated, investigated, and summarized. The writing is concise and avoids unnecessary technical ornamentation, which I find appealing.\n\nInsightful experimental scope. The paper systematically studies several important dimensions of this task (prompting strategy, training scheme, focal length handling, and supervision density), offering valuable insights for the community.\n\nStrong empirical performance. The model achieves competitive or superior results compared to prior methods, demonstrating practical effectiveness.", "weaknesses": "Overemphasis on visual prompting novelty. Prompting in the visual space instead of textual space is useful, but not highly innovative by itself. The paper seems to overstate the novelty of this contribution, dedicating perhaps too much discussion to it relative to its conceptual depth.\n\nSFT vs. RL conclusion lacks depth. The conclusion that SFT outperforms RL feels premature. RL performance can vary significantly with algorithm choices (e.g., GRPO, PPO, DPO), reward design (e.g., continuous metric rewards vs. token-based or discrete depth-bin rewards), and training strategies (e.g., RL after SFT rather than combined in one stage). A more thorough investigation—or at least acknowledgment of these factors—would strengthen the claim.\n\nFocal-length generalization clarity. The discussion on focal-length generalization is somewhat unclear. It appears the model is trained with unified focal-length settings and expected to generalize to unseen focal lengths during inference. Clarifying the exact assumption and mechanism would be helpful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aNqYwL9chF", "reviewer": "Reviewer_r3mG", "summary": "The goal of this paper is to investigate whether it is possible to fine-tune a pre-trained VLM to achieve high accuracy in per-pixel metric depth estimation without modifications to the model architecture or training objective.\n\nThe authors fix the model architecture and conduct a set of experiments to investigate the best way to incorporate task-specific information for 3D depth estimation into the model's input prompt to achieve high-accuracy depth predictions. They fine-tune VLM to perform the 3D depth estimation task by generating a target distance in text modality.\n\n\nThe contributions of this work are as follows:\n1) an approach for training VLMs to achieve high accuracy in per-pixel depth estimation\n2) an accompanying benchmark dataset (mix of public datasets) adapted to the training and evaluation of VLMs on the 3D depth estimation task.", "questions": "1) Fine-tuned model does achieve the result comparable to specialized pure vision models, but it is fine-tuned just on the per-pixel depth estimation task. What is the difference vs expert pure vision approaches, except that the model is trained to output the text? Even in \"Beyond metric depth\" experiments, when the model is trained on five other tasks (which in fact are all depth estimation related), what original VLM properties, except executing these 5 tasks on which it was trained, are preserved?\n\n2) Related to the above question: L37-39: Is it still a VLM if it is trained to execute one or five specific tasks? \n\n3) L117-118: how numerical 3D labels are chosen?\n\n4) L324: \"We tune the prompt for VLMs that are not trained directly on our task to maximize their performance\". How was the prompt tuning done? Were all VLMs, selected as baselines, benchmarked using visual prompting?\n\n5) L420: It -> it.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1) Approach works with two different VLM architectures.\n2) It is the first VLM-fine-tuning-based approach for metric depth estimation that reaches specialised pure vision models.", "weaknesses": "1) It is not clear how the proposed pretraining unlocks 3D understanding. It looks like authors use the terms \"3D understanding\" and \"per-pixel 3D metric depth estimation\" interchangeably. 3D understanding is much broader than per-pixel 3D metric depth estimation. See questions.\n2) High training cost. The model is trained on \"\"128 H100 GPUs for about 2-4 days\"\" compared to 6 days on 16 NVIDIA 4090 with half precision (best monocular metric depth estimation baseline UniDepthV2).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cWekWNljgo", "reviewer": "Reviewer_w53k", "summary": "This paper introduces DepthLM, which enables Vision-Language Model (VLM) to perform metric depth estimation with accuracy comparable to specialized vision models. The authors identify that VLM’s weakness in depth estimation is due to issues in pixel reference and camera ambiguity. To address this, DepthLM employs visual prompting for pixel localization and intrinsic-conditioned augmentation to unify focal lengths across datasets. Using only sparse text-based supervision (1 or 2-4 labeled pixels per image in large-scale training ), DepthLM outperforms large proprietary models like GPT-5 and Gemini-2.5-Pro and reaches parity with state-of-the-art pure vision models such as UniDepth.", "questions": "1. In training, DepthLM conducts image resizing to unify focal length. Does inference also require unifying focal lengths, which requires camera intrinsics? If I understand correctly, unifying focal length during training is resizing all images to the same size, then by conducting random cropping, the model can generalize to different image sizes during inference. My question is, since the focal length is not unified during inference, could authors elaborate more about how the model can generalize to different image sizes with different focal lengths during inference?\n\n2. This paper mentions that “Image diversity is more important than label density”. The experiment provides only an increase in the label density after the training samples (I assume it means the number of images) larger than 16M. I am quite curious about the contribution of label density and image diversity to the model’s performance, respectively. Could the authors provide the results that show the performance under the combination of different image diversity and label density? For example, draw several lines for Figure 4 (c), each line means sampling different numbers of pixels per training image. I know this experiment would require extensive computational resources, so it’s totally OK if only part of such an experiment could be shown.\n\n3. How are the pixels sampled during evaluation? How many pixels are sampled per image, and how are they sampled? I didn’t find it in the paper.\n\n4. How are labelled pixels sampled in each image? (How) Will the sampling strategy affect the performance in training?", "limitations": "", "rating": 10, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. DepthLM achieves expert-level metric depth estimation without changing the architecture or using complex losses in mainstream VLM. It’s very astonishing that VLM can beat expert depth estimation models in monocular depth estimation.\n2. It’s also very astonishing that by querying a few pixels in each image, and tuning VLM to estimate depth for a few pixels of each image, VLM can beat expert depth estimation models. This finding is very interesting and meaningful.\n3. The DepthLM framework extends beyond monocular depth estimation to other 3D reasoning tasks, like spatial VQA or robot manipulation, using one unified vision-language model. I think if general VLM can be tuned to be very good at depth estimation, it can also benefit downstream application that requires precise distance or spatial measurement, like robot navigation and manipulation.", "weaknesses": "**Major:**\n\n1. The method requires significant computational resources. As stated in the paper, the training requires 23M-60M samples, with 2-4 pixels sampled per image, using 128 H100 GPU for 2-4 days. Nevertheless, I think if the training process can be integrated into the instruction-tuning stage of general VLM to improve depth perception ability, the computational overhead is acceptable.\n\n**Minor:**\n\nSince DepthLM explores leveraging VLM for depth estimation, the authors might consider discussing or citing a few related papers that also investigate VLM for depth estimation:\n1. Zhang, R., Zeng, Z., Guo, Z., & Li, Y. (2022, October). Can language understand depth?. In Proceedings of the 30th ACM International Conference on Multimedia (pp. 6868-6874).\n2. Hu, X., Zhang, C., Zhang, Y., Hai, B., Yu, K., & He, Z. (2024). Learning to adapt clip for few-shot monocular depth estimation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (pp. 5594-5603).\n3. Chen, W., Shi, C., Ma, C., Li, W., & Dong, S. (2024). DepthBLIP-2: Leveraging Language to Guide BLIP-2 in Understanding Depth Information. In Proceedings of the Asian Conference on Computer Vision (pp. 2939-2953).\n4. Zeng, Z., Wang, D., Yang, F., Park, H., Soatto, S., Lao, D., & Wong, A. (2024). Wordepth: Variational language prior for monocular depth estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 9708-9719).\n5. Cui, B., Huang, Y., Bai, L., & Ren, H. (2025). TR2M: Transferring Monocular Relative Depth to Metric Depth with Language Descriptions and Scale-Oriented Contrast. arXiv preprint arXiv:2506.13387.\n6. Zeng, Z., Wu, Y., Park, H., Wang, D., Yang, F., Soatto, S., ... & Wong, A. (2024). Rsa: Resolving scale ambiguities in monocular depth estimators through language descriptions. Advances in neural information processing systems, 37, 112684-112705.\n7. Zhang, W., Liu, H., Li, B., He, J., Qi, Z., Wang, Y., ... & Jin, X. (2025). Hybrid-grained Feature Aggregation with Coarse-to-fine Language Guidance for Self-supervised Monocular Depth Estimation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 6678-6692).\n8. Zeng, Z., Ni, J., Wang, D., Rim, P., Chung, Y., Yang, F., ... & Wong, A. (2024). PriorDiffusion: Leverage Language Prior in Diffusion Models for Monocular Depth Estimation. arXiv preprint arXiv:2411.16750.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)