Title: SSR: Enhancing Depth Perception in Vision-Language Models via Rationale-Guided Spatial Reasoning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Visual-Language Model
Abstract: Despite impressive advancements in Visual-Language Models (VLMs) for multi-modal tasks, their reliance on RGB inputs limits precise spatial understanding. Existing methods for integrating spatial cues, such as point clouds or depth, either require specialized sensors or fail to effectively exploit depth information for higher-order reasoning. To this end, we propose a novel Spatial Sense and Reasoning method, dubbed SSR, a novel framework that transforms raw depth data into structured, interpretable textual rationales. These textual rationales serve as meaningful intermediate representations to significantly enhance spatial reasoning capabilities. Additionally, we leverage knowledge distillation to compress the generated rationales into compact latent embeddings, which facilitate resource-efficient and plug-and-play integration into existing VLMs without retraining. To enable comprehensive evaluation, we introduce a new dataset named SSR-CoT, a million-scale visual-language reasoning dataset enriched with intermediate spatial reasoning annotations, and present SSRBench, a comprehensive multi-task benchmark. Extensive experiments on multiple benchmarks demonstrate SSR substantially improves depth utilization and enhances spatial reasoning, thereby advancing VLMs toward more human-like multi-modal understanding. Project page: [https://yliu-cs.github.io/SSR](https://yliu-cs.github.io/SSR).


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces SSR (Spatial Semantic Reasoning), a novel paradigm that redefines depth integration in vision-language models (VLMs) through interpretable and structured rationale-language representations. Furthermore, the proposed framework transforms raw depth data into latent embeddings via knowledge distillation, thereby reducing computational costs. SSR provides a plug-and-play method to improve the spatial reasoning capabilities of existing VLMs. Finally, for a comprehensive evaluation, the author constructs a large-scale vision-language dataset (SSR-COT) and a comprehensive multi-task benchmark (SSR-BENCH).\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. SSR demonstrates superior comprehension of depth maps and effectively enhances the spatial reasoning capabilities of the VLM.\n\n2. The collection, annotation, and quality evaluation pipeline for the large-scale vision-language dataset (SSR-CoT) are well-designed, providing high-quality spatial Chain-of-Thought (CoT) reasoning data.\n\n3. The SSRBench benchmark thoroughly assesses VLM spatial reasoning performance.\n\nMajor Weaknesses:\n\nQ1. My primary concern regards the training-free SSR_{PAP} module, which demonstrates only marginal performance gains on certain tasks, particularly on the proposed SSRBench. While this limited improvement is understandable given that pretrained VLMs have never seen similar depth latent features before, could the authors explain why the SSR method can operate training-free and be directly plugged into a pretrained VLM model?\n\nQ2. Following Q1, if SSR undergoes Stage 2 training, would this lead to pretrained knowledge forgetting or a degragation in inherent reasoning capabilities? For example, how would SSR-7B perform on other general- or common-sense reasoning benchmarks compared to its backbone, Qwen2.5-VL?\n\nQ3. For the method part, if SSR does not convert depth information into latent features but directly generates language representations as additional text input for the VLM, how would this affect spatial reasoning performance?\n\nMinor Weaknesses:\n\nQ4. In Table 3, while the 7B backbone (Qwen2.5-VL) outperforms the 3B version on all metrics, why does SSR-3B surpass SSR-7B on certain metrics? Is this due to random variability, or does the effectiveness of the proposed method (SSR) diminish as model parameters scale up?\n\nQ5. In Figure 1, the right-side VQA example contains an error: the question is multiple-choice, yet the model responds \"No.\"\n\nQUESTIONS:\nIf the authors can address all of my major concerns, I will raise my rating; otherwise, I will lower the current rating. Please correct me if I've misunderstood any aspects of the work.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on enhancing and evaluating spatial understanding leveraging depth images and reasoning traces or rationales. First of all, authors present two datasets, SSR-CoT and SSR-Bench, which are built semi-automatically from existing datasets, such as LLaVA-CoT, Visual-CoT, VO-CoT and SpatialQA. Both datasets are VQA datasets, where rationales are added together with the answers. SSR-CoT is built for training purposes, and SSR-Bench is directly extracted from SSR-CoT. As the second contribution, authors present a new framework to build VLMs with spatial understanding. The main idea is to leverage RGB and depth images together with rationales to train a new module called MIDI. MIDI is designed to produce latent tokens that aggregate the information from RGB, depth and answer rationales, which can be used as the input for a standard VLM for general VQA tasks. Authors evaluate their approach on various datasets, including their SSR-Bench showing improvements.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n1. The SSR-CoT and SSR-Bench datasets are interesting.\n2. Leveraging depth images estimated from monocular RGB images is also an interesting idea.\n3. The plug-and-play nature of the latent tokens learnt by MIDI, which could be used for any VLM (although this is not proven in the experimentation).\n\n**Weaknesses:**\n1. The writing of the paper can be improved in my opinion. The paper lacks clarity at many points and makes some claims that are very arguable in my opinion (I list some of those points in the Questions section). \n2. Many of the design decisions are not justified. Examples: i) the selection of vision encoders for RGB and depth images (why CLIP for one and SigLIP for the other?), ii) the usage of Mamba for the MIDI module (why not a standard transformer-based LM?), iii) why do they use an LLM to align MIDI latent tokens semantically and not the VLM itself (which has an LLM that could be used and seems a more natural option), iv) why do they use Qwen2.5 as the alignment LLM and not another LLM? I'm not saying the design decisions are wrong, but I think those decisions should be justified in the paper. \n3. The experimental methodology has many flaws from my point of view. \n- I think some baselines are missing to support the main claims of the paper, which are, to the best of my understanding, that depth images and language-based rationales are key to improve spatial understanding of VLMs. One of the baselines is Qwen2-5-VL (their base VLM) directly fine-tuned in SSR-CoT, without the MIDI component, but with the rationales provided by the dataset. Another important baseline could be their whole system but without the depth images. I think those baselines are needed to really understand the contribution of depth images and rationales in the results. \n- The selection of evaluation datasets is not justified and is arguable. I think that some spatial understanding datasets are missing, such as Visual Spatial Reasoning [1] and What's Up [2]. On the other hand, I don't understand why MME is used, when the dataset is not designed for spatial understanding (and I have no evidence spatial understanding is key for improved results; I may have missed something here). \n- The evaluation protocol is weak. They use a judge LLM, but they do not provide any human correlation information. They blindly believe the output of the judge, which is not a good practice. On the other hand, the selection of the judge LLM is not suitable. In general, LLMs tend to evaluate better their own outputs, and as far as I know, it is not recommended to use the same LLM family to evaluate competing LLMs. In this case, their system is based on Qwen2.5-VL, whose backbone LLM is Qwen-2.5, and the judge is also a Qwen2.5 LLM (the 14B variant).\n\n[1] Liu, Fangyu, Guy Emerson, and Nigel Collier. \"Visual spatial reasoning.\" Transactions of the Association for Computational Linguistics 11 (2023): 635-651.\n\n[2] Kamath, Amita, Jack Hessel, and Kai-Wei Chang. \"What’s “up” with vision-language models? Investigating their struggle with spatial reasoning.\" In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 9161-9175. 2023.\n\nQUESTIONS:\n1. In line 20, authors state: \"relying solely on RGB is inadequate for accurately capturing spatial information such as relative positions and distances\". I do not agree with this statement. Indeed, the depth information used by your system comes from RGB images, which means that in principle, that information could be represented by VLMs in their latent representations. Do you have any evidence to support this claim?\n2. From line 37 to line 43: I understand this text is written to motivate your work. However, I think it is too vague. You should better specify what VLMs are missing for spatial understanding. By the way, what does this sentence mean? \"existing methods incorporate depth explicitly without capitalizing on its inferential value\". Could you clarify it please?\n3. From line 47 to 53: similar to the previous. I think the text is too vague. You should better motivate the usage of language-based rationales. \n4. Typos: lines 77 and 78: there are two \"has\" that sould be \"have\".\n5. Line 113: Please, rewrite the sentence. \n6. Line 155: \"an efficient language within the MIDI\" -> \"an efficient language model within the MIDI\"?\n7. Equations 1 and 2 are the same loss function, aren't they? Why do you use two different names to refer to them? \"Standard causal modeling loss\" and \"standard cross-entropy loss for auto-regressive generation\". Did I miss anything here?\n8. Line 200: authors mention that they perform sampling quality checks. Can you explain how you do so?\n9. Line 245: detailed hyperparameters in Appendix A, but how did you set those hyperparameters? I couldn't find any information about the procedure. \n10. Table 4: I don't know MME in detail, but it's very weird to me to see the employed performance metric (MME reports summation). It is very difficult to understand the differences in performance between different systems. Could you clarify this point better, please?\n11. Table 5: It is not clear to me what PAP means here. I think you refer to the usage of latent tokens generated by MIDI as the result of the Stage 1 training, directly as the input of VLMs, but without any Stage 2 training. Is that right? In that case, please make it more clear in the text. \n12. Figure 6: I don't understand this figure and its motivation. Legends for the axes are missing and the text in Section 5.3 is not very descriptive. It may be my fault, but I read it three times and I don't understand what this is showing us.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to address a common limitation in existing vision-language models (VLMs), where they rely primarily on RGB images and thus lack accurate spatial understanding capabilities. To alleviate this limitation, the paper introduces the new Spatial Sense and Reasoning (SSR) approach, which leverages depth information to generate structured and interpretable rationales in text. Additionally, the approach also uses knowledge distillation to compress the abovementioned rationales into latent embeddings. To train the resulting model, the authors curate the new SSR-CoT dataset as well as introduce SSRBENCH which is a benchmark that evaluates on multiple downstream tasks.\n\nSTRENGTHS AND WEAKNESSES:\nStrength 1 - In terms of clarity and quality of the paper, the paper is relatively well-written and motivated. The model figures are informative and especially helpful in helping the reader to understand the proposed SELongVLM approach.\n\nStrength 2 - With regards to the significance of the work, this paper introduces an interesting method to integrate depth information, which is well-aligned with the latent language embedding space of the base LLM. This can be especially useful and practical since the introduced Mamba-based Image-Depth Interpreter (MIDI) modoule can be integrated into existing VLMS without having to retrain them. This helps to save time and cost for adapting such VLMs to new tasks or improving their performance directly on benchmarks. \n\nStrength 3 - This work also introduces a new large-scale training dataset SSR-CoT and the SSRBENCH evaluation benchmark. The former can be useful for the research community to integrate into their existing data curation and training pipelines for training VLMs that are more capable of spatial reasoning. \n\nWeakness 1 - The proposed SSR approach somewhat lacks novelty. While integrating depth information for spatial reasoning in VLMs effectively is an important research problem, there are already numerous recent works such as SpatialVLM, SpatialBot and SpatialRGPT that have explored adding depth and spatial information to VLMs. It is unclear how different SSR is from the abovementioned three works.\n\nWeakness 2 - More importantly and related to Weakness 1, the empirical results included in the paper do not compare to some VLMS that are specifically targeted at spatial reasoning, such as SpatialVLM and SpatialRGPT among many others. It will be much more insightful if such empirical comparisons are included. \n\nWeakness 3 - Finally, while the reported performance of SSR across multiple benchmarks appear to be strong, it is difficult to understand the relative importance of the different components proposed in the paper.  For instance, the paper does not include any detailed ablation on whether using depth information alone could provide substantial performance gains without generating the intermediate chain-of-thoughts rationale.\n\nQUESTIONS:\nPlease look at the abovementioned weaknesses. It will be much more insightful to include empirical comparisons to more VLMs that are targeted at spatial reasoning and more ablations.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nTo enhance spatial reasoning, SSR translates raw depth data into structured textual rationales as intermediate representations. These rationales are distilled into latent embeddings via the MIDI module and then passed as tokens into a vision-language model (VLM) to answer spatial reasoning questions. Experiments show that SSR significantly improves spatial reasoning performance across multiple tasks while preserving general VLM capabilities.\n\nIn addition, the authors introduce SSR-COT, a million-scale vision-language dataset with spatial annotations, and SSRBENCH, a comprehensive benchmark for evaluating spatial understanding. Together, these resources support robust evaluation and training of spatial reasoning models.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. Novel depth integration approach: The transformation of raw depth information into interpretable textual rationales offers a more meaningful way to incorporate spatial cues compared to direct depth feature fusion, enabling better spatial reasoning.\n\n2. Comprehensive evaluation framework: The introduction of both a large-scale dataset (SSR-COT with over 1 million samples) and the SSRBENCH benchmark provides valuable resources for the community to evaluate spatial reasoning capabilities.\n\nWeaknesses\n\n1. Limited analysis of failure modes: The paper lacks a qualitative discussion of when and why the proposed approach fails, or how it improves upon baseline models in specific scenarios.\n\n2. Unclear computational overhead: While the paper introduces the MIDI module, it does not provide a clear analysis of inference-time computational cost or memory usage compared to a vanilla vision-language model.\n\nQUESTIONS:\n1. Could the authors provide qualitative examples or a detailed analysis of common failure cases to better understand when and why the method underperforms and outperforms baseline models?\n\n2. How does the proposed MIDI module affect efficiency? Can the authors report the computational or memory overhead compared to a standard vision-language model during training and inference?\n\nLIMITATIONS:\nLimitations have been discussed.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-03"}