Understanding Depth and Height Perception in Large Visual-Language Models

## Abstract

Abstract Geometric understanding is crucial for navigating and interacting with our environment. While large Vision Language Models (VLMs) demonstrate impressive capabilities, deploying them in real-world scenarios necessitates a comparable geometric understanding in visual perception. In this work, we focus on the geometric comprehension of these models; specifically targeting the depths and heights of objects within a scene. Our observations reveal that, although VLMs excel in basic geometric properties perception such as shape and size, they encounter significant challenges in reasoning about the depth and height of objects. To address this, we introduce GeoMeter , a suite of benchmark datasets—encompassing Synthetic 2D, Synthetic 3D, and Real-World scenarios—to rigorously evaluate these aspects. We benchmark 17 state-of-the-art VLMs using these datasets and find that they consistently struggle with both depth and height perception. Our key insights include detailed analyses of the shortcomings in depth and height reasoning capabilities of VLMs and the inherent bias present in these models.
This study aims to pave the way for the development of VLMs with enhanced geometric understanding, crucial for real-world applications.

## 1 Introduction

In recent years, the AI community has significantly focused on integrating visual and natural language inputs, notably in Visual Question Answering (VQA) systems. These systems analyze images and answer questions about them, showing substantial advancements in understanding basic visual concepts such as shape identification [], object detection [], and the spatial relationships [, , ] by using large Visual Language Models (VLMs). These models have excelled in processing complex text and visual inputs due to their strong visual understanding capability, leading to applications in image captioning, visual question answering, image text retrieval, and so on.

The ability to understand visual properties such as size, shape, depth, and height is fundamental to visual understanding, yet many existing Visual Question Answering (VQA) benchmarks [, , , , ] do not specifically focus on the depth and height perception capabilities of Vision Language Models (VLMs). Accurate perception of these dimensions is vital for practical applications like surveillance, navigation, and assistive technologies. The lack of accurate depth and height understanding in VLMs can lead to serious consequences, such as misjudging the proximity of objects, which could result in catastrophic outcomes in real-world scenarios.

Despite VLMs’ abilities to recognize object shapes and sizes, their depth and height reasoning often relies on learned size/shape cues rather than actual spatial analysis, potentially influenced by biases from training data []. Alternatively, models might estimate the depth based on the apparent size of objects, without genuine inter-object reasoning. Additionally when faced with multiple choices, VLMs might also show bias towards certain answers, influenced by the prevalence of similar data during training. An example illustrated in Figure 1 shows one of the most popular closed-source VLM, GPT-4V [] incorrectly assessing the depth relationship between two cats, highlighting the model’s reliance on visual cues that conflict with the actual spatial arrangement. The other wrong perception examples of GPT-4V shown in Figure 1 underscores the need for more focused benchmarks and training approaches that enhance true depth and height perception in VLMs, ensuring they perform reliably in complex, real-world environments.

In this paper, we propose GeoMeter, a benchmark specifically designed to evaluate the depth and height reasoning capabilities of Vision Language Models (VLMs).
GeoMeter comprises approximately 4.04k unique images and 11.3k image-text pairs across three distinct datasets: Synthetic 2D, Synthetic 3D, and Real-World. The synthetic datasets focus on basic 2D and 3D shapes like rectangles, circles, cubes, cylinders etc.
The Real-World dataset consists of image captures from indoor scenes.
The development of synthetic datasets featuring basic shapes aims to genuinely test the visual reasoning capabilities of models, focusing on their ability to process visual information without relying on familiar real-world cues. Conversely, the real-world dataset is designed to assess how well models can discern depth and height in new, previously unseen images. Our motivation comes from concerns about test time data leakage that could arise when models, trained on vast existing datasets, encounter images during testing that they may have already seen during training. By using unique datasets, we seek to ensure a more accurate evaluation of the model’s true visual interpretation abilities.

We extensively analyze our proposed benchmarks on 17 recent open-source and closed-source models for the VQA task. Our findings indicate that from the studied models, (1) Even though VLMs have visual reasoning capability in terms of basic geometric understanding, they struggle in depth and height perception. (2) Closed-source models exhibit a greater performance gap between synthetic and real data compared to open models. (3) Generally models show better depth perception than height. (4) Models show inherent biases towards certain options when presented with advanced perception tasks.

Overall, our contributions can be summarized as follows:

- •
We introduce GeoMeter, a new benchmark study aimed at probing depth and height awareness of VLMs.
- •
We propose 3 datasets to study this problem, namely, Synthetic 2D, Synthetic 3D, and Real-World.
- •
We present an extensive analysis on depth and height perception on 17 open and closed-source VLMs to gain insight on these models’ behavior and their inherent biases.

## 2 Related Works

##### Visual Language Models (VLMs).

The field of AI has undergone a significant transformation with the advent of vision language models (VLMs), which are trained on extensive multimodal datasets and are versatile across numerous applications [, ]. These models have shown remarkable performance in language and vision-related tasks, e.g. recognition, reasoning, etc. VLMs are models with a pre-trained LLM backbone and a vision encoder; which are aligned by using different methods. Recent closed-source VLMs such as GPT-4 [], Gemini [], Claude [] showcase a strong potential for tasks that require understanding and processing information across different modalities. Additionally, various openly available VLMs such as LLaVA [], LLaVA-NeXT [], Bunny [] etc. also have comparative performance with the closed-source models across different vision-language tasks. All of these VLMs are trained on massive amount of public and proprietary data, making them a strong performer of general reasoning.

##### Visual Question Answering.

Several works and benchmarks have probed VLMs to understand what they are learning in terms of spatial reasoning, object understanding, object-attribute relationship [, , , , , , , ] and geometric property understanding [, , ]. Various works have also explored the visual limitations of VLMs [, , , ]. Most of these benchmarks contain generic questions which can be used to probe the VLMs spatial reasoning and visual understanding and VLMs limitation in these tasks. However, these are insufficient to understand whether or not models truly understand advance concepts like depth and height.
Even though the VQA task has been explored for a long time, we see that the general understanding of VLMs in the context of geometric properties like depth and height perception is rather unexplored. Our proposed benchmark contains image-text pairs that probe the depth and height perception of the VLMs without requiring mathematical knowledge.

## 3 Benchmark and Evaluation

### 3.1 Datasets

Our benchmark consists of three datasets: Synthetic 2D, Synthetic 3D, and Real-World. They are designed to test model performance on depth and height perception tasks, utilizing unique identifiers as diverse query attributes for question generation. Table 1, Figure 2 and Figure 3 respectively show the dataset statistics, sample images and sample image-text pair of our proposed datasets. More samples from each dataset is given in the supplementary. The dataset generation can be divided into two parts - Image generation (Section 3.1.1) and Question generation (Section 3.1.2).

**Table 1: Dataset statistics of our proposed benchmark. Here Query attributes are unique identifiers for the object of interest. MCQ and T/F respectively denote Multiple Choice Questions and True/False questions.**
| Dataset | Category | Task | Images | Question Type | Questions | Query attributes | Img-Text pairs |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Synthetic 2D | Depth | VQA | 1200 | MCQ, T/F | 2400 | Color, Numeric label (random and patterned) | 4800 |
| Height | 1200 | 2400 | Color, Numeric label (random) |  |  |  |  |
| Synthetic 3D | Depth | VQA | 800 | MCQ, T/F | 3200 | Color, Material | 6400 |
| Height | 800 | 3200 |  |  |  |  |  |
| Real-world | Depth | VQA | 43 | MCQ | 30 | Numeric label (random) | 100 |
| Height | MCQ | 70 |  |  |  |  |  |

Figure: Figure 2: Samples from the proposed benchmarks. Here each samples are shown with random query attributes- color and numeric label for Synthetic 2D, color and material for Synthetic 3D and numeric label for Real-World dataset.
Refer to caption: extracted/5823729/images/Geom_Bench_image_stuff-23.png

Figure: Figure 3: Sample image-text pair from the Synthetic 2D dataset. Here the image contains 5 shapes and labeled with random numeric labels which are used as query attributes in the prompt. Prompt template shows the basic template for each image-text pair of all our benchmark, where the prompt example is the actual prompt for this image. The prompt example is appended with either MCQ or True/False type question.
Refer to caption: extracted/5823729/images/Geom_Bench_image_stuff-21.png

#### 3.1.1 Image Generation

Our proposed synthetic datasets are divided into two categories - Depth and Height, with each image containing a real-world scene as a background to enhance realism.

Synthetic 2D: The Synthetic 2D dataset includes 2400 images and 4800 unique questions. The Depth category consists of 1200 images (600 with 3 shapes and 600 with 5 shapes), featuring rectangles, triangles, or circles that partially overlap to create a depth illusion, with unique identifiers such as colors, and numeric labels.
The Height category also has 1200 images (600 with 3 towers and 600 with 5 towers), where each tower consists of four rectangles with random dimensions. In half of the images one of the towers is placed on a horizontal black strip that is treated as a raised platform. This category includes two sets: one with all towers at the same height and another with a randomly chosen tower on a raised platform, with unique identifiers being color and label. All of the towers are labeled sequentially.

Synthetic 3D: The Synthetic 3D dataset comprises 1600 images and 6400 unique questions, created based on the existing CLEVR dataset []. Images are generated by randomly sampling a scene graph and rendering it using Blender [], with random jittering of light and camera positions. Unique identifiers for the objects include color and material (shiny "metal" and matte "rubber"). The Depth category contains 800 images (400 with 3 shapes and 400 with 5 shapes) featuring randomly positioned cubes, spheres, and cylinders. These shapes are colored from a palette of eight colors and two materials, with increased horizontal and vertical margins than original CLEVR images between objects to reduce ambiguous spatial relationships. The Height category includes 800 images (400 with 3 towers, and 400 with 5 towers). Each tower consists of four cubes, with random sizes and colors. In some images, the bottom-most cube is black and matte "rubber" material to denote it as an elevated base-plane, making that tower essentially 3 cubes high.

Real-world Dataset: The Real-world dataset comprises of 43 images and 100 unique questions across both depth and height category that we have collected featuring commonly used office objects in their typical settings. Instead of labeling specific objects in each image, we have designated random regions with arbitrary numeric values. This approach is intended to test the models’ ability to associate regions with labels, which may or may not include recognizable objects. The objective is to assess the model’s genuine capability in depth and height perception, requiring it to rely on actual visual reasoning for understanding spatial dimensions of regions.

#### 3.1.2 Question Generation

The method used for generating questions is consistent across all our proposed datasets. Each question is a Description prompt appended with an Answer format instruction. The description prompt contains some general information about the scene providing semantic cues to the given image; followed by the actual question and answer format instruction. For example,
"[more information] Provide depth/height ordering for the shapes <question items> in the image. [more information]" is a descriptive prompt. This is followed by "From the given options: <answer set>, select the correct answer [more information]." which is an answer format instruction.

The question items is a list containing <query attribute> appended by <shape>. Here <query attributes> is one of the unique identifiers of the dataset.
For example in the question item "green metal cube", "green metal" is the <query attribute> and <cube> is the shape. The answer set contains all possible valid values (<query attribute> + <shape>) to that given prompt. To generate both the question items and answer set, we read through the scene graph and run depth-first search on it to generate valid unambiguous values of object-pair relationship. For each image, there are two types of questions - MCQ and True/False.

### 3.2 Model Variants

We perform our benchmark evaluation on 17 state-of-the-art multi-modal models. All of our chosen VLMs are trained on very large (public and/or proprietary) datasets. The selected VLMs can be categorized into 14 open-source and 3 closed-sourced models.

LLaVA, LLaVA-NeXT [, ] are a family of large open-source multimodal models capable of visual reasoning. It connects the CLIP visual encoder [] with the Vicuna language decoder []. We evaluated our benchmark on the following LLaVA model: LLaVA 1.5 7B, LLaVA 1.5 13B; and LLaVA-NeXT models: LLaVA 1.6 Mistral 7B, LLaVA 1.6 Vicuna 7B, LLaVA 1.6 Vicuna 13B.

Fuyu-8B [] is a more efficient open-source multimodal model that uses a decoder-only transformer architecture. Unlike traditional multimodal models, it bypasses the need for an image encoder by linearly projecting image patches into the transformer’s first layer, supporting arbitrary image resolutions and reducing both training and inference complexity.

Bunny [] is a lightweight open-source family of multimodal models offering flexible combinations of vision encoders and LLM backbones, aligned through a cross-modality projector. We evaluated Bunny-v1.0-3B, Bunny-v1.0-4B, Bunny-v1.1-4B, and Bunny-Llama-3-8B-V.

InstructBLIP [], another open-source family of models leverage the BLIP-2 [] architecture for visual instruction tuning, with the distinction that the text prompt is also fed to the Q-Former. We evaluated InstructBLIP-Vicuna-7B and Instruct-BLIP-Flan-T5-XL.

LLaMA-Adapter[] is a parameter-efficient visual instruction model with superior multimodal reasoning, fine-tuning LLaMA []. We evaluated LLaMA-Adapter v2-multimodal.

MiniGPT-4 [] aligns a frozen visual encoder from BLIP-2 [] with the frozen Vicuna LLM using a projection layer for multimodal visual reasoning tasks.

GPT-4 [] is a closed-source multimodal conversational model by OpenAI, based on a transformer architecture, pre-trained on large datasets and fine-tuned with Reinforcement Learning from Human Feedback (RLHF) []. We evaluated GPT-4V, and GPT-4o.

Claude [] is a closed-source multimodal model by Anthropic with competitive performance against other closed-source models. We evaluated Claude 3 Opus.

### 3.3 Evaluation Metrics

We evaluate our benchmark on the task of visual question answering (VQA), with accuracy being the performance metric on MCQ and True/False type questions. Evaluation is done across query attributes and number of shapes on probing the VLMs’ depth and height perception.

### 3.4 Implementation Details

All models are used in accordance to the provided evaluation code and model weights. The closed-source models were accessed through APIs which have been provided through a paywall by the corresponding developing team of those models. For the True/False questions, the ground truth is randomly selected between True and False. For MCQ, the order of the given options are randomly generated, and ground truth is always randomly placed in one of those options.

**Table 2: Performance comparison of the studied models on proposed benchmark. The reported results are averaged across depth and height category, query attributes and shapes with top scores in bold. Here, T/F denotes True/False type questions.**
|  | Model | Synthetic 2D | Synthetic 3D | Real-World |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  | MCQ | T/F | MCQ | T/F | MCQ |  |
| Open | LLaVA 1.5 7B | 28.8 | 50.5 | 28.0 | 49.8 | 53.0 |
| LLaVA 1.5 13B | 17.8 | 52.5 | 29.0 | 51.3 | 47.0 |  |
| LLaVA 1.6 Mistral 7B | 22.1 | 52.2 | 26.7 | 48.7 | 39.0 |  |
| LLaVA 1.6 Vicuna 7B | 17.1 | 51.7 | 28.6 | 50.0 | 45.0 |  |
| LLaVA 1.6 Vicuna 13B | 28.2 | 54.2 | 32.5 | 52.7 | 48.0 |  |
| Bunny-v1.0-3B | 24.1 | 50.1 | 17.1 | 37.1 | 45.0 |  |
| Bunny-v1.0-4B | 24.2 | 52.6 | 19.9 | 39.3 | 52.0 |  |
| Bunny-v1.1-4B | 26.6 | 52.3 | 26.9 | 44.4 | 51.0 |  |
| Bunny-Llama-3-8B-V | 27.9 | 50.2 | 26.9 | 43.2 | 47.0 |  |
| Fuyu-8B | 8.6 | 53.0 | 19.4 | 43.2 | 30.0 |  |
| InstructBLIP-Flan-T5-XL | 10.8 | 47.4 | 37.5 | 52.1 | 41.0 |  |
| InstructBLIP-Vicuna-7B | 28.3 | 49.0 | 38.1 | 53.8 | 39.0 |  |
| LLaMA-Adapter-v2-Multimodal | 22.9 | 48.8 | 32.7 | 52.4 | 38.0 |  |
| MiniGPT-4 | 25.0 | 50.4 | 39.4 | 56.3 | 40.0 |  |
| Closed | GPT-4V | 25.5 | 54.0 | 35.2 | 50.5 | 61.0 |
| GPT-4o | 30.8 | 56.7 | 38.5 | 52.4 | 60.0 |  |
| Claude 3 Opus | 29.0 | 51.9 | 36.2 | 49.9 | 48.0 |  |

Figure: Figure 4: Depth and height perception performance on the proposed Synthetic 2D, Synthetic 3D and Real-World dataset on MCQ and True/False (T/F) questions. D, H, A respectively denote depth, height and average performance. For example, 2D(D) MCQ and 2D(H) MCQ corresponds to respectively Synthetic 2D depth and height performance on MCQ questions. Y-axis denotes the average performance across shape and query attributes and X-axis denotes all the evaluated models.
Refer to caption: extracted/5823729/images/Unknown-91.png

### 3.5 Results

The performance of the selected models on the VQA task for MCQ and True/False type questions on the proposed benchmarks are shown in Table 2, where each row corresponds to the average accuracy across all different query attributes and shapes. Depth and height category wise results are presented in Figure 4. Additional results across all query attributes and shapes are reported in the supplementary.

## 4 Analysis and Discussion

### 4.1 Model Behavior Analysis

Models show basic visual reasoning capability but struggles in advance perception tasks.
We developed a specialized dataset called Synthetic 2D Basic containing 30 image-text pairs (some samples shown in Figure 5 left)to evaluate the fundamental visual reasoning capabilities of Vision Language Models (VLMs). This dataset focuses on basic geometric tasks like line understanding, shape recognition, shape counting, and assessing spatial relationships between shapes. The initial assessments using MCQs demonstrate high performance by models on these basic tasks, as detailed in Figure 5 right. Despite this proficiency in simple visual properties, results from Figure 4 highlight that these same models struggle significantly with depth and height perception tasks involving similar shapes. This discrepancy underscores the benchmark’s value in identifying gaps in VLMs’ capabilities to handle more complex spatial reasoning, beyond mere shape recognition.

Figure: Figure 5: Model behavior on basic understanding of shapes and size on our created Synthetic 2D Basic dataset (samples on the left). Performance of selected models on this dataset is shown in right. Here, LU, SI, SC and SR respectively denote line understanding, shape identification, shape counting and spatial reasoning. Y-axis denotes performance accuracy of different categories and X-axis denotes evaluated models.
Refer to caption: extracted/5823729/images/Geom_Bench_image_stuff-36.png

Closed models exhibit a greater performance discrepancy between synthetic and real data compared to open models.

Figure: Figure 6: Performance comparison of open and closed models on synthetic and real data. Here synthetic data performance denotes average performance of both synthetic 2D and 3D data. There is greater discrepancy between synthetic and real data performance in closed models than open models.
Refer to caption: extracted/5823729/images/Geom_Bench_image_stuff-34.png

Figure: Figure 7: Height perception of open and closed models on Synthetic 2D and Real-World data. Here we show the prediction LLaVA 1.5 7B and GPT 4o. Although GPT 4o can do accurate height prediction on the real-world data, it fails on the same task on the synthetic data; indicating discrepancy in performance.
Refer to caption: extracted/5823729/images/Geom_Bench_image_stuff-29.png

The performance data from Figures 4 and 6 shows clear differences in how closed and open models perform on synthetic versus real datasets. Closed models excel in real-world scenarios, likely due to training on proprietary datasets, but show less adaptability on synthetic data, demonstrating a notable performance gap. One such scenario is depicted in Figure 7 where GPT-4o performs well in real-world height perception task but struggles with similar task in synthetic settings. In contrast, open models, while generally less accurate, exhibit a smaller performance disparity between synthetic and real data; however, this does not make them superior over closed models. Rather it shows their general lesser perception of depth and height in both synthetic and real world setting.

The data shown in Figure 4 indicates that both open and closed models struggle more with height perception compared to depth in MCQ type questions, with depth perception generally aided by occlusion cues and height perception challenged by the complexity of assessing vertically stacked objects. This disparity suggests that models are more adept at interpreting scenarios with obscured views than accurately evaluating detailed vertical arrangements. The comparatively poor performance in height perception is likely due to gaps in training, such as a lack of emphasis on diverse height configurations and potential biases in model architecture towards simpler visual cues. Overall, while models perform better in depth perception, they still show limitations in comprehensively handling more complex spatial tasks, underscoring an area for improvement in advanced spatial property perception.

Closed models are more robust to increased scene density compared to open models. Figure 8 shows that the performance of models on MCQs assessing the number of shapes in Synthetic 2D and 3D datasets declines as scene density increases from 3 to 5 shapes. Open-source models like LLaVA and Bunny experience a more pronounced performance drop with increased scene complexity, while closed-source models demonstrate better resilience, suggesting they are more capable of handling visual reasoning in denser environments.

Figure: Figure 8: Performance comparison of depth perception with varying scene density. Here, Y-axis denotes the performance on scene complexity of Synthetic 2D and Synthetic 3D datasets’ on MCQ questions. The X-axis denotes all the evaluated models.
Refer to caption: extracted/5823729/images/Unknown-93.png

Model performance is generally not influenced by query attributes.
The performance analysis in Figure 9 reveals that changing query attributes generally does not significantly affect the average performance of most models across depth and height categories. Both the Synthetic 2D and 3D datasets show stable performance across attributes like color and label or color and material, indicating that models consistently handle different attributes within these visual categories.

Figure: Figure 9: Performance comparison of depth perception with varying query attributes. Here, Y-axis denotes the performance on query attributes of Synthetic 2D and Synthetic 3D datasets’ on MCQ questions. The X-axis denotes all the evaluated models.
Refer to caption: extracted/5823729/images/Unknown-94.png

### 4.2 Model Bias Analysis

We conducted further analysis on the type of prompts to study any inherent biases in the models could be influencing their performance on MCQ and True/False type questions on a smaller subset (1600 image-text pairs uniformly selected from the depth and height categories) of the Synthetic 3D dataset.

Some open-source models are more biased towards picking True over False than others.
The performance of some open-source models on True/False questions tends to hover around 50% (Table 2), suggesting they might not be effectively distinguishing between true and false statements, potentially defaulting to random guesses. This is highlighted by experiments showing similar outcomes (Figure 10 left) when ground truth is random versus always set to "True," and a significant performance decline when it is always "False," indicating a bias towards predicting "True." Models like Bunny and Fuyu exhibit the most substantial drops, suggesting a stronger true bias in open-source models compared to closed-source models, which generally show greater resilience to this bias. This pattern points to a lack of robust decision-making in handling True/False questions among these models.

Figure: Figure 10: Model bias analysis. Left: Effect of ground truth value in True/False questions. GT-R denotes randomly set ground truth between true and false; whereas GT-T/F denotes ground truth always true or always false. Right: Effect of ground truth ordering in choices of MCQs. GT-C1 and GT-Ab denotes ground truth being choice 1 and not present respectively. The Y-axis denotes the average performance and X-axis denotes all the evaluated models.
Refer to caption: extracted/5823729/images/Unknown-100.png

Some open source models are more biased towards picking the first choice in case of MCQ.

Experiments reveal that while closed-source models show consistent performance across various MCQ ground truth placements, open-source models like Bunny and Fuyu display a significant bias towards selecting the first option, especially when ground truth is positioned as the first choice (Figure 10 right). Their performance drops when the correct option is absent, indicating a struggle with correctly identifying the true answer or a "None of the above" choice, suggesting a tendency towards random selections. The LLaVA models also prefer the first option, but to a lesser extent, highlighting a general bias in open-source models towards the first choice in MCQ settings.

## 5 Conclusion

In this benchmark, we evaluated large VLMs on depth and height perception across three new datasets: Synthetic 2D, Synthetic 3D, and Real-World. Our comprehensive evaluation highlights several key findings: (1) Models display basic visual reasoning capabilities but struggle with advanced perception tasks like depth and height understanding. (2) Closed models exhibit a greater performance discrepancy between synthetic and real data compared to open models, suggesting lesser robustness in synthetic environments. (3) Models generally perform better in depth perception than in height. (4) There is an inherent bias in how models respond to advanced perception tasks.
While our study focused primarily on depth and height, the exploration of broader geometric reasoning aspects remains a promising area for future research. Enhancing these capabilities can significantly improve the utility of VLMs in real-world applications, marking an exciting direction as these models continue to advance.

## 6 Additional Results

In this section we will provide additional results for the different dataset benchmarks.

### 6.1 Quantitative Evaluation

Table 3, Table 4 present detailed results for the synthetic 2D dataset; and Table 5, Table 6 present detailed results for the synthetic 3D dataset. All of these results examine the impact of scene complexity (3 shapes vs 5 shapes), query attributes (color, labels), and question types (MCQ and True/False) on depth and height perception (respectively). While the main paper reports average results, the individual category-specific outcomes offer deeper insights. For instance, performance deteriorates with increased scene complexity (5 shapes) for many open-source models, highlighting the superior robustness of closed-source models under these conditions. Additionally, changes in query attributes show minimal impact on performance for most models, indicating their resilience to variations in query types. Additionally, Figure 11 presents the accuracy performance for True/False questions within the Real-World dataset. Consistent with previous results for True/False questions, this figure also indicates that the performance typically hovers around 50% accuracy for this dataset. However, the performance of the closed models are much better than that of the open models, proving their superiority over the open models on real-world data.

### 6.2 Qualitative Examples

Figure 14 displays sample predictions from both open and closed models, highlighting their challenges with depth and height perception. The examples particularly emphasize the models’ inaccuracies, especially in synthetic data scenarios, showcasing their limitations in spatial understanding. This figure includes predictions from the best-performing models in both the open (LLaVA 1.5 7B) and closed (GPT 4o) categories.

Figures 15 and 16 present examples from the Synthetic 2D dataset, including the specific prompts for both MCQ and True/False questions, serving as visual aids for the evaluations discussed. Similarly, Figures 17 and 18, along with Figures 19 and 20, showcase samples and corresponding prompts from the Synthetic 3D and Real-World datasets, respectively. These figures provide insights into the different scenarios and questions used to assess depth and height perception across various data types. Additionally, Figure 21 features image-text pairs from the Synthetic 2D Basic dataset, highlighting the initial stages of evaluating the models’ capabilities in recognizing basic properties. This collection of figures effectively illustrates the range and focus of the datasets employed to test the perceptual abilities of the models.

**Table 3: Performance of the studied models on proposed Synthetic-2D depth category. Evaluation is done on the VQA task on MCQ and True/False type questions. Color, RL, PL are the query attributes. Here, RL, PL respectively denotes random numeric label, patterned numeric label.**
|  | Model | Depth-3 shapes | Depth-5 shapes |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | MCQ | T/F | MCQ | T/F |  |  |  |  |  |  |  |  |  |
|  |  | Color | RL | PL | Color | RL | PL | Color | RL | PL | Color | RL | PL |
| Open | LLaVA 1.5 7B | 48.0 | 37.5 | 54.5 | 49.0 | 54.5 | 47.0 | 36.5 | 31.0 | 39.0 | 45.0 | 56.0 | 49.5 |
| LLaVA 1.5 13B | 36.5 | 21.0 | 29.0 | 52.0 | 57.0 | 54.0 | 35.5 | 15.0 | 11.0 | 54.5 | 53.0 | 54.0 |  |
| LLaVA 1.6 Mistral 7B | 44.0 | 34.5 | 25.0 | 55.5 | 54.5 | 52.5 | 28.5 | 24.0 | 11.0 | 54.0 | 56.0 | 54.0 |  |
| LLaVA 1.6 Vicuna 7B | 37.0 | 20.5 | 13.0 | 54.5 | 50.5 | 49.5 | 29.0 | 7.0 | 1.0 | 50.5 | 52.5 | 55.0 |  |
| LLaVA 1.6 Vicuna 13B | 35.0 | 42.0 | 62.0 | 45.5 | 53.5 | 72.0 | 28.0 | 35.5 | 32.0 | 56.0 | 54.0 | 62.5 |  |
| Bunny-v1.0-3B | 41.5 | 40.5 | 38.5 | 48.0 | 45.5 | 54.0 | 31.0 | 30.0 | 13.5 | 46.5 | 52.5 | 55.0 |  |
| Bunny-v1.0-4B | 38.0 | 47.0 | 33.5 | 55.5 | 55.5 | 55.5 | 26.5 | 29.5 | 22.5 | 52.5 | 53.0 | 53.0 |  |
| Bunny-v1.1-4B | 45.5 | 47.5 | 33.5 | 52.5 | 55.5 | 55.5 | 34.0 | 36.0 | 31.5 | 52.5 | 53.0 | 53.0 |  |
| Bunny-Llama-3-8B-V | 34.5 | 45.0 | 46.0 | 41.0 | 58.5 | 51.5 | 27.5 | 36.5 | 48.0 | 48.5 | 53.5 | 46.0 |  |
| Fuyu-8B | 33.5 | 17.0 | 4.5 | 58.5 | 55.5 | 55.5 | 30.0 | 15.5 | 3.0 | 53.5 | 53.0 | 53.0 |  |
| InstructBLIP-Flan-T5-XL | 45.5 | 8.5 | 0.0 | 44.5 | 44.5 | 44.5 | 32.0 | 40.0 | 0.0 | 47.0 | 47.0 | 47.0 |  |
| InstructBLIP-Vicuna-7B | 43.5 | 40.0 | 59.0 | 49.5 | 44.0 | 43.0 | 32.0 | 31.0 | 34.0 | 46.5 | 47.5 | 46.0 |  |
| LLaMA-Adapter-v2-Multimodal | 41.0 | 40.0 | 39.5 | 48.5 | 45.5 | 45.5 | 31.0 | 30.0 | 33.0 | 47 | 45.5 | 45.5 |  |
| MiniGPT-4 | 42.0 | 41.5 | 43.0 | 52.0 | 51.5 | 51.5 | 34.0 | 32.0 | 30.0 | 48.5 | 47.5 | 47.5 |  |
| Closed | GPT-4V | 45.0 | 49.0 | 41.5 | 54.5 | 57.0 | 61.5 | 38.5 | 37.0 | 40.5 | 56.0 | 58.5 | 53.0 |
| GPT-4o | 47.5 | 44.5 | 47.0 | 55.5 | 58.5 | 70.5 | 49.5 | 36.5 | 36.0 | 62.0 | 59.0 | 52.0 |  |
| Claude 3 Opus | 47.5 | 40.5 | 50 | 51.5 | 51.5 | 56.5 | 36.5 | 36.0 | 41.0 | 52.5 | 51.5 | 56.0 |  |

Figure: Figure 11: Performance of Real-World dataset on True/False type questions. Here Y axis denotes accuracy whereas X axis denotes the models evaluated. Difference between average accuracy between open and closed models denote superiority of closed models on real-world data.
Refer to caption: extracted/5823729/images/Unknown-102.png

**Table 4: Performance of the studied models on proposed Synthetic-2D height category. Evaluation is done on the VQA task on MCQ and True/False type questions. Color, Label are the query attributes. Here, SP, $\overline{\mbox{SP}}$ respectively denote w/ step, and w/o step.**
|  | Model | Height-3 towers $\overline{\mbox{SP}}$ | Height-3 towers SP |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | MCQ | T/F | MCQ | T/F |  |  |  |  |  |
|  | Color | Label | Color | Label | Color | Label | Color | Label |  |
| Open | LLaVA 1.5 7B | 15.5 | 18.0 | 50.0 | 54.0 | 21.0 | 16.5 | 49.5 | 57.0 |
| LLaVA 1.5 | 15.5 | 9.0 | 49.0 | 54.0 | 14.5 | 10.0 | 49.0 | 56.9 |  |
| LLaVA 1.6 Mistral 7B | 16.0 | 17.0 | 50.5 | 55.5 | 14.0 | 15.5 | 49.5 | 53.0 |  |
| LLaVA 1.6 Vicuna 7B | 14.0 | 19.0 | 49.0 | 55.0 | 18.5 | 18.0 | 50.0 | 58.0 |  |
| LLaVA 1.6 Vicuna 13B | 19.0 | 19.0 | 49.5 | 54.0 | 13.5 | 20.5 | 49.5 | 57.0 |  |
| Bunny-v1.0-3B | 13.5 | 17.5 | 49.0 | 51.0 | 18.5 | 20.0 | 49.0 | 57.0 |  |
| Bunny-v1.0-4B | 18.0 | 16.5 | 49.0 | 54.0 | 16.0 | 12.5 | 49.0 | 57.0 |  |
| Bunny-v1.1-4B | 11.0 | 18.5 | 49.0 | 54.0 | 19.0 | 15.0 | 49.0 | 57.0 |  |
| Bunny-Llama-3-8B-V | 15.0 | 15.5 | 49.0 | 54.5 | 14.5 | 18.0 | 49.0 | 53.5 |  |
| Fuyu-8B | 0.0 | 0.0 | 45.5 | 55.0 | 0.0 | 0.0 | 53.5 | 55.0 |  |
| InstructBLIP-Flan-T5-XL | 0.5 | 0.5 | 51.0 | 46.0 | 0.0 | 0.5 | 51.0 | 43.0 |  |
| InstructBLIP-Vicuna-7B | 19.0 | 16.0 | 52.0 | 54.0 | 21.0 | 20.5 | 52.5 | 57.0 |  |
| LLaMA-Adapter-v2-Multimodal | 11.0 | 9.0 | 52.0 | 50.0 | 13.0 | 10.0 | 53.0 | 50.0 |  |
| MiniGPT-4 | 13.0 | 12.0 | 54.0 | 52.5 | 15.0 | 14.0 | 54.0 | 51.5 |  |
| Closed | GPT-4V | 6.5 | 7.0 | 48.0 | 55.5 | 3.0 | 10.0 | 48.5 | 56.0 |
| GPT-4o | 21.0 | 17.0 | 57.0 | 53.0 | 17.5 | 15.5 | 51.5 | 56.5 |  |
| Claude 3 Opus | 15.0 | 13.5 | 50.5 | 51.5 | 16.0 | 18.5 | 50.0 | 56.0 |  |
|  | Model | Height-5 towers $\overline{\mbox{SP}}$ | Height-5 towers SP |  |  |  |  |  |  |
|  | MCQ | T/F | MCQ | T/F |  |  |  |  |  |
|  | Color | Label | Color | Label | Color | Label | Color | Label |  |
| Open | LLaVA 1.5 7B | 14.0 | 14.0 | 46.0 | 47.0 | 14.0 | 18.5 | 51.5 | 51.0 |
| LLaVA 1.5 13B | 12.0 | 9.0 | 52.0 | 49.0 | 8.5.0 | 8.0 | 49.0 | 48.0 |  |
| LLaVA 1.6 Mistral 7B | 16.0 | 14.5 | 46.0 | 46.0 | 17.5 | 20.5 | 48.0 | 51.0 |  |
| LLaVA 1.6 Vicuna 7B | 16.0 | 13.5 | 51.5 | 49.5 | 16.0 | 15.0 | 48.5 | 49.0 |  |
| LLaVA 1.6 Vicuna 13B | 16.5 | 16.0 | 52.0 | 49.0 | 20.0 | 14.5 | 49.0 | 49.0 |  |
| Bunny-v1.0-3B | 13.0 | 11.5 | 50.5 | 44.0 | 12.5 | 19.5 | 49.0 | 50.5 |  |
| Bunny-v1.0-4B | 16.0 | 14.5 | 52.0 | 49.0 | 14.0 | 17.0 | 49.0 | 49.0 |  |
| Bunny-v1.1-4B | 14.5 | 13.0 | 52.0 | 49.0 | 12.0 | 18.0 | 49.0 | 49.0 |  |
| Bunny-Llama-3-8B-V | 15.0 | 15.0 | 52.0 | 47.5 | 14.5 | 21.0 | 49.0 | 49.5 |  |
| Fuyu-8B | 0.0 | 0.0 | 52.5 | 51.5 | 0.0 | 0.0 | 49.0 | 46.5 |  |
| InstructBLIP-Flan-T5-XL | 0.0 | 1.5 | 48.0 | 51.0 | 0.0 | 1.5 | 51.0 | 51.0 |  |
| InstructBLIP-Vicuna-7B | 15.0 | 11.0 | 52.5 | 49.0 | 15.0 | 16.0 | 48.5 | 49.0 |  |
| LLaMA-Adapter-v2-Multimodal | 10.5 | 8.5 | 51.0 | 52 | 9.5 | 9.0 | 50.0 | 51.5 |  |
| MiniGPT-4 | 13.5 | 10.0 | 52.0 | 50.0 | 12.0 | 10.5 | 51.0 | 49.5 |  |
| Closed | GPT-4V | 17.5 | 12.5 | 51.5 | 50.0 | 14.0 | 6.5 | 50.0 | 49.0 |
| GPT-4o | 18.0 | 18.5 | 59.5 | 50.0 | 19.0 | 19.0 | 51.0 | 52.0 |  |
| Claude 3 Opus | 19.5 | 14.0 | 48.5 | 51.5 | 13.0 | 19.5 | 47.5 | 48.5 |  |

Figure: Figure 12: Prompt engineering using chain of thought prompting. Here the intermediate reasoning steps introduced in the engineered prompts of the Synthetic 3D dataset is denoted by a dashed box.
Refer to caption: extracted/5823729/images/Geom_Bench_image_stuff-98.png

**Table 5: Performance of the studied models on proposed Synthetic-3D height category. Evaluation is done on the VQA task on MCQ and True/False type questions. Color, ColMat are the query attributes. Here, ColMat denotes color+material**
|  | Model | Depth-3 shapes | Depth-5 shapes |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | MCQ | T/F | MCQ | T/F |  |  |  |  |  |
|  | Color | ColMat | Color | ColMat | Color | ColMat | Color | ColMat |  |
| Open | LLaVA 1.5 7B | 49.1 | 42.5 | 59.4 | 53.8 | 43.1 | 37.5 | 55.7 | 50.4 |
| LLaVA 1.5 13B | 51.3 | 45.9 | 61.9 | 58.4 | 37.3 | 35.1 | 50.3 | 44.3 |  |
| LLaVA 1.6 Mistral 7B | 47.1 | 45.3 | 51.9 | 50.6 | 34.8 | 30.8 | 50.3 | 48.9 |  |
| LLaVA 1.6 Vicuna 7B | 48.8 | 47.3 | 61.9 | 58.3 | 40.2 | 32.9 | 45.9 | 40.2 |  |
| LLaVA 1.6 Vicuna 13B | 51.8 | 50.3 | 64.2 | 61.2 | 48.3 | 42.9 | 50.2 | 45.9 |  |
| Bunny-v1.0-3B | 34.8 | 29.3 | 40.2 | 35.8 | 21.9 | 18.3 | 34.8 | 29.8 |  |
| Bunny-v1.0-4B | 34.2 | 30.8 | 45.3 | 43.2 | 28.2 | 23.2 | 34.9 | 30.7 |  |
| Bunny-v1.1-4B | 45.2 | 40.3 | 44.2 | 42.9 | 40.2 | 38.3 | 48.3 | 42.9 |  |
| Bunny-Llama-3-8B-V | 44.2 | 42.1 | 45.2 | 40.8 | 40.8 | 35.9 | 40.8 | 38.3 |  |
| Fuyu-8B | 41.8 | 38.4 | 59.3 | 51.8 | 30.5 | 27.5 | 48.3 | 47.2 |  |
| InstructBLIP-Flan-T5-XL | 58.3 | 54.2 | 55.3 | 51.3 | 61.9 | 59.3 | 54.9 | 53.8 |  |
| InstructBLIP-Vicuna-7B | 57.4 | 56.3 | 56.9 | 55.4 | 60.2 | 57.3 | 59.9 | 58.6 |  |
| LLaMA-Adapter-v2-Multimodal | 52.9 | 48.3 | 47.3 | 44.2 | 59.8 | 56.8 | 57.8 | 54.7 |  |
| MiniGPT-4 | 60.3 | 56.3 | 57.8 | 54.8 | 65.3 | 62.9 | 60.3 | 54.8 |  |
| Closed | GPT-4V | 54.3 | 50.1 | 63.9 | 60.2 | 45.3 | 40.9 | 48.4 | 43.2 |
| GPT-4o | 59.9 | 52.9 | 65.9 | 60.3 | 50.3 | 44.3 | 50.3 | 44.8 |  |
| Claude 3 Opus | 56.3 | 53.9 | 57.3 | 52.3 | 47.3 | 43.2 | 51.8 | 47.4 |  |

**Table 6: Performance of the studied models on proposed Synthetic-3D height category. Evaluation is done on the VQA task on MCQ and True/False type questions. Color, ColMat are the query attributes. Here, ColMat, SP, $\overline{\mbox{SP}}$ respectively denotes color+material, w/ step, and w/o step.**
|  | Model | Height-3 towers $\overline{\mbox{SP}}$ | Height-3 towers SP |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | MCQ | T/F | MCQ | T/F |  |  |  |  |  |
|  | Color | ColMat | Color | ColMat | Color | ColMat | Color | ColMat |  |
| Open | LLaVA 1.5 7B | 20.3 | 12.9 | 48.2 | 40.8 | 18.8 | 8.1 | 46.3 | 40.3 |
| LLaVA 1.5 13B | 22.8 | 18.3 | 52.1 | 48.9 | 19.9 | 15.8 | 48.2 | 45.9 |  |
| LLaVA 1.6 Mistral 7B | 21.9 | 18.7 | 49.9 | 42.7 | 18.3 | 12.8 | 47.9 | 44.3 |  |
| LLaVA 1.6 Vicuna 7B | 20.8 | 18.9 | 48.7 | 44.8 | 18.7 | 12.7 | 49.7 | 43.8 |  |
| LLaVA 1.6 Vicuna 13B | 24.9 | 19.8 | 50.7 | 47.3 | 20.8 | 17.3 | 50.2 | 45.9 |  |
| Bunny-v1.0-3B | 12.4 | 9.4 | 51.4 | 50.4 | 9.4 | 5.3 | 42.9 | 40.3 |  |
| Bunny-v1.0-4B | 14.9 | 10.4 | 51.8 | 48.3 | 12.9 | 10.5 | 44.3 | 41.7 |  |
| Bunny-v1.1-4B | 15.9 | 12.7 | 54.8 | 52.6 | 13.7 | 11.8 | 50.3 | 48.5 |  |
| Bunny-Llama-3-8B-V | 16.3 | 12.8 | 55.7 | 53.9 | 14.9 | 13.9 | 52.9 | 49.3 |  |
| Fuyu-8B | 9.3 | 7.9 | 40.2 | 35.4 | 5.9 | 3.9 | 37.9 | 34.7 |  |
| InstructBLIP-Flan-T5-XL | 25.1 | 20.9 | 53.8 | 50.3 | 22.9 | 20.4 | 50.3 | 48.2 |  |
| InstructBLIP-Vicuna-7B | 24.9 | 21.9 | 54.3 | 52.9 | 20.8 | 18.9 | 52.7 | 49.3 |  |
| LLaMA-Adapter-v2-Multimodal | 23.9 | 20.3 | 49.3 | 47.8 | 20.2 | 18.7 | 48.2 | 45.8 |  |
| MiniGPT-4 | 26.9 | 24.8 | 54.8 | 53.7 | 24.8 | 20.4 | 53.8 | 51.8 |  |
| Closed | GPT-4V | 28.8 | 25.9 | 48.3 | 48.0 | 27.1 | 26.9 | 46.0 | 43.9 |
| GPT-4o | 30.5 | 28.9 | 50.9 | 49.2 | 28.9 | 27.8 | 49.3 | 46.8 |  |
| Claude 3 Opus | 28.3 | 24.0 | 51.8 | 48.3 | 26.1 | 22.0 | 47.3 | 43.0 |  |
|  | Model | Height-5 towers $\overline{\mbox{SP}}$ | Height-5 towers SP |  |  |  |  |  |  |
|  | MCQ | T/F | MCQ | T/F |  |  |  |  |  |
|  | Color | ColMat | Color | ColMat | Color | ColMat | Color | ColMat |  |
| Open | LLaVA 1.5 7B | 12.9 | 10.4 | 48.3 | 42.3 | 10.4 | 9.3 | 47.3 | 43.8 |
| LLaVA 1.5 13B | 13.9 | 11.3 | 50.3 | 49.2 | 11.8 | 10.5 | 49.3 | 47.3 |  |
| LLaVA 1.6 Mistral 7B | 11.0 | 9.3 | 50.4 | 47.3 | 10.3 | 8.3 | 47.0 | 46.9 |  |
| LLaVA 1.6 Vicuna 7B | 13.9 | 10.3 | 51.9 | 49.2 | 11.8 | 10.8 | 50.8 | 47.1 |  |
| LLaVA 1.6 Vicuna 13B | 15.9 | 12.3 | 54.1 | 50.3 | 12.9 | 9.3 | 52.9 | 48.3 |  |
| Bunny-v1.0-3B | 9.2 | 4.2 | 34.3 | 28.4 | 7.3 | 6.9 | 33.2 | 30.9 |  |
| Bunny-v1.0-4B | 11.9 | 9.3 | 35.3 | 30.4 | 9.3 | 5.3 | 34.3 | 33.9 |  |
| Bunny-v1.1-4B | 13.9 | 11.4 | 39.3 | 36.3 | 12.9 | 10.2 | 37.3 | 33.9 |  |
| Bunny-Llama-3-8B-V | 13.3 | 12.1 | 38.3 | 37.9 | 10.3 | 9.9 | 36.3 | 35.9 |  |
| Fuyu-8B | 4.2 | 1.8 | 35.3 | 30.0 | 0.0 | 0.0 | 32.8 | 31.9 |  |
| InstructBLIP-Flan-T5-XL | 19.8 | 18.9 | 47.2 | 42.1 | 16.3 | 15.9 | 42.9 | 38.3 |  |
| InstructBLIP-Vicuna-7B | 18.3 | 17.9 | 46.3 | 45.8 | 17.0 | 16.9 | 43.9 | 42.7 |  |
| LLaMA-Adapter-v2-Multimodal | 15.3 | 12.8 | 48.3 | 48.0 | 13.9 | 12.8 | 47.4 | 45.4 |  |
| MiniGPT-4 | 20.8 | 19.3 | 53.2 | 50.2 | 19.2 | 16.0 | 49.3 | 47.3 |  |
| Closed | GPT-4V | 19.3 | 17.3 | 48.4 | 47.8 | 18.3 | 16.9 | 47.0 | 46.3 |
| GPT-4o | 22.6 | 21.9 | 51.9 | 50.3 | 20.9 | 19.6 | 49.4 | 47.4 |  |
| Claude 3 Opus | 21.9 | 19.3 | 49.3 | 47.0 | 19.7 | 15.9 | 48.9 | 44.8 |  |

### 6.3 Prompt Engineering

Figure: Figure 13: Performance evaluation with chain of thought prompting on subset of Synthetic 3D dataset.
Refer to caption: extracted/5823729/images/Unknown-104.png

Chain of thought prompting enhances problem-solving by guiding models through logical reasoning steps, similar to human cognitive processes. To provide models with additional contextual information regarding visual cues with the help of intermediate reasoning, we implemented chain-of-thought prompting following []. To assess its effectiveness, we selected a small subset (100 images, 100 questions) of the Synthetic 3D dataset from the depth category. We manually generated chain-of-thought prompts and rewrote the original standard prompts to include these intermediate reasoning steps, as illustrated in Figure 12. We then evaluated two top-performing models (LLaVA 1.5 7B from open models and GPT 4o from closed models) using these prompts, with results shown in Figure 13. The evaluation revealed only minor performance improvements, indicating that chain of thought prompting did not significantly enhance model performance. This suggests that models might already be performing intermediate reasoning with the original standard. While this exact method did not result in a significant performance boost, other techniques might. Future research should focus on automating and refining prompt generation, testing on larger datasets, incorporating more contextual information, and comparing different prompting techniques to fully leverage the potential of chain-of-thought prompting.

## 7 Limitations

This research on the depth and height perception of vision language models (VLMs) using synthetic and real-world datasets highlights several areas for further exploration. Incorporating temporal dynamics and expanding the scope to include higher-order reasoning tasks will contribute to a more comprehensive understanding of VLM capabilities in terms of depth and height. Additionally, investigating potential biases in depth and height perception could help in developing fairer and more equitable VLM systems. These future directions not only address current limitations but also aim to enhance the practical applications of VLMs in diverse real-world scenarios.

## 8 Broader Impact

To our understanding, there are no negative societal impacts of our work. The goal of this work was to evaluate the depth and height perception capabilities of models that may later be used in real-world settings. This research provides insights into the depth and height perception capabilities of vision language models (VLMs), significantly impacting practical applications like autonomous driving, augmented reality, and assistive technologies. This work not only advances theoretical understanding but also opens up new possibilities for real-world applications.

## 9 Computational Resources

All experiments were run on an internal cluster. Each run used a single NVIDIA GPU, with memory ranging from 16GB-24GB.

Figure: Figure 14: Depth and height perception of open and closed models. Here we show the prediction of LLaVA 1.5 7B and GPT 4o. Real-World dataset has more accurate predictions than Synthetic Datasets, showing discrepancy in model performance. Here Q and A respectively denote Question and Ground Truth Answer. Green and Red boxes respectively denote correct and incorrect prediction.
Refer to caption: extracted/5823729/images/geombench-visuals.drawio-12.png

Figure: Figure 15: Samples from Synthetic 2D dataset - depth category. Here each row represents one image and its corresponding prompt along with MCQ and True/False questions. First three rows show samples for labels as query attribute, whereas last three rows show samples for color as query attribute.
Refer to caption: extracted/5823729/images/geombench-visuals.drawio-3.png

Figure: Figure 16: Samples from Synthetic 2D dataset - height category. Here each row represents one image and its corresponding prompt along with MCQ and True/False questions. First three rows show samples for labels as query attribute, whereas last three rows show samples for color as query attribute
Refer to caption: extracted/5823729/images/geombench-visuals.drawio-4.png

Figure: Figure 17: Samples from Synthetic 3D dataset - depth category. Here each row represents one image and its corresponding prompt along with MCQ and True/False questions. First three rows show samples for color as query attribute, whereas last three rows show samples for color+material as query attribute
Refer to caption: extracted/5823729/images/geombench-visuals.drawio-5.png

Figure: Figure 18: Samples from Synthetic 3D dataset - height category. Here each row represents one image and its corresponding prompt along with MCQ and True/False questions. First three rows show samples for color as query attribute, whereas last three rows show samples for color+material as query attribute
Refer to caption: extracted/5823729/images/geombench-visuals.drawio-6.png

Figure: Figure 19: Samples from Real-World dataset - depth category. Here each row represents one image and its corresponding prompt along with MCQ and True/False questions. Numeric labels are used as query attribute
Refer to caption: extracted/5823729/images/geombench-visuals.drawio-9.png

Figure: Figure 20: Samples from Real-World dataset - height category. Here each row represents one image and its corresponding prompt along with MCQ and True/False questions. Numeric labels are used as query attribute
Refer to caption: extracted/5823729/images/geombench-visuals.drawio-10.png

Figure: Figure 21: Samples from Synthetic 2D Basic dataset. Here each two rows respectively represent line understanding, shape identification, shape counting and spatial relationship categories. Each row shows one image and its corresponding prompt along with the MCQ.
Refer to caption: extracted/5823729/images/geombench-visuals.drawio-11.png

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: See the weakness above.
soundness: 3
strengths: S 1. Assessing VLMs' capabilities in height and depth perception is novel and largely not emphasized by existing VQA tasks.

S 2. Clear and detailed experiment presentation. The figures are informative. An in-depth analysis of the performances is provided.

S 3 Very reasonable ablation experiments. 
  - Examined the VLMs' ability to recognize and understand shapes. It is interesting to see that all the tested models can easily understand the lines and the shapes but struggle with depth and height. 
  - Chain-of-though prompting is experimented with to validate that the unsatisfactory performance cannot be addressed by simply improving the prompting techniques. This helps prove that the limitation is inherent to existing VLMs.
summary: This work proposes a benchmark to evaluate large Vision and Language Models (VLMs) on their capabilities in perceiving and understanding height and depth from single images in the format of visual question answering. They curated the dataset using synthetic shapes and symbols in both 2D and 3D and benchmarked 18 open or closed-source VLMs. Their results reveal that current VLMs still struggle with perceiving depth and height from single images, although it is easy for human evaluators.
weaknesses: W 1.  The design choice of using simple synthetic shapes as the dataset is reasonably justified. Still, the reviewer thinks it is necessary to incorporate data sampled from the real world so that the performance on the benchmark can better indicate the performance in real life where VLMs are applied. So far this part has been missing and all data is synthetic. In other words, how consistent is the performance between this benchmark and the performance in real-life scenarios?

W 2. Similar to W 1, while the background adds realism to the data, it might be a good idea to also incorporate data with a clean and white background. The purpose is to show how much difficulty in the task is introduced by the complexity of visual input rather than depth and height. Will the performance improve if the background is clean? If so, by how much?

W 3. Add experiments on VLMs with a vision focus. While many VLMs are included in the paper, it is missing some recent ones that contributed to improving the visual perception abilities of VLMs, for example [1] and [2]. While they are cited in the paper, they are not included in the experiments. Given the content of the task, adding experiments on these VLMs (if available) seems necessary.

W 4. While exposing the perception limitations of current VLMs is a welcomed contribution, the paper will make a greater contribution to the community if it can design some new approach to try to mitigate the failures and enhance a VLM on perceiving height and depth. [3], cited in this paper, is a great example.

---
[1]  Chen, Boyuan, et al. "Spatialvlm: Endowing vision-language models with spatial reasoning capabilities." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[2] Tong, Shengbang, et al. "Cambrian-1: A fully open, vision-centric exploration of multimodal llms." arXiv preprint arXiv:2406.16860 (2024).

[3] Tong, Shengbang, et al. "Eyes wide shut? exploring the visual shortcomings of multimodal llms." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

---

confidence: 4
contribution: 1
initial_rating: 3
presentation: 3
questions: I don't agree with the claim that "This study paves the way for developing VLMs with enhanced geometric understanding". Could you explain explicitly which way can be inspired by this paper?
soundness: 3
strengths: - This paper proposes two synthetic datasets to evaluate the depth and height understanding capabilities of exiting VLMs. The datasets are generated programmatically to ensure randomness. 

- This paper provides a comprehensive analysis of 18 open-source and closed-source VLMs.
summary: This paper mainly investigates the spatial reasoning capabilities of existing Vision Language Models (VLMs) regarding depth and height understanding. To achieve this, it proposes two synthetic benchmark datasets focusing on 2D and 3D scenarios, named GeoMeter-2D and GeoMeter-3D respectively. Each image is generated by inserting objects with random shapes and numeric labels(or materials) into a real natural image. The prompt template is designed to determine which of the given objects is on the top and provide the height ordering of the given objects to evaluate the capabilities of estimating depth and height. The authors analyze the proposed benchmark datasets on 18 recent open-source and closed-source models.
weaknesses: - The contribution of this paper is quite limited. The proposed benchmarks are too easy as they only contain some conceptual questions about easy spatial relationships among different objects. Some complex relationships should also be covered, like "Which object is on the left ?". And it would be great if some numerical questions could be provided as well. 

- The conclusion from the analysis is not so insightful or informative. From the analysis of several VLMs, the authors find that most of them lack depth and height perception ability. I don't think this conclusion is something new to the community, there are already several papers proposed to enhance the spatial reasoning ability of the existing VLMs.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: N/A
soundness: 2
strengths: 1. The presentation of this paper is clear and easy to follow. 
2. The coverage of 18 VLMs is comprehensive. Moreover, human performance is also reported, which helps understand the difficulty of the proposed questions.
summary: This paper proposes a new benchmark for evaluating the depth and height understanding capability of the vision language foundation models (VLMs). The benchmark has two subsets, one in 2D and another in 3D, both consisting of rendered images with synthetic objects placed on random real backgrounds. The models are asked about the depth order of the objects, or the height of a stack of rectangles. The questions are formatted into either multiple-choice or true-or-false. The authors tested the performance of 18 VLMs, including both open-source and closed-source ones, as well as human performance. All VLMs perform poorly on the benchmark.
weaknesses: 1. Limited novelty and contribition for the depth task. The proposed 3D benchmark is highly based on existing the CLEVR dataset. CLEVR already contains questions requires reasoning about spatial relationships among objects, partially covering the depth task proposed by the authors. The CLEVR dataset has even higher complexity (up to 10 objects, compared to 5 used by the authors) and is more photo-realistic than the images shown in the paper. The authors should discuss more about the connection with CLEVR and what new aspects can the proposed dataset bring.
2. The height understanding task is novel to me, but importance of this task need justification. Depth is important for 3D understanding in human vision, but height is less straightforward, especially under the toyish setting of counting the height of a stack of building blocks. The authors should give some concrete example about how solving the proposed height task could benefit real-world applications of VLMs. Also the black platform makes the task a bit confusing to me, and the failure of VLMs may simply because they don't understand the setting.
3. The scope of the the task is limited (only 2, depth and height), and only synthetic images are used. This is a problem as more diverse and realistic benchmarks already exist. For example, a recent work BLINK[1] covers more fundamental vision tasks (including depth) with real-world images. The authors should discuss the connection with it in related work. Also since the authors use only synthetic images, it would be helpful to add discussions about the advantages or disadvantages of using synthetic images vs real-world images for benchmarking.

Overall, I feel the paper is trying to analyze an important and interesting question, i.e., the spatial relationship perception ability of VLMs. However, I feel that the current version of this paper doesn't meed the ICLR standard due to the limited scope, novelty and diversity of the proposed dataset. I would recommend the authors to cover a wider range of vision tasks in a more realistic setting. For example, covering more 3D vision tasks such as visual correspondence, object pose estimation, camera pose estimation, or using real-world images to make the benchmark more realistic.

Minor 1: L59 mentions 2 cats in Fig 1, but I see no cats in the figure. Is this a typo/mis-inserted figure?
Minor 2: From my understanding, Fig 4 and Tab 2 are basically displaying the same result, with Tab 2 simply merging the numbers for depth and height. This is kind of redundant and confusing, and I would recommend only keeping the table.

[1] Blink: Multimodal large language models can see but not perceive. ECCV 2024.

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 3
questions: * It's not clear how this benchmark applies to real-world problems.
* The reason for using different materials isn't well explained, given that all objects seem to have different colors. Some samples include the material tags, while others do not (Figure 14).
* Many samples don't fit well with their backgrounds. This might make the benchmark less useful. Would a plain white background, like in CLEVR, work better?
* The numbers in GeoMeter-2D are tiny. This might be hard for VLMs (or even for humans) to recognize since VLMs often use low-res images like around 200-300 as input.
* In GeoMeter-3D (Height), the flat base often doesn't match the background image. This could confuse the model.
* It's not clear if the background matters for this test. If not, the test might be more about tricking the model than testing general height/depth recognition capabilities.
* The benchmark uses a limited range of shapes, colors, and textures. Even though there are many samples (and with diverse backgrounds), the foreground objects all look similar.
* The paper mentions that some open-source models are more biased toward picking True over False or the first option in multiple-choice questions. However, if this is true, how can these models perform well on other general VQA benchmarks?
soundness: 2
strengths: The paper explores the spatial reasoning abilities of current Vision Language Models, an important area that currently lacks a proper benchmark in the community.
summary: The paper introduces a synthetic benchmark suite for Vision Language Models (VLMs) that includes both 2D and 3D scenarios. This suite is designed to assess the current capabilities of VLMs in understanding height and depth. The evaluation and analysis are conducted on 18 different Vision Language Models.
weaknesses: My main concern with this benchmark is how it was designed and whether it truly tests VLMs' ability to recognize height and depth. There are serious doubts about whether the benchmark samples properly measure these abilities or if there are flaws in the benchmark design or evaluation process. The test results, which seem to show random guessing from most models, add to these concerns. The analysis provided with the results also doesn’t offer much explanation as it fails to clarify why the models performed the way they did (even SOTA closed-source models seem random on True/False types) or give useful insights into the benchmark. Please see the details of the questions below.