LENS: Multi-level Evaluation of Multimodal Reasoning with Large Language Models

## Abstract

Abstract Multimodal Large Language Models (MLLMs) have achieved significant advances in integrating visual and linguistic information, yet their ability to reason about complex and real-world scenarios remains limited.
The existing benchmarks are usually constructed in the task-oriented manner without guarantee that different task samples come from the same data distribution, thus
they often fall short in evaluating the synergistic effects of lower-level perceptual capabilities on higher-order reasoning. To lift this limitation, we contribute Lens , a multi-level benchmark with 3.4K contemporary images and 60K+ human-authored questions covering eight tasks and 12 daily scenarios, forming three progressive task tiers, i.e. , perception, understanding, and reasoning.
One feature is that each image is equipped with rich annotations for all tasks.
Thus, this dataset intrinsically supports to evaluate MLLMs
to handle image-invariable prompts,
from basic perception to compositional reasoning.
In addition, our images are manully collected from the social media, in which 53 % percent 53 53\% 53 % were published later than Jan. 2025.
We evaluate 15+ frontier MLLMs such as Qwen2.5-VL-72B, InternVL3-78B, GPT-4o and two reasoning models QVQ-72B-preview and Kimi-VL.
These models are released later than Dec. 2024, and none of them achieve an accuracy greater than 60 % percent 60 60\% 60 % in the reasoning tasks.
Project page: https://github.com/Lens4MLLMs/lens .
ICCV 2025 workshop page: https://lens4mllms.github.io/mars2-workshop-iccv2025/

## 1 Introduction

Multimodal Large Language Models (MLLMs) have emerged as a rapidly advancing field in artificial intelligence, demonstrating substantial improvements in visual content recognition and multimodal reasoning zhu2025internvl3; bai2025qwen2; wu2024deepseek; team2024gemini; team2025gemma.
Despite their promising capabilities, MLLMs continue to face significant challenges in interpreting complex and real-world visual environments that are inherently dynamic, diverse, and grounded in physicality. However, existing benchmarks remain limited in their ability to evaluate
multi-level reasoning.

Early evaluations were largely based on classical computer vision tasks everingham2010pascal; lin2014microsoft; yu2016modeling and their integration with natural language. The real-world knowledge was often superficial, resulting in weak alignment between visual input and linguistic output. Secondly, these benchmarks are typically constructed under closed-world assumptions, lacking the inter-task consistency needed to assess reasoning across modalities fu2024mme; li2024survey. As a result, the absence of quantitative multi-level evaluation hinders meaningful comparison across MLLMs.

Figure: Figure 1: Illustration of the task split in Lens.
Refer to caption: x1.png

More recent benchmarks have begun to shift toward open-world evaluation and multimodal reasoning tasks wu2024v; zang2025contextual. While this represents progress, current
benchmarks do not adequately assess the nuanced performance necessary to evaluate MLLMs’ progression towards human-like intelligence in real-world settings.
They require largely primary visual comprehension and fall short of measuring higher-order reasoning and spatial understanding li2024embodied; yue2024mmmu; liu2024mmbench. Furthermore, data distributions often differed between tasks, so that high performance in perceptual tasks did not necessarily translate into strong inference capabilities in more complex integrated multimodal tasks fan2025scaling.
As a result, they ignore the synergistic effect of the combinations of lower-order perceptual abilities on higher-order reasoning and are hard to provide a fine-grained assessment.

In this study, we propose a hierarchical and comprehensive evaluation framework Lens specifically designed to assess the multimodal capabilities in real-world scenarios. Our benchmark focuses not only on isolated tasks but also on the integration of perception, understanding, and reasoning—three core tiers essential for intelligent multimodal systems. As shown in Figure [1](https://arxiv.org/html/2505.15616v1#S1.F1), Lens encompasses eight tasks, systematically organized into three hierarchical tiers with eight subtasks, and it comprises 3.4K real-world photographs and 60K+ human-authored questions, in 12 diverse scenarios—including streets, stations, schools, homes, and more, which can be roughly divided into three themes: “Home”, “Education”, and “City”, and we visualize the high-frequency words under different themes in Figure [2](https://arxiv.org/html/2505.15616v1#S1.F2). 53% of the images are from 2025 and more than 80% of the images are from after September 2024, ensuring the content reflects contemporary environments.

In contrast to traditional closed-set benchmarks that rely on fixed label sets or rigid taxonomies, our framework adopts an open-set configuration, allowing queries to be posed in natural language and grounded in authentic photographic content. This design enables evaluation of model performance in complex, ambiguous, and information-rich settings, better aligning with real-time human demands. Moreover, our benchmark introduces a series of task-oriented challenges, such as calculating checkout amounts from receipts, determining public transport schedules from signage, or inferring human activities in household scenes. These tasks share the same image source and necessitate the integration of visual perception with external knowledge and logical reasoning, encouraging models to move beyond recognition toward functional intelligence, makes Lens able to evaluate the synergistic effects of lower-level perceptual
abilities (e.g., object detection, localization) on higher-order reasoning tasks. To succeed in Lens, models must jointly process multimodal input, recall domain knowledge, and conduct multi-step reasoning to arrive at valid conclusions and our experimental results show that Lens is challenging for current SOTA models.

In summary, Lens makes the following contributions:

- •
Realistic and Up-to-Date Evaluation: By leveraging a newly collected set of high-resolution, naturalistic images, our benchmark evaluates the latest multimodal reasoning models in settings that closely reflect real-world complexity.
- •
Multi-Level Evaluation: It supports fine-grained and interpretable evaluation across three core dimensions—perception, understanding, and reasoning—providing a comprehensive view of a model’s multimodal competence.
- •
Synergistic Capability Evaluation: Unlike existing benchmarks that often assess tasks in isolation, our framework emphasizes the synergistic effects of lower-level perceptual abilities (e.g., object detection, localization) on higher-order reasoning tasks (e.g., inference, spatial understanding).
- •
Towards Generalizable Intelligence: By capturing both perceptual and reasoning performance in integrated tasks, our benchmark helps identify the gaps between current model capabilities and the requirements of human-aligned reasoning systems and measure the shortcomings of current models (Qwen2.5-VL achieved less than 45% accuracy on reasoning tasks).

Figure: Figure 2: Three core themes, “Education”, “City”, and “Home”, along with their word clouds of the scenario distributions by name size.
Refer to caption: x2.png

## 2 Related work

### 2.1 Benchmarks for Visual Capability of MLLMs.

The capability of Visual Perception, Understanding and Reasoning is a foundational aspect of
understanding benchmarks, which involves the ability to recognize and localize multiple objects, interpret various visual elements with complex emotional or implicit cues and summarize visual information for feedback and decision making.
Specifically, Perception in MLLMs involves the classification, detection of basic visual objects (e.g., dog, cat) and attributes (e.g., color, lighting). These low-level
perceptual capabilities are crucial for various applications,
including recognition systems zhao2024octopus and visual quality enhancement zhang2024q. Understanding represents
a sophisticated level of image understanding
that focuses on the detailed and nuanced aspects of
visual content. It includes recognizing and interpreting
the visual-linguistic concepts, such as text recognition (OCRBench liu2024ocrbench), Visual Grounding (RefCOCO yu2016modeling, FineCops-Ref liu2024finecops, HC-RefLoCo wei2024large) and Referring Expression Generation (Visual genome)krishna2017visual, which refers to the model’s ability to accurately link visual
elements with corresponding textual descriptions. Although tasks at this level begin to involve visual and textual alignment, they still do not require reasoning or external knowledge. For higher-order capability, reasoning in MLLMs involves advanced event understanding and deep meaning extraction from multimodal data. These capabilities include interpreting and responding to complex emotional cues across multiple modalities cheng2024emotion, deriving subtle implicit meanings from visual and contextual information liu2023multilingual, and a range of other competencies, including knowledge acquisition, language generation, spatial awareness, and cultural context integration rachabatuni2024context.

#### 2.1.1 Reasoning Capability of MLLMs.

MLLMs have demonstrated remarkable reasoning capabilities, largely facilitated by test-time scaling dong2022survey; wei2022chain, which allows feeding prompted samples and
context. This capability has been further enhanced by chain-of-thought (CoT) prompting wei2022chain, which
enables LLMs to generate coherent intermediate reasoning steps toward the final answer. Previous
studies have shown that LLMs benefit from manually written demonstrations as well as zero-shot
prompting outputs. However, due to the domain gap between various modalities, the current reasoning capability of MLLMs in the complex real-world environment still limited.
To address this limitation, researchers have focused
on enhancing the reasoning capability of MLLMs in both the training and prompting paradigms. Flamingo alayrac2022flamingo bridges the gap between these two modalities by pre-training on interleaved
visual and textual data. Some other works, such as Shikra chen2023shikra and Ferret you2023ferret, leverage visual grounding data xiao2024towards; yao2024visual to achieve fine-grained vision-language alignment. Further more, some methods employs the external knowledge to focus on important visual details, like V* wu2024v, Marvel jiang2024marvel, and ICAL
sarch2024vlm, collecting a series of visual reasoning steps as training data.
More recently, with the emergence of DeepSeek-R1 guo2025deepseek demonstrating strong potential in Large Language Model (LLM) reasoning, research efforts have begun to explore reasoning-centric models and R1-style reinforcement learning strategies for understanding complex visual scenes and tasks. These studies huang2025vision; shen2025vlm; liu2025visual particularly emphasize the long-chain reasoning capabilities within Multimodal Large Language Models (MLLMs), aiming to enhance their performance in handling intricate visual-linguistic reasoning challenges.

## 3 Dataset and Benchmark

Figure: Figure 3: Lens consists of eight sub-tasks at three levels. Perception tasks focus on recognizing object attribute and counting. Understanding tasks emphasizes localization and inter-object relationships, requiring a integration of fine-grained visual context. Reasoning tasks demand the use of external knowledge beyond the visual input and involve multi-step, complex reasoning processes to arrive at the correct answer.
Refer to caption: x3.png

### 3.1 Data Curation Process

#### 3.1.1 Data Collection.

The image data collection in our benchmark focuses on real-world scenes to ensure diversity, representativeness, and practicality for visual perception, understanding and reasoning tasks. To this end, we first defined a set of common real-life scenarios that are highly relevant to typical human visual experiences. The selection principle was that each visual scene should contain distinguishable and representative semantic content. For example, street scenes are usually populated with cars, pedestrians, and storefronts, while indoor environments like classrooms often involve students, teachers, and educational materials. To avoid regional or cultural bias and ensure a broad distribution of content, we collected images from multiple international social media platforms, including X (formerly Twitter), Instagram, Weibo, and Xiaohongshu. These platforms were chosen due to their global user bases and diverse content coverage across regions and lifestyles. During the collection process, we strictly complied with the copyright and licensing regulations of each platform, ensuring that data was collected only from publicly accessible posts and that no images were downloaded from sources explicitly prohibiting data reuse or redistribution. Moreover, to facilitate the evaluation of multiple subtasks within the same image (e.g., detection, visual grounding, OCR, scene knowledge inference), we curated images that exhibit rich semantic content while maintaining scene clarity. Complex or ambiguous images were manually filtered out to avoid introducing noise that could hinder benchmarking or evaluation consistency. At last, images containing sensitive personal information, such as visible faces, identifiable personal details, or private life scenarios, were either excluded or processed to blur or mask sensitive regions, to mitigate privacy risks.

Table: Table 1: Comparison with other recently released multimodal benchmarks.

Figure: Figure 4: Lens covers a wide range of images and annotations, from fine-grained recognition and spatial localization to complex reasoning over extended thought processes. Notably, each image is annotated with labels corresponding to all subtasks concurrently, enabling comprehensive evaluation.
Refer to caption: x4.png

#### 3.1.2 Task design and Annotation process.

To construct a comprehensive and diverse benchmark, we recruited over 50 undergraduate and graduate students as human annotators to assist in the process of question collection and task annotation. These annotators were carefully trained to ensure high annotation quality and consistency. As shown in Figure [3](https://arxiv.org/html/2505.15616v1#S3.F3), the generated questions were divided into three major categories: Perception, Understanding and Reasoning. For Perception and Understanding, they primarily target the model’s ability to perceive visual objects and align them accurately with natural language descriptions. They emphasize fine-grained visual grounding and object recognition rather than abstract reasoning. At last, reasoning-based questions aims to evaluate the model’s ability to understand user intent and reason based on external knowledge, commonsense, physical laws, or background information beyond the purely visual content of the image. Based on these assessment dimensions, we compare Lens with related multimodal benchmarks in Table [1](https://arxiv.org/html/2505.15616v1#S3.T1) and formulated our challengeing open-ended, language-driven tasks as follows:

Object Counting (OC): Estimating the number of object instances described by a free-form expression, often under complex conditions like occlusion, scale variation, or clutter.

Object Detection (OD): Localizing objects within an image by generating bounding boxes paired with corresponding class labels. In order to better match the real-life scenarios and practical applications, we construct more than 300 fine-grained object categories based on natural language.

Object Existence Determination (OE): Determining whether a particular object, which described by a detailed expression, exists in the image without requiring spatial localization.

Relation Extraction (RE): Identifying semantic relationships (e.g., “holding”, “next to”, “wearing”) between pairs of objects to facilitate structured scene understanding. And we added questions about the objects that do not exist in the images to evaluate model’s ability to suppress hallucinations.

Visual Grounding (VG): Localizing an image region that corresponds to a natural language expression, linking linguistic references to fine-grained visual content.

Region-wise OCR (OCR): Recognizing and transcribing text within a region, which specified by coordinates or description, facilitating fine-grained interleaved image-text understanding.

Spatial Relationship Comprehension (SRC): Understanding geometric relationships (e.g., “above” and “to the left front to”) between objects within diverse 3D views, supporting visual-spatial reasoning. Compared to some rudimentary or synthetic spatial understanding datasets johnson2017clevr; li2023super; liu2023visual, our data is more realistic in emphasizing spatial location understanding under real-world scenarios as well as 2D images acquired by cameras or cell phones.

Scene Knowledge Inference (SKI): Inferring high-level semantic and functional information about the scene or making decision based on the visual contents, incorporating context, commonsense knowledge, and visual cues beyond explicit visual entities. Compared to the regular visual reasoning dataset, Lens additionally distinguish between “thought paths” and “final answers”, differentiated by the <think> token, aiming to provide richer and finer-grained information for potential test-time scaling tests and R1-style reinforcement learning.

#### 3.1.3 Quality control.

In addition to the primary task annotations, we also enriched each image with supplementary metadata to support traceability, temporal analysis, and contextual scene understanding. Specifically, we labeled the metadata of Annotator ID (the pseudonymized identifier linking each image to the annotator responsible for generating its questions and task labels), which allows for annotator-specific quality tracking while preserving privacy. Time of Online Publication (the original timestamp when the image was published on the internet) and Scene Category (high-level semantic label describing the type of scene) are also labeld to facilitate temporal studies, filter out outdated content and organize the dataset.
Further more, we perform two steps of data cleaning. In the first stage, suspected duplicates were reviewed by the authors to identify and eliminate any duplications. The second stage involves distributing the problems among different co-authors for format and typo checking. This step requires annotators to ensure adherence to a standardized format, undertaking necessary corrections where deviations are found. Representative examples of the final cleaned data are visualized in Figure [4](https://arxiv.org/html/2505.15616v1#S3.F4).

Figure: Figure 5: Statistical analysis of our dataset. We visualize the temporal distribution of the images for different scenarios, size distribution of images and bounding box annotaions, and number of QA entries for different tasks, demonstrating the timeliness and diversity of our data.
Refer to caption: x5.png

Figure: Figure 6: The normalized probability distributions of low-level attributes from different scenes. Scenes with flat peaks show more diversity, while those with sharp peaks have similar features.
Refer to caption: x6.png

### 3.2 Data Analysis.

Our work aims to construct a dataset that is not only comprehensive and dynamic but also emphasizes reasoning ability practices. In the following analysis, we demonstrate the strengths of our benchmark in terms of diversity of images and annotations and we visualizate the quantitative results.

First, Real-Time and Various Visual Content:
Unlike traditional static image datasets, our benchmark incorporates temporally aware content and real-time data. As shown in Figure [5](https://arxiv.org/html/2505.15616v1#S3.F5) (a) and (b), more than 50% of the images in our dataset were collected in 2025, and approximately 70% were collected in November 2024 and beyond, which avoids potential data leakage.
Many images reflect dynamic scenes (e.g., crowded streets, interactive environments) captured at different times and locations, aligning with real-world AI deployment scenarios.

Second, in our dataset, the coverage of a wide range of object categories, scene types, and bounding box annotations further support diverse downstream tasks from detection to high-level semantic inference and Interleaved Image-Text understanding. As illustrated in Figure [5](https://arxiv.org/html/2505.15616v1#S3.F5) (c), the high resolution of the images in our dataset makes it challenging for fine-grained understanding of the model and supports evaluation across varying input sizes. Additionally, as shown in Figure [5](https://arxiv.org/html/2505.15616v1#S3.F5) (d), the various objects are labeled with different sizes of bounding boxes to meet the needs of multi-scale object detection, visual grounding and region-wise OCR evaluation.

Further more, beyond perception, our dataset facilitates reasoning-oriented research by supporting tasks that require: Spatial reasoning (e.g., understanding object layouts and geometric relationships).Relational inference (e.g., extracting interactions between objects). Commonsense knowledge application (e.g., inferring the feasibility of a behavior or scene functionalities). Cross-modal alignment (e.g., grounding free-form language to specific visual content). We also analyze the question-answer pairs distribution of these tasks and Figure [5](https://arxiv.org/html/2505.15616v1#S3.F5) (e) shows that over 60% of the questions in the dataset go beyond simple recognition, explicitly encouraging models to reason about the scene, context, and user intent.

At last, we counted five low-level visual attributes, including lighting, contrast, color, blur, and spatial information (SI) in hosu2017konstanz, to assess the statistical difference between different scenes. As shown in Figure [6](https://arxiv.org/html/2505.15616v1#S3.F6), the normalized probability density curves of low-level visual attributes across different scenes are consistent with human perceptual preferences. Scenes with regulated lighting conditions (e.g., classrooms, airports, and stations) demonstrate sharp peaks near x $\approx$ 0 in the illumination curves (density > 0.5), indicating constrained variations in brightness. In contrast, domestic environments (e.g., living rooms, bedrooms, and kitchens) display broader illumination distributions, suggesting more diverse and adaptive light sources. Furthermore, functional scenes such as bedrooms, bathrooms, and kitchens exhibit sharp, concentrated peaks in color distributions (peak density $\approx$ 0.5), implying greater structural regularity or visual normativity in specific visual attributes.

## 4 Evaluation

### 4.1 Evaluation Models

To illustrate the difficulty of our benchmark and evaluate the latest advances in current research, we evaluate various MLLMs belonging to three major categories: Closed-source
generalist MLLMs, such as GPT-4o achiam2023gpt and Gemini2.5 Pro team2024gemini. Open-source generalist MLLMs like Qwen2.5-VL bai2025qwen2, Deepseek-VL2 wu2024deepseek, Gemma3 team2025gemma, InternVL3 zhu2025internvl3. Multimodal reasoning models QvQ-preview and Kimi-VL-thinking, focusing on advanced reasoning capabilities.
The release dates of these models are distributed from Dec. 2024 to Apr. 2025.

### 4.2 Evaluation Strategy

To ensure a fair and efficient assessment of model performance across our benchmark, we adopt two evaluation strategies for main results. For perception and understanding tasks, models were evaluated based on their direct outputs without additional inference-time computations. For complex reasoning tasks, which require deeper multi-step inference, we allow models to generate multiple candidate responses per question and the final prediction is then selected via majority voting liu2025bag.
For qualitative judgment, we follow prior work wang2023large and employ a large language model (e.g., GLM4-flash glm2024chatglm) as an automatic evaluator. The LLM is prompted to produce multiple pieces of evaluation evidence for calibration, comparing the model-generated responses against human-annotated answers, aiming to offer a consistent framework for evaluating model performance across diverse tasks.

Further more, to assess the synergistic effects of perception and understanding on reasoning, we analyze the cross-task performance patterns of different models. Leveraging the benchmark’s unified visual source—where all tasks share the same image set—we enable comparative evaluation of model behavior. So Lens supports detailed per-task and cross-task analysis, facilitating insight into how foundational capabilities contribute to multimodal reasoning performance.

### 4.3 Evaluation Results

**Table 2: Comparison of state-of-the-art methods on Lens. We evaluate object detection (OD) performance using AP50 lin2014microsoft, visual grounding (VG) performance with ACC$@0.5$ xiao2024towards, and use accuracy as the metric for other tasks. Task abbreviations follow the definitions provided in Section [3.1](https://arxiv.org/html/2505.15616v1#S3.SS1). “MoE 1B/3B” denotes 3B Mixture of Experts model with 1B parameters activated. “N/A” denotes the official documentation does not confirm that the model is applicable for the task. Best performing models are shaded in red.**
| Methods | Model<br>size | Perception | Understanding | Reasoning |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| OC | OD | OE | RE | VG | OCR | SRC | SKI |  |  |
| MLLM (closed source) |  |  |  |  |  |  |  |  |  |
| GPT-4o | - | 54.32 | N/A | 85.09 | 72.77 | N/A | 42.86 | 51.14 | 55.20 |
| Gemini2.5-Pro | - | 60.18 | 47.40 | 86.59 | 76.52 | 25.61 | 61.95 | 56.20 | 59.31 |
| Open source |  |  |  |  |  |  |  |  |  |
| Deepseek-VL2-tiny | MoE 1B/3B | 56.22 | 21.12 | 72.11 | 58.73 | 16.09 | 44.01 | 38.97 | 45.12 |
| Deepseek-VL2 | MoE 4.5B/27B | 61.41 | 46.08 | 77.68 | 69.18 | 42.47 | 48.76 | 44.58 | 49.50 |
| Gemma3 | 4B | 38.85 | N/A | 71.88 | 62.98 | N/A | 27.03 | 39.53 | 45.18 |
| Gemma3 | 12B | 44.65 | N/A | 73.21 | 62.78 | N/A | 33.98 | 43.33 | 48.56 |
| InternVL3 | 2B | 55.81 | 18.39 | 71.96 | 64.49 | 15.22 | 45.51 | 40.56 | 48.59 |
| InternVL3 | 9B | 55.63 | 25.79 | 77.49 | 67.18 | 18.18 | 48.79 | 44.69 | 51.32 |
| InternVL3 | 38B | 62.78 | 43.44 | 81.60 | 71.37 | 24.98 | 51.72 | 47.18 | 51.85 |
| InternVL3 | 78B | 61.38 | 47.44 | 84.87 | 74.93 | 27.24 | 54.21 | 49.39 | 55.17 |
| Qwen2.5-VL | 3B | 58.76 | 35.16 | 74.01 | 66.52 | 39.44 | 52.43 | 40.33 | 46.50 |
| Qwen2.5-VL | 7B | 58.35 | 37.75 | 83.75 | 71.58 | 40.11 | 61.65 | 46.28 | 48.87 |
| Qwen2.5-VL | 32B | 62.25 | 39.93 | 83.60 | 74.57 | 41.15 | 65.64 | 51.66 | 51.54 |
| Qwen2.5-VL | 72B | 59.75 | 43.48 | 85.67 | 75.98 | 44.98 | 68.51 | 53.65 | 54.79 |
| Reasoning model |  |  |  |  |  |  |  |  |  |
| QVQ-Max | 72B | 49.95 | N/A | 85.37 | 74.01 | N/A | 58.67 | 50.80 | 58.86 |
| Kimi-VL-thinking | MoE 2.8B/16B | 46.87 | N/A | 72.77 | 48.16 | N/A | 30.21 | 29.40 | 36.44 |

We evaluate a suite of state-of-the-art Multimodal Large Language Models on our benchmark, which spans three tiers and eight tasks. Results, as shown in Table [2](https://arxiv.org/html/2505.15616v1#S4.T2), reveal insights into model scaling, inter-task dependencies, and capability gaps in current MLLMs.

Model Scaling and General Trends.
We observe a consistent performance gain with increased model size in both closed- and open-source models. For example, Qwen2.5-VL improves steadily from 3B to 72B, achieving top performance on reasoning tasks (SRC: 53.65%, SKI: 54.79%). InternVL3 shows similar gains in OD, rising from 18.39% (2B) to 47.44% (78B), though performance saturates at higher scales. These trends confirm that scaling remains a key driver for multimodal reasoning, albeit with diminishing returns in some subtasks.

Perception: Foundation for Higher Cognition.
Perception-level tasks form the backbone of visual reasoning. Closed-source models like Gemini2.5-Pro and GPT-4o excel at OE (86.59% and 85.09%, respectively), although OD support is lacking. Among open-source models, Deepseek-VL2 and Qwen2.5-VL-72B deliver competitive OD and OC performance. Notably, models with stronger perception capabilities tend to exhibit superior reasoning performance, highlighting the foundational role of low-level visual understanding.

Understanding: Progress and Bottlenecks.
Understanding tasks assess models’ ability to interpret structured visual semantics with textual information. Gemini2.5-Pro leads in RE (76.52%) and OCR (61.95%), showcasing robust relational and textual grounding. However, VG remains a bottleneck even for large-scale models like InternVL3-78B (27.24%) and Qwen2.5-VL-72B (44.98%), suggesting persistent challenges in fine-grained spatial-semantic alignment.

Reasoning: High-Level Generalization.
Reasoning tasks are the most demanding. Closed-source models such as GPT-4o and Gemini2.5-Pro achieve strong results (51.14%/56.20% on SRC and 55.20%/59.31% on SKI). Among open-source models, Qwen2.5-VL-72B leads, while the reasoning-specialized QVQ-Max approaches closed-source performance (58.86% on SKI) despite lacking OD and VG capabilities. This suggests that explicit reasoning models can partially compensate for perceptual limitations, likely relying on test-time scaling rather than grounded perception.

Figure: Figure 7: Statistical analysis of model accuracy and synergies between different tasks.
Refer to caption: x7.png

### 4.4 Synergistic effects analysis

To analyze the cross-task performance patterns of different models, we perform a statistical analysis of the synergies between different tasks and visualized the results as in Figure [7](https://arxiv.org/html/2505.15616v1#S4.F7). We compute the Pearson correlation coefficients between Perception and Understanding tasks and observe notable interdependencies. OC and RE exhibit a strong positive correlation of $0.73$, while OE and OCR show a similarly significant correlation of $0.67$. These results indicate that effective performance in perception directly contributes to understanding, which in turn underpins downstream reasoning. Scatter plot visualizations further confirm these links, OCR, in particular, correlates strongly with both SRC and SKI, underscoring its central role in enabling semantic reasoning. Linear regression analyses reinforce these findings: OE and OCR are strong predictors of SRC, while OC and RE significantly influence SKI, highlighting how object-level detection and relational reasoning jointly support high-level inference. Finally, we apply second-order polynomial regression and the feature importance analysis of regression coefficients reveals task-specific contributions. These insights collectively demonstrate the layered structure of visual reasoning pipelines, where perception and understanding stages must be well-aligned to support robust inference.

## 5 Conclusion

We contribute Lens, a multi-level benchmark designed to evaluate Multimodal Large Language Models (MLLMs) across perception, understanding, and reasoning. Unlike prior benchmarks, Lens aligns all tasks to the same set of realistic, contemporary images, enabling fine-grained analysis of how low-level visual capabilities support higher-order reasoning.

## References

- [1]
Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Yuchen Duan, Hao Tian, Weijie Su, Jie Shao, et al.
Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models.
arXiv preprint arXiv:2504.10479, 2025.
- [2]
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al.
Qwen2. 5-vl technical report.
arXiv preprint arXiv:2502.13923, 2025.
- [3]
Zhiyu Wu, Xiaokang Chen, Zizheng Pan, Xingchao Liu, Wen Liu, Damai Dai, Huazuo Gao, Yiyang Ma, Chengyue Wu, Bingxuan Wang, et al.
Deepseek-vl2: Mixture-of-experts vision-language models for advanced multimodal understanding.
arXiv preprint arXiv:2412.10302, 2024.
- [4]
Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al.
Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.
arXiv preprint arXiv:2403.05530, 2024.
- [5]
Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al.
Gemma 3 technical report.
arXiv preprint arXiv:2503.19786, 2025.
- [6]
Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman.
The pascal visual object classes (voc) challenge.
International journal of computer vision, 88:303–338, 2010.
- [7]
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.
Microsoft coco: Common objects in context.
In Computer vision–ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part v 13, pages 740–755. Springer, 2014.
- [8]
Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg.
Modeling context in referring expressions.
In European Conference on Computer Vision, pages 69–85. Springer, 2016.
- [9]
Chaoyou Fu, Yi-Fan Zhang, Shukang Yin, Bo Li, Xinyu Fang, Sirui Zhao, Haodong Duan, Xing Sun, Ziwei Liu, Liang Wang, et al.
Mme-survey: A comprehensive survey on evaluation of multimodal llms.
arXiv preprint arXiv:2411.15296, 2024.
- [10]
Lin Li, Guikun Chen, Hanrong Shi, Jun Xiao, and Long Chen.
A survey on multimodal benchmarks: In the era of large ai models.
arXiv preprint arXiv:2409.18142, 2024.
- [11]
Penghao Wu and Saining Xie.
V?: Guided visual search as a core mechanism in multimodal llms.
In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13084–13094, 2024.
- [12]
Yuhang Zang, Wei Li, Jun Han, Kaiyang Zhou, and Chen Change Loy.
Contextual object detection with multimodal large language models.
International Journal of Computer Vision, 133(2):825–843, 2025.
- [13]
Manling Li, Shiyu Zhao, Qineng Wang, Kangrui Wang, Yu Zhou, Sanjana Srivastava, Cem Gokmen, Tony Lee, Erran Li Li, Ruohan Zhang, et al.
Embodied agent interface: Benchmarking llms for embodied decision making.
Advances in Neural Information Processing Systems, 37:100428–100534, 2024.
- [14]
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al.
Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi.
In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9556–9567, 2024.
- [15]
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al.
Mmbench: Is your multi-modal model an all-around player?
In European conference on computer vision, pages 216–233. Springer, 2024.
- [16]
David Fan, Shengbang Tong, Jiachen Zhu, Koustuv Sinha, Zhuang Liu, Xinlei Chen, Michael Rabbat, Nicolas Ballas, Yann LeCun, Amir Bar, et al.
Scaling language-free visual representation learning.
arXiv preprint arXiv:2504.01017, 2025.
- [17]
Chuyang Zhao, YuXin Song, Junru Chen, Kang Rong, Haocheng Feng, Gang Zhang, Shufan Ji, Jingdong Wang, Errui Ding, and Yifan Sun.
Octopus: A multi-modal llm with parallel recognition and sequential understanding.
Advances in Neural Information Processing Systems, 37:90009–90029, 2024.
- [18]
Zicheng Zhang, Haoning Wu, Erli Zhang, Guangtao Zhai, and Weisi Lin.
Q-bench: A benchmark for multi-modal foundation models on low-level vision from single images to pairs.
IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
- [19]
Yuliang Liu, Zhang Li, Mingxin Huang, Biao Yang, Wenwen Yu, Chunyuan Li, Xu-Cheng Yin, Cheng-Lin Liu, Lianwen Jin, and Xiang Bai.
Ocrbench: on the hidden mystery of ocr in large multimodal models.
Science China Information Sciences, 67(12):220102, 2024.
- [20]
Junzhuo Liu, Xuzheng Yang, Weiwei Li, and Peng Wang.
Finecops-ref: A new dataset and task for fine-grained compositional referring expression comprehension.
arXiv preprint arXiv:2409.14750, 2024.
- [21]
Fangyun Wei, Jinjing Zhao, Kun Yan, Hongyang Zhang, and Chang Xu.
A large-scale human-centric benchmark for referring expression comprehension in the lmm era.
Advances in Neural Information Processing Systems, 37:69566–69587, 2024.
- [22]
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al.
Visual genome: Connecting language and vision using crowdsourced dense image annotations.
International journal of computer vision, 123:32–73, 2017.
- [23]
Zebang Cheng, Zhi-Qi Cheng, Jun-Yan He, Kai Wang, Yuxiang Lin, Zheng Lian, Xiaojiang Peng, and Alexander Hauptmann.
Emotion-llama: Multimodal emotion recognition and reasoning with instruction tuning.
Advances in Neural Information Processing Systems, 37:110805–110853, 2024.
- [24]
Chen Cecilia Liu, Fajri Koto, Timothy Baldwin, and Iryna Gurevych.
Are multilingual llms culturally-diverse reasoners? an investigation into multicultural proverbs and sayings.
arXiv preprint arXiv:2309.08591, 2023.
- [25]
Pavan Kartheek Rachabatuni, Filippo Principi, Paolo Mazzanti, and Marco Bertini.
Context-aware chatbot using mllms for cultural heritage.
In Proceedings of the 15th ACM Multimedia Systems Conference, pages 459–463, 2024.
- [26]
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Tianyu Liu, et al.
A survey on in-context learning.
arXiv preprint arXiv:2301.00234, 2022.
- [27]
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al.
Chain-of-thought prompting elicits reasoning in large language models.
Advances in neural information processing systems, 35:24824–24837, 2022.
- [28]
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al.
Flamingo: a visual language model for few-shot learning.
Advances in neural information processing systems, 35:23716–23736, 2022.
- [29]
Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao.
Shikra: Unleashing multimodal llm’s referential dialogue magic.
arXiv preprint arXiv:2306.15195, 2023.
- [30]
Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang, Liangliang Cao, Shih-Fu Chang, and Yinfei Yang.
Ferret: Refer and ground anything anywhere at any granularity.
arXiv preprint arXiv:2310.07704, 2023.
- [31]
Linhui Xiao, Xiaoshan Yang, Xiangyuan Lan, Yaowei Wang, and Changsheng Xu.
Towards visual grounding: A survey.
arXiv preprint arXiv:2412.20206, 2024.
- [32]
Ruilin Yao, Shengwu Xiong, Yichen Zhao, and Yi Rong.
Visual grounding with multi-modal conditional adaptation.
In Proceedings of the 32nd ACM International Conference on Multimedia, pages 3877–3886, 2024.
- [33]
Yifan Jiang, Kexuan Sun, Zhivar Sourati, Kian Ahrabian, Kaixin Ma, Filip Ilievski, Jay Pujara, et al.
Marvel: Multidimensional abstraction and reasoning through visual evaluation and learning.
Advances in Neural Information Processing Systems, 37:46567–46592, 2024.
- [34]
Gabriel Sarch, Lawrence Jang, Michael Tarr, William W Cohen, Kenneth Marino, and Katerina Fragkiadaki.
Vlm agents generate their own memories: Distilling experience into embodied programs of thought.
Advances in Neural Information Processing Systems, 37:75942–75985, 2024.
- [35]
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al.
Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.
arXiv preprint arXiv:2501.12948, 2025.
- [36]
Wenxuan Huang, Bohan Jia, Zijie Zhai, Shaosheng Cao, Zheyu Ye, Fei Zhao, Yao Hu, and Shaohui Lin.
Vision-r1: Incentivizing reasoning capability in multimodal large language models.
arXiv preprint arXiv:2503.06749, 2025.
- [37]
Haozhan Shen, Peng Liu, Jingcheng Li, Chunxin Fang, Yibo Ma, Jiajia Liao, Qiaoli Shen, Zilun Zhang, Kangjia Zhao, Qianqian Zhang, et al.
Vlm-r1: A stable and generalizable r1-style large vision-language model.
arXiv preprint arXiv:2504.07615, 2025.
- [38]
Ziyu Liu, Zeyi Sun, Yuhang Zang, Xiaoyi Dong, Yuhang Cao, Haodong Duan, Dahua Lin, and Jiaqi Wang.
Visual-rft: Visual reinforcement fine-tuning.
arXiv preprint arXiv:2503.01785, 2025.
- [39]
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al.
Segment anything.
In Proceedings of the IEEE/CVF international conference on computer vision, pages 4015–4026, 2023.
- [40]
Wujian Peng, Sicheng Xie, Zuyao You, Shiyi Lan, and Zuxuan Wu.
Synthesize diagnose and optimize: Towards fine-grained vision-language understanding.
In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13279–13288, 2024.
- [41]
Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie.
Eyes wide shut? exploring the visual shortcomings of multimodal llms.
In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9568–9578, 2024.
- [42]
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al.
Imagenet large scale visual recognition challenge.
International journal of computer vision, 115:211–252, 2015.
- [43]
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al.
Laion-5b: An open large-scale dataset for training next generation image-text models.
Advances in neural information processing systems, 35:25278–25294, 2022.
- [44]
Zhecan Wang, Garrett Bingham, Adams Wei Yu, Quoc V Le, Thang Luong, and Golnaz Ghiasi.
Haloquest: A visual hallucination dataset for advancing multimodal reasoning.
In European Conference on Computer Vision, pages 288–304. Springer, 2024.
- [45]
Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al.
The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale.
International journal of computer vision, 128(7):1956–1981, 2020.
- [46]
Weiyun Wang, Yiming Ren, Haowen Luo, Tiantong Li, Chenxiang Yan, Zhe Chen, Wenhai Wang, Qingyun Li, Lewei Lu, Xizhou Zhu, et al.
The all-seeing project v2: Towards general relation comprehension of the open world.
In European Conference on Computer Vision, pages 471–490. Springer, 2024.
- [47]
Holger Caesar, Jasper Uijlings, and Vittorio Ferrari.
Coco-stuff: Thing and stuff classes in context.
In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1209–1218, 2018.
- [48]
Hao Shao, Shengju Qian, Han Xiao, Guanglu Song, Zhuofan Zong, Letian Wang, Yu Liu, and Hongsheng Li.
Visual cot: Advancing multi-modal language models with a comprehensive dataset and benchmark for chain-of-thought reasoning.
Advances in Neural Information Processing Systems, 37:8612–8642, 2024.
- [49]
Yunqiu Xu, Linchao Zhu, and Yi Yang.
Mc-bench: A benchmark for multi-context visual grounding in the era of mllms.
arXiv preprint arXiv:2410.12332, 2024.
- [50]
Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier.
From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions.
Transactions of the association for computational linguistics, 2:67–78, 2014.
- [51]
Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik.
Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models.
In Proceedings of the IEEE international conference on computer vision, pages 2641–2649, 2015.
- [52]
Yunjie Tian, Tianren Ma, Lingxi Xie, and Qixiang Ye.
Chatterbox: Multimodal referring and grounding with chain-of-questions.
Proceedings of the AAAI Conference on Artificial Intelligence, 39(7):7401–7409, Apr. 2025.
- [53]
Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick.
Clevr: A diagnostic dataset for compositional language and elementary visual reasoning.
In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2901–2910, 2017.
- [54]
Zhuowan Li, Xingrui Wang, Elias Stengel-Eskin, Adam Kortylewski, Wufei Ma, Benjamin Van Durme, and Alan L Yuille.
Super-clevr: A virtual benchmark to diagnose domain robustness in visual reasoning.
In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14963–14973, 2023.
- [55]
Fangyu Liu, Guy Emerson, and Nigel Collier.
Visual spatial reasoning.
Transactions of the Association for Computational Linguistics, 11:635–651, 2023.
- [56]
Vlad Hosu, Franz Hahn, Mohsen Jenadeleh, Hanhe Lin, Hui Men, Tamás Szirányi, Shujun Li, and Dietmar Saupe.
The konstanz natural video database (konvid-1k).
In 2017 Ninth international conference on quality of multimedia experience (QoMEX), pages 1–6. IEEE, 2017.
- [57]
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al.
Gpt-4 technical report.
arXiv preprint arXiv:2303.08774, 2023.
- [58]
Fan Liu, Wenshuo Chao, Naiqiang Tan, and Hao Liu.
Bag of tricks for inference-time computation of llm reasoning.
arXiv preprint arXiv:2502.07191, 2025.
- [59]
Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui.
Large language models are not fair evaluators.
arXiv preprint arXiv:2305.17926, 2023.
- [60]
Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Dan Zhang, Diego Rojas, Guanyu Feng, Hanlin Zhao, et al.
Chatglm: A family of large language models from glm-130b to glm-4 all tools.
arXiv preprint arXiv:2406.12793, 2024.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 4
presentation: 3
questions: 1. Could the authors clarify the rationale for categorizing SRC as a reasoning task, given that its scope appears to overlap with understanding-level tasks?
2. Would it be possible for the authors to include or discuss more detailed qualitative error analyses to better illustrate common failure patterns and model limitations across different task levels?
soundness: 2
strengths: 1. LENS unifies eight well-defined tasks over shared images, allowing systematic analysis of inter-level dependencies.
2. Ensures methodological transparency and ethical compliance through detailed dataset documentation and privacy handling.
summary: The paper presents LENS, a multi-level benchmark for evaluating multimodal large language models on perception, understanding, and reasoning tasks within a unified dataset. LENS contains over 60K human-written questions built on 3.4K real-world images, enabling analysis of interdependencies among visual reasoning levels. The authors further propose SMEC, a self-driven multi-expert collaboration framework that improves reasoning performance without external tools. Experiments on 15+ open and closed-source models reveal strong correlations between perception and reasoning performance and highlight persistent challenges in complex reasoning.
weaknesses: 1. The boundaries between perception, understanding, and reasoning tasks are not clearly defined, and some tasks (e.g. SRC as a reasoning task) overlap in scope.
2. Lacks qualitative or fine-grained error analysis, which limits understanding of how and why models fail across different levels.
3. Font size in Figure 6 should be improved for better readability.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 4
questions: Follow-up questions presented along with the weaknesses
soundness: 3
strengths: 1. Hierarchical evaluation:
The benchmark’s three-tiered structure -  spanning perception, understanding, and reasoning - with a consistent data distribution across tasks, enables a more causal interpretation of how lower-level perceptual abilities influence higher-level reasoning (if any).

2. Scale of human annotations:
In terms of human-authored data, the dataset is impressively large and aligns with the current scale of contemporary multimodal benchmarks.

3. Up-to-date and well-curated dataset:
The dataset is well-defined and systematically structured, with 70% images sourced after November 2024 and 50% after January 2025. The images are high-resolution and reflect contemporary visual content, ensuring relevance to real-world scenarios beyond academic testbeds.

4. Strong empirical analysis:
One of the paper’s strongest aspects lies in its thorough analysis and experiments - particularly the dataset exploration in Fig. 4, cross-task synthetic analysis in Fig. 6, and detailed discussions in Sections 2.3, 4.3, 4.4, 4.5.

5. Interesting new SMEC framework:
The proposed Self-Driven Multi-Expert Collaboration (SMEC) is an interesting  idea where the same MLLM, prompted with diverse role-specific instructions, can collaboratively address complex reasoning tasks. It consistently outperforms direct prompting, self-reflection, and majority voting. This approach also raises an intriguing direction for future work on redundancy in self-reflection or voting setups - showing that structured role specialization may better harness the diverse knowledge and reasoning abilities embedded within a single MLLM?

6. Clarity and presentation:
The paper is clearly written and easy to follow, with well-organized appendices that are rich in detail and effectively referenced throughout the main text.
summary: This paper presents Lens, a large-scale, multi-level benchmark for evaluating multimodal reasoning in large vision-language models (MLLMs). 

Unlike prior task-specific datasets, Lens uses a unified image set - 3.4K high-resolution, non-commercially licensed social media image - paired with over 60K human-authored questions across 12 real-world scenarios. Each image supports eight tasks organized into three progressive tiers: perception, understanding, and reasoning, enabling a structured evaluation of how lower-level perceptual skills contribute to higher-level reasoning, while minimizing the effects of data distribution across task categories.

Comprehensive experiments on 15+ state-of-the-art MLLMs, including GPT-4o, Qwen2.5-VL, InternVL3, and reasoning MLLMs like QVQ-Max, and Kimi-VL, reveal strong interdependencies between low-level tasks, like perception and high level tasks like reasoning.

Lastly, the paper also introduces Self-Driven Multi-Expert Collaboration (SMEC), a novel framework in which an MLLM simulates a panel of specialized agents via self-generated, role-specific prompts. SMEC can dynamically refine or even redefine these roles as needed to address a given instruction without external supervision. The authors demonstrate that this approach outperforms established baselines such as majority voting, self-reflection, and direct prompting.
weaknesses: Weaknesses and Questions

1. Limited scenario diversity: While the dataset spans three main scenarios -  education, city, and home - it would benefit from broader coverage. For example, incorporating workplace, outdoor, or social-interaction settings could better capture real-world multimodal reasoning. Additionally, introducing samples grounded in logical, mathematical, or scientific domains, following the same hierarchical design principles, would strengthen the benchmark’s comprehensiveness.

2. Number and organization of tasks per tier: The benchmark currently features a relatively limited number of tasks within each tier. It is unclear whether this was an intentional design choice (e.g., grouping subtasks to facilitate downstream synergy analysis between task tiers) or simply a byproduct of human annotation diversity.

a) Did the authors design subtasks with cross-tier synergy in mind?
b) How do they envision scaling the benchmark to include additional domains (e.g., logic, mathematics, science)?

3. Clarifications on SMEC framework:
The Self-Driven Multi-Expert Collaboration (SMEC) framework is conceptually strong but would benefit from further detail and analysis. Section 3 and the appendix provide helpful descriptions, yet several aspects remain underspecified:

a) How exactly is the diversity metric defined and measured?
b)  Why did the authors choose 3500 samples subset, and how representative it is of the entire benchmark? Do results in Table 3 scale to the entire dataset?
c) What are the types of roles the model tends to generate, and which roles contribute most to performance gains? 
d) are roles domain-centric  (e.g., geometry, culture, ethics) or task-centric roles (e.g., perception vs. reasoning)? Which is better and what scenario? 
e)Could the framework be extended to multi-model setups, where different MLLMs act as specialized experts?

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: There is a formatting problem of the citations in the paper. To cite a paper, \citep should be used instead of \cite.
soundness: 3
strengths: 1. Hierarchical Benchmark Design。 The "Lens" benchmark introduces a valuable multi-level structure (Perception, Understanding, Reasoning) where all tasks are annotated on the same set of images. This is a strength as it uniquely enables the evaluation of synergistic effects。

2. Contemporary and Relevant Dataset. The dataset is built from 3.4K contemporary images manually collected from social media, with a large portion (53%) published after January 2025. This freshness is crucial for fairly evaluating modern MLLMs and mitigating the risk of data contamination from older, widely used training sets.

3. Constructive Contribution with SMEC. Beyond just identifying a problem, the paper proposes a solution with the "Self-Driven Multi-Expert Collaborative Framework" (SMEC). This framework offers an innovative, tool-free method for enhancing MLLM reasoning by simulating a panel of experts. The positive results from SMEC add significant value to the paper.
summary: This paper introduces "Lens," a new multi-level benchmark for evaluating multimodal large language models (MLLMs). The authors argue that existing benchmarks fail to assess how foundational perceptual skills (like object detection) contribute to higher-order reasoning. Lens addresses this by providing 3.4K contemporary images, each annotated for a hierarchy of tasks progressing from perception to understanding and reasoning. The paper evaluates over 15 recent MLLMs and finds that even top models struggle with the reasoning tasks, scoring below 60%. To address this, the authors also propose SMEC, a framework where an MLLM uses self-generated prompts to simulate a "panel of experts" to collaborate on complex reasoning, showing improved performance. A key finding is the confirmation of synergistic effects, where improving low-level perceptual tasks directly facilitates better performance on high-level reasoning.
weaknesses: 1. Limited to Static Images. As acknowledged by the authors, the benchmark is confined to static images. This scope does not capture the complexities of real-world multimodal reasoning, which often involves video, temporal sequences, audio, or long-form narrative understanding.

2. Potential Impracticality of SMEC. The SMEC framework relies on an iterative, multi-step process of generating expert prompts and synthesizing answers. This implies a significant increase in computational overhead and latency at inference time, which could make the approach impractical for real-time applications. The paper demonstrates effectiveness but does not deeply analyze this efficiency trade-off.  For example, if using several small models, how about just using a larger powerful model? Under the same compute, which method is better?

---

confidence: 2
contribution: 1
initial_rating: 2
presentation: 1
questions: •	In which task is the interleaved image-text feature used? Figure 7 does not show the example. 

•	The second link on foot page opens Instagram webpage, not developer agreement. The reviewer didn’t understand the relation of the link and text in the paper. 

•	Do the authors get legal permissions from social media platforms X, Instagram, Weibo and RedNote to use their data? Although some processes, like erasing facial information, have been done for privacy concerns, the images are still copyrighted. For example, the copyright regulations of RedNote state that “RedNote's trademarks, logos, and content are protected by intellectual property laws. You may not use our intellectual property without explicit permission.” According to these cases the authors should get permission from the platforms to use their data for research purposes. If permission has been granted for data use, including documentation of this in the Appendix, is recommended. 
RedNote: https://red-note.co/terms-of-service 

•	It would be valuable to report human performance on the Lens benchmark and the performance comparison between humans and models. 

•	The Related Work section does not clearly articulate how Lens differs from existing benchmarks, particularly in terms of visual and reasoning capabilities. It would be helpful to clarify what unique contributions or challenges Lens introduces that are not addressed by prior datasets.

Suggestions:

•	The organization of the paper could be improved. For instance, placing the 'Related Work' section in the Appendix makes it harder to assess the motivation and novelty, whereas some less critical elements like Figure 2 are included in the main text
soundness: 2
strengths: •	The data is manually filtered and annotated by human verifiers.

•	The process of SMEC is explained very detailly.  

•	Synergistic effects evaluation provides detailed examination of cross-tasks performance.
summary: This paper introduces Lens, a benchmark for evaluating MLLMs on multi-modal reasoning and SMEC, a framework for MLLMs to solve the tasks with self-generated expert opinions. The Lens consists of eight tasks under three main categories: perception, understanding and reasoning. Images (3.4K) of the benchmark are collected manually from variety social media sources. The results of the experiments on Lens show that low-level visual tasks can affect the higher-order reasoning process. When the models apply SMEC for Scene Knowledge Inference task, the accuracy of the models increases with more iterations.
weaknesses: •	The overall writing quality of the paper could be improved. Some sections lack clarity, and certain sentences are difficult to understand.

•	While the paper highlights multi-level evaluation and integrated tasks as key contributions, these concepts are not sufficiently explained or exemplified in the paper.

•	The paper does not clearly justify the need for introducing a new benchmark. Several existing benchmarks already include some of the tasks used in the paper. It remains unclear why extending existing benchmarks with additional tasks would not be sufficient to analyze cross-task interactions. 

•	The authors did not explain why the community needs a new benchmark. Other benchmarks have already included some tasks. By adding other task-related questions to the existing benchmarks and data, the same effect can be achieved. In that case, the evaluation of the synergistic effect does not require generating a new benchmark. 

•	It remains unclear how the proposed SMEC framework distinguishes itself from prior work in terms of novelty and contribution. The related work section does not address prior efforts involving expert-based or modular task-solving frameworks. 

•	The experimental validation of SMEC appears limited, as its performance is only evaluated on a single task (Scene Knowledge Inference). To validate its effectiveness, it would be valuable to test SMEC across other tasks and other datasets.


Decision: Conditional Accept (Poster)