ActiView: Evaluating Active Perception Ability for Multimodal Large Language Models
Authors: Ziyue Wang, Chi Chen, Fuwen Luo, Yurui Dong, Yuanchi Zhang, Yuzhuang Xu, Xiaolong Wang, Peng Li, Yang Liu
Accepted at: acl25

ActiView: Evaluating Active Perception Ability for Multimodal Large Language Models

Ziyue Wang*1, Chi Chen*1, Fuwen Luo*1, Yurui Dong3, Yuanchi Zhang1, Yuzhuang Xu1, Xiaolong Wang1, Peng Li*2,4, Yang Liu*1,2,4,5
1Dept. of Comp. Sci. & Tech., Institute for AI, Tsinghua University, Beijing, China
2Institute for AI Industry Research (AIR), Tsinghua University, Beijing, China
3School of Management, Fudan University, Shanghai, China
4Shanghai Artificial Intelligence Laboratory, Shanghai, China
5Jiangsu Collaborative Innovation Center for Language Competence, Jiangsu, China

Abstract

Active perception, a crucial human capability, involves setting a goal based on the current understanding of the environment and performing actions to achieve that goal. Despite significant efforts in evaluating Multimodal Large Language Models (MLLMs), active perception has been largely overlooked. To address this gap, we propose a novel benchmark named ActiView to evaluate active perception in MLLMs. We focus on a specialized form of Visual Question Answering (VQA) that eases and quantifies the evaluation yet challenging for existing MLLMs. Meanwhile, intermediate reasoning behaviors of models are also discussed. Given an image, we restrict the perceptual field of a model, requiring it to actively zoom or shift its perceptual field based on reasoning to answer the question successfully. We conduct extensive evaluation over 30 models, including proprietary and open-source models, and observe that restricted perceptual fields play a significant role in enabling active perception. Results reveal a significant gap in the active perception capability of MLLMs, indicating that this area deserves more attention. We hope that ActiView could help develop methods for MLLMs to understand multimodal inputs in more natural and holistic ways.1

1 Introduction

The advent of Multimodal Large Language Models (MLLMs) has marked a significant milestone in the realm of artificial intelligence, demonstrating capabilities that are increasingly approaching human-like performance (OpenAI, 2023; Liu et al., 2023c; Ye et al., 2024b). This advancement, while promising, also presents new challenges and opportunities for evaluating these models. As a result, the landscape of MLLM evaluation is rapidly evolving, with numerous benchmarks being developed to either comprehensively evaluate models (Fu et al., 2023; Liu et al., 2023d) or to analyze specific aspects of their capabilities (Liu et al., 2023a; Lu et al., 2023; Luo et al., 2024; Xiao et al., 2024; Li et al., 2024b; Nie et al., 2024; Qian et al., 2024).

Despite the extensive efforts devoted to MLLM evaluation, active perception (Bajcsy, 1988; Bajcsy et al., 2018) remains underexplored. Active perception involves understanding the reasons for sensing, choosing what to perceive, and determining the methods, timing, and locations for achieving that perception (Bajcsy et al., 2018). This is important because in the real world, the desired information often does not appear directly in the center of one’s field of vision. Instead, it requires individuals to move their field of view, locate details, and filter out distracting information. For example, in Figure 1, suppose we are looking for information in a giant painting. We need to first shift our view to locate the specific area and then possibly zoom

![Active perception allows humans or models to perform more complex tasks by actively seeking and processing relevant information. In this paper, we evaluate two key active perception abilities for MLLMs: 1) shifting, as real-world scenarios often present limited views and require shifts to obtain new perspectives, and 2) zooming, which helps enhance perception by zooming out for a broader view and zooming in for details.](page_1012_370_388_246.png)

Figure 1: Active perception allows humans or models to perform more complex tasks by actively seeking and processing relevant information. In this paper, we evaluate two key active perception abilities for MLLMs: 1) shifting, as real-world scenarios often present limited views and require shifts to obtain new perspectives, and 2) zooming, which helps enhance perception by zooming out for a broader view and zooming in for details.

*These authors contribute equally.
**Corresponding authors: Peng Li and Yang Liu.
1Codes and data will be available at https://github.com/THUNLP-MT/ActiView
<table>
  <tr>
    <th>Benchmarks</th>
    <th>Evaluation Target</th>
    <th colspan="2">Change of Per. Fields</th>
    <th>Num. Evaluation Instances</th>
    <th>Annotator</th>
  </tr>
  <tr>
    <th></th>
    <th></th>
    <th>Shifting</th>
    <th>Zooming</th>
    <th>Img</th>
    <th></th>
  </tr>
  <tr>
    <td>MME (Fu et al., 2023)</td>
    <td>Visual comprehension</td>
    <td>x</td>
    <td>x</td>
    <td>1.1k</td>
    <td>1.3k</td>
    <td>Manual</td>
  </tr>
  <tr>
    <td>MMBench (Liu et al., 2023d)</td>
    <td>Visual comprehension</td>
    <td>x</td>
    <td>x</td>
    <td>1.8k</td>
    <td>1.8k</td>
    <td>Manual + Auto</td>
  </tr>
  <tr>
    <td>MM-Vet (Yu et al., 2023)</td>
    <td>Integrated capabilities</td>
    <td>x</td>
    <td>x</td>
    <td>200</td>
    <td>218</td>
    <td>Manual*</td>
  </tr>
  <tr>
    <td>Seed-Bench (Li et al., 2023b)</td>
    <td>Visual comprehension</td>
    <td>x</td>
    <td>x</td>
    <td>1.9k*</td>
    <td>24k</td>
    <td>Auto</td>
  </tr>
  <tr>
    <td>BLINK (Fu et al., 2024b)</td>
    <td>Visual perception</td>
    <td>✓</td>
    <td>✓</td>
    <td>7.3k</td>
    <td>3.8k</td>
    <td>Manual</td>
  </tr>
  <tr>
    <td>VIP-Bench (Biernacki et al., 2021)</td>
    <td>Understanding of visual prompt</td>
    <td>x</td>
    <td>x</td>
    <td>303</td>
    <td>303</td>
    <td>Manual</td>
  </tr>
  <tr>
    <td>HallusionBench (Liu et al., 2023a)</td>
    <td>Hallucination</td>
    <td>x</td>
    <td>x</td>
    <td>346</td>
    <td>1.1k</td>
    <td>Manual + Auto</td>
  </tr>
  <tr>
    <td>LogicVista (Xiao et al., 2024)</td>
    <td>Visual logical reasoning</td>
    <td>x</td>
    <td>x</td>
    <td>448</td>
    <td>448</td>
    <td>Manual</td>
  </tr>
  <tr>
    <td>CNT (Roberts et al., 2023)</td>
    <td>Geographic and Geospatial</td>
    <td>✓</td>
    <td>✓</td>
    <td>345</td>
    <td>345</td>
    <td>Manual</td>
  </tr>
  <tr>
    <td>V* (Wu and Xie, 2023)</td>
    <td>Fine-grained visual search</td>
    <td>x</td>
    <td>x</td>
    <td>191</td>
    <td>191</td>
    <td>Manual</td>
  </tr>
  <tr>
    <td>ActiView (Ours)</td>
    <td>Active perception</td>
    <td>✓</td>
    <td>✓</td>
    <td>314</td>
    <td>1,625</td>
    <td>Manual</td>
  </tr>
</table>

Table 1: Comparison with other benchmarks for MLLMs. “Per. Fields”: Perceptual Fields. 1.9k*: Videos. Manual*: A mixture of manual annotation and data from existing benchmarks. Our benchmark focuses on evaluating active perception abilities via changes in visual perceptual fields, including shifting for compensating missing information, zooming for fine-grained details in the current fields, and a combination of both to mimic real-world scenarios.

in to gather detailed information. Intuitively, active perception not only enables a person or model to accomplish more complex tasks, but also has the potential to serve as a good indicator of the level of intelligence of a model, making it a critical capability that warrants thorough evaluation.

However, existing multimodal evaluation benchmarks are not well-suited to assess active perception capabilities. Table 1 summarizes several widely used or recently proposed multimodal evaluation benchmarks, most of them assess models in static perceptual field settings, where models process information presented directly to them without requiring active exploration or dynamic adjustments to their field of view. BLINK (Fu et al., 2024b), V* (Wu and Xie, 2023), and CNT (Roberts et al., 2023) are exceptions, as they utilize dynamic perceptual fields. However, they only consider either shifting or zooming of the field of view in specific scenarios, which are insufficient for measuring active perception capabilities. Therefore, there is a clear need for new evaluation frameworks that can adequately capture active perception abilities across diverse and dynamic environments.

To fill this gap, we introduce a novel benchmark specifically designed to evaluate Active perception through View changes (ActiView). Given the difficulty of comprehensively evaluating such capabilities across all possible scenarios, ActiView focuses on a series of tasks that are feasible to evaluate, yet still present significant challenges to current models. We manually curate a diverse set of instances, each including question-answer pairs and reasoning clues, and follows the Visual Question Answering (VQA) (Antol et al., 2015) format but exhibits additional features: 1) Each question requires an understanding of multiple detailed visual clues in the image to answer accurately. 2) View constraints are imposed, allowing models to perceive only a partial field of view of the full image at a time. This setup explicitly requires models to perform view shifting and zooming to gather necessary information and eliminate potential distractions, simulating the active perception process in real life. 3) In addition to answering visual questions, intermediate reasoning behaviors, such as view selection, also contribute to the evaluation.

Results from over 30 models reveal that these models generally lag behind in active perception. For instance, the strong proprietary model, GPT-4o, only achieved an average score of 66.40% with our designed evaluation pipelines for fundamental abilities, which is notably lower then the human score of 84.67%. Regarding another pipeline that allows models to flexibly integrate these fundamental abilities, GPT-4o achieves 69.54%, implying that combining fundamental active perception abilities can contribute to improvements. Moreover, the average performance gap between proprietary models and open-source models in active perception is considerably smaller within our designed pipelines than those observed in tasks from previous research. Recent small open-source models, in particular, exhibit approaching GPT-4o results. Experimental results suggest that models tend to perform better when given a complete image but struggle to develop a holistic understanding when presented with even all the separate and constrained perceptual fields. These findings highlight the need for further research in active perception and the value of our benchmark for advancing this field.
Figure 2: Examples of ActiView, exhibiting the following features: i) requiring focusing on multiple fine-grained regions; ii) requiring distinguishing distracting information from the entire image; iii) requiring moving of perceptual fields to obtain sufficient visual information to answer questions. During evaluation, models will be given an initial view cropped from the original image as shown above. Visual Information: human-annotated visual clues.

2 Related Works

2.1 MLLM Benchmarks

Extensive efforts have been devoted to developing MLLM evaluation benchmarks (Table 1), covering a wide range of capabilities, including visual comprehension (Fu et al., 2023; Liu et al., 2023d; Fu et al., 2024a), visual perception (Fu et al., 2024b), hallucination (Liu et al., 2023a), and mathematical and logical reasoning (Lu et al., 2023; Xiao et al., 2024). However, most of them rely on a static view of the input image, which is not suited for assessing active perception. While BLINK (Fu et al., 2024b) involves view shifting, and both V* (Wu and Xie, 2023) and CNT (Roberts et al., 2023) require view zooming, active perception is not a prerequisite for solving their evaluation questions, making them insufficient for comprehensive active perception evaluation. In contrast, our benchmark considers both view shifting and zooming, with questions specifically designed to necessitate active perception for answering, which makes it a more robust framework towards active perception evaluation.

2.2 Active Perception in MLLMs

Although MLLMs have attracted extensive interest, less effort has been dedicated to improving the active perception capability of MLLMs. One line of research focuses on improving the ability of processing high-resolution images by using higher-resolution ViTs (Ye et al., 2024b), slicing high-resolution images and then concatenate them (Liu et al., 2024), or directly using LLMs to process raw patches of any resolution (Li et al., 2023a). The other line emphasizes visual search for fine-grained details. SEAL (Wu and Xie, 2023) fine-tunes a framework of two MLLMs to follow the visual search mechanism for precise visual grounding, and V-IRL (Yang et al., 2024) proposes an active detection strategy to improve the comprehension of real-world geospatial information. Despite these efforts, our evaluation results reveal that existing MLLMs still generally lack active perception capabilities. Our benchmark will shed light on evaluating and enhancing active perception in MLLMs.
3 ActiView

Our benchmark examines active perception abilities of models through different perceptual fields, where Actively zooming and shifting of Views (ActiView) are required. We summarize zooming and shifting as core components of active perception, as depicted in Figure 1, which allow us to quantitatively evaluate active perception abilities in both separate and integrated manners. ActiView imitates the behavior of active perception by providing models with a constraint initial view, either a cropped field of the original image or a full image at reduced resolution. As shown in Figure 2, models should search for missing yet critical information through view zooming and/or shifting, while eliminating distractions caused by redundant content within the view.

3.1 Benchmark Overview

When perceiving an image, humans instinctively focus on three principle aspects: the depicted environment, the primary objects, and the events in which these objects are involved. Similarly, we categorize questions in our benchmark into three main types, as shown in Figure 2, which are further divided into eight sub-classes according to the type of visual information and features required to answer the questions. Due to page limitations, detailed descriptions and typical examples for each sub-class are provided in Appendix A.3. Below is a concise overview of categories in ActiView:

• Environment-centric (Type I) involves three sub-classes. Geo-Localization (Geo-Loc) focuses on geographical features unique to specific countries or cities, requiring models to identify geographical locations implied in the image. Orientation (Orient) challenges models to reason from information of natural orientation. Daily-location (Daily-Loc) distincts from Geo-Loc by centering on everyday locations that could appear in most cities, without being tied to a particular country or city.
• Object-centric (Type II) tasks go beyond simple grounding that directly ask for the attributes or relations of objects, by challenging models with distracting information. Three subclasses are Object-attribute (Obj-Attr), that focuses on identifying objects attributes out of distractions that potentially mislead the model; Object-relation (Obj-Rel), that examines spatial relationships among objects while the questions do not explicitly ask for them; and Counting (Count-Dis), that involves counting objects while handling similar but distracting elements that can lead to incorrect answers.
• Event-centric (Type III) focuses on the interactions between humans and objects, such as actions and activities. This category is divided according to the number of objects involved in the target event, including Event-single (Event-S) which focuses on events involving a single item or person; and Event-multi (Event-M) that happens among multiple items or people.

![The statistical distribution of our benchmark.](page_749_186_355_246.png)

Figure 3: The statistical distribution of our benchmark.

3.2 Data Curation and Statistics

Our dataset is manually curated, including image collection, question and option annotation, and visual clue identification. To assess active perception abilities, which require zooming for fine-grained details and shifting for capturing missing information, we select images featuring multiple fine-grained objects and complex scenes or events. Detailed descriptions of the collected images and annotation guidelines can be found in Appendix A. Annotators are also required to identify visual clues to support their answers, as shown in the “Visual Information” columns in Figure 2. To prevent models from selecting answers based solely on the provided options, we adopt a set of more flexible annotation rules than the typical two- or four-option format. Option count ranges from two to seven, many of which are derived directly from the images. For instance, in the Geo-Loc task shown in Figure 2, the options all correspond to visual clues in the image, such as flags representing the UK, France, Spain, and Hungary. Furthermore, for options comprised by numbers, they are arranged in random order to avoid biased predictions.

The experiments of automatic data generation are discussed in Appendix K. In summary, we found that powerful models, such as GPT-4V and GPT-4o, fail to satisfy our annotation guidelines.
Example: Question: In which country is the photo taken? Options: A. UK   B. France   C. Spain   D. Hungary

View selection                Question answering
Please select views that could help you answer the question {Question}.
Given the selected views, answer the question:
{Question} {Options}
(a). Zooming Q : Evaluation for the ability to locate and determine fine-grained details

Missing-view examination      Question answering
This is the upper-right view of the image. Do you require more visual information to answer question {Question}?
This is the lower-right view of the image.
*Note: Keep adding views until all views are used or the model responses "No".
These are your selected views. Please answer the question: {Question} {Options}
(b). Shifting ▶: Evaluation for the ability to shift perceptual fields for missing information

Operation determination        Question answering
Given the full image, you can either ZOOM to a sub-view and/or SHIFT to other views that could help you answer the question {Question}.
Zooming and/or Shifting
Given the selected views, answer question:
{Question} {Options}
(c). Mixed: Mixed operation of shifting and zooming (without specifying the operation).

Figure 4: Evaluation pipelines as described in §4. (a) Zooming requires models to select multiple regions to zoom in. It tests one of the fundamental active perception abilities. (b) Shifting challenges models to ask for more necessary information. It tests the other fundamental active perception abilities. (c) Mixed simulates human behavior when shifting perceptual fields for missing information. It is more flexible and applicable in real life compare to the previous two fundamental abilities. Note that while we provide an example in the figure where model delete a zoomed sub-view, the deletion behavior is NOT required. It is to address the compound features of the mixed pipeline (c) compare to the other fundamental pipelines (a) and (b).

They struggle with hallucination when processing multiple images, and fall short on distinguish between visual facts in the image and external world knowledge not present in the image.

Statistics. We collected 314 images and annotated 325 questions, with distribution of categories shown in Figure 3. To further enrich the diversity, each question corresponds to 5 different evaluation instances, assessing active perception across varying components and difficulties. In total, 1,625 evaluation instances are curated. On average, there are 3.24 options and 2.64 sub-views containing visual clues per question, highlighting that a single view is often insufficient for answering accurately, and that the ability to comprehend multiple images jointly is crucial for our benchmark.

4 Evaluation

For thorough investigation, we design three evaluation pipelines for different operations of perceptual fields as illustrated in Figure 4, including two individual pipelines for core components, and a mixed pipeline incorporating both. We set up five different initial views for each question-image pairs, where a full image of limited resolution is used for zooming and mixed pipelines, and four constrained views are applied for the shifting pipeline. These correspond to the 1,625 evaluation instances mentioned in previous section. Due to page limitation, details of pipelines can be found in Appendix E, and evaluated models are discussed in Appendix D.

4.1 Pipelines for Individual Component

We separately investigate two core components, zooming and shifting. The zooming pipeline evaluates the ability to locate and determine necessary fine-grained information. As shown in Figure 4 (a), this pipeline contains two stages, view selection and question answering. In this pipeline, models first select sub-views to zoom in given the initial view, the full image of size \( w \times h \), then answer the question based on these zoomed views. The
<table>
  <tr>
    <th rowspan="2">Metrics</th>
    <th colspan="4">Proprietary Models</th>
    <th colspan="6">Multi-image Open-source Models</th>
    <th colspan="5">Single-image Open-source Models</th>
  </tr>
  <tr>
    <th>Gemini-1.5-pro</th>
    <th>GPT-4o</th>
    <th>Claude3.5 Sonnet</th>
    <th>Qwen2.5-VL VL-7B</th>
    <th>DeepSeek VL2 -7B</th>
    <th>Idefics3 V 2.6 -7B</th>
    <th>MiniCPM V 2.6 -7B</th>
    <th>mPLUG-Owl3 V 2.6 -8B</th>
    <th>LlAVA Intern V 2.6 -8B</th>
    <th>Mantis Phi-3.5 -vision</th>
    <th>GLM-4V9</th>
    <th>InternVL LlAVA</th>
    <th>MGIM</th>
    <th>SEAL</th>
  </tr>
  <tr>
    <td>P_{shift}</td>
    <td>79.00</td>
    <td>79.40</td>
    <td>81.40</td>
    <td>73.72</td>
    <td>71.66</td>
    <td>71.31</td>
    <td>69.13</td>
    <td>68.25</td>
    <td>69.08</td>
    <td>71.03</td>
    <td>64.88</td>
    <td>69.60</td>
    <td>74.93</td>
    <td>70.48</td>
    <td>65.46</td>
    <td>66.24</td>
    <td>65.92</td>
  </tr>
  <tr>
    <td>R_{shift}</td>
    <td>62.63</td>
    <td>69.03</td>
    <td>67.64</td>
    <td>76.24</td>
    <td>55.30</td>
    <td>57.04</td>
    <td>57.04</td>
    <td>68.51</td>
    <td>46.41</td>
    <td>57.04</td>
    <td>58.39</td>
    <td>61.70</td>
    <td>68.57</td>
    <td>58.35</td>
    <td>68.22</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>F_1</td>
    <td>82.87</td>
    <td>73.94</td>
    <td>73.92</td>
    <td>72.10</td>
    <td>62.42</td>
    <td>52.14</td>
    <td>62.50</td>
    <td>68.41</td>
    <td>57.91</td>
    <td>52.06</td>
    <td>33.74</td>
    <td>40.46</td>
    <td>43.47</td>
    <td>43.47</td>
    <td>43.47</td>
    <td>41.44</td>
    <td>67.05</td>
  </tr>
  <tr>
    <td>ACC_{QA}</td>
    <td>72.31</td>
    <td>68.62</td>
    <td>71.69</td>
    <td>68.92</td>
    <td>65.85</td>
    <td>58.15</td>
    <td>61.85</td>
    <td>60.92</td>
    <td>65.23</td>
    <td>56.00</td>
    <td>60.62</td>
    <td>56.62</td>
    <td>56.92</td>
    <td>62.77</td>
    <td>68.92</td>
    <td>34.77</td>
    <td>54.77</td>
  </tr>
  <tr>
    <td>ACC_{Shift-R}</td>
    <td>67.08</td>
    <td>67.08</td>
    <td>65.23</td>
    <td>68.62</td>
    <td>65.54</td>
    <td>61.85</td>
    <td>54.77</td>
    <td>51.69</td>
    <td>53.54</td>
    <td>54.77</td>
    <td>52.92</td>
    <td>50.46</td>
    <td>56.92</td>
    <td>53.85</td>
    <td>51.69</td>
    <td>48.62</td>
    <td>42.77</td>
  </tr>
  <tr>
    <td>ACC_{Shift-E}</td>
    <td><b>67.38</b></td>
    <td>66.77</td>
    <td>66.15</td>
    <td>67.08</td>
    <td>62.77</td>
    <td>59.83</td>
    <td>61.23</td>
    <td>56.31</td>
    <td>57.23</td>
    <td>59.70</td>
    <td>55.38</td>
    <td>54.15</td>
    <td>60.62</td>
    <td>52.92</td>
    <td>52.31</td>
    <td>48.00</td>
    <td>42.77</td>
  </tr>
  <tr>
    <td>ACC_{Shift-M}</td>
    <td>65.54</td>
    <td>65.23</td>
    <td>60.31</td>
    <td>67.38</td>
    <td>64.31</td>
    <td>59.69</td>
    <td>58.15</td>
    <td>55.69</td>
    <td>52.31</td>
    <td>53.23</td>
    <td>52.92</td>
    <td>50.15</td>
    <td>56.00</td>
    <td>52.92</td>
    <td>49.32</td>
    <td>47.69</td>
    <td>40.02</td>
  </tr>
  <tr>
    <td>ACC_{Shift-H}</td>
    <td>67.69</td>
    <td>64.31</td>
    <td>61.85</td>
    <td>68.00</td>
    <td>64.62</td>
    <td>60.31</td>
    <td>55.69</td>
    <td>53.54</td>
    <td>48.62</td>
    <td>52.00</td>
    <td>52.31</td>
    <td>45.54</td>
    <td>52.92</td>
    <td>51.08</td>
    <td>48.00</td>
    <td>50.15</td>
    <td>40.62</td>
  </tr>
  <tr>
    <td>Average ACC</td>
    <td><b>68.00</b></td>
    <td>66.40</td>
    <td>65.05</td>
    <td><b>68.00</b></td>
    <td>65.11</td>
    <td>59.88</td>
    <td>58.34</td>
    <td>55.63</td>
    <td>55.39</td>
    <td>55.14</td>
    <td>54.83</td>
    <td>51.38</td>
    <td>56.68</td>
    <td>54.71</td>
    <td>54.03</td>
    <td>45.84</td>
    <td>44.07</td>
  </tr>
  <tr>
    <td>ACC w/ Human Clues</td>
    <td>72.00</td>
    <td>73.54</td>
    <td>72.31</td>
    <td>70.77</td>
    <td>68.31</td>
    <td>60.92</td>
    <td>62.77</td>
    <td>60.62</td>
    <td>64.92</td>
    <td>73.23</td>
    <td>59.38</td>
    <td>58.15</td>
    <td><b>74.46</b></td>
    <td>68.00</td>
    <td>67.69</td>
    <td>68.31</td>
    <td>56.92</td>
  </tr>
</table>

Table 2: Results of evaluation of individual components, following shifting and zooming pipelines. We list results of some widely-discussed models here, and refer readers to Table 12 for more details. The human performance is **84.67%** referring to Table 7 in Appendix B. “Average AVG”: average scores of question answering accuracy of all settings. The best scores of each row are **bolded** and the best scores in the other model types are **highlighted**.

<table>
  <tr>
    <th rowspan="2">Metrics</th>
    <th colspan="4">Proprietary Models</th>
    <th colspan="6">Multi-image Open-source Models</th>
    <th colspan="5">Single-image Open-source Models</th>
  </tr>
  <tr>
    <th>Claude3.5 Sonnet</th>
    <th>GPT-4o</th>
    <th>Gemini-1.5-pro</th>
    <th>Qwen2.5-VL VL-7B</th>
    <th>Qwen2.5-VL VL2 -7B</th>
    <th>DeepSeek VL2 -7B</th>
    <th>Qwen2.5-VL V 2.6 -7B</th>
    <th>MiniCPM V 2.6 -7B</th>
    <th>Idefics3 V 2.6 -8B</th>
    <th>mPLUG-Owl3 V 2.6 -7B</th>
    <th>GLM-4V9</th>
    <th>InternVL LlAVA</th>
    <th>MGIM</th>
    <th>SEAL</th>
  </tr>
  <tr>
    <td>#zoom</td>
    <td>2.30</td>
    <td>1.61</td>
    <td>1.82</td>
    <td>1.88</td>
    <td>2.51</td>
    <td><b>2.65</b></td>
    <td>1.21</td>
    <td>1.31</td>
    <td>1.16</td>
    <td>2.59</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>#shift</td>
    <td><b>3.15</b></td>
    <td>1.23</td>
    <td>1.65</td>
    <td>1.59</td>
    <td>2.17</td>
    <td>1.74</td>
    <td>1.73</td>
    <td>0.39</td>
    <td>0.59</td>
    <td>1.49</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>#view (diff)</td>
    <td>1.89 (-0.75)</td>
    <td>1.35 (-1.29)</td>
    <td>1.14 (-1.50)</td>
    <td>1.26 (-1.38)</td>
    <td>1.23 (-0.52)</td>
    <td><b>2.43 (-0.21)</b></td>
    <td>0.94 (-1.70)</td>
    <td>0.94 (-1.70)</td>
    <td>0.58 (-2.06)</td>
    <td>1.43 (-1.21)</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>ACC</td>
    <td><b>72.00</b></td>
    <td>69.54</td>
    <td>68.92</td>
    <td>70.77</td>
    <td>65.54</td>
    <td>65.23</td>
    <td>64.62</td>
    <td>64.00</td>
    <td>62.15</td>
    <td>59.69</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
</table>

Table 3: Experimental results of mixed pipeline for integrated components. “#zoom”: average zooming operations; “#shift”: average shifting operations. The most count is marked in blue and the least count in yellow. “#view”: average used views; “diff”: #view-view annotated, indicating the difference between actual selected views and the average sub-views containing visual clues (2.64). “ACC”: question answering accuracy. The best results for #view (diff) and ACC are **bolded** and the second best are **underlined**.

selected sub-views are resized to \( w \times h \), the same as the initial view, to enable a zooming operation. Note that a “None” selection is permitted.

The shifting pipeline addresses the ability to navigate perceptual fields incrementally, mimicking real-world scenarios where complete context is unavailable. It measures the ability to shift perceptual fields for missing information and to infer the answer jointly based on constrained perceptual fields. This is also a two-stage pipeline as in Figure 4 (b). To simulate the movement of human eyes, the model begins with an initial view of size \( w \times h \), and determines if the current views are sufficient for answering the question. If more views are needed, adjacent views will present until the answer can be inferred. Furthermore, we assign four difficulty levels according to human-annotated visual clues contained in the initial views, namely “Shift-R”, “Shift-E”, “Shift-M”, and “Shift-H”, with corresponding settings are detailed in Appendix E.

4.2 Pipeline for Integrated Components

In addition to the individual pipelines, we also implement an automated mixed setting, **mixed pipeline**, that does not specify the type of active perception ability required. As illustrated in Figure 4 (c), models must independently decide whether to zoom, shift, or use both to address different perceptual fields. In contrast to the zooming pipeline, where models answer questions based on all selected views, the mixed pipeline allows models to discard irrelevant views after selection. Unlike the shifting pipeline, the mixed pipeline also provides access to the full image view in addition to cropped sub-views. The mixed pipeline emphasizes model autonomy and requires models to account for all views, including the full one, to ensure unbiased operation decision and view selection. Otherwise, it is at risk of reverting to either zooming or shifting tasks, limiting its evaluation. The need for autonomy, strict adherence to instructions, and comprehensive understanding of all views makes this pipeline suitable only for the most advanced and recent multi-image models.

Templates for the above pipelines are listed in Appendix J.2, J.3 and J.4, respectively. In addition, we enable a general VQA evaluation, where models answer visual questions given full images without zooming or shifting, to serve as the reference for assessing the difficulty of our created benchmark. Detailed prompts are provided in Appendix J.1.

4.3 Processing of Views

In this paper, we focus on the interleaved multi-image setting, which is more practical and natural
than the single-image setting. Multi-image models can naturally comprehend several views at one time during evaluating, allowing us to directly format the images and text in interleaved form. For fairness, we also design two evaluation methods for single-image models, including flatly concatenating multiple images into single one, and converting previously selected images into textual descriptions to retain their major information.

To enable zooming and shifting operations, images are split into four sub-views, ensuring unbiased evaluation across all models, regardless of their training data or image processing strategies. We also discuss different splitting methods (Appendix F.1), and input view processing techniques, including image processing (Appendix F.2) and textual form conversion (Appendix F.3).

5 Results and Analysis

Experimental results of individual components, zooming and shifting, are in listed Table 2. We adopt accuracy as the evaluation metric for question answering, together with measurements of view selection (detailed in Appendix G). Due to space constraints, we only list selected high-performance models in these two tables, and provide elaborated results of all evaluated models in Appendix H, including scores for each categories, sub-classes and difficulties. Experimental results of integrated components from the mixed pipeline are reported in Table 3. The number of candidate options ranges from two to seven, with a random choice baseline of 33.95% on our benchmark. We also conduct human and text-only evaluation to assess the difficulty and robustness of our benchmark. The average performance among six testers is 84.67%, indicating that while our benchmark is feasible for humans, it can still be challenging. The text-only evaluation implies that questions in ActiView cannot be solved solely by commonsense knowledge within models, and that visual information is crucial for completing the tasks.

5.1 Main Results

Results of evaluation for individual components.
We draw four key findings from the pipelines for individual components of active perception.

First, as shown in Table 2, all evaluated models outperform random guessing, indicating their potential to maintain active perception abilities of zooming and shifting. However, even the best proprietary models fall significantly behind human. Second, although proprietary models achieve better overall performances compared to open-source models, the performance gap between these two categories are considerably smaller compared to gaps observed in other tasks from previous research. Moreover, the gap is becoming smaller for some recent released open-source models such as Qwen2.5-VL, which achieves the same highest average score of 68.00% as Gemini-1.5-pro. Third, among open-source models, multi-image models largely outperform single-image models, particularly in shifting evaluations with constrained views. Finally, we observe that the view selection scores, F_1 in particular, are highly related to the final performance. Lower scores of precision, recall and F_1 stand for more unnecessary information given, which also correlates with lower QA performances.

For results of mixed evaluation in Table 3, we observe that the evaluated models benefit from enabling complex active perception and often outperform individual zooming or shifting on average. Notably, MiniCPM-V 2.6 (64.00%) and Idefics3-8B-Llama3 (62.15%) surpass the accuracy of given human-annotated views (62.77% and 60.92%, respectively, from Table 18); and Claude 3.5 Sonnet and Qwen2.5-VL-7B achieve equivalent performance of given human-annotated visual clues.

<table>
  <tr>
    <th></th>
    <th>Human*</th>
    <th>Random</th>
    <th>Text-only (GPT-4o)</th>
  </tr>
  <tr>
    <td>ACC</td>
    <td>84.67</td>
    <td>33.95</td>
    <td>2.45</td>
  </tr>
</table>

Table 4: Human level performances, random choice result (averaged over 10k runs), and text-only evaluation. Detailed discussion of human evaluation is provided in Appendix B, and text-only evaluation across different models is provided in Appendix C.

Results of evaluation for integrated components.
The mixed pipeline encourages models to zoom and/or shift perceptual fields autonomously, mimicking human behaviors and highlighting the effectiveness of active perception. However, during experiments, we noticed that some large multi-image models, typically released months or a year ago, failed to follow instructions in the mixed evaluation, generating irrelevant responses or selecting invalid views, disrupting the mixed process. In contrast, recent models, regardless of size, successfully handled the this evaluation. With the reported counts of operations, we can conclude that models that actively zoom and shift views are likely to present higher question answering scores, thereby exhibiting better active perception ability.
<table>
  <tr>
    <th>Models</th>
    <th>W/ Clues</th>
    <th>Full</th>
    <th>Zooming</th>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>73.54</td>
    <td>67.38</td>
    <td>68.62</td>
  </tr>
  <tr>
    <td>Qwen2-VL-7B</td>
    <td>65.85</td>
    <td>63.08</td>
    <td>64.62</td>
  </tr>
  <tr>
    <td>Qwen2.5-VL-3B</td>
    <td>66.15</td>
    <td>65.85</td>
    <td><b>66.15</b></td>
  </tr>
  <tr>
    <td>Qwen2.5-VL-7B</td>
    <td>70.77</td>
    <td>67.08</td>
    <td>68.92</td>
  </tr>
  <tr>
    <td>InternVL2-8B</td>
    <td>73.23</td>
    <td>58.15</td>
    <td>56.00</td>
  </tr>
  <tr>
    <td>Owen2-VL-7B</td>
    <td>65.85</td>
    <td>63.08</td>
    <td>64.62</td>
  </tr>
  <tr>
    <td>Idefics3-8B</td>
    <td>60.92</td>
    <td>59.08</td>
    <td>58.15</td>
  </tr>
  <tr>
    <td>SEAL</td>
    <td>56.92</td>
    <td>48.31</td>
    <td>54.77</td>
  </tr>
</table>

Table 5: Performance comparison among providing annotation clues (W/ Clues), full images without applying perceptual constraints (Full), and our designed zooming pipeline (Zooming).

Discussion of the necessity of active perception.
To address the effectiveness and significance of active perception, we conduct comparison among three settings: i) providing annotation clues, ii) full images without applying perceptual constraints, and iii) our designed zooming pipeline, to demonstrate the usefulness of active perception, and also highlight the current limitations of models in active perception. We select several high-performance models from Table 2, and report corresponding results in Table 5, where active perception improves over Full for both manually guided scenarios (W/ Clues) and model automated scenarios (Zooming). Results suggest that if models could accurately identify the necessary views (such as obtaining the human-annotated clues), their performance could be further improved via active perception. More case studies are further discussed in Appendix I to support the necessity of active perception.

5.2 Analysis of Different Pipelines

Impact of selected views  Our pipelines involve selecting useful view in their first stages. The reliability of selected views plays a crucial role in the following question answering stage. We refer readers to Appendix H.3 for elaborate discussion on corresponding results (in Table 18).

Overall, lower selection recall tends to correlate with lower VQA accuracy. For example, Idefics3-8B-Llama3 and InternVL2-8B present the lowest recalls (41.09%) among multi-image models, leading to lower accuracies for zooming evaluation, 56.00% and 58.15%, respectively. We also investigate the performance when given groundtruth views that contain human-annotated clues. Generally, models are prompted to generate more accurate answers compared to the pure zooming setting. However, mPLUG-Owl3, Gemini-1.5-pro, and LLaVA-OneVision are only exceptional, whose performance slightly decrease when given visual clues. We argue that they are better at the question answering task rather than exhibiting active perception ability. Additionally, we observe that shifting evaluations tend to require more views for answering questions compared to zooming evaluation, yet often results in inferior overall performance, indicating that models lack the ability to actively shifting perceptual fields under constraints. Thus, we believe that more attention should be paid to evaluating and enhancing active perception abilities of MLLMs given limited perceptual fields.

Performance for different difficulty levels  Generally, the accuracy of question answering and the recall of view selection decrease as the difficulties of the initial views increases. As shown by typical results of LLaVA-OneVision and GLM-4V-9B in Table 2, the gaps between easy and hard settings are as large as 8.61% and 7.70%, respectively. However, exceptions exist for Gemini-1.5-pro and Idefics3, demonstrating different reasons, where one is caused by the recall of selected views, and the other lies in the order of relevant views. Gemini-1.5-pro presents higher recall on Shift-H due to higher selection recall. Idefics3 maintains the same recall for all different settings, but achieves a higher accuracy on Shift-H. We hypothesize that the gain comes from the order of input views, where hard-level evaluation starts with less relevant views while appends more informative views at the end of input image sequence when all the views are selected. Please refer to Appendix H.2 for detailed analysis on shifting evaluation.

5.3 Analysis of View Processing Strategies

We investigate these two aspects in Appendix F.1 and Appendix F.3, respectively. For the splitting settings, the adopted 4 sub-image setting provides fair and reliable evaluation results, which is not only effective and efficient, but also demonstrate a good balance between zooming and shifting evaluations. For the strategy of converting image into text, on the contrary, we observe significant drops of results on both zooming and shifting evaluations for most of investigated models. This suggests that the resizing issue in image concatenation strategy has only a minor impact on the performance. Please refer to Appendix F.1 and Appendix F.3 for details.

5.4 Potential Improving Approaches

In this section, we present a simple experiment to show that existing methods can enhance of ac-
<table>
  <tr>
    <th rowspan="2">Models</th>
    <th colspan="2">Zero-shot</th>
    <th colspan="2">One-shot</th>
  </tr>
  <tr>
    <th>ACC<sub>QA</sub></th>
    <th>\( P_{select} \)</th>
    <th>ACC<sub>QA</sub></th>
    <th>\( P_{select} \)</th>
  </tr>
  <tr>
    <td>Idefics3-8B</td>
    <td>58.15</td>
    <td>71.31</td>
    <td>61.23 (+3.08)</td>
    <td>73.80 (+2.49)</td>
  </tr>
  <tr>
    <td>Qwen2-VL-7B</td>
    <td>64.62</td>
    <td>72.55</td>
    <td>64.91 (+0.29)</td>
    <td>73.19 (+0.64)</td>
  </tr>
  <tr>
    <td>Qwen2.5-VL</td>
    <td>68.92</td>
    <td>73.58</td>
    <td>69.85 (+0.93)</td>
    <td>73.72 (+0.14)</td>
  </tr>
</table>

Table 6: One-shot results on Zooming QA accuracy and view selection performance.

tive perception. However, their effectiveness varies across models and remains limited, highlighting the need for more focused efforts in advancing active perception capabilities. We apply one-shot In-Context Learning (ICL) to guide models in selecting views that are truly required and informative. As shown in Table 6, ICL benefits active perception, as it does to improve the performance of other tasks, yet the degree of improvement is inconsistent and remains limited for some models, such as vision-language models of Qwen family.

6 Conclusion

This paper introduces ActiView, a novel benchmark designed to evaluate the active perception abilities of MLLMs. ActiView simulates real-world scenarios by imposing view constraints on images, requiring models to perform view shifting and/or zooming to gather necessary information for answering questions. Our results indicate that current MLLMs exhibit significantly lower active perception capabilities compared to humans, and that active perception abilities of models will be markedly enhanced by allowing inputs in multi-image interleaved structures. We also observed that models tend to perform better on our zooming evaluations compared to shifting evaluations. This suggests that the evaluated models lack the ability to combine their understandings of constrained perceptual fields to form a holistic perspective of the complete image or the full scene. We hope our benchmark will inspire further research in this critical area.

Acknowledgments

This work is supported by the National Key R&D Program of China (2022ZD0160502), the National Natural Science Foundation of China (No. 62276152). We are truly grateful to Tong Su, Ruqin Zhao, Zhicheng Guo, TongTong Lin, and Qi Wang for conducting the human evaluation, and we appreciate Yang Li for providing the image that helped expand our benchmark.

We are grateful for the majority of involved reviewers in the past year, and the AC in the Febru-

ary round of ACL Rolling Review. These insightful suggestions and thoughtful discussions greatly helped us refine our work and bring it to ACL 2025. At the same time, we are aware of the complexities and imperfections inherent in the peer review process, including the presence of low-effort or unconstructive feedback (even some ridiculous comments) from a minority of reviewers, which we have encountered and felt disappointed. We believe that academic research should be guided by the pursuit of meaningful contributions to future progress, rather than solely optimizing benchmark scores or following trends. We hope that future reviewing processes continue to evolve toward greater rigor, respect, and impact.

Limitations

In this study, we utilize the form of VQA and intermediate reasoning behaviors to assess active perception abilities of models. While pipeline presents significant challenges for current multimodal language models (MLLMs), it does not encompass all aspects of active perception. For instance, it overlooks factors such as perspective distortion, multi-sensor integration, the incorporation of more dynamic or interactive environments, and real-time manipulation. With the development of reasoning ability of models such as OpenAI o1, o3 and DeepSeek-R1, we believe the active perception ability will also emerge or be improved. Moreover, techniques like tool learning and multi-agent collaboration could potentially enhance active perception performance based on existing MLLMs, making these areas worthy for future exploration and improvement. Additionally, active perception could also be used to improve other challenging topics such as hallucination and real-life application such as auto-driving. Also, Considering that these exceed the scope of a single conference paper, we do not include them in this paper, and we solely evaluates the inherent active perception capabilities of the MLLMs, aiming at an in-depth investigation of our primary focus.

References

2024. Phi-3 technical report: A highly capable language model locally on your phone. Preprint, arXiv:2404.14219.

Anthropic. 2024. The Claude 3 Model Family: Opus, Sonnet, Haiku.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. 2015. VQA: visual question answering. In 2015 IEEE International Conference on Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015, pages 2425–2433. IEEE Computer Society.

Ruzena Bajcsy. 1988. Active perception. Proceedings of the IEEE, 76(8):966–1005.

Ruzena Bajcsy, Yiannis Aloimonos, and John K Tsotsos. 2018. Revisiting active perception. Autonomous Robots, 42:177–196.

Lauren Biernacki, Meron Zerihun Demissie, Kidus Birkayehu Workneh, Galane Basha Namomsa, Plato Gebremedhin, Fitisum Assamnew Andargie, Brandon Reagen, and Todd Austin. 2021. VIP-bench: a benchmark suite for evaluating privacy-enhanced computation frameworks. In 2021 International Symposium on Secure and Private Execution Environment Design (SEED), pages 139–149. IEEE.

Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. 2024. How far are we to GPT-4V? closing the gap to commercial multimodal models with open-source suites. ArXiv preprint, abs/2404.16821.

Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. 2023. InterVL: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. ArXiv preprint, abs/2312.14238.

Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven C. H. Hoi. 2023. InstructBLIP: towards general-purpose vision-language models with instruction tuning. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023.

Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhang Qiu, Zhihui Yang, and Jie Tang. 2022. GLM: General language model pretraining with autoregressive blank infilling. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 320–335, Dublin, Ireland. Association for Computational Linguistics.

Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. 2023. MME: A comprehensive evaluation benchmark for multimodal large language models. Preprint, arXiv:2306.13394.

Chaoyou Fu, Yuhuan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Rennui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. 2024a. Video-MME: the first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. ArXiv preprint, abs/2405.21075.

Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A Smith, Wei-Chiu Ma, and Ranjay Krishna. 2024b. Blink: multimodal large language models can see but not perceive. ArXiv preprint, abs/2404.12390.

Dongfu Jiang, Xuan He, Huaye Zeng, Cong Wei, Max Ku, Qian Liu, and Wenhui Chen. 2024. Mantis: interleaved multi-image instruction tuning. ArXiv preprint, abs/2405.01483.

Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloé Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross B. Girshick. 2023. Segment anything. In IEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023, pages 3992–4003. IEEE.

Hugo Laurencón, Léo Tronchon, Matthieu Cord, and Victor Sanh. 2024. What matters when building vision-language models? ArXiv preprint, abs/2405.02246.

Hugo Laurencón, Andrés Marafioti, Victor Sanh, and Léo Tronchon. 2024. Building and better understanding vision-language models: insights and future directions.

Bo Li, Peiyuan Zhang, Jingkang Yang, Yuanhan Zhang, Fanyi Pu, and Ziwei Liu. 2023a. OtterHD: A high-resolution multi-modality model. ArXiv preprint, abs/2311.04219.

Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. 2024a. LLaVA-OneVision: Easy visual task transfer. ArXiv preprint, abs/2408.03326.

Bohao Li, Yuying Ge, Yi Chen, Yixiao Ge, Ruimao Zhang, and Ying Shan. 2024b. SEED-Bench-2-Plus: benchmarking multimodal large language models with text-rich visual comprehension. ArXiv preprint, abs/2404.16790.

Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. 2023b. SEED-Bench: Benchmarking multimodal llms with generative comprehension. Preprint, arXiv:2307.16125.

Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, and Jiaya Jia. 2024c. Mini-gemini: mining the potential of multi-modality vision language models. ArXiv preprint, abs/2403.18814.
Fuxiao Liu, Tianrui Guan, Zongxia Li, Lichang Chen, Yaser Yaoob, Dinesh Manocha, and Tianyi Zhou. 2023a. Hallusionbench: you see what you think? or you think what you see? an image-context reasoning benchmark challenging for gpt-4v (ision), llava-1.5, and other multi-modality models. ArXiv preprint, abs/2310.14566.

Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2023b. Improved baselines with visual instruction tuning. ArXiv preprint, abs/2310.03744.

Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. 2024. LLaVA-NeXT: Improved reasoning, OCR, and world knowledge.

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023c. Visual instruction tuning. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023.

Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahuai Lin. 2023d. MMBench: is your multi-modal model an all-around player? Preprint, arXiv:2307.06281.

Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhushou Li, Hao Yang, Yaofeng Sun, Chengqi Deng, Hanwei Xu, Zhenda Xie, and Chong Ruan. 2024. Deepseek-vl: Towards real-world vision-language understanding. Preprint, arXiv:2403.05525.

Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. 2023. Mathvista: evaluating mathematical reasoning of foundation models in visual contexts. ArXiv preprint, abs/2310.02255.

Fuwen Luo, Chi Chen, Zihao Wan, Zhaolu Kang, Qidong Yan, Yingjie Li, Xiaolong Wang, Siyu Wang, Ziyue Wang, Xiaoyue Mi, et al. 2024. CODIS: Benchmarking Context-Dependent Visual Comprehension for Multimodal Large Language Models. ArXiv preprint, abs/2402.13607.

Jiahao Nie, Gongjie Zhang, Wenbin An, Yap-Peng Tan, Alex C Kot, and Shijian Lu. 2024. MMRel: A relation understanding dataset and benchmark in the mllm era. ArXiv preprint, abs/2406.09121.

OpenAI. 2023. GPT-4 technical report. ArXiv preprint, abs/2303.08774.

OpenAI. 2024. Hello GPT-4o. Accessed on June 16, 2024.

Yusu Qian, Hanrong Ye, Jean-Philippe Fauconnier, Peter Grasch, Yinfei Yang, and Zhe Gan. 2024. MIA-Bench: Towards better instruction following evaluation of multimodal llms. ArXiv preprint, abs/2407.01509.

Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. 2024. Gemini 1.5: unlocking multimodal understanding across millions of tokens of context. ArXiv preprint, abs/2403.05530.

Jonathan Roberts, Timo Lüddecke, Rehan Sheikh, Kai Han, and Samuel Albanie. 2023. Charting new territories: exploring the geographic and geospatial capabilities of multimodal llms. ArXiv preprint, abs/2311.14656.

Qwen Team. 2025. Qwen2.5-vl.

Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyong Lin. 2024a. Qwen2-VL: Enhancing vision-language model’s perception of the world at any resolution.

Ziyue Wang, Chi Chen, Yiqi Zhu, Fuwen Luo, Peng Li, Ming Yan, Ji Zhang, Fei Huang, Maosong Sun, and Yang Liu. 2024b. Browse and Concentrate: comprehending multimodal content via prior-llm context fusion. ArXiv preprint, abs/2402.12195.

Penghao Wu and Saining Xie. 2023. V*: Guided visual search as a core mechanism in multimodal llms. ArXiv preprint, abs/2312.14135.

Zhiyu Wu, Xiaokang Chen, Zizheng Pan, Xingchao Liu, Wen Liu, Damai Dai, Huazuo Gao, Yiyang Ma, Chengye Wu, Bingxuan Wang, Zhenda Xie, Yu Wu, Kai Hu, Jiawei Wang, Yaofeng Sun, Yukun Li, Yishi Piao, Kang Guan, Aixin Liu, Xin Xie, Yuxiang You, Kai Dong, Xingkai Yu, Haowei Zhang, Liang Zhao, Yisong Wang, and Chong Ruan. 2024. Deepseek-vl2: Mixture-of-experts vision-language models for advanced multimodal understanding. Preprint, arXiv:2412.10302.

Yijia Xiao, Edward Sun, Tianyu Liu, and Wei Wang. 2024. LogicVista: Multimodal LLM Logical Reasoning Benchmark in Visual Contexts. ArXiv preprint, abs/2407.04973.

Jihan Yang, Runyu Ding, Ellis Brown, Xiaojuan Qi, and Saining Xie. 2024. V-IRL: Grounding Virtual Intelligence in Real Life.

Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. 2024. MiniCPM-V: A GPT-4V level MLLM on your phone. ArXiv preprint, abs/2408.01800.

Jiabo Ye, Haiyang Xu, Haowei Liu, Anwen Hu, Ming Yan, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. 2024a. mPLUG-Owl3: towards long image-sequence understanding in multi-modal large language models. ArXiv preprint, abs/2408.04840.
Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Anwen Hu, Haowei Liu, Qi Qian, Ji Zhang, and Fei Huang. 2024b. mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13040–13051.

Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. 2023. MM-vet: Evaluating large multimodal models for integrated capabilities. ArXiv preprint, abs/2308.02490.

Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. From recognition to cognition: Visual commonsense reasoning. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 6720–6731. Computer Vision Foundation / IEEE.

Haozhe Zhao, Zefan Cai, Shuzheng Si, Xiaojian Ma, Kaikai An, Liang Chen, Zixuan Liu, Sheng Wang, Wenjuan Han, and Baobao Chang. 2024. MMICL: Empowering vision-language model with multi-modal in-context learning. In The Twelfth International Conference on Learning Representations.

Appendices

A Data Details

A.1 Image Collection

To ensure the clearness of useful visual details, the collected original image should be of high resolution. In practice, we collected images of three resolution levels, including 1920 × 1040, 2250 × 1500, and 5184 × 3456, which are originated from VCR dataset (Zellers et al., 2019), SA-1B dataset (Kirillov et al., 2023), and photos taken in daily life. At the beginning of the image collection process, 30 images are collected from photos taken from daily activities, which are then served as pilots and standards for manually expanding the data scale from SA-1B dataset (Kirillov et al., 2023) and VCR dataset (Zellers et al., 2019). These images should include rich and fine-grained visual details.

A.2 Rules for Annotation

We provide a concise version of instruction used during annotation. For each of the images, annotators should follow the following instructions:

• Questions: (1) Questions should be objective which have one and only one answer regarding the images. (2) The participation of multiple visual clues are preferred. They can be in the same or different regions of the image.
• Options: (1) Options originated from the image itself are preferred. (2) The numeric options should be arrange randomly, neither descending nor ascending order. (3) Options cannot be opposite to each other, except for “Yes” or “No”. (4) The number of options are not restricted to 4, you can provide as many options as long as they are reasonable and are closely related to the question and the image.
• Distraction: Annotator should provide distracting visual clues that could lead to wrong answer (if any).
• Clues: regions in the image that contribute to your annotated answer.

A.3 Detailed Category Description

When perceiving an image, humans intuitively focus on three principle aspects: the environment depicted in the image, the primary objects, and the event that these objects are engaged in. Correspondingly, we summarize the questions in our benchmark into three main categories, environment-centric (Type I), object-centric (Type II), and event-centric (Type III) categories, which are further divided into eight sub-classes according to the specific type of visual information and visual features used for answering the questions, as displayed in Figure 2. For the environment-centric category, three sub-classes are developed:

• Geo-Localization (Geo-Loc) focuses on geographical features that are unique to a country or a city, and requires models to identify geographical locations depicted in target images. Typical questions are “Where is this place located?”, “In which country is the photo taken?”, and etc. Images in this class usually contain unique landmarks such as the Eiffel Tower in Paris and the Atomium in Brussels.
• Orientation (Orient) challenges models to exploit natural orientation information for answering the questions, such as the position of shadows, the position of the sun, and the directional information on street signs. Questions of this type include “Is this a sunset or a sunrise?”, “Where is the sunlight coming from?” and etc.
• Daily-location (Daily-Loc). To distinguish from Geo-Loc, this sub-class concentrates on locations in everyday life that could appear in most of the cities and are not unique to a certain city or country. Images in this sub-class usually depict scenes of museums, restaurants,
shops, etc. The corresponding questions include “Where is this picture most likely taken?”, “Is there a music school nearby?”, and etc.

For the object-centric category, we expect models to exhibit abilities beyond simple grounding tasks that directly ask for the attributes or relations of objects. Questions for this category usually involve distracting information from images, and require models to precisely understand the intentions. Sub-classes are demonstrated as follows:

• Object-attribute (Obj-Attr) addresses objects attributes while distracting information, that potentially lead to incorrect answers, appears in the images. Shown by the Obj-Attr case in Figure 2, the highest price, 69 per kilogram, corresponds to papaya rather than watermelon.
• Object-relation (Obj-Rel) concentrates on the spatial relationships among multiple objects, while the questions do not directly ask for the spatial relationships. It requires models to reason for the correct answer via spatial information. Figure 2 displays an Obj-Rel case in which models should be aware of the relative positions of the feet of the person to the water.
• Counting (Count-Dis). Although it focuses on the number of objects, different from the counting tasks in other benchmarks (Fu et al., 2023; Yu et al., 2023), there are similar but distracting information about the targets in our images. These distracting objects easily confuse models and challenge the abilities to understand and strictly follow instructions. As the Count-Dis case in Figure 2, the jerky on the table are distracting to the answer of question “How many pieces of jerky are hanging on the wall?”.

The event-centric category focuses on the interactions of humans and items, such as movements, actions and activities. This category is divided according to the number of objects involved in the target event as following:

• Event-single (Event-S). There is only one item or person involved in the target event. For example, the image for Event-S in Figure 2 shows one person driving without other people presenting in the image.
• Event-multi (Event-M). Different from Event-S, events of this type happen among multiple items or people. In the Event-M case in Figure 2, the “woman in blue” is engaged in a photo shooting activity in which she is posing and another person is taking photo for her. It requires models to distinguish the event or events that each entities are engaged in.

<table>
  <tr>
    <th>Annotator</th>
    <th>Background</th>
    <th>ACC</th>
    <th>ACC*</th>
    <th>Consis.</th>
  </tr>
  <tr>
    <td>User1</td>
    <td>CS</td>
    <td>73.33</td>
    <td>85.00</td>
    <td>100.00</td>
  </tr>
  <tr>
    <td>User2</td>
    <td>Med</td>
    <td>71.67</td>
    <td>75.00</td>
    <td>93.33</td>
  </tr>
  <tr>
    <td>User3</td>
    <td>Telecom</td>
    <td>85.00</td>
    <td>90.00</td>
    <td>100.00</td>
  </tr>
  <tr>
    <td>User4</td>
    <td>CS</td>
    <td>81.67</td>
    <td>88.33</td>
    <td>83.33</td>
  </tr>
  <tr>
    <td>User5</td>
    <td>CS</td>
    <td>76.67</td>
    <td>85.00</td>
    <td>95.00</td>
  </tr>
  <tr>
    <td>User6</td>
    <td>Art</td>
    <td>70.00</td>
    <td>78.83</td>
    <td>83.33</td>
  </tr>
  <tr>
    <th>Average</th>
    <th></th>
    <th>77.53</th>
    <th>84.67</th>
    <th>94.20</th>
  </tr>
</table>

Table 7: Human level performance and question consistency. Consis.: human-annotated consistency of question-image-option-groundtruth. ACC: accuracy of answering the questions without assistance (i.e., the accuracy for “Human” evaluation). ACC*: accuracy of answering the questions with the help of Internet (i.e., the accuracy for “Human*” evaluation).

B Human Evaluation

We sampled 60 questions for human-level test, and recruit 6 testees, who did not participate in image collection and question annotation, to evaluate the human level performance of our benchmark. These testees are from diverse backgrounds, including computer science (CS), telecommunication (Telecom), Medicine (Med), and Art.

For a fair comparison with the MLLMs, we employ two settings, including a “Human” evaluation that asks testees to answer questions all by themselves, and a “Human*” evaluation that allows testees to use the Internet and LLMs for the required knowledge, because these testees may not be exposed to knowledge that never appear in their everyday life, which MLLMs have already seen in the training data. Note that in the “Human*” evaluation, directly search for the answer to the questions are forbidden. Referring to the question in Figure 4 as an example, testees may search for “what does the national flag of UK/France/Spain/Hungary look like?”, which may provide extra knowledge that helps them to answer the original question. Manual evaluation achieves an average accuracy of 84.67%, which is more than doubled of the random choice result (33.95%), while some models present only slightly higher accuracies compared to the random result. These indicate the potential for models to get improved.

We ask testees to vote for the consistency of the annotated question-image-option-groundtruth
<table>
  <tr>
    <th>Model</th>
    <th>Claude</th>
    <th>GPT-4o</th>
    <th>Qwen2-VL</th>
    <th>MiniCPM-V 2.6</th>
    <th>Idefics3</th>
    <th>Brote-IM-XL</th>
  </tr>
  <tr>
    <th>ACC</th>
    <td>2.14</td>
    <td>2.45</td>
    <td>23.38</td>
    <td>26.77</td>
    <td>44.92</td>
    <td>40.00</td>
  </tr>
  <tr>
    <th>ACC(guess)</th>
    <td>26.07</td>
    <td>37.73</td>
    <td>42.77</td>
    <td>41.54</td>
    <td>47.38</td>
    <td>40.00</td>
  </tr>
</table>

Table 8: Results of text-only evaluation. ACC: answer with commonsense only without random guessing. ACC(guess): guess the answer according to commonsense.

quadruple for the investigation of the reliability of our benchmark. The consistency score represents if the testee agree with these quadruples and find the groundtruth answers and the provided options are practical and reasonable. Our benchmark is reliable indicated by a consistency of 94.2%.

B.1 Analysis of Human Performances

We assess the difficulty and reliability of our benchmark upon human performances in Table 7. We employ two settings for human evaluation, where “Human” asks annotators to answer questions only by themselves, and “Human*” allows annotators to use the Internet and LLMs for extra knowledge that could help answer the questions. This “Human*” evaluation aims at fair comparison as humans may not be exposed to knowledge that never appear in their everyday life, while most of MLLMs should be aware of these knowledge from the training data. Human presents an average accuracy of 77.53%, suggesting that our benchmark is challenging even for human. For a fair comparison with large models, “Human*” achieves an average of 84.67% by allowing searching for world knowledge from the Internet or using LLMs. Human performances (84.67%) are more than doubled of the random result (33.95%), while some models present only slightly higher accuracies compared to the random result. These indicate the potential for models to get improved.

C Text-only Evaluation

We provide a text-only evaluation to measure the amount of commonsense answers with providing images in our benchmark. We conducted two experiments:

• Commonsense-only evaluation. This evaluation aims at measuring the amount of questions that can be answered only via commonsense knowledge without searching for visual clues in the image. The template is as follows: “Please answer questions based on you commonsense knowledge. If you are not able to answer the question based soly on the commonsense knowledge you’ve acquired, please response with ‘None’. Question Options Your answer:”

• Commonsense and data bias evaluation. Considering that current models are trained with a large amount of data and various tasks, they could potentially memories the most frequent answers given a image-question pair. We implement another template to evaluate the amount of data that can be correctly guessed without corresponding context. The template is as follows: “Please answer questions based on you commonsense knowledge. If you are not able to answer, please select a most probable one. Question Options Your answer:”

Results for these text-only evaluations are listed in Table 8. This table indicates that questions in our benchmark cannot be simply answered via commonsense, where two powerful models GPT-4o and Claude achieves only 2.45% and 2.14% for commonsense-only evaluation. The row of ACC(guess) presents results of generating the most probable answers, reflecting the bias obtained from the training corpus. The differences between these two type of evaluation are caused by the ability of instruction-following. We found that Idefics3 and Brote-IM-XL present weaker instruction-following ability compared to other models in this table, that they still exhibit a behavior of guessing when commonsense cannot be used to answer the questions.

Overall, our benchmark requires elaborate observation of the given images and comprehensive understanding of image-question pairs, which cannot be solved simply by commonsense.

D Models

We investigate both proprietary and open-source models. The proprietary models include widely discussed GPT-4o (OpenAI, 2024), Gemini-1.5-pro (Reid et al., 2024), and Claude 3.5 Sonnet (Anthropic, 2024). For open-source models, we carefully select recent and commonly used models of different structures and of difference scales, such as
<table>
  <tr>
    <th>Models</th>
    <th>LLM Backbone</th>
    <th>Vision Encoder</th>
  </tr>
  <tr>
    <td colspan="3"><i>APIs</i></td>
  </tr>
  <tr>
    <td>GPT-4o (OpenAI, 2024)</td>
    <td>gpt-4o</td>
    <td></td>
  </tr>
  <tr>
    <td>Gemini-1.5-pro (Reid et al., 2024)</td>
    <td>gemini-1.5-pro</td>
    <td></td>
  </tr>
  <tr>
    <td>Claude 3.5 Sonnet (Anthropic, 2024)</td>
    <td>claude-3.5-sonnet-20240620</td>
    <td></td>
  </tr>
  <tr>
    <th colspan="3">Open-Source Models</th>
  </tr>
  <tr>
    <td>GLM-4V-9B (Du et al., 2022)</td>
    <td>GLM-4-9B</td>
    <td>CLIP</td>
  </tr>
  <tr>
    <td>SEAL (Wu and Xie, 2023)</td>
    <td>Vicuna-7B</td>
    <td>CLIP ViT-L/14</td>
  </tr>
  <tr>
    <td>InternVL-Vicuna-7B (Chen et al., 2023)</td>
    <td>Vicuna-7B</td>
    <td>InternViT</td>
  </tr>
  <tr>
    <td>InternVL-Vicuna-13B (Chen et al., 2023)</td>
    <td>Vicuna-13B</td>
    <td>InternViT</td>
  </tr>
  <tr>
    <td>InternVL-Vicuna-13B-448px (Chen et al., 2023)</td>
    <td>Vicuna-13B</td>
    <td>InternViT-300M-448px</td>
  </tr>
  <tr>
    <td>InternVL-2.8B (Chen et al., 2024)</td>
    <td>internlm2_5-7b-chat</td>
    <td>InternViT-300M-448px</td>
  </tr>
  <tr>
    <td>MiniCPM-Llama3-V-2.5 (Yao et al., 2024)</td>
    <td>Llama-3-8B</td>
    <td>SigLip-400M</td>
  </tr>
  <tr>
    <td>MiniCPM-V 2.6 (Yao et al., 2024)</td>
    <td>Qwen2-7B</td>
    <td>SigLip-400M</td>
  </tr>
  <tr>
    <td>LLaVA-1.6-13B (Liu et al., 2024)</td>
    <td>Vicuna-13B</td>
    <td>CLIP-ViT/L/14</td>
  </tr>
  <tr>
    <td>LLaVA-1.6-7B (Liu et al., 2024)</td>
    <td>Vicuna-7B</td>
    <td>CLIP-ViT/L/14</td>
  </tr>
  <tr>
    <td>LLaVA-OneVision-7B (Li et al., 2024a)</td>
    <td>Qwen2-7B</td>
    <td>SO400M</td>
  </tr>
  <tr>
    <td>Phi-3.5-Vision (abd, 2024)</td>
    <td>Phi-3.5</td>
    <td>CLIP-ViT-L-16-336</td>
  </tr>
  <tr>
    <td>mPLUG-Owl2-7B (Ye et al., 2024b)</td>
    <td>Llama-2-7B</td>
    <td>CLIP-ViT/L/14</td>
  </tr>
  <tr>
    <td>mPLUG-Owl3-7B (Ye et al., 2024a)</td>
    <td>Qwen2-7B</td>
    <td>Siglip-400m</td>
  </tr>
  <tr>
    <td>Qwen2-VL-5B (Wang et al., 2024a)</td>
    <td>Qwen2-7B</td>
    <td>OpenCLIP-ViT-bigG</td>
  </tr>
  <tr>
    <td>Qwen2.5-VL-3B (Team, 2025)</td>
    <td>Qwen2.5</td>
    <td>trained from scratch</td>
  </tr>
  <tr>
    <td>Qwen2.5-VL-7B (Team, 2025)</td>
    <td>Qwen2.5</td>
    <td>trained from scratch</td>
  </tr>
  <tr>
    <td>Deepseek-VL-7B (Lu et al., 2024)</td>
    <td>Deepseek</td>
    <td>Siglip-large-patch16-384</td>
  </tr>
  <tr>
    <td>Deepseek-VL-2.0 (Wu et al., 2024)</td>
    <td>Deepseek2</td>
    <td>Siglip-400m</td>
  </tr>
  <tr>
    <td>Mantis (Jiang et al., 2024)</td>
    <td>LLaMA-3</td>
    <td>Siglip-400m</td>
  </tr>
  <tr>
    <td>Idefics2-8B (Laurençon et al., 2024)</td>
    <td>Mistral-7B</td>
    <td>Siglip-400m</td>
  </tr>
  <tr>
    <td>Idefics2-8B-base (Laurençon et al., 2024)</td>
    <td>Mistral-7B</td>
    <td>Siglip-400m</td>
  </tr>
  <tr>
    <td>Idefics3-8B-Llama3(Laurençon et al., 2024)</td>
    <td>Mistral-7B</td>
    <td>Siglip-400m</td>
  </tr>
  <tr>
    <td>MMICL-XXL (Zhao et al., 2024)</td>
    <td>FlanT5-XXL-11B</td>
    <td>EVA-G</td>
  </tr>
  <tr>
    <td>Brote-IM-XXL (Wang et al., 2024b)</td>
    <td>FlanT5-XXL-11B</td>
    <td>EVA-G</td>
  </tr>
  <tr>
    <td>MMICL-XL (Zhao et al., 2024)</td>
    <td>FlanT5-XL-3B</td>
    <td>EVA-G</td>
  </tr>
  <tr>
    <td>Brote-IM-XL (Wang et al., 2024b)</td>
    <td>FlanT5-XL-3B</td>
    <td>EVA-G</td>
  </tr>
  <tr>
    <td>Mini-Gemini-7B-HD (Li et al., 2024c)</td>
    <td>LLaMA-3</td>
    <td>CLIP-L</td>
  </tr>
  <tr>
    <td>Mini-Gemini-7B (Li et al., 2024c)</td>
    <td>LLaMA-3</td>
    <td>CLIP-L</td>
  </tr>
</table>

Table 9: The versions of LLM backbone and vision encoder of our evaluated models. For proprietary models, we provide the API version we used.

model families of MiniCPM-V (Yao et al., 2024), LLaVA (Liu et al., 2023b,c), mPLUG-Owl (Ye et al., 2024b,a), Idefics (Laurençon et al., 2024; Laurençon et al., 2024), and etc. Since the awareness of fine-grained details and instruction-aware visual features are significant indicators during evaluation, we also include models specifically optimised on these aspects, such as SEAL (Wu and Xie, 2023) for fine-grained details understanding, and Brote (Wang et al., 2024b) which is trained from InstructBLIP (Dai et al., 2023) for instruction-aware and multi-image comprehension. Details of these models are listed in Table 9. Considering models of different scales, we include a total of 27 models. These models are divided into two types, single-image models that accepting only one image per input, such as LLaVA-1.6 (Liu et al., 2023b) and MiniCPM-Llama3-V-2.5 (Yao et al., 2024); and multi-image models that allow more than one images to appear in the same input, such as Brote and Idefics. We describe the approaches for integrating multiple views into the input for the two types of models in Appendix F.2 and Appendix F.3.

E Evaluation Pipelines

This section will discuss motivations and settings of each pipelines in detail.

Zooming pipeline. It focuses on one of the fundamental factors, zooming, and evaluates the ability to locate and determine fine-grained information necessary to answer questions. As illustrated in Figure 4 (a), this pipeline contains two stages, the view selection and the question answering stages. To simulate the zooming operation, models are required to first select sub-views to be zoomed given the initial view, then answer questions based on these zoomed views. The initial view used in this pipeline is the full image with size \( w \times h \). Each of the selected sub-views will be resized to size \( w \times h \), the same as the initial view. In Figure 4 (a), the zoomed right-upper view is resized as a \( w \times h \) image, and so does the zoomed left-lower view.
Afterwards, models answer the question given the two zoomed views. Please refer to Appendix J.2 for prompt templates.

Shifting pipeline. It addresses the other fundamental factor, shifting, and emphasizes the ability to navigate perceptual fields incrementally, mimicking real-world scenarios where full context is unavailable. It evaluates the ability to shift perceptual fields for missing information and to deduce the answer given perceived perceptual fields following templates in Appendix J.3. This is also a two-stage pipeline as in Figure 4 (b). To simulate the movement of human eyes, models are presented with an initial view, size \( w \times h \), which is a cropped field from the original image, and are asked to determine if the current views are sufficient for answering. Upon receiving positive responses, models are prompted to produce answer given the current view or views. If the model requires more views to infer the answer, an adjacent view will be given until the model can answer the question. For this pipeline, we further assign different difficulties according to human-annotated visual clues contained in the initial views as follows:

• Shift-R: randomly selected initial views.
• Shift-E: easy-level evaluation, where initial views contain at least one entire visual clue for answering the question.
• Shift-M: medium-level evaluation, where initial views contain only partial visual clues for answering the question.
• Shift-H: hard-level evaluation, where no visual clues appear in the initial views.

Mixed pipeline. While the above pipelines permit either zooming or shifting individually, we also implement an automated mixed setting that does not specify the type of active perception ability required. As illustrated in Figure 4 (c), models must independently decide whether to zoom and/or shift to different perceptual fields. Unlike the zooming pipeline, where the model answers questions based on all selected views, in the mixed pipeline, a view would be discarded after selection if the model recognizes it as irrelevant to the question. Compared to the shifting pipeline, the mixed pipeline also provides access to the full image view in addition to cropped sub-views. Appendix J.4 records the employed prompt templates. This pipeline requires models to account for all the sub-views and the full image for unbiased operation determination and view selection. Otherwise, it is at risk of reverting to zooming or shifting evaluation without sufficient and unconverted visual information. Therefore, the mixed pipeline emphasizes the autonomy of models and is only applied to multi-image models.

<table>
  <tr>
    <th>Model</th>
    <th>Splits</th>
    <th>Zooming</th>
    <th>Shift-R</th>
    <th>AVG</th>
  </tr>
  <tr>
    <td>LLaVA-1.6 7B</td>
    <td>4</td>
    <td>68.92</td>
    <td>51.69</td>
    <td>60.31</td>
  </tr>
  <tr>
    <td>LLaVA-1.6 7B</td>
    <td>6</td>
    <td>73.23</td>
    <td>53.85</td>
    <td>63.54</td>
  </tr>
  <tr>
    <td>LLaVA-1.6 7B</td>
    <td>8</td>
    <td>72.92</td>
    <td>48.61</td>
    <td>60.77</td>
  </tr>
  <tr>
    <td>LLaVA-1.6 7B</td>
    <td>9</td>
    <td>66.46</td>
    <td>46.16</td>
    <td>56.31</td>
  </tr>
  <tr>
    <td>LLaVA-1.6 7B</td>
    <td>16</td>
    <td>69.23</td>
    <td>46.15</td>
    <td>57.69</td>
  </tr>
  <tr>
    <td>LLaVA-1.6 13B</td>
    <td>4</td>
    <td>65.23</td>
    <td>53.85</td>
    <td>59.54</td>
  </tr>
  <tr>
    <td>LLaVA-1.6 13B</td>
    <td>6</td>
    <td>71.69</td>
    <td>46.46</td>
    <td>59.07</td>
  </tr>
  <tr>
    <td>LLaVA-1.6 13B</td>
    <td>8</td>
    <td>71.84</td>
    <td>44.00</td>
    <td>57.92</td>
  </tr>
  <tr>
    <td>LLaVA-1.6 13B</td>
    <td>9</td>
    <td>72.00</td>
    <td>43.69</td>
    <td>57.84</td>
  </tr>
  <tr>
    <td>LLaVA-1.6 13B</td>
    <td>16</td>
    <td>73.31</td>
    <td>43.23</td>
    <td>58.27</td>
  </tr>
</table>

Table 10: Experimental results of different splits.

F Discussion on Image Splitting and Processing Strategies

F.1 Image Splitting Settings

In our final pipelines, the original images are equally split into 4 views. We also conduct experiments of splitting into more views and report the results in Table 10. We found that the 4 sub-image setting is able to derive fair and reliable evaluation results, which is not only effective but also efficient. More splits require additional inference time and resources (e.g., the context length, GPU memory, etc.), but they only yield similar trends and conclusions compared to 4 sub-image setting.

Additionally, there are two issues with more splits. First, it is challenging for the ability to process multiple images and understand their relationships. As shown in the table above, when increasing the number of splits, LLaVA-1.6-7b degrades from 60.31 to 57.69 (-2.62) on average, and LLaVA-1.6-13b decreases 1.27 on average. Although increasing the splits would increase the performance of zooming evaluation, the performance of shifting is remarkably decreased. As we focus on active perception concerning both zooming and shifting, a split of 4 would present a decent balance. Second, the necessary information would be more likely to be split into different tiles, causing information loss.

F.2 Processing of Views

The question answering stage of the zooming, shifting, and mixed pipelines, as well as the missing view examination stage of the shifting pipeline,
<table>
  <tr>
    <th>Model</th>
    <th>Visual Info. Type</th>
    <th>Zooming</th>
    <th>Shift-R</th>
  </tr>
  <tr>
    <td>LLaVA-1.6 7B</td>
    <td>Image concatenation<br>Textual descriptions</td>
    <td>68.92<br>60.31 <span style="color:red">-8.61</span></td>
    <td>51.69<br>53.83 <span style="color:green">+2.14</span></td>
  </tr>
  <tr>
    <td>LLaVA-1.6 13B</td>
    <td>Image concatenation<br>Textual descriptions</td>
    <td>65.23<br>60.00 <span style="color:red">-5.23</span></td>
    <td>45.85<br>43.69 <span style="color:green">+2.16</span></td>
  </tr>
  <tr>
    <td>mPLUG-Owl2 7B</td>
    <td>Image concatenation<br>Textual descriptions</td>
    <td>55.38<br>62.77 <span style="color:green">+7.39</span></td>
    <td>47.38<br>54.15 <span style="color:green">+6.77</span></td>
  </tr>
  <tr>
    <td>MiniCPM-Llama3-V-2.5</td>
    <td>Image concatenation<br>Textual descriptions</td>
    <td>61.25<br>61.25 <span style="color:red">-0</span></td>
    <td>60.92<br>60.31 <span style="color:red">-0.61</span></td>
  </tr>
</table>

Table 11: Experimental results providing single-image models with captions as compensation for the invisibility of previous images.

require multi-image inputs if multiple views are selected. In this paper, we primarily focus on the interleaved multi-image setting, since it is more practical and natural compared to the single-image setting. Multi-image models can naturally read and understand multiple views at one time (in the form of different images) during evaluating, and we directly format the images and text in an interleaved format. However, we also propose methods for evaluating powerful single-image models. For these models, we employ two strategies to enable simultaneous understanding of different views. One is to concatenate the required views into a single flattened image, and the other preserves merely the current view as an image while converting the remainings into textual descriptions. The following subsection discusses this in detail.

F.3 Strategies of Processing Multiple Images for Single-image Models

For all the pipelines, multiple views might be selected depending on the response of models, which can be naturally handled by multi-image models. However, for models that only accepts single image per input, we apply different image processing approaches for zooming and shifting pipelines. For the shifting pipeline, we proposed to concatenate the selected views or convert them into textual descriptions to fit the information of multiple images into a single input. The concatenation refer to stitch the images selected views together from left to right to form a single image as the input for the model. This is applicable for both missing view examination stage and question answering stage. For the question answering stage in zooming pipeline, if multiple views were selected in the first stage of our pipelines, we will use the each selected view to ask questions sequentially. After obtaining answers, if the model answers correctly based on any of the views, we consider it a complete and successful view selection.

In addition to direct processing of images, we also propose and investigate methods to deliver visual information by converting images into textual descriptions. This enables single-image models to “see” multiple images in the form of text inputs. This method can be applied to both shifting and zooming settings. When multiple views are required, we preserve merely the current view in the form of image, while converting the remainings into textual descriptions via the prompt “Please describe the image:”. Results of typical single-image models, LLaVA-1.6, mPLUG-Owl2 and MiniCPM-Llama3-V-2.5 are shown in Table 11.

For the strategy of converting image into text, it is supposed to be a compensation for the image concatenation strategy to avoid images being resized. On the contrary, we observe significant drops of results on both zooming and shifting evaluations for most of the investigated models, indicating that the resizing issue of image concatenation strategy has minor influence on the performance. Moreover, the operation to converting images into textual descriptions introduces the influence of other abilities that interferes the evaluation of active perception abilities.

G Measurements of View Selection

We follow the recall, precision and F1 metrics to evaluate the performance of the view selection for zooming setting and the missing view examination for shifting settings. We denote the selected views containing human-annotated clues as \( TP_{op} \), where \( op \) refers to either “zoom”, “shift” or “mix”. \( FN_{op} \) refers to views that contain human-annotated clues but are not selected for answering questions, and \( FP_{op} \) refers to views selected but do not contain human-annotated clues. Finally, the precision,
\( P_{select} \), is calculated as follows:

\[
P_{select} = \frac{TP_{op}}{TP_{op} + FP_{op}}, \quad op \in \{zoom, shift, mix\}.
\]

(1)

It measures the proportion of selected views that are actually relevant. It reflects the model ability to avoid unnecessary or irrelevant views. A higher precision indicates that the model is more efficient in identifying only the information necessary for answering the question. The recall, \( R_{select} \), is calculated as follows:

\[
R_{select} = \frac{TP_{op}}{TP_{op} + FN_{op}}, \quad op \in \{zoom, shift, mix\}.
\]

(2)

This recall score measures the proportion of views correctly identified by the model out of all views containing human-annotated clues. It reflects the model ability to capture the required information. A higher recall indicates that the model is less likely to miss important views during the zooming process. Accordingly, F1 score of view selection, \( F_1 \) is computed as:

\[
F_1 = \frac{2 \cdot P_{select} \cdot R_{select}}{P_{select} + R_{select}}
\]

(3)

H Experimental Results

We reported the full results of 27 models in Table 12. This table preserves the conclusions as discussed in by Table 2. The detailed results of each categories are listed in Table 13, Table 14, Table 15, Table 16, and Table 17, for zooming, Shift-R, Shift-E, Shift-M, and Shift-H, respectively.

H.1 Analysis of Results on Zooming Evaluation

We notice that for the zooming evaluation, except for InternVL and LLaVA-1.6, single-image models fail to achieve equivalent or comparable results (comparing with full image setting), and present performance gap of as large as 29.23% (for Mini-Gemini-7B) where the zooming results are much lower. These imply that some single models are unaware of the location of key visual information required by the target question. On the contrary, multi-image models present comparable or even better scores under the zooming evaluation.

We summarise the zooming results on subclasses from Table 13, that the environment-centric category (including Geo-Loc, Orient, and Daily-Loc) presents significantly higher scores than object-centric and event-centric categories. The reason lies in the fact that questions in environment-centric category require more visual commonsense that most of models learnt from the vast training data. We also notice that Idefics2-8B-base even enlarges the performance gap between environment-centric category and the others by around 40%, which demonstrate extremely unbalanced capabilities of exploiting inherent commonsense and observed visual clues. The most challenging types of instances are Orient, Count-Dis and Event-S, that present even halved scores compared to the other sub-classes. Surprisingly, some of evaluated single-image models achieve better scores or perform equally compared to powerful proprietary models for the zooming evaluation, especially mPLUG-Owl2-7B regarding the object-centric category. We hypothesis that this model possesses strong object recognition ability and is less affected by object hallucination compared to other MLLMs.

H.2 Analysis of Results on Shifting Evaluation

The shifting pipeline aims at mimicking the scenario when humans look for more visual information by shifting the perceptual fields, the previously perceived views cannot be simply erased from memory, and new views are integrated incrementally. Results of Shift-R evaluation are shown in Table 14, and the level-specified shifting evaluation are listed in Table 15, Table 16 and Table 17. Similar to that of zooming evaluation, results on environment-centric category are significantly better than the ones on object-centric and event-centric categories. The results of proprietary models are better than the results of open-source models, and that models for multiple images perform better than models for single image. We observe a trend where, as the difficulty increases, the superiority of open-source multi-image models becomes more evident.

There is an overall trend for all the sub-classed that the accuracy decreases as the difficulty is getting increased. As shown by typical results for LLaVA-OneVision and GLM-4V-9B in Table 2, the gaps between Shift-E and Shift-H are as large as 8.61% and 7.70%, respectively. However, exceptional performances are identified for Gemini-1.5-pro, Idefics3-8B-Llama3, and Mini-Gemini-7B-HD, where the results of Shift-H even outperform the results of Shift-E. One of the reason could be the recall of selected views. For Gemini-1.5-pro, the recall for Shift-H is 47.73%, over 1 point higher than Shift-E (45.29%). We conclude that Gemini-
<table>
  <tr>
    <th rowspan="2">Models</th>
    <th colspan="3">Zooming</th>
    <th colspan="5">Shifting</th>
    <th rowspan="2">Model AVG</th>
  </tr>
  <tr>
    <th>Full Image</th>
    <th>Zooming</th>
    <th>Single View</th>
    <th>Shift-R</th>
    <th>Shift-E</th>
    <th>Shift-M</th>
    <th>Shift-H</th>
  </tr>
  <tr>
    <td colspan="9"><i>proprietary models</i></td>
  </tr>
  <tr>
    <td>Gemini-1.5-pro</td>
    <td>73.85</td>
    <td>72.46</td>
    <td>58.46</td>
    <td>67.08</td>
    <td>67.38</td>
    <td>65.54</td>
    <td>67.69</td>
    <td>68.00</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>67.38</td>
    <td>68.62</td>
    <td>61.23</td>
    <td>67.08</td>
    <td>66.77</td>
    <td>65.23</td>
    <td>64.31</td>
    <td>66.40</td>
  </tr>
  <tr>
    <td>Claude 3.5 Sonnet</td>
    <td>72.92</td>
    <td>71.69</td>
    <td>54.46</td>
    <td>65.23</td>
    <td>66.15</td>
    <td>60.31</td>
    <td>61.85</td>
    <td>65.05</td>
  </tr>
  <tr>
    <td colspan="9"><i>Open-source models for multiple images as input</i></td>
  </tr>
  <tr>
    <td>Qwen2.5-VL-7B</td>
    <td>67.08</td>
    <td>68.92</td>
    <td>47.69</td>
    <td>68.62</td>
    <td>67.08</td>
    <td>67.38</td>
    <td>68.00</td>
    <td>68.00</td>
  </tr>
  <tr>
    <td>Qwen2.5-VL-3B</td>
    <td>65.85</td>
    <td>66.15</td>
    <td>55.08</td>
    <td>65.32</td>
    <td>65.85</td>
    <td>65.54</td>
    <td>65.32</td>
    <td>65.64</td>
  </tr>
  <tr>
    <td>DeepSeek-VL2</td>
    <td>70.46</td>
    <td>65.85</td>
    <td>58.15</td>
    <td>65.54</td>
    <td>65.23</td>
    <td>64.31</td>
    <td>64.62</td>
    <td>65.11</td>
  </tr>
  <tr>
    <td>Qwen2-VL</td>
    <td>63.08</td>
    <td>64.62</td>
    <td>54.46</td>
    <td>61.23</td>
    <td>62.77</td>
    <td>64.31</td>
    <td>61.85</td>
    <td>62.96</td>
  </tr>
  <tr>
    <td>Idefics3-8B-Llama3</td>
    <td>59.08</td>
    <td>58.15</td>
    <td>53.23</td>
    <td>61.85</td>
    <td>59.38</td>
    <td>59.69</td>
    <td>60.31</td>
    <td>59.88</td>
  </tr>
  <tr>
    <td>MiniCPM-V 2.6</td>
    <td>64.62</td>
    <td>61.85</td>
    <td>54.46</td>
    <td>54.77</td>
    <td>61.23</td>
    <td>58.15</td>
    <td>55.69</td>
    <td>58.34</td>
  </tr>
  <tr>
    <td>mPLUG-Owl3</td>
    <td>62.46</td>
    <td>60.92</td>
    <td>54.15</td>
    <td>51.69</td>
    <td>56.31</td>
    <td>55.69</td>
    <td>53.34</td>
    <td>55.63</td>
  </tr>
  <tr>
    <td>LLaVA-OneVision</td>
    <td>64.92</td>
    <td>65.23</td>
    <td>56.92</td>
    <td>53.54</td>
    <td>57.23</td>
    <td>52.31</td>
    <td>48.62</td>
    <td>55.39</td>
  </tr>
  <tr>
    <td>InternVL2-8B</td>
    <td>58.15</td>
    <td>56.00</td>
    <td>45.85</td>
    <td>54.77</td>
    <td>59.70</td>
    <td>53.23</td>
    <td>52.00</td>
    <td>55.14</td>
  </tr>
  <tr>
    <td>Mantis</td>
    <td>59.08</td>
    <td>60.62</td>
    <td>52.92</td>
    <td>52.92</td>
    <td>55.38</td>
    <td>52.92</td>
    <td>52.31</td>
    <td>54.83</td>
  </tr>
  <tr>
    <td>Idefics2-8B</td>
    <td>61.85</td>
    <td>61.85</td>
    <td>55.69</td>
    <td>53.23</td>
    <td>52.92</td>
    <td>49.49</td>
    <td>49.49</td>
    <td>54.58</td>
  </tr>
  <tr>
    <td>Brote-IM-XXL-3B</td>
    <td>54.77</td>
    <td>54.46</td>
    <td>55.69</td>
    <td>51.38</td>
    <td>51.08</td>
    <td>52.62</td>
    <td>47.69</td>
    <td>51.45</td>
  </tr>
  <tr>
    <td>Ph-3.5-Vision</td>
    <td>55.08</td>
    <td>56.62</td>
    <td>48.92</td>
    <td>50.46</td>
    <td>54.15</td>
    <td>50.15</td>
    <td>45.54</td>
    <td>51.38</td>
  </tr>
  <tr>
    <td>DeepSeek-VL-7B</td>
    <td>53.23</td>
    <td>53.23</td>
    <td>49.85</td>
    <td>50.15</td>
    <td>49.85</td>
    <td>51.69</td>
    <td>51.69</td>
    <td>51.32</td>
  </tr>
  <tr>
    <td>Idefics2-8B-base</td>
    <td>52.62</td>
    <td>48.62</td>
    <td>47.69</td>
    <td>49.54</td>
    <td>50.77</td>
    <td>47.69</td>
    <td>47.69</td>
    <td>48.86</td>
  </tr>
  <tr>
    <td>Brote-IM-XXL-11B</td>
    <td>53.85</td>
    <td>54.77</td>
    <td>49.23</td>
    <td>49.85</td>
    <td>50.77</td>
    <td>44.92</td>
    <td>43.69</td>
    <td>48.80</td>
  </tr>
  <tr>
    <td>MMICL-XXL-11B</td>
    <td>51.69</td>
    <td>49.54</td>
    <td>50.15</td>
    <td>49.85</td>
    <td>46.77</td>
    <td>45.54</td>
    <td>48.31</td>
    <td>48.31</td>
  </tr>
  <tr>
    <td>MMICL-XXL-3B</td>
    <td>49.85</td>
    <td>49.85</td>
    <td>44.31</td>
    <td>44.92</td>
    <td>48.92</td>
    <td>45.85</td>
    <td>44.31</td>
    <td>46.77</td>
  </tr>
  <tr>
    <td colspan="9"><i>Open-source models for single image as input</i></td>
  </tr>
  <tr>
    <td>MiniCPM-Llama3-V-2.5</td>
    <td>63.87</td>
    <td>61.25</td>
    <td>54.47</td>
    <td>60.92</td>
    <td>60.31</td>
    <td>59.38</td>
    <td>58.46</td>
    <td>60.06</td>
  </tr>
  <tr>
    <td>GLM-4V-9B</td>
    <td>67.08</td>
    <td>56.92</td>
    <td>53.85</td>
    <td>56.92</td>
    <td>60.62</td>
    <td>56.00</td>
    <td>52.92</td>
    <td>56.68</td>
  </tr>
  <tr>
    <td>InternVL-Vicuna-13B</td>
    <td>56.92</td>
    <td>62.77</td>
    <td>52.31</td>
    <td>53.85</td>
    <td>52.92</td>
    <td>52.92</td>
    <td>51.08</td>
    <td>54.71</td>
  </tr>
  <tr>
    <td>LLaVA-1.6 7B</td>
    <td>55.08</td>
    <td>68.92</td>
    <td>50.15</td>
    <td>51.69</td>
    <td>52.31</td>
    <td>49.23</td>
    <td>48.00</td>
    <td>54.03</td>
  </tr>
  <tr>
    <td>InternVL-Vicuna-7B</td>
    <td>55.38</td>
    <td>65.23</td>
    <td>51.70</td>
    <td>52.92</td>
    <td>51.38</td>
    <td>50.77</td>
    <td>48.62</td>
    <td>53.78</td>
  </tr>
  <tr>
    <td>LLaVA-1.6 13B</td>
    <td>56.92</td>
    <td>65.23</td>
    <td>52.31</td>
    <td>48.85</td>
    <td>55.08</td>
    <td>52.62</td>
    <td>48.92</td>
    <td>53.54</td>
  </tr>
  <tr>
    <td>InternVL-Vicuna-13B-448px</td>
    <td>50.46</td>
    <td>57.85</td>
    <td>45.54</td>
    <td>48.31</td>
    <td>48.92</td>
    <td>48.92</td>
    <td>48.92</td>
    <td>50.46</td>
  </tr>
  <tr>
    <td>mPLUG-Owl2-7B</td>
    <td>55.08</td>
    <td>55.38</td>
    <td>52.30</td>
    <td>47.88</td>
    <td>46.46</td>
    <td>46.46</td>
    <td>46.46</td>
    <td>48.37</td>
  </tr>
  <tr>
    <td>Mini-Gemini-7B-HD</td>
    <td>53.66</td>
    <td>34.47</td>
    <td>31.70</td>
    <td>48.62</td>
    <td>47.69</td>
    <td>50.15</td>
    <td>45.85</td>
    <td>48.35</td>
  </tr>
  <tr>
    <td>SEAL</td>
    <td>48.31</td>
    <td>54.77</td>
    <td>42.77</td>
    <td>42.15</td>
    <td>42.77</td>
    <td>40.02</td>
    <td>40.62</td>
    <td>44.07</td>
  </tr>
  <tr>
    <td>Mini-Gemini-7B</td>
    <td>47.08</td>
    <td>17.85</td>
    <td>47.38</td>
    <td>39.38</td>
    <td>38.15</td>
    <td>38.15</td>
    <td>36.00</td>
    <td>33.91</td>
  </tr>
</table>

Table 12: The evaluation of active perception abilities on our benchmark, including zooming (for limited resolution scenarios), and shifting (for scenarios of limiting the field of views). “Model AVG”: average scores of column “Zooming”, “Shift-R”, “Shift-E”, “Shift-M”, and “Shift-H”. The best scores of each column are bolded and the best scores in each model types are highlighted.

1.5-pro achieves higher accuracy on Shift-H due to the acquisition of more proper views. While Idefics3 presents a different trend. It maintains a recall of 74.64% from Shift-E to Shift-H, but achieves a higher accuracy on Shift-H. There is another potential reason that the performance gain of this model comes from the order of input views. The hard-level evaluation starts with less relevant views and appends more useful views at the end of the image sequence, and the performance of these models are more significantly influenced by the order of presented images compared to the rest models. The degradation of performance is more remarkable for the environment-centric and the object-centric categories compared to the event-centric category. Regarding the increasing of difficulty for the environment-centric and the object-centric categories, we observe gaps of about 10% for models such as LLaVA-OneVision, Idefics2-8B, Brote, MMICL, GLM-4V-9B and Mini-Gemini-7B. These observations indicate that different initial perceptual fields have distinct impacts on instances that requiring demanding attention on subtle changes of fine-grained objects. Results show that GPT-4o consistently outperforms other models in the average score of the environment-centric category, implying robust event capture and understanding capabilities in multi-image scenarios.

H.3 Analysis of View Selection

Our evaluation pipelines involve selecting useful view in their first stages. The reliability of selected views plays a crucial role in the following question answering stage. We compute the recall of used views following Equation 2 in Appendix G, and include results in Table 18, along with accuracy of providing models with groundtruth views. Overall, lower selection recall tends to correlate with lower question answering accuracy. For example, Idefics3-8B-Llama3 and InternVL2-8B present the lowest recalls (41.09%) among multi-image models in Table 18, leading to lower zooming evaluation
<table>
  <tr>
    <th rowspan="2">Models</th>
    <th colspan="1">Type I AVG</th>
    <th colspan="1">Type II AVG</th>
    <th colspan="1">Type III AVG</th>
  </tr>
  <tr>
    <th>Geo-Loc Orient Daily-Loc</th>
    <th>Obj-Attr Obj-Rel Count-Dis</th>
    <th>Event-M Event-S</th>
  </tr>
  <tr>
    <th colspan="4" style="text-align:center;">proprietary models</th>
  </tr>
  <tr>
    <td>Gemini-1.5-pro</td>
    <td>91.89 60.00 92.68 83.33</td>
    <td>80.00 65.22 51.06 65.73</td>
    <td>85.29 57.50 70.27</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>94.59 63.33 85.37 82.44</td>
    <td>66.00 54.35 46.81 56.64</td>
    <td>76.47 65.00 70.27</td>
  </tr>
  <tr>
    <td>Claude 3.5 Sonnet</td>
    <td><b>97.30</b> 50.00 87.80 80.56</td>
    <td>72.00 67.39 42.55 60.84</td>
    <td>82.35 75.00 <b>78.38</b></td>
  </tr>
  <tr>
    <th colspan="4" style="text-align:center;">Open-source models for multiple images as input</th>
  </tr>
  <tr>
    <td>Qwen-2-VL</td>
    <td>97.30 50.00 80.49 77.78</td>
    <td>68.00 65.22 40.43 58.04</td>
    <td>58.82 57.50 58.11</td>
  </tr>
  <tr>
    <td>Idiefcs3-X8B-Llama3</td>
    <td>89.19 56.67 73.17 74.07</td>
    <td>60.00 54.35 29.79 48.25</td>
    <td>58.82 50.00 54.05</td>
  </tr>
  <tr>
    <td>MiniCPM-V 2.6</td>
    <td>86.49 46.67 80.49 73.15</td>
    <td>54.00 56.52 31.91 47.55</td>
    <td>61.76 42.50 51.35</td>
  </tr>
  <tr>
    <td>mPLUG-Owl3</td>
    <td>89.19 53.33 80.49 75.93</td>
    <td>64.00 60.87 36.17 53.85</td>
    <td>58.82 47.50 52.70</td>
  </tr>
  <tr>
    <td>LlVA-OneVision</td>
    <td>91.89 46.67 87.80 77.78</td>
    <td>74.00 58.70 42.55 58.74</td>
    <td>61.76 57.50 59.46</td>
  </tr>
  <tr>
    <td>InternVL2-XB</td>
    <td>75.68 56.67 70.73 68.52</td>
    <td>60.00 47.83 25.53 44.76</td>
    <td>61.76 57.50 59.46</td>
  </tr>
  <tr>
    <td>Mantis</td>
    <td>89.19 41.38 80.00 72.64</td>
    <td>72.00 54.35 44.68 57.34</td>
    <td>54.55 51.28 52.78</td>
  </tr>
  <tr>
    <td>Idiefcs-8B</td>
    <td>89.19 46.67 85.37 81.26</td>
    <td>72.00 43.48 23.45 34.17</td>
    <td>55.33 30.00 33.00</td>
  </tr>
  <tr>
    <td>Brote-IM-XL-1B</td>
    <td>86.49 60.00 81.77 68.52</td>
    <td>60.00 43.48 40.43 48.25</td>
    <td>44.12 47.50 45.95</td>
  </tr>
  <tr>
    <td>Idiefcs2-8B-base</td>
    <td>89.19 56.67 78.05 75.93</td>
    <td>42.00 39.13 29.79 37.06</td>
    <td>23.53 35.00 29.73</td>
  </tr>
  <tr>
    <td>Brote-IM-XXL-1B</td>
    <td>86.49 33.33 80.49 69.44</td>
    <td>58.00 43.48 34.04 45.45</td>
    <td>58.82 45.00 51.35</td>
  </tr>
  <tr>
    <td>MMICL-XXL-11B</td>
    <td>67.57 53.33 65.85 62.96</td>
    <td>52.00 36.96 34.04 41.26</td>
    <td>58.82 35.00 45.95</td>
  </tr>
  <tr>
    <td>MMICL-XL-3B</td>
    <td>70.27 43.33 68.29 62.04</td>
    <td>58.00 34.78 36.17 43.36</td>
    <td>38.24 50.00 44.59</td>
  </tr>
  <tr>
    <th colspan="4" style="text-align:center;">Open-source models for single image as input</th>
  </tr>
  <tr>
    <td>MiniCPM-Llama3-V-2.5</td>
    <td>86.49 53.33 75.61 73.15</td>
    <td>64.00 43.48 31.91 46.85</td>
    <td>50.00 50.00 50.00</td>
  </tr>
  <tr>
    <td>GLM-4V-9B</td>
    <td>78.38 53.33 75.61 70.37</td>
    <td>60.00 45.65 31.91 46.15</td>
    <td>61.76 55.00 58.11</td>
  </tr>
  <tr>
    <td>InternVL-Vicuna-13B</td>
    <td>72.97 43.33 85.37 69.44</td>
    <td>68.00 58.70 29.79 52.45</td>
    <td>73.53 72.50 72.97</td>
  </tr>
  <tr>
    <td>LlVA-1.6-7B</td>
    <td>91.89 <b>66.67</b> 87.80 <b>83.33</b></td>
    <td>76.00 60.87 44.68 60.84</td>
    <td>79.41 65.00 71.62</td>
  </tr>
  <tr>
    <td>InternVL-Vicuna-7B</td>
    <td>86.49 56.67 70.73 68.52</td>
    <td>64.00 43.48 23.45 36.77</td>
    <td>61.76 57.50 61.46</td>
  </tr>
  <tr>
    <td>LlVA-1.6-13B</td>
    <td>94.59 56.67 90.24 82.41</td>
    <td>78.00 69.57 36.17 61.54</td>
    <td>64.71 <b>77.50</b> 71.62</td>
  </tr>
  <tr>
    <td>InternVL-Llacna-13B-448px</td>
    <td>48.65 53.33 63.41 55.56</td>
    <td>74.00 58.70 36.17 56.64</td>
    <td>64.71 62.50 63.51</td>
  </tr>
  <tr>
    <td>mPLUG-Owl2-7B</td>
    <td>91.89 60.00 90.24 82.41</td>
    <td><b>84.00</b> <b>71.74</b> <b>51.06</b> <b>69.23</b></td>
    <td>70.59 70.00 70.72</td>
  </tr>
  <tr>
    <td>Mini-Gemini-7B-HD</td>
    <td>62.16 26.67 26.83 38.89</td>
    <td>26.00 43.48 27.66 32.17</td>
    <td>44.12 25.00 33.78</td>
  </tr>
  <tr>
    <td>SEAL</td>
    <td>70.27 46.67 63.41 61.11</td>
    <td>64.00 50.00 44.68 53.15</td>
    <td>41.18 55.00 48.65</td>
  </tr>
  <tr>
    <td>Mini-Gemini-7B</td>
    <td>37.84 16.67 21.95 25.93</td>
    <td>6.00 17.39 8.51 10.49</td>
    <td>20.59 20.00 20.27</td>
  </tr>
</table>

Table 13: Results on all sub-classes of zooming evaluation.

<table>
  <tr>
    <th rowspan="2">Models</th>
    <th colspan="1">Type I AVG</th>
    <th colspan="1">Type II AVG</th>
    <th colspan="1">Type III AVG</th>
  </tr>
  <tr>
    <th>Geo-Loc Orient Daily-Loc</th>
    <th>Obj-Attr Obj-Rel Count-Dis</th>
    <th>Event-M Event-S</th>
  </tr>
  <tr>
    <th colspan="4" style="text-align:center;">proprietary models</th>
  </tr>
  <tr>
    <td>Gemini-1.5-pro</td>
    <td>91.89 50.00 82.93 76.85</td>
    <td><b>71.00</b> 50.00 <b>55.32</b> 68.84</td>
    <td>70.59 57.50 63.51</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td><b>94.59</b> 63.33 80.49 <b>85.86</b></td>
    <td>74.00 50.00 42.55 55.94</td>
    <td>73.53 67.50 70.27</td>
  </tr>
  <tr>
    <td>Claude 3.5 Sonnet</td>
    <td>91.89 53.33 80.49 77.78</td>
    <td>72.00 52.17 40.43 45.24</td>
    <td>64.71 <b>77.50</b> 66.22</td>
  </tr>
  <tr>
    <th colspan="4" style="text-align:center;">Open-source models for multiple images as input</th>
  </tr>
  <tr>
    <td>Qwen-2-VL</td>
    <td>91.89 50.00 85.37 77.78</td>
    <td>72.00 54.35 38.30 55.24</td>
    <td>52.94 45.00 48.65</td>
  </tr>
  <tr>
    <td>Idiefcs3-X8B-Llama3</td>
    <td>89.19 53.33 83.37 77.78</td>
    <td>64.00 50.00 42.55 52.45</td>
    <td>61.76 52.50 56.76</td>
  </tr>
  <tr>
    <td>MiniCPM-V 2.6</td>
    <td>89.19 53.33 73.17 75.93</td>
    <td>64.00 47.83 25.53 46.15</td>
    <td>40.76 42.50 44.59</td>
  </tr>
  <tr>
    <td>mPLUG-Owl3</td>
    <td>81.08 43.33 73.17 69.79</td>
    <td>70.00 34.78 19.15 41.96</td>
    <td>55.88 40.00 47.30</td>
  </tr>
  <tr>
    <td>LlVA-OneVision</td>
    <td>91.89 46.67 73.17 62.04</td>
    <td>64.00 42.10 31.91 45.45</td>
    <td>35.29 60.00 48.65</td>
  </tr>
  <tr>
    <td>InternVL2-XB</td>
    <td>78.38 50.00 80.49 71.30</td>
    <td>62.00 41.30 19.15 46.85</td>
    <td>52.94 40.00 45.95</td>
  </tr>
  <tr>
    <td>Mantis</td>
    <td>91.89 40.00 70.73 69.44</td>
    <td>70.00 39.13 19.15 39.86</td>
    <td>61.76 52.50 56.76</td>
  </tr>
  <tr>
    <td>Idiefcs-8B</td>
    <td>75.68 60.00 70.98 69.44</td>
    <td>60.00 39.13 19.15 38.36</td>
    <td>47.08 46.85 48.65</td>
  </tr>
  <tr>
    <td>Brote-IM-XL-1B</td>
    <td>86.49 43.33 78.05 71.30</td>
    <td>56.00 34.69 23.40 38.46</td>
    <td>50.00 27.50 37.84</td>
  </tr>
  <tr>
    <td>Idiefcs2-8B-base</td>
    <td>89.19 46.67 85.37 68.52</td>
    <td>56.00 47.83 31.91 45.45</td>
    <td>55.88 32.50 43.24</td>
  </tr>
  <tr>
    <td>Brote-IM-XXL-1B</td>
    <td>70.27 40.00 65.85 60.19</td>
    <td>56.00 47.83 34.04 46.15</td>
    <td>52.94 32.50 41.89</td>
  </tr>
  <tr>
    <td>MMICL-XXL-11B</td>
    <td>67.57 53.33 63.41 60.19</td>
    <td>52.00 45.65 38.30 44.55</td>
    <td>41.18 32.50 36.49</td>
  </tr>
  <tr>
    <td>MMICL-XL-3B</td>
    <td>32.43 50.00 68.55 50.00</td>
    <td>32.00 39.13 29.79 34.97</td>
    <td>32.35 32.50 32.43</td>
  </tr>
  <tr>
    <th colspan="4" style="text-align:center;">Open-source models for single image as input</th>
  </tr>
  <tr>
    <td>MiniCPM-Llama3-V-2.5</td>
    <td><b>94.59</b> 36.67 82.93 74.07</td>
    <td>66.00 50.00 48.94 55.24</td>
    <td>41.18 57.50 50.00</td>
  </tr>
  <tr>
    <td>GLM-4V-9B</td>
    <td>86.49 53.33 80.49 75.00</td>
    <td>62.00 30.43 40.43 44.76</td>
    <td>55.88 52.50 54.05</td>
  </tr>
  <tr>
    <td>InternVL-Vicuna-13B</td>
    <td>62.16 46.67 50.09 57.41</td>
    <td>64.00 54.35 48.25 58.82</td>
    <td>60.00 59.46</td>
  </tr>
  <tr>
    <td>InternVL-Vicuna-7B</td>
    <td>72.97 50.00 60.98 62.04</td>
    <td>60.00 45.65 34.04 46.85</td>
    <td>58.82 47.50 51.53</td>
  </tr>
  <tr>
    <td>InternVL-Vicuna-13B-448px</td>
    <td>50.09 54.09 63.41 51.85</td>
    <td>56.00 41.30 19.15 38.46</td>
    <td>50.00 32.50 38.65</td>
  </tr>
  <tr>
    <td>mPLUG-Owl2-7B</td>
    <td>72.97 53.33 43.90 56.48</td>
    <td>56.00 43.48 25.53 41.96</td>
    <td>58.82 42.50 50.00</td>
  </tr>
  <tr>
    <td>Mini-Gemini-7B-HD</td>
    <td>72.97 33.33 21.95 25.93</td>
    <td>6.00 17.39 8.51 10.49</td>
    <td>20.59 20.00 20.27</td>
  </tr>
  <tr>
    <td>SEAL</td>
    <td>59.46 46.67 43.90 50.00</td>
    <td>36.00 39.13 29.79 34.97</td>
    <td>32.35 32.50 32.43</td>
  </tr>
</table>

Table 14: Results on each sub-classes of Shift-R, shifting with random initial views.

accuracies of 56.00% and 58.15%, respectively.

For zooming evaluation, we also investigate the performance when the given groundtruth views that contain human-annotated clues. Generally, models are prompted to generate more accurate answers compared to the pure zooming setting. However, mPLUG-Owl3, Gemini-1.5-pro, and LLaVA-OneVision are only exceptional, whose perfor-
<table>
  <tr>
    <th rowspan="2">Models</th>
    <th colspan="3">Type I</th>
    <th colspan="2">Type II</th>
    <th colspan="2">Type III</th>
  </tr>
  <tr>
    <th>Geo-Loc Orient Daily-Loc</th>
    <th>AVG</th>
    <th>srv</th>
    <th>Obj-Attr Obj-Rel Count-Dis</th>
    <th>AVG</th>
    <th>Event-M Event-S</th>
    <th>AVG</th>
  </tr>
  <tr>
    <td colspan="8"><i>proprietary models</i></td>
  </tr>
  <tr>
    <td>Gemini-1.5-pro</td>
    <td>91.89</td>
    <td>63.33</td>
    <td><b>90.24</b></td>
    <td><b>83.33</b></td>
    <td>68.00</td>
    <td>50.00</td>
    <td><b>48.94</b></td>
    <td><b>55.94</b></td>
    <td><b>79.41</b></td>
    <td>52.50</td>
    <td>64.86</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>97.30</td>
    <td>53.33</td>
    <td>85.37</td>
    <td>83.41</td>
    <td>62.40</td>
    <td>52.17</td>
    <td>44.68</td>
    <td>53.85</td>
    <td>73.53</td>
    <td>B2.50</td>
    <td><b>72.97</b></td>
  </tr>
  <tr>
    <td>Claude 3.5 Sonnet</td>
    <td>94.59</td>
    <td>70.00</td>
    <td>80.49</td>
    <td>82.41</td>
    <td>66.00</td>
    <td>52.17</td>
    <td>38.30</td>
    <td>52.45</td>
    <td>70.59</td>
    <td>65.00</td>
    <td>67.57</td>
  </tr>
  <tr>
    <td colspan="8"><i>Open-source models for multiple images as input</i></td>
  </tr>
  <tr>
    <td>Qwen2-VL</td>
    <td>94.59</td>
    <td>53.33</td>
    <td>85.37</td>
    <td>70.63</td>
    <td>68.00</td>
    <td>60.87</td>
    <td>40.43</td>
    <td>56.64</td>
    <td>50.00</td>
    <td>50.00</td>
    <td>50.00</td>
  </tr>
  <tr>
    <td>Idafcons-38-BLlama3</td>
    <td>89.19</td>
    <td>46.67</td>
    <td>82.93</td>
    <td>75.00</td>
    <td>60.00</td>
    <td>56.52</td>
    <td>40.43</td>
    <td>52.45</td>
    <td>58.82</td>
    <td>42.50</td>
    <td>50.00</td>
  </tr>
  <tr>
    <td>MiniCPM-V 2.6</td>
    <td>89.19</td>
    <td>63.33</td>
    <td>78.05</td>
    <td>77.78</td>
    <td><b>76.00</b></td>
    <td>58.70</td>
    <td>23.40</td>
    <td>53.15</td>
    <td>52.94</td>
    <td>52.50</td>
    <td>52.70</td>
  </tr>
  <tr>
    <td>mPLUG-Ow13</td>
    <td>83.78</td>
    <td>46.67</td>
    <td>78.05</td>
    <td>71.30</td>
    <td>62.00</td>
    <td>52.17</td>
    <td>27.66</td>
    <td>47.55</td>
    <td>52.94</td>
    <td>50.00</td>
    <td>51.35</td>
  </tr>
  <tr>
    <td>LLaVA-OneVision</td>
    <td>70.27</td>
    <td>43.33</td>
    <td>70.13</td>
    <td>62.96</td>
    <td><b>76.00</b></td>
    <td>60.87</td>
    <td>29.79</td>
    <td>55.94</td>
    <td>58.82</td>
    <td>45.00</td>
    <td>51.35</td>
  </tr>
  <tr>
    <td>InternV1.2-SE</td>
    <td>83.78</td>
    <td>63.33</td>
    <td>63.41</td>
    <td>70.37</td>
    <td>66.00</td>
    <td>56.52</td>
    <td>36.17</td>
    <td>53.15</td>
    <td>44.12</td>
    <td>67.50</td>
    <td>56.76</td>
  </tr>
  <tr>
    <td>Mantis</td>
    <td>91.89</td>
    <td>36.67</td>
    <td>70.73</td>
    <td>68.52</td>
    <td>72.00</td>
    <td>52.17</td>
    <td>24.00</td>
    <td>49.65</td>
    <td>50.00</td>
    <td>45.00</td>
    <td>47.30</td>
  </tr>
  <tr>
    <td> dafür-8B</td>
    <td>83.78</td>
    <td>53.33</td>
    <td>63.28</td>
    <td>70.37</td>
    <td>66.00</td>
    <td>56.52</td>
    <td>36.17</td>
    <td>53.15</td>
    <td>44.12</td>
    <td>67.50</td>
    <td>56.76</td>
  </tr>
  <tr>
    <td>Brote-IM-XL-3B</td>
    <td>83.78</td>
    <td>53.33</td>
    <td>58.29</td>
    <td>58.33</td>
    <td>64.00</td>
    <td>50.00</td>
    <td>44.68</td>
    <td>53.15</td>
    <td>41.18</td>
    <td>50.00</td>
    <td>33.24</td>
  </tr>
  <tr>
    <td>Gmund2-8B-base</td>
    <td>81.08</td>
    <td>43.33</td>
    <td>85.37</td>
    <td>72.22</td>
    <td>62.00</td>
    <td>39.13</td>
    <td>25.53</td>
    <td>42.66</td>
    <td>48.11</td>
    <td>56.18</td>
    <td>38.61</td>
  </tr>
  <tr>
    <td>Brote-IM-XXL-11B</td>
    <td>64.86</td>
    <td>33.33</td>
    <td>60.98</td>
    <td>54.63</td>
    <td>58.00</td>
    <td>52.17</td>
    <td>46.81</td>
    <td>52.45</td>
    <td>43.81</td>
    <td>42.50</td>
    <td>41.89</td>
  </tr>
  <tr>
    <td>MMICL-XXL-11B</td>
    <td>62.16</td>
    <td>36.67</td>
    <td>60.98</td>
    <td>54.63</td>
    <td>62.00</td>
    <td>41.30</td>
    <td>46.81</td>
    <td>50.35</td>
    <td>41.18</td>
    <td>42.50</td>
    <td>41.89</td>
  </tr>
  <tr>
    <td>MMICL-XL-3B</td>
    <td>56.76</td>
    <td>46.67</td>
    <td>68.29</td>
    <td>58.33</td>
    <td>56.00</td>
    <td>45.65</td>
    <td>40.43</td>
    <td>47.55</td>
    <td>41.18</td>
    <td>35.00</td>
    <td>37.84</td>
  </tr>
</table>

Table 15: Results on sub-classes of Shift-E (the easy-level shifting evaluation), where initial views contain clues for answering the question.

<table>
  <tr>
    <th rowspan="2">Models</th>
    <th colspan="3">Type I</th>
    <th colspan="2">Type II</th>
    <th colspan="2">Type III</th>
  </tr>
  <tr>
    <th>Geo-Loc Orient Daily-Loc</th>
    <th>AVG</th>
    <th>srv</th>
    <th>Obj-Attr Obj-Rel Count-Dis</th>
    <th>AVG</th>
    <th>Event-M Event-S</th>
    <th>AVG</th>
  </tr>
  <tr>
    <td colspan="8"><i>proprietary models</i></td>
  </tr>
  <tr>
    <td>Gemini-1.5-pro</td>
    <td>89.19</td>
    <td>60.00</td>
    <td><b>81.48</b></td>
    <td>70.00</td>
    <td>47.83</td>
    <td>42.55</td>
    <td><b>53.85</b></td>
    <td><b>73.53</b></td>
    <td>60.00</td>
    <td>66.22</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>94.59</td>
    <td>53.33</td>
    <td>85.37</td>
    <td>79.63</td>
    <td>64.00</td>
    <td>52.17</td>
    <td>42.55</td>
    <td>53.15</td>
    <td>67.65</td>
    <td>70.00</td>
    <td>68.92</td>
  </tr>
  <tr>
    <td>Claude 3.5 Sonnet</td>
    <td>89.19</td>
    <td>46.67</td>
    <td>75.61</td>
    <td>72.22</td>
    <td>66.00</td>
    <td>54.35</td>
    <td>40.43</td>
    <td>53.85</td>
    <td>58.88</td>
    <td>52.50</td>
    <td>54.05</td>
  </tr>
  <tr>
    <td colspan="8"><i>Open-source models for multiple images as input</i></td>
  </tr>
  <tr>
    <td>Qwen2-VL</td>
    <td>91.89</td>
    <td>50.00</td>
    <td>85.37</td>
    <td>77.78</td>
    <td><b>76.00</b></td>
    <td>60.87</td>
    <td>40.43</td>
    <td><b>59.44</b></td>
    <td>55.88</td>
    <td>52.50</td>
    <td>54.05</td>
  </tr>
  <tr>
    <td>IdAfcons-38-BLlama3</td>
    <td>86.49</td>
    <td>50.00</td>
    <td>84.03</td>
    <td>74.07</td>
    <td>64.00</td>
    <td>52.17</td>
    <td>40.43</td>
    <td>52.45</td>
    <td>55.88</td>
    <td>50.00</td>
    <td>52.70</td>
  </tr>
  <tr>
    <td>MiniCPM-V 2.6</td>
    <td>86.49</td>
    <td><b>63.33</b></td>
    <td>73.17</td>
    <td>75.00</td>
    <td>60.00</td>
    <td>56.52</td>
    <td>40.28</td>
    <td>56.52</td>
    <td>55.88</td>
    <td>55.00</td>
    <td>51.58</td>
  </tr>
  <tr>
    <td>mPLUG-Ow13</td>
    <td>81.46</td>
    <td>46.67</td>
    <td>61.79</td>
    <td>69.29</td>
    <td>62.00</td>
    <td>52.17</td>
    <td>40.35</td>
    <td>47.17</td>
    <td>55.88</td>
    <td>50.00</td>
    <td>53.41</td>
  </tr>
  <tr>
    <td>LIAA-OneVision</td>
    <td>56.76</td>
    <td>46.67</td>
    <td>63.41</td>
    <td>64.81</td>
    <td>62.00</td>
    <td>55.62</td>
    <td>27.66</td>
    <td>42.88</td>
    <td>47.17</td>
    <td>41.20</td>
    <td>57.20</td>
  </tr>
  <tr>
    <td>InternV1.2-SE</td>
    <td>78.38</td>
    <td>50.00</td>
    <td>63.41</td>
    <td>64.38</td>
    <td>66.00</td>
    <td>43.48</td>
    <td>39.15</td>
    <td>33.16</td>
    <td>47.55</td>
    <td>44.12</td>
    <td>50.00</td>
    <td>47.30</td>
  </tr>
  <tr>
    <td>Mantis</td>
    <td>89.19</td>
    <td>36.67</td>
    <td>65.85</td>
    <td>67.54</td>
    <td>62.00</td>
    <td>54.35</td>
    <td>19.15</td>
    <td>45.45</td>
    <td>55.88</td>
    <td>42.50</td>
    <td>46.85</td>
  </tr>
  <tr>
    <td> dafür-8B</td>
    <td>83.78</td>
    <td><b>63.33</b></td>
    <td>73.17</td>
    <td>67.59</td>
    <td>56.00</td>
    <td>43.48</td>
    <td>21.28</td>
    <td>40.56</td>
    <td>52.94</td>
    <td>50.00</td>
    <td>51.35</td>
  </tr>
  <tr>
    <td>Brote-IM-XL-3B</td>
    <td>48.05</td>
    <td>36.67</td>
    <td>63.41</td>
    <td>50.93</td>
    <td>54.00</td>
    <td>50.08</td>
    <td>44.68</td>
    <td>49.65</td>
    <td>42.12</td>
    <td>50.00</td>
    <td>39.19</td>
  </tr>
  <tr>
    <td>Gmund2-8B-base</td>
    <td>75.68</td>
    <td>43.33</td>
    <td>82.93</td>
    <td>69.44</td>
    <td>52.20</td>
    <td>41.30</td>
    <td>28.78</td>
    <td>22.86</td>
    <td>41.18</td>
    <td>42.50</td>
    <td>32.43</td>
  </tr>
  <tr>
    <td>Brote-IM-XXL-11B</td>
    <td>56.76</td>
    <td>43.00</td>
    <td>63.41</td>
    <td>51.85</td>
    <td>42.00</td>
    <td>41.30</td>
    <td>42.55</td>
    <td>41.96</td>
    <td>44.12</td>
    <td>44.76</td>
    <td>38.73</td>
  </tr>
  <tr>
    <td>MMICL-XXL-11B</td>
    <td>56.76</td>
    <td>47.00</td>
    <td>58.74</td>
    <td>58.03</td>
    <td>50.00</td>
    <td>45.24</td>
    <td>30.28</td>
    <td>43.15</td>
    <td>40.47</td>
    <td>35.91</td>
    <td>38.00</td>
  </tr>
  <tr>
    <td>MMICL-313-13B</td>
    <td>70.43</td>
    <td>43.33</td>
    <td>48.78</td>
    <td>59.22</td>
    <td>52.20</td>
    <td>34.84</td>
    <td>23.78</td>
    <td>24.47</td>
    <td>40.00</td>
    <td>33.78</td>
    <td>50.00</td>
  </tr>
</table>

Table 16: Results on each sub-classes of Shift-M (the medium-level shifting evaluation), where initial views contain only partial clues for answering the questions.

formance slightly degrade when given the visual clues. We argue that these models are better at the question answering task rather than exhibiting active perception ability. Additionally, we observe that shifting evaluations tend to require more views to be used for answering questions than zooming evaluations, yet it often results in inferior overall performance compared to zooming. For the shifting
<table>
  <tr>
    <th rowspan="2">Models</th>
    <th colspan="3">Type I</th>
    <th colspan="3">Type II</th>
    <th colspan="3">Type III</th>
  </tr>
  <tr>
    <th>AVG</th>
    <th>Geo-Loc Orient Daily-Loc.</th>
    <th>Obj-Attr Obj-Rel Count-Dis.</th>
    <th>AVG</th>
    <th>Obj-Attr Obj-Rel Count-Dis.</th>
    <th>AVG</th>
    <th>Event-M Event-S</th>
    <th>AVG</th>
  </tr>
  <tr>
    <td colspan="9"><i>proprietary models</i></td>
  </tr>
  <tr>
    <td>Gemini-1.5-pro</td>
    <td>91.89</td>
    <td>66.67</td>
    <td>90.24</td>
    <td>84.26</td>
    <td>70.00</td>
    <td>50.00</td>
    <td>46.81</td>
    <td>55.94</td>
    <td>73.53</td>
    <td>57.50</td>
    <td>64.86</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>91.89</td>
    <td>53.33</td>
    <td>82.93</td>
    <td>77.77</td>
    <td>66.00</td>
    <td>47.83</td>
    <td>34.04</td>
    <td>49.65</td>
    <td>79.41</td>
    <td>65.00</td>
    <td>71.62</td>
  </tr>
  <tr>
    <td>Claude 3.5 Sonnet</td>
    <td>91.89</td>
    <td>46.67</td>
    <td>73.17</td>
    <td>72.22</td>
    <td>68.00</td>
    <td>52.17</td>
    <td>34.04</td>
    <td>51.75</td>
    <td>67.65</td>
    <td>62.50</td>
    <td>64.86</td>
  </tr>
  <tr>
    <td colspan="12"><i>Open-source models for multiple images as input</i></td>
  </tr>
  <tr>
    <td>Qwen2-VL</td>
    <td>91.89</td>
    <td>50.00</td>
    <td>82.93</td>
    <td>76.85</td>
    <td>78.00</td>
    <td>52.17</td>
    <td>40.43</td>
    <td>57.34</td>
    <td>50.00</td>
    <td>47.50</td>
    <td>48.65</td>
  </tr>
  <tr>
    <td>Idefics3-SB-Llama3</td>
    <td>83.78</td>
    <td>53.33</td>
    <td>85.37</td>
    <td>75.93</td>
    <td>68.00</td>
    <td>52.17</td>
    <td>36.17</td>
    <td>52.45</td>
    <td>58.82</td>
    <td>47.50</td>
    <td>52.70</td>
  </tr>
  <tr>
    <td>MiniCPM-V 2.6</td>
    <td>86.49</td>
    <td>50.00</td>
    <td>75.61</td>
    <td>72.22</td>
    <td>64.00</td>
    <td>50.00</td>
    <td>23.40</td>
    <td>46.15</td>
    <td>52.94</td>
    <td>47.50</td>
    <td>50.00</td>
  </tr>
  <tr>
    <td>mPLUG-Owl13</td>
    <td>78.38</td>
    <td>43.33</td>
    <td>70.73</td>
    <td>65.74</td>
    <td>54.00</td>
    <td>50.00</td>
    <td>25.53</td>
    <td>43.36</td>
    <td>58.82</td>
    <td>52.50</td>
    <td>55.41</td>
  </tr>
  <tr>
    <td>InternVL2-SB</td>
    <td>64.86</td>
    <td>40.00</td>
    <td>65.85</td>
    <td>58.33</td>
    <td>62.00</td>
    <td>54.35</td>
    <td>34.04</td>
    <td>50.35</td>
    <td>52.94</td>
    <td>40.00</td>
    <td>45.95</td>
  </tr>
  <tr>
    <td>LlVA-OneVision</td>
    <td>56.76</td>
    <td>40.00</td>
    <td>63.41</td>
    <td>54.63</td>
    <td>54.00</td>
    <td>52.17</td>
    <td>27.66</td>
    <td>44.76</td>
    <td>55.88</td>
    <td>40.00</td>
    <td>47.30</td>
  </tr>
  <tr>
    <td>Mantis</td>
    <td>86.49</td>
    <td>36.67</td>
    <td>63.41</td>
    <td>63.89</td>
    <td>66.00</td>
    <td>50.00</td>
    <td>19.15</td>
    <td>45.45</td>
    <td>58.82</td>
    <td>40.00</td>
    <td>48.65</td>
  </tr>
  <tr>
    <td>Idefics3-SB</td>
    <td>62.16</td>
    <td>60.67</td>
    <td>65.85</td>
    <td>64.63</td>
    <td>52.00</td>
    <td>41.30</td>
    <td>23.40</td>
    <td>43.16</td>
    <td>52.85</td>
    <td>42.50</td>
    <td>47.30</td>
  </tr>
  <tr>
    <td>Brote-IM-XL-3B</td>
    <td>54.00</td>
    <td>43.33</td>
    <td>52.78</td>
    <td>52.78</td>
    <td>52.00</td>
    <td>36.96</td>
    <td>43.36</td>
    <td>50.00</td>
    <td>35.00</td>
    <td>41.89</td>
    <td>33.78</td>
  </tr>
  <tr>
    <td>Idefics2-SB-base</td>
    <td>81.08</td>
    <td>46.67</td>
    <td>75.61</td>
    <td>69.44</td>
    <td>50.00</td>
    <td>43.48</td>
    <td>19.15</td>
    <td>37.76</td>
    <td>41.18</td>
    <td>27.50</td>
    <td>33.78</td>
  </tr>
  <tr>
    <td>Brote-IM-XXL-11B</td>
    <td>56.76</td>
    <td>30.00</td>
    <td>60.98</td>
    <td>50.93</td>
    <td>44.00</td>
    <td>34.78</td>
    <td>38.30</td>
    <td>39.16</td>
    <td>50.00</td>
    <td>35.00</td>
    <td>41.89</td>
  </tr>
  <tr>
    <td>MMICL-XXL-11B</td>
    <td>64.86</td>
    <td>46.67</td>
    <td>58.54</td>
    <td>57.41</td>
    <td>50.00</td>
    <td>30.43</td>
    <td>36.17</td>
    <td>39.16</td>
    <td>50.00</td>
    <td>32.50</td>
    <td>40.54</td>
  </tr>
  <tr>
    <td>MMICL-XL-3B</td>
    <td>43.24</td>
    <td>43.33</td>
    <td>68.29</td>
    <td>52.78</td>
    <td>46.00</td>
    <td>32.61</td>
    <td>38.30</td>
    <td>39.16</td>
    <td>52.94</td>
    <td>32.50</td>
    <td>41.89</td>
  </tr>
  <tr>
    <td colspan="12"><i>Open-source models for single image as input</i></td>
  </tr>
  <tr>
    <td>MiniCPM-Llama3-V 2.5</td>
    <td>91.89</td>
    <td>36.67</td>
    <td>78.05</td>
    <td>71.30</td>
    <td>60.00</td>
    <td>45.65</td>
    <td>42.55</td>
    <td>49.65</td>
    <td>50.00</td>
    <td>55.00</td>
    <td>52.70</td>
  </tr>
  <tr>
    <td>GLM-4V-9B</td>
    <td>89.19</td>
    <td>50.00</td>
    <td>73.17</td>
    <td>72.22</td>
    <td>40.00</td>
    <td>34.78</td>
    <td>38.30</td>
    <td>37.76</td>
    <td>58.82</td>
    <td>50.00</td>
    <td>54.05</td>
  </tr>
  <tr>
    <td>InternVL-Vicuna-13B</td>
    <td>56.76</td>
    <td>36.67</td>
    <td>60.98</td>
    <td>52.78</td>
    <td>62.00</td>
    <td>50.00</td>
    <td>21.28</td>
    <td>44.76</td>
    <td>61.70</td>
    <td>60.00</td>
    <td>60.81</td>
  </tr>
  <tr>
    <td>InternVL-Vicuna-7B</td>
    <td>59.46</td>
    <td>43.33</td>
    <td>63.41</td>
    <td>56.48</td>
    <td>52.00</td>
    <td>45.65</td>
    <td>23.40</td>
    <td>43.56</td>
    <td>55.88</td>
    <td>50.00</td>
    <td>52.70</td>
  </tr>
  <tr>
    <td>LlVA-1.6 13B</td>
    <td>51.38</td>
    <td>43.33</td>
    <td>52.78</td>
    <td>52.78</td>
    <td>58.00</td>
    <td>43.16</td>
    <td>23.40</td>
    <td>43.56</td>
    <td>55.88</td>
    <td>40.00</td>
    <td>48.65</td>
  </tr>
  <tr>
    <td>InternVL-Vicuna-13B-448px</td>
    <td>48.65</td>
    <td>40.00</td>
    <td>60.98</td>
    <td>50.93</td>
    <td>58.00</td>
    <td>47.83</td>
    <td>29.79</td>
    <td>45.45</td>
    <td>50.00</td>
    <td>52.50</td>
    <td>51.35</td>
  </tr>
  <tr>
    <td>mPLUG-Owl2-7B</td>
    <td>56.76</td>
    <td>40.00</td>
    <td>56.10</td>
    <td>51.85</td>
    <td>58.00</td>
    <td>45.65</td>
    <td>25.53</td>
    <td>43.36</td>
    <td>52.94</td>
    <td>42.50</td>
    <td>47.30</td>
  </tr>
  <tr>
    <td>Mini-Gemini-7B-HD</td>
    <td>70.27</td>
    <td>50.00</td>
    <td>51.22</td>
    <td>57.41</td>
    <td>52.00</td>
    <td>47.83</td>
    <td>29.79</td>
    <td>43.36</td>
    <td>58.82</td>
    <td>47.50</td>
    <td>52.70</td>
  </tr>
  <tr>
    <td>SEAL</td>
    <td>56.76</td>
    <td>36.67</td>
    <td>51.22</td>
    <td>49.07</td>
    <td>54.00</td>
    <td>34.78</td>
    <td>23.40</td>
    <td>37.76</td>
    <td>29.41</td>
    <td>37.50</td>
    <td>33.78</td>
  </tr>
  <tr>
    <td>Mini-Gemini-7B</td>
    <td>64.86</td>
    <td>43.33</td>
    <td>51.22</td>
    <td>53.70</td>
    <td>36.00</td>
    <td>28.26</td>
    <td>23.40</td>
    <td>29.37</td>
    <td>26.47</td>
    <td>20.00</td>
    <td>22.97</td>
  </tr>
</table>

Table 17: Results on each sub-classes of Shift-H (the hard-level shifting evaluation), where initial views do not display clues for answering the questions. Models should decide whether to shift to the next view all by themselves.

<table>
  <tr>
    <th rowspan="2">Models</th>
    <th colspan="2">Zooming</th>
    <th colspan="2">Shift-R</th>
  </tr>
  <tr>
    <th>ACC_GT</th>
    <th>ACC_QA</th>
    <th>#view</th>
    <th>ACC_QA</th>
    <th>#view</th>
  </tr>
  <tr>
    <td colspan="6"><i>Multi-image Models</i></td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>73.54</td>
    <td>68.62</td>
    <td>69.03</td>
    <td>2.29</td>
    <td><b>67.08</b></td>
    <td>60.54</td>
    <td>3.26</td>
  </tr>
  <tr>
    <td>mPLUG-Owl13</td>
    <td>60.62</td>
    <td>60.92</td>
    <td>68.57</td>
    <td>2.66</td>
    <td>51.69</td>
    <td><b>74.62</b></td>
    <td>4.00</td>
  </tr>
  <tr>
    <td>Claude 3.5 Sonnet</td>
    <td>72.31</td>
    <td>71.69</td>
    <td>67.64</td>
    <td>2.19</td>
    <td>65.23</td>
    <td>45.52</td>
    <td>2.47</td>
  </tr>
  <tr>
    <td>Qwen2-VL</td>
    <td>65.85</td>
    <td>64.62</td>
    <td>64.61</td>
    <td>2.35</td>
    <td>61.23</td>
    <td><b>74.62</b></td>
    <td>4.00</td>
  </tr>
  <tr>
    <td>Gemini-1.5-pro</td>
    <td>72.00</td>
    <td>72.00</td>
    <td>62.63</td>
    <td>2.60</td>
    <td><b>67.08</b></td>
    <td>46.33</td>
    <td>2.46</td>
  </tr>
  <tr>
    <td>MiniCPM-V 2.6</td>
    <td>62.77</td>
    <td>62.77</td>
    <td>58.85</td>
    <td>3.20</td>
    <td>53.54</td>
    <td>51.88</td>
    <td>3.08</td>
  </tr>
  <tr>
    <td>LlVA-OneVision</td>
    <td>64.92</td>
    <td>62.53</td>
    <td>66.67</td>
    <td>2.35</td>
    <td>53.54</td>
    <td>71.44</td>
    <td>2.02</td>
  </tr>
  <tr>
    <td>Idefics3-SB-Llama3</td>
    <td>60.92</td>
    <td>58.15</td>
    <td>41.09</td>
    <td>1.52</td>
    <td>61.85</td>
    <td><b>74.62</b></td>
    <td>4.00</td>
  </tr>
  <tr>
    <td>InternVL2-SB</td>
    <td>73.23</td>
    <td>56.00</td>
    <td>41.09</td>
    <td>1.53</td>
    <td>54.77</td>
    <td>45.75</td>
    <td>2.61</td>
  </tr>
  <tr>
    <td colspan="6"><i>Single-image Models</i></td>
  </tr>
  <tr>
    <td>InternVL-Vicuna-13B</td>
    <td>68.00</td>
    <td>62.77</td>
    <td>83.47</td>
    <td>3.31</td>
    <td>53.85</td>
    <td>69.73</td>
    <td>3.75</td>
  </tr>
  <tr>
    <td>LlVA-1.6 13B</td>
    <td>67.69</td>
    <td>68.92</td>
    <td>68.57</td>
    <td>2.65</td>
    <td>51.69</td>
    <td><b>74.62</b></td>
    <td>4.00</td>
  </tr>
  <tr>
    <td>SEAL</td>
    <td>56.92</td>
    <td>54.77</td>
    <td>68.22</td>
    <td>2.74</td>
    <td>42.15</td>
    <td>71.48</td>
    <td>3.83</td>
  </tr>
  <tr>
    <td>MiniCPM-Llama3-V 2.5</td>
    <td>62.20</td>
    <td>61.25</td>
    <td>61.26</td>
    <td>2.46</td>
    <td>53.85</td>
    <td>63.56</td>
    <td>3.42</td>
  </tr>
  <tr>
    <td>mPLUG-Owl2-7B</td>
    <td>67.38</td>
    <td>55.38</td>
    <td>47.61</td>
    <td>1.97</td>
    <td>47.38</td>
    <td><b>74.62</b></td>
    <td>4.00</td>
  </tr>
  <tr>
    <td>GLM-4V-9B</td>
    <td><b>74.46</b></td>
    <td>56.92</td>
    <td>30.62</td>
    <td>1.08</td>
    <td>56.92</td>
    <td>50.17</td>
    <td>2.64</td>
  </tr>
</table>

Table 18: Results of view selection ("R_{select}") and the accuracy given groundtruth views ("ACC_GT", 2.64 views on average) that contain human-annotated visual clues. "ACC_QA": accuracy of question answering for zooming and shifting. "#view": average counts of selected views.

evaluation, some models keep shifting view until all four views are inquired. However, this does not necessarily support a better accuracy, as some views contain redundant information that might distract the model during reasoning. Additionally, we observe shifting evaluations tend to require more views to be used for answering questions than zooming evaluations, yet it often results in inferior overall performance compared to zooming. This is because some of the current advanced models struggle to either move their field of views for necessary visual details, or screen out distracting information. Therefore, we believe that more attention should be paid to evaluating and enhancing active perception abilities of MLLMs given constraint perceptual fields.

I Case Study

In this section, we demonstrate three cases for the three proposed pipelines in Figure 5, and provide additional examples of integrating human-annotated visual clues hints in Figure 6.

I.1 Analysis of Cases from Each Pipelines

These results are generated by GPT-4 models and Gemini-1.5 models. Case (a) stands for the zooming evaluation, where models successfully identify the view containing useful information and generate the correct result. Case (b) illustrates a failure in the Shift-R evaluation, where all the models continue shifting to new views until all views are used. Though including the correct views, the additional views severely distract the reasoning process, where three out of four employed models produce incorrect answers. To explore how human-like mixed evaluation affects the visual reasoning
Question: What place is it?
Options: A. A train Station.   B. An airport.   C. A gymnasium.   D. A natatorium.

Zooming Evaluation:
GPT-4o: B. An airport.   GPT-4V: B. An airport.
Gemini-1.5-pro: B. An airport.   Gemini-1.5-flash: B. An airport.

(a) A successful case of zooming evaluation

Question: How much does the most expensive watermelon in the picture cost per kilogram?
Options: A. 29.   B. 39.   C. 59.   D. 69.

Shifting Evaluation w/o Hint:
GPT-4o: D. 69.
GPT-4V: D. 69.
Gemini-1.5-pro: C. 59.
Gemini-1.5-flash: D. 69.

(b) A failure case of shifting-R evaluation

Question: How much does the most expensive watermelon in the picture cost per kilogram?
Options: A. 29.   B. 39.   C. 59.   D. 69.

Mixed Evaluation:
("zoom": "1", "3",
"reason": "Both 1 and 3 contain signs with price details.")
{"zoom_keep": "1",
"shift": "2", "reason":
"3 does not contain watermelons, 2 contains watermelons."}
C. 59

(c) A case of mixed evaluation that the model independently decide whether to zoom and/or shift to different perceptual fields

Figure 5: Cases for each evaluation pipelines. (a) a succeeded zooming case, (b) a failed shifting case, and (c) a mixed case that successfully corrects the wrong answer produced by (b). Model selected views for case (a) and (b) are placed to the right of example frames, and used views for case (c) are shown with in its frame as the selection of views changes during the evaluation process.

process, we further exam this failure case using GPT-4o. As shown in Figure 5 case (c), GPT-4o first zooms into the “upper left” and “upper right” views, then discards the “upper right” view and shifts to the “lower left” one, which finally leads to the correct answer. Notably, in the final preserved views, distracting information (the highest price tag on papaya, “69”) is screened out. This indicates that GPT-4o exhibits decent active perception abilities to move the field of view, locate details, and filter out distracting information.

I.2 Cases of Giving Human-annotated Clues
We present a case study of ActiView in Figure 6. The first question targets at the most expensive watermelon, and only two out of four price tags, the ‘39’ and ‘59’ ones, are standing for the prices of watermelons. A distracting information appears at the “69” price tag that corresponds to papayas instead of watermelons. Models easily mislead by the most expensive tag “69” during evaluation. However, when we provide the models with the view of the price tags and remind them to focus on these tags, both GPT-4o and GPT-4V models correctly answer the question, indicating that actively perceiving key information helps improve model performance. While Gemini-1.5-pro gives the correct answer both with and without hints, and Gemini-1.5-pro fails to benefit from the hints. The second question asks models to recognize the place of the picture. Although it may be difficult to distinguish at first glance, we can still identify this place as an airport from some details, such as a airline’s logo. Since there isn’t a need to extract much information from the image, and there is little distracting information, all the four models answered the question correctly both with and without hints.

The right side of Figure 6 shows a comparison between the attention areas selected autonomously by GPT-4o and the areas highlighted by the hints we provided. It can be observed that when facing some difficult problems, although the model selects all the regions, it is unable to actively retrieve all the necessary details, thus lacking some essential information for answering the question. When the questions are relatively simple, the model successfully identify important information and gives the correct answer. This indicates that the GPT-4o model possesses a limited level of active perception capability and it still has room for improvement. We have also observed similar conclusions for other models.

J Prompt Template

In this section, we will provide detailed templates used for evaluation pipelines depicted in Figure 4.
Figure 6: Two cases of ActiView benchmark when given human-annotated visual clues for shifting and zooming evaluation. Left: The questions and answers of models. Right: We show the location of the visual clues we provided in the original image, as well as the areas chosen by GPT-4o model. For the first case, GPT-4o chooses all the areas, and for the second case, it chooses all the areas except the one in the bottom right corner.

J.1 Templates for General Question Answering

The general VQA template that requires models to answer questions given images is as following:

An Example Prompt for General Question Answering

Carefully analysis this image <image>, and answer the question from the given options.
Question: <question> Options: <options>.
Answer:

We develop a different template for two of our evaluated models, SEAL and MGM series. These models are optimized especially on VQA tasks, and sometimes fail to strictly following long textual instructions. Therefore, we use a simple and straightforward template to prompt these models for answers as follows:

An Example Prompt for Question Answering(SEAL and MGM)

<question> <options>. Answer:<image>

J.2 Templates for Zooming Evaluation

Here are templates used in the two stages of zooming pipeline depicted in Figure 4 (a). Note that the term “description_of_splits” refers to the positions of the views that guide the model to shift and select views. “description_of_splits” varies depending on how the views are divided. Taking 4 sub-image for example, it is described as “1 is the upper-left part, 2 is the lower-left part, 3 is the upper-right part, and 4 is the lower-right part.” The model should then response with “1, 2, 3, and/or 4” to select the appropriate views. The prompts are as follows:

An Example Prompt for View Selection

This is the full image <image>, which is split in to <num_splits> equal parts, numbered from 1 to <num_splits>, where <description_of_splits>.
===
Response with the number of part (at least one part, at most <num_splits> parts), that must be used to answer the question. The question is: <question>
===
Do not directly answer the given question. Response with the selected number of parts, split by ' ' if there are multiple selections. Your Response:

An Example Prompt for Zooming Question Answering

Image 0 is the full image. <zoomed_images>
These are your selected part from the full image to be zoomed for details for answering the question. Please answer question according to the given images from the the given options. Question: <question> Options: <option>. Answer:

J.3 Templates for Shifting Evaluation

Here are templates used in the two stages of zooming pipeline depicted in Figure 4 (b).
An Example Prompt for Missing-view Examination

You will be presented with a partial image and a question concerning the full image. image 0 is <image0>, is the <image_view> part of the full image. Given image 0, please determine if you need more visual information to answer the question: <question>
===
Do not directly answer the question. If you can answer the question without more visual information, response with NO. Otherwise, response with other image parts you need to see given this <image_view> part, you can choose from these views: <view_options>.
Your Response:

An Example Prompt for Operation Determination (1)

You will be presented with a full image <image> and a corresponding question to answer. The image is split in to <num_splits> equal parts, numbered from 1 to <num_splits>, where <description_of_splits>.
You can check for detailed visual information via zooming operation that zoom in to your selected part or parts with from the above numbers. Response with the the numbers of parts you wish to zoom in, or response with “none” if you don’t need to can check for details.
The question is: <question>
You should not directly answer the question. You should generate the a json dict containing 2 fields:
- “part”: type str, the selected numbers of index of parts, split by “,”, or ‘none’ if no zooming required;
- “reason”: type str, why you choose these parts.
Your response:

An Example Prompt for Shifting Question Answering

These are parts of an image. <all_required_views>. Carefully analysis these images and pay attention to their original position. Answer the question from the given options. Question: <question>. Options: <option>. Answer:

J.4 Templates for Mixed Evaluation

Here are templates used for the mixed pipeline depicted in Figure 4 (c). We design two templates for regarding the type of current view. We apply template “Operation Determination”(1) from the followings for the full images, and apply template “Operation Determination”(2) from the followings for zoomed views. Templates are as follows:

An Example Prompt for Operation Determination (2)

You will be presented with a partial image and a question concerning the full image. image 0 is <image0>, is the <image_view> part of the full image. Given image 0, please determine if you need more visual information to answer the question: <question>
===
Your are given a full image <image> and a corresponding question to answer. The image is split in to <num_splits> equal parts, numbered from 1 to <num_splits>, where <description_of_splits>. Your have chosen to zoom in to these parts, <zoomed_images>, for detailed checking if they can help to ansewr the question.
Question: <question> Options: <option>.
Now, there are two operations: “keep” and “shift”.
- “keep”: choose none or more parts from the zoomed ones to answer the question;
- “shift”: you can shift to the rest parts to answer questions or answer question with none sub-parts.
You should not directly answer the question. You should return you answer in a json dict containing two fields:
- “zoom_keep”: type str, the index numbers of required parts split by “,”, or “none” if the zoomed parts are useless;
- “shift”: type str, the index numbers of the rest parts, that are useful to the question split by “,”, or “none” if you don’t wish to shift.
Your response:
An Example Prompt of Quesiton Ansewring for Mixed Pipeline

Image 0 is the full image. <image_views> <image_view_desc> These are your selected part of image that must be used to answer the question. Please answer question according to the given images from the the given options. Question: <question> Options: <option>. Answer:

K  Attempts of Automatic Data Generation

In this last section, we discuss our experiments of automatic data generation, and analyse why powerful models like GPT-4V fail to accomplish this task. We will discuss the process and demonstrate typical failure cases in the following sections.

K.1  Automatic Data Generation Process

In the process of automatic data generation, we used the GPT-4V model for the following experiments:

• Step 1: We applied heuristic prompts on public datasets to encourage GPT to generate creative annotations across all types.

• Step 2: We selected the types that showed the best performance in automatic annotation and conducted batch annotation specifically for these types.

• Step 3: We manually filtered a subset of data that could be used.

In Step 1, we not only employed heuristic prompts to encourage GPT to generate diverse annotations but also specified the annotation types and their precise meanings (provided as candidates, encouraging the model to select from them). We restricted the annotation fields and types, and provided several manually curated examples as few-shot instances. Considering that some images in public datasets may not be suitable for our task, we allowed GPT to return “None” for images deemed unsuitable for annotation. The filtered annotation data were then re-evaluated using a scoring prompt, where we provided our annotation types and requirements, instructing GPT to rank the annotated data to assess its suitability.

In Step 2, we found that GPT performed best in annotating data of the counting type (based on a combination of manual inspection of the annotation results and GPT’s automatic scoring). Therefore, we decided to use GPT for automatic annotation of counting-type data. Considering that some public datasets (such as VCR) contain images with more than one type of bounding box, we processed different bounding box types in batches for each image to ensure that only one type of object was counted at a time.

Detailed prompt templates are attached in the third sub-section of this section.

K.2  Cases of Unsuccessful Generations of GPT-4V

We provide two typical cases demonstrating why GPT-4V fail to generate usable instances. The corresponding image is Figure 7. For the case regarding the left image, it presents a typical encountered issue case of hallucination and speculation without a factual basis. Given this image, GPT-4V produces the following annotations prompted by Step 1:

{
    "question": "Which of the following best describes the setting based on the appearance and arrangement of the glass items on the table?",
    "options": ["A casual family dinner", "A quick lunch at a fast food restaurant", "An official or formal meeting", "An outdoor picnic"],
    "answer": 2,
    "groundtruth": "The setting seems to be an official or formal meeting given the presence of multiple large, elegant glasses on the table, which suggest formal drinkware typically used in such settings."
}

The question and annotated answer posed by GPT-4V makes certain assumptions about the image that this scenario shows “An official or formal meeting”. The question is not answerable concerning only this image, where it could refer to either a meeting or a dinner. Moreover, the other options except for annotated answer does not match the image in any circumstances, and can be easily eliminated without any further observation of the image. The answers does not strictly follow the given ground truth (i.e., the answer to the question cannot be rigorously inferred from the visual clues in the image), where the glasses do not support the reasoning. For the case of the right image, it presents a typical failure case from Step 2. Regarding this image, GPT-4V generates an ambiguous question “How many umbrellas can be seen in the image?”, where there are some small visible objects could potentially be umbrellas as well.
Figure 7: Left: example of the automatic annotation results from Step 1. Right: example of the automatic annotation results from Step 2, where the question annotated by GPT is “How many umbrellas can be seen in the image?”

K.3 Prompt Templates Used for Automatic Generation

Here, we give the prompt used for automatic annotation in Step 1 and Step 2.

Heuristic prompt used for automatic annotation in Step 1

The clues for marking information in several bboxes in this picture are: {clues}
Based on several bboxes and corresponding clues, please design a question that requires the model to synthesize the information in these bboxes (at least two, and can only be answered based on the information in the bboxes and the clues corresponding to the annotated information). You only need to ask the question, and there is no need to repeat the clue again.
Note that the existence of bbox (including its ID information) cannot be mentioned in the question. Questions and reasoning should be based on objective facts as much as possible instead of subjective guessing. But at the same time, you should also avoid grounding questions and questions that can be answered without pictures (including questions like what someone in the picture is doing)
Next, mark me the corresponding information in the following format:
1. “question” (str)
2. “options” (list)
3. “abilities” (list): choose from “zoom in”, “zoom out”, “shifting” (it is mentioned in the analysis and is not mentioned at the beginning)
4. “answer” (int, index of option)
5. “order”: the order in which the pictures cut out of the bbox and the entire picture are displayed (the list is given in the order of reasoning, all of which are ints, representing the id corresponding to the bbox on the picture, if it is a complete picture, it is 0)

Heuristic prompt used for automatic annotation in Step 1 (Continue)

6. “groundtruth”: Give the reasons and complete reasoning process for answering the question
7. “number_of_operations”: For example, first zoom in and then move the angle of view, it is two operations
You must give me the answer in the following json-string format (not code block) and dont say anything else:
({
“question”: question(str),
“options”: options(list),
“abilities”: abilities(list),
“answer”: answer_index(int),
“order”: order(list),
“groundtruth”: groundtruth(str),
“number_of_operations”: number of the operations(int)
Scoring prompt used for automatic annotation in Step 1

We want to design a question about the picture to test the active perception ability of the respondent. Here are the requirements:
- You will be provided with an image and information of bboxes in it. You should design a question that requires the respondent to synthesize the information in these bboxes.
- While designing the questions, you must follow these rules:
  - The question should be based on the information in the given bboxes.
  - The question requires the respondent to obtain information from the field of view of these bboxes as a basis, identify irrelevant information on the picture, and move the field of view of different bboxes to obtain more information before answering the question.
  - Differences between options should be distinct. And options must not be conflict to each other.
  - There should be one and only one correct answer among all options.
  - The evidence or clues for answering the question must be visible in the image. Also, you should realize the following conditions:
    - The answers must not require the respondent guess subjectively.
    - You cannot generate questions require simple object grounding, e.g., what is the object in a certain region, what is the color of an object, etc.
    - The existence of bbox and visual clues (including their ID information) cannot be mentioned in the question nor in the options.

Scoring prompt used for automatic annotation in Step 1 (Continue)

You should score the annotation through the rules given above. Here are the predefined levels for scoring, where level D is the worst and level A is the best:
- Level D: no reasonable questions can be generated for the given image by strictly following our rules.
- Level C: the question contains subjective guesses and judgments, rather than strictly following the rules(e.g. infer the location from the architectural style/image style rather than some grounding signs and texts etc.)
- Level B: the question can be answered via simple captioning of the pictures(like using ViT or OCR to caption the picture and ask the language model to answer the question with out the picture), or can be answered via pure common sense reasoning.
- Level A: the question is cleverly designed and is completely based on the information in the picture. It requires the respondents to visit different parts marked on the image for comprehensive reasoning, which fully complies with the above marking rules.

Remember, if any subjective guess seems to appear, or anything that requires inferring from knowledge outside the image, or anything that does not follow our rule strictly (including asking for some weirded questions etc.), do not hesitate to assign a low level.

Here's the annotation information of the given picture:
{annotation}

You must give me the answer in the following json-string format(not code block) and don't say anything else:
{{"score": string, choose from “A”, “B”,”C”, “D”,
“reason”: string, explain why you give this score
Prompt used for automatic annotation in Step 2

You are an annotator to design questions and options for given images. Here are the guidebook for you:
===
Overall task description: You will be presented with an image, please generate a question, corresponding options and answer to the question, and some other information that help the reasoning process as well.
===
Detailed requirements you **must** follow:
- You must design the problem in the following type:
  Counting with restricted information or extending reasoning based on counting. For example, there are lots of products in the image, but only a part of them are on sale, you can ask for the number of on sale products. Options are list of numbers.
Candidates:
- How many people are wearing black hat?
- How many products are on discount?
- Which color of umbrellas are the most numerous in the picture?
But remember, you *cannot* ask common sense questions like how many objects are there in the picture, which can be answered without reasoning.
- **Simple grounding questions are NOT allowed**, such as (but not restricted to): “what is xxx object?”, “What is the color/style of xxx?”, and etc.
- For answers:
  - By referring to the image, there must exists one and only one answer, without any ambiguities and subjective guesses.
  - The evidence for answering the question must be visible in the image.
  - Objective reasoning are not allowed.
  - DO NOT rely on information that does not exist in the image.
- For options:
  - The differences between generated options should be distinct.
  - There should be one and only one correct answer among all options.
  - Options must not be conflict to each other.
===

Prompt used for automatic annotation in Step 2 (Continue)

The requirements of the generated data format are as follows:
1. “question” (str, start with wh words or prep + wh words)
2. “options” (list)
3. “abilities” (list): choose from “zoom in”, “zoom out”, “shift”
4. “answer” (int, index of correction option, starting from 0)
5. “groundtruth”: Give the reasons and complete reasoning process for answering the question
6. “operations”: For example, first zoom in to a region and then moving to a different region, counted as two operations
===
Here are some bounding boxes and their type for you to refer to:
{boxes}
The items in these bounding boxes are all {type} The questions you ask must be about the information within the bounding boxes and strictly meet the requirements and question types given to you above.
===
If it is impossible to come up with required questions, you should response with “question”: (str)“None” in json-string format(not code block). Otherwise, you must generate response in the following json-string format(not code block) and dont say anything else:
{{“question”: question(str),
“options”: options(list),
“abilities”: abilities(list),
“answer”: answer_index(int),
“order”: order(list),
“groundtruth”: groundtruth(str),
“operations”: number of the operations(int),
}}
===
Please generate response for the given image that **strictly follow** the above requirements:

Reviewer paper summaries: This paper introduce a novel benchmark ActiView to evaluate active perception abilities in MLLMs. Extensive experiments are conducted over 27 models and observe a gap in performance between human and current MLLMs.

This paper addresses the gap in evaluating MLLMs on active perception tasks by proposing a novel benchmark named ActiView. Specifically, it considers active perception as a Visual Question Answering (VQA) task, where the perceptual field of MLLMs is restricted, requiring them to actively zoom or shift within a given image based on reasoning to answer questions. The experiments were conducted over 27 proprietary and open-source models. The results indicated that current MLLMs capabilities are significantly lower than human performance, but active perception abilities of models could be enhanced through multi-image interleaved structures.
Summary of strengths: 1. This paper introduce a novel benchmark focusing on evaluation of the active perception abilities in MLLMs
2. The benchmark contains well-defined sub-classes and categories covering comprehensive aspects of studying active perception abilities.
3. Extensive experiments on  27 models demonstrates a gap in performance between human and current MLLMs on this task.

The paper is well structured and easy to follow.

The paper introduces a new MLLM benchmark, ActiView, with eight sub-classes categorized by different types of visual information and visual features as a QA task focusing on active perception.

The experiments are detailed and demonstrate the performance of current proprietary and open-source MLLMs on ActiView benchmark.
Summary of weaknesses: 1. How realistic are the scenarios in the benchmark, the example in the Figure 1 illustrated a scenario where a single picture is super-resolution therefore requires the model to actively shifts or zooming. But for the scenarios in the benchmark as illustrated in Figure 2, how to demonstrate that shift and zoom are actually useful? 
2. Follow-up the first point, a baseline of simply incorporate the whole image is needed to illustrative the effectiveness and importance of active perception abilities. As current MLLMs are becoming good at high-resolution images, their performance should be tested.

For zooming pipeline evaluating, it is worth checking the performance for the view selection stage as the question answering stage depends on the result of selection stage.

It is better to see if there are any improvements with one-shot or few-shot learning for MLLMs on the ActiView benchmark. Not all models evaluated in the paper need to be tested, but selecting a few would be beneficial.
Comments, suggestions and typos: 1. At line 221, "Is this a sunset or sunrise", how can the model determine the answer if it's just one static image. I assume it would require a continuous views of several images to determine the answer.

lower then -> lower than (line 451).

It will be helpful if the proposed benchmark is released.
Overall assessment scores: 3

3.5
Metareview score: 3
Metareview: This paper introduces ActiView, a new benchmark designed to evaluate the active perception capabilities of Multimodal Large Language Models (MLLMs). Active perception, as defined in this work, involves an MLLM being able to interact with an image by zooming or shifting its perceptual field to gather necessary information to answer a question. This benchmark specifically focuses on a specialized form of Visual Question Answering (VQA) where the model's initial view of an image is restricted, requiring it to actively explore the image to find the answer.
Summary of reasons to publish: - The paper introduces a novel benchmark, ActiView, which addresses a gap in the evaluation of MLLMs by focusing on active perception, a capability not thoroughly assessed by existing benchmarks.

- ActiView is unique in its design, requiring models to actively interact with images through zooming and shifting, simulating a more human-like approach to visual understanding.

- The benchmark includes a diverse set of questions across multiple subcategories, allowing for detailed analysis of model performance and identification of specific areas needing improvement.