ActiView: Evaluating Active Perception Ability for Multimodal Large Language Models
Authors: Ziyue Wang, Chi Chen, Fuwen Luo, Yurui Dong, Yuanchi Zhang, Yuzhuang Xu, Xiaolong Wang, Peng Li, Yang Liu


Reviewer paper summaries: This paper proposes a benchmark specifically designed to evaluate the active perception capabilities of Multilevel Language Models (MLLMs), named ActiView. The evaluation perspectives are divided into three major categories: environment-centric (Type I), object-centric (Type II), and event-centric (Type III), and encompass eight sub-categories in total. 

For a question, two perception settings—zooming and shifting—are introduced to evaluate the models. During the evaluation, MLLMs are guided to actively zoom and shift views in the format of Visual Question Answering (VQA). Moreover, for each of the two settings, a distinct evaluation pipeline has been designed for the models being evaluated.

Using this benchmark, the authors have assessed mainstream existing open-source and proprietary MLLMs, revealing deficiencies in their active perception capabilities.

Contribution:
1) A benchmark assessing active perception abilities of MLLMs across diverse and dynamic environments, which was largely overlooked in previous work.
2) A meticulously manual annotated dataset that covers multiple evaluation perspectives of MLLMs’ active perception abilities .
3) The evaluated results reveal that there exists a significant gap in the active perception capability of MLLMs, which deserves further exploration and improvement.
Summary of strengths: 1)This is a novel idea that specifically evaluates the active perception capabilities of MLLMs, which is highly relevant to the real-world applications of MLLMs and has not been emphasized in previous benchmarks.
2) Authors introduce two settings -- zooming and shifting -- to simulate the humans’ action when perceiving the real-world visual information, which is plausible and feasible.
3) Authors meticulously annotate their dataset and divide eight sub-categories in total to conduct an extensive and fine-grained evaluation for MLLMs.
Summary of weaknesses: 1)The model inference process displayed by the author in Figure 1 is quite appealing. However, in the actual evaluations, it seems that the method shown in Figure 1 where combining zooming and shifting within a single question is not adopted. Instead, each question is evaluated separately using either zooming or shifting, making the evaluation approach somewhat one-sided.

2)In the evaluation, there is a subcategory called Number Counting, which is a type of problem that requires MLLMs to carefully observe each area to arrive at the correct answer. This is because the object to be counted may exist in every corner of the image. However, the authors state in the paper, “For models that can only accept single image inputs, in zooming question answering, we do not provide them with the complete view of the image. Instead, they are given the image of the view selected in the previous round of dialogue.”(Appendix B.1) This means that one-image input models can only see information from one area, which seems likely to lead to evaluation failures in Number Counting questions. 

3)In Appendix B.1, authors state that “In shifting question answering, the views selected by the model in the first round are stitched together from left to right to form an image as the input for the model.” Combined that bottom contents of Figure 4, I consider the final input image for one-image input models maybe a flat image if there are multiple views selected in the first round, which could lead to information loss for one-image input models that use CLIP ViT to resize this flat image to a fixed resolution.

4)In the left-bottom block of Appendix F, authors show the prompt of MLLMs that “This is the full image <image>, ..., where <description_of_splits>.” I am confused about what <description_of_splits> here refers to. If this means the description of the splited image content, it seems that an information leak because it prematurely informs the model about the content of the image.
Comments, suggestions and typos: Overall, this paper suggests a good direction, but the implementation details in the evaluation of the models could be improved.
Overall assessment scores: 3.5