VOILA: Evaluation of MLLMs For Perceptual Understanding and Analogical Reasoning

Authors: Nilay Yilmaz, Maitreya Patel, Yiran Lawrence Luo, Tejas Gokhale, Chitta Baral, Suren Jayasuriya, Yezhou Yang
Primary area: datasets and benchmarks
Keywords: abstract reasoning, relational reasoning, perceptual understanding, MLLMs, visual analogy

Abstract:
Multimodal Large Language Models (MLLMs) have become a powerful tool for integrating visual and textual information. Despite their exceptional performance on visual understanding benchmarks, measuring their ability to reason abstractly across multiple images remains a significant challenge. To address this, we introduce VOILA, a large-scale, open-ended, dynamic benchmark designed to evaluate MLLMs' perceptual understanding and abstract relational reasoning. VOILA employs an analogical mapping approach in the visual domain, requiring models to generate an image that completes an analogy between two given image pairs, reference and application, without relying on predefined choices. Our experiments demonstrate that the analogical reasoning tasks in VOILA present a challenge to MLLMs. Through multi-step analysis, we reveal that current MLLMs struggle to comprehend inter-image relationships and exhibit limited capabilities in high-level relational reasoning. Notably, we observe that performance improves when following a multi-step strategy of least-to-most prompting. Comprehensive evaluations on open-source models and GPT-4o show that on text-based answers, the best accuracy for challenging scenarios is 13% (LLaMa 3.2) and even for simpler tasks is only 29% (GPT-4o), while human performance is significantly higher at 70% across both difficulty levels.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 6, 5

Reviews and discussion:
<thread>
creation date:
1730105847649

thread_type:
official_review

<thread_object>
creator: reviewer_pi5r

<content>
summary:
The paper introduces VOILA, a benchmark designed to assess multimodal large language models (MLLMs) on tasks requiring visual perception and abstract relational reasoning. VOILA primarily focuses on visual analogical reasoning, where models must generate an image that completes an analogy given two pairs of images. The benchmark includes two subsets, VOILA-WD (with distractions) and VOILA-ND (without distractions), testing MLLMs on varying levels of complexity. Contributions:
1. A dataset creation pipeline allowing large-scale generation of visual analogy questions, and a large-scale, open-ended benchmark to evaluate MLLMs’ high-level visual reasoning capabilities. 
2. Comprehensive evaluation of state-of-the-art MLLMs, highlighting performance gaps in relational reasoning compared to human baselines.
3. Ablation studies reveal model limitations in handling complex visual relationships, suggesting areas for improvement in future MLLM development.

strengths:
1. The proposed problem is a novel multimodal reasoning problem, which is well-defined and advances abstract relational reasoning in MLLMs​.
2. The benchmark design is comprehensive, encompassing various rule configurations and levels of task difficulty.
3. The results reveal the weaknesses of current multimodal LLMs.

weaknesses:
1. The paper does not address why the proposed problem is an essential and meaningful problem to solve. In what applications/scenarios is the capability essential?
2. The comparison between the model and human performance may not be fair: (1) Humans are given two examples. (2) It seems the task for humans is to choose properties from available options, while MLLMs need to predict those properties.
3. Table 3 takes up a lot of space and has a lot of numbers, but it is barely mentioned or explained in the text.

questions:
1. The prompt4 for generating images seems unnecessary for evaluating the "reasoning" capabilities of MLLMs. Also, the reviewer is curious how the authors use GPT4o to generate images since it is not clear whether they call another image generation model to generate images in the web interface. And it seems gpt4o API does not support image output. 
2. Why Table 4 and Table 5 only use the selected models for the ablation study?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730621681663

thread_type:
official_review

<thread_object>
creator: reviewer_qyey

<content>
summary:
This paper introduces a new benchmark for evaluating multi-modal large language models’ perceptual understanding and abstract relational reasoning through analogy completion. It contributes a dataset creation pipeline, a large evaluation dataset consisting of 10K+ questions, and extensive evaluations of both open-source and proprietary models on this benchmark as well as detailed analyses of their performance on different subtasks.

strengths:
- The idea of using visual analogy completion for evaluating multi-modal large language models is well motivated and somewhat unique 
- The visual analogy completion problem is well defined and formulated, with clear definitions of the three properties and four rules.  
- The dataset has been manually cleaned to ensure that the generated images match the texts.

weaknesses:
- The experiments can be more complete. Currently, most evaluations focus on the first three steps of the analogy completion task with only a few data points on the image generation stage. 
- The paper presentation can be improved. For example, the writing can be improved to highlight the most insightful and interesting takeaways. 
- While the visual analogy task is interesting, the reviewer is uncertain about the practical value of this benchmark especially given the abundance of multi-modal benchmarks – for example, why should the community use this benchmark to evaluate models’ relation understanding instead of existing VQA benchmarks that can also test for relation understanding?

questions:
- Did the authors include examples of each subtask in the prompts? 
- The authors mention they evaluated Emu-2 on the image generation subtask in 5.1, but Table 3 is missing its numbers? 
- In Figure 6, it’s interesting that LlaMa 3.2 is the only model with higher performance on Voila-WD than Voila-ND, while all other models exhibit large drops from Voila-ND to Voila-WD regardless of their absolute accuracies. Can the authors provide insights into why this occurs for LlaMa 3.2?
- Nit: table 4, it’s VOILA-ND in both columns – should one of them be VOILA-WD instead?

Suggestions: 
- While the authors motivate the visual analogy task by saying that it tests for higher-order reasoning, they break down the task into four subtasks, each of which is an easier task (despite being still difficult to the model). While the reviewer agrees with the authors that this task decomposition enables fine-grained analyses of models’ weaknesses, it’d also be interesting to see an evaluation of the end-to-end analogy completion task. 
- Table 2 can highlight the performance drops from the previous to the next stage
- The authors mentioned that models struggle with the visual analogy task when images are combined in a collage format due to resolution constraints. It’d be interesting to see if this finding holds with high resolution image collages for multi-modal models that support the AnyRes strategy such as Llava-OneVision.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730682661510

thread_type:
official_review

<thread_object>
creator: reviewer_jyff

<content>
summary:
This paper proposed a new benchmark named VOILA, aiming to evaluate the performance of visual language models in understanding abstract relation understanding by designing visual analogy questions.
Experiments show that many models can describe the images and idenitify the relationship between images, but cannot apply the relationship very reliably.

strengths:
1. The visual analogy perspective of evaluating vision language models is interesting.
2. The paper gives a very detailed the description of how the benchmark is collected.

weaknesses:
1. The analogies considered in this paper seems to be restricted to only numbers, subjects, and actions.

questions:
1. One line of related works I think this paper need to mention is on the benchmarking of multiple image understanding, such as works of Muirbench and MIRB, especially MIRB, which has already included visual analogy as one sub task for testing current state-of-the-art visual language models.
2. The data generation pipeline uses diffusion models to geerate images from given text prompts. I'm wondering how reliable is the diffusion model? how many images are discarded. Perhaps include a performance of human on the final dataset can answer this (I noticed that in table 3 there are some human performance, but why is human performance only tested on the 'Applying relationship' subset?)
3. In my understanding, this paper only considers analogy on properties like number, subject, and action. Is these enough to cover all possible visual combinations that can form analogies?



Muirbench: https://arxiv.org/abs/2406.09411
MIRB: https://arxiv.org/abs/2406.12742
</content>
</thread_object>

</thread>
<thread>
creation date:
1730696691294

thread_type:
official_review

<thread_object>
creator: reviewer_ykdc

<content>
summary:
The authors propose a new benchmark called VOILA which evaluates the perceptual understanding and analogical reasoning capabilities of multimodal language models, particularly its reasoning capabilities across multiple images. VOILA requires a MLLM to understand relations between images and generate a new image that follows the pattern. This task is open-ended and results are evaluated using a strong LLM. Results indicate that even the best MLLM performs much worse than an average human (30% vs 70%)

strengths:
This is generally a well organized and well written paper.
The proposed benchmark is interesting and highlights a critical shortcoming in existing MLLMs
The benchmark is well curated, there is also manual filtering involved to ensure quality

weaknesses:
This task is not exactly novel, [1] propose a new task that evaluates visual cognition of MLLMs. The new novelty comes from the requirement that MLLM generate the output image. 
The evaluation is not grounded in existing psychological assessments for instance the reasoning used in [1] is widely used in neurodevelopmental and neuropsychological research.
GPT4o seems to be poor at identifying relationships (from image) in VOILA, does this affect the evaluation which in turn also uses GPT4o. 

References
[1] https://arxiv.org/abs/2406.10424

questions:
Can the authors conduct a small experiment to validate that their evaluation strategy is accurate?
The prompt to identify relation asks the model to identify differences in subject, types and actions all in a single prompt. What if this problem was broken down to three sub-prompts? It seems intuitive that performance should improve
</content>
</thread_object>

</thread>
<thread>
creation date:
1730795050735

thread_type:
official_review

<thread_object>
creator: reviewer_pczi

<content>
summary:
The paper introduces VOILA, a benchmark specifically designed to evaluate Multimodal Large Language Models (MLLMs) in perceptual understanding and abstract relational reasoning across images. By using analogical mapping, VOILA requires MLLMs to generate an image that completes an analogy between two image pairs, testing the models' relational reasoning without predefined answer choices. The benchmark includes challenging tasks, with models struggling to match human performance, especially in higher-level reasoning steps. Performance improves with least-to-most prompting strategies, but there remains a substantial gap between the best-performing model and human results.

strengths:
• VOILA stands out by focusing on abstract visual analogies, making it a valuable addition to existing MLLM evaluation benchmarks, particularly in assessing perceptual and relational reasoning.

• The multi-step reasoning approach and comprehensive ablation studies offer a detailed examination of the current limitations in MLLMs.

weaknesses:
• The paper emphasizes that this is a dynamic benchmark. For a static benchmark, once a configuration is provided, it can be fully generated and then annotated for correctness by human evaluators. However, as a dynamic benchmark, once it is generated, how to ensure the correctness of the benchmark and how to make it scalable, which remain challenging and difficult to guarantee. This issue is unsolved in this paper.

• VOILA’s reliance on GPT-4o for model evaluation makes the evaluation process both resource-intensive and costly for practitioners.

questions:
Please refer to Weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1732390306451

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global Response to Reviewers

comment:
We greatly appreciate the valuable and constructive feedback provided by the reviewers. We are truly gratified to see their unanimous positive evaluations, which acknowledge the strengths of our work across multiple dimensions.

- The reviewers consistently acknowledge that **VOILA reveals the shortcomings of state-of-the-art MLLMs** (pCZi, YkdC, pi5r) supported by **detailed and comprehensive evaluations**. (pCZi, qyEy, pi5r).
- Reviewers pCZi and pi5r recognize the significance of VOILA in **contributing valuable advancements to abstract relational reasoning** within MLLMs.
- Reviewers qyEy and pi5r acknowledge VOILA’s **contribution to the dataset creation pipeline**, noting its **well-defined and comprehensive structure**, which includes clear definitions of various properties, rule configurations, and difficulty levels.
- Reviewer Jyff emphasizes the critical stage of the **well-curated** VOILA task —**applying relationships**—which highlights the distinction between human reasoning ability and MLLMs.
- Additionally, we are pleased that reviewer qyEy described our study as **unique, well-motivated, and well-formulated**, and Reviewer pi5r acknowledged VOILA task as a **novel multimodal reasoning problem**.

We have addressed each reviewer’s comments in detail individually. Below, we summarize our responses to the key questions and offer additional clarifications.

## Clarification of the Significance of Analogical Reasoning and VOILA
Analogical reasoning is a critical cognitive ability for effective problem-solving, decision-making, and learning strategies. A key aspect of analogical reasoning is the transfer of knowledge from previous relational learnings to novel concepts, which distinguishes it from other reasoning tasks. By identifying similarities between past experiences and new situations and applying relational patterns, humans enhance their learning process, solve problems more effectively, and anticipate the outcomes of their decisions. 

In this work, we created VOILA, an analogical reasoning task, to investigate how well current MLLMs can transfer learned patterns to new domains, essential for generalization. By evaluating the models' performance on VOILA, supported by comprehensive analysis, we aim to reveal their limitations and demonstrate the gap in analogical reasoning abilities between humans and MLLMs. 

## Summary of Ablation Studies
- **Ablation on the Accuracy of GPT-4o in the Evaluation Process** : As requested by Reviewer YkdC, we conducted a thorough comparison of human performance and GPT-4o performance across evaluation steps and properties. Our findings indicate that GPT-4 achieves high accuracy in comparing and scoring ground truth and model answers, with a maximum error rate of only 10%. 

- **Ablation on Relation Extraction Step Using Three Sub-prompts** : In response to Reviewer YkdC's request, we conducted an additional ablation study by using three sub-prompts instead of one to reveal the relationships between properties in images. We observed that GPT-4o's ability to identify relationships remains unaffected by the number of properties requested in the prompt. 

- **Ablation on the AnyRes Method for Image Collages** : As requested by Reviewer qyEy, we performed additional ablations on Llava-One-Vision model using AnyRes approach, processing both image collages and three sequential images as input. Our findings indicate that the AnyRes method enhances single-image resolution and narrows the performance gap between processing image collages and multiple separate images.

- **Ablation on Performance Gap Between Human and MLLMs**: In response to Reviewer pi5r, we conducted an additional ablation study with GPT-4o on VOILA-WD under the same human evaluation conditions, incorporating the direct answering approach. The model was provided with two example questions and property option lists. We observed that the new setting did not increase the performance of MLLLMs, as compared to our previous study using the least-to-most (L2M) approach with step-by-step instructions.

- **Ablation on CoT Effect at "Applying Relationship" Step**: In response to Reviewer pi5r, we conducted an ablation study with GPT-4o on VOILA-WD to test the impact of the CoT approach at the third step. We provided two textual examples and rationales for the model. Our findings indicate that the CoT approach performs similarly to the L2M method which implements a multi-step reasoning process with detailed instructions.

Once again, we thank the reviewers for their time and detailed feedback. We are glad that we resolved most of the queries from the reviewers. We believe that this fruitful discussion has made our paper stronger.
</content>
</thread_object>

</thread>


Metareview:
metareview:
### Summary of Scientific Claims and Findings
This paper introduces **VOILA**, a benchmark for evaluating Multimodal Large Language Models (MLLMs) on their ability to perform abstract relational reasoning and perceptual understanding through visual analogies. VOILA presents a novel problem structure that integrates dynamic, large-scale datasets with varying levels of complexity (VOILA-ND and VOILA-WD). The key task requires MLLMs to generate images or textual outputs that complete an analogy based on provided image pairs, testing models' ability to identify, understand, and apply relationships between properties such as numbers, subjects, and actions. Experimental results demonstrate that while current MLLMs can perform simple tasks such as property identification, they struggle significantly with higher-order relational reasoning, especially when distraction rules are introduced.

### Strengths of the Paper
1. **Novel Contribution**: VOILA introduces a unique problem structure, focusing on abstract analogical reasoning, which is distinct from existing benchmarks like MaRs-VQA and MIRB.
2. **Comprehensive Benchmark**: The dynamic dataset generation process, covering over 6.4M analogy questions, makes VOILA scalable and adaptable to different configurations.
3. **Thorough Evaluation**: Extensive experiments reveal the limitations of current MLLMs in high-level reasoning tasks, with detailed ablation studies exploring various task aspects.
4. **Impactful Insights**: The paper highlights significant gaps in MLLMs’ reasoning capabilities, motivating further research in multimodal reasoning.

### Weaknesses and Missing Elements
1. **Practical Significance**: While the benchmark is well-defined, its practical applications and relevance to real-world scenarios could be better articulated.
2. **Evaluation Noise**: The use of external models like DALL-E 3 for image generation introduces potential noise in evaluating reasoning capabilities.
3. **Presentation Gaps**: Dense tables and some unclear explanations limit accessibility. Insights from results could be highlighted more prominently.
4. **Limited Novelty in Methods**: Although the problem is novel, the methods (e.g., Least-to-Most prompting) align with prior work, making the approach feel incremental.
5. **Image Generation Challenges**: There is limited error analysis explaining why models fail at the image generation stage and how this step contributes uniquely to reasoning evaluation.

### Recommendation and Justification
The paper makes a significant contribution by introducing VOILA, a novel benchmark for evaluating multimodal large language models (MLLMs) on abstract analogical reasoning and perceptual understanding, a largely unexplored area in current research. VOILA offers a scalable, dynamic dataset and reveals critical gaps in MLLMs' reasoning capabilities compared to humans, supported by robust experiments and detailed ablation studies. The authors effectively addressed most reviewer concerns, including the benchmark’s novelty, relevance, and evaluation methodologies, with additional experiments and clarifications. While some issues remain, such as evaluation noise in the image generation step and limited practical applications discussed, these do not diminish the paper’s overall merit. Given its potential to drive future advancements in multimodal reasoning and its alignment with reviewer consensus post-rebuttal, this work is a valuable addition to the field.