VOILA: Evaluation of MLLMs For Perceptual Understanding and Analogical Reasoning
Authors: Nilay Yilmaz
Maitreya Patel
Yiran Lawrence Luo
Tejas Gokhale
Chitta Baral
Suren Jayasuriya
Yezhou Yang
Venue: ICLR 2025 Poster (2025)
Keywords: abstract reasoning
relational reasoning
perceptual understanding
MLLMs
visual analogy
Primary area: datasets and benchmarks
TL;DR: To assess MLLMs' perceptual understanding and abstract relational reasoning capacity, we present VOILA which applies an analogical mapping strategy to the visual domain.

Abstract: Multimodal Large Language Models (MLLMs) have become a powerful tool for integrating visual and textual information. Despite their exceptional performance on visual understanding benchmarks, measuring their ability to reason abstractly across multiple images remains a significant challenge. To address this, we introduce VOILA, a large-scale, open-ended, dynamic benchmark designed to evaluate MLLMs' perceptual understanding and abstract relational reasoning. VOILA employs an analogical mapping approach in the visual domain, requiring models to generate an image that completes an analogy between two given image pairs, reference and application, without relying on predefined choices. Our experiments demonstrate that the analogical reasoning tasks in VOILA present a challenge to MLLMs. Through multi-step analysis, we reveal that current MLLMs struggle to comprehend inter-image relationships and exhibit limited capabilities in high-level relational reasoning. Notably, we observe that performance improves when following a multi-step strategy of least-to-most prompting. Comprehensive evaluations on open-source models and GPT-4o show that on text-based answers, the best accuracy for challenging scenarios is 13% (LLaMa 3.2) and even for simpler tasks is only 29% (GPT-4o), while human performance is significantly higher at 70% across both difficulty levels.

Published as a conference paper at ICLR 2025
VOILA: EVALUATION OF MLLMs FOR PERCEPTUAL UNDERSTANDING AND ANALOGICAL REASONING
Nilay O Maitreya Patel
Yiran Lawrence Luo
Tejas Gokhale
Chitta Baral "
Suren Jayasuriya Yezhou Yang
? Arizona State University University of Maryland, Baltimore County
ABSTRACT
Multimodal Large Language Models (MLLMs) have become a powerful tool for integrating visual and textual information. Despite their exceptional performance visual understanding benchmarks, measuring their ability to reason abstractly across multiple images remains a significant challenge. To address this, we intro- duce VOIIA, large-scale, open-ended, dynamic benchmark designed to evaluate MLLMs' perceptual understanding and abstract relational reasoning. VOILA em- ploys analogical mapping approach in the visual domain, requiring models generate an image that completes an analogy between two given image pairs, ref- erence and application, without relying on predefined choices. Our experiments demonstrate that the analogical reasoning tasks in VOILA present a challenge to MILMs. Through multi-step analysis, we reveal that current MLLMs struggle to comprehend inter-image relationships and exhibit limited capabilities in high- level relational reasoning. Notably, we observe that performance improves when following a multi-step strategy of least-to-most prompting. Comprehensive eval- uations on open-source models and GPT-4o show that on text-based answers, the best accuracy for challenging scenarios is 13% (LLaMa 3.2) and even for simpler tasks only 29% (GPT-4o), while human performance is significantly higher at 70% across both difficulty levels.
INTRODUCTION
Multimodal Large Language Models (MLLMs) have made remarkable strides in advancing human- level language processing and visual perception in tasks such as image captioning (Vinyals et al., 2015), visual question answering (Agrawal et al., 2016), object detection, and scene understanding (Bochkovskiy et al., 2020). While these advancements are promising, perceptual reasoning tasks such as relational and analogical reasoning. where models must infer and understand visual infor- mation, remain significant challenge. Achieving human-level cognitive intelligence in these tasks demands greater attention and development.
According to Bloom's taxonomy of educational objectives, creation, rather than evaluation, requires the highest cognitive skills in the learning process (Bloom et al., 1956). However, many current mul- timodal reasoning tasks (Wang et al., 2024e; Plummer et al., 2016) rely multiple-choice formats, where models select a solution from a predefined set. Although this approach provides insight into the learning and understanding capabilities of model, we argue that it fails reveal the model's ability to engage in high-level cognitive tasks involving the interpretation of visual context and ab- stract reasoning. To attain human-level cognitive intelligence, MLLMs must go beyond evaluating options; they must generate solutions for complex tasks that require ad advanced reasoning skills. Ex- isting studies on open-ended visual reasoning tasks (Zellers al., 2019; Zhang et al., 2019; Johnson et al., 2016), 6), which assess MLLMs' cognitive capabilities, are limited in scope and do not fully explore these higher-order reasoning abilities.
In response to these challenges, we introduce VOILA: an open-ended reasoning benchmark de- signed to evaluate whether MIL.Ms possess vision-level understanding and relational reasoning ca- pabilities. Our benchmark focuses on an analogical reasoning task which has been developed as
"Correpooniigg author; nyilmaz3 aasimmaz3@ssu.ddan Code and data: githuh. nuihub.com/nlllmm/Voila

---

Published as a conference paper at ICLR 2025
VOILA with Distractions (VOILA-WD)
VOILA without Distractions (VOILA- XC))
?
?
?
?
Figure 1: Examples of visual analogy questions from the VOILA benchmark with distractions (VOILA-WD) and without distractions (VOILA-ND) The aim is to generate an image that completes the analogy problem following perceptual and relational reasoning. Each visual analogy question has specific rule configuration that leads the answer. The questions in VOILA-WI WD benchmark can apply the distraction rule when no relational pattern exists between images. For the examples shown above, the answers that complete the VOILA-ND analogy by following the relation rules are "two bears driving a car" and "two female children swimming"". The answers to the VOILA-WD analogy are "four [any subject] reading" and "two male children doing anything".
cognitive assessment of problem-solving, decision-making classification, and learning processes (Goswami, 1992). Analogical reasoning consists of diverse atomic abilities; perceptual understand- ing, mapping abstract relationships between visual contents (Gentner, 1983), and transferring rela- tional patterns to novel cases. These integral sub-tasks are essential for achieving a coherent solution to the analogy problem. A key aspect of analogical reasoning is the transfer of knowledge from pre- viously learned relations to new concepts. The task involves two image pairs: reference pair and an application pair. The reference pair shares both visual and abstract contextual information. The goal is to infer the relationship and apply it to predict the unknown content in the application pair. few examples are shown in Figure 1.
Our benchmark incorporates multi-step reasoning process, which is crucial for analyzing where MLLMS encounter difficulties. This process allows for a detailed examination of the models' limi- tations about the key properties of the task. To introduce varying levels of difficulty, we created two sub-datasets: VOTLA-WD and VOTLA-NO Our findings indicate that VOILA-WE presents a greater challenge than VOILA-ND ND due to the introduction of visual distractions where a certain property of the image ( subject, number, task) is irrelevant to the analogical reasoning. Experimental results reveal that sfate-of-thh-ar MLLMs struggle VOILA. Although GPT-4o reaches 79% accuracy in the description of the image and 43% precision the identification of the relationship stages, struggles to produce correct answers to complete the analogy. LLaMa 3.2 achieves the highest per- formance, attaining 13% accuracy in implementing the relationship stage on VOILA-WE WD. Interest- ingly, GPT-4o outperforms other models on VOILA-ND, achieving an accuracy of 29% in applying relationships. However, human performance significantly surpasses these results, achieving 71% and 69% accuracy on VOILA-W and VOILA-ND, respectively.
We observe that least-to-most (L2M) prompting (Zhou et al., 2023) improves model accuracy com- pared to direct answering strategies. Input formats also influence model performance: using sequen- tial images instead of a single image collage results in an average of 40% improvement. Addition- ally, the image generation step notably reduces model performance. We conduct an ablation study on VOIIA-Wi WD using GPT-4o, showing that even when ground truth information (image descriptions or relationships) is provided at each step, the model achieves only 17% accuracy on the applying relations step. Another ablation study the VOI benchmark demonstrates that the model's performance improves by 27% when textual information used instcad of visual input. We describe experimental setup and findings in Sections 4 and 5 in detail.
Our contributions and findings are summarized below:
We present VOILA, large-scale, open-ended benchmark evaluate MLLMs' high-level visual reasoning capabilities. We introduce a method for the dynamic creation of extensive visual anal- ogy questions utilizing text-to-image models.
2

---

Published as a conference paper at ICLR 2025
Rules
Text
Data Cleaning
Building Image Anatogics
VOLLA
Frumowork
Mdetification
Dufisstion
Figure 2: Dataset creation pipeline of VOILA.
We assess state-of-the-ar MLLMs on VOILA, revealing significant performance gap between humans (71%) and MLLMs (13%).
We conduct comprehensive investigation of factors influencing performance: image format, prompting techniques, distraction rules, input information types, and provision of ground truths.
2 RELATED WORK
Visual Analogical Reasoning. Visual Analogical Reasoning is complex task that requires mod- els to recognize abstract similarities between visual contexts and underlying relational rules and to apply these rules to novel visual content to generate solutions. Raven's Progressive Matrices (RPMs) (Raven et al., 1938), introduced in 1938, are one of the earliest visual analogical reasoning tests, designed to assess human intelligence by completing 3x3 matrix of visual patterns based on rela- tional reasoning. Recent visual analogy benchmarks, such as VISALOGY (Sadeghi et al., 2015) and VASR (Bitton al., 2022), offer valuable perspectives but have limitations. VISALOGY, which is not publicly available, focuses on attributes and actions in Google images with manual annotations, while VASR emphasizes general image understanding rather than feature-based relationships In contrast, VOILA introduces more complex and dynamic approach to visual analogical reasoning. incorporates diverse subject relationships and rule-based structures and manipulates up to three properties at time. Additionally, VOILA introduces distraction elements to increase task difficulty, requiring models to discover and filter out the irrelevant changes among properties while solving analogy questions. Unlike static datasets, VOILA allows the generation of over 6.4M distinct visual analogy scenarios across 14 subject types, 13 actions, and 4 numeric values by adjusting flexible property-rule configuration, offering scalable and adaptable evaluation platform for MILMs.
Prompting. Prompting strategies are critical to improving the performance of MLLMs, especially in complex reasoning tasks. Zero-shot prompting provides models with task instructions without examples, while few-shot prompting includes example-based prompts to guide the model toward the correct answer (Brown et al., 2020). Chain-of- breaking down problems into sequential steps, allowing the model to tackle each sub-problem structured manner (Wei et al., 2022). Two common CoT approaches include zero-shot CoT (Kojima et al., 2022), which encourages step-by-step reasoning without examples, and few-shot CoT (Wei et al., 2022), which includes rationales explanations to guide reasoning. Research has shown that few-shot CoT, by providing exemplar rationales, often outperforms zero-shot approaches (Zhang al., 2024). Another method, Least-to-Mo tost (L2M) prompting (Zhou et al., 2023), adopts a similar multi-phase structure as CoT but without the use of rationales. Instead, L,2 progressively breaks down the task, using the solution to each sub-problem input for the next. In VOILA, extend the use of L2M to gain deeper understanding of MLLMs' behavior across sub-tasks, from recognizing visual content generating accurate images based on relational reasoning.
(CoT) prompting enhances reasoning by
Multimodal Reasoning Benchmarks. Multimodal reasoning benchmarks have been instrumen- tal in ad advancing the evaluation of MLLMs, integrating both textual and visual information to assess models' capabilities across a variety of domains. Domain-speciffe specific benchmarks like ScienceQA (Lu et al., 2022), A-OK VQA (Schwenk et al., 2022), Math-Vin Vision (Wang et al., 2024b), and MMMU- Pro (Yue et al., 2024) focus on specialized knowledge, such as scientific, mathematical, and visual question-answerrng reasoning tasks. Meanwhile, benchmarks such as CompBench (Kil et al., 2024), MMRel (Nie et al., 2024), MARVEL (Jiang et al., 2024b), and ScanReason (Zhu et al., 2024) ad- dress more generalized multimodal relational reasoning. Several studies also focus multi-step reasoning tasks, where models must process information sequentially, such as VisualCoT (Shao al., 2024), LogicVista (Xiao al., 2024), and VideoCoT (Wang al., 2024d), while datasets
3

---

Published as a conference paper at ICLR 2025
such MuirBench (Wang al., 2024a), MIRB (Zhao al., 2024) and MANTIS-EWS (Jiang al., 2024a) assess multiple image reasoning. Visual action planning (Gokhale et al., 2019) and visual procedural planning (Chang al., 2020; Su et al., 2024) has also been explored to find relationships between pairs of images. However, VOILA distinguishes itself by focusing on high-order abstract relations and knowledge transfer across multiple images, challenging models not only under- stand but also to generate both images and text while applying correct relational reasoning. These positions VOILA unique benchmark for testing MLLMs' higher-level cognitive abilities.
CONSTRUCTING THE VOILA BENCHMARK
The VOILA benchmark designed to evaluate the abstract reasoning capabilities of MLLMs. This task challenges models to process perceptual information and apply relational reasoning by in- terpreting visual content from three given images generate fourth image according specified pattern. VOILA is a large-scale dataset that dynamically generates visual analogy questions based on demand and configuration. The dataset can generate over 6.4M questions, distributed across 19 unique structures and utilizing total of 7,280 images which makes VOILA highly scalable and adaptable to various configurations. Figure illustrates the dataset creation pipeline of VOILA.
3.1 DATASET CREATION PIPELINE
Property Identification. The VOILA dataset is generated using an image analogy framework (A A' :: B'). Each of the first three images contains distinct properties that form the basis for the visual analogy questions. We identified three key properties: the number of subjects, subject type, and action. In the VOILA benchmark, each question includes three images (J1, I2, Iz), with each image containing corresponding properties (n4, 8;, P. A total of 14 subjects, 4 numbers, and 13 actions were used to create the image dataset. For further details regarding the categorical information of the property types, please refer Appendix A.I.
To structure each visual analogy question, four types of rules are applied in VOILA: Stable, Change, Arithmetic, and Distraction. The rules are assigned to the properties as outlined in Table 1, with each image containing rule-property pairs ((r..:)). Let N symbolize the number of the subject property, P1, P3 and P3 represent the same property (subject type action) but different values. The rule patterns are defined as follows:
Rule Definition.
Stable: The property value in the first image is the same as in the second image:
Pi :: P2 Po.
Change: The property value in the first image changes in the second image:
P1 P2 :: P1 Pz.
Arithmetic: The number of subjects changes by either increasing or decreasing from the first image to the second image. N 1:
N₂ N4 :: N1 Ng + 4 2-2= 1
Distraction:
The property values except the number of subjects different in three images.
There is no correlation among these values, so P is a distraction. After applying increase or decrease changes, if N S O, then N is a distraction:
P1 P2 : Pa ANY.N. .N : N₁ :: N2 ANY + 4 = 3 3: 2 3 = -.
Text Prompt and Image Generation. To ensure that the relationships between images are easily recognizable, the images VOILA must be clear and object-centered. We employ the open-source SDXL model, which generates high-quality images based on a simple text prompt structure that includes the number subjects, subject types, and actions, for example "Two dogs walking". Com- plex overly detailed prompts can lead to incorrect image generation (Podell et al., 2023), so maintain straightforward prompt structures. After generating text prompts, the images for the anal- ogy questions are produced using the SDXL pipeline, with output resolution set 1024 x 1024 and a guidance scale of 8, which controls the fidelity to the text prompt. For each prompt, 30 images are generated. Appendix A.2 provides examples of generated images and their text descriptions.
4

---

Published as a conference paper at ICLR 2025
Table 1: VOILA contains analogies with three properties and four rules applied to these properties.
Properties
Stable
Change
Arithmetic
Distraction
Action Number of Subjects
V
Subject Type
Algorithm Visual Analogy Generation
Three property arrays: reumberrs], subjects[. actioons actionsl Question arra:: mmageo, Imager, Imagez, Images]] Define rules Rg where y = 1, 2, 2, 3,4 (Stable, Changes, Arithmetic, Distraction) Define analogy structures A, where = .,., 19
Input:
Output:
Each structure A, contains variables n., St, and different rule configurations for to do
74 < applyRy, (reembberr))) {Generate possible permutations of properties] 8 < applyR,, (subjecct)))
CoplyRy((ctionnn
combinations{] + combineeroperrtes((u,, 8,, as, count) (Generate random selections with balanced distribution}
for rt, s. a in combinations] do
for x to 3 do
Imagez
findImage(n (ar]. a[xr), alx),, index) {Find images that matches requested properties.
Use the index provide variety of images.}
end for Q < (Imageo, Imager, Imagez, Imagess)
end for end for End
Data Cleaning. essential to verify the alignment between the text prompts and the generated images. Some images may not correspond correctly to their prompts (see Appendix A.3), making them unsuitable for visual analogy construction. These images were manually filtered, and from this process, retained 10 images per prompt, resulting in total of 7,280 diverse images.
Building Image Analogies. To construct visual analogy questions, we combine image features using predetermined rules. Each question follows a specific rule-property configuration, with the number of properties undergoing changes determining the dataset configuration. Since VOILA in- cludes three properties, it proposes three different analogy structures. The number of cases required for each structure is calculated by pairing unchanged properties with unmodified rules. Given that Arithmetic and Change rules modify properties, the remaining two rules are used for the process.
n!!
C -=rn-p
pl (n D)!
where r is the number of rules (excluding Arithmetic and Change), is the total number of prop- erties, and indicates the number of changed properties. This yields 19 unique cases for three configurations: cases for property change, cases for 2 property changes, and case where all properties change. Appendix A.5 details the possible cases and assigned rules for each property.
To create a dynamic VOILA benchmark, we propose the Visual Analogy Generation strategy (see Algorithm 1), which takes input properties, structure, and rule configurations and outputs the cor- responding image sequence for each VOITA question. Each structure assigned property-rule configuration, and the number of distinct analogy questions for each structure is calculated based on rule-property permutations, including dual permutations for the Stable and Change rules, and length-3 permutations for the Distraction rule. Unique number combinations for Arithmetic and Distraction rules are manually calculated to eliminate redundancy (see Appendix A.5 for details).
In VOILA. the analogy questions equally distributed among the 19 structures, ensuring distinct combination of properties for each question. Images can reused across different configurations, preserving uniqueness while distributing image usage evenly.
5

---

Published as a conference paper at ICLR 2025
mageue Cauge
Funluntonn
images
Prempt Deecribe contace these sentence
Eqpaceed Ourgar maTRe SUSDECT acEOn
irage
Pooulti
REXSRATT
nunbtr suraeds, subject lypos octions in the format Triege: Dosorittor
Imagg 37 pla with
bycongainggt procceetees to
Mentry ni valeicrafipe
gour dluuh
Prompt inwntty the changed and unchanged pripertiss chssved tetwoon the for and soond images, tocueing on rumber autpeet, yyes,
connec aax
Kation inseend from playping socor meeding waak
othenciia an
Apo the miltonatii to thidd Image
Prorei Radle the skerined unchenged or did propetaioro Ue imagge ursee
preeed im aap "jumbed e. juuject action jactinn Lite tellw as to desemri mine pispeniae to the Nurth Image.
trage
This
proper rurrare constan bobeaa Irage trogge pespenty
The Inwaseertte
3ar pispert (eot udiig number of sutjern changee between Imaget and sereen image mnalimege the peoparly vodor
Hocels
number- & exfject i nobthsacco action sanding back
enage
3. To demamiie the number of bgaote the burth imags aap lecisase cacruasa can abouveed from image imagge rumber subteers in
Gersauting the image
Liemeno 2 Innnisaeneee
mags
description (putper)
lile (acten??
Figure 3: VOILA multi-step reasoning and evaluation pipeline. The top section illustrates two visual input formats. The left side of the MLLMs connection displays the four primary tasks along with their corresponding prompts, while the right side presents the expected outcomes for each task. The results scored in the evaluation stage utilizing GPT-4o and ground truths.
EXPERIMENTAL SETUP
VOILA Test Dataset. For evaluating MLLMs, the VOILA benchmark is split into two distinct test datasets: VOILA-WA WWD, which includes Distraction rules, and VOILA-ND, which excludes them. VOILA-WD consists of 10K unique questions, applying all four rules across 19 structures, while VAILA-ND comprises 3.6K questions with seven configurations and three rules (see Appendix A.4 for details). Each dataset contains 527 questions per configuration, with 728 images annotated with image contents, relationship explanations, and descriptions of the requested image.
Models. We evaluated several baseline models on both VOILA-WE WD and VOILA-NU GPT-4o (Achiam et al., 2023), Emu2-37B (Sun al., 2024), Qwen2-VL--BBInstruue (Wang al., 2024c), CogVLM2-19B (Hong et al., 2024), SEED-LLaMA-SB (Ge et al., 2023), Llama 3.2-11B, and MolmoE- 7B (Deitke et al., 2024). Models incapable of producing visual output were excluded from the image generation task.
Image Input. Given that some baseline models, such as Cog VLM2 do not support multiple image inputs, we implemented visual input formats: sequential presentation of the three images and an image collage combining the three images into a single visual representation.
Prompting. We applied the Least-to-Most (L2M) prompting strategy (Zhou et al., 2023) and man- ually decomposed the visual analogy task into four sub-problems: (1) understanding the visual content, (2) identifying relationships between images, (3) applying those relationships to the third image, and (4) generating the content of the fourth image. Instead of using sub-questions, ployed sub-instructions, asking the models to solve each sub-task sequentially, with the previous answer appended the next problem. This structured reasoning process allowed performance at each sub-task. We tested various prompts on the baseline models using both L2M and direct answer approaches. The prompts used for multi-step reasoning and direct answering are detailed in Appendix C.I.
em-
evaluate
Evaluation. Performance on VOILA is assessed at each step based on correct property prediction. Using GPT-4o and four distinct text prompts, scored model responses for each step, see Figure

---

Published as a conference paper at ICLR 2025
Table 2: Evaluation results of VOILA using least-to-most (L2M) (Zhou al., 2023) prompting.
LAILAW (VOILA w/ Distraction)
VOILA-ND (VOILA w/o Distraction)
Describing Images
Identifying Relations
Applying Relationship
Describing Images
Identifying Relations
Applying Relationship
Method
Human (AMTurks)
71.36
69.69
Image Collage
COG-VLM2
56.05
14.54
0.41
57.93
19.13
6.39
Dwen-VL.2-Chat
48.41
12.25
0.52
44.80
9.87
3.77
GPT-4o
65.37
24.93
3.94
64.45
25.00
19,43
LLaMa 3,2
68.28
29.00
13.16
67.88
24.45
6.83
Three Separate Images
MolmoE
7.76
0.65
0.08
8.34
0.38
1.00
Qwen-VL2 VL2-Chat
77.80
20.58
0.85
75.80
21.20
5,20
GPT-4o
78.94
42.79
6.44
78.53
38.60
29.03
3. In the first phase, the models' ability to understand the visual content is evaluated. Given three images l4 and their properties Pt and ground truth descriptions G₁, the in this phase is calculated as:
f (():) - -(0 (P, (n,, (,)) = I,G and E [0,2]
(2)
otherwise.
This scoring strategy is applied across the first three phases, where models evaluated whether they correctly identify the properties of the images. In the second phase, we assess the models' ability to extract relationships between the images. In the third phase, evaluate the application of these relationships to a new domain. In the final step. the model-generated images are assessed using a VQA-style approach. For each property, we generated three questions based on the ground truth text and used GPT-4o to answer these questions in relation to the generated image. A similar property-bas based scoring method applied evaluate the generated images. Detailed evaluation prompts for each step are provided in Appendix B.
Human Performance. We also evaluated human performance on the visual analogy task using the Amazon Mechanical Turk (MTurk) platform. A total of 440 distinct analogy questions, incor- porating different rule configurations, were presented to participants. Instruction sets and example questions, with and without Distraction rules, were provided during the experiment. Human par- ticipants were tasked with predicting the properties of the missing fourth image. The results were collected and evaluated against the ground truth text. Additional details on the MTurk human eval- uation study are available in Appendix D.I.
EXPERIMENTAL RESULTS
MAIN RESULTS
We evaluated the high-level reasoning abilities of state-of-the-art MILMs on the VOILA benchmark. Table 2 presents the step-by-step accuracy of the models on both VOILA-W) and VOILA-MA Al- though both GPT-4o and Qwen VL2 achieved peak performance with an average accuracy of 78% during the image description stage, GPT-4o distinguished itself with exceptional results in relational reasoning by reaching an average precision of 40% in both datasets. the application relation- ship step, while GPT-4o outperformed other models by 29%, emerging as the top performer on VOILA ND, LLaMa 3.2 took the lead on VOILA-WA WD with a 13% accuracy rate, surpassing GPT- 4o by 6.7%. This result shows LLaMa 3.2's enhanced ability to identify distractions compared to other models. However, human participants still significantly outperformed all MLLMs, particularly in understanding relationships and making inferences. The performance gap between human par- ticipants and the top perfoorrnnnng models-LLaMi 3.2 on VOILA-W WD and GPT-4o on VOILA-ND equals approximately 58% and 40%, respectively.

---

Published as a conference paper at ICLR 2025
Vels with Detinction
Vola without Detracton
20-
76
70-
Aoune (N0
Adun
20
Dise images
locralyy ng ficlsions
Asshiigg toctenshi
Genciaing Image
bing iages
Monkiying reladors
Aspbiig for Rocanoniii
Ganciating Image
Gapli KP (C)
LaMe 22 GPT-4e (C)
SEED-LWAADU
QVEN-A- (C)
QWENN-VV (3)
Figure 4: Accuracy of VOILA WD and VOILA ND at each step, respectively.
Model accuracy notably declines after each reasoning step, illustrating the growing difficulty as tasks become more complex. Figure 4 shows the accuracy of models across the four reasoning stages on VOILA-WD and VOILA-ND. While most models achieve over 50% accuracy in the initial image understanding stage, their performance drops sharply in the second stage, where they are required to interpret relationships. This decline continues the third stage, where models apply these relationships to generate inferences for the third image. Due to the limited image generation capabilities baseline models, we evaluated this step only on GPT-4o, Seed-LLaMa, and Emu- Across both datasets, performance dropped the image generation stage (see Appendix B.2). These results demonstrate that current MI.LMs struggle with relational understanding and their inference accuracy. Additional details on successful and failed cases are available Appendix B.10.
5.2 PROPERTY SUCCESS OF MODELS
To further analyze how baseline models understand and predict properties, we evaluated model per- formance based three properties: the number of subjects, subject type, and action, across each reasoning step on VOILA-WD and VOILA-ND. Figure 5 highlights the property-base accuracy of four models that perform best VOILA-WD. Each model exhibited different strengths in identify- ing properties at each step. QWEN2-VI, for instance, exhibited strong capabilities in identifying numbers and subject types during the initial two steps, but its performance notably decreased when applying relationships, particularly for these properties similar to CogVLM-2. GPT-4o maintained high accuracy in predicting numbers during the first and second stages but experienced signifi- cant decrease of 60% in the relation application phase on VOILA-WD, followed by an additional 6% decline in the image generation step. Although QWEN2- VL and GPT-4o performed best dur- ing the first stage, QWEN2-VL VL struggled more than GPT-4o to identify the relationships. LLaMa 3.2 achieves the most balanced performance across categories, maintaining relatively high aceu- racy in the complex task of relationship application. While transferring the relationship step is the most challenging part for CogVLM-2 QWEN2-VL VL, and GPT-4o, LLaMa 3.2 struggles more with identifying relationships VOILA-WD. Further property analysis for VOILA-WI is provided in Appendix B.3.
5.3 L2M VS DIRECT ANSWER
We conducted experiments comparing Least-to-Most (L2M) prompting with direct answering ap- proaches on VOILA-BA WD and VOILA-MI ND. Table 3 summarizes the results, showing that L2M prompting consistently improves model performance by breaking down the visual analogy task into sub-problems. This approach leads to higher accuracy in both settings, with a particularly strong impact LODLA-WD, which involves that L2M prompting enhances the reasoning process by encouraging models to solve each part of the problem incrementally. Additional results direct answering are available in Appendix B.
complex rule configurations. These findings suggest
8

---

Published as a conference paper at ICLR 2025
Wo2 (1.21)
QWENZ-VL (30 (1.2M)
LLAMA 3.2 (L2M)
GPT-44 (a) (1.2M)
28
Figure 5: Accuracy of baseline MLLMs regarding properties at each step on VOILA-WD.
Accuracy of the models in the third step using L2M and direct answering.
Table 3:
Table 4: Accuracy the models the third step using L2M on image collage and sequential image settings.
VOILA-WD
VOIIA-BE HD
Direct Answer
Direct Answer
VO-
VOILA-NU
Models
L2M
Image Collage
Soquential Images
Image Collage
Sequential Images
Models
COG-VI TOG-VLM2 (IC)
0.41
0.18
6.39
3.57
GPT-4o (3l)
6.44
0.9
29.03
16.94
QWEN VL2 (L2M)
0.52
0.85
3.77
6.8
Sced LLama 14B
2.99
1.38
2.35
2.03
GPT-4o (1.2M)
3.94
6.44
19.43
29.03
5.4 IMAGE COLLAGE VS SEQUENTIAL IMAGE
We examined the impact of imput formats image collage versus sequential images- on model per- formance in VOILA-WI and VOILA Table 4 compares the results for QW and GPT-4o, which accept both input formats. Our findings indicate that input configuration has a significant ef- feet on visual perception: performance dropped by approximately 40% when models were presented with image collages compared to sequential images. This pattern of performance degradation was consistent across both VOILA and VOILA-MO ND, highlighting that models struggle with visual analogy tasks when images are combined in collage format, which is counterintuitive and can be at- tributed the image resolution constraints. However, our additional study with LLaVA-One Vision (Li et al., 2024) detailed in Appendix B.6, demonstrates that the AnyRes approach enhances the interpretation of collaged images and helps to reduce the performance differences.
5.5 VOILA-WD VS VOILA-ND
According to the accuracy outcomes of mod- els on both VOI and VOILA-NA -N bench- marks, all models, excluding LLaMa 3.2, per- form better addressing the VCILA-ND ques- tions, see Figure 6. The accuracy of best- performer GPT-4o. dropped by 22% when solv- ing VOILA-WO questions. The results demon- strate that implementing the Distraction rule increases the difficulty level of the VOILA. benchmark and proves that VOILA-WD intro- duces more complex challenges compared to VOILA-NU We also analyzed the rule-based performances of LLaMa 3.2 and discovered that applies the Distraction rule better than other rules (Arithmetic and Stable), particu- larly in number property which explains why it achieves better results the VOILA-WD unlike other models.
13,0
15.38
QUB
6.0 1.34
GF7140
0F1-49
QWEER2
LIWW 5.2
Cog VLMZ
OPFT
QUEEM2-V VL.
Figure Performance of the top-8 models on VOILA-ND and VOILA-WD.

---

Published as a conference paper at ICLR 2025
5.6 ERROR ANALYSIS
To assess GPT-4o's performance on the evaluation task and to quantify the differences between human and GPT-4o evaluations, examined 50 visual analogy questions and 180 responses, 30 corresponding to the image generation step, answered by various models using diverse rule config- urations including the distraction rule. The results show that GPT-4o has an error rate of up 10% per step, and its challenges in recognizing relationships do not influence the benchmark assessment procedure. Additional details of the analysis are available in Appendix B.5
ABLATION STUDY
6.1 MODEL PERFORMANCE WITH ACCESS TO GROUND TRUTH INFORMATION
To better understand model behavior each reasoning step, we conducted an ablation study on GPT-4o by providing ground truth inputs starting from the second phase. This study aims to evalu- ate the model's 's reasoning abilities under ideal conditions. Using L2M prompting on the LOILA-WD benchmark, we provided GPT-4o with ground truth image descriptions at phase 2. The model's ' per- formance in identifying relationships increased significantly, reaching 97%, indicating that GPT-4o can effectively analyze changes in text descriptions when given correct inputs. However, when we provided the model with ground truth relationships in the third step, its performance dropped dramat- ically to 17%, well below human performance (71%). ).). These results suggest that GPT-4o struggles to apply known relationships to new visuals, revealing limited reasoning in practical inference.
6.2 How DOES VISUAL INFORMATION AFFECT PERFORMANCE?
We also investigated how visual versus textual information affects model performance on the anal- ogy task by conducting ablation study on VOILA-NO ND. In this experiment, we used direct an- swering prompt without any explanation of the rules. Two experiments were conducted with GPT- 4o, one using three sequential images input and the other using text descriptions of those images. When processing image data, GPT-4o achieved an accuracy of 22%, while with textual input, its ac- curacy to 49%. These results highlight the importance of input format in MLLM performance, exposing gap between visual and textual reasoning abilities.
We also analyze the results based the number of correctly answered questions on DOILA-ND Fig- ure 7 demonstrates the rule-based performance com- parison of models that accept visual and textual in- formation. The outcomes of both models show that the Arithmetic rule included in rule numbers 6 and influences GPT-4o's performance. The models present better accuracy when Stable or Change rules are applied to the number property. We also ob- served that models achieve the lowest accuracy in predicting the properties in rule where all proper- ties in question change at time.
lec inptt G77-40
400
Figure 7: Correct number of questions in VOILA-ND.
CONCLUSION
We introduced VOILA, large-scale, open-ended, and dynamic reasoning benchmark designed to evaluate the visual understanding and analogical reasoning capabilities of state-of-the-art MLLMs. VOILA comprises two sub-tasks: the more complex VOILA-WD and the simpler VOILA-ND. Our evaluations revealed that humans outperform the best-performing models by substantial margin of 58% on VOILA-WO WD and 40% on VOILA-NE ND. These results demonstrate that current MLLMS not only struggle to generate accurate visual or textual outputs but also lack the ability to recognize and apply relational reasoning across images. The significant performance gap between humans and MLLMs underscores the limitations of current models in higher-level cognitive tasks. We anticipate that VOILA will serve as a rigorous benchmark for advancing MLLMs, systematically evaluating their ability tackle complex reasoning tasks that demand human-like intelligence.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
NY is supported by the Republic of Türkiye Ministry of National Education. MP, CB, and YY are supported by US NSF RI grant #2132724. TG supported by the SURFF award from UMBC ORCA. We thank the NSF NAIRR initiative, the Research Computing (RC) at Arizona State Uni- versity (ASU), and cr8dl.ai for their generous support in providing computing resources. The views and opinions of the authors expressed herein do not necessarily state or reflect those of the funding agencies and employers.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.086 087744 2023. 6
Aishwarya Agrawal, Jiasen Lu, Stanislaw Antol, Margaret Mitchell, C. Lawrence Zitnick, Dhruv Batra, and Devi Parikh. Vqa: Visual question answering, 2016. URL abs/1505 00468.
Yonatan Bitton, Ron Yosef, Eli Strugo, Dafna Shahaf, Roy Schwartz, and Gabriel Stanovsky, Vasr: Visual analogies of situation recognition, 2022. URL https://arxiv/ org/ abb/2212. 04542. 3
B. S. Bloom, M. B. Engelhart, J. Furst, W. H. Hill, and D. R. Krathwohl. Taxonomy of educational objectives. The classification of educational goals. Handbook Cognitive domain. Longmans Green, New York, 1956. 1
Alexey Bochkovskiy, Chien Yao Wang, and Hong- Yuan Mark Liao. Yolov4: Optimal speed and accuracy of object detection, 2020. URL http::ss///xxxv.org////// http:/abb/220...../0 10934.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners, 2020. URL https: / arxiv. org/ abs/ 2005. 14165. 3
Chien-Yi Yi Chang, De-An Huang, Danfei Xu, Ehsan Adeli, Li Fei-Fei, and Juan Carlos Niebles. Procedure planning in instructional videos. In European Conference Computer Vision, pp.
334-350. Springer, 2020.
Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jac Sung Park, Moham- madreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, Jiasen Lu, Taira Anderson, Erin Bransom, Kiana Ehsani, Huong Ngo, YenSung Chen, Ajay Patel, Mark Yatskar, Chris Callison- Burch, Andrew Head, Rose Hendrix, Favyen Bastani, Eli VanderBilt, Nathan Lambert, Yvonne Chou, Arnavi Chheda, Jenna Sparks, Sam Skjonsberg, Michael Schmitz, Aaron Sarnat, Byron Bischoff, Pete Walsh, Chris Newell, Piper Wolters, Tanmay Gupta, Kuo-Hao Zeng, Jon Borchardt, Dirk Groeneveld, Jen Dumas, Crystal Nam, Sophie Lebrecht, Caitlin Wittlif, Carissa Schoenick, Oscar Michel, Ranjay Krishna, Luca Weihs, Noah A. Smith, Hannaneh Hajishirzi, Ross Girshick, Ali Farhadi, and Aniruddha Kembhavi. Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models, 2024. URL https:s:/// /ttp///wvvv////////////.////.wvl org/ abs / 2409249.... 17146.
Yuying Ge, Sijie Zhao, Ziyun Zeng, Yixiao Ge, Chen Li, Xintao Wang, and Ying Shan. Making Ilama see and draw with seed tokenizer, 2023. URL httph:tt//// httpp:://axxivoorrt wwwpw//wwwgg//////////////s/ 01218. 6
Dedre Gentner. Structre-mapping A theoretical framework for analogy. Cognitive Sci- ence, 7(2):155-170. 1983. ISSN 0364-0213. doi: http:://oo.og///1110000666661111 80009-3. URL http://wwwcaeeeeeceeecccccc/ec//
11

---

Published as a conference paper at ICLR 2025
Tejas Gokhale, Shailaja Sampat, Zhiyuan Fang, Yezhou Yang, and Chitta Baral. Cooking with blocks: A recipe for visual reasoning on image-pairs. In Proceedings of the IEEE/CVF Confer- on Computer Vision and Pattern Recognition Workshops, pp. 5 5-8, 2019. 4
Usha Goswami. Analogical Reasoning in Children. Psychology Press, 1992.
Wenyi Hong, Weihan Wang, Ming Ding, Wenmeng Yu, Qingsong Lv, Yan Wang, Yean Cheng, Shiyu Huang, Junhui Ji, Zhao Xue, Lei Zhao, Zhuoyi Yang. Xiaotao Gu, Xiaohan Zhang. Guanyu Feng, Da Yin, Zihan Wang, Ji Qi, Xixuan Song, Peng Zhang, Debing Liu, Bin Xu, Juanzi Li, Yuxiao Dong, and Jie Tang. Cogvlm2: Visual language models for image and video understanding, 2024.
URL https: / aawiiiiiiiv.///.i org/ abs/ 2408 6500.
Dongfu Jiang, Xuan He, Huaye Zeng, Cong Wei, Max W.F. Ku, Qian Liu, and Wenhu Chen. Mantis: Interleaved multi-image instruction tuning. Transactions on Machine Learning Research, 2024, 2024a. URL https: opwnrevieweeeeeeeeeeeeeeeereeeee net/ forum? Lerrumii=skLtttuaaal
Yifan Jiang, Jiarui Zhang, Kexuan Sun, Zhivar Sourati, Kian Ahrabian, Kaixin Ma. Filip Ilievski, and Jay Pujara. Marvel: Multidimensional abstraction and reasoning through visual evaluation and learning, 2024b. URL https: //tp:v/aaavvvv/.a org/ / 2404 13591.
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross Girshick. Clevr: A diagmostic dataset for compositional language and elementary visual reasoning, 2016. URL https: /ttpvaaiiivvv..////v///aaa/aaaiv org abs hatg/abs/lii 612. 06890,
Jihyung Kil, Zheda Mai, Justin Lee, Zihe Wang, Kerrie Cheng, Lemeng Wang, Ye Liu, Arpita Chowdhury, and Wei-Lun Chao. Compbench: A comparative reasoning benchmark for multi- modal Ilms, 2024. URL https::/ //aass///axvvaaaa/ orggabs//407..u 16837.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. arXiv preprint arXiv:2205.11916 2022. 3
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer, 2024. URL atttps://arxtt org/ abs /20///24400 408.008000 03326. 6. 9,
Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In The 36th Conference Neural Information Processing Systems (NeurIPS), 2022. 3
Jiahao Nie, Gongjie Zhang, Wenbin An, Yap-Peng Tan, Alex C. Kot, and Shijian Lu. Mmrel: A relation understanding dataset and benchmark in the milm era, 2024. URL https://arxix: 6 09121.
Bryan A. Plummer, Liwei Wang, Chris M. Cervantes, Juan C. Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models, 2016. URL https: /ttr/aai.v.orreeeeeeeee/ wtt.or/g/aa//////o /10... 04870.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhom, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis, 2023. URL https:tt///// /tt//aaiivvv...//// http://///aaaaooo/////////////// httpgg/bb/23337/ 01922.
John Raven, J. C. Raven, J. H. Court, Psychological Corporation, J. C. Raven, et al. Raven's 's Pro- gressive Matrices. Western Psychological Services, 1938. 3
Fereshteh Sadeghi, C. Lawrence Zitnick, and Ali Farhadi. Visalogy: Answering visual analogy questions, 2015. URL https:://axx/ti wwwrexiivv..rveeeomm /ats//b///// www.////...........//eeebbatt.ta /1100. 08 9833.
Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mot-
taghi. A-okvqa: A benchmark for visual question answering using world knowledge, 2022. URL https: / wxwvv org/ 2206. 718...
12

---

Published as a conference paper at ICLR 2025
Hao Shao, Shengju Qian, Han Xiao, Guanglu Song. Zhuofan Zong, Letian Wang, Yu Liu, and Hong- sheng Li. Visual cot: Advancing benchmark for chain-of-thougge reasoning, 2024. URL https:t///// peattps///axivvee http://wwww//////////////// 16999. 3
language models with comprehensive dataset and
Ying Su, Zhan Ling, Haochen Shi, Cheng Jiayang, Yauwai Yim, and Yangqiu Song. Actplan- Ik: Benchmarking the procedural planning ability of visual language models in household activities. In Proceedings the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 14953-14965, 2024. 4
Quan Sun, Yufeng Cui. Xiaosong Zhang, Fan Zhang, Qiying Yu, Zhengxiong Luo, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative multimodal models in-context learners, 2024. URL https:: /arxivvvv.////////// hargg/bb//312/ 13286. 6
Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. Show and tell: A neural image caption generator, 2015. URL https: / aawiiv org/ abs/1411. 4555. 1
Fei Wang. Xingyu Fu, James Y. Huang, Zekun Li, Qin Liu, Xiaogeng Liu, Mingyu Derek Ma, Nan Xu, Wenxuan Zhou, Kai Zhang. Tianyi Lorena Yan, Wenjie Jacky Mo, Hsiang-Hui Liu, Pan Lu, Chunyuan Li, Chaowei Xiao, Kai-Wei Chang, Dan Roth, Sheng Zhang, Hoifung Poon, and Muhao Chen. Muirbench: A comprehensive benchmark for robust multi-image understanding, 2024a. URL https:tp////// arriv.iiiiiv/..i org/ abs/ 2406.0060000 09411. 4
Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Mingjie Zhan, and Hongsheng Li. Measuring multi- modal mathematical reasoning with math-vision dataset, 2024b. URL htt.tttsss//ii..ooo/. abs/ / 220224022..2 14804. 3
Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang. Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution, 2024c. URL htwps:tt/////. httppii/////v.v.//////eee/ org/ orggabs//aab////// 12191. 6
Yan Wang, Yawen Zeng, Jingsheng Zheng, Xiaofen Xing, Jin Xu, and Xiangmin Xu. Videocot: A video ofain-of-thought dataset with active annotation tool, 2024d. URL httht:///aaxtt org/abs/2a 2407. 05355.
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. Mmlu-pro: more robust and challenging multi-task language understanding benchmark, 2024c. URL https: / //a//aiivv////////////////////aa org/ abs/ 2406.000000 01574.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed H. Chi, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903. 2022. 3
Yijia Xiao, Edward Sun, Tianyu Liu, and Wei Wang. Logicvista: Multimodal 1lm logical reasoning benchmark in visual contexts, 2024. URL https: http://wxwww.rr..//o 2tsb//47 /2t.//2277074.....
Xiang Yue, Tianyu Zheng, Yuansheng Ni, Yubo Wang, Kai Zhang, Shengbang Tong, Yuxuan Sun, Ming Yin, Botao Yu, Ge Zhang, et al. Mmmu-pro: more robust multi-discipline multimodal understanding benchmark. arXiv preprint arXiv:2409.028813 2024. 3
Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. From recognition cognition: Visual commonsense reasoning, 2019, URL https: /
hbc::/bb//lie /1811. 10830. 1
Chi Zhang, Feng Gao, Baoxiong Jia, Yixin Zhu, and Song-Chun Zhu. Raven: A dataset for relational and analogical visual reasoning. 2019. URL wtR.httss///axiiv.ooo aaag///b//iie 1903. 02741.
Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. Multimodal chain-of tf-houht- reasoning in language models, 2024. URL https: /arxivv. org/ abs/ 3302.00922 3
13

---

Published as a conference paper at ICLR 2025
Bingchen Zhao, Yongshuo Zong, Letian Zhang, and Timothy Hospedales. Benchmarking multi- image understanding in vision and language models: Perception, knowledge, reasoning, and multi-hop reasoning. arXiv preprint, 2024. 4
Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuur- mans, Claire Cui, Olivier Bousquet, Quoc Le, and Ed Chi. Least-to-most prompting enables com- plex reasoning in large language models, 2023. URL htttps://arxdlv.or///bas. 10625. 2, 7
Chenming Zhu, Tai Wang, Wenwei Zhang, Kai Chen, and Xihui Liu. Scanreason: Empowering 3d visual grounding with reasoning capabilities, 2024. URL 2407. 01525.
14

---

Published as a conference paper at ICLR 2025
A THE VOILA DATASET
A. PROPERTY DETAILS
We select three properties utilize in the VOILA: subject types, number of subjects and actions. In total 14 distinct subject types used in the images: cat, dog, fox, hamster, rabbit wolf, bear, and monkey, male child, female child, man, woman, senior man, and senior woman. Also, each image includes one subject type. The numbers selected for the dataset are 1, 2, and 4. We want to restrict numbers to avoid creating more complex configurations. Action property includes 13 physical activities: playing soccer, driving car, ice-skating, walking, swimming, jumping, typing, writing, digging a hole, carrying something, reading, running, and eating food.
A.2 GENERATED IMAGES BY SDXL
Utilizing the SDXL text-to-image model, we generated 30 images for each prompt using various properties. Some of the generated images are provided in Figures 8, 9, 10, and 11 with their text prompts.
Figure 8:
Figure 9;
Figure 11: Two hamsters carrying something
Figure 10:
Four brown bears driving car
Three senior women writing
One dog swimming
A.3 IMAGE CLEANING
Some of the images generated by the model obtain some problems like object hallucinations and depicting the action clearly. To eliminate these failure cases, we filter the images manually. Figure 12 shows some of the faulty images generated by the model.
(a) Three male adolescents swimming
(b) Two white foxes writing
Figure 12: Unmatched images with text prompt.
A.4 VOILA STRUCTURE
The configuration of VOILA benchmarks is determined by the rules and the number propertics. Table 5 shows all possible rule settings for three property-based cases using all rules that represent
15

---

Published as a conference paper at ICLR 2025
the configuration of VOILA-WD. On the other hand, Table 6 shows seven distinct cases excluding the Distraction rule.
Table 5: In total 19 cases are required to generate visual analogy questions in VOILA-WD
Rule
Action
Subject Type
Number of Subject
Change
Stable
Stable
Change
Stable
Distraction
Change
Distraction
Stable
Change
Distraction
Distraction
Stable
Change
Stable
Stable
Change
Distraction
Distraction
Change
Stable
Distraction
Change
Distraction
Stable
Stable
Arithmetic
10
Stable
Distraction
Arithmetic
Distraction
Stable
Arithmetic
Distraction
Distraction
Arithmetic
13
Change
Change
Stable
14
Change
Change
Distraction
Stable
Change
Arithmetic
Distraction
Change
Arithmetic
17
Change
Stable
Arithmetic
18
Change
Distraction
Arithmetic
Change
Change
Arithmetic
Table 6: In total 7 cases are required to generate visual analogy questions in VOILA-ND.
Rule
Action
Subject Type Number of Subject
Change
Stable
Stable
2
Stable
Change
Stable
Stable
Stable
Arithmetic
4
Change
Change
Stable
5
Stable
Change
Arithmetic
6
Change
Stable
Arithmetic
7
Change
Change
Arithmetic
Table 7: Number of different cases created by rules.
Properties
Stable
Change
Arithmetic Distraction
Action
156
156
22308
Number of Subjects
12
16
24
Subject Type
182
182
30576
A.5 CASE NUMBERS
To caleulate how many different cases we can generate using properties, we implement dual permu- tations for the Stable and Change rules and permutation of three for the Distraction rules. Potential cases the number subjects are manually calculated. Table 7 shows the number various pos- sibilities for each property after assigning the rules. For 14 subject types, the Stable and Change rules generate 182 cases that can be used in the visual analogy questions. Since the Distraction number has large amount of data variation, I fixed the generated number of data according the permutation results of Changes and Stable rules. Utilizing these variations, we can create more than 6.5M analogy questions.
16

---

Published as a conference paper at ICLR 2025
B EXPERIMENT RESULTS
B.1 DIRECT ANSWERING
The reasoning workflow is applied with final determined prompts progressively for each model. We tested the direct answering approach for some benchmark models on both VOILA-WD and VOILA-ND. The models are tested on zero-shot prompting approach without multi-step process. The steps of image description and understanding the relations skipped in that study to directly receive the answer from the model. The results are provided in Table 8.
Table 8: Evaluation results of VOIIA using di- rect answering Applying Relationship. In this setting, there are no immediate results for De- scribing Images and Identifying Relations.
Table 9: Evaluation results of VOILA Gener- aling Images. L2M = least-to-most prompting, DA direct answering.
Method
VOILA-ND
VOIIA-NI
Method
VOILA-WD VOILA-ND
Human (AMTurks)
71.36
69.69
Image Collage GPT-4o (I.2M)
2.76
13.01
Image Collage
Seed LLama 14B
2.14
0.76
Three Separate Images
GPT-4o (L2M)
4.93
20.76
Three Separate Images
GPT-4o (DA)
0.67
11.3
GPT-40
0.90
16.94
Seed LLama 14B (DA)
0.11
Emu-2
0.85
1.08
Emu-2 (DA)
0.03
0.11
Seed LLama 14B
1.35
2.03
B.2
IMAGE GENERATION
Image generation is the last step of the visual analogy questions. To answer the question cor- rectly, MLLMs that are capable of creating visual outputs, need to generate the accurate images aligned with the ground truths. We tested GPT-4o and Seed L.LaMa on VOILA-WD and VOILA-ND datasets to evaluate their performance of image generation after the relational inference step. Since VOILA-ND has less challenging questions, the models perform better on VOILA-N ND rather than VOILA-WA WD which obtains some distraction rules. Since the models struggle to maintain their per- formance until the last step, the accuracy of the image generation performance is below 5% of the best-performing model (GPT-4o) VOILA-WD. The more detailed results are provided in Table 9.
B.3 VOILA WD PROPERTY-BASEL EVALUATION
We comprehensively analyze the performance of the baseline models against the accuracy of the properties of each step. Table 10 and Figure 5 show the performance of various models in VOILA-WD with respect to different configurations such as using L2M directly answering and entering an image collage or three sequential images. The results demonstrate that GPT-4o per- forms better than other models in understanding and identifying relations across all properties in visual content on VOILA-WD. However, LLaMa 3.2 shows more thriving performance in applying relationships subject types and actions.
B.4 VOILA-ND PROPERTY-BASEE EVALUATION
We evaluated models regarding property based performance at each step on VOILA-ND under dif- ferent configurations, including the use of 1.2M versus direct answering, as well the input format of an image collage compared to three sequential images. As the distraction rule affects the applica- tion relationships stage, the model performances at the initial two stages show a similar pattern as in the VOILA WD dataset. The results provided in Table 11 and Figure 13 demonstrate that all models, excluding LLaMa 3.2, achieved better performance on VOILA-ND and increased the accuracy of property predictions in the final step. In contrast, LLaMa 3.2 exhibited a decrease in performance at the last stage, particularly in the application of number relations. Notably, GPT-4o outperformed other models in applying number and subject relations.
17

---

Published as a conference paper at ICLR 2025
Table 10: Performance of various models on different steps VOILA-WD. Model names: IC image collage, 31 = three sequential images, L2M = least-to-most prompting.
Describing Images
Identifying Relations
Applying Relationship
Generuting Image
Subject
Action
Number Subject
Number
Subject
Number Subject
Humanss (MTurk)
93.6
82.3
91.4
Cog (L2M)
71.1
94.1
78.2
34.3
50.6
37.3
6.6
12.6
26.9
(L2M)
30.7
32.6
27.8
31.2
30.0
32.9
QWEN2-VL (L2M)
62.0
89.2
76.1
32.2
53.6
38.8
3.16
21.8
1.9
QWEN2-VL (31) (L2M)
96.9
95.8
81.0
58.9
57.9
41.1
11.6
17.6
20.8
MolmoEE(11 (1.2M)
14.57
32.22
22.18
6.61
10.92
8.02
5.62
10.06
10.63
LLaMa 3.2 (IC) (1.2M)
79.2
91.3
87.6
50.2
58.5
49.9
29.7
40.1
39.1
SEED-LLaMa (3D (L2M)
87.3
93.7
43.8
12.1
19.7
24.8
GPT-40 (L.M))
86.0
95.9
78.8
68.1
70.9
46.5
25.4
33.8
33.2
3.4
3.7
3.2
GPT-4o (31XL2M)
99.6
97.0
81.7
94.3
78.5
55,9
33.8
36.5
35.0
5.6
6.1
GPT-4o (3I)
10.9
13.7
27.5
0.8
0.8
0.8
SEED-LLaMM (31)
5.1
16.9
25.8
1.1
0.7
0.7
Fmm-2
3.0
12.5
0.2
0.2
0.1
CogVLM2
7.8
8.2
23.7
SEBD-LLaMa
Table 11: Performance of various models on different steps VOILA-ND. Model names: IC = image collage, three separate images, 1.2M = least-to- to-toosttttttt prompting.
Describing Images
Identifying Relations
Applying Relationship
Genevating Image
Model
Number Subject Action
Number
Subject
Action
Number
Subject
Action
Number Subject Action
Humans (MTurk)
92.4
85.4
CL2 (L2M)
95
QWEN2-VL (L2M)
56.8
86.7
74.9
30.7
54
34.9
19.8
25.8
21.7
QWEN2-VL (3XXL2M)
96.9
95.4
61.9
57.5
42.4
15.8
27
27.2
MolmoE (3)) (1.2M)
15.5
37.6
26.9
6.23
10.6
8.9
22
14.2
LLaMa 3.2 (IC) (1.2M)
77.8
90.9
88.4
50.9
60.9
49.2
15.1
50.1
48.3
SEED-LLaMa (3DLL2M)
77.9
86.8
73.1
48.1
54,9
37.4
7.2
27.2
GPT--4o (L2M)
84.8
95.9
78.9
69.2
74.3
45.8
45.8
61.2
44.9
15.9
17.9
16.9
GPT-4033)LL2M)
GPT-40(31)
28.9
45.5
18.7
14.4
15.1
14.6
Emu-2
6.2
8.7
13.1
0.4
0.7
0.3
C12VLM2
17.9
SEED-LLaMa (3D)
68
7.8
24.3
SEED-LLaMa
156 Comsn:
ComPrE
Datatonert
the Consent
Relatcaflp
Reladorshgg
94 Conner
Relidrett
Aclis
Figure 13: Property performance of models each step VOILA-ND
B.5 GPT-40 EVALUATION PERFORMANCE
We measured the gap between human and GPT-4 evaluations utilizing confusion tables for each step including all attributes. total 50 visual analogy questions and 180 responses were evaluated.
True Positive (TP): The number of cases where both the human and the GPT-4o agree that the response is correct.
False Negative (FN): The number of cases where the human expresses the answer is correct,
but the GPT-40 says it incorrect.
18

---

Published as a conference paper at ICLR 2025
False Positive (FP): The number of cases where the human says the answer is incorrect, but the GPT-4o states is correct.
True Negative (TN): The number cases where both the human and GPT-4o agree that the response is incorrect.
Accuracy (Agreement): The number of cases where both the human and the GPT-4o agree. Accuracy (Agreement rate) = (TP + TN) (TP TN + + FN)
First, calculated false negative and false positive cases between human and GPT-4o evaluations for each step, attribute, and question answer. Then we computed the accuracy also called the agree- ment rate. The results of each step regarding the attributes provided in Tables 12, 13, 14, and 15. The "Question" in tables represents the question answer merged with three properties (number subject + action).
The results show the agreement rate between human evaluation and GPT-4o was 91% to describe images, 94% to identify relationships, 92% apply relationships, and 91% to generate images. Although the attribute agreement rate is the lowest at 74%, the precision to answer questions, which require integration properties, exceeds 91%
Table Step
Describing Images
Table 13: Step Identifying Relations
Attribute
FP
FN
TP
TN
Accuracy
Attribute
FP
FN
TP
TN
Accuracy
Number
7
142
95%
Number
5
39
78%
Subject
5
5
140
93%
Subject
12
37
74%
Action
14
131
87%
Action
8
40
80%
Question
7
137
91%
Question
2
47
94%
Table 14: Step Applying Relationships
Table 15: Step Generating Images
Attribute
FP
FN
TP
TN
Accuracy
Attribute FP
FN
TP
TN
Accuracy
Number
8
39
78%
Number
2
23
77%
Subject
4
7
39
78%
Subject
4
25
83%
Action
4
38
76%
Action
4
24
80%
Question
2
2
46
92%
Question
2
27
90%
B.6 ANYRES STRATEGY WITH IMAGE COLLAGES
To investigate whether the AnyRes strategy would improve the performance of collaged images, we experimented with the ALLVVA-OneVisiee (Li al., 2024) on VOILA-ND utilizing both image collage and multiple image formats. We tested the model only in the first stage where the images were processed. The model utilizing image collage achieves an accuracy 53%, while model using multiple images performs slightly better, with 57% accuracy in describing images. Table 16 summarizes the results, showing that the AnyRes approach improves the image resolution and closes the performance gap between the process of image collage and multiple images.
Table 16: Performance of LLaVA-One Vision different input types
Method
Input Type
Number Subject
Action
Total
VaaVA-OOeVisia Image Collage
63.8%
94.5%
84.3%
53%
VA-Ve-Oee Vision
Three Sequential Images
67.9%
73.7%
67.5%
57.5%
B.7 IMPACT OF SUB-PROMPTS ON IDENTIPYING RELATIONSHIPS
We experimented with GPT-4o on the VOILA-WE WD dataset to discover the impact of employing sub-prompts for determining property relationships between first and second images. For a fair
19

---

Published as a conference paper at ICLR 2025
comparison, froze the first-step answers and requested the model to find whether the num- ber/subject/action changed or remained the same from the first image to the second image. Af- ter merging the results from three sub-tasks, achieved similar accuracy of 42% with a single prompt experiment. The accuracy of properties is also similar with 94% for numbers, 79% for sub- jects, and 56% for action. The relationship identification performance of GPT-4o with single and three sub-prompts, provided in Table 17 demonstrates that GPT-4o's ability to identify relationships not affected by the number of properties asked in the prompt.
Table 17: Performance of GPT-4o on identifying relationships with different prompting approaches
Model
Approach
Number
Subject
Action
Total
GPT-4o
Single Prompt
94.3%
78.5%
42.8% 56.3%
55.9%
GPT-4o
Three Sub-prompts
94.1%
79.3%
42.3%
B.8 COT L2M FOR APPLYING RELATIONS
To evaluate the effectiveness of CoT in the "Applying Relationship" step, we conducted an exper- iment utilizing GPT-4o on the VOILA-WA WD dataset. For a fair comparison of implementing CoT and L2M for step 3, we froze the first and second-step answers and provided two textual examples and their rationales. We requested from model to find the properties of the fourth image by provid- ing previous sub-task answers. The result of the study provided in Table shows that the L.2M approach with detailed instructions performs slightly better than the CoT approach for this task.
Table 18: Performance of GPT-4o with different approaches
Model
Approach Inputs
Number Subject
Action
Total
GPT-4o CoT
Two Examples
48.8%
33.1
29%
5.96%
GPT-4o L2M
Instructions
33.8%
36.5%
35.0%
6.44%
B.9 HUMAN VS MODELS USING EXAMPLES AND OPTIONS
We conducted an ablation study using GPT-40 on the VOILA-WD dataset to evaluate the perfor- mance gap between human and model responses in the same conditions where two example ques- tions and property options are provided. As the human evaluation was performed in one step, we set up the experiment for the direct answering approach. In addition to the example images and lists, we also provided detailed rationales to support the solutions. The experimental results provided in Table 19 demonstrate that offering examples and option lists impacts the model's performance similar to the L2M approach employing step-by-step instructions. Although the model selects each property answer from the given list and analyzes example images and logic, its performance significantly below the human performance, with notable 65% accuracy gap.
Table 19: Performance comparison between human evaluation and models
Method Approach
Inputs
Number Subject
Action
Total
Human
Direct Answering
Two Examples and List 93.6%
82.3%
91.4%
71.36%
GPT-4o Direct Answering
Two Examples and List
46.6%
25.28%
64.4%
6.8%
GPT-4o
L2M
Instructions
33.8%
36.5%
35.0%
6.44%
B.10 FAILURE CASE
An example of how visual analogy questions evaluate the MLLMs step-by-step is provided in Figure 14. In the example, we employ GPT-4o using the 1.2M approach with sequence of three images.
20

---

Published as a conference paper at ICLR 2025
audjects, and uctioro in the formut Tmagge:
Descrptin
carving beskets with bread, Image two hamosera simubarneusrt
iderrity the ot engiedaand and undt prpidpnpertoon observed from
imege imago ri number autioots, types.
SSttect typa change from
black rocenta in image to rabbms image Actions
change leaping the image conying
Norw jumber
action remane bechergelbbbbeee image and image
the identified unchenged and chunged propertics the image 3io predictshe fourth image. oie me enswer for the fourth image the format of anower number - (numbed) sutject (sutpect] action faction. Use the totlowing rules defermins the propertios for the fourth image:
1.ff property remains constant between image and image the property fourth image
value propetty
Image
2.lf property peoperty (eectuding
subjectal changee
image image property image
subjoots
lees than number lany.
ary subject any action
corrying
Figure 14: Failure L2M case of GPT-4o VOILA-WD
B.11 SUCCESS CASE
An example of how GPT-4o successfully reaches the solution is provided in Figure 15. To show all stages of the task, including image generation, utilize GPT-4o in the example.
21

---

Published as a conference paper at ICLR 2025
Deecrita the content thiss imaage in secrenco ruguuubrr auuuttects, sudject tyoes, and actional formut trrgge Descriptior
O
Imrage 1: Three wom play secour held Imrepe bears are playing fiets.
Identity chumped and anchernged propertten observed from and proparties.
prpperiee: sudtect changes from image to polar bears in image the action changes from playing scccer saliiigngg Unstanged rppertie: and image
Apoty identifted unchanged changed propertes
the lmmgge to predict tourth imaggs.
the fourth image turmat The unsweria
number (numberl [sunject] (action))
leacluding number subjects)
between Imaget image
property value from image fourth image. Othenwise, set 1:10 any'.
3. determine the number suttjecos the fourth image, apply the increase or decteart cate observed from image to image 2 the number autjectt Immege 3. the roouti is loso than one, oot rumber prop coppenyyi any!
The antwerr number 33, aubject any action swimming.
based following
swimming
Figure 15: Success L2M case of GPT-4o on VOILA-WD
B.12 EVALUATION PROMPTS STEP BY STEP
After receiving the answers from MILMs for using the L2M method, we execute the evaluation pipeline which consists of multiple stages. Figures 16, 17, 18, and 19 show the evaluation of the models' outputs step by step.
22

---

Published as a conference paper at ICLR 2025
we shorc comparnggte prowaked toats the groud truhh nox eon image
numbers sullest types, and oction. Comparing samiaa suteest tyons, duch "mees" versue manern.
"oonior connect aorig poie sachpripers propprty enwiion gran trroos properties corner for imag, then nosigs
potnn imagrrta matchang.
Ground butt Imagge these childrun outing ragi timee dogo unting food Imagge muse
Texta companc: tmage Three bapa and uating food
pundeurs fetmage Three
atting and losting el boera food in park saiiiiin 3: hour bopananntt eyeggiicaalll. by guiha puth. colpur uocio following informution imagle amparnat
an.
subjuct tapks ir..
actors
In "numbert
boddect lape
Thaage maapee-teett marshingh
Sadees seseeleeee
Figure 16: Step evaluation
Givs ma the Prwancee by compatte proppredeed teen to ground trth nuthoott devertinntt the mode connectly
Identhee unotaanged changed pmpartes tetwwen iaga imag impartee campare otherzile thre srpprrtis Groundmum tnes: humber remrinn conature dy Action inmain conatonn exting feod. sygjchtppels changed foommmat Mieiee dugs.
property.
pelnttor identhy changes sccuraty.
tin Changed imaagg
compane
Chingged Propert Subpoot
of foolin pasi ('iie Lnditinggd Proprii periessin fn Number Susiponts mame tives.
Wete mueuuttut BBC wepoot with lowinng information eech mmgg without axplonationn
rumben
setlors in. Alentlyy chergiaciid
Trumber's 1,
"sutject typelt
change"1:0
Figure 17: Step 2 evaluation
23

---

Published as a conference paper at ICLR 2025
Giwwme ecoewity comporing
ground out newniraf mode
itenritis the prppattee of sure uittern sinpertypss ant nolorn rot it wi uth eas any for prosety, onyyy courte connct smodded arooha formha pooperty Ommerti connect
fordiing
Groundlruth vodo lour uriin
localte compare. Hecarnss tumbe coutuoos dugs eccion cnnniinggt ona giosypally
wasst slpoct utlowing snlo pioomttion eachinege without oxplamaiinn
runderiint
oudeesl bpaeirr,
"rumbert: 1 "subject
Figure 18: Step 3 evaluation
by answering following three image with "yes" or "nc." Assign i point for each "yes" response, and points each response, the oulput object
Questions: "Is there a dog?"
"Is there four subject?"
Are subjects subjlccs
"gr:
Figure 19: Step 4 evaluation
24

---

Published as a conference paper at ICLR 2025
C PROMPTS
C.I PROMPT SELECTING
To find the best-performing parameters, tested various text prompts on MILMs on VOILA and VOILA-WI utilizing 0-shot and ceast-to-most prompting. Since VOILA-WD includes distrac- tions, we modify the text prompts for VOILA-ND by excluding the explanation of distraction rules. Figure 20 displays diverse zero-shot prompts tried before selecting the final instructions. For L2M prompting, we need multiple text instructions for each step. Various prompts are tested for each sub-prue problem in multiple-ster step reasoning process, see Figures 20, 21, 22, and 23.
You understand
visuel three images, looking et changes in numbers, subject types, and actions. To find the changes the number you need to look at the rate of increase or decrease.
wes
To relationships between
questions:
images by
Give the (number) subject - (subject) action (action)" the subject number in the fourth image is lees than one, the number the fourth image is 'any' if there are different subject types in all, three images, the subjeect in the fourth image 'any. If there are different actions in all three images, the action in the fourth image 'amy.'
Then "The relationship the third image determine the fourth
what fourth
You relationships observed actions. The recation of numbers depencis on the rate of increase or decrease the subject in the third image is different from the supjects in the first and second images, the subject in the fourth image can be anything. the action the third image different from the actions anything. What the fourth image? "The
assistant
visual analogy questions three and find the fourth image numbers, subjects,
job figure how
second images
the first second imagees, the action the founth subject (subject)
number (number)
(action)"
Completee the visual analogy by comparing et subject type, number, and action properties: is to as C To find the changas in number you need to look at increasing or decreasing artthmetlc operations, If the number of the fourth image smaller then or subjects and
different each three images, please 'any' this Find the fourth (number) subject (subject) action (action)"
format of "The
Complete the visual analogy by comparing at subject type, number and action properties: to as is to? Answer in format of "The answer is: number (number) subject - (subject) action (action?"
Complete the viuual ansingy, A to B as C is D, by comparing at object type, number and action proparties. Number can be find in decreasing incressing order. What is the number object, object type action fourth image?
Figure 20: Zero-shot prompt selection.
Prompt Describe the content the images in format number (number]] subjjctt- (gubjj)) (uummbbeissjjjjeiseteeeeeee
(action)
Prompt 1: Decoribe the contort throe images one sentenoe usinggnungmmrrrr.. subjects, and actiono in the format of "Imago: Description"
Prompt image consists images. content images sentence ueing numbers, subjects,
Figure 21: First prompt selection of L2M process
Prompt 2: Ident the relationehip between the first and second locking changed stable pattern numbers, subject types, and actions.
Prompt identt subject types, and sction properties If the property in the firs: image is di different from the second image, use the format of "property is charged from X to y" use "prpperyy
ged and unchenged properties auserved between first second images, tocusing number subjects,
constant
Prompt Ident the changed and unchangad properties observed between the first and second imAges, tocusing number subjacts, decrease
subject types, action properties. For number ubjects, cons the change
Figure 22: Second prompt selection of L2M process.
25

---

Published as a conference paper at ICLR 2025
Prompt Apply unchanged changed number, subject, action properbies constant, itremains the same in the fourth image. the progerty is changed, and it's the same the first image, then the property the fourth image will be the same in the second image. Otherwise, it can be any. What is the fourth image? Write the answer in the formet of "The fourth image number - (number) subject (subject) action (action)"
image to fourth image property
Prompt S: Apply the unchanged and changed property rules to the third image to predict the fourth image. What the fourth image? "The answer is number - (number) subject (subject) action - [action] If property except number changed and different in all three images, the propertyin the 'sny'.
fourth
Prompt Apply the unchanged changed number, constant, remains same fourth image. subject changed, and is same the first image, then the subject will oe the same in the second image, Otherwise it can be 'any. an action is constent, rimaina the same in the fourth image. the action changed, and la the same the first imags, then the action in the fourth image will be the same in the second imaga. Otherwiae, it can lany.. number [numbar)
action properties the third
fourth
subject
image
What the fourth image?
answer format "The fourth
(subject) action (action)".
Apply
predict number sutject, subject action fourth image:
1. property the same first second Images, property value from third image the fourth Image.
2.If a property lexcluding number of subjects] changes bewween the first and second images and is the same in the first and third Images, set the property value from the second image to the fourth image. Otherwise, it bny.. 3.Apply the
from first the seconci Image the third If
than
number to lnyy..
you prediction as: The answer is number " (number) subject - (subject) action (ection)'
Prompt Study the content of the three images in series, Ide ntity the changed and unchanged properties between the first image and the second Image. If property remains constant between first and second images, the corresponding property in the fourth image will have the same value that of the third image. It property /ecoluding the number of subjects) changes between the first second images image.
the first and
respective property Predicithe description of the hidden fourth image format of The answer is number - ([numbrr) subject (subject) action (action)
images,
image
the
Figure 23: Third prompt selection of L2M process.
D HUMAN EXPERIMENT
D.1 MTURK INSTRUCTIONS
The human study was conducted by using the Amazon Mechanical Turk platform. Participants solve 440 various analogy questions in total. The instructions and example questions of the study were provided in Figure 24.
Parcaptual Understanding and Relational Reasoning with Visual Analogy Questions
Consent firm I1 Ehc tut Lime ane pal neltctading this stuby ploaru bake pame lime seading form an endima wain Bail agaa with Sunneting ng thk indeame then the tave nend connent torm and syree partituate tet stady Di meve rad and spre with consent torm
uncersoneng resoerrpeoeesseee
edtorness, and egatication.
vetsooes
phevacalio sto kammioio The goet
the atomalgp adoormno
the new sell of tirageect the appliiribe paik
Your tase
unil your For caen you will get two rilennns and apploomtn imapy Your goot lovaare the bert cowrer on rerget image from droptton ni
image pain syrk
app target inage
Quick Guide
1) tivitee moneraes relueen antrneue inagee you oan mon repect typ- nanveer and actice pro partaes 20 toy te of the gropprtio merioonnc olboro do not vhttlyy lie velation shap tpt reullllll by not proviible tolully thered wichi magee, phroe selved "Srytheng" Teption
bulld understaning
wranpii solied Mific
Closs mestorac
?
?
Figure 24: Human experiment of MTurk platform.
26


Reviews:
[{"review_id": "hBhoyZTOuW", "reviewer": "Reviewer_pCZi", "summary": "The paper introduces VOILA, a benchmark specifically designed to evaluate Multimodal Large Language Models (MLLMs) in perceptual understanding and abstract relational reasoning across images. By using analogical mapping, VOILA requires MLLMs to generate an image that completes an analogy between two image pairs, testing the models' relational reasoning without predefined answer choices. The benchmark includes challenging tasks, with models struggling to match human performance, especially in higher-level reasoning steps. Performance improves with least-to-most prompting strategies, but there remains a substantial gap between the best-performing model and human results.", "questions": "Please refer to Weaknesses", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "• VOILA stands out by focusing on abstract visual analogies, making it a valuable addition to existing MLLM evaluation benchmarks, particularly in assessing perceptual and relational reasoning.\n\n• The multi-step reasoning approach and comprehensive ablation studies offer a detailed examination of the current limitations in MLLMs.", "weaknesses": "• The paper emphasizes that this is a dynamic benchmark. For a static benchmark, once a configuration is provided, it can be fully generated and then annotated for correctness by human evaluators. However, as a dynamic benchmark, once it is generated, how to ensure the correctness of the benchmark and how to make it scalable, which remain challenging and difficult to guarantee. This issue is unsolved in this paper.\n\n• VOILA’s reliance on GPT-4o for model evaluation makes the evaluation process both resource-intensive and costly for practitioners.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "b6Sdd8GO4A", "reviewer": "Reviewer_YkdC", "summary": "The authors propose a new benchmark called VOILA which evaluates the perceptual understanding and analogical reasoning capabilities of multimodal language models, particularly its reasoning capabilities across multiple images. VOILA requires a MLLM to understand relations between images and generate a new image that follows the pattern. This task is open-ended and results are evaluated using a strong LLM. Results indicate that even the best MLLM performs much worse than an average human (30% vs 70%)", "questions": "Can the authors conduct a small experiment to validate that their evaluation strategy is accurate?\nThe prompt to identify relation asks the model to identify differences in subject, types and actions all in a single prompt. What if this problem was broken down to three sub-prompts? It seems intuitive that performance should improve", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "This is generally a well organized and well written paper.\nThe proposed benchmark is interesting and highlights a critical shortcoming in existing MLLMs\nThe benchmark is well curated, there is also manual filtering involved to ensure quality", "weaknesses": "This task is not exactly novel, [1] propose a new task that evaluates visual cognition of MLLMs. The new novelty comes from the requirement that MLLM generate the output image. \nThe evaluation is not grounded in existing psychological assessments for instance the reasoning used in [1] is widely used in neurodevelopmental and neuropsychological research.\nGPT4o seems to be poor at identifying relationships (from image) in VOILA, does this affect the evaluation which in turn also uses GPT4o. \n\nReferences\n[1] https://arxiv.org/abs/2406.10424", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AjxYfaqFJ1", "reviewer": "Reviewer_Jyff", "summary": "This paper proposed a new benchmark named VOILA, aiming to evaluate the performance of visual language models in understanding abstract relation understanding by designing visual analogy questions.\nExperiments show that many models can describe the images and idenitify the relationship between images, but cannot apply the relationship very reliably.", "questions": "1. One line of related works I think this paper need to mention is on the benchmarking of multiple image understanding, such as works of Muirbench and MIRB, especially MIRB, which has already included visual analogy as one sub task for testing current state-of-the-art visual language models.\n2. The data generation pipeline uses diffusion models to geerate images from given text prompts. I'm wondering how reliable is the diffusion model? how many images are discarded. Perhaps include a performance of human on the final dataset can answer this (I noticed that in table 3 there are some human performance, but why is human performance only tested on the 'Applying relationship' subset?)\n3. In my understanding, this paper only considers analogy on properties like number, subject, and action. Is these enough to cover all possible visual combinations that can form analogies?\n\n\n\nMuirbench: https://arxiv.org/abs/2406.09411\nMIRB: https://arxiv.org/abs/2406.12742", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The visual analogy perspective of evaluating vision language models is interesting.\n2. The paper gives a very detailed the description of how the benchmark is collected.", "weaknesses": "1. The analogies considered in this paper seems to be restricted to only numbers, subjects, and actions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "N98e48Yqrj", "reviewer": "Reviewer_qyEy", "summary": "This paper introduces a new benchmark for evaluating multi-modal large language models’ perceptual understanding and abstract relational reasoning through analogy completion. It contributes a dataset creation pipeline, a large evaluation dataset consisting of 10K+ questions, and extensive evaluations of both open-source and proprietary models on this benchmark as well as detailed analyses of their performance on different subtasks.", "questions": "- Did the authors include examples of each subtask in the prompts? \n- The authors mention they evaluated Emu-2 on the image generation subtask in 5.1, but Table 3 is missing its numbers? \n- In Figure 6, it’s interesting that LlaMa 3.2 is the only model with higher performance on Voila-WD than Voila-ND, while all other models exhibit large drops from Voila-ND to Voila-WD regardless of their absolute accuracies. Can the authors provide insights into why this occurs for LlaMa 3.2?\n- Nit: table 4, it’s VOILA-ND in both columns – should one of them be VOILA-WD instead?\n\nSuggestions: \n- While the authors motivate the visual analogy task by saying that it tests for higher-order reasoning, they break down the task into four subtasks, each of which is an easier task (despite being still difficult to the model). While the reviewer agrees with the authors that this task decomposition enables fine-grained analyses of models’ weaknesses, it’d also be interesting to see an evaluation of the end-to-end analogy completion task. \n- Table 2 can highlight the performance drops from the previous to the next stage\n- The authors mentioned that models struggle with the visual analogy task when images are combined in a collage format due to resolution constraints. It’d be interesting to see if this finding holds with high resolution image collages for multi-modal models that support the AnyRes strategy such as Llava-OneVision.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The idea of using visual analogy completion for evaluating multi-modal large language models is well motivated and somewhat unique \n- The visual analogy completion problem is well defined and formulated, with clear definitions of the three properties and four rules.  \n- The dataset has been manually cleaned to ensure that the generated images match the texts.", "weaknesses": "- The experiments can be more complete. Currently, most evaluations focus on the first three steps of the analogy completion task with only a few data points on the image generation stage. \n- The paper presentation can be improved. For example, the writing can be improved to highlight the most insightful and interesting takeaways. \n- While the visual analogy task is interesting, the reviewer is uncertain about the practical value of this benchmark especially given the abundance of multi-modal benchmarks – for example, why should the community use this benchmark to evaluate models’ relation understanding instead of existing VQA benchmarks that can also test for relation understanding?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3TbpwEAkgr", "reviewer": "Reviewer_pi5r", "summary": "The paper introduces VOILA, a benchmark designed to assess multimodal large language models (MLLMs) on tasks requiring visual perception and abstract relational reasoning. VOILA primarily focuses on visual analogical reasoning, where models must generate an image that completes an analogy given two pairs of images. The benchmark includes two subsets, VOILA-WD (with distractions) and VOILA-ND (without distractions), testing MLLMs on varying levels of complexity. Contributions:\n1. A dataset creation pipeline allowing large-scale generation of visual analogy questions, and a large-scale, open-ended benchmark to evaluate MLLMs’ high-level visual reasoning capabilities. \n2. Comprehensive evaluation of state-of-the-art MLLMs, highlighting performance gaps in relational reasoning compared to human baselines.\n3. Ablation studies reveal model limitations in handling complex visual relationships, suggesting areas for improvement in future MLLM development.", "questions": "1. The prompt4 for generating images seems unnecessary for evaluating the \"reasoning\" capabilities of MLLMs. Also, the reviewer is curious how the authors use GPT4o to generate images since it is not clear whether they call another image generation model to generate images in the web interface. And it seems gpt4o API does not support image output. \n2. Why Table 4 and Table 5 only use the selected models for the ablation study?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. The proposed problem is a novel multimodal reasoning problem, which is well-defined and advances abstract relational reasoning in MLLMs​.\n2. The benchmark design is comprehensive, encompassing various rule configurations and levels of task difficulty.\n3. The results reveal the weaknesses of current multimodal LLMs.", "weaknesses": "1. The paper does not address why the proposed problem is an essential and meaningful problem to solve. In what applications/scenarios is the capability essential?\n2. The comparison between the model and human performance may not be fair: (1) Humans are given two examples. (2) It seems the task for humans is to choose properties from available options, while MLLMs need to predict those properties.\n3. Table 3 takes up a lot of space and has a lot of numbers, but it is barely mentioned or explained in the text.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)