Beyond Perception: Evaluating Abstract Visual Reasoning through Multi-Stage Task
Authors: Yanbei Jiang, Yihao Ding, Chao Lei, Jiayang Ao, Jey Han Lau, Krista A. Ehinger
Accepted at: acl25

Beyond Perception: Evaluating Abstract Visual Reasoning through Multi-Stage Task

Yanbei Jiang1, Yihao Ding1,2, Chao Lei1, Jiayang Ao1
Jey Han Lau1, Krista A. Ehinger1
1The University of Melbourne, 2University of Sydney
yanbeij@student.unimelb.edu.au
jeyhan.lau@gmail.com
kehinger@unimelb.edu.au

Abstract
Current Multimodal Large Language Models (MLLMs) excel in general visual reasoning but remain underexplored in Abstract Visual Reasoning (AVR), which demands higher-order reasoning to identify abstract rules beyond simple perception. Existing AVR benchmarks focus on single-step reasoning, emphasizing the end result but neglecting the multi-stage nature of reasoning process. Past studies found MLLMs struggle with these benchmarks, but it doesn’t explain how they fail. To address this gap, we introduce MultiSTAR, a Multi-Stage AVR benchmark based on RAVEN, to assess reasoning across varying levels of complexity. Additionally, existing metrics like accuracy only focus on the final outcomes while do not account for the correctness of intermediate steps. Therefore, we propose a novel metric, MSEval, which considers the correctness of intermediate steps in addition to the final outcomes. We conduct comprehensive experiments on MultiSTAR using 17 representative close-source and open-source MLLMs. The results reveal that while existing MLLMs perform adequately on basic perception tasks, they continue to face challenges in more complex rule detection stages. The dataset and code are available at https://github.com/YanbeiJiang/MultiSTAR

1 Introduction
Multimodal Large Language Models (MLLMs) demonstrate proficiency in addressing a wide array of visual-text inquiries and show strong multimodal understanding ability in tasks such as visual question answering (Goyal et al., 2017; Marino et al., 2019; Ding et al., 2023), image captioning (Saito et al., 2023; Vinyals et al., 2015; Jiang et al., 2024a), and visual grounding (He et al., 2024; Deng et al., 2021). These tasks focus on evaluating the models’ capability to understand real-world or domain-specific knowledge. However, Abstract Visual Reasoning (AVR) presents a different challenge, focusing on a model’s ability to identify and reason through abstract patterns, relationships, and rules. A well-known example of AVR tasks is RAVEN (Raven, 2003; Zhang et al., 2019), as shown in the left half of Figure 1. The solver needs to select the correct panel from a answer set to complete a 3x3 problem matrix by deducing the visual rules governing the grid’s arrangement. For instance, by analyzing the colors of each panel, one might observe the color remains consistent across each row. Unlike other multimodal tasks in real-world scenarios, AVR focuses on reasoning about arbitrary visual elements, serving as a robust benchmark for evaluating the zero-shot reasoning capabilities of MLLMs in visual contexts (Mańdziuk and Żychowski, 2019; Santoro et al., 2018).

Previous works have consistently shown that AVR tasks pose challenges for MLLMs in zero-shot inference settings. Despite recent advancements like Chain-of-Thought prompting (Ahbarian et al., 2024; Gendron et al., 2024) and the inclusion of oracle captions (Zhang et al., 2024), models continue to perform at near-random levels on these tasks. The AVR datasets used commonly in these evaluations like RAVEN primarily focus on single-step end-to-end reasoning (i.e., giving the models the questions and asking them to derive the final answer; left half of Figure 1) (Santoro et al., 2018; Nie et al., 2020; Cao et al., 2024). However, this design deviates from the human reasoning process, which often involves sequential steps: starting with single-panel perception, progressing to panel comparisons, and finally deducing the underlying rules before solving the puzzle. Previous datasets often omit these intermediate stages, and as such cannot evaluate MLLM’s step-by-step reasoning capabilities and identify where they struggle within the reasoning process. This highlights the need for benchmarks that assess intermediate perception and reasoning processes. Additionally, existing
Figure 1: Left: RAVEN puzzle. The correct answer is 1. Right: Direct Answer subtask, where questions are independent for each configuration; Logical Chain subtask, where information from previous stages is used to assist in answering the current stage. All questions here focus on the concept of Number.

evaluation metrics like accuracy measure only the performance of the final answer without considering the correctness of intermediate steps., and so they are unable to assess partial success.

To address these limitations, we introduce MultiStAR, a Multi-Stage Abstract Visual Reasoning dataset based on RAVEN, to evaluate MLLMs on the intermediate steps in the reasoning process. As shown in the right half of Figure 1, the dataset is divided into two sub-tasks, each focusing on different aspects of reasoning. The first sub-task, referred to as Direct Answer, evaluates model performance at varying levels of complexity to assess perception and reasoning abilities at each individual step. Using template-based methods, we generate questions based on RAVEN, ranging from basic object recognition to advanced comparison, pattern recognition, and rule inference. This approach ensures comprehensive coverage of reasoning patterns. The second sub-task called Logical Chain, emphasizes how models measure and maintain logical correlations across reasoning steps. Using puzzles from the RAVEN as the final question, we decompose the reasoning process into a sequence of subproblems in a bottom-up manner. Each stage in this chain links the current reasoning task to its dependent subproblems, requiring the model to combine current information with outputs from previous stages. To assess the correctness of intermediate steps, we introduce a novel metric, MSEval, which provides a more fine-grained assessment of the model’s reasoning process for the logical chain task. MSEval uses the correct answer probabilities at each stage to compute the joint probability across the reasoning process. This approach considers both the correctness of the current stage and all dependent intermediate steps.

In summary, our contributions are: 1) we introduce the MultiStAR benchmark, designed to evaluate models across different stages of reasoning through two subtasks, allowing for a more granular analysis of their performance throughout the reasoning process; 2) we present a novel metric that incorporates the correctness of the current stage as well as the accuracy of its dependent intermediate steps; and 3) we perform extensive experiments on a wide range of state-of-the-art MLLMs, our results indicate that current VLLMs still lack the ability to navigate multi-stage logical dependencies effectively.

2 Related Work

Visual reasoning benchmarks have evolved to assess the capacity of AI models on various tasks including compositional (Johnson et al., 2017), commonsense (Gao et al., 2022; Li et al., 2024), scientific (Hiippala et al., 2021; Saikh et al., 2022; Yue et al., 2024), and abstract visual reasoning. Both commonsense and scientific reasoning tasks require real-world knowledge and a prior understanding of specific domains. Abstract Visual Reasoning (AVR) is the main focus of this work, and it primarily involves a classification task where models select an answer from a fixed set of choices
<table>
  <tr>
    <th>Dataset</th>
    <th>Num. of Images</th>
    <th>Num. of QA pairs</th>
    <th>Reasoning Domain (Task Focus)</th>
    <th>Question Generation</th>
    <th>Answer Type</th>
    <th>Functional Program</th>
    <th>Multi-Step Structure</th>
  </tr>
  <tr>
    <td>CLEVR (Johnson et al., 2017)</td>
    <td>100K</td>
    <td>1M</td>
    <td>Compositional (3D shapes)</td>
    <td>Template</td>
    <td>Open QA</td>
    <td>✓</td>
    <td>x</td>
  </tr>
  <tr>
    <td>CRIC (Gao et al., 2022)</td>
    <td>96K</td>
    <td>494K</td>
    <td>Commonsense (Daily life)</td>
    <td>Template</td>
    <td>Open QA</td>
    <td>✓</td>
    <td>x</td>
  </tr>
  <tr>
    <td>AJD (Hippala et al., 2021)</td>
    <td>5K</td>
    <td>15K</td>
    <td>Scientific (Science diagram)</td>
    <td>Manual</td>
    <td>MCQA</td>
    <td>x</td>
    <td>x</td>
  </tr>
  <tr>
    <td>ScienceQA (Saikia et al., 2022)</td>
    <td>10K</td>
    <td>21K</td>
    <td>Scientific (Science problems)</td>
    <td>Manual</td>
    <td>MCQA</td>
    <td>x</td>
    <td>x</td>
  </tr>
  <tr>
    <td>MMU (Yue et al., 2024)</td>
    <td>11.5K</td>
    <td>11.5K</td>
    <td>Scientific (Exam questions)</td>
    <td>Manual</td>
    <td>MCQA</td>
    <td>x</td>
    <td>x</td>
  </tr>
  <tr>
    <td>SEED-Bench (Li et al., 2024)</td>
    <td>19K</td>
    <td>19K</td>
    <td>Commonsense (Spatial, temporal)</td>
    <td>Neural</td>
    <td>MCQA</td>
    <td>x</td>
    <td>x</td>
  </tr>
  <tr>
    <td>MARVEL (Jiang et al., 2024b)</td>
    <td>0.8K</td>
    <td>3K</td>
    <td>Abstract shapes</td>
    <td>Template</td>
    <td>Open QA</td>
    <td>x</td>
    <td>x</td>
  </tr>
  <tr>
    <th>MultiStAR (Direct Answer)</th>
    <td>8.1K</td>
    <td>21.7K</td>
    <td>Abstract shapes</td>
    <td>Template & Neural</td>
    <td>MCQA</td>
    <td>✓</td>
    <td>x</td>
  </tr>
  <tr>
    <th>MultiStAR (Logical Chain)</th>
    <td>0.50K</td>
    <td>3.92K</td>
    <td>Abstract shapes</td>
    <td>Template & Neural</td>
    <td>MCQA</td>
    <td>✓</td>
    <td>✓</td>
  </tr>
</table>

Table 1: Comparison of various VQA datasets. Template: generated using predefined rules, Manual: written by humans, Neural: generated using large language models, Template & Neural: generated using predefined rules and rewritten by large language models. Open QA: free-text answers, MCQA: Multiple-Choice Question Answering. Functional Program: Indicates whether the dataset is automatically created by functional programs. Multi-Step Structure: Highlights whether the dataset includes a hierarchical structure with interdependent reasoning steps.

![Left: Direct Answer subtask, showcasing six configurations along with their corresponding examples. Right: Logical Chain task, presenting a partial view of the logical chain (see full chain and the chain designing rationale in Appendix A.3). Examples are provided for one specific path in the chain.](page_104_670_1237_246.png)

Figure 2: Left: Direct Answer subtask, showcasing six configurations along with their corresponding examples. Right: Logical Chain task, presenting a partial view of the logical chain (see full chain and the chain designing rationale in Appendix A.3). Examples are provided for one specific path in the chain.

based on abstract patterns and rules (Mańdziuk and Żychoński, 2019; Zhang et al., 2019; Santoro et al., 2018; Nie et al., 2020). A few other AVR benchmarks address generative tasks, where models are tasked with recreating elements that fit within a given visual sequence, introducing additional complexity by evaluating a model’s creative reasoning capabilities (Chollet, 2019; Moskvichev et al., 2023). The most similar benchmark to ours is MARVEL (Jiang et al., 2024b), which targets AVR tasks and extends reasoning diversity with six core patterns across geometric and abstract shapes. It also includes basic perception questions to assess visual comprehension. However, MARVEL is still limited in its capacity to analyze intermediate reasoning steps. Table 1 shows the key statistics and features comparison of major multimodal reasoning datasets alongside our proposed MultiStAR benchmark.

3 Multi-stage Evaluation Benchmark

3.1 Task Definition and Configuration

Our dataset comprises two sub-tasks, Direct Answer and Logical Chain, both derived from RAVEN but with distinct reasoning patterns and focuses.

3.1.1 Direct Answer

To uncovering where the MLLMs likely to succeed or struggle in the individual stages, this sub-task explores AVR across multiple levels, which is divided into six configurations, shown in Figure 2:
a) One Panel Basic Perception (1P-B): The puzzle image consists of a single panel \( I = p \), focusing on basic perception questions, such as determining the number of objects, the shape, or the position of a single object, without requiring any comparison.
b) One Panel Comparison (1P-C): The puzzle image remains a single panel, but questions require intra-panel attribute comparisons.
c) Two Panels Comparison (2P): The puzzle image consists of two panels, \( I = (p_1, p_2) \), requiring cross-panel comparisons.
d) One Row Rule Deduction (IR): The puzzle image is a single row of three panels, \( I = (\mathbf{p}_1, \mathbf{p}_2, \mathbf{p}_3) \), and the task involves identifying a rule that governs the sequence.

e) Two Row Rule Deduction (2R): The puzzle image contains the first two rows, each with three panels, denoted as \( I = (\{\mathbf{p}_{1,1}, \mathbf{p}_{1,2}, \mathbf{p}_{1,3}\}, \{\mathbf{p}_{2,1}, \mathbf{p}_{2,2}, \mathbf{p}_{2,3}\}) \). The task is to find a rule that applies to both rows.

f) RAVEN puzzle (Final): The original puzzle from RAVEN dataset.

Formally, given an puzzle image \( I \) (which consist of one or more panels p) and a question \( q \), the task is to select an answer \( a \) from a set of \( k \) multiple-choice options:

\[
a^* = \arg\max_{a \in \mathcal{A}} P(a \mid I, q)
\]

where \( \mathcal{A} = \{a_1, \ldots, a_k\} \) is the answer set.

3.1.2 Logical Chain

To measure the sequential steps of the reasoning process required to reach the final answer, rather than evaluating stages in isolation, the second Logical Chain task extends reasoning across multiple subproblems, introducing dependencies between stages to form a coherent logical chain. As illustrated in the right part of Figure 2, each node represents a stage question, and edges are connected if the previous information is necessary to answer the current stage.

This task consists of five stages, similar to Direct Answer subtask: IP (Merged IP-B and IP-C), 2P, IR, 2R and Final. Specifically, each node \( t \) involves predicting an answer \( a_t \) based on the current question \( q_t \), the current image \( I_t \), and information from prior stages \( \mathbf{H}_t \), the task is defined as:

\[
a_t^* = \arg\max_{a_t \in \mathcal{A}_t} P(a_t \mid I_t, q_t, \mathbf{H}_t)
\]

\[
\mathbf{H}_t = \{\text{Re-Format}(q_j, a_j) \mid j \in \mathcal{D}_t\}
\]

where \( \mathbf{H}_t \) represents the set of prior information, as determined by the pre-defined logical chain \( \mathcal{D}_t \), specifying one or more nodes that current node \( t \) depends on. As the images referenced by prior questions change across different stages, we use a rule-based program to reformat each dependent question \( q_j \) and the generated answer \( a_j \), appending this prior information before the current question to construct the input for the current node. Details of this program are provided in Appendix A.4.2.

3.2 Dataset Creation

Data Sources: Our MultiStAR dataset is derived from the RAVEN dataset, which its associated XML files provide objects details and ground-truth logical rules for generating each puzzle.

Define Templates: We pre-define question templates for all six configurations, each template including a question format, constraints, an answer space, and a corresponding function sequence, as illustrated in Figure 3. Overall, we created 25 distinct templates, details are shown in Appendix A.2.

Question Generation: By leveraging the puzzle information and the pre-defined templates, we implement an automated template-based generation process to efficiently produce large-scale question-answer pairs. Firstly, to enrich question formats and linguistic diversity, we employ GPT-4o (OpenAI, 2024) to rewrite the templates. Then, following a methodology similar to the CLEVR dataset (Johnson et al., 2017), we design functional programs that execute a sequence of functions. For instance, as shown in Figure 3, the program “Scene Retrieve → Panel Retrieve → Filter Unique → Shape Query → Compare Shape” identifies the puzzle matrix, retrieves the relevant panel <P>, locates objects at positions <X1> and <X2>, queries their shapes, and compares them to determine the ground-truth answer. And lastly, the multiple-choice options are sampled from the answer space.

Subtasks Creation: To create the Direct Answer subtask, we first sample XML files from RAVEN, and for each XML file, we generate one question for each template. During question formation, placeholders (e.g., <X1>, <X2>) are replaced with randomly selected any possible values consistent with the value ranges and constraints. Next, we create the Logical Chain sub-task by first filtering out those templates that do not contribute useful information for building the logical chain (i.e. the question does not provide necessary input for its child nodes). To simplify chain construction, the first two one-panel configurations, IP-B and IP-C, are combined into a single stage representing one-panel information. During question generation, one question is created for each node, with placeholders such as panel <P> replaced by the values corresponding to the current node’s position in the chain. For instance, if there are three IP nodes in the chain, they correspond to panels 1, 2, and 3, respectively. Questions are then grouped by attributes such as number and position, aligning with
how the chain is constructed. Finally, we assign the previous nodes for each question to establish the edges between nodes. Detailed analysis of our dataset MultiStAR, such as the question length and function distribution, please see Appendix A.1.

Human Verification: To evaluate the quality of automatically generated question-answer pairs, we also conduct human study based on three aspects, Correctness, Clarity and Content Validity. The results show our dataset performing well across all aspects, see Appendix A.6.1 for details.

3.3 Evaluation Metrics

We use accuracy for the Direct Answer subtask, as it directly aligns with the task of selecting the correct answer from multiple choices. However, for the Logical Chain subtask, accuracy alone does not consider intermediate reasoning steps, focusing only on the end result. To address this limitation and better align with the step-by-step reasoning process, we introduce a new metric, MSEval. As the example illustrated in Figure 4, the score for the 1R node is designed to aggregate from all its related nodes, which include three 1P nodes, two 2P nodes, and the 1R node itself. This aggregation captures the interconnectivity between nodes in the logical chain. And to reflect their contribution to the reasoning process, it assigns a weight to each of these nodes based on their importance. Specifically,

Aggregated Outcomes: To aggregated the intermediate steps outcomes across the chain, MSEval chooses to compute a joint probability of the current node and all dependent nodes as the product of their conditional probabilities. To prevent disregarding the model’s performance when it is incorrect, each conditional probability is derived by measuring the probability assigned to the ground truth. Using logits from the model’s final layer for the answer choices (e.g., A, B, C, D), the correct answer logit \( p_j^{(i)} \) is transformed into a probability via the softmax function. This is defined as:

\[
\text{Joint Prob}_t^{(i)} = \text{Norm}(P(a_t^{(i)} = a_t^{(i)*}, \Phi_t^{(i)} | \xi_t^{(i)})) \\
= \prod_{j \in \mathcal{D}_t \cup \{t\}} \exp(\frac{p_j^{(i)}}{\epsilon_j^{(i)}})
\]

\[
p_j^{(i)} = P(a_j^{(i)} = a_j^{(i)*} | H_j^{(i)}) = \frac{\exp(z_{a_j^{(i)*}})}{\sum_{k \in \mathcal{A}_j^{(i)}} \exp(z_k)}
\]

where \( z_{a_j^{(i)*}} \) is the logit for the correct answer \( a_j^{(i)*} \), and \( \mathcal{A}_j^{(i)} \) is the set of all possible answers, \( \epsilon_j^{(i)} = \frac{1}{|\mathcal{A}_j^{(i)}|} \) represents the random rate. The term \( \exp(\cdot) \) normalizes the probability to account for varying numbers of choices for all nodes. \( \Phi_t^{(i)} = \{a_j^{(i)} = a_j^{(i)*} \mid j \in \mathcal{D}_t\} \) represents the correct answer probability for all dependent nodes, \( \xi_t^{(i)} = \{H_j^{(i)} \mid j \in \mathcal{D}_t \cup \{t\}\} \) represents a set of prior information for each node.

Weighted Importance: The joint probability does not account for the relative importance of each node

![Our MultiStAR dataset generation pipeline.](page_186_120_1077_393.png)

Figure 3: Our MultiStAR dataset generation pipeline.

![This example is the MSEval score calculation for the 1R node, depends on 1P, 2P and 1R itself. The corresponding weights are denoted as w_a through w_f.](page_1012_563_393_181.png)

Figure 4: This example is the MSEval score calculation for the 1R node, depends on 1P, 2P and 1R itself. The corresponding weights are denoted as \( w_a \) through \( w_f \).
in the chain. To address this, we introduce a weight for each node based on its influence on the current node, as measured by conditional mutual information (CMI). CMI is obtained by altering the set of all possible answers \( \mathcal{A}_j^{(i)} \) at node \( j \) while keeping the outputs of all other nodes (\( \mathcal{D}_t^{(i)} \setminus \{j\} \)) fixed. We then observe how the model’s outputs \( \mathcal{A}_{j \rightarrow t}^{(i)} \) at the current node \( t \) change. If \( \mathcal{A}_{j \rightarrow t}^{(i)} \) changes significantly, the CMI is higher, resulting in a higher weight. As raw CMI values may vary in scale, we take a normalization of the conditional mutual information (NCMI). This process is defined as:

\[
\text{CMI}(i, j, t) = \text{CMI}(\mathcal{A}_j^{(i)}, \mathcal{A}_{j \rightarrow t}^{(i)} | \mathcal{D}_t^{(i)} \setminus \{j\}) \\
= H(\mathcal{A}_{j \rightarrow t}^{(i)} | \mathcal{D}_t^{(i)} \setminus \{j\}) \\
+ H(\mathcal{A}_j^{(i)} | \mathcal{D}_t^{(i)} \setminus \{j\}) \\
- H(\mathcal{A}_{j \rightarrow t}^{(i)}, \mathcal{A}_j^{(i)} | \mathcal{D}_t^{(i)} \setminus \{j\})
\]

\[
\text{NCMI}(i, j, t) = \frac{\exp(\text{CMI}(i, j, t))}{\sum_{k \in \mathcal{D}_t \cup \{t\}} \exp(\text{CMI}(i, k, t)))
\]

where \( H(\cdot) \) denotes entropy. Note that for current node \( t \), we have \( \mathcal{A}_t^{(i)} = \mathcal{A}_{t \rightarrow t}^{(i)} \), so that current node would always have the highest impact to itself.

We apply the NCMI to each node’s conditional probability to compute its weighted contribution to the reasoning process. To simplify the formulation, we apply a log to the expression. The final MSEval score for stage \( t \), instance \( i \) is computed as:

\[
\text{MSEval}_t^{(i)} = \log \prod_{j \in \mathcal{D}_t \cup \{t\}} (\exp(\frac{p_j^{(i)}}{\epsilon_j^{(i)}})) \text{NCMI}(i, j, t) \\
= \sum_{j \in \mathcal{D}_t \cup \{t\}} \text{NCMI}(i, j, t) \cdot \frac{p_j^{(i)}}{\epsilon_j^{(i)}}
\]

\[
\text{MSEval}_t^{(i)} = \sum_{j \in \mathcal{D}_t \cup \{t\}} w_j^{(i)} \cdot \frac{p_j^{(i)}}{\epsilon_j^{(i)}}
\]

As MSEval relies on access to the logits of the model’s final layer, which can only be applied to open-source models. More details about MSEval, such as algorithm Pseudo Code and computational cost, are shown in Appendix A.8.

4 Results

In the Direct Answer subtask, we evaluate 17 representative MLLMs in zero-shot setting, including both open-source and close-source models. For open-source models, we have vanilla pre-trained (i.e., without instruction-tuning) and instruction-tuned models. Additionally, we consider a range of model sizes to ensure a comprehensive evaluation. The model settings and the input prompt details are shown in Appendix A.4.1.

In the Logical Chain subtask, we evaluate six models which performed well in the Direct Answer subtask in zero-shot setting. To compare how much benefit the models gain from prior information, we also evaluate the models without providing prior information (\( \mathbf{H}_t = \emptyset \)), serving as a baseline for comparison. In this case, as we only consider the current stage, the MSEval score simplifies to

\[
\text{MSEval}_t^{(i)} = \frac{p_t^{(i)}}{|\mathcal{A}_t^{(i)}|}
\]

The model settings and the input prompt details are shown in Appendix A.4.2.

To establish the human performance on the subproblems within MultiStAR, we conduct a human study on a crowd-sourcing platform in which participants solved a 10% subset of the benchmark. We do not evaluate human performance on the original RAVEN puzzles, but instead use the result reported in Zhang et al. (2019). Please see Appendix A.6.2 for more details about the human study.

4.1 Result Analysis

Direct Answer: Table 2 compares the performance of various MLLMs on the Direct Answer task. The two close-source models, GPT-4o and Gemini-1.5-pro, outperform others in 1P-B, 1R and 2R stages. GPT-4o achieves an impressive 88.07% accuracy for basic object-oriented questions within a single panel, highlighting its strong capability to recognize simple visual patterns. Gemini achieves the best results in rule deduction tasks for both one-row and two-row configurations, indicating its superior ability to process more complex visual inputs and perform logical reasoning effectively. Among open-source models, pre-trained models generally perform worse than instruction-tuned ones, which is no surprise as the pre-trained models are unlikely to be able follow the prompt/instructions. Qwen2-VL-Instruct-72B achieves the best performance on cross-panel comparison tasks. Interestingly, we also see that it has a very strong performance in the original RAVEN task (“Final”), and we suspect there is probably contamination since it struggles with the more complex intermediate reasoning tasks (“1R” and “2R”). We also think Idefics2-8B maybe be contaminated, again for the same reason.
We performed an error analysis (see Appendix A.4.2) and identified that the models mainly face challenges with deep reasoning errors. Additionally, the decoder-only models (Qwen2-VL, InternVL2 and NVLM-D) tend to perform much better than encoder-decoder models (InstructBLIP and LLaVA-v1.5). This suggests that decoder-only architectures may be more effective for step-by-step reasoning to some extent. However, it’s important to note that the decoder-only models are newer and the performance gap may be attributed to other factors such as larger or more diverse training data or better training objectives.

Interestingly, as the questions become increasingly complex and require deeper reasoning, a noticeable decline in performance is observed across all models, gradually approaching the random baseline. While models demonstrate strong performance on basic perception tasks, they struggle significantly with deeper reasoning challenges. In contrast, human performance remains stable at above 60% with increasing complexity. This highlights the substantial gap between model and human performance, emphasizing the limitations of current MLLMs in understanding and reasoning at a level comparable to humans. Another finding is that performance generally improves with larger model sizes — the size of the base language model (see Appendix A.9.1 for detailed visual analysis). This suggests that these VLLMs might be primarily relying on its language model for tackling these AVR tasks.

Logical Chain: Table 3 shows the performance of MLLMs on each stage of the Logical Chain task. Results are reported as accuracy and MSEval scores. Among all models, Gemini-1.5-pro achieves the best performance on the first four stages. Among the open-source models, Qwen2-VL-72B outperforms others in both accuracy and MSEval, although as mentioned earlier we think the model has been trained with RAVEN and we should probably interpret these results with caution (same for Idefics2-8B). When comparing results with and without prior information, all models show better performance when prior information is available in both metrics, highlighting their capacity to benefit from step-by-step reasoning, even when some generated previous answers might be incorrect. For visual representation of the percentage increase, see to Appendix A.7.

Interestingly, for the final stage involving the

<table>
  <tr>
    <th></th>
    <th>IP-B</th>
    <th>IP-C</th>
    <th>2P</th>
    <th>TR</th>
    <th>2R</th>
    <th>Final</th>
  </tr>
  <tr>
    <th colspan="7">Close-Source Models</th>
  </tr>
  <tr>
    <td>GPT-4o (2024)</td>
    <td>88.1</td>
    <td>72.7</td>
    <td>54.0</td>
    <td>40.0</td>
    <td>31.6</td>
    <td>12.1</td>
  </tr>
  <tr>
    <td>Gemini-1.5-pro (2023)</td>
    <td>83.2</td>
    <td>75.0</td>
    <td>50.0</td>
    <td>46.9</td>
    <td>37.8</td>
    <td>11.6</td>
  </tr>
  <tr>
    <th colspan="7">Pre-trained Open-Source Models</th>
  </tr>
  <tr>
    <td>Qwen-VL-7B (2023)</td>
    <td>17.5</td>
    <td>24.7</td>
    <td>22.5</td>
    <td>15.8</td>
    <td>12.2</td>
    <td>12.3</td>
  </tr>
  <tr>
    <td>Idefics2-8B (2024)</td>
    <td>17.2</td>
    <td>33.0</td>
    <td>27.3</td>
    <td>19.8</td>
    <td>21.4</td>
    <td>12.3</td>
  </tr>
  <tr>
    <td>xGen-MM-4B (2024)</td>
    <td>40.2</td>
    <td>31.8</td>
    <td>12.5</td>
    <td>24.1</td>
    <td>23.9</td>
    <td>3.4</td>
  </tr>
  <tr>
    <th colspan="7">Instruction-Tuned Open-Source Models</th>
  </tr>
  <tr>
    <td>Instructblip-7B (2023)</td>
    <td>27.5</td>
    <td>37.1</td>
    <td>27.7</td>
    <td>14.0</td>
    <td>13.5</td>
    <td>11.6</td>
  </tr>
  <tr>
    <td>Instructblip-13B (2023)</td>
    <td>29.4</td>
    <td>39.0</td>
    <td>26.9</td>
    <td>25.0</td>
    <td>23.0</td>
    <td>14.3</td>
  </tr>
  <tr>
    <td>LLaVA-v1.5-7B (2024)</td>
    <td>47.8</td>
    <td>50.2</td>
    <td>32.9</td>
    <td>27.1</td>
    <td>25.7</td>
    <td>13.3</td>
  </tr>
  <tr>
    <td>LLaVA-v1.5-13B (2024)</td>
    <td>59.6</td>
    <td>47.6</td>
    <td>15.9</td>
    <td>26.7</td>
    <td>26.9</td>
    <td>11.3</td>
  </tr>
  <tr>
    <td>Idefics2-8B* (2024)</td>
    <td>85.1</td>
    <td>65.5</td>
    <td>42.0</td>
    <td>37.0</td>
    <td>36.8</td>
    <td>29.9</td>
  </tr>
  <tr>
    <td>xGen-MM-4B (2024)</td>
    <td>81.2</td>
    <td>47.9</td>
    <td>21.9</td>
    <td>24.0</td>
    <td>25.6</td>
    <td>2.4</td>
  </tr>
  <tr>
    <td>Qwen2-VL-2B (2024)</td>
    <td>40.4</td>
    <td>42.7</td>
    <td>22.8</td>
    <td>13.7</td>
    <td>11.4</td>
    <td>9.9</td>
  </tr>
  <tr>
    <td>Qwen2-VL-7B* (2024)</td>
    <td>64.8</td>
    <td>56.6</td>
    <td>47.9</td>
    <td>31.0</td>
    <td>33.2</td>
    <td>24.3</td>
  </tr>
  <tr>
    <td>Qwen2-VL-72B* (2024)</td>
    <td>86.9</td>
    <td>77.8</td>
    <td>60.2</td>
    <td>45.5</td>
    <td>21.9</td>
    <td>63.7</td>
  </tr>
  <tr>
    <td>NVLM-D-72B (2024)</td>
    <td>80.5</td>
    <td>67.1</td>
    <td>45.3</td>
    <td>39.1</td>
    <td>31.3</td>
    <td>12.7</td>
  </tr>
  <tr>
    <td>Intern-VL2-2B (2024)</td>
    <td>54.0</td>
    <td>48.7</td>
    <td>27.3</td>
    <td>26.9</td>
    <td>23.9</td>
    <td>10.1</td>
  </tr>
  <tr>
    <td>Intern-VL2-8B (2024)</td>
    <td>63.0</td>
    <td>54.5</td>
    <td>34.2</td>
    <td>23.2</td>
    <td>23.4</td>
    <td>14.6</td>
  </tr>
  <tr>
    <th>Random</th>
    <td>39.9</td>
    <td>23.0</td>
    <td>26.2</td>
    <td>25.0</td>
    <td>25.0</td>
    <td>12.5</td>
  </tr>
  <tr>
    <th>Human</th>
    <td>98.5</td>
    <td>88.9</td>
    <td>69.1</td>
    <td>62.1</td>
    <td>63.3</td>
    <td>84.4<sup>†</sup></td>
  </tr>
</table>

Table 2: The answer accuracy of MLLMs for the Direct Answer subtask. The best results are highlighted in bold. * denotes contaminated models that may have been trained with RAVEN. † means the number is taken from from Zhang et al. (2019).

RAVEN puzzle, prior information appears to provide limited utility for most models (exception: Gemini). This aligns with previous findings that chain-of-thought reasoning models struggle to solve RAVEN puzzles (Ahrabian et al., 2024; Genron et al., 2024). However, MSEval scores tell a different story. Almost all models (exception: Qwen2-VL-72B) see an improvement in MSEval even though there is no change (or even a drop) in accuracy when prior information is incorporated. Since MSEval evaluates correctness across intermediate and current stages, it reveals that even though prior information might not help MLLMs solve the final stage question, it helps them with the intermediate tasks.

Another finding is the MSEval score for Qwen2-VL-72B declines when provided with prior information, which is not consistent with accuracy. This indicates its weakness in addressing intermediate stages that were likely not part of its pre-training. Despite errors in earlier stages, the model still performs well on the final question, suggesting it relies on memorizing patterns from the final stage rather than demonstrating a strong understanding of the logical reasoning behind the task. This highlights a critical limitation in current MLLMs, while they may achieve impressive results in isolated cases, their ability to generalize and reason through
<table>
  <tr>
    <th>Metric</th>
    <th>Prior</th>
    <th>IP</th>
    <th>2P</th>
    <th>1R</th>
    <th>2R</th>
    <th>Final</th>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>w/o</td>
    <td>73.8</td>
    <td>39.1</td>
    <td>34.7</td>
    <td>28.9</td>
    <td>15.7</td>
  </tr>
  <tr>
    <td></td>
    <td>w</td>
    <td>73.8</td>
    <td>43.9</td>
    <td>41.8</td>
    <td>50.6</td>
    <td>10.0</td>
  </tr>
  <tr>
    <td>Gemini-1.5-pro</td>
    <td>w/o</td>
    <td>75.5</td>
    <td>61.6</td>
    <td>49.6</td>
    <td>44.6</td>
    <td>5.7</td>
  </tr>
  <tr>
    <td></td>
    <td>w</td>
    <td><b>75.5</b></td>
    <td><b>64.4</b></td>
    <td><b>52.6</b></td>
    <td><b>57.1</b></td>
    <td>18.6</td>
  </tr>
  <tr>
    <td>Idefics2-8B*</td>
    <td>w/o</td>
    <td>57.8</td>
    <td>39.6</td>
    <td>34.4</td>
    <td>35.1</td>
    <td>25.7</td>
  </tr>
  <tr>
    <td></td>
    <td>w</td>
    <td>57.8</td>
    <td>37.8</td>
    <td>36.6</td>
    <td>42.4</td>
    <td>25.7</td>
  </tr>
  <tr>
    <td>MSEval</td>
    <td>w/o</td>
    <td>2.02</td>
    <td>1.29</td>
    <td>1.29</td>
    <td>1.27</td>
    <td>1.24</td>
  </tr>
  <tr>
    <td></td>
    <td>w</td>
    <td>2.02</td>
    <td>1.48</td>
    <td>1.51</td>
    <td>1.51</td>
    <td>1.44</td>
  </tr>
  <tr>
    <td>Qwen2-VL-72B*</td>
    <td>w/o</td>
    <td>74.1</td>
    <td>56.0</td>
    <td>45.1</td>
    <td>42.9</td>
    <td>64.3</td>
  </tr>
  <tr>
    <td></td>
    <td>w</td>
    <td>74.1</td>
    <td>57.8</td>
    <td>47.3</td>
    <td>54.2</td>
    <td><b>65.7</b></td>
  </tr>
  <tr>
    <td>MSEval</td>
    <td>w/o</td>
    <td>2.54</td>
    <td>1.95</td>
    <td>1.79</td>
    <td>1.70</td>
    <td>5.14</td>
  </tr>
  <tr>
    <td></td>
    <td>w</td>
    <td>2.54</td>
    <td>2.13</td>
    <td>2.12</td>
    <td>2.10</td>
    <td>3.31</td>
  </tr>
  <tr>
    <td>Intern-VL-2-8B</td>
    <td>w/o</td>
    <td>54.4</td>
    <td>35.9</td>
    <td>21.6</td>
    <td>21.6</td>
    <td>18.6</td>
  </tr>
  <tr>
    <td></td>
    <td>w</td>
    <td>54.4</td>
    <td>41.9</td>
    <td>31.6</td>
    <td>33.5</td>
    <td>17.1</td>
  </tr>
  <tr>
    <td>MSEval</td>
    <td>w/o</td>
    <td>1.75</td>
    <td>1.09</td>
    <td>0.90</td>
    <td>0.91</td>
    <td>1.18</td>
  </tr>
  <tr>
    <td></td>
    <td>w</td>
    <td>1.75</td>
    <td>1.38</td>
    <td>1.30</td>
    <td>1.18</td>
    <td>1.26</td>
  </tr>
  <tr>
    <td>NVLM-D-72B</td>
    <td>w/o</td>
    <td>66.1</td>
    <td>42.4</td>
    <td>37.8</td>
    <td>23.5</td>
    <td>8.6</td>
  </tr>
  <tr>
    <td></td>
    <td>w</td>
    <td>66.1</td>
    <td>45.2</td>
    <td>39.1</td>
    <td>43.3</td>
    <td>7.1</td>
  </tr>
  <tr>
    <td>MSEval</td>
    <td>w/o</td>
    <td>2.25</td>
    <td>1.20</td>
    <td>1.28</td>
    <td>1.02</td>
    <td>0.76</td>
  </tr>
  <tr>
    <td></td>
    <td>w</td>
    <td>2.25</td>
    <td>1.65</td>
    <td>1.69</td>
    <td>1.62</td>
    <td>1.41</td>
  </tr>
  <tr>
    <td>Random</td>
    <td>-</td>
    <td>31.1</td>
    <td>31.7</td>
    <td>25.0</td>
    <td>25.0</td>
    <td>12.5</td>
  </tr>
  <tr>
    <td>MSEval</td>
    <td>-</td>
    <td>1.00</td>
    <td>1.00</td>
    <td>1.00</td>
    <td>1.00</td>
    <td>1.00</td>
  </tr>
</table>

Table 3: The Accuracy (Acc) and MSEval scores for the Logical Chain task. * denotes contaminated models that may have been trained with RAVEN. The highest accuracy are highlighted in bold. The highest MSEval are highlighted in underline. w/o: without prior, w: with prior. See Appendix A.4.2 for results of more models.

multi-stage logical dependencies remains inadequate. To further verify the MSEval’s effectiveness, we also conduct qualitative analysis in section 6.

5 Discussion

What insights can be drawn from each attribute’s performance? From Figure 5, the “number” attribute is the easiest to recognize in higher level configurations (2P, 1R, 2R), while “position” is the most easily identified in low-level, single-panel settings. Some attributes achieve accuracy above 90%, indicating that the models exhibit strong counting and spatial reasoning capabilities. However, they struggle with attributes like “color” and “size”, particularly in high-level configurations.

![Breakdown analysis of five attributes for Gemeni-1.5-pro and Qwen2-VL-72B on the Direct Answer task.](page_1012_120_388_312.png)

Figure 5: Breakdown analysis of five attributes for Gemeni-1.5-pro and Qwen2-VL-72B on the Direct Answer task. See Appendix A.9.2 for more models.

Given the ground truth (i.e., correct) answer for intermediate steps, how does it influence the final results? Table 4 highlights that the correctness of the prior information is important. For example, the 1R stage benefits significantly from the insights about each panel and intra-panel comparisons. The 2R stage also sees substantial gains, as it mainly relies on double-checking information from the 1R stage without requiring additional changes in most cases. However, the final stage experiences a negative impact despite the inclusion of correct rules. This may be attributed to the complexity of the visual input, which contains numerous objects, making it challenging for the model to effectively apply the given rules. And for the Qwen2-VL-72B model, its tendency to memorize patterns might turn these ground truths into noise.

<table>
  <tr>
    <th>Prior Info</th>
    <th>IP</th>
    <th>2P</th>
    <th>1R</th>
    <th>2R</th>
    <th>Final</th>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>w/o</td>
    <td>73.8</td>
    <td>39.1</td>
    <td>34.7</td>
    <td>28.9</td>
    <td>15.7</td>
  </tr>
  <tr>
    <td></td>
    <td>GT</td>
    <td>73.8</td>
    <td>49.2</td>
    <td>66.0</td>
    <td>93.8</td>
    <td>14.3</td>
  </tr>
  <tr>
    <td>Gemini-1.5-pro</td>
    <td>w/o</td>
    <td>75.5</td>
    <td>61.6</td>
    <td>49.6</td>
    <td>44.6</td>
    <td>5.7</td>
  </tr>
  <tr>
    <td></td>
    <td>GT</td>
    <td><b>75.5</b></td>
    <td><b>62.4</b></td>
    <td><b>68.9</b></td>
    <td><b>79.5</b></td>
    <td>7.1</td>
  </tr>
  <tr>
    <td>Qwen2-VL-72B*</td>
    <td>w/o</td>
    <td>74.1</td>
    <td>56.0</td>
    <td>45.1</td>
    <td>42.9</td>
    <td>64.3</td>
  </tr>
  <tr>
    <td></td>
    <td>GT</td>
    <td>74.1</td>
    <td>61.9</td>
    <td><b>70.7</b></td>
    <td><b>92.2</b></td>
    <td><b>55.7</b></td>
  </tr>
  <tr>
    <td>NVLM-D-72B</td>
    <td>w/o</td>
    <td>66.1</td>
    <td>42.4</td>
    <td>37.8</td>
    <td>23.5</td>
    <td>8.6</td>
  </tr>
  <tr>
    <td></td>
    <td>GT</td>
    <td>66.1</td>
    <td>51.9</td>
    <td>66.4</td>
    <td><b>96.0</b></td>
    <td>7.14</td>
  </tr>
</table>

Table 4: The accuracy with incorporating ground truth information at each stage for Logical Chain task.

How much do previous stages influence the current stage? A key step in our MSEval metrics is measuring the relative importance of intermediate dependent stages to the current stage using NCMI. This allows us to assess how each step in the chain depends on prior stages, helping verify whether previous information is useful and if the designed chain is logically sound. As shown in Table 5, prior information often has significant weight on the current stage, except for the position attribute at “1P to 2P”. This suggests that querying object position in a single panel has little impact on determining if positions are the same across panels.

How do variations in handling long prompts affect model outcomes? Injecting prior information into prompts significantly increases their
Figure 6: The top two rows are dependent stages (All Correct), the bottom row is current stage (Incorrect).

<table>
  <tr>
    <th>Attributes</th>
    <th>1P to 2P</th>
    <th>(1P2P) to 1R</th>
    <th>1R to 2R</th>
    <th>2R to F</th>
  </tr>
  <tr>
    <td rowspan="5">Qwen2-VL-72B</td>
    <td>Number</td>
    <td>0.40</td>
    <td>0.54</td>
    <td>0.33</td>
  </tr>
  <tr>
    <td>Position</td>
    <td>0.21</td>
    <td>0.48</td>
    <td>0.39</td>
  </tr>
  <tr>
    <td>Shape</td>
    <td>0.37</td>
    <td>0.51</td>
    <td>0.27</td>
    <td>0.42</td>
  </tr>
  <tr>
    <td>Color</td>
    <td>0.36</td>
    <td>0.53</td>
    <td>0.33</td>
  </tr>
  <tr>
    <td>Size</td>
    <td>0.37</td>
    <td>0.51</td>
    <td>0.32</td>
  </tr>
  <tr>
    <td rowspan="5">NVLM-D-72B</td>
    <td>Number</td>
    <td>0.40</td>
    <td>0.56</td>
    <td>0.37</td>
  </tr>
  <tr>
    <td>Position</td>
    <td>0.21</td>
    <td>0.50</td>
    <td>0.42</td>
  </tr>
  <tr>
    <td>Shape</td>
    <td>0.40</td>
    <td>0.52</td>
    <td>0.35</td>
    <td>0.63</td>
  </tr>
  <tr>
    <td>Color</td>
    <td>0.38</td>
    <td>0.54</td>
    <td>0.38</td>
  </tr>
  <tr>
    <td>Size</td>
    <td>0.39</td>
    <td>0.54</td>
    <td>0.38</td>
  </tr>
</table>

Table 5: Average NCMI weight assigned to all dependent stages, grouped by each attribute.

length (see Appendix A.1 for details), making it more challenging for models to focus on critical details. To address this issue, we proposed two methods: (1) adding HTML tags to structure the prompt by separating prior information, background, and questions, enabling the model to clearly distinguish each part, and (2) formatting the prompt as a PDF document with distinct sections and titles. Table 6 demonstrates that HTML tagging provides some benefits, but not so much for the PDF approach (see Appendix A.9.3 for further details and examples of the conversion methods).

<table>
  <tr>
    <th></th>
    <th>Prior</th>
    <th>1P</th>
    <th>2P</th>
    <th>1R</th>
    <th>2R</th>
    <th>Final</th>
  </tr>
  <tr>
    <td rowspan="3">GPT-4o</td>
    <td>Vanilla</td>
    <td>73.8</td>
    <td>43.9</td>
    <td>41.8</td>
    <td>50.6</td>
    <td>10.0</td>
  </tr>
  <tr>
    <td>Struct.</td>
    <td><b>82.2</b></td>
    <td>64.4</td>
    <td>47.8</td>
    <td>50.9</td>
    <td>8.6</td>
  </tr>
  <tr>
    <td>Doc.</td>
    <td>80.8</td>
    <td>44.8</td>
    <td>31.1</td>
    <td>24.9</td>
    <td>10.0</td>
  </tr>
  <tr>
    <td rowspan="3">Gemini-1.5-pro</td>
    <td>Vanilla</td>
    <td>75.5</td>
    <td>64.4</td>
    <td>52.6</td>
    <td>57.1</td>
    <td>18.6</td>
  </tr>
  <tr>
    <td>Struct.</td>
    <td>70.6</td>
    <td>66.4</td>
    <td>52.9</td>
    <td><b>57.8</b></td>
    <td>17.1</td>
  </tr>
  <tr>
    <td>Doc.</td>
    <td>69.6</td>
    <td>51.0</td>
    <td>36.7</td>
    <td>33.1</td>
    <td>14.3</td>
  </tr>
  <tr>
    <td rowspan="3">Qwen2-VL-72B*</td>
    <td>Vanilla</td>
    <td>74.1</td>
    <td>57.8</td>
    <td>47.3</td>
    <td>54.2</td>
    <td>65.7</td>
  </tr>
  <tr>
    <td>Struct.</td>
    <td><b>77.2</b></td>
    <td><b>67.7</b></td>
    <td><b>55.1</b></td>
    <td>53.6</td>
    <td>61.4</td>
  </tr>
  <tr>
    <td>Doc.</td>
    <td>76.5</td>
    <td>63.1</td>
    <td>50.2</td>
    <td>46.6</td>
    <td>24.3</td>
  </tr>
</table>

Table 6: The accuracy of three prompting techniques for prior information for Logical Chain task. Vanilla: Pure Text, Struct.: Structure (HTML), Doc.: Document.

6 Qualitative Analysis

To highlight the advantages of our MSEval metric over traditional accuracy, we provide several concrete examples across different scenarios. Figure 6 shows a case where the final answer to the current question is incorrect, resulting in an accuracy score of 0.0. However, the model demonstrates strong performance in intermediate steps, correctly solving the one-panel and two-panel comparisons with high confidence. Additionally, the logits for the correct answer "A" are only slightly lower than the highest logits. By considering these factors, the MSEval metric produces a more reasonable score, reflecting the model's partial success. Further examples, including cases where the current question is correct but the intermediate steps are not, are provided in Appendix A.10.

7 Conclusion

In this work, we propose the MultiStAR benchmark, to evaluate MLLMs on the intermediate steps in the reasoning process through two tasks, Direct Answer and Logical Chain. While current models perform well on basic perception tasks, they face significant challenges with deeper reasoning stages. Our findings also indicate that models have the potential to benefit from step-by-step reasoning. However, despite extensive training yielding impressive results in isolated scenarios, their ability to handle logical dependencies remains limited. We introduce a metric MSEval, which can be applied to a variety of reasoning tasks beyond visual reasoning, including domains such as mathematics and science, where multi-step logic is critical, provided there are clearly defined chains.
8 Limitation

The intermediate task generation method we propose is restricted to datasets with explicitly defined object attributes (in our case, they are documented in the XML files in RAVEN). This limits its applicability to other AVR datasets, as they often lack such metadata.

The logical chain design in our dataset is not perfect. In some cases, prior information is insufficient for the current stage, such as instances in the one-row rule deduction stage where the rule might involve “Three Different Numbers”, in this case, we also need the information about the second row. To make the chain construction easier, we design chains at the “Corpus-Level”, meaning they are fixed across all instances. Future work could explore automatic “Instance-Level” chain construction methods to enable models to dynamically generate chains based on patterns within individual examples.

Acknowledgments

This project is supported by the Australian Research Council Discovery Grant awarded to Jey Han Lau (ID: DP240101006).

References

Kian Aharian, Zhivar Sourati, Kexuan Sun, Jiarui Zhang, Yifan Jiang, Fred Morstatter, and Jay Pujara. 2024. The curious case of nonverbal abstract reasoning with multi-modal large language models. In Proceedings of Thirty Seventh Conference on Learning Theory.

Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966.

Xu Cao, Bolin Lai, Wenqian Ye, Yunsheng Ma, Jörg Heintz, Jintai Chen, Jianguo Cao, and James M Rehg. 2024. What is the visual cognition gap between humans and multimodal llms? arXiv preprint arXiv:2406.10424.

Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. 2024. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. arXiv preprint arXiv:2404.16821.

François Chollet. 2019. On the measure of intelligence. arXiv preprint arXiv:1911.01547.

Wenliang Dai, Nayeon Lee, Boxin Wang, Zhuolin Yang, Zihan Liu, Jon Barker, Tuomas Rintamaki, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. 2024. Nvlm: Open frontier-class multimodal llms. arXiv preprint arXiv:2409.11402.

Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. 2023. Instructclip: Towards general-purpose vision-language models with instruction tuning. Preprint, arXiv:2305.06500.

Jiajun Deng, Zhengyuany Yang, Tianlang Chen, Wengang Zhou, and Houqiang Li. 2021. Transvg: End-to-end visual grounding with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1769–1779.

Yihao Ding, Siwen Luo, Hyunsuk Chung, and Soyeon Caren Han. 2023. Vqa: A new dataset for real-world vqa on pdf documents. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 585–601. Springer.

Difei Gao, Ruiping Wang, Shiguang Shan, and Xilin Chen. 2022. Cric: A vqa dataset for compositional reasoning on vision and commonsense. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(5):5561–5578.

Gael Gendron, Qiming Bao, Michael Witbrock, and Gillian Dobbie. 2024. Large language models are not strong abstract reasoners yet. In ICLR 2024 Workshop: How Far Are We From AGI.

Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6904–6913.

Ruozhen He, Paola Cascante-Bonilla, Ziyan Yang, Alexander C Berg, and Vicente Ordonez. 2024. Improved visual grounding through self-consistent explanations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13095–13105.

Tuomo Hiippala, Malihe Alikhani, Jonas Haverinen, Timo Kallikoski, Evgeniya Logacheva, Serafina Orekhova, Aino Tuomainen, Matthew Stone, and John A Bateman. 2021. Ai2d-rst: A multimodal corpus of 1000 primary school science diagrams. Language Resources and Evaluation, 55:661–688.

Yanbei Jiang, Krista A Ehinger, and Jey Han Lau. 2024a. Kale: An artwork image captioning system augmented with heterogeneous graph. arXiv preprint arXiv:2409.10921.

Yifan Jiang, Jiarui Zhang, Kexuan Sun, Zhivar Sourati, Kian Aharian, Kaixin Ma, Filip Ilievski, and Jay Pujara. 2024b. Marvel: Multidimensional abstraction and reasoning through visual evaluation and learning. arXiv preprint arXiv:2404.13591.
Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. 2017. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2901–2910.

J Richard Landis and Gary G Koch. 1977. The measurement of observer agreement for categorical data. biometrics, pages 159–174.

Hugo Laurençon, Léo Tronchon, Matthieu Cord, and Victor Sanh. 2024. What matters when building vision-language models? arXiv preprint arXiv:2405.02246.

Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. 2024. Seed-bench: Benchmarking multimodal large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13299–13308.

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2024. Visual instruction tuning. Advances in neural information processing systems, 36.

Jacek Mańdziuk and Adam Żychowski. 2019. Deepiq: A human-inspired ai system for solving iq test problems. In 2019 International Joint Conference on Neural Networks (IJCNN), pages 1–8. IEEE.

Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. 2019. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEE/cvf conference on computer vision and pattern recognition, pages 3195–3204.

Arsenii Kirillovich Moskvichev, Victor Vikram Odouard, and Melanie Mitchell. 2023. The conceptarc benchmark: Evaluating understanding and generalization in the arc domain. Transactions on machine learning research.

Weili Nie, Zhiding Yu, Lei Mao, Ankit B Patel, Yuke Zhu, and Anima Anandkumar. 2020. Bongard-logo: A new benchmark for human-level concept learning and reasoning. Advances in Neural Information Processing Systems, 33:16468–16480.

OpenAI. 2024. https://openai.com/index/hello-gpt-40/.

Jean Raven. 2003. Raven progressive matrices. In Handbook of nonverbal assessment, pages 223–237. Springer.

Tanik Saikh, Tirthankar Ghosal, Amish Mittal, Asif Ekbal, and Pushpak Bhattacharyya. 2022. Scienceqa: A novel resource for question answering on scholarly articles. International Journal on Digital Libraries, 23(3):289–301.

Kuniaki Saito, Kihyuk Sohn, Xiang Zhang, Chun-Liang Li, Chen-Yu Lee, Kate Saenko, and Tomas Pfister. 2023. Pic2word: Mapping pictures to words for zero-shot composed image retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19305–19314.

Adam Santoro, Felix Hill, David Barrett, Ari Morcos, and Timothy Lillicrap. 2018. Measuring abstract reasoning in neural networks. In International conference on machine learning, pages 4477–4486.

Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805.

Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. 2015. Show and tell: A neural image caption generator. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3156–3164.

Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhi-hao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. 2024. Qwen-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191.

Le Xue, Manli Shu, Anas Awadalla, Jun Wang, An Yan, Senthil Purushwalkam, Honglu Zhou, Viraj Prabhu, Yutong Dai, Michael S Ryoo, et al. 2024. xgen-mm (blip-3): A family of open large multimodal models. arXiv preprint arXiv:2408.08872.

Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruqiu Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. 2024. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9556–9567.

Chi Zhang, Feng Gao, Baoxiong Jia, Yixin Zhu, and Song-Chun Zhu. 2019. Raven: A dataset for relational and analogical visual reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5317–5327.

Yizhe Zhang, He Bai, Ruixiang Zhang, Jiatao Gu, Shuangfei Zhai, Josh Susskind, and Navdeep Jaitly. 2024. How far are we from intelligent visual deductive reasoning? arXiv preprint arXiv:2403.04732.
A Appendix

A.1 Dataset Analysis

Figure 7a compares question lengths across different reasoning VQA datasets. Our dataset stands out with a roughly even distribution of question lengths, unlike other datasets that predominantly focus on shorter questions, which makes our dataset more challenging for MLLMs. Figure 7b illustrates the proportion of functional programs used in the dataset, showing a wide variety of functions, with query_rule being slightly more frequent. Figure 7c highlights the number of multiple-choice options for each configuration, where differences in the number of choices arise due to constraints in the answer space for some configurations. Figure 7d presents the input prompt length for each stage in the logical chain task, comparing settings with and without prior information. Incorporating prior information from earlier stages significantly increases the maximum prompt length to 261.2 tokens, posing a challenge for MLLMs to parse effectively.

A.2 Generation Template

Table 7 outlines the question templates used for the Direct Answer Task. Notably, it is impractical to ask questions about the color or size of a single object, as these attributes are represented by numerical values (e.g., color as 255 or size as 8), which models cannot interpret meaningfully. Therefore, questions about size and color are excluded from the basic one-panel tasks. Instead, comparative questions such as "darker" or "smaller" are included in the one-panel comparison tasks. Two constraints are applied during template creation: Not_Equal(P, P2) ensures that Panels P and P2 are different, and Same_Row(P, P2) ensures that Panels P and P2 belong to the same row. For answer spaces exceeding four options, a sampling method is used to limit the choices to a maximum of four. Table 8 lists all possible values that each placeholder can take, as well as the complete set of rules for each attribute in the rule deduction configuration. After the placeholder values are assigned, references to "Panel <P>" are replaced with "this panel" to enhance clarity and readability.

A.3 The Full Logical Chain

Figure 8 presents the full view of our pre-defined logical chain, while Table 9 provides corresponding question examples for each node in the chain. The chain’s ultimate objective is to solve the original RAVEN puzzle, where each row contains rules for attributes such as number, position, shape, size, and color. Intuitively, we connect each attribute’s rule deduction phase to the final phase, operating under the assumption that knowing all hidden rules of the RAVEN puzzle provides sufficient information to solve it.

To determine the rules, we expand the reasoning scope from one row to two rows. For one-row rule deduction, we link single-panel perception and two-panel comparison to ensure that with panel-level details and inter-panel comparisons, the rule can be identified. This logical chain is crafted to mimic human problem-solving behavior: focusing first on single-panel perception, followed by panel comparisons, then deducing the first-row rule and validating it with the second row.

However, this handcrafted chain design has limitations. First, it may not align with the model’s actual reasoning process, which can cause discrepancies in performance. Additionally, to simplify chain construction, we designed it at the "Corpus-Level," meaning it remains fixed across all instances. This approach sometimes results in insufficient prior information for certain stages. For example, in one-row rule deduction, a rule like "the number of objects distributes three distinct values across panels, rotating through each possible permutation" may require second-row information to resolve. These limitations highlight the need for more flexible and instance-specific logical chain designs in future work.

A.4 Input Prompt And Model Settings

A.4.1 Direct Answer

Model Settings: All MLLMs are tested under their default settings under the environment of Huggingface ¹, the transformer package version in python is 4.39.2 for NVLM-D-72B model and 4.46.0 for all others.

Prompt Details: The RAVEN dataset includes various puzzle settings, such as Left-Right, Up-Down, and In-Out, where rules are applied separately to distinct parts of the panels (Figure 9). To address these settings, when we decompose the problem into subproblems, we treat each part independently. For instance, there are separate question sets for the left and right sections of the panels in the Left-Right setting, with the question explicitly

¹ https://huggingface.co/
(a) Question Length Compare    (b) Function Distribution    (c) Num. Choice Distribution    (d) Prompt Length

Figure 7: The left three panels (a), (b), and (c) present analyses of the Direct Answer task, while the right panel (d) focuses on the Logical Chain task.

![The full logical chain: To arrive at the final answer, we incorporate rules from all five attributes.](page_184_352_1082_180.png)

Figure 8: The full logical chain: To arrive at the final answer, we incorporate rules from all five attributes.

stating which part is being addressed. To clarify the panel structure, an additional sentence is appended to the question:

• Left-Right: The panel is divided into two sections by a vertical line, separating the left side from the right side, with objects possibly present in both sections.

• Up-Down: The input panel is split by a horizontal line, separating the top side from the bottom side, with objects possibly present in both sections.

• In-Out: The panel is divided into two regions: an outer structure and an inner structure, with objects possibly present in both regions.

This extra information is unnecessary for other settings. The complete prompt format is: [Extra Setting Info] Question: [question] Please select one of the following: [choices]. The answer should be one of A, B, C, D.

![The original RAVEN puzzle, includes seven puzzle settings.](page_184_1092_1082_180.png)

Figure 9: The original RAVEN puzzle, includes seven puzzle settings.

A.4.2 Logical Chain
Model Settings: All MLLMs are tested under their default settings under the environment of Huggingface 2, the transformer package version in python is 4.39.2 for NVLM-D-72B model and 4.46.0 for all others. In addition, to handle the length of our prompts, we increase the maximum token length to 2048.

When prior information is injected, a rule-based program is used to convert the information into text and integrate it into the prompt. This transformation is necessary because the images referenced in prior questions are not the same as those in the current question, making it impossible to directly reuse them.

For example, if the prior question is "How many objects in this panel?" and the current question is "Comparing the number of objects in the left and right panel," the phrase "this panel" cannot directly correspond to "left panel" or "right panel." To address this, we transform "this panel" into a more specific term, such as "left panel" or "right panel." Table 11 outlines the transfer rules for Number and Position. Similar patterns are applied for other attributes, which are not listed here for brevity.

After the prior information is transformed, the prompt is structured as follows: [Extra Setting Info] Below is the information generated from the previous steps, please be aware that it may or may not contain errors: [[Prior Info 1], [Prior Info 2], ...] Question: [question] Please select one of the following: [choices]. The answer should be one of A, B, C, D.

More results: Table 10 shows the results three additional models InstructBlip-13B, LLaVA-1.5-13B, and xgen-mm. Prior information appears to provide limited utility for these models, all of them

2https://huggingface.co/
are just around random baselines except xgen-mm have relative good basic perception ability.

A.5 Error Analysis

![Bar chart showing errors distribution for each model under the settings of without and with prior.](page_184_328_670_246.png)

Figure 10: Errors distribution for each model under the settings of without and with prior.

A.5.1 Errors from Explanation

To further investigate model performance, we conduct an error analysis for the Logical Chain subtask. Models are asked to generate with explanations alongside answers, and we manually reviewed all output explanations when the models predict incorrect answer. Errors are classified into four types:

• Perception Error: This occurs when the model misinterprets visual inputs, such as object numbers or shapes. In the provided example, there should be four objects in the left panel and three in the right panel, but the model fails to recognize this correctly.

• Reasoning Error: This involves incorrect logic applied to correctly perceived inputs. In this example, the model accurately identifies the objects and their edge numbers in each panel. However, due to flawed reasoning, it incorrectly concludes that the number of edges is decreasing, which is not the case.

• Unrelated Information Error: This error type refers to the generation of incomplete sentences or unrelated information. Here, the question asks about the position of objects, but the explanation provided by the model focuses on shapes and sizes, which are irrelevant.

• Propagation Error: This occurs when the model fails to detect or correct inaccuracies in prior information. In this example, the prior information is already incorrect, but the model does not identify or address these inaccuracies, leading to an incorrect answer.

Figure 10 reveals that reasoning errors are the most common, followed by perception errors, with unrelated-information and propagation errors being rare. Gemini exhibits the lowest perception error rate, while GPT-4o shows the lowest reasoning error rate. Notably, injecting prior information significantly reduces perception errors, demonstrating that prior knowledge enhances models’ understanding of visual inputs, but it does not help with reasoning error.

A.5.2 Unanswerable Question

We conducted a small experiment by making all the answer choices incorrect, this can be used to verify the models robustness. In this experiment, we defined hallucination as the model’s failure to recognize that the question is "non-answerable". We defined two approaches for setting unanswerable questions:

• Same Attribute: All choices are incorrect while retaining the same attribute as the question.

• Changed Attribute: The choices are changed to a different attribute (e.g., the question asks about the number of objects, but the choices are their positions).

We sampled 10 examples for each stage (60 in total) and tested them on GPT-4o, manually reviewing the output explanations. As shown in Table 12, interestingly, as the questions become more difficult, particularly at the final step, the model increasingly fails to distinguish unanswerable questions, resulting in higher rates of hallucination. Furthermore, under Setting 2, where the attribute is changed, the model exhibits a greater likelihood of hallucination, as it struggles to recognize the shift in attributes.

A.6 Human Studies and and Inter-Participant Agreement

To evaluate the subjective quality of human performance in our study, we conducted two separate parts: Part A and Part B. Part A focuses on evaluation of the quality of our automatically generated dataset, while Part B focuses on testing human reasoning abilities across different stages of complexity. For both Part A and Part B, a Consent Form and a Plain Language Statement are provided to the annotators prior to the annotation process. These documents must be read and agreed upon before they can proceed with the annotations.
A.6.1 Part A
Part A involved five research students who participated in answering a series of abstract reasoning questions. This section aimed to evaluate the quality of the dataset generated by our template-based methods. Since the Direct Answer and Logical Chain share the same pool of templates, and Direct Answer covers all templates, we chose to focus on assessing the quality of the Direct Answer component. A random sample of 620 questions was selected for this evaluation. To ensure participants clearly understood the tasks and evaluation criteria, a detailed guide was provided at the beginning of the questionnaire (see Figure 11). An example question from the questionnaire is shown in Figure 12.

![Background Information:
You are tasked with reviewing a series of question-answer (QA) pairs related to an abstract visual reasoning dataset. This dataset has been designed to test reasoning abilities across different stages of complexity. The stages range from simple, single-panel reasoning to more complex comparisons between panels.

Dataset Stages for current task A:
Basic Perception (Single Panel):
• Focus: Understanding basic properties such as number, position, and shape of objects in a single panel.
• No comparisons are involved in this stage.

Each stage involves different types of reasoning, such as identifying patterns within a single panel or comparing objects across multiple panels. Your feedback will help us evaluate the quality of these questions based on three key indicators: Correctness, Clarity, and Content Validity.

Explanation of Evaluation Criteria:
1. Correctness:
   • This aspect assesses whether the answer provided for the question is correct.
   • Questions are considered correct if the answer corresponds accurately to the visual information presented in the panel.
2. Clarity:
   • This aspect evaluates how clear and understandable the question is.
   • Questions are considered clear if they are easy to understand.
3. Content Validity:
   • This aspect checks if the question is suitable for the task stage in which it is presented.
   • Questions are considered valid if they are appropriate for their respective stage.
   • The dataset is divided into five stages, such as single-panel reasoning or multi-panel comparisons.
   • Evaluate if the question is appropriate for the specific reasoning type it represents.

Instructions for Evaluators:
Please carefully review each QA pair and assess it using the three questions provided. Your input will be critical in refining the dataset to ensure it accurately reflects the intended reasoning tasks and provides clear, valid questions and answers.](page_184_370_591_312.png)

Figure 11: A detailed guide provided to participants at the beginning of the questionnaire for Part A.

To thoroughly assess the human performance in Part A, we used three key indicators: Correctness, Clarity, and Content Validity.

Correctness assesses whether the answer provided for the question is correct. Evaluators were asked to determine if the provided answer accurately corresponded to the visual information presented in the panels. This involved a careful comparison between the answer and the visual data to ensure accuracy.

Clarity evaluates how clear and understandable the question is. Evaluators considered whether the question was phrased clearly and was easy to understand. They assessed if the wording made sense, if any terms were ambiguous, and whether someone without prior knowledge could easily comprehend the question. This indicator is crucial for ensuring that the questions are accessible and interpretable by all participants.

Content validity checks if the question is suitable for the task stage in which it is presented. Evaluators examined whether the content of the question aligned with the current task stage. The dataset is divided into five types, such as one panel basic perception or two panel comparisons. Participants needed to ensure that the question was appropriate for the specific reasoning type it represented. This indicator ensures that each question is relevant and appropriately challenging for its designated stage.

The metrics used to evaluate performance in Part A included correctness, clarity, and content validity, with positive rates for each metric provided in Table 13. The positive rate is the proportion of questions answered by "Yes". The results indicate that the participants in Part A performed exceptionally well across all metrics, with Correctness, Clarity, and

<table>
  <tr>
    <th colspan="2">Task A (Basic Perception), Question 1</th>
  </tr>
  <tr>
    <td>Question:</td>
    <td>Identify the number of objects located in this panel.</td>
  </tr>
  <tr>
    <td>Answer:</td>
    <td>1</td>
  </tr>
  <tr>
    <td colspan="2"></td>
  </tr>
  <tr>
    <td colspan="2">Is the provided answer correct for this question? *</td>
  </tr>
  <tr>
    <td></td>
    <td>○ Yes<br>○ No</td>
  </tr>
  <tr>
    <td colspan="2">Is the question phrased clearly and easy to understand? *</td>
  </tr>
  <tr>
    <td></td>
    <td>○ Yes<br>○ No</td>
  </tr>
  <tr>
    <td colspan="2">Does the content of the question align with the current task stage? *</td>
  </tr>
  <tr>
    <td></td>
    <td>○ Yes<br>○ No</td>
  </tr>
</table>

Figure 12: Sample Question from the Questionnaire for Part A.
Content Validity scores consistently high. This suggests that the questions were well-designed and comprehensible, and the participants were able to provide accurate answers.

A.6.2 Part B
Part B utilized the Prolific crowdsourcing platform3 to recruit 162 participants who were subjected to the same set of abstract reasoning questions as those given to the MLLMs. The objective of this part was to evaluate human performance on our dataset, enabling a comparison between human and model capabilities. Participants received a detailed guide at the beginning of the questionnaire, which included task descriptions and several examples, as shown in Figure 13. The guide varied depending on the stage of the Direct Answer task, but for this section, we include only the One-Panel Basic Perception stage. And similar to Part A, due to Direct Answer covering all templates, we chose to focus on assessing the human performance of the Direct Answer component. Each question in the questionnaire for Part B included a image and a multiple-choice question, as illustrated in Figure 14.

![A detailed guide provided to participants at the beginning of the questionnaire for Part B. This guide focuses on One-Panel Basic Perception.](page_184_728_563_246.png)

Figure 13: A detailed guide provided to participants at the beginning of the questionnaire for Part B. This guide focuses on One-Panel Basic Perception.

The performance metrics for Part B are summarized in Table 14. The performance for Part B show a noticeable decline in positive rates, particularly for more complex tasks such as Two Panel Compare, One Row, and Two Rows. This decline highlights the increased difficulty of these tasks and suggests that the broader participant pool found these questions more challenging.

Inter-Participant Agreement. To quantify the inter-participant agreement across participants for Part B studies, we computed Fleiss’ kappa scores (Landis and Koch, 1977) across different question types. The Fleiss’ Kappa scores for each question types are provided in Table 15.

The high Fleiss’ Kappa score for One-Panel Basic (0.9711) indicates strong agreement among the participants, this is mainly due to the simplicity of One-Panel Basic questions. However, the lower scores for Two-Panel and rule deduction phase highlight the increased difficulty and the significant variability in human interpretation for these more complex tasks.

![Sample Question from the One-Panel Basic Perception Questionnaire for Part B.](page_728_728_363_246.png)

Figure 14: Sample Question from the One-Panel Basic Perception Questionnaire for Part B.

A.7 Performance Increase with Prior Info

Tables 15 and 16 present the percentage increase in Accuracy and MSEval metrics, respectively. It is evident that, except for the Final stage, all other stages show improved performance, with MSEval and Accuracy metrics closely aligned in these cases. However, in the Final stage, while Accuracy does not show a significant increase for the four open-source models, MSEval suggests some improvement due to the incorporation of rule information for solving the final RAVEN puzzle. An exception is observed with Qwen2-VL-72B, which may already perform well on RAVEN. Incorporating information from earlier stages might introduce

3https://www.prolific.com/
misleading details, leading to a significant performance drop.

A.8 Additional Details about MSEval

A.8.1 Algorithm Pseudo Code

Algorithm 1 shows the details Pseudo Code for our proposed MSEval metrics.

A.8.2 Computational Cost

The computational complexity of the algorithm can be expressed as:

\[
O(N \cdot |\mathcal{E}_t| \cdot |A|)
\]

where:

• \( N \): The number of samples.

• \( |\mathcal{E}_t| \): The number of edges in the logical chain (dependency relationships between nodes).

• \( |A| \): The number of possible choices for each node.

In most cases, \( |A| \), the number of possible choices per node, is typically equal to 4. As a result, the computational complexity simplifies to:

\[
O(4 \cdot N \cdot |\mathcal{E}_t|) \quad \text{or simply} \quad O(N \cdot |\mathcal{E}_t|),
\]

which is effectively linear with respect to both the number of instances (\( N \)) and the number of edges (\( |\mathcal{E}_t| \)) in the logical chain. By reducing the number of edges or employing a smaller logical chain, the computational cost can be significantly minimized, ensuring better scalability and efficiency, especially for large datasets or complex logical dependencies. This simplification highlights the importance of optimizing the chain structure to maintain computational feasibility. The actual time cost for each open-source model we tested is shown in Table 16.

A.9 Discussion Additional Materials

A.9.1 Model Parameters Trend

Figures 17, 18, 19, 20, 21, 22, and 23 demonstrate that performance generally improves with larger model parameter sizes across stages, except for the 2R and Final stages. For these two stages, most models perform below the random baseline. The differences in model performance are primarily attributed to the varying sizes of their language encoders, highlighting the significant role of a robust language encoder in overall performance. However, despite the observed improvements, a noticeable gap persists between model and human performance. This discrepancy may arise from the complexity of the visual input, which poses challenges for models in fully understanding and integrating multimodal information.

A.9.2 Attribute Break-Down Analysis

Figure 24 illustrates the attribute-level performance breakdown of two open-source models and four closed-source models evaluated on the logical chain task. Gemini, GPT-4o, and the two larger models, Qwen2-VL and NVLM-D, exhibit similar trends: the Number attribute achieves the highest performance in more complex stages (2P, 1R, and 2R), while Position dominates in lower-level stages (1P-C and 1P-B). In contrast, smaller models like Idefics2 and Intern2-VL struggle with the Number attribute but perform relatively better on Position, indicating that these models are less sensitive to counting tasks but demonstrate better spatial reasoning.

A.9.3 Handling Long Prompts

Table 17 presents the accuracy and MSEval scores for three prompting techniques incorporating prior information in the Logical Chain task. For GPT-4o, Qwen2-VL, and Gemini, the use of HTML tags yields significant performance improvements. Additionally, for GPT-4o and Qwen2-VL, the Document-based prompting also demonstrates notable benefits. However, for other models, these two techniques show a negative impact. In this case, the MSEval results are consistent with the accuracy outcomes. List 1 shows an example of HTML structured prompts, while Figure 25 shows an example of Document structured prompts.

<html>
  <body>
    <h1>
      In this visual puzzle, you are given two panels. Each panel divided into two sections by a vertical line, separating the <strong>left</strong> side from the <strong>right</strong> side, with objects might present in both sections. Below is the information generated from the previous steps, please be aware that it may or may not contain errors:
    </h1>
  </body>
</html>
<div>
<h2>Panel Information</h2>
<ul>
<li>There are 1 objects in the <strong>left</strong> part of the <strong>left</strong> panel.</li>
<li>There are 2 objects in the <strong>left</strong> part of the <strong>right</strong> panel.</li>
</ul>
</div>
<div>
<h2>Question</h2>
<p>Consider only the <strong>left</strong> part of the two panels in the image. Does the <strong>left</strong> panel contain the same number of objects, more objects, or fewer objects than the <strong>right</strong> panel?
Please select one of the following:</p>
<ul>
<li>A: More</li>
<li>B: The same</li>
<li>C: Fewer</li>
</ul>
<p>The answer should be one of A, B, C.</p>
</div>
</body>
</html>

Listing 1: HTML Structure for Handling Long Prompt

A.10 Qualitative Analysis

Lists 2 to 6 present five examples that illustrate the advantages of our MSEval metric over traditional accuracy. For instance, in List 5, all the intermediate steps are incorrect, yet the model arrives at the correct answer with only a small confidence margin (31.375 compared to the second-highest confidence of 31.125). In this case, the model’s performance is not truly effective, as it is unclear how it managed to produce the correct answer despite incorrect intermediate steps. Unlike traditional accuracy, which would mark this as fully correct, MSEval appropriately penalizes such cases by assigning a low score.

Conversely, a reverse scenario is shown in List 4, where traditional accuracy marks the result as entirely incorrect. However, the model correctly answers all intermediate steps, and the probability of the correct answer is very close to the highest confidence value. In this situation, MSEval assigns a relatively high score, reflecting the model’s partial success and rewarding its correct reasoning process.
Figure 16: MSEval percentage increase after incorporating prior info.

Figure 17: The average accuracy trend for the Direct Answer task as model sizes gradually increase. The trend line is derived using Gaussian smoothing, and the average accuracy is calculated by averaging the results across all five stages.

Figure 18: The One-Panel Basic Perception accuracy trend for the Direct Answer task as model sizes gradually increase. The trend line is derived using Gaussian smoothing.

Figure 19: The One-Panel Comparison accuracy trend for the Direct Answer task as model sizes gradually increase. The trend line is derived using Gaussian smoothing.

Figure 20: The Two-Panels Comparison accuracy trend for the Direct Answer task as model sizes gradually increase. The trend line is derived using Gaussian smoothing.
Algorithm 1 Overall Workflow

Input:
Logical Chain \( \mathcal{D}_t \)
Define: \( S_t = \{ t \} \cup \mathcal{D}_t \)
Model logits \( Z = \{ z_j^{(i)} \mid j \in S_t, i = 1, \ldots, N \} \)
All possible choices for each node \( \{ \mathcal{A}_j^{(i)} \mid j \in S_t \} \)
Output: MSEval score for stage t: MSEval_t
Step 1: Compute Conditional Probabilities
for each sample \( i = 1 \) to \( N \) do
    for each node \( j \in S_t \) do
        Compute probability \( p_j^{(i)} \leftarrow \frac{\exp(z_j^{(i)})}{\sum_{k \in \mathcal{A}_j^{(i)}} \exp(z_k^{(i)})}
    end for
end for
Step 2: Compute Conditional Mutual Information
for each sample \( i = 1 \) to \( N \) do
    for each node \( j \in S_t \) do
        Alter \( \mathcal{A}_j^{(i)} \) to generate perturbed outputs \( \mathcal{A}_{j \rightarrow t}^{(i)} \)
        Compute:
        \[
        \text{CMI}(i, j, t) \leftarrow H(\mathcal{A}_{j \rightarrow t}^{(i)} \mid \mathcal{D}_t^{(i),-j})
            + H(\mathcal{A}_j^{(i)} \mid \mathcal{D}_t^{(i),-j})
            - H(\mathcal{A}_{j \rightarrow t}^{(i)}, \mathcal{A}_j^{(i)} \mid \mathcal{D}_t^{(i),-j})
        \]
    end for
end for
Step 3: Normalize Conditional Mutual Information
for each sample \( i = 1 \) to \( N \) do
    for each node \( j \in S_t \) do
        Compute:
        \[
        \text{NCMI}(i, j, t) \leftarrow \frac{\exp(\text{CMI}(i, j, t))}{\sum_{k \in S_t} \exp(\text{CMI}(i, k, t))}
        \]
    end for
end for
Step 4: Compute MSEval Score for Each Sample
for each sample \( i = 1 \) to \( N \) do
    Initialize \( \text{MSEval}_t^{(i)} \leftarrow 0 \)
    for each node \( j \in S_t \) do
        Compute
        \[
        \epsilon_j^{(i)} = \frac{1}{|\mathcal{A}_j^{(i)}|}
        \]
        Update:
        \[
        \text{MSEval}_t^{(i)} \leftarrow \text{MSEval}_t^{(i)} + \text{NCMI}(i, j, t) \cdot \frac{p_j^{(i)}}{\epsilon_j^{(i)}}
        \]
    end for
end for
Step 5: Average MSEval Across All Samples
Compute:
\[
\text{MSEval}_t \leftarrow \frac{1}{N} \sum_{i=1}^N \text{MSEval}_t^{(i)}
\]
Return: MSEval_t

Figure 21: The One-Row Deduction accuracy trend for the Direct Answer task as model sizes gradually increase. The trend line is derived using Gaussian smoothing.

![Accuracy trend for 1R Config vs. Model Parameters](page_1016_180_377_246.png)

Figure 22: The Two-Rows Deduction accuracy trend for the Direct Answer task as model sizes gradually increase. The trend line is derived using Gaussian smoothing.

![Accuracy trend for 2R Config vs. Model Parameters](page_1016_495_377_246.png)

Figure 23: The Final accuracy trend for the Direct Answer task as model sizes gradually increase. The trend line is derived using Gaussian smoothing.

![Accuracy trend for Final Config vs. Model Parameters](page_1016_810_377_246.png)
Figure 24: The Radar Chart for all six models in Logical Chain subtask.

RAVEN Matrix Puzzle

You are presented with a 3x3 grid of panels, called the "Problem Matrix." The last panel is missing and marked with a "?" symbol. Below the matrix, there is a set of 8 possible answer options labeled from 1 to 8. Your task is to determine which panel from the answer set (1-8) correctly fits the missing position in the problem matrix. The pattern in the matrix follows some hidden rules that apply row by row (horizontally).

Rules:
Here are some rules derived from previous steps, which might be useful to derive the answer; please be aware that they may contain errors:
Number - The number of objects in the last panel equals the sum of the objects in the previous two panels.
Position - The position of objects in the last panel is the union of positions from the previous two panels.
Shape - The edge number of shape gradually decreases by 1.
Size - The size of objects in the last panel is the difference between the sizes in the previous two panels.
Color - The color of objects in the last panel is the sum of the colors in the previous two panels.

Figure 25: Document Structure for Handling Long Prompt.
<table>
  <tr>
    <th>Question Pattern</th>
    <th>Question Example</th>
    <th>Attribute</th>
    <th>Constraints</th>
    <th>Answer Space</th>
  </tr>
  <tr>
    <th colspan="5">One-Panel Basic</th>
  </tr>
  <tr>
    <td>How many objects are in panel &lt;P&gt;?</td>
    <td>How many objects are in panel 1?</td>
    <td>Number</td>
    <td>NA</td>
    <td>[1,2,3,4,5,6,7,8,9]</td>
  </tr>
  <tr>
    <td>What is the shape of the object at &lt;X&gt; in panel &lt;P&gt;?</td>
    <td>What is the shape of the object at top-left in panel 1?</td>
    <td>Shape</td>
    <td>NA</td>
    <td>[ "triangle", "square", "pentagon", "hexagon", "circle" ]</td>
  </tr>
  <tr>
    <td>Where is the &lt;S&gt; positioned in panel &lt;P&gt;?</td>
    <td>Where is the triangle positioned in panel 1?</td>
    <td>Position</td>
    <td>NA</td>
    <td>[ "Left", "Right", "Top", "Down", "Bottom-Left", ... ]</td>
  </tr>
  <tr>
    <th colspan="5">One-Panel Comparison</th>
  </tr>
  <tr>
    <td>In panel &lt;P&gt;, is the shape of the object on the &lt;X&gt; have the same, more, or fewer edges compared to the object on the &lt;X2&gt;? (Note: The edge number increases in the following order: triangle, square, pentagon, hexagon, circle)</td>
    <td>In this panel, is the shape of the object on the left have the same, more, or fewer edges compared to the object on the right? (Note: The edge number increases in the following order: triangle, square, pentagon, hexagon, circle)</td>
    <td>Shape</td>
    <td>Not_Equal(X, X2)</td>
    <td>[ "The same", "Fewer", "More" ]</td>
  </tr>
  <tr>
    <td>In panel &lt;P&gt;, does the object on the &lt;X&gt; the same, smaller or larger in size compared to the object on the &lt;X2&gt;?</td>
    <td>In panel 1, does the object on the top-left the same, smaller or larger in size compared to the object on the bottom-right?</td>
    <td>Size</td>
    <td>Not_Equal(X, X2)</td>
    <td>[ "The same", "Smaller", "Larger" ]</td>
  </tr>
  <tr>
    <td>In panel &lt;P&gt;, does the object on the &lt;X&gt; the same, darker or brighter in color compared to the object on the &lt;X2&gt;?</td>
    <td>In panel 1, does the object on the top-left the same, darker or brighter in color compared to the object on the bottom-right?</td>
    <td>color</td>
    <td>Not_Equal(X, X2)</td>
    <td>[ "The same", "Darker", "Brighter" ]</td>
  </tr>
  <tr>
    <td>In panel &lt;P&gt;, where is the &lt;S&gt; relative to the &lt;S2&gt;?</td>
    <td>In panel 1, where is the triangle relative to the square?</td>
    <td>Position</td>
    <td>Not_Equal(S, S2)</td>
    <td>[ "Left", "Right", "Above", "Below", ... ]</td>
  </tr>
  <tr>
    <td>Are all objects in panel &lt;P&gt; of the same shape?</td>
    <td>Are all objects in panel 1 of the same shape?</td>
    <td>Shape</td>
    <td>NA</td>
    <td>[ "Yes", "No" ]</td>
  </tr>
  <tr>
    <td>Are all objects in panel &lt;P&gt; of the same size?</td>
    <td>Are all objects in panel 1 of the same size?</td>
    <td>Size</td>
    <td>NA</td>
    <td>[ "Yes", "No" ]</td>
  </tr>
  <tr>
    <td>Are all objects in panel &lt;P&gt; of the same color?</td>
    <td>Are all objects in panel 1 of the same color?</td>
    <td>Color</td>
    <td>NA</td>
    <td>[ "Yes", "No" ]</td>
  </tr>
  <tr>
    <th colspan="5">Two-Panels Comparison</th>
  </tr>
  <tr>
    <td>Does panel &lt;P&gt; contain the same number of objects, more objects, or fewer objects than panel &lt;P2&gt;?</td>
    <td>Does panel 1 contain the same number of objects, more objects, or fewer objects than panel 2?</td>
    <td>Number</td>
    <td>Not_Equal(P, P2), Same_Row(P, P2)</td>
    <td>[ "The same", "More", "Fewer" ]</td>
  </tr>
  <tr>
    <td>Is the shape of all the objects in panel &lt;P&gt; have the same, more, or fewer edges compared to the objects in panel &lt;P2&gt;? If the shapes within either panel are already different from each other, select 'Not Comparable.' (Note: The edge number increases in the following order: triangle, square, pentagon, hexagon, circle)</td>
    <td>Is the shape of all the objects in panel 1 have the same, more, or fewer edges compared to the objects in panel 2? If the shapes within either panel are already different from each other, select 'Not Comparable.' (Note: The edge number increases in the following order: triangle, square, pentagon, hexagon, circle)</td>
    <td>Shape</td>
    <td>Not_Equal(P, P2), Same_Row(P, P2)</td>
    <td>[ "The same", "Fewer", "More", "Not comparable" ]</td>
  </tr>
  <tr>
    <td>Is the size of all the objects in panel &lt;P&gt; the same as, smaller or larger than the objects in panel &lt;P2&gt;? If the sizes within either panel are already different from each other, select 'Not Comparable.'</td>
    <td>Is the size of all the objects in panel 1 the same as, smaller or larger than the objects in panel 2? If the sizes within either panel are already different from each other, select 'Not Comparable.'</td>
    <td>Size</td>
    <td>Not_Equal(P, P2), Same_Row(P, P2)</td>
    <td>[ "The same", "Smaller", "Larger", "Not comparable" ]</td>
  </tr>
</table>
<table>
  <tr>
    <th>Question Pattern</th>
    <th>Question Example</th>
    <th>Attribute</th>
    <th>Constraints</th>
    <th>Answer Space</th>
  </tr>
  <tr>
    <th colspan="5">Two-Panels Comparison</th>
  </tr>
  <tr>
    <td>Is the color of all the objects in panel <P1> the same as, darker or brighter than the objects in panel <P2>? If the colors within either panel are already different from each other, select 'Not Comparable.'</td>
    <td>Is the color of all the objects in panel 1 the same as, darker or brighter than the objects in panel 2? If the colors within either panel are already different from each other, select 'Not Comparable.'</td>
    <td>Color</td>
    <td>Not_Equal(P1, P2), Same_Row(P1,P2)</td>
    <td>[ "The same", "Darker", "Brighter", "Not comparable" ]</td>
  </tr>
  <tr>
    <td>Is the position of all the objects in panel <P1> the same as the objects in panel <P2>?</td>
    <td>Is the position of all the objects in panel 1 the same as the objects in panel 2?</td>
    <td>Position</td>
    <td>Not_Equal(P1, P2), Same_Row(P1,P2)</td>
    <td>[ "Yes", "No" ]</td>
  </tr>
  <tr>
    <th colspan="5">One-Row Rule Deduction</th>
  </tr>
  <tr>
    <td>Examine the three panels in the image from left to right and identify the rule that governs the number of the objects.</td>
    <td>-</td>
    <td>Number</td>
    <td>NA</td>
    <td>See Number Rule in Table 8</td>
  </tr>
  <tr>
    <td>Examine the three panels in the image from left to right and identify the rule that governs the position of the objects.</td>
    <td>-</td>
    <td>Position</td>
    <td>NA</td>
    <td>See Position Rule in Table 8</td>
  </tr>
  <tr>
    <td>Examine the three panels in the image from left to right and identify the rule that governs the shape of the objects.</td>
    <td>-</td>
    <td>Shape</td>
    <td>NA</td>
    <td>See Shape Rule in Table 8</td>
  </tr>
  <tr>
    <td>Examine the three panels in the image from left to right and identify the rule that governs the size of the objects.</td>
    <td>-</td>
    <td>Size</td>
    <td>NA</td>
    <td>See Size Rule in Table 8</td>
  </tr>
  <tr>
    <td>Examine the three panels in the image from left to right and identify the rule that governs the color of the objects.</td>
    <td>-</td>
    <td>Color</td>
    <td>NA</td>
    <td>See Color Rule in Table 8</td>
  </tr>
  <tr>
    <th colspan="5">Two-Rows Rule Deduction</th>
  </tr>
  <tr>
    <td>Inspect the first row of three panels from left to right and inspect the second row of three panels from left to right and determine a rule applicable to both rows that governs the number of objects.</td>
    <td>-</td>
    <td>Number</td>
    <td>NA</td>
    <td>Same as One-Row</td>
  </tr>
  <tr>
    <td>Inspect the first row of three panels from left to right and inspect the second row of three panels from left to right and determine a rule applicable to both rows that governs the position of objects.</td>
    <td>-</td>
    <td>Position</td>
    <td>NA</td>
    <td>Same as One-Row</td>
  </tr>
  <tr>
    <td>Inspect the first row of three panels from left to right and inspect the second row of three panels from left to right and determine a rule applicable to both rows that governs the shape of objects.</td>
    <td>-</td>
    <td>Shape</td>
    <td>NA</td>
    <td>Same as One-Row</td>
  </tr>
  <tr>
    <td>Inspect the first row of three panels from left to right and inspect the second row of three panels from left to right and determine a rule applicable to both rows that governs the size of objects.</td>
    <td>-</td>
    <td>Size</td>
    <td>NA</td>
    <td>Same as One-Row</td>
  </tr>
  <tr>
    <td>Inspect the first row of three panels from left to right and inspect the second row of three panels from left to right and determine a rule applicable to both rows that governs the color of objects.</td>
    <td>-</td>
    <td>Color</td>
    <td>NA</td>
    <td>Same as One-Row</td>
  </tr>
</table>

Table 7: Question pattern templates with corresponding example questions. There are 25 templates in total. 3 templates for One-Panel Basic. 7 templates for One-Panel Comparison. 5 templates for Two-Panels Comparison. 5 Templates for One-Row Rule Deduction. 5 Templates for Two-Rows Rule Deduction.
<table>
  <tr>
    <th>Category</th>
    <th>Value Ranges</th>
  </tr>
  <tr>
    <th colspan="2">Placeholders</th>
  </tr>
  <tr>
    <td>Position (&lt;X&gt;)</td>
    <td>center, left, right, top, bottom, top-left, top-right, bottom-left, bottom-right, top-left, top-center, top-right, middle-left, middle-center, middle-right, bottom-left, bottom-center, bottom-right, outer-part, inner-part, top-left of the inner part, top-right of the inner part, bottom-left of the inner part, bottom-right of the inner part</td>
  </tr>
  <tr>
    <td>Panel (&lt;P&gt;)</td>
    <td>0, 1, 2, 3, 4, 5, 6, 7</td>
  </tr>
  <tr>
    <td>Shape (&lt;S&gt;)</td>
    <td>triangle, square, pentagon, hexagon, circle</td>
  </tr>
  <tr>
    <th colspan="2">Rules</th>
  </tr>
  <tr>
    <td>Number Rule</td>
    <td>The number of objects gradually decreases by 1; The number of objects remains constant; The number of objects gradually increases by 1; The number of objects distributes three distinct values across panels, rotating through each possible permutation of these values; The number of objects in the last panel equals the sum of the objects in the previous two panels; The number of objects in the last panel equals the difference between the objects in the previous two panels; No clear rule is present</td>
  </tr>
  <tr>
    <td>Position Rule</td>
    <td>If an object is in the first panel but not in the second at corresponding position, it appears in the third panel; The position of objects in the last panel is the union of positions from the previous two panels; Three distinct position settings across panels, rotating through each possible permutation of these settings; The position of objects does not change across panels; No clear rule is present</td>
  </tr>
  <tr>
    <td>Color Rule</td>
    <td>The color of objects gradually darkens by a constant amount each time; The color of objects gradually brightens by a constant amount each time; The color of objects in the last panel is the sum of the colors in the previous two panels; The color of objects in the last panel is the difference between the colors in the previous two panels; Three distinct colors across panels, rotating through each possible permutation of these colors; The color remains constant; No clear rule is present</td>
  </tr>
  <tr>
    <td>Size Rule</td>
    <td>The size of objects gradually increases by a constant amount each time; The size of objects gradually decreases by a constant amount each time; The size of objects in the last panel is the sum of the sizes in the previous two panels; The size of objects in the last panel is the difference between the sizes in the previous two panels; Three distinct sizes across panels, rotating through each possible permutation of these sizes; The size remains constant; No clear rule is present</td>
  </tr>
  <tr>
    <td>Shape Rule</td>
    <td>The edge number of shape gradually decreases by 1; The edge number of shape gradually increases by 1; Three distinct shapes across panels, rotating through each possible permutation of these shapes; The shape remains constant; No clear rule is present</td>
  </tr>
</table>

Table 8: Pre-defined placeholder value ranges and rules for five attributes
<table>
  <tr>
    <th>Attributes</th>
    <th>Logical Chain Stage Example</th>
  </tr>
  <tr>
    <td rowspan="4">Number</td>
    <td><b>1P:</b> How many objects are in the panel?<br>
      <b>2P:</b> Does the left panel contain the same number of objects, more objects, or fewer objects than the right panel?<br>
      <b>1R:</b> Inspect the three panels in the image from left to right and identify the rule that dictates the number of objects.<br>
      <b>2R:</b> Inspect the first row of three panels from left to right and inspect the second row of three panels from left to right and determine a rule applicable to both rows that governs the number of objects.
    </td>
  </tr>
  <tr>
    <td rowspan="4">Position</td>
    <td><b>1P:</b> Where is the circle positioned in the panel?<br>
      <b>2P:</b> Is the position of all the objects in the left panel the same as the objects in the right panel?<br>
      <b>1R:</b> Examine the three panels in the image from left to right and identify the rule that governs the position of the objects.<br>
      <b>2R:</b> Examine the three panels in the first row, then the three panels in the second row, both from left to right, and derive a rule that applies to both rows in relation to the position of objects.
    </td>
  </tr>
  <tr>
    <td rowspan="4">Shape</td>
    <td><b>1P:</b> What is the shape of the object at center in the panel?<br>
      <b>2P:</b> Is the shape of all the objects in the left panel have the same, more, or fewer edges compared to the objects in the right panel?<br>
      <b>1R:</b> Inspect the three panels in the image from left to right and identify the rule that dictates the shape of objects.<br>
      <b>2R:</b> Analyze the first row of three panels from left to right, followed by the second row of three panels, and identify a common rule that dictates the shape of objects in both rows.
    </td>
  </tr>
  <tr>
    <td rowspan="4">Size</td>
    <td><b>1P:</b> Are all objects in the panel of the same size?<br>
      <b>2P:</b> Is the size of all the objects in the left panel the same as, smaller or larger than the objects in the right panel?<br>
      <b>1R:</b> Analyze the three panels in the image from left to right and uncover the rule that governs the size of objects.<br>
      <b>2R:</b> Review the first row of three panels in sequence from left to right, then do the same for the second row, and determine a shared rule that governs the size of objects in both rows.
    </td>
  </tr>
  <tr>
    <td rowspan="4">Color</td>
    <td><b>1P:</b> Are all objects in the panel of the same color?<br>
      <b>2P:</b> Is the color of all the objects in panel <P> the same as, darker or brighter than the objects in panel <P2>?<br>
      <b>1R:</b> Inspect the three panels in the image from left to right and identify the rule that dictates the color of objects.<br>
      <b>2R:</b> Examine the three panels in the first row, then the three panels in the second row, both from left to right, and derive a rule that applies to both rows in relation to the color of objects.
    </td>
  </tr>
  <tr>
    <td rowspan="1">Final</td>
    <td>You are presented with a 3x3 grid of panels, called the <i>Problem Matrix</i>. The last panel is missing and marked with a "?" symbol. Below the matrix, there is a set of 8 possible answer options labeled from 1 to 8. Your task is to determine which panel from the answer set (1-8) correctly fits the missing position in the problem matrix. The pattern in the matrix follows some hidden rules that apply row by row (horizontally). Please select the number (from 1 to 8) of the panel that completes the pattern.</td>
  </tr>
</table>

Table 9: A full logical chain with the examples for five stages.
<table>
  <tr>
    <th rowspan="2">Metric</th>
    <th colspan="6">Prior</th>
  </tr>
  <tr>
    <th>1P</th>
    <th>2P</th>
    <th>1R</th>
    <th>2R</th>
    <th>Final</th>
  </tr>
  <tr>
    <td rowspan="2">Acc</td>
    <td>w/o</td>
    <td>31.94</td>
    <td>28.09</td>
    <td>22.00</td>
    <td>23.82</td>
    <td>7.14</td>
  </tr>
  <tr>
    <td>w</td>
    <td>31.94</td>
    <td>28.09</td>
    <td>22.13</td>
    <td>23.45</td>
    <td>8.57</td>
  </tr>
  <tr>
    <td>InstructBLIP</td>
    <td>MSEval</td>
    <td>w/o</td>
    <td>1.000</td>
    <td>1.000</td>
    <td>1.007</td>
    <td>1.000</td>
    <td>0.848</td>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td>w</td>
    <td>1.000</td>
    <td>1.000</td>
    <td>1.004</td>
    <td>1.004</td>
    <td>0.959</td>
  </tr>
  <tr>
    <td rowspan="2">Acc</td>
    <td>xGen-MM</td>
    <td>w/o</td>
    <td>60.48</td>
    <td>23.27</td>
    <td>25.82</td>
    <td>21.64</td>
    <td>14.29</td>
  </tr>
  <tr>
    <td></td>
    <td>w</td>
    <td>60.48</td>
    <td>23.18</td>
    <td>20.18</td>
    <td>26.36</td>
    <td>5.71</td>
  </tr>
  <tr>
    <td>MSEval</td>
    <td>w/o</td>
    <td>2.006</td>
    <td>0.794</td>
    <td>0.965</td>
    <td>0.953</td>
    <td>1.290</td>
  </tr>
  <tr>
    <td></td>
    <td>w</td>
    <td>2.006</td>
    <td>1.082</td>
    <td>0.812</td>
    <td>0.985</td>
    <td>1.013</td>
  </tr>
  <tr>
    <td rowspan="2">Acc</td>
    <td>Llava-13b</td>
    <td>w/o</td>
    <td>30.91</td>
    <td>31.55</td>
    <td>23.09</td>
    <td>22.36</td>
    <td>15.71</td>
  </tr>
  <tr>
    <td></td>
    <td>w</td>
    <td>30.91</td>
    <td>31.09</td>
    <td>22.36</td>
    <td>21.64</td>
    <td>15.71</td>
  </tr>
  <tr>
    <td>MSEval</td>
    <td>w/o</td>
    <td>1.059</td>
    <td>0.997</td>
    <td>0.941</td>
    <td>0.910</td>
    <td>1.005</td>
  </tr>
  <tr>
    <td></td>
    <td>w</td>
    <td>1.059</td>
    <td>1.009</td>
    <td>0.944</td>
    <td>0.929</td>
    <td>0.975</td>
  </tr>
  <tr>
    <td rowspan="2">Acc</td>
    <td>Random</td>
    <td>-</td>
    <td>31.1</td>
    <td>31.7</td>
    <td>25.0</td>
    <td>25.0</td>
    <td>12.5</td>
  </tr>
  <tr>
    <td>MSEval</td>
    <td>-</td>
    <td>1.00</td>
    <td>1.00</td>
    <td>1.00</td>
    <td>1.00</td>
    <td>1.00</td>
  </tr>
</table>

Table 10: The Accuracy (Acc) and MSEval scores for the Logical Chain task. w/o: without prior, w: with prior.
<table>
  <tr>
    <th>Attribute</th>
    <th>Stage</th>
    <th>Numbers</th>
    <th>Output</th>
  </tr>
  <tr>
    <td rowspan="3"><b>Number</b></td>
    <td>single_panel</td>
    <td>[\"1\"]</td>
    <td>There are {answer_str} objects in the left panel.</td>
  </tr>
  <tr>
    <td></td>
    <td>[\"2\"]</td>
    <td>There are {answer_str} objects in the right panel.</td>
  </tr>
  <tr>
    <td></td>
    <td>[\"3\"]</td>
    <td>There are {answer_str} objects in the right panel.</td>
  </tr>
  <tr>
    <td rowspan="2"></td>
    <td>two_panels</td>
    <td>[\"1\", \"2\"]</td>
    <td>The left panel has {answer_str} objects compared to the middle panel.</td>
  </tr>
  <tr>
    <td></td>
    <td>[\"2\", \"3\"]</td>
    <td>The middle panel has {answer_str} objects compared to the right panel.</td>
  </tr>
  <tr>
    <td></td>
    <td>one_row</td>
    <td>Any</td>
    <td>The rule for the number of objects in the first row is: {answer_str}.</td>
  </tr>
  <tr>
    <td rowspan="5"><b>Position</b></td>
    <td>single_panel</td>
    <td>[\"1\"]</td>
    <td>Where is the (\w+) positioned in the panel? becomes: There is a \1 positioned in the left panel.</td>
  </tr>
  <tr>
    <td></td>
    <td>[\"2\"], [\"3\"]</td>
    <td>Where is the (\w+) positioned in the panel? becomes: There is a \1 positioned in the right panel.</td>
  </tr>
  <tr>
    <td>two_panels</td>
    <td>[\"1\", \"2\"]</td>
    <td>If answer_str is Yes, "The position of all the objects in the left panel is the same as the objects in the middle panel." Otherwise, "The position of all the objects in the left panel is not the same as the objects in the middle panel."</td>
  </tr>
  <tr>
    <td></td>
    <td>[\"2\", \"3\"]</td>
    <td>If answer_str is Yes, "The position of all the objects in the middle panel is the same as the objects in the right panel." Otherwise, "The position of all the objects in the middle panel is not the same as the objects in the right panel."</td>
  </tr>
  <tr>
    <td>one_row</td>
    <td>Any</td>
    <td>The rule for the position of objects in the first row is: {answer_str}.</td>
  </tr>
</table>

Table 11: Rule-based program of attribute Number and Position. The Stage represents the prior stage. (\w+) represents the word here will be put in the position of \1.

<table>
  <tr>
    <th>Setting</th>
    <th>1P-C</th>
    <th>1P-B</th>
    <th>2P</th>
    <th>1R</th>
    <th>2R</th>
    <th>Final</th>
  </tr>
  <tr>
    <td>Setting 1</td>
    <td>0/10</td>
    <td>3/10</td>
    <td>1/10</td>
    <td>8/10</td>
    <td>7/10</td>
    <td>10/10</td>
  </tr>
  <tr>
    <td>Setting 2</td>
    <td>7/10</td>
    <td>4/10</td>
    <td>8/10</td>
    <td>10/10</td>
    <td>10/10</td>
    <td>10/10</td>
  </tr>
</table>

Table 12: Performance comparison of different settings across various stages. Setting 1: Same Attribute; Setting 2: Changed Attribute. The ratio is Hallucinations / Total:
<table>
  <tr>
    <th>Metric</th>
    <th>One Panel Basic</th>
    <th>One Panel Compare</th>
    <th>Two Panel Compare</th>
    <th>One Row</th>
    <th>Two Rows</th>
  </tr>
  <tr>
    <td>Correctness</td>
    <td>0.98</td>
    <td>0.97</td>
    <td>0.96</td>
    <td>0.93</td>
    <td>0.94</td>
  </tr>
  <tr>
    <td>Clarity</td>
    <td>0.96</td>
    <td>0.97</td>
    <td>0.94</td>
    <td>0.95</td>
    <td>0.99</td>
  </tr>
  <tr>
    <td>Content Validity</td>
    <td>0.99</td>
    <td>0.99</td>
    <td>0.99</td>
    <td>1.00</td>
    <td>1.00</td>
  </tr>
</table>

Table 13: Human performance (positive rates) for Part A across different question types.

<table>
  <tr>
    <th>1P-B</th>
    <th>1P-C</th>
    <th>2P</th>
    <th>1R</th>
    <th>2R</th>
  </tr>
  <tr>
    <td>98.52</td>
    <td>88.89</td>
    <td>69.08</td>
    <td>62.12</td>
    <td>63.33</td>
  </tr>
</table>

Table 14: Human performance (positive rates) for Part B across different question types.

<table>
  <tr>
    <th>Task</th>
    <th>1P-B</th>
    <th>1P-C</th>
    <th>2P</th>
    <th>1R</th>
    <th>2R</th>
  </tr>
  <tr>
    <td>Kappa Scores</td>
    <td>0.9711</td>
    <td>0.7830</td>
    <td>0.4988</td>
    <td>0.4443</td>
    <td>0.4075</td>
  </tr>
</table>

Table 15: Fleiss’ Kappa Scores for Inter-Participant Agreement across different question types.

<table>
  <tr>
    <th>Model</th>
    <th>Running Time</th>
  </tr>
  <tr>
    <td>Idefics2-8B</td>
    <td>7H 24M</td>
  </tr>
  <tr>
    <td>Intern2-VL-8B</td>
    <td>14H 54M</td>
  </tr>
  <tr>
    <td>Qwen2-VL-Instruct-72B</td>
    <td>5D 3H 50M</td>
  </tr>
  <tr>
    <td>NVLM-D-72B</td>
    <td>5D 0H 13M</td>
  </tr>
</table>

Total Questions: 3.92K

Device: 2 × A100 80G GPU

Table 16: Actual Running Time for Each Model. D: Day, H: Hour, M: Minute

<table>
  <tr>
    <th>Metric</th>
    <th>Prior</th>
    <th>1P</th>
    <th>2P</th>
    <th>1R</th>
    <th>2R</th>
    <th>Final</th>
  </tr>
  <tr>
    <td rowspan="3">GPT-4o</td>
    <td>Vanilla</td>
    <td>73.8</td>
    <td>43.9</td>
    <td>41.8</td>
    <td>50.6</td>
    <td>10.0</td>
  </tr>
  <tr>
    <td>Acc</td>
    <td><b>82.2</b></td>
    <td>64.4</td>
    <td>47.8</td>
    <td>50.9</td>
    <td>8.6</td>
  </tr>
  <tr>
    <td>Doc.</td>
    <td>80.8</td>
    <td>44.8</td>
    <td>31.1</td>
    <td>24.9</td>
    <td>10.0</td>
  </tr>
  <tr>
    <td rowspan="3">Gemini</td>
    <td>Vanilla</td>
    <td>75.5</td>
    <td>64.4</td>
    <td>52.6</td>
    <td>57.1</td>
    <td>18.6</td>
  </tr>
  <tr>
    <td>Acc</td>
    <td>70.6</td>
    <td>66.4</td>
    <td>52.9</td>
    <td><b>57.8</b></td>
    <td>17.1</td>
  </tr>
  <tr>
    <td>Doc.</td>
    <td>69.6</td>
    <td>51.0</td>
    <td>36.7</td>
    <td>33.1</td>
    <td>14.3</td>
  </tr>
  <tr>
    <td rowspan="3">Qwen2-VL (72B)</td>
    <td>Vanilla</td>
    <td>74.1</td>
    <td>57.8</td>
    <td>47.3</td>
    <td>54.2</td>
    <td>65.7*</td>
  </tr>
  <tr>
    <td>Acc</td>
    <td>77.2</td>
    <td><b>67.7</b></td>
    <td><b>55.1</b></td>
    <td>53.6</td>
    <td>61.4</td>
  </tr>
  <tr>
    <td>Doc.</td>
    <td>76.5</td>
    <td>63.1</td>
    <td>50.2</td>
    <td>46.6</td>
    <td>24.3</td>
  </tr>
  <tr>
    <td rowspan="3">NVLM-D (72B)</td>
    <td>Vanilla</td>
    <td>2.54</td>
    <td>1.95</td>
    <td>1.79</td>
    <td>1.70</td>
    <td>5.14*</td>
  </tr>
  <tr>
    <td>MSEval</td>
    <td>2.64</td>
    <td>2.46</td>
    <td>2.37</td>
    <td>2.17</td>
    <td>3.11</td>
  </tr>
  <tr>
    <td>Doc.</td>
    <td>2.62</td>
    <td>2.33</td>
    <td>2.26</td>
    <td>1.90</td>
    <td>1.88</td>
  </tr>
  <tr>
    <td rowspan="3">Idefics2 (8B)</td>
    <td>Vanilla</td>
    <td>66.1</td>
    <td>45.2</td>
    <td>39.1</td>
    <td>43.3</td>
    <td>7.1</td>
  </tr>
  <tr>
    <td>Acc</td>
    <td>45.6</td>
    <td>25.3</td>
    <td>23.1</td>
    <td>36.8</td>
    <td>10.0</td>
  </tr>
  <tr>
    <td>Doc.</td>
    <td>18.7</td>
    <td>11.3</td>
    <td>17.5</td>
    <td>14.2</td>
    <td>20.0</td>
  </tr>
  <tr>
    <td rowspan="3">Intern2-VL (8B)</td>
    <td>Vanilla</td>
    <td>2.25</td>
    <td>1.20</td>
    <td>1.28</td>
    <td>1.02</td>
    <td>0.76</td>
  </tr>
  <tr>
    <td>MSEval</td>
    <td>1.84</td>
    <td>0.87</td>
    <td>0.99</td>
    <td>0.98</td>
    <td>0.79</td>
  </tr>
  <tr>
    <td>Doc.</td>
    <td>0.93</td>
    <td>0.78</td>
    <td>0.88</td>
    <td>0.94</td>
    <td>0.96</td>
  </tr>
  <tr>
    <td rowspan="3">Vanilla</td>
    <td>57.8</td>
    <td>37.8</td>
    <td>36.6</td>
    <td>42.4</td>
    <td>25.7</td>
  </tr>
  <tr>
    <td>Acc</td>
    <td>46.2</td>
    <td>30.5</td>
    <td>36.9</td>
    <td>43.5</td>
    <td>18.6</td>
  </tr>
  <tr>
    <td>Doc.</td>
    <td>27.2</td>
    <td>23.6</td>
    <td>9.6</td>
    <td>6.9</td>
    <td>15.7</td>
  </tr>
  <tr>
    <td rowspan="3">Vanilla</td>
    <td>2.02</td>
    <td>1.48</td>
    <td>1.51</td>
    <td>1.51</td>
    <td>1.44</td>
  </tr>
  <tr>
    <td>MSEval</td>
    <td>1.59</td>
    <td>1.18</td>
    <td>1.25</td>
    <td>1.33</td>
    <td>1.27</td>
  </tr>
  <tr>
    <td>Doc.</td>
    <td>1.04</td>
    <td>1.01</td>
    <td>0.97</td>
    <td>0.98</td>
    <td>1.00</td>
  </tr>
  <tr>
    <td rowspan="3">Vanilla</td>
    <td>54.4</td>
    <td>41.9</td>
    <td>31.6</td>
    <td>33.5</td>
    <td>17.1</td>
  </tr>
  <tr>
    <td>Acc</td>
    <td>48.4</td>
    <td>31.0</td>
    <td>20.7</td>
    <td>27.3</td>
    <td>7.1</td>
  </tr>
  <tr>
    <td>Doc.</td>
    <td>23.1</td>
    <td>30.2</td>
    <td>17.1</td>
    <td>17.1</td>
    <td>8.6</td>
  </tr>
  <tr>
    <td rowspan="3">Vanilla</td>
    <td>2.02</td>
    <td>1.48</td>
    <td>1.51</td>
    <td>1.51</td>
    <td>1.44</td>
  </tr>
  <tr>
    <td>MSEval</td>
    <td>1.52</td>
    <td>1.18</td>
    <td>1.10</td>
    <td>1.00</td>
    <td>0.97</td>
  </tr>
  <tr>
    <td>Doc.</td>
    <td>1.02</td>
    <td>1.00</td>
    <td>1.00</td>
    <td>0.97</td>
    <td>0.92</td>
  </tr>
</table>

Table 17: The Accuracy (Acc) and MSEval scores of three prompting techniques for the Logical Chain task. Vanilla: Pure Text, Struct.: Structure (HTML), Doc.: Document. The highest accuracy are highlighted in bold. The highest MSEval are highlighted in underline.
---------------------------
Dependent Stage Name: single_panel_1_left
Dependent Stage Question: Are all objects in the left part of the panel of the same color?
Dependent Stage Choice: ['A: Only one object', 'B: No', 'C: Yes']
Dependent Stage Ground Truth: A
Dependent Stage Logits: {'A': 11.125, 'B': 11.125, 'C': 20.75}
Dependent Stage Generated Answer: C
---------------------------
Dependent Stage Name: single_panel_2_left
Dependent Stage Question: Are all objects in the left part of the panel of the same color?
Dependent Stage Choice: ['A: No', 'B: Only one object', 'C: Yes']
Dependent Stage Ground Truth: B
Dependent Stage Logits: {'A': 11.125, 'B': 10.5625, 'C': 19.875}
Dependent Stage Generated Answer: C
---------------------------
Dependent Stage Name: single_panel_3_left
Dependent Stage Question: Are all objects in the left part of the panel of the same color?
Dependent Stage Choice: ['A: Only one object', 'B: No', 'C: Yes']
Dependent Stage Ground Truth: A
Dependent Stage Logits: {'A': 10.375, 'B': 10.375, 'C': 19.75}
Dependent Stage Generated Answer: C
---------------------------
Dependent Stage Name: two_panels_1_2_left
Dependent Stage Question: Consider only the left part of the two panels in the image. Is the color of all the objects in the left panel the same as, darker or brighter than the objects in the right panel? If the colors within either panel are already different from each other, select 'Not Comparable.'
Dependent Stage Choice: ['A: Not comparable', 'B: The same', 'C: Darker', 'D: Brighter']
Dependent Stage Ground Truth: C
Dependent Stage Logits: {'A': 16.5, 'B': 17.75, 'C': 16.375, 'D': 15.0625}
Dependent Stage Generated Answer: B
---------------------------
Dependent Stage Name: two_panels_2_3_left
Dependent Stage Question: Consider only the left part of the two panels in the image. Is the color of all the objects in the left panel the same as, darker or brighter than the objects in the right panel? If the colors within either panel are already different from each other, select 'Not Comparable.'
Dependent Stage Choice: ['A: Not comparable', 'B: Darker', 'C: Brighter', 'D: The same']
Dependent Stage Ground Truth: B
Dependent Stage Logits: {'A': 18.875, 'B': 18.0, 'C': 16.125, 'D': 19.5}
Dependent Stage Generated Answer: D
---------------------------
Current Stage:
Current Stage Name: one_row_left
Current Stage Question: Look at the three panels in the image from left to right, paying attention only to the left portions of each panel, and identify the rule that controls the color of objects.
Current Stage Choice: ['A: The color of objects in the last panel is the sum of the colors in the previous two panels.', 'B: The color of objects gradually brightens by a constant amount each time.', 'C: The color of objects gradually darkens by a constant amount each time.', 'D: The color of objects in the last panel is the difference between the colors in the previous two panels.']
Current Stage Ground Truth: B
Current Stage Logits: {'A': 20.0, 'B': 21.375, 'C': 20.5, 'D': 19.75}
Current Stage Generated Answer: B
---------------------------
Accuracy: 1.0
MSEval: 1.066
MSEval Random Baseline: 1.0

Listing 2: An instance of high accuracy but low MSEval occurs since the LLM NVLM-D-72B generates a current-stage answer consistent with the ground truth, while earlier dependent stages produce inconsistent results.
---------------------------
Dependent Stage Name: single_panel_1_left
Dependent Stage Question: What is the shape of the object in the left part of the panel ?
Dependent Stage Choice: ['A: circle', 'B: hexagon', 'C: triangle', 'D: square']
Dependent Stage Ground Truth: B
Dependent Stage Logits: {'A': 17.125, 'B': 24.5, 'C': 16.0, 'D': 16.0}
Dependent Stage Generated Answer: B
---------------------------
Dependent Stage Name: single_panel_2_left
Dependent Stage Question: What is the shape of the object in the left part of the panel ?
Dependent Stage Choice: ['A: triangle', 'B: pentagon', 'C: square', 'D: hexagon']
Dependent Stage Ground Truth: B
Dependent Stage Logits: {'A': 17.5, 'B': 23.875, 'C': 17.125, 'D': 15.8125}
Dependent Stage Generated Answer: B
---------------------------
Dependent Stage Name: single_panel_3_left
Dependent Stage Question: What is the shape of the object in the left part of the panel ?
Dependent Stage Choice: ['A: hexagon', 'B: pentagon', 'C: square', 'D: triangle']
Dependent Stage Ground Truth: C
Dependent Stage Logits: {'A': 17.375, 'B': 16.875, 'C': 24.125, 'D': 17.375}
Dependent Stage Generated Answer: C
---------------------------
Dependent Stage Name: two_panels_1_2_left
Dependent Stage Question: Consider only the left part of the two panels in the image. Is the shape of all the objects in the left panel have the same, more, or fewer edges compared to the objects in the right panel? If the shapes within either panel are already different from each other, select 'Not Comparable.' (Note: The edge number increases in the following order: triangle, square, pentagon, hexagon, circle)
Dependent Stage Choice: ['A: Not comparable', 'B: Fewer ', 'C: The same', 'D: More']
Dependent Stage Ground Truth: D
Dependent Stage Logits: {'A': 21.625, 'B': 20.875, 'C': 20.0, 'D': 22.25}
Dependent Stage Generated Answer: D
---------------------------
Dependent Stage Name: two_panels_2_3_left
Dependent Stage Question: Consider only the left part of the two panels in the image. Is the shape of all the objects in the left panel have the same, more, or fewer edges compared to the objects in the right panel? If the shapes within either panel are already different from each other, select 'Not Comparable.' (Note: The edge number increases in the following order: triangle, square, pentagon, hexagon, circle)
Dependent Stage Choice: ['A: The same', 'B: Not comparable', 'C: More ', 'D: Fewer']
Dependent Stage Ground Truth: C
Dependent Stage Logits: {'A': 20.875, 'B': 21.75, 'C': 21.875, 'D': 20.25}
Dependent Stage Generated Answer: C
---------------------------
Current Stage:
Current Stage Name: one_row_left
Current Stage Question: Look at the three panels in the image from left to right, paying attention only to the left portions of each panel, and identify the rule that controls the shape of objects. (Note: The edge number increases in the following order: triangle, square, pentagon, hexagon, circle)
Current Stage Choice: ['A: The edge number of shape gradually decreases by 1.', 'B: The edge number of shape gradually increases by 1.', 'C: No clear rule is present.', 'D: The shape remains constant.']
Current Stage Ground Truth: A
Current Stage Logits: {'A': 21.75, 'B': 22.75, 'C': 19.25, 'D': 16.625}
Current Stage Generated Answer: B
---------------------------
Accuracy: 0.0
MSEval: 2.506839853582758
MSEval Random Baseline: 1.0

Listing 3: An instance of low accuracy but high MSEval arises as the LLM NVLM-D-72B generates a current-stage answer inconsistent with the ground truth, despite earlier dependent stages producing consistent results.
---------------------------
Dependent Stage Name: single_panel_1_left
Dependent Stage Question: What is the shape of the object in the left part of the panel ?
Dependent Stage Choice: ['A: circle', 'B: hexagon', 'C: triangle', 'D: square']
Dependent Stage Ground Truth: B
Dependent Stage Logits: {'A': 21.125, 'B': 26.375, 'C': 20.375, 'D': 22.375}
Dependent Stage Generated Answer: B
---------------------------
Dependent Stage Name: single_panel_2_left
Dependent Stage Question: What is the shape of the object in the left part of the panel ?
Dependent Stage Choice: ['A: triangle', 'B: pentagon', 'C: square', 'D: hexagon']
Dependent Stage Ground Truth: B
Dependent Stage Logits: {'A': 22.5, 'B': 25.5, 'C': 21.25, 'D': 24.625}
Dependent Stage Generated Answer: B
---------------------------
Dependent Stage Name: single_panel_3_left
Dependent Stage Question: What is the shape of the object in the left part of the panel ?
Dependent Stage Choice: ['A: hexagon', 'B: pentagon', 'C: square', 'D: triangle']
Dependent Stage Ground Truth: C
Dependent Stage Logits: {'A': 22.875, 'B': 21.125, 'C': 27.5, 'D': 23.625}
Dependent Stage Generated Answer: C
---------------------------
Dependent Stage Name: two_panels_1_2_left
Dependent Stage Question: Consider only the left part of the two panels in the image. Is the shape of all the objects in the left panel have the same, more, or fewer edges compared to the objects in the right panel? If the shapes within either panel are already different from each other, select 'Not Comparable.' (Note: The edge number increases in the following order: triangle, square, pentagon, hexagon, circle)
Dependent Stage Choice: ['A: Not comparable', 'B: Fewer', 'C: The same', 'D: More']
Dependent Stage Ground Truth: D
Dependent Stage Logits: {'A': 30.5, 'B': 30.25, 'C': 30.75, 'D': 30.875}
Dependent Stage Generated Answer: D
---------------------------
Dependent Stage Name: two_panels_2_3_left
Dependent Stage Question: Consider only the left part of the two panels in the image. Is the shape of all the objects in the left panel have the same, more, or fewer edges compared to the objects in the right panel? If the shapes within either panel are already different from each other, select 'Not Comparable.' (Note: The edge number increases in the following order: triangle, square, pentagon, hexagon, circle)
Dependent Stage Choice: ['A: The same', 'B: Not comparable', 'C: More', 'D: Fewer']
Dependent Stage Ground Truth: C
Dependent Stage Logits: {'A': 30.375, 'B': 29.75, 'C': 30.375, 'D': 30.25}
Dependent Stage Generated Answer: C
---------------------------
Current Stage:
Current Stage Name: one_row_left
Current Stage Question: Look at the three panels in the image from left to right, paying attention only to the left portions of each panel, and identify the rule that controls the shape of objects. (Note: The edge number increases in the following order: triangle, square, pentagon, hexagon, circle)
Current Stage Choice: ['A: The edge number of shape gradually decreases by 1.', 'B: The edge number of shape gradually increases by 1.', 'C: No clear rule is present.', 'D: The shape remains constant.']
Current Stage Ground Truth: A
Current Stage Logits: {'A': 32.75, 'B': 33.0, 'C': 31.25, 'D': 30.5}
Current Stage Generated Answer: B
---------------------------
Accuracy: 0.0
MSEval: 2.228
MSEval Random Baseline: 1.0

Listing 4: An instance of low accuracy but high MSEval arises as the LLM Intern-VL2-8B generates a current-stage answer inconsistent with the ground truth, despite earlier dependent stages producing consistent results.
---------------------------
Dependent Stage Name: single_panel_1_right
Dependent Stage Question: Are all objects in the right part of the panel of the same size?
Dependent Stage Choice: ['A: No', 'B: Yes', 'C: Only one object']
Dependent Stage Ground Truth: C
Dependent Stage Logits: {'A': 31.125, 'B': 31.125, 'C': 29.625}
Dependent Stage Generated Answer: A
---------------------------
Dependent Stage Name: single_panel_2_right
Dependent Stage Question: Are all objects in the right part of the panel of the same size?
Dependent Stage Choice: ['A: No', 'B: Yes', 'C: Only one object']
Dependent Stage Ground Truth: C
Dependent Stage Logits: {'A': 31.0, 'B': 31.125, 'C': 29.5}
Dependent Stage Generated Answer: B
---------------------------
Dependent Stage Name: single_panel_3_right
Dependent Stage Question: Are all objects in the right part of the panel of the same size?
Dependent Stage Choice: ['A: Only one object', 'B: No', 'C: Yes']
Dependent Stage Ground Truth: A
Dependent Stage Logits: {'A': 31.375, 'B': 31.375, 'C': 31.625}
Dependent Stage Generated Answer: C
---------------------------
Dependent Stage Name: two_panels_1_2_right
Dependent Stage Question: Consider only the right part of the two panels in the image. Is the size of all the objects in the left panel the same as, smaller or larger than the objects in the right panel? If the sizes within either panel are already different from each other, select 'Not Comparable.
Dependent Stage Choice: ['A: Not comparable', 'B: Smaller', 'C: Larger', 'D: The same']
Dependent Stage Ground Truth: C
Dependent Stage Logits: {'A': 30.875, 'B': 30.75, 'C': 30.25, 'D': 30.75}
Dependent Stage Generated Answer: A
---------------------------
Dependent Stage Name: two_panels_2_3_right
Dependent Stage Question: Consider only the right part of the two panels in the image. Is the size of all the objects in the left panel the same as, smaller or larger than the objects in the right panel? If the sizes within either panel are already different from each other, select 'Not Comparable.
Dependent Stage Choice: ['A: Not comparable', 'B: Smaller', 'C: The same', 'D: Larger']
Dependent Stage Ground Truth: D
Dependent Stage Logits: {'A': 30.25, 'B': 30.125, 'C': 30.5, 'D': 30.125}
Dependent Stage Generated Answer: C
---------------------------
Current Stage:
Current Stage Name: one_row_right
Current Stage Question: Analyze the three panels in the image from left to right, concentrating only on the right areas of each panel, and determine the rule that dictates the size of objects.
Current Stage Choice: ['A: The size of objects gradually decreases by a constant amount each time.', 'B: The size of objects in the last panel is the difference between the sizes in the previous two panels.', 'C: The size remains constant.', 'D: Three distinct sizes across panels, rotating through each possible permutation of these sizes.']
Current Stage Ground Truth: D
Current Stage Logits: {'A': 31.125, 'B': 30.375, 'C': 31.0, 'D': 31.375}
Current Stage Generated Answer: D
---------------------------
Accuracy: 1.0
MSEval: 0.955
MSEval Random Baseline: 1.0

Listing 5: An instance of high accuracy but low MSEval occurs since the LLM Intern-VL2-8B generates a current-stage answer consistent with the ground truth, while earlier dependent stages produce inconsistent results.
---------------------------
Dependent Stage Name: single_panel_1_right
Dependent Stage Question: What is the shape of the object in the right part of the panel?
Dependent Stage Choice: ['A: circle', 'B: square', 'C: hexagon', 'D: pentagon']
Dependent Stage Ground Truth: C
Dependent Stage Logits: {'A': 0, 'B': 0, 'C': 30.625, 'D': 0}
Dependent Stage Generated Answer: C
---------------------------
Dependent Stage Name: single_panel_2_right
Dependent Stage Question: What is the shape of the object in the right part of the panel?
Dependent Stage Choice: ['A: pentagon', 'B: triangle', 'C: square', 'D: circle']
Dependent Stage Ground Truth: B
Dependent Stage Logits: {'A': 0, 'B': 30.5, 'C': 0, 'D': 0}
Dependent Stage Generated Answer: B
---------------------------
Dependent Stage Name: single_panel_3_right
Dependent Stage Question: What is the shape of the object in the right part of the panel?
Dependent Stage Choice: ['A: triangle', 'B: hexagon', 'C: square', 'D: circle']
Dependent Stage Ground Truth: C
Dependent Stage Logits: {'A': 0, 'B': 0, 'C': 30.875, 'D': 0}
Dependent Stage Generated Answer: C
---------------------------
Dependent Stage Name: two_panels_1_2_right
Dependent Stage Question: Consider only the right part of the two panels in the image. Is the shape of all the objects in the left panel have the same, more, or fewer edges compared to the objects in the right panel? If the shapes within either panel are already different from each other, select 'Not Comparable.' (Note: The edge number increases in the following order: triangle, square, pentagon, hexagon, circle)
Dependent Stage Choice: ['A: The same', 'B: More', 'C: Fewer', 'D: Not comparable']
Dependent Stage Ground Truth: B
Dependent Stage Logits: {'A': 0, 'B': 30.0, 'C': 0, 'D': 0}
Dependent Stage Generated Answer: B
---------------------------
Dependent Stage Name: two_panels_2_3_right
Dependent Stage Question: Consider only the right part of the two panels in the image. Is the shape of all the objects in the left panel have the same, more, or fewer edges compared to the objects in the right panel? If the shapes within either panel are already different from each other, select 'Not Comparable.' (Note: The edge number increases in the following order: triangle, square, pentagon, hexagon, circle)
Dependent Stage Choice: ['A: Not comparable', 'B: Fewer', 'C: More', 'D: The same']
Dependent Stage Ground Truth: B
Dependent Stage Logits: {'A': 0, 'B': 28.25, 'C': 0, 'D': 0}
Dependent Stage Generated Answer: B
---------------------------
Current Stage:
Current Stage Name: one_row_right
Current Stage Question: Inspect the three panels in the image from left to right, focusing exclusively on the right parts of each panel, and uncover the rule that governs the shape of objects. (Note: The edge number increases in the following order: triangle, square, pentagon, hexagon, circle)
Current Stage Choice: ['A: The shape remains constant.', 'B: Three distinct shapes across panels, rotating through each possible permutation of these shapes.', 'C: No clear rule is present.', 'D: The edge number of shape gradually increases by 1.']
Current Stage Ground Truth: B
Current Stage Logits: {'A': 0, 'B': 0, 'C': 31.67, 'D': 0}
Current Stage Generated Answer: C
---------------------------
Accuracy: 0.0
MSEval: 2.345087186311839
MSEval Random Baseline: 1.0

Listing 6: An instance of low accuracy but high MSEval arises as the LLM Qwen2-VL-72B generates a current-stage answer inconsistent with the ground truth, despite earlier dependent stages producing consistent results.

Reviewer paper summaries: The paper introduces a multi-stage abstract visual reasoning benchmark based on the RAVEN dataset, aimed to evaluate intermediate reasoning steps in multimodal large language models (MLLMs). It also proposes a new metric, MSEval, which assesses the correctness of intermediate steps.
Summary of strengths: **Comprehensive Experiments**: The paper presents extensive experiments across 17 multimodal large language models (MLLMs), including both open-source and closed-source variants.

**Well-motivated Problem**: The authors highlight the gap in existing benchmarks that focus solely on end-to-end single-step reasoning, overlooking the multi-stage nature of human-like reasoning.
Summary of weaknesses: **Limited Question Types in Perception/Comparison:**
The questions are narrowly tailored to the RAVEN dataset (attribute, one row/two row rule deduction), limiting the generality and applicability of the approach.

**Overreliance on RAVEN Dataset:** 
While RAVEN is a foundational dataset for AVR tasks, the absence of a new dataset limits the benchmark's novelty. Expanding the benchmark with diverse datasets, particularly those containing varied reasoning tasks beyond geometric patterns, could enhance its utility.

**Restricted Applicability of MSEval:**
The reliance of MSEval on access to logits restricts its applicability to open-source models. This significantly reduces its potential impact in real-world settings where closed-source models dominate.

**Lack of justification of MSEval:** 
MSEval captures joint probability as a measure of correctness but lacks deeper justification or exploration of why this metric was chosen over others.
Comments, suggestions and typos: N/A
Overall assessment scores: 2.5