LogicVista: Multimodal LLM Logical Reasoning Benchmark in Visual Contexts

## Abstract

Abstract We propose LogicVista, an evaluation benchmark that assesses the integrated logical reasoning capabilities of multimodal large language models (MLLMs) in Vis ual contexts. Recent advancements in MLLMs have demonstrated various fascinating abilities, from crafting poetry based on an image to performing mathematical reasoning. However, there is still a lack of systematic evaluation of MLLMs’ proficiency in
logical reasoning tasks, which are essential for activities like navigation and puzzle-solving. Thus we evaluate general logical cognition abilities across 5 logical reasoning tasks encompassing 9 different capabilities, using a sample of 448 multiple-choice questions. Each question is annotated with the correct answer and the human-written reasoning behind the selection, enabling both open-ended and multiple-choice evaluation. A total of 8 MLLMs are comprehensively evaluated using LogicVista. Code and Data Available at https://github.com/Yijia-Xiao/LogicVista .

## 1 Introduction

Recent advancements in Large Language Models (LLMs) are gradually turning the vision of a generalist AI agent into reality. These models exhibit near-human expert-level performance across a variety of tasks and have recently been augmented with visual understanding capabilities, enabling them to tackle even more complex visual challenges. This branch of work, led by proprietary projects such as GPT-4 [] and Flamingo [], as well as open-source efforts like LLaVA [], Mini-GPT4 [], enhances existing LLMs by incorporating visual comprehension. These models, known as Multimodal Large Language Models (MLLMs), use LLMs as the foundation for processing information and generating reasoned outcomes [], thereby bridging the gap between language and vision.

Recent MLLMs have demonstrated a range of impressive abilities, such as writing poems based on an image [], engaging in mathematical reasoning [], and even aiding in medical diagnosis []. To evaluate the performance of these models, various benchmarks have been proposed, as shown in Figure. 1 targeting the performance on common tasks such as objects recognition [], text understanding in images [], or mathematical problem solving []. However, as seen in Figure. 1, there is a notable shortage of benchmarks for MLLMs’ abilities in critical logical reasoning tasks that underlie most tasks. Perception and reasoning are two representative abilities of high-level intelligence that are used in unison during human problem-solving processes.

Many current MLLM datasets have focused solely on perception tasks, which require fact retrieval where the MLLM identifies and retrieve relevant information from a scene. However, complex multimodal reasoning, such as interpreting graphs [], everyday reasoning, critical thinking, and problem-solving [, ] requires a combination of perception and logical reasoning. Proficiency in these reasoning skills is a reliable indicator of cognitive capabilities required for performing specialized or routine tasks across different domains. To our knowledge, MathVista [] is the only benchmark that attempts to evaluate multimodal logical reasoning, but its scope is limited to mathematical-related reasoning. For a better understanding of how MLLMs perform on general reasoning tasks, there is a need for a comprehensive and general visual reasoning benchmark.

Figure: Figure 1: Capabilities and reasoning skills of various existing benchmarks. Traditional benchmarks seldom assess reasoning skills, whereas LogicVista emphasizes the fundamental capacities necessary for solving specific problems, going beyond simple recognition or math tasks.
Refer to caption: extracted/5714025/figures/ours1.png

We argue that a universal comprehensive evaluation benchmark should have the following characteristics: (1) cover a wide range of logical reasoning tasks, including deductive, inductive, numeric, spatial, and mechanical reasoning; (2) present information in both graphical and Optical Character Recognition (OCR) formats to accommodate different types of data inputs; and (3) facilitate convenient quantitative analysis for rigorous assessment and comparison of model performance.

To this end, we present a comprehensive MLLM evaluation benchmark, named LogicVista, which meets all these criteria:

- •
LogicVista covers 5 representative categories of logical reasoning tasks: inductive ($sample=107$), deductive ($sample=93$), numerical ($sample=95$), spatial ($sample=79$), and mechanical ($sample=74$).
- •
LogicVista includes a variety of capabilities, ranging from diagrams ($sample=330$), OCR, ($sample=234$), patterns ($sample=105$), graphs ($sample=67$), tables ($sample=70$), 3D shapes ($samples=45$), puzzles ($samples=256$), sequences ($samples=76$), and physics ($samples=69$).
- •
All images, instructions, solution, and reasoning are manually annotated and validated.
- •
With our instruction design “please select from A, B, C, D, and E." and our LLM answer evaluator, we can assess different reasoning skills and capabilities and easily perform quantitative statistical analysis based on the natural language output of MLLMs. Additionally, We provide more in-depth human-written explanations for why each answer is correct, allowing for thorough open-ended evaluation.

As shown in Figure. 1, LogicVista covers a wide range of reasoning capabilities and evaluates them comprehensively. For instance, answering the question “Which of these images is the top view of the given object" in Figure 1(b) requires not only recognizing the objects’ orientation but also the ability to spatially reason over the object from a different perspective. Since these questions and diagrams are presented without context, they effectively probe the MLLM’s underlying ability rather than relying on contextual cues from the surrounding real-life environment.

Furthermore, we provide two evaluation strategies with our annotations: multiple-choice question (MCQ) evaluation and open-ended evaluation. Our annotation of MCQ choices along with our LLM evaluator allows quick evaluations of answers provided by MLLMs. Additionally, our annotation of the reasoning and thought process behind each MCQ enables open-ended evaluation, capturing the nuances of the MLLM responses and identifying which reasoning steps were correct or incorrect.

We comprehensively evaluate the performance of 8 representative open and closed source MLLMs on 448 tasks across 5 main logical reasoning categories. LogicVista’s evaluation strategy allows users to see a detailed breakdown of an MLLM’s performance on each reasoning skill and capability. This approach provides more insights than a single overall score, enabling users to better understand the specific skills in which a model excels or needs improvement.

## 2 Related Works

**Table 1: Comparision with related vision-language benchmarks.**
|  | VQAv2 [, ] | COCO  [] | TextCaps [] | Contextual [] | MM-vet [] | MathVista [] | VisIT-Bench [] | LogicVista |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Number of Logical Reasoning Skills Tested | 0 | 0 | 1 | 1 | 1 | 2 | 1 | 5 |
| Number of Multimodal Capabilities Tested | 1 | 1 | 2 | 2 | 6 | 12 | 2 | 9 |
| Dataset Size | 204,721 | 330,000 | 28,000 | 506 | 217 | 6,141 | 592 | 448 |
| Scene and Object Recognition | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
| Inductive Reasoning | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ | ✗ | ✓ |
| Deductive Reasoning | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ |
| Numerical Reasoning | ✗ | ✗ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
| Spatial Reasoning | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ |
| Mechanical Reasoning | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ |
| Answer Choice Explanations | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ | ✗ | ✓ |
| Human Annotation | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
| Human Evaluation | ✗ | ✓ | ✓ | ✓ | ✗ | ✓ | ✓ | ✗ |
| Auto/GPT-4 Evaluation | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
| Open-ended Evaluation | ✗ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |

Multimodal Language Models
The field of vision-language models [, , , , , , , , , ] has made significant progress towards achieving a cohesive understanding and generation of both visual and linguistic information. This progress is largely driven by the remarkable generalization and quality capabilities of recent large language models (LLMs) [, , , ]. As a result, there has been a surge in the development of MLLMs that aim to integrate the diverse capabilities of vision and language for complex multimodal tasks.

Efforts to create these multimodal generalist systems include enhancing LLMs with multi-sensory processing abilities, as demonstrated by innovative projects like Frozen [], Flamingo [], PaLM-E [], and GPT-4 []. Recent releases of open-source LLMs [, , ] have further propelled research in this field, leading to the development of OpenFlamingo [], LLaVA [], MiniGPT-4 [], Otter [], InstructBLIP [], among others [, , ]. Additionally, multimodal agents [, , ] have been explored for their ability to link various vision tools with LLMs [, ], aiming to enhance integrated vision-language capabilities

Vision-Language Benchmarks
Traditional vision-language benchmarks have focused on assessing specific capabilities, including visual recognition [], generating image descriptions [, ], and other specialized functions such as understanding scene text [, , ], commonsense reasoning [], mathematical reasoning [], instruction following  [], and external knowledge incorporation []. While some benchmarks incorporate reasoning  [], they are often presented in real-life contexts, which may reduce the task to mere recognition based on contextual cues.

The emergence of general MLLMs has highlighted the need for updated vision-language benchmarks that encompass complex multimodal tasks requiring comprehensive vision-language skills. Our benchmark, LogicVista, aligns closely with recent evaluation studies like MM-Vet and MMBench [, ], which aim to provide thorough evaluations of MLLMs through well-designed evaluation samples. A key distinction of LogicVista lies in its focus on integrated vision-language capabilities, offering deeper insights beyond mere model rankings.

LLM-Based Evaluation.
LogicVista adopts an open-ended LLM-based evaluation approach, which facilitates the generation and assessment of diverse answer styles and question types beyond the limitations of binary or multiple-choice responses. This innovative method leverages the capabilities of large language models (LLMs) for comprehensive model evaluation, a technique that has been effectively applied in natural language processing (NLP) tasks [, , , ]. Our findings indicate that this LLM-based evaluation framework is not only versatile but also robust, enabling a unified and flexible assessment across various modalities. By accommodating a wide range of answer styles and question types, this approach enhances evaluation depth and breadth, which contributes to a more thorough understanding of model performance.

## 3 Data annotation and organization

Figure: (a)
Refer to caption: x1.png

### 3.1 Data Sources

To ensure the integrity and quality of LogicVista’s evaluations, we have implemented a stringent data collection and curation process specifically designed to prevent data leakage detailed in Figure. 2. Our approach involves sourcing and annotating our samples from proprietary sources that require licenses, registration, payment, or a combination of these barriers to access. This methodology is critical to minimizing the risk that our benchmark data has been previously seen or utilized in the training of other multi-modal models. We prioritized sourcing data from closed sources to further reduce the potential of data leakage.

- •
Licensed Access: We obtain data from sources that require formal licensing, ensuring the data is used solely for research purposes and not freely available for general use or scraping on the internet.
- •
Registration Requirements: Some of our data sources mandate user registration and account verification, adding an additional layer of access control to ensure that the data remain restricted and not easily accessible.
- •
Paid Content: We utilize paid sources where content is accessible only through purchase or subscription, further restricting the data from being freely available on the internet.

Additionally, we obtained permission from the creators of IQ tests and other evaluation materials included in our dataset. This permission specifically allows the use of their content for research purposes, ensuring the data’s legitimacy and accuracy.

### 3.2 Annotation and Data Collection

LogicVista consists of images designed to assess the underlying reasoning capacities of MLLMs. Using real-life scenes as explicit tests of logical reasoning can be challenging, as they often contain context clues that AI agent can use to deduce answers without directly reasoning through the scene. Therefore, LogicVista presents multiple-choice questions across 9 explicit capabilities that specify the type of reasoning required, without the additional context of real-life scenes typically found in intelligence and reasoning tests. The dataset is manually collected and annotated from various licensed intelligence test sources. Over a period of 3 months, 5 annotators extracted images, correct answers, and explanations when available. The explanations detailing the reasoning behind answer choices were extensively annotated and cross-validated among annotators, ensuring data integrity through multiple rounds of quality checks. The data is structured in JSON format to facilitate easy retrieval and processing in our evaluation pipeline. For our evaluation, we focused on summarizing five reasoning skills spanning two multimodal capabilities. For detailed examples of these reasoning skills and capabilities, please refer to Appendix. A and Appendix. B.

Figure: Figure 3: Proportion of reasoning skills and capabilities. On the left is the proportion of questions belonging to each reasoning skill. These proportions add up to $100\%$ as each skill is independent of another. On the right is the proportion of questions belonging to each multi-modal capability. These do not add up to $100\%$ due to the use of mixed capabilities.
Refer to caption: x3.png

#### 3.2.1 Capabilities

We distinguish multimodal capabilities from reasoning skills, considering these capabilities fundamental to understanding a multimodal scene and extracting information. Capabilities refer to the modalities through which logical reasoning questions are delivered. To ensure comprehensive coverage in LogicVista, we have defined a diverse array of 9 capabilities for evaluation. This diversity guarantees that LogicVista thoroughly assess various logical situations that an MLLM may encounter in everyday reasoning.
Figure 3 demonstrates how LogicVista contains a balanced mix of capabilities, including samples that utilize multiple capabilities to solve a problem.

- •
Diagrams: Simple flow diagrams and logical diagrams (e.g., Markov diagrams).
- •
OCR: Text embedded within an image (e.g., “gas station” in an image of a gas station).
- •
Patterns: Repeated sequences such as a series of diagrams, numbers, shapes, and objects (e.g., identifying patterns in how a box moves through repeated images of boxes).
- •
Graphs: Mathematical graphs with axes (e.g., graphs of $y=2x$ and $y=x^{2}$).
- •
Tables: Data tables (e.g., pie charts and T-tables).
- •
3D Shapes: The ability to understand and differentiate 3D objects from 2D ones (e.g., recognizing a 3D shape in different rotations).
- •
Puzzles: Puzzles with logical implications embedded within the shapes (e.g., chess puzzles).
- •
Sequences: Sequences of related items or objects (e.g., predicting the next item in a sequence).
- •
Physics: Situations involving physics (e.g., diagrams of projectile motion).

#### 3.2.2 Reasoning Skills

The reasoning skills of interest for this benchmark are based on common critical thinking and problem-solving skills used by humans in various contexts. For our evaluation, we summarize these into the following five skills. For our evaluation, we summarize these to include the following 5 skills. As seen in Figure 3, LogicVista encompasses a wide range of all these reasoning skills:

- •
Inductive Reasoning:
The ability to infer the next entry in a pattern given a set of observations. This involves making generalizations based on specific observations to form an educated guess. It moves from many specific observations to a generalization. For example, observing that John gets a stomach ache when he eats dairy products leads to the inductive conclusion that he is likely lactose intolerant.
- •
Deductive Reasoning:
The ability to conclude a specific case from a general principle or pattern. This involves moving from the general to the specific. For example, from the statement “all men are mortal,” one can deduce that “John is mortal” because John is a man.
- •
Numerical Reasoning:
The ability to read arithmetic problems in an image and solve the math equations. For example, given the equation “10 + 10 = ?,” the answer would be “20.”
- •
Spatial Reasoning:
The ability to understand the spatial relationships between objects and patterns and reason with those relationships. For example, seeing an unfolded box and understanding what the box would look like when folded.
- •
Mechanical Reasoning:
The ability to recognize a physical system and solve equations based on that system or answer questions about it. For example, seeing a set of three gears and understanding which gears will turn clockwise and which will turn counterclockwise.

### 3.3 LLM-based Multiple Choice Answer Extractor

Figure: Figure 4: Pipeline of evaluating open-ended LMM outputs using MCQ answer choice extraction.
Refer to caption: x4.png

LLMs generate non-deterministic and open-ended responses [, ], making direct evaluation challenging. To address this, we use an LLM evaluator to compare these open-ended responses to our annotations as detailed in 4. This evaluator can assess both MCQ answer choices and the MLLM’s reasoning behind those selections, as both elements are included in our annotations. This step is achieved by feeding various contexts such as the question, and the available choices, along with the LLM-generated answers to an extraction LLM (GPT, LLaMA, etc.). Based on the provided rich context, the LLM can generate the selected letter answer choice. The final output is also repeatedly validated and if the validation fails, the extraction repeats with the provided feedback to obtain correct results.

## 4 Evaluation Setup

**Table 2: Summary of the MLLMs used for evaluations in this study.**
| Model | Size | Language Model | Vision Model |
| --- | --- | --- | --- |
| LLaVA-Vicuna-7B | 7B | Vicuna-7B | CLIP ViT-L/14 |
| LLaVA-Vicuna-13B | 13B | Vicuna-13B | CLIP ViT-L/336px |
| LLaVA-NeXT-Mistral-7B | 7B | Mistral-7B | CLIP ViT-L/14 |
| LLaVA-NeXT-Vicuna-7B | 7B | Vicuna-7B | CLIP ViT-L/14 |
| LLaVA-NeXT-Vicuna-13B | 13B | Vicuna-13B | CLIP ViT-L/336px |
| LLaVA-NeXT-Nous-Hermes-Yi-34B | 34B | Nous Hermes 2-Yi-34B | CLIP ViT-L/336px |
| MiniGPT-4-7B | 7B | Vicuna-7B | BLIP-2 Q-Former |
| MiniGPT-4-13B | 13B | Vicuna-13B | BLIP-2 Q-Former |
| Otter-9B | 9B | MPT-7B | CLIP ViT-L/14 |
| GPT-4 Vision | N/A(^1^11N/A: Not disclosed) | N/A | N/A |
| BLIP-2 | 2.7B | OPT-2.7B | EVA-ViT-G |
| Pix2Struct | 1.3B | ViT | ViT |
| InstructBLIP-Vicuna-7B | 7B | Vicuna-7B | BLIP-2 Q-Former |
| InstructBLIP-Vicuna-13B | 13B | Vicuna-13B | BLIP-2 Q-Former |
| InstructBLIP-FLAN-T5-xl | 3B | FLAN-T5 XL | BLIP-2 Q-Former |
| InstructBLIP-FLAN-T5-xxl | 11B | FLAN-T5 XXL | BLIP-2 Q-Former |

To evaluate the performance of MLLMs on LogicVista, we selected a range of representative models detailed in Table. 2. Specifically, we chose8 models for evaluation, including LLaVA [, ], MiniGPT4 [], Otter [], GPT-4 Vision [], BLIP-2 [], and InstructBLIP [] We also included pix2struct [] which has been fine-tuned to understand chart and diagram data.

Each model generated outputs using the LogicVista dataset. Our LLM-based multiple-choice extractor was then employed to isolate the multiple-choice selections from the MLLMs’ outputs (which often appear as full-sentence responses rather than single letters) and compare them to the ground truth answers. The overall logical reasoning score is calculated as follows:

$$ $S=\frac{\sum_{n=1}^{N}s_{i}}{N}*100\%$ (1) $$

Here, $S$ represents the overall score, $s_{i}$ indicate whether a sample $i$ is evaluated as correct or not (regardless of category), and $N$ is the total number of samples. The score for each reasoning skill subcategory is calculated as:

$$ $S_{LR}=\frac{\sum_{n=1}^{N_{LR}}s_{i}}{N_{LR}}*100\%$ (2) $$

where $S_{LR}$ represents the score for a specific reasoning skill category, $N_{LR}$ is the total number of samples in that reasoning skill category, and $s_{i}$ indicate whether a sample $i$ from that category was evaluated as correct. Similarly, the score for each multi-modal capability is calculated as:

$$ $S_{c}=\frac{\sum_{n=1}^{N_{c}}s_{i}}{N_{c}}*100\%$ (3) $$

where $S_{c}$ represents the score for a specific capability, $N_{c}$ is the total number of samples in that capability, and $s_{i}$ indicates whether a sample $i$ in the capability category is evaluated correctly.

## 5 LogicVista Benchmarking and Performance Interpretation

### 5.1 Logical Reasoning Skills

We present the performance results of various multimodal LLMs on LogicVista. Table 3 outlines the outcome for these models across five logical reasoning categories. We analyzed models of different architectures and sizes, benchmarking them against a random baseline that assumes an average of five choices per question in the LogicVista dataset. Our findings indicate that many models perform below expectations, often yielding results that are worse than random guessing. This outcome is somewhat anticipated, given that most training data for multimodal LLMs and LLMs are derived from classical computer vision datasets such as COCO, which focus on recognition tasks rather than complex reasoning.

Traditional benchmarks typically emphasize recognition tasks, resulting in a lack of emphasis on reasoning tasks during both training and evaluation phases. This is evident from the observation that while many models excel on recognition-based benchmarks like COCO, TextVQA, and MM-vet, they often struggle to outperform a random baseline on logical reasoning tasks.

**Table 3: LogicVista evaluation results for various multimodal LLMs on each logical reasoning skill are presented as $\%$, with the highest possible accuracy being $100\%$. The highest-scoring models are highlighted in green and the lower-scoring models are highlighted in yellow.**
| Model | Inductive | Deductive | Numerical | Spatial | Mechanical |
| --- | --- | --- | --- | --- | --- |
| LLAVA7B | 29.91% | 29.03% | 26.32% | 25.32% | 36.49% |
| LLAVA13B | 18.69% | 31.18% | 20.00% | 27.85% | 24.32% |
| otter9B | 31.78% | 24.73% | 18.95% | 18.99% | 21.62% |
| GPT4 | 23.36% | 54.84% | 24.21% | 21.52% | 41.89% |
| BLIP2 | 17.76% | 23.66% | 23.16% | 24.05% | 18.92% |
| LLAVANEXT-7B-mistral | 16.82% | 34.41% | 23.16% | 21.52% | 22.97% |
| miniGPTvicuna7B | 10.28% | 9.68% | 7.37% | 3.80% | 27.03% |
| miniGPTvicuna13B | 13.08% | 23.66% | 10.53% | 10.13% | 17.57% |
| pix2struct | 12.15% | 6.45% | 2.11% | 7.59% | 17.57% |
| instructBLIP-vicuna-7B | 4.67% | 21.51% | 24.21% | 2.53% | 22.97% |
| instructBLIP-vicuna-13B | 3.74% | 10.75% | 18.95% | 5.06% | 17.57% |
| instructBLIP-flan-t5-xl | 23.36% | 22.58% | 22.11% | 7.59% | 33.78% |
| instructBLIP-flan-t5-xxl | 17.76% | 30.11% | 24.21% | 20.25% | 22.97% |
| LLAVANEXT-7B-vicuna | 26.17% | 21.51% | 25.26% | 27.85% | 29.73% |
| LLAVANEXT-13B-vicuna | 22.43% | 22.58% | 26.32% | 26.58% | 25.68% |
| LLAVANEXT-34B-NH | 20.56% | 52.69% | 30.53% | 24.05% | 40.54% |

Upon closer examination, we find that models perform best on deductive, numerical, and mechanical reasoning tasks. These types of reasoning are more prevalent in real-life scenarios, which makes models more adept at handling them. For example, deductive reasoning can be applied in predicting a character’s actions based on a scene, while numerical reasoning is crucial in solving arithmetic visual tasks. Mechanical reasoning involves understanding physical principles and interactions.

In contrast, induction and spatial reasoning are less frequently encountered in standard training data, potentially explaining the lower performance of models in these areas. These insights underscore the necessity for enhanced training and evaluation methodologies that prioritize reasoning tasks to bolster the logical reasoning capabilities of multimodal LLMs.

### 5.2 Visual Capabilities

In Table 4, we present the results of multimodal LLMs on logical reasoning tasks across diagrammatic and OCR mediums. Generally, we observe that OCR tasks tend to perform better than diagrammatic tasks. This difference stems from the nature of traditional computer vision tasks, which often prioritize recognizing prominent objects (“landmarks”) in a scene, such as distinct cars, planes, people, or balls. Diagrams, in contrast, lack such prominent features and mainly consist of lines and shapes, making it challenging for models to extract intricate relationships between objects.

In OCR tasks, once the text is accurately extracted from the image, the remainder of the reasoning task relies on the underlying LLM’s ability to process and interpret the content. This process typically bypasses the complexities of multimodal reasoning, leading to better performance on OCR tasks compared to diagrammatic tasks. These findings highlight the necessity for enhanced evaluation methodologies tailored to diagrammatic reasoning in multimodal LLMs, as current approaches may overlook critical details inherent in these types of tasks.

**Table 4: LogicVista evaluation results on various multimodal LLMs across each multi-modal capability. Accuracy results are presented as $\%$, with a maximum possible accuracy of $100\%$. Models achieving the highest scores are highlighted green, while lower-scoring models are highlighted yellow.**
| Model | Diagram | OCR | Patterns | Graphs | Tables | 3D Shapes | Puzzles | Sequences | Physics |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| LLAVA7B | 29.70% | 28.21% | 30.47% | 25.37% | 25.71% | 22.22% | 28.52% | 25.00% | 43.48% |
| LLAVA13B | 21.52% | 22.65% | 16.19% | 16.42% | 20.00% | 31.11% | 26.17% | 15.79% | 26.09% |
| otter9B | 23.64% | 20.51% | 30.48% | 14.93% | 22.86% | 13.33% | 26.17% | 26.32% | 24.64% |
| GPT4 | 26.06% | 39.74% | 20.95% | 20.90% | 22.86% | 31.11% | 31.25% | 28.95% | 47.83% |
| BLIP2 | 20.30% | 21.79% | 20.00% | 17.91% | 24.29% | 17.78% | 22.27% | 15.79% | 20.29% |
| LLAVANEXT-7B-mistral | 20.30% | 26.92% | 21.90% | 23.88% | 22.86% | 13.33% | 22.27% | 23.68% | 30.43% |
| miniGPTvicuna7B | 10.91% | 11.54% | 12.38% | 7.46% | 8.57% | 11.11% | 9.77% | 7.89% | 23.19% |
| miniGPTvicuna13B | 12.73% | 17.52% | 12.38% | 10.45% | 11.43% | 11.11% | 14.84% | 6.58% | 20.29% |
| pix2struct | 9.39% | 8.55% | 10.48% | 0.00% | 4.29% | 11.11% | 10.55% | 11.84% | 14.49% |
| instructBLIP-vicuna-7B | 11.82% | 21.37% | 7.62% | 22.39% | 22.86% | 6.67% | 10.55% | 0.00% | 24.64% |
| instructBLIP-vicuna-13B | 10.91% | 13.68% | 5.71% | 19.40% | 15.71% | 11.11% | 6.25% | 2.63% | 18.84% |
| instructBLIP-flan-t5-xl | 20.30% | 22.22% | 20.00% | 17.91% | 22.86% | 13.33% | 18.36% | 15.79% | 33.33% |
| instructBLIP-flan-t5-xxl | 20.91% | 24.36% | 22.86% | 20.90% | 25.71% | 20.00% | 21.09% | 14.47% | 21.74% |
| LLAVANEXT-7B-vicuna | 26.67% | 23.08% | 26.67% | 20.90% | 27.14% | 33.33% | 26.56% | 19.74% | 30.43% |
| LLAVANEXT-13B-vicuna | 25.15% | 22.65% | 23.81% | 20.90% | 27.14% | 26.67% | 24.61% | 15.79% | 27.54% |
| LLAVANEXT-34B-NH | 27.58% | 39.32% | 25.71% | 28.36% | 32.86% | 26.67% | 30.86% | 21.05% | 46.37% |

### 5.3 Relationship between Model Size and Performance

Figure 5 presents a comparative analysis of the model size and the average score achieved across all logical reasoning tasks and capabilities. Each plot includes a shaded region denoting the 95% confidence interval for the regression estimate, visually representing the uncertainty associated with the regression line. Dot sizes in the scatter plot indicate the number of models with identical parameter counts, illustrating the distribution density. This visual evidence strongly suggests a positive correlation between larger model sizes and improved performance in LogicVista. Specifically, as model size increases, performance tends to improve, indicating that larger models may have greater capacity to handle complex patterns and reasoning tasks.

## 6 Conclusion

Reasoning skills are critical for solving complex tasks and serve as the foundation for many challenges that humans expect AI agents to tackle. However, the exploration of reasoning abilities in multimodal LLM agents remains limited, with most benchmarks and training datasets predominantly focused on traditional computer vision tasks like recognition. For multimodal LLMs to excel in critical thinking and complex tasks, they must comprehend the underlying logical relationships inherent in these challenges.

Figure: Figure 5: correlation between model size and average accuracy. The scatter plot uses varying dot sizes to represent the density of models with identical sizes.
Refer to caption: x5.png

To address this gap, we introduce LogicVista, a novel benchmark designed to evaluate multimodal LLMs through a comprehensive assessment of logical reasoning capabilities. This benchmark features a dataset of 448 samples covering five distinct reasoning skills, providing a robust platform for evaluating cutting-edge multimodal models. Our evaluation aims to shed light on the current state of logical reasoning in multimodal LLMs.

To facilitate straightforward evaluation, we employ an LLM-based multiple-choice question-answer extractor, which helps mitigate the non-deterministic nature often associated with multimodal LLM outputs. While LogicVista primarily focuses on explicit logical reasoning tasks isolated from real-life contexts, this approach represents a crucial step toward understanding fundamental reasoning skills. However, it is equally important to explore how AI agents perform tasks that blend abstract reasoning with real-world scenarios, a direction that will guide our future research endeavors.

## Acknowledgements

We extend our sincere appreciation to the student researchers at the University of California, Los Angeles, for their diligent efforts in the manual annotation and validation of our dataset: Evan Li, Srinath Saikrishnan, Lawrence Li, and Oscar Cooper Stern.

## Appendix A Examples of LogicVista Logical Reasoning Data

**Table 5: Three samples requiring inductive logical reasoning skills.**
| (Case A) |  |
| --- | --- |
|  |  |
| Q: | Which choice (A, B, C, or D) completes the series? |
| Answer: | D |
| Reasoning: | In this example, there are two rules to be applied. The first is that the circle moves counter-clockwise in the hexagon. It follows that, in the following diagram, the circle will be in the upper corner of the hexagon, pointing to D as the answer. To confirm this, the second rule can be applied, according to which the position of the black triangle alternates between the bottom left and the top right. Thus, in the following diagram, the black triangle will need to be in the upper right corner of the hex. The answer is therefore definitely D. |
| Logical Reasoning Skill: | Inductive |
| Required capability | Diagram |

**Table 6: Three samples requiring inductive logical reasoning skills (Case B).**
| (Case B) |  |
| --- | --- |
|  |  |
| Q: | Two grids containing colored symbols and following a common rule are presented. In the block on the right, four additional grids are presented. The candidate must find the two grids that follow the same rule out of these four options. What options (A, B, C, or D) follow this same rule? |
| Answer: | B, D |
| Reasoning: | In this example, it is easy to see that the rule governing the two grids on the left is: that blue triangles are present in each of the two bottom lines. This rule is followed in the two grids on the right. |
| Logical Reasoning Skill: | Inductive |
| Required capability | Diagram, OCR |

**Table 7: Three samples requiring inductive logical reasoning skills (Case C).**
| (Case C) |  |
| --- | --- |
|  |  |
| Q: | Who is the odd-one-out? Select answers from A-I. |
| Answer: | G |
| Reasoning: | Element G constitutes the exception and is therefore the correct answer. |
| Logical Reasoning Skill: | Inductive |
| Required capability | Diagram |

**Table 8: Three samples requiring deductive logical reasoning skills (Case A).**
| (Case A) |  |
| --- | --- |
|  |  |
| Q: | Which is the correct answer according to the image? Select from 1-5? |
| Answer: | 3 |
| Reasoning: | Using deductive reasoning, the only logical answer is 3. To get to this answer, you need to simplify the given facts. All famous sports players are footballers, and all footballers are fit and healthy. We can not deduce that all footballers are famous sports people, as we have not got that information. We can not deduce that all famous people are fit and healthy, because the fact is about famous sports people. This is the logical answer. This information is not given; all footballers are fit and healthy but we can not logically link that all fit and healthy people are footballers. This is obviously incorrect, as gender is not mentioned at all in the question. |
| Logical Reasoning Skill: | Deductive |
| Required capability: | OCR |

**Table 9: Three samples requiring deductive logical reasoning skills (Case B).**
| (Case B) |  |
| --- | --- |
|  |  |
| Q: | What is the correct answer to the question in the image? Select from A-D. |
| Answer: | C |
| Reasoning: | The vast majority of swallows are blue so the answer must be C: there is a blue swallow. |
| Logical Reasoning Skill: | Deductive |
| Required capability: | OCR |

**Table 10: Three samples requiring deductive logical reasoning skills (Case C).**
| (Case C) |  |
| --- | --- |
|  |  |
| Q: | What is produced is determined by the people. Select from A, B, and C. (A) True (B)False (C)Insufficient Information? |
| Answer: | A |
| Reasoning: | Line 1 states that the people determine what is produced. Line 2 states that the government is made up of the people. Therefore, the people determine what is produced. This is a syllogism. Thus, this statement is true. |
| Logical Reasoning Skill: | Deductive |
| Required capability: | OCR |

**Table 11: Three samples requiring numerical logical reasoning skills (Case A).**
| (Case A) |  |
| --- | --- |
|  |  |
| Q: | Which share had the largest difference between the highest and lowest price over the last 12 months? Select from A, B, C, D and E. (A) Huver Co. (B) Drebs Ltd (C) Fevs Plc (D) Fauvers (E) Steapars |
| Answer: | C |
| Reasoning: | Step 1- Calculate the difference between the maximum and the minimum prices. Huver Co. = 1,360 - 860 = 500 Drebs Ltd = 22 - 11 = 11 Fevs Plc = 1,955 - 1,242 = 713 Fauvers = 724 - 464 = 260 Steapars = 2,630 - 2,216 = 414. Tip: Notice the wording of the question is asking for the share with the largest absolute change in price, NOT the largest percentage change, which would have been Drebs Ltd. If the question had wanted the percentage change it would have used the word percentage. Thus the correct answer is (C) Fevs Plc |
| Logical Reasoning Skill: | Numerical |
| Required capability: | OCR |

**Table 12: Three samples requiring numerical logical reasoning skills (Case B).**
| (Case B) |  |
| --- | --- |
|  |  |
| Q: | Reyes Heslop had a target for Leisure profits to be a quarter of their total profits. Assuming profits in other areas remain the same, by how much did the Leisure profits miss this target? Select from A, B, C, D and E. (A) 31.8 million (B) 32.4 million (C) 32.7 million (D) 33.2 million (E) 33.4 million |
| Answer: | D |
| Reasoning: | Step 1- Calculate the total Reyes Heslop profits across all areas other than Leisure. (6.3 + 7.2 + 5.0) + (3.8 + 5.8 + 4.4) + (3.6 + 5.9 + 4.5) + (6.2 + 5.1 + 3.5) = 61.3 million. Step 2- This needs to be 1/4 of all profits for the condition to be met. Therefore all profits, across all sectors, would be 61.3 / 75% = 81.7333 million. Step 3- Now we look at the difference between actual and target Leisure profits. Actual = (4.6 + 7.4 + 5.2) = 17.2 Target = (81.7333 - 61.3) = 20.4333 Shortfall = 3.2333 (millions) Thus the correct answer is (D) 33.2 million |
| Logical Reasoning Skill: | Numerical |
| Required capability: | Diagram, OCR |

**Table 13: Three samples requiring numerical logical reasoning skills (Case C).**
| (Case C) |  |
| --- | --- |
|  |  |
| Q: | Which space experienced the smallest reduction in kWh used between 1990 and 2000? Select from A, B, C, and D. (A) Office Space (B) Print Room (C) Meeting Rooms (D) PC Room |
| Answer: | D |
| Reasoning: | Step 1- Calculate the value of kWh for 1990 and 2000 for each of the rooms. Room 1990 per kWh 2000 per kWh Meeting Rooms 2.04 2.10 Office Space 6.97 5.85 Print Room 2.55 1.80 PC Room 3.40 3.15 Kitchen 2.04 2.10 Step 2- Subtract the kWh for 2000 from that of 1990 for each of the rooms. Room change (1990 - 2000) kWh Meeting Rooms -0.06 Office Space 1.12 Print Room 0.75 PC Room 0.25 Kitchen -0.06 Step 3- Look for the smallest positive value. Negative values represent an increase between 1990 and 2000. Tip- You only need to perform 4 calculations, as two of the rooms have the same values. Thus, the correct answer is (D) PC Room. |
| Logical Reasoning Skill: | Deductive |
| Required capability: | Diagram, OCR |

**Table 14: Three samples requiring spatial logical reasoning skills (Case A).**
| (Case A) |  |
| --- | --- |
|  |  |
| Q: | Which figure is a rotation of the object? Select from A, B, C, and D. (A) (B) (C) (D) |
| Answer: | B |
| Reasoning: | The answer is B. |
| Logical Reasoning Skill: | Spatial |
| Required capability: | Diagram |

**Table 15: Three samples requiring spatial logical reasoning skills (Case B).**
| (Case B) |  |
| --- | --- |
|  |  |
| Q: | Which figure can be formed with the given piece? Select from A, B, C, and D. (A) (B) (C) (D) |
| Answer: | C |
| Reasoning: | The answer is C. |
| Logical Reasoning Skill: | Spatial |
| Required capability: | Diagram |

**Table 16: Three samples requiring spatial logical reasoning skills (Case C).**
| (Case C) |  |
| --- | --- |
|  |  |
| Q: | To which object does the given top view correspond? Select from A, B, C, and D. (A) (B) (C) (D) |
| Answer: | A |
| Reasoning: | The answer is A. |
| Logical Reasoning Skill: | Spatial |
| Required capability: | Diagram |

**Table 17: Three samples requiring mechanical logical reasoning skills (Case A).**
| (Case A) |  |
| --- | --- |
|  |  |
| Q: | A non-pressurised cylindrical metal tank filled with air is submerged underwater. As the air escapes, the tank gradually moves deeper underwater. Which statement provides the best reason for this motion? Select from A, B, C, D, and E. (A) The bubbles provide a downward thrust on the tank (B) The metal increases in density so it gets heavier (C) The bubbles lower the density of the water which lowers its buoyancy (D) Water replaces the air in the tank which makes it heavier (E) Impossible to tell |
| Answer: | D |
| Reasoning: | As air escapes the available space is quickly replaced with water, so the tank’s density becomes the same as that of the water and with the added weight and density of the tank itself continues to sink. |
| Logical Reasoning Skill: | Mechanical |
| Required capability: | Diagram |

**Table 18: Three samples requiring mechanical logical reasoning skills (Case B).**
| (Case B) |  |
| --- | --- |
|  |  |
| Q: | It is a cold winter outside and a well-insulated house has its heater turned on. The front door is opened and cold air rushes in. If the wind speed outside is very low, how would the cold air enter the house? Select from A, B, C, D, and E. (A) Scenario A, the cold air will flow towards the floor (B) Scenario B, the cold air will flow towards the ceiling (C) A combination of A and B (D) The cold air will not enter the house (E) Impossible to tell |
| Answer: | A |
| Reasoning: | Cold air sinks, whereas hot air rises. The house and the air inside it are warmer than the outside air temperature, so if these two systems (house and outside) were to be suddenly connected (door opening) the cold air would sink and the hot air would sit above the cold air until the heat transferred between the two. |
| Logical Reasoning Skill: | Mechanical |
| Required capability: | Diagram |

**Table 19: Three samples requiring mechanical logical reasoning skills (Case C).**
| (Case C) |  |
| --- | --- |
|  |  |
| Q: | In which direction does the orange gear rotate? Select from A, B, and C. (A) Clockwise (B) Counterclockwise (C) No rotation |
| Answer: | A |
| Reasoning: | The correct answer is clockwise. |
| Logical Reasoning Skill: | Mechanical |
| Required capability: | Diagram |

## Appendix B Examples of Different LogicVista Capabilities Data

**Table 20: Three samples of diagram, OCR, and mixed LogicVista data (Case A).**
| (Case A) |  |
| --- | --- |
|  |  |
| Q: | Which ball is the heaviest? Select from A, B, C, and D. (A) A (B) B (C) C (D) CAN NOT SAY |
| Answer: | D |
| Reasoning: | The correct answer is D. |
| Logical Reasoning Skill: | Mechanical |
| Required capability: | Diagram |

**Table 21: Three samples of diagram, OCR, and mixed LogicVista data (Case B).**
| (Case B) |  |
| --- | --- |
|  |  |
| Q: | Select from A, B, C, and D. (A) banana (B) scissors (C) empty plastic soda bottle (D) wooden pencil |
| Answer: | B |
| Reasoning: | The correct answer is B because scissors have metal and are most likely to sink. |
| Logical Reasoning Skill: | Deductive |
| Required capability: | OCR |

**Table 22: Three samples of diagram, OCR, and mixed LogicVista data (Case C).**
| (Case C) |  |
| --- | --- |
|  |  |
| Q: | Which of the following statements is false regarding legal sector spending between Year 4 and projected Year 5? Select from A, B, C, D, and E. (A) IT consulting will increase by 35 million. (B) IT consulting will match that of year 2. (C) IT software will exceed IT consulting. (D) Spending on IT hardware will decline. (E) None of these. |
| Answer: | D |
| Reasoning: | Step 1- Check in turn whether each statement is true or false: a) The projected spend on IT consulting is projected to increase by 35 million. Option A is true. b) The projected spend on IT consulting is 320 million, which matches year 2. Option B is true. c) The projected spend on IT software is 330 million and for IT consulting it is 320 million. Option C is true. d) There are increases projected for IT hardware, IT software, and consulting, therefore “spending on IT hardware will decline” is not true. The option for D is false. e) We see that option D is false, so E cannot be the correct answer. Thus the correct answer is (D) Spending on IT hardware, software, and consulting is projected to decline. |
| Logical Reasoning Skill: | Numerical |
| Required capability: | Diagram, OCR |

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: - The abbreviation of MCQ needs to be clarified the first time using it.
- The pie chart in the middle of Figure 3 mixes with the right chart.
soundness: 2
strengths: - A newly proposed human-annotated benchmark for MLLMs' logical reasoning abilities, sourced from gated private datasets.
- The dataset is indeed challenging considering the low scores of Claude and GPT-4o, with extensive analysis on cases and different components.
summary: This paper presents LogicVista, a benchmark for evaluating MLLMs' logical reasoning abilities. It includes 448 multi-choice questions spanning 11 abilities, with human-annotated rationales. Experiments on 11 MLLMs show there remains a huge gap for improvement.
weaknesses: Although I believe it will be a valuable data resource (if the authors agree to open-source), my main concern is the **necessity** of this dataset in these aspects:

(1) There are already many datasets in the field of reasoning, such as MathVista [1], MMMU [2] and ScienceQA [3], with **some subsets even overlapping with LogicVista** (such as IQTest in MathVista).

(2) As the dataset is sourced from 15 private IQ tests, why are **IQ tests** used in this dataset specifically designed for logical reasoning? Need some citations to support this claim.

(3) The **categorisation** of skills, broad and specific capabilities need further explanations or authorized references. Did you refer to previous MLLM evaluation research? For example, FLASK [4] refers to this QA taxonomy [5] for the definition of skills.

(4) Regarding the **quantity** of this dataset, there are only 448 questions, much smaller than datasets in this category, such as ScienceQA (21208 questions) and MathVista (6141 questions).

[1] MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts https://arxiv.org/abs/2310.02255

[2] MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI https://arxiv.org/abs/2311.16502

[3] Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering https://scienceqa.github.io/

[4] FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Sets https://arxiv.org/abs/2307.10928

[5] QA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering and Reading Comprehension https://arxiv.org/abs/2107.12708

**Some minor weaknesses:**

- Lack of human annotation details: As the authors state there is cross-validation during annotation, what is the inter-annotator agreement? How much payment is given to each annotator? How many hours does it take to finish annotation?

- Reliability of LLM-based evaluator on the chain-of-thought: What is the agreement rate between human-based and LLM-based evaluators on the rationales? Only when the agreement is acceptable can the LLM-based evaluation be convincing.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: I don't have questions.
soundness: 3
strengths: 1. Innovative Multimodal Reasoning Evaluation: The paper systematically evaluates the logical reasoning capabilities of multimodal large language models (MLLMs), covering five core areas: deductive reasoning, inductive reasoning, numerical reasoning, spatial reasoning, and mechanical reasoning. This evaluation fills gaps in current methodologies and holds significant potential for advancing research in the multimodal domain.
2. Rigorous Data Source: The paper uses authorized IQ test data, which avoids the common issue of public data leakage and ensures fairness in the evaluation. This data is more reflective of the models’ reasoning capabilities, rather than merely testing memory or simple inference abilities.
3. Multidimensional Evaluation Approach: By combining MCQ and Chain-of-Thought (CoT) methods, the paper efficiently quantifies the model’s selection ability while also deeply analyzing its reasoning process, balancing both evaluation depth and efficiency.
4. Scalability and Continuous Updates: The introduction of crowdsourced annotation tools ensures the dataset’s scalability and future updates, laying a solid foundation for iterative evaluations.
5. Clear Language and Logical Presentation: The paper is clearly written, with well-structured logic and organized presentation of experimental results, aiding readers in understanding its contributions.
summary: The paper proposes the LogicVista benchmark dataset, aimed at systematically evaluating the performance of multimodal large language models in visual reasoning tasks. The data is sourced from authorized intelligence tests and employs a dual evaluation method combining multiple-choice questions (MCQ) and chain-of-thought (CoT). It covers inductive, deductive, numerical, spatial, and mechanical reasoning. The experiments reveal an imbalance in model performance across different reasoning tasks, with particular limitations in complex spatial reasoning tasks.
weaknesses: 1. Limited Data Source: Although the IQ test data is rigorous, it is relatively concentrated in scope, lacking coverage of other fields (e.g., scientific reasoning, language understanding). The dataset size is also limited, potentially hindering the ability to fully capture the model’s performance across a broader range of tasks.
2. Lack of Depth in Benchmark Design and Ablation Studies: The paper primarily uses random and frequentist baselines, which provide limited insights into the deeper aspects of reasoning. The absence of ablation studies restricts the analysis of how different model components contribute to performance. Introducing more complex baselines and conducting ablation studies could enhance the interpretability of the results.
3. Redundant Logical Expressions: The distinction between the “Broad Capabilities” and “Specific Capabilities” sections in 4.2 is somewhat redundant, particularly in the discussion of OCR and diagram tasks. Simplifying these sections and clearly distinguishing between the two could improve the clarity of the paper’s logic.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: I would like to ask the authors about the weakness and how they can improve them.
soundness: 2
strengths: 1. This approach covers a variety of logical reasoning tasks, allowing for a comprehensive assessment of the model’s performance across different logical contexts.
2. The data sources are drawn from authorized intelligence tests, which effectively ensures data privacy and novelty, reducing the risk of training data leakage.
3. The evaluation methods are rich, combining multiple-choice (MCQ) and chain-of-thought (CoT) approaches, adding depth to the assessment. This allows for effective evaluation of the model’s reasoning process as well as precise measurement of answer accuracy.
summary: This paper presents a variety of logical reasoning tasks, allowing for a comprehensive assessment of the model’s performance across different logical contexts.
weaknesses: 1. LogicVista presents significant difficulty differences across reasoning tasks, leading to unbalanced performance. For example, while models perform relatively well in deductive and mechanical reasoning, they perform less effectively in inductive, numerical, and spatial reasoning tasks. This variation may affect the fairness of overall performance evaluations.
2. The paper notes that current visual encoders face substantial limitations in recognizing spatial and abstract relationships, particularly in complex spatial reasoning and 3D pattern recognition tasks, where models tend to underperform. However, the paper lacks specific improvement suggestions, such as ways to enhance visual encoders or improve training data to boost reasoning ability, which somewhat constrains future developmental direction.
3. There is a lack of ablation studies to deeply analyze the specific contributions of each component to overall performance. Ablation studies can reveal the independent impact of each module, helping to better understand model composition and optimization paths.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: **Major Points:**

- Is there theoretical justification for the representativeness of the five selected reasoning task categories?
- Would it be beneficial to include an error analysis section?
- Uncertain if the significance of this work aligns with the claims made.

**Minor Comments:**

- In line 50, is GLoRE applied to an MLLM or just an LLM?
- When "MCQ-based" and "CoT-based" are first mentioned, it may be clearer to use the full terms, such as "Chain of Thought-based (CoT-based)."
soundness: 3
strengths: - Originality: provides a new, high-quality dataset
- Quality: strict measures are taken to prevent data leakage; well-designed experiments
- Clarity: clear data collection and experimental procedures.
- Significance: compensates for the lack of benchmarks specifically designed to test logical reasoning.
summary: This paper presents a new benchmark, LogicVista, to assess MLLMs' logical reasoning capabilities, addressing an overlooked area in AI evaluation. It includes 448 annotated tasks across five reasoning types (inductive, deductive, numerical, spatial, mechanical) and nice task categories, providing a systematic test of MLLMs' logical reasoning.
weaknesses: Relatively small dataset — 448 samples

In my assessment, a high-quality benchmark typically excels in at least one of three areas:
1. **Data Quality**: The benchmark offers exceptionally high-quality data that often serves as an evaluation standard within its field, as exemplified by VQAv2 and MMMU.
2. **Tooling and Resources**: It provides innovative tools or resources, such as novel code architectures, metrics, or other practical assets that advance usability and applicability.
3. **Research Insight**: The benchmark highlights a previously overlooked problem or dimension, encouraging new avenues for research and fostering deeper understanding.

While the quality of this work is solid, it falls short of providing substantial new insights or advancements in these areas. As a result, its impact on the field may be limited.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 2
questions: Please refer to the Weaknesses.
soundness: 2
strengths: 1. The paper introduces a new evaluation benchmark, LogicVista, focusing on evaluating visual logical reasoning abilities.
2. The structure of the paper is clear, and it thoroughly explains the design motivation, data collection methods, evaluation models, and result analysis of LogicVista.
summary: This paper proposes an evaluation benchmark named LogicVista, which is designed to assess the logical reasoning abilities of Multimodal Large Language Models (MLLMs) in visual contexts. LogicVista comprehensively evaluates 11 existing MLLMs through five types of logical reasoning tasks. These tasks cover inductive, deductive, numerical, spatial, and mechanical reasoning, using 448 multiple-choice questions (MCQs) with correct answers and human-written reasoning annotations. The evaluation methods include both MCQ and open-ended Chain-of-Thought (CoT) analysis to better understand the models' strengths and limitations. Experimental results show that while some models perform well in deductive reasoning, most models score lower in other reasoning categories.
weaknesses: 1. Although LogicVista covers various logical reasoning tasks, the sample size of 448 may be insufficient to fully capture the performance of MLLMs in real-world complex scenarios.
2. The tasks in LogicVista mainly focus on basic logical reasoning, such as mechanical and inductive reasoning. There is a lack of sufficient coverage for higher-level complex reasoning tasks, such as multi-step reasoning or continuous multimodal reasoning.
3. The paper mentions the risk of data leakage in many benchmarks, indicating that MLLMs might have encountered some test data during training. Although LogicVista avoids publicly available internet data in its dataset selection, it does not provide detailed mechanisms to verify the complete independence of all samples.