MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts
Authors: Pan Lu
Hritik Bansal
Tony Xia
Jiacheng Liu
Chunyuan Li
Hannaneh Hajishirzi
Hao Cheng
Kai-Wei Chang
Michel Galley
Jianfeng Gao
Venue: ICLR 2024 oral (2024)
Keywords: large language models
large multimodal models
mathematical reasoning
vision-language reasoning
foundation models and their evaluations
Primary area: datasets and benchmarks
TL;DR: We introduce MathVista, a novel benchmark for evaluating mathematical reasoning capabilities within visual contexts, and conduct extensive experiments on 11 foundation models.

Abstract: Large Language Models (LLMs) and Large Multimodal Models (LMMs) exhibit impressive problem-solving skills in many tasks and domains, but their ability in mathematical reasoning in visual contexts has not been systematically studied. To bridge this gap, we present MathVista, a benchmark designed to combine challenges from diverse mathematical and visual tasks. It consists of 6,141 examples, derived from 28 existing multimodal datasets involving mathematics and 3 newly created datasets (i.e., IQTest, FunctionQA, and PaperQA). Completing these tasks requires fine-grained, deep visual understanding and compositional reasoning, which all state-of-the-art foundation models find challenging. With MathVista, we have conducted a comprehensive, quantitative evaluation of 12 prominent foundation models. The best-performing GPT-4V model achieves an overall accuracy of 49.9%, substantially outperforming Bard, the second-best performer, by 15.1%. Our in-depth analysis reveals that the superiority of GPT-4V is mainly attributed to its enhanced visual perception and mathematical reasoning. However, GPT-4V still falls short of human performance by 10.4%, as it often struggles to understand complex figures and perform rigorous reasoning. This significant gap underscores the critical role that MathVista will play in the development of general-purpose AI agents capable of tackling mathematically intensive and visually rich real-world tasks. We further explore the new ability of self-verification, the application of self-consistency, and the interactive chatbot capabilities of GPT-4V, highlighting its promising potential for future research. The project is available at https://mathvista.github.io/.

Published as a conference paper at ICLR 2024
MATHVISTA: EVALUATING MATHEMATICAL REASON- ING OF FOUNDATION MODELS IN VISUAL CONTEXTS
Pan Lu Hritik Bansal', Tony Jiacheng Liu", Chunyuan Li Hannaneh Hajishirzi", Hao Cheng". Kai-Wei Chang', Michel Galley'. Jianfeng Gao UCLA, UUniversity of Washington, MMicrosoft Research, Redmond https: //math/istaa.... httma//vsttaaiittu.b.b... github.
ABSTRACT
Large Language Models (LLMs) and Large Multimodal Models (LMMs) exhibit impressive problem-solving skills in many tasks and domains, but their ability in mathematical reasoning in visual contexts has not been systematically studied. To bridge this gap, we present MATIVISTA, benchmark designed to combine challenges from diverse mathematical and visual tasks. consists of 6,141 ex- amples, derived from 28 existing multimodal datasets involving mathematics and 3 newly created datasets (i.e., IQTest, FunctionQA, and PaperQA). Completing these tasks requires fine-grained, deep visual understanding and compositional reasoning, which all state-of-the-ar foundation models find challenging.
With MATHVISTA, we have conducted a comprehensive, quantitative evaluation of 12 prominent foundation models. The best-performing GPT-4V model achieves an overall accuracy of 49.9%, substantially outperforming Bard, the second -bett performer, by 15.1%. %. Our in-depth analysis reveals that the superiority of GPT- 4V is mainly attributed to its enhanced visual perception and mathematical rea- soning. However, GPT-4V still falls short of human performance by 10.4%, as it often struggles to understand complex figures and perform rigorous reasoning. This significant gap underscores the critical role that MATUVISTA will play in the development of ceneera-ppupose Al agents capable of tackling mathematically intensive and visually rich real-world tasks. We further explore the new ability of self-verification, the application of self-consistencyy and the interactive chatbot capabilities of GPT-4V, highlighting its promising potential for future research.
INTRODUCTION
Mathematical reasoning stands as a testament to the intricacies of human intelligence (Kahneman, 201 I). It requires rigorous logical thinking, domain-specific knowledge, and the ability to engage in multistep reasoning processes (Lightman al., 2023). This complexity is observed not only in textual scenarios but also significantly in visual contexts. When assessing a child's mathematical reasoning capabilities, problems are often designed encompass visual contexts (Stipek & Iver, 1989; Pollitt et al., 2020). At the same time, Al agents with strong mathematical reasoning capa- bilities visual contexts have wide range of real-world applications, such as solving complex problems (Seo et al., 2015; Wang et al., 2017), addressing logical queries about statistical data (Wu al., 2023; Yang et al., 2023a; Liu al., 2024b), and assisting in theorem proving and scientific discovery in advanced research (Taylor et al., 2022; Dong et al., 2023; Trinh et al., 2024).
Numerous datasets have been curated assess the mathematical reasoning abilities of AI sys- tems, with most presented purely in text form. Some datasets such ChariQA (Lu et al., 2021a; Dahlgren Lindström & Abraham, 2022; Masry et al., 2022) have explored mathematical reasoning in rision-language settings. However, these datasets tend to either focus specific tasks, like math word problems, or particular visual contexts, such as geometry problems or bar charts. General- purpose visual question answering (VQA) datasets natural scenes contain only small portion of questions necessitating mathematical reasoning, leaving a comprehensive investigation of vision- language reasoning within mathematical framework largely unexplored.
On the other hand, Large Language Models (LLMs) (OpenAI, 2022; 2023a) and Large Multimodal Models (LMMss) (Google, 2023; OpenAI, 2023b; Team al., 2023) have exhibited impressive

---

Published as a conference paper at ICLR 2024
Random Chance
LLaVA
PoT GPT-4
Multimodal Bard
GPT-4V (Playground)
Human
Geometry Reasoning
Function
Bar
Arithmetic Reasoning
Geometry Diagram
Chart
Logical Reasoning
Line Plot
Abstract
Scene
10
30
70
Natural
10
30
70
Algebraic Reasoning
Image
Other
Numetic Commonsense
Puzzle
Table
Test
Statistical
Scatter
Synthetic Scene
Scientific Reasoning
Reasoning
Plot
Scientific Figure
(a) Mathematical rcasoning
(b) Visual context
Figure I: Accuracies of one leading LLM (i.e., PoT GPT-4), four prominent LMMs, random chance, and human performance on our proposed MATHVISTA across mathematical reasoning and visual context types. PoT GPT-4 is textual, program- aidedd LLM augmented with the Bard caption and OCR text. GPT-4V is manually evaluated via the playground chatbot.
problem-solving skills in many tasks and domains. Recently, some studies have aimed to augment existing LLMs with mathematical and scientific reasoning capabilities using external tools (Lu et al., 2023a; Wang al., 2023b). However, the ability of these foundation models to perform mathemat- ical reasoning in visual contexts has not been systematically examined. Therefore, it is essential to develop new benchmark to (1) facilitate the development of mathematical reasoning systems in visually intensive scenarios, and (2) evaluate the research progress of LLMS and L.Ms, especially their capabilities in solving rigorous reasoning tasks.
In this paper, we present MATIVVISTA, consolidated Mathematical reasoning benchmark in Visual contexts. We propose task taxonomy to guide the development of MATUVISTA: (1) we identify seven mathematical reasoning types: algebraic reasoning. arithmetic reasoning. geometry reason- ing, logical reasoning, numeric common sense, scientific reasoning, and statistical reasoning; (2) we focus on five primary tasks: figure question answering (FQA), geomelry problem solving (GPS), math word problem (MWP), textbook question answering (TQA), and visual question answering (VQA); and (3) we encompass diverse array of visual contexts, including natural images, ge- ometry diagrams, abstract scenes, synthetic scenes, as well as various figures, charts, and plots. MATIVVIST incorporates 28 existing multimodal datasets, including 9 math-targeted question an- swering (MathQA) datasets and 19 VQA datasets. In addition, we have created three new datasets (i.e., IQTest, FunctionQA, PaperQA) which are tailored to evaluating logical reasoning puzzle test figures, algebraic reasoning over functional plots, and scientific reasoning with academic paper figures, respectively. Overall, MATHVISTA consists of 6,141 examples, with 736 of them being newly curated (Table I). To facilitate fine-grained evaluation, examples are annotated with meta- data, including question type, answer type, task category, grade level, visual context, and required reasoning skills. Detailed descriptions of data collection can be found in $2, SC, and $D.
We conduct extensive experiments on MATI ISTA evaluate the reasoning abilities of 12 founda- tion models known for their leading performance in mathematical and multimodal reasoning. This ensemble includes three LLMs (i.e, ChatGPT, GPT-4, Claude-2), two proprietary LMMs (i.e., GPT- 4V, Bard), and seven open-source LMMs. For LLMs, we examine zero-shot and few-shot settings using two prompting strategies: chain-of- tothhught (CoT) (Wei et al., 2022b) and program-of-thoughh (PoT) (Chen et al., 2022b). These LLMs can also be augmented with off-the-shelf visual models for image captioning and OCR. We establish a human performance baseline by engaging qualified human annotators with high school diploma higher. We show that MATHVISTA featuring ad- vanced topics such as college curricula and scientific reasoning, is a very challenging benchmark, with human performance reaching only 60.3% accuracy.
Our results indicate that CoT GPT-4, the best-performing LLM without visual tool augmentations, achieves overall accuracy of 29.2%. Multimodal Bard, the best-performing LMM,, achieves
2

---

Published as a conference paper at ICLR 2024
7
5
64
test 003
cluate
11
?
216
tean bloo co1
changamz
00
2
aitarder 500
vcurt-233
Sper-sourced
Question: Find the missing value in this math puzzle.
staderode--10
oasotistia ofi 1- 20
Solution:
(5 (5-4)²- -
Question: Which function is monotonic in range [0, pi]?
AgentBench Overall Score
(7 (7-33-64
Question: What the performance gap in the AgentBench Overall Score between the worst API-based I.I.M and the best open-soureed LLM?
(8 (8-2)=216
Choices:
Similarly, (11 (11-89-
(A) the red one (B) the blue one
So the missing value is Answer: 27
(C) both (D) of them
Answer: (B) the blue
Answer: 0.16
Category: Math-targeted
Category: Math-targeted
Category: Math-targeted
Task: Figure question answering
Task: Textbook question answering
Task: Figure question answering
Context: Puzzle test
Context: Function plot
Context: Scientific figure
Grade: Elementary school
Grade: College
Grade: College
Math: Logical reasoning
Math: Algehraic reasoning
Math: Scientific reasoning
(a) IQTest
(b) FunctionQA
(c) PaperQA
Figure 2: Examples of our newly annotated datasets: IQTest, FunctionQA, and PaperQA.
34.8% ($3.3), which attains only 58% of human performance (34.8% vs 60.3%). When augmented with Bard captions and OCR text, PoT GPT-4 obtains 33.9%, closely matching Multimodal Bard ($3.4). Further analysis indicates that the Multimodal Bard model failures arise from incorrect cal- culations and ballucinations caused by visual perception and textual reasoning ($3.5).
With MATHVISTA we report, for the first time, comprehensive quantitative and qualitative eval- uation of GPT-4V (OpenAI, 2023b), the latest multimodal version of GPT-4. Remarkably, GPT-4V achieves a state-of-the-art accuracy of 49.9%, significant improvement of 15.1 1% over Multimodal Bard. As illustrated in Figure 1, GPT-4V even surpasses human performance on set of tasks in- volving algebraic reasoning and complex visual contexts, which include tables and function plots. Nevertheless, a 10.4% gap in overall accuracy remains when compared to the human bascline, leav- ing plenty of room for model improvement. Our in-depth analysis ($H) reveals that the superiority of GPT-4V is mainly attributed to its strong capabilities visual perception and mathematical reason- ing. We further highlight its emergent ability for self-verification ($H.5), the use of self-comsistency ($H.6), and its ability to drive goal-directed multi-turn human-A Al dialogues ($H.7).
2
THE MATHVISTA DATASET
2.1 COLLECTION GUIDELINES
Our benchmark, MATHVISTA, is motivated to bridge the notable gap in existing benchmarks, which primarily evaluate mathematical reasoning in textual contexts. It adheres to the following collection guidelines: (1) it covers multiple tasks and topics to mirror real-world applications; (2) it incorpo- rates diverse visual contexts and mathematical skills to foster a well-rounded evaluation; (3) it offers varying levels of challenge to effectively uncover the potential limitations of current models; and (4) it provides robust settings for deterministic evaluations.
The taxonomy for this work is introduced as follows: We identify seven types of mathematical rea- soning: algebraic reasoning, arithmetic reasoning, geometry reasoning, logical reasoning, numeric common sense, scientific reasoning, and statistical reasoning, with detailed definitions provided in SC.1 and examples shown in $C.2. We focus on five primary tasks: figure question answering (FQA), which centers around statistical reasoning over multiple charts and plots; geometry problem solving (GPS), which deals with geometrical topics; math word problem (MWP), which involves arithmetic reasoning in everyday scenarios; textbook question answering (TQA), which usually tails knowledge-intensive intensive reasoning on scientific topics and figures; and visual question answering (VQA). Furthermore, objective is to account for diverse array of visual contexts, including
3

---

Published as a conference paper at ICLR 2024
natural images, geometry diagrams, abstract scenes, synthetic scenes, multiple charts and plots, scientific figures, tables, function plots, puzzle test figures, and more, with examples shown in $C.3.
2.2 DATA COLLECTION
Collection of MathQA datasets. We collected nine MathQA datasets in multimodal settings, in- cluding four for GPS, two for MWP with visual contexts of synthetic scenes, abstract diagrams, and tables, and two for TQA on college curricula (see $C.4). Annotations such as solutions, programs, parsing results, and grounded theorems are also collected, providing demonstration examples for LLMs. Each source dataset is limited to up 400 examples ensure a balanced representation of each source in our final compiled benchmark. In total, we collected 2,666 examples.
Review and collection of VQA datasets. Many existing VQA datasets feature instances requiring mathematical reasoning abilities, such as arithmetic operations or numeric common sense. Incor- porating these datasets enhances problem diversity in terms of tasks, domains, visual contexts, and reasoning skills involved. We reviewed more than 70 datasets, collecting 19 of them that contain math-related instances and are publicly available, as listed in $C.4. Since these datasets are not orig- inally math-targeted we initially designed heuristic rules to automatically select examples likely to involve mathematical reasoning from a large pool of candidates. Examples with numeric an- swers those containing quantity words (as listed in $D.1) in the questions were selected. This automatic filtration yielded 4,949 VQA-forma examples, though some false positive examples re- mained. Therefore, mine if they involve mathematical reasoning (more details in $ D.2). Utilizing majority voting and limiting each source dataset 400 examples, we finalized a collection of 2.739 examples.
engaged three expert annotators manually label these examples to deter-
Collection of three new datasets. While the source datasets we collected encompass multiple visual contexts and mathematical reasoning abilities, certain scenarios remain unaddressed: logical reasoning on puzzle test diagrams, statistical reasoning on functional plots, and scientific reasoning on academic figures. address these gaps, we introduced three new datasets: IQTest, FunctionQA, and PaperQA, with examples illustrated in Figure 2. IQTest comprises 228 examples requiring in- ductive reasoning, abstract thinking, pattern prediction, and calculations, sourced from puzzle test figures online learning platforms. FunctionQA. with 400 examples, emphasizes subtle visual per- ceptions of functional plots and algebraic reasoning concerning variables, expressions, equations, and functions. PaperQA is novel dataset featuring questions derived from informative academic il- lustrations, including tables, figures, and charts from online education resources, with 107 examples sourced from papers released in August 2023 on Huggingface! To ensure data quality, all questions were manually annotated by graduate students in STEM fields and further refined through a rigorous review process.
2.3 METADATA ANNOTATION
Fine-grainer metadata facilitates a comprehensive analysis of models' reasoning capabilities across various aspects. To this end, we annotate the examples MATHVISTA with information including question type, answer type, language, source, category, task, grade level, and visual context, which can be accurately obtained from the details provided in the source datasets. MATHVISTA features seven different types of mathematical reasoning abilities, categorized in Table 3 (SC.1). Coarse la- bels of mathematical reasoning can be automatically obtained from the details of the source datasets. To verify the quality of automatic annotation, expert annotators manually label the mathematical rea- soning categories from seven candidates for 1,000 examples, using the annotation tool illustrated in $D.4. The results show that 94.1 % of the examples from automatic and human annotations have the exact same set of reasoning types, while 98. 79% of the individual labels are identical, indicating that the automatic annotation for the labeling of mathematical reasoning is highly accurate.
2.4 DATA PREPARATION AND RELEASE
VATI consists of 6,141 examples, divided into two subsets: testmini and test. testmini con- tains 1,000 examples, intended for model development validation or for those with limited comput-
Cectps://huggingfaee wotppaaaccccppppaaa..ppppp
4

---

Published as a conference paper at ICLR 2024
Statistic
Number
Geo 6.5%
PanerQl
Total questions
6,141
Gepmesry3k
Chamaa 3.7%
-muliple-choice questions
3,392 (55.2%)
2,2
FFree-orm questions
2.749 (44.8%)
6.5%
Figuna 6.5%
Questions with annotations
5,261 (85.6 (85.6%)
Questions newly annotated
736 (12.0%)
UniGeo 6.5%
Unique number of images
5,487
GPS
FQA 26.8%
DVQA
Unique number of questions Unique number of answers
4,746
GEOS
21.5%
6.5%
1,464
GLEVR-Math 6.5%
MWP 19.5%
TQA 15.3%
704 SciBench
Source datasets
31
Existing VQA dutasets
19
IconQA 6.5%
VQA
A120
Existing MathQA datasets
16.9%
Function@A
4%
Our newly annotated datasets
3
Tab 6.8%
Visual context (image) classes
19
Super- Supe:-CLEVR
6.5%
6.2%
XVQA
3.1%
vOAs.
Maximum question length
213
Maximum answer length
27
Maximum choice number
8
Average question length
15.6
Average answer length
1.2
Figure 3: Source dataset distribution of MATHVISTA.
Average choice number
3.4
FQA: figure question answering, GPS: geometry prob- lem solving, MWP: math word problem, TQA: textbook question answering, VQA: visual question answering.
Table I: Key statistics of MATIVVSTA.
ing resources. The lest features the remaining 5,141 examples for standard evaluation. Notably, the answer labels for test will not be publicly released to prevent data contamination, and we will maintain an online evaluation platform. To ensure that each source dataset is well represented in testmini and to maintain a distribution in testmini closely resembling the whole set, we adopted this sampling strategy: (1) first, randomly sample questions with a threshold number of for each source dataset; (2) then, randomly sample the remaining questions for each source dataset on its proportion in the entire set. The KL Divergence and Total Variation (TV) distance between the testmini set and the entire set are 0.008 and 0.035, respectively, suggesting that festmini is close to the distribution of the whole set. We also conducted several quality checks to address any unidentified errors.
2.5 DATA ANALYSIS
The main statistics of MATHVISTA presented in Table There two types of questions: multiple-choice and free-form. Answers to free-form questions are categorized as integers, float- ing numbers, lists. The large unique number of images, questions, and answers ensures pattern diversity in MATIVVITA. MATHVISTA is derived from 31 source datasets, including three newly annotated datasets to address the missing types of mathematical reasoning over specific visual con- texts. Further details on data analysis are available in SE.
3 EXPERIMENTS
In this work, we aim to conduct qualitative and quantitative studies to provide a systematic evaluation of existing foundation models for mathematical reasoning in visual contexts using MATHVISTA.
3.1 EVALUATION PROTOCOLS
Recent LLMs and LMMs have been instructed to generate long responses in conventional settings instead of short text. Therefore, we propose a new strategy for benchmarking MATH VISTA, unlike using desinnd or template matching rules (Lu et al., 2022). The evaluation process consists of three stages: response generation, answer extraction, and score calculation. Initially, the base- lines generate responses given the input query, which incorporates the task description, the question, the choices, and the metadata, using the template defined in Table (SF33.. Next, the short answer text is extracted from the detailed response. We propose an answer extractor ($F2) based on LLMs such as GPT-4, inspired by its remarkable ability for text processing (Wei et al., 2022b). A prelim- inary study of 200 examples shows that GPT-4 can extract the answer text with more than 99.5% accuracy. Finally, the extracted answer is normalized to a required answer format (c.g., an option letter integer), and the target metric scores are computed. Taking advantage of the fact that
5

---

Published as a conference paper at ICLR 2024
Model
Input |ALL
FQA
GPS MWP TQA VQA |LG ARI
GEO
LOG NUM
SCI STA
Heuristics baselines
Random chance
17.9
18.2
21.6
3.8
19.6
26.3
21.7
14.7
20.1
13.5
8.3
17.2
16.3
Frequent guess
26.3
22.7
34.1
20.4
31.0
24.6
33.1
18.7
31.4
24.3
19.4
32.0
20.9
Large Language Models (LLMs)
Zero-shot ChatGPT
Q only
23.5
21.9
26.9
9.1
38.6
23.5
27.7
15.9
25.7
21.6
9.9
20.5
Zero-shot GPT-4
only
26.1
22.3
37.0
7.0
39.2
27.4
33.6
17.4
35.6
16.2
9.2
45.8
19.5
Zero-shot Claude-2
Q only
26.4
21.9
34.1
13.4
36.1
29.1
32.8
20.4
33.3
13.5
12.1
36.4
20.5
2-shot CoT Claude-2
Q only
24.4
18.6
29.8
9.7
33.5
34.1
29.2
19.0
28.0
5.4
13.9
36.9
18.9
2-shot CoT CharGPT
Q only
26.8
20.1
36.5
8.6
44.9
28.5
35.6
17.0
33,5
21.6
14.6
45.9
17.9
2-shot CoT GPT-4
Q only
29.2
20.1
44.7
8.6
46.2
31.3
41.6
19.3
41.0
18.9
13.9
47.5
18.9
2-shot PoT ChatGPT
Q only
25.1
19.0
30.8
16.1
38.0
25.7
29.9
19.8
29.3
24.3
19.4
38.5
16.9
2-shot PoT GPT-4
Q only
26.0
20.1
33.2
8.1
44.9
28.5
32.7
16.7
31.0
24.3
13.2
48.4
18.3
Augmented Large Language Models (Augmented-LLLM)
2-shot COT Claude-2
Q. I. 33.2
26.0
31.7
35.5
48.1
30.2
32.4
32.3
33,0
16.2
17.4
54.9
36.2
2-shot CoT ChatGPT
Q. 1..
33.2
27.5
29.3
36.0
49.4
29.1
31.0
32.9
31.0
16.2
17.4
50.8
37.2
2-shot CoT GPT-4
Q. To.
33.2
31.7
31.2
28.5
33.5
32.2
13.5
12.5
37.9
2-shot PoT CharGPT
Q. To.
26.8
24.5
26.4
23.7
33.5
27.9
27.8
26.1
28.0
18.9
13.2
33.6
29.9
2-shot PoT GPT-4
Q T., Is
33.9
39.4
30.6
39.9
31.3
37.4
31.7
41.0
18.9
44.3
Large Multimodal Models (LMMs)
EDEFICS-9--Insee Instructtt
Q.1 19.8
21.6
21.1
6.5
25.9
24.0
22.1
15.0
19.8
18.9
9.9
24.6
18.1
mPLUG-OWW-LLLMAN
Q.
22.2
22.7
23.6
10.2
27.2
27.9
23.6
19.2
23.9
13.5
12.7
21.4
miniGPT+LLaMMA2227B
23.1
18.6
26.0
13.4
30.4
30.2
28.1
21.0
24.7
16.2
16.7
25.4
17.9
LLaMA-Adapter-V22. V2-7B
Q.1
23.9
21.2
25.5
11.3
32.3
31.8
26.3
20.4
24.3
24.3
13.9
29.5
18.3
LLaVAR
Q,
25.2
21.9
25.0
16.7
34.8
30.7
24.2
22.1
23.0
13.5
15.3
42.6
21.9
InstructBLIP- PBstcccBLI--Vicuma-BB
Q.I
25.3
23.1
20.7
18.3
32.3
35.2
21.8
27.1
20.7
18.9
20.4
33.0
23.1
LLVA-LLaMA-22133B
Q.
26.1
26.8
29.3
16.1
32.3
26.3
27.3
20.1
28.8
24.3
18.3
37.3
25.1
Multimodal Bard
Q.
34,8
26.0
47.1
29.6
48.7
26.8
46.5
28.6
47.8
13.5
14.9
47.5
33.0
GPT-4V (Playground)
Q.
49.9
43.1
50.5
57.5
65.2
38.0
53.0
49.0
51.0
21.6
20.1
63.1
55.8
Hummn
Human performance
Q.
60.3
59.7
48.4
73.0
63.2
55.9
50.9
59.2
51.4
40.7
53.8
64.9
63.9
Table 2: Accuracy scores the testmini subset of MATHVISTA. Input: Q: question, I: image, Ie image caption, I;: OCR text detected the image. ALL: overall accuracy. Task types: FQA: figure question answering, GPS: geometry problem solving. MWP: math word problem, TQA: text- book question answering. VQA: visual question answering. Mathematical reasoning types: ALG: algebraic reasoning. ARI: arithmetic reasoning, GEO: geometry reasoning, LOG: logical reasoning, NUM: numeric commonsense, SCI: scientific reasoning, STA: statistical reasoning. The highest scores among models in each section and overall are highlighted in blue and red, respectively.
the instances in MATIIVISTA are either multiple-choice questions for textual answers or free-form questions for numerical answers, accuracy scores used metrics for deterministic evaluation.
3.2 EXPERIMENTAL SETUP
We evaluate the models on MATHVISTA under three setups: (a) Text-Only LLMs including ChatGPT (OpenAI, 2022), GPT-4 (OpenAI, 2023a), and Claude-2 (Anthropic, 2023) in zero-shot and two-shot settings with Chain-of-Thouglee (CoT) (Wei al., 2022b) and Program-of Saggram-of-Thougee (PoT) (Chen et al., all,, 2022b), (b) Auggentted-LMMs where the LLMs are provided with additional visual information including the generated image captions from Multimodal Bard (Google, 2023) and the detected OCR text from EasyOCR (JaidedAI, 2020), (c) LMMs that include open-source models such as IDEFICS-9D (Laurençon et al., 2023), APLUGGOWL-LLAMAA LLaMA-2-7B (Zhu et al., 2023a), LLaMA- V2-7B (Gao et al., 2023), InstructBLIP. Vicuuaa 7B (Dai et al., 2023), LLaVA-LLaMA-22
(Ye et al., 2023), miniGPT-4-
(Liu et al., 2024a), LLaVAR Zhang et al. (2023c), and proprietary models such Bard and GPT-4V. Since GPT-4V does not offer API access, resorted to manually evaluating it using the playground chatbot. We provide the prompts for LLMs and the hyperparameters used for LMMs in SF.

---

Published as a conference paper at ICLR 2024
3.3 EXPERIMENTAL RESULTS
We compare the performance of several models, including Text-only LLMs, Augmented LLMs, and LMMs on VATHVISTA Table We include random chance (i.e., one of the options multiple- choice questions, and empty in the free-form questions) and frequency guess ($F.1) naive base- lines. Additionally, we established human performance baseline using Amazon Mechanical Turk. Eligible human annotators must have satisfactory annotating history, successfully pass qualifica- tion examples, and possess a high school degree or higher. We asked each annotator to complete five questions within 20 minutes. Further details can be found in $F.6.
Among only LLMs, all models outperform the random baselines, with the 2-shot GPT-4 using thain-of-thouuht (CoT) prompting achieving 29.2%. The limited performance of text-only LLMs suggests that our dataset requires models to reason within visual contexts for optimal results. When equipped with image captions and detected OCR text, augmented L.MSs exhibit superior perfor- mance compared to their text-only counterparts on MATHVISTA. Specifically, the best-performing augmented LLM is the 2-shot GPT-4 employing program-of of-uhh (PoT) prompting, which scores 33.9%. This model generates Python programs for execution, thereby promoting rigorous reasoning.
On the LMM side, Multimodal Bard scores a 34.8% accuracy, which is only 58% of human perfor- mance at 60.3%. Notably, the best-performing GPT-4V model achieves 49. 49,9%, marking a substan- tial 15.1 1% improvement over Bard: however, it still falls 10.4% short of human performance. These gaps highlight that there is significant scope for further improvements on our benchmark. The open-source models (IEFFCC to LLaVA) achieve underwhelming performance on MATH This can be attributed to their lack of math reasoning capabilities, text recognition (useful for math word problems), shape detection (useful for geometrical problems), and chart understanding. No- tably, these models utilize different model architectures for processing the vision (e.g.. OpenCLIP, CLIP, Vit-G) and language (e.g.. LLaMA-1, LLaMA-2), different alignment strategies (e.g.. MLP projection in LLaVA. Q-former in InstructBLIP. visual abstractor in mPLUGOwl), and instruction tuning data (e.g.. 150K instruction-responsse pairs from LLaVA data, 3,500 instruction-response pairs from miniGPT-4). While fine-tuned with instruction-oolloiin data from text-rich images, VLR does not perform well, indicating that strong text recognition abilities do not guarantee high performance on MATHVISTA, which requires comprehensive visual perception and mathemat- ical reasoning. This underscores that there are immense possibilities for innovations in model, data, or training objectives to improve the zero-shot performance of LMMs on MATHVISTA.
3.4 FINE-GRAINED RESULTS
We also report fine-grained scores for a comprehensive study of the capabilities of existing models across different tasks (Table 2), mathematical reasoning abilities (Table 2, Figures 1, 33), visual con- text types (Figures 1, 34), and grade levels (Figure 35). Remarkably, GPT-4V surpasses most other baselines in various categories, with exceptions in problems related to logical reasoning and numeric commonsense reasoning. Notably. GPT-4V surpasses human performance not only in tasks like ge- ometry problem solving (GPS), textbook question answering (TQA), and mathematical reasoning skills such as algebraic reasoning but also in visual contexts including function plots, geometry diagrams, seatter plots, and tables. Please refer to SG.2, SG.3, and $G.4 for more detailed analysis.
We perform an ablation study on the augmented LLMs and present the results in Table 36 (see &G.5). The gap in the performance of the Augmented LLMs can be attributed to poor image captions, which may not adequately describe the math in visual contexts, the inability of the OCR to detect shapes useful for gcometrical reasoning, and the lack of mathematical reasoning capabilities. An in-depth study of GPT-4V can be found in H..
3.5 QUALITATIVE ANALYSIS
Success and failure analysis of Multimodal Bard. In $3.3, we observe that Multimodal Bard achieves the highest average accuracy on MATUVISTA. Here, we analyze its predictions through human evaluation to understand its mode of success and failure. To do so, we ask the human workers, from Amazon Mechanical Turk (AMT), to study Bard's predictions given the math question, its associated image, and the ground truth from MATHVISTA dataset for 250 instances. Specifically, workers were instructed to decide whether the predictions contained the correct answer with the
7

---

Published as a conference paper at ICLR 2024
Correct Ans, w/ Correct Exp.
21.2%
Correct Reasoping w/ Hallucination
Correct Ans, w/ Partial Exp.
6.8%
Correct Ans, w/ Wrang
8.1%
Correct Reasoning Wrong Calculations
Applicable 4.5%
Incorrect Ans, w/ Correct Exp.
3.2%
Reasoning Hallucination
40.6%
Incorrect. Ans, w/ Partial Exp.
11.7%
Incorrect Ans, w/ Wrong
64.6%
Incorrect Reasoning Wrong Calculations
12.4%
10
20
30
40
50
10
20
30
40
50
(a) Errors in answers and explanations
(b) Types of wrong explanations
Figure 4: Error analysis of Bard results: (a) presents errors in answers and explanations; (b) delves into the details of wrong explanations. Notations: "Answer" is "Ans.", "Explanation" is "Exp.", "Partially Correct" "Partial", and "Not applicable" refers unanswerable indeterminate cases.
i
Question: Is the function
Question: Find
3
(f to R) injective?
2z 15
Choices:
2
Choices:
(A)
(B)
(A) Yes
(C) 12 (D)
(B) No
Response from Bard:
Response from Bard:
We can use the Law of Cosines to solve for 2. The Law of Cosines states that for triangle with side lengths b, and and angle measure 0 opposite side the following 2ab cos(O)
The function f(x) = x² not injective because it takes two different inputs, -1 and 1, to the same output, 1. This means that there is no one-to-one correspondence between the domain and the range of the function.
equation holds: c² a²
In this case, have 7. 9. and 0 90 (because RRT S right triangle). Plugging these values into the Law of Cosines, get: z² 7 + 2(7) cos cos (90")
In the image, we can see that the graph of the function passes through the points (-1, 1) and (I, 1). This means that f(-1) = f(1) = 1. So, if we have two different imputs. -1 and 1, they both map the same output, This violates the definition an injective function.
Since cos(90") 0, we can simplify the equation to get:
x² 49
Taking the square rool of both sides, we Since /130 not among the answer choices, we round z to nearest integer, which 12
130.
Therefore, the answer is (B).
Correct output: (B) No
Correct output: (C) 12
(a) Correct answer and explanation
(b) Correct answer but wrong explanation
Figure 5: Two examples from Bard. In (b), Bard does not correctly identify the geometry symbols and relationships. The accurate correct should identify the isosceles triangle and apply properties.
correct explanation. If the workers find that the model's explanation incorrect, they had choose whether the wrong explanation was due to various failure modes such as incorrect reasoning with hallucination or wrong calculations. In our setup, we define hallucination as an introduction of incorrect facts, in the model explanation, that is not mentioned in the context of the image question (e.g., in Figure 39 and Figure 40). More details can be found SF...
We present the distribution of the quality of Bard's predictions, judged by the human annotators, in Figure 4 (a). We find that 44.6% of the Bard's predictions had incorrect answers with incorrect explanations. Interestingly, we observe that Bard responds with partial (6.8%) completely (8.1%) incorrect explanations despite giving the correct answer to the input image and question, highlight- ing its failure reach the correct answer for the wrong reasons. In Figure 4 (b), present the distribution over possible reasons when Bard provides incorrect explanations. Notably, we find that 49.6% of its responses contain hallucinations. Our analysis highlights that hallucination source of errors in the generative foundation models (Lu et al., 2023c; et al., 2023). We also observe that the model responds with correct reasoning but either hallucinates (18.6%) or performs wrong calculations (19.5%) leaving an overall impression of being wrong explanation.
major
Qualitative examples of Multimodal Bard. We also present a few qualitative examples of Bard's predictions. In Figure 5 (a). find that Bard generates the correct answer with the correct expla- nation. including detecting the correct function (i.e., f(x) = x²) and analyzing its properties (i.e., injective) answer the question. However, in Figure (b), observe that the model provides the
8

---

Published as a conference paper at ICLR 2024
Laps driven
Day
Number laps
(acl
Thursday
[(x)-
Friday
f(
Saturday
Sunday
2
Monday
race car driver kept track how many laps he drove in the past 5 days.
Question:
Question: What this function most likely be?
Choices:
What is the mode of the numbers?
(A) polynomial (B) trigonometric
Detected OCR text:
(C) exponential (D) logarithmic
([55, 9]. Laps driven).
Caption from Bard:
([21, 341, Day),
The image shows graph of a function (a) = on a graph paper. The axis is the horizontal axis and the axis is the vertical axis. The graph is horizontal line that passes through the point (0,c). c). The text "y =
([81, 371, Number of laps),
([1, 64], Thursday). ([129, 67]. 53),
([3, 991, Friday), ([129, 99], 53),
that the y-value The text "(0. means that
([1, 126]. Saturday). ((129, 129J.
([3, 161], Sunday), ([129, 161], 55),
all points on the point (0, c) is on the graph. The text "f(x) = c" means that the function f(z is equal to for all values of a. The graph of constant function is horizontal line. In this case, the horizontal line passes through the point (0, 3), the constant value of the function is 3.
graph
([1, 188), Monday), ([129, 191], 55)
Response from GPT-4:
def mode (laps)
return (set (laps), key -lups.count)
Response from GPT-4:
laps [53, 53, 53, 55, 55] print (mode (laps))
The function shown in the image a constant function, which is a type of polynomial function. Thus, the answer is A.
Correct output: 53
Correct output: (A) polynomial
(a) Correct answer and code
(b) Correct answer with partially correct outputs
Figure 6: Two examples from GPT-4. GPT-4 depends on the qualities of the generated caption and detected OCR texts. In (b), some information is incorrect, though the final answer is correct.
correct answer (i.e, 12) but with an incorrect explanation (i.e., using the law of cosines when the question requires an understanding of the properties of isosceles triangles). We present more ex- amples in $G.9, Overall, our analysis of Bard highlights its modes of failure detail, which could guide future foundation model design to address these issues.
Qualitative examples of Augmented GPT-4. Augmented with external visual models, CoT GPT- 4 and PoT GPT-4 are able achieve comparable performance with Multimodal Bard. As shown in Figure 6 (a), provided with the accurate OCR text detected in the image, PoT GPT-4 accurately understands the structural information of the image and generates code snippet to perform precise statistical reasoning. In Figure 6 (b), the caption provides some accurate descriptions of the image (e.g.. f(x) c) along with hallucination (e.g., y 3, the line passes through (0, 3)) caused by the external Bard model. Although CoT GPT- predicts the correct answer given the partially correct information, the qualities of visual information augmented by external models have an impact on the accurate visual perception and thus the final mathematical reasoning performance. Examples in $G. 10 show failure cases due to hallucination caused by external visual models.
4 CONCLUSION
In this work, we introduce MATHVISTA, a benchmark designed to systematically analyze the math- ematical reasoning capabilities of state-of-the-an models in visually complex scenarios. Our evalu- ation of 12 prominent foundation models highlights that significant advancements have been made, especially with the GPT-4V model. However, substantial gap of 10.4% still exists between GPT- 4V. the best-performing model, and human performance. This disparity sets a clear direction for future research, emphasizing the need for models that seamlessly integrate mathematical rea- soning with visual comprehension. Moreover, our exploration of GPT-4V's selffierriication,, self- consistency, and chathot interactions offers valuable insights for future investigations.

---

Published as a conference paper at ICLR 2024
REFERENCES
lean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems (NeurIPS), 35:
23716-23736, 2022. 20
Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Kedzioskkdiii Yejin Choi, and Hannaneh Hajishirzi. Mathqa: Towards interpretable math word problem solving with operation- based for- malisms. In Proceedings of the 2019 Conference of the North American Chapter of the Associ- ation for Computational Linguistics: Human Language Technologies (NAACL), pp. 2357-2367, 2019. 20
Anthropic. Claude 2, 2023. URL http:///wwwww.attoorooo..ooo//// 20
2. 6,
Anthropic. The claude model family: Opus, sonnet, haiku, 2024. URL https://www
anthropic. coomnewws /aludee-eeeeeeeeee 3- 3-familly ly. 20
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zit- nick, and Devi Parikh. VQA: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pp. 2425-2433 2015. 20,
Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, et al. OpenFlamingo: An open- source framework for training large autoregressive vision-language models. arXiv preprint arXiv:2308.011900 2023. 20
Yonatan Bitton, Hritik Bansal, Jack Hessel, Rulin Shao, Wanrong Zhu, Anas Awadalla, Josh Gard- ner, Rohan Taori, and Ludwig Schimdt. VisIT-Bench: benchmark for vision-language instruc- tion following inspired by real-world use. In Neural Information Processing Systems (NeurIPS) Track on Datasets and Benchmarks, 2023. 20
Nitzan Bitton-Guetta, Yonatan Bitton, Jack Hessel, Ludwig Schmidt, Yuval Elovici, Gabriel Stanovsky, and Roy Schwartz. Breaking common sense: WHOOPS! A vision-and-languaae benchmark of synthetic and compositional images. arXiv preprint arXiv:2303.07274. 2023. 20
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportu- nities and risks of foundation models. arXiv preprint arXiv:2108.0725. 2021. 20
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in Neural Information Processing Systems 33:r1PP)),3311877-11 2020. 20
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Ka- mar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712, 2023. 20
Jie Cao and Jing Xiao. An augmented benchmark dataset for geometric question answering through dual parallel text encoding. In Proceedings of the 29th International Conference on Computa- tional Linguistics, pp. 151 1511-, 2022. 20, 27
Shuaichen Chang, David Palzer, Jialin Li, Eric Lusser, and Ningchuan Xiao, MapQA: A dataset for question answering on choropleth maps. arXiv preprint arXiv:2211.08545, 2022. 20, 27
Jiaqi Chen, Tong Li, Jinghui Qin, Pan Lu, Liang Lin, Chongyu Chen, and Xiaodan Liang. UniGeo: Unifying geometry logical reasoning via reformulating mathematical expression. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 3313-3323, 2022a. 20, 27
10

---

Published as a conference paper at ICLR 2024
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared
Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained code. arXiv preprint arXiv:2107.03374 2021. 20
Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W Cohen. Program of thoughts prompt- ing: Disentangling computation from reasoning for numerical reasoning tasks. arXiv preprint arXiv:2211.12588 2022b. 2, 6,
Wenhu Chen, Ming Yin, Max Ku, Pan Lu, Elaine Wan, Xueguang Ma, Jianyu Xu, Tony Xia, and Xinyi Wang. TheoremQA: A theorem-driven question answering dataset. In The 2023 Conference on Empirical Methods Natural Language Processing (EMNLP), 2023. 21, 27
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110. 2021. 20
Adam Dahlgren Lindström and Savitha Sam Abraham. CLEVR-Math: A dataset for composi- tional language, visual and mathematical reasoning. In 16th International Workshop on Neural- Symbolic Learning and Reasoning, NeSy 2022, Windsor, UK, september 28-30, 2022., volume 3212. CEUR-WS, 2022. 1, 20, 27
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. InstructBLIP: Towards general-purpose vision- language models with instruction tuning, 2023. 20, 39
Qingxiu Dong, Li Dong, Ke Xu, Guangyan Zhou Yaru Hao, Zhifang Sui, and Furu Wei. Large language model for science: A study on vs. NP. arXiv preprint arXiv:2309.0560 2023.
Iddo Drori and Nakul Verma. Solving linear algebra by program synthesis. arXiv preprint
arXiv:2111.08171, 2021. 2021. 21
Iddo Drori, Sarah Zhang, Reece Shuttleworth, Leonard Tang, Albert Lu, Elizabeth Ke, Kevin Liu, Linda Chen, Sunny Tran, Newman Cheng, et al. A neural network solves, explains, and generates university math problems by program synthesis and few-shot learning at human level. Proceed- ings of the National Academy of Sciences, 119(2))e2123433113 2022.
Simon Frieder, Luca Pinchetti, Ryan-Rhys Griffiths, Tommaso Salvatori. Thomas Lukasiewiez, Philipp Christian Petersen, Alexis Chevalier, and Julius Berner. Mathematical capabilities of chatgpt. In 37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks, 2023. 20
Lingyue Fu, Huacan Chai, Shuang Luo, Kounianhua Du, Weiming Zhang, Longteng Fan, Jiayi Lei, Renting Rui, Jianghao Lin, Yuchen Fang, et al. CodeApex: bilingual programming evaluation benchmark for large language models. arXiv preprint arXiv:2309.019::
2023. 20
Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, Hongsheng Li, and Qiao. LLaMA-Adapter V2: Parameter-efficient visual instruction model. arXiv preprint ar 20000030...1000 2023. 6, 20
Peng Gao, Renrui Zhang, Chris Liu, Longtian Qiu, Siyuan Huang, Weifeng Lin, Shitian Zhao, Shijie Geng, Ziyi Lin, Peng Jin, Kaipeng Zhang, Wenqi Shao, Chao Xu, Conghui He, Junjun He, Hao Shao, Pan Lu, Hongsheng Li, and Yu Qiao. Sphinx-x: Scaling data and parameters for family of multi-modal large language models. arXiv preprint arXiv:2402.059::
2024. 20
Google. Bard, 2023. URL https: /bbaddaadd//////// www.soooooggccoooooooooooo com/ 1, 6, 20
Yash Goyal, Tejas Khot, Douglas Summers-Star Staan Dhruv Batra, and Devi Parikh. Making the V in VQA matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6904-6913, 2017. 20, 27
Jia-Chen Gu. Hao-Xiang Xu, Jun- Yu Ma, Pan Lu, Zua Ling, Kai- Wei Chang, and Nanyun Peng. Model editing can hurt general abilities of large language models. arXiv preprint 2oXiv:240.004700 2024. 20
11

---

Published as a conference paper at ICLR 2024
Danna Gurari, Qing Li, Abigale Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. VizWiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 3608-3617, 2018. 20, 27
Wenlong Huang, Picter Abbeel, Deepak Pathak, and Igor Mordatch. Language models zero-shot planners: Extracting actionable knowledge for embodied agents. In International Conference on Machine Learning (ICLR), pp. 9118-9147, PMLR, 2022. 20
Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Yao Fu, et al. C-eval: A multi-level multi-discipline chinese evalua- tion suite for foundation models. Advances in Neural Information Processing Systems (NeurIPS), 36, 2024. 20
JaidedAI. EasyOCR: Ready-to-use OCR, 2020. URL
com/ Jtnp//aaieeedAAA
EasyOCR, 6
Anya Ji, Noriyuki Kojima, Noah Rush, Alane Suhr, Wai Keen Vong, Robert D Hawkins, and Yoav Artzi. Abstract visual reasoning with tangram shapes. arXiv preprint arXiv:2211.16492, 2022. 20
Ziwei Ji, Naycon Lee, Rita Frieske, Tiezheng Yu. Dan Su, Yan Xu, Etsuko Ishii Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12): 2023. 8
Kushal Kafle, Brian Price, Scott Cohen, and Christopher Kanan. DVQA: Understanding data visu- alizations via question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5648-5656, 2018. 20, 27
Daniel Kahneman. Thinking, fast and slow. macmillan, 2011. 1
Samira Ebrahimi Kahou, Vincent Michalski, Adam Atkinson, Ákos Kádár, Adam Trischler, and Yoshua Bengio. FigureQA: An annotated figure dataset for visual reasoning. arXiv preprint 1rXiv:1710.07300 2017. 20,
Aniruddha Kembhavi. Mike Salvato, Eric Kolve, Minjoon Sco, Hannanch Hajishirzi, and Ali Farhadi. A diagram is worth dozen images. In Computer Vision-ECCV 2016: 14th Euro- pean Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV 14, pp. Springer, 2016. 20, 27
Aniruddha Kembhavi, Minjoon Sco, Dustin Schwenk, Jonghyun Choi, Ali Farhadi, and Hannanch Hajishirzi. Are you smarter than sixth grader? Textbook question answering for multimodal machine comprehension. In Proceedings the IEEE Conference on Computer Vision and Pattern recognition, pp. 4999-50000 2017. 20,
Jason J Lau, Soumya Gayen, Asma Ben Abacha, and Dina Demner-Fushman. A dataset of clinically generated visual questions and answers about radiology images. Scientific data, 5(1):1-10, 2018. 20, 27
Hugo Laurençon, Lucile Saulnier, Léo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander M. Rush, Douwe Kiela, Matthieu Cord, and Victor Sanh. OBELICS; An open web-scale filtered dataset of interleaved image-text documents, 2023. 6, 39
Kenton Lee, Mandar Joshi, Iulia Raluca Ture, Hexiang Hu, Fangyu Liu, Julian Martin Eisenschlos, Urvashi Khandelwal, Peter Shaw, Ming- Wei Chang, and Kristina Toutanova. Pix2Struct: Screen- shot parsing as pretraining for visual language understanding. In International Conference on Machine Learning (ICLR), pp. 18893-18912. PMLR, 2023. 20
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301. /2597, 2023a. 39
12

---

Published as a conference paper at ICLR 2024
Yunxin Li, Longyue Wang, Baotian Hu, Xinyu Chen, Wanqi Zhong, Chenyang Lyu, and Min Zhang- A comprehensive evaluation of gpt-4v on knowledge-intent inteensive visual question answering. arXiv preprint arXiv:2311.075330 2023b. 39
Zhuowan Li, Xingrui Wang. Elias Stengel-Eskin, Adam Kortylewski, Wufei Ma, Benjamin Van Durme, and Alan L Yuille. SLEVR-CLEVR: A virtual benchmark to diagnose domain ro- bustness in visual reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14963-14973, 2023c. 20, 27
Thomas Liao, Rohan Taori, Inioluwa Deborah Raji, and Ludwig Schmidt. Are we learning yet? A meta review of evaluation failures across machine learning. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021. 20
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. arXiv preprint arXiv:2305.20050, 2023. 1
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740-755. Springer, 2014. 20
Fangyu Liu, Francesco Piccinno, Syrine Krichene, Chenxi Pang, Kenton Lee, Mandar Joshi, Yasemin Altun, Nigel Collier, and Julian Martin Eisenschlos. MatCha: Enhancing visual lan- guage pretraining with math reasoning and chart derendering. arXiv preprint 1rXiv:2212.09662, 2022. 20
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems (NeurIPS), 36, 2024a. 6, 20
Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, al. AgentBench: Evaluating LLMs agents. arXiv preprint arXiv:2308.08 238.023008.,3688, 2023a. 20
Xiao Liu, Zirui Wu, Xueqing Wu, Pan Lu, Kai-We Chang, and Yansong Feng. Are iims capable of data-based statistical and causal reasoning? benchmarking advanced quantitative reasoning with data. arXiv preprint arXiv:2402. 2024b.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. MMBench: Is your multi-modal model all-around player? arXiv preprint arXiv:2307.06281, 2023b. 20
Yuliang Liu, Zhang Li, Hongliang Li, Wenwen Yu, Mingxin Huang, Dezhi Peng, Mingyu Liu, Min- grui Chen, Chunyuan Li, Lianwen Jin, al. On the hidden mystery of OCR in large multimodal models. arXiv preprint arXiv:2305.07895 2023c. 20
Pan Lu, Ran Gong, Shibiao Jiang. Liang Qiu, Siyuan Huang, Xiaodan Liang, and Song-Chun Chun Zhu. Inter-GPe GPS; Interpretable geometry problem solving with formal language and symbolic reasoning. In The 59th Annual Meeting of the Association for Computational Linguistics (ACI.), 2021a. 1,
20, 21, 27
Pan Lu, Liang Qiu, Jiaqi Chen, Tony Xia, Yizhou Zhao, Wei Zhang, Zhou Yu, Xiaodan Liang, and Song-Chun Zhu. IconQA: A new benchmark for abstract diagram understanding and visual lan- guage reasoning. In The 35th Conference Neural Information Processing Systems (NeurIPS) Track on Datasets and Benchmarks, 2021b. 20, 27
Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai- Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In The 36th Conference on Neural Information Processing Systems (NeurIPS). 2022. 5, 20, 27
13

---

Published as a conference paper at ICLR 2024
Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai-Wei Chang, Ying Nian Wu, Song- Zhu, and Jianfeng Gao. Chameleon: Plug-and-play compositional reasoning with large language mod- cls. In The 37th Conference Neural Information Processing Systems (NeurIPS), 2023a. 2, 37
Pan Lu, Liang Qiu, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, Tanmay Rajpurohit, Peter Clark, and Ashwin Kalyan. Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning. In International Conference Learning Representations (ICLR), 2023b. 21
Pan Lu, Liang Qiu, Wenhao Yu, Sean Welleck, and Kai-Wei Chang. A survey of deep learning for mathematical reasoning. In The 61st Annual Meeting of the Association for Computational Linguistics (ACL), 2023c. 8, 20
Ahmed Masry, Xuan Long Do, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. ChartQA: A bench- mark for question answering about charts with visual and logical reasoning. In Findings of the Association for Computational Linguistics: ACL 2022, pp. 2263-2279, 2022. 1, 20,00,2 27
Ahmed Masry, Parsa Kavehzadeh, Xuan Long Do., Enamul Hoque, and Shafiq Joty. UniChart: A universal vision-language pretrained model for chart comprehension and reasoning. arXiv preprint arXiv:2305.14700 2023. 20
Minesh Mathew, Viraj Bagal, Rubèn Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawa- har. Infographics VAA. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1697-1706, 2022. 20, 27
Nitesh Methani, Pritha Ganguly, Mitesh M Khapra, and Pratyush Kumar. PlotQA: Reasoning scientific plots. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1527-1536, 2020. 20, 27
Swaroop Mishra, Matthew Finlayson, Pan Lu, Leonard Tang, Sean Welleck, Chitta Baral, Tanmay Rajpurohit, Oyvind Tafjord, Ashish Sabharwal, Peter Clark, and Ashwin Kalyan. LILA: A unified benchmark for mathematical reasoning. In The 2022 Conference on Empirical Methods in Natural
Language Processing (EMNLP), 2022. 20
Shaghayegh Mobasher, Ghazal Zamaninejad, Maryam Hashemi. Melika Nobakhtian, and Sauleh Eetemadi. Pars ParCCQA- A benchmark for visual question answering and image captioning in persian. people, 101:404, 2022. 20
Harsha Nori, Nicholas King, Scott Mayer McKinney, Dean Carignan, and Eric Horvitz. Capabilities of GPT-4 on medical challenge problems. arXiv preprint UrXiv:2303.13377 2023. 20
OpenAI. Chatgpt, 2022. URL htt..//ttpe///gggaaa...////////a
20
OpenAl. GPT-4 technical report. arXiv preprint arxiv:23333.8774,22002 2023a. 1, 6, 20
OpenAl. GPT-4V(ision) system card, 2023b. URL https:// wtt.aaa/...oeeeeee.eeeaa
pet-4vv-ysttem-card. 1,3
Baolin Peng. Michel Galley, Pengcheng He, Hao Cheng, Yujia Xie, Yu Hu, Qiuyuan Huang, Lars Liden, Zhou Yu, Weizhu Chen, and Jianfeng Gao. Check your facts and try again: Improv- ing large language models with external knowledge and automated feedback. arXiv preprint 302.:22302.12813.. 2023. 97
Rachel Pollitt, Caroline Cohrssen, and Wee Tiong Seah. Assessing spatial reasoning during play: Educator observations, assessment and curriculum planning. Mathematics Education Research Journal, 32(2):331-365 2020. 1
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. LAION-5B: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems (NeurIPS), 35:25278-25294 2022. 20
14

---

Published as a conference paper at ICLR 2024
Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A-OKVQA: A benchmark for visual question answering using world knowledge. European Conference on Computer Vision, pp. 146-162. Springer, 2022. 20, 27
Minjoon Seo, Hannaneh Hajishirzi, Ali Farhadi, Oren Etzioni, and Clint Malcolm. Solving geometry problems: Combining text and diagram interpretation. In Proceedings of the 2015 conference on empirical methods in natural language processing, pp. 1466-1476, 2015. 1, 20, 27
Sanket Shah, Anand Mishra, Naganand Yadati, and Partha Pratim Talukdar. KVQA: Knowledge- aware visual question answering. In Proceedings of the AAAI conference on artificial intelligence, pp. 8876-8884, 2019. 20, 27
Wenqi Shao, Yutao Hu, Peng Gao, Meng Lei, Kaipeng Zhang, Fanqing Meng, Peng Xu, Siyuan Huang, Hongsheng Li, Yu Qiao, al. Tiny LVLM-cHub: Early multimodal experiments with bard. arXiv preprint 2rXiv:2308.03722 2023. 20
Piyush Sharma, Nan Ding. Sebastian Goodman, and Radu Soricut. Conceptual Captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp- 2556-2565, 2018. 20
Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. HuggingGPT:: Solving ai tasks with chatgpt and its friends in huggingface. arXiv preprint arXiv:2303. 17580, 2023. 37
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Mareus Rohrbach. Towards VQA models that read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8317-8326, 2019. 20, 27
Deborah Stipek and Douglas Mac Iver. Developmental change in children's assessment of intellec- tual competence. Child development, pp. 521-538, 1989.
Liangtai Sun, Yang Han, Zihan Zhao, Da Ma, Zhennan Shen, Baocai Chen, Lu Chen, and Kai Yu. SciEval: A multi-level large language model evaluation benchmark for scientific research. arXiv preprint arXiv:2308. 13149, 2023. 20
Sanaz Talaifar and William B Swann. Self-verification theory. Encyclopedia of personality and individual differences, pp. 4813 4821, 2020. 97
John Chong Min Tan and Mehul Motani. Large language model (Im) as a system of multiple expert agents: An approach to solve the abstraction and reasoning corpus (arc) challenge. arXiv preprint arXiv:2310.05144 2023. 21
Leonard Tang. Elizabeth Ke, Nikhil Singh, Bo Feng, Derek Austin, Nakul Verma, and Iddo Drori. Solving probability and statistics problems by probabilistic program synthesis at human level and predicting solvability. In International Conference Artificial Intelligence in Education, pp- 612-615. Springer, 2022. 21
Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. Galactica: large language model for science. arXiv preprint (.X.::22210000:: 2022.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312.1 1805, 2023. 1, 20
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Erie Hambro, Faisal Azhar, et al. LLaMA: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023. 20
Trieu H Trinh, Yuhuai Wu, Quoc V Le, He He, and Thang Luong. Solving olympiad geometry without human demonstrations. Nature, 62577995):476-4482 2024.
15

---

Published as a conference paper at ICLR 2024
Ruocheng Wang, Eric Zelikman, Gabriel Poesia, Yewen Pu, Nick Haber, and Noah D Goodman. Hypothesis search: Inductive reasoning with language models. arXiv preprint wrXiv:2309.05660 2023a. 21
Xiaoxuan Wang, Ziniu Hu, Pan Lu, Yanqiao Zhu, Jieyu Zhang. Satyen Subramaniam, Arjun R Loomba, Shichang Zhang, Yizhou Sun, and Wei Wang. SciBench: Evaluating college-level sci- entific problem-solving abilities of large language models. arXiv preprint arXiv:2307. 10635, 2023b. 2, 20, 27
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint 2rXhh:2208./1171, 2022. 103
Yan Wang, Xiaojiang Liu, and Shuming Shi. Deep neural solver for math word problems. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing 2017.
(EMNLP), pp.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yo- gatama, Maarten Bosma, Denny Zhou, Donald Metzler, al. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682 2022a. 20
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903 2022b. 2, 5, 6, 21, 103
Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebastian Gehrmann, Prab- hanjan Kambadur, David Rosenberg, and Gideon Mann. BloombergGPT: A large language model for finance. arXiv preprint arXiv:2303. 17564, 2023. 1
Peng Xu, Wenqi Shao, Kaipeng Zhang, Peng Gao, Shuo Liu, Meng Lei, Fanqing Meng, Siyuan Huang, Yu Qiao, and Ping Luo. LVLM-eHub: A comprehensive evaluation benchmark for large vision-language models. arXiv preprint arXiv:2306.0922
2023. 20
Hongyang Yang, Xiao- Xiao-Yang Liu, and Christina Dan Wang. FinGPT: Open-source financial large language models. arXiv preprint arXiv:2306.06031, 2023a.
Zhengyuan Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Chung-Ching Saig-Ching Lin, Zicheng Liu, and Li- juan Wang. The Dawn of LMMs: Preliminary explorations with gpt-4v(ision). arXiv preprint arXiv:2309. 17421, 2023b. 97
Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. mPlug-Owl: Modularization empowers large language mod- els with multimodality. arXiv preprint arXiv:2304. 14178, 2023. 6, 20
Da Yin, Liunian Harold Li, Ziniu Hu, Nanyun Peng, and Kai-Wei Chang. Broaden the vision: Geo-diverse visual commonsense reasoning. arXiv preprint arXiv:2109.06880 2021. 20
Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. MM-Ves Vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308. 320.002900 2023. 20
Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. From recognition to cognition: Visual commonsense reasoning. Proceedings of the IEEE/CVF conference on computer vision and paltern recognition, pp. 6720-6731, 2019. 20
Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao, and Yu Qiao. LLaMA-Adapter. Efficient fine-tuning of language models with zero-init attention. In International Conference on Learning Representations (ICLR), 2024. 20
Xiang Zhang, Senyu Li, Zijun Wu, and Ning Shi. Lost in translation: When gpt-4v (ision) can't see eye to eye with text. a aasion-language-conisttenng analysis of vllms and beyond. arXiv preprint arXiv:2310. 12520, 2023a. 21
16

---

Published as a conference paper at ICLR 2024
Xiaoman Zhang, Chaoyi Wu, Ziheng Zhao, Weixiong Lin, Ya Zhang, Yanfeng Wang, and Weidi Xie. PMC-VQA: Visual instruction tuning for medical visual question answering. arXiv preprint arXiv:2305.10415, 2023b. 20, 27
Yanzhe Zhang, Ruiyi Zhang, Jiuxiang Gu, Yufan Zhou, Nedim Lipka, Diyi Yang, and Tong Sun. LLaVAR: Enhanced visual instruction tuning for text-rich image understanding. arXiv preprint
arXiv:2306.17107, 2023c. 6.20
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. MiniGPT-4: En- hancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304. 10592, 2023a. 6, 20
Wanrong Zhu, Jack Hessel, Anas Awadalla, Samir Yitzhak Gadre, Jesse Dodge, Alex Fang, Young- jae Yu, Ludwig Schmidt, William Yang Wang, and Yejin Choi. Multimodal C4: An open, billion- scale corpus of images interleaved with text. In Neural Information Processing Systems (NeurIPS) Track Datasets and Benchmarks, 2023b. 20
17

---

Published as a conference paper at ICLR 2024
CONTENTS
Related Work
20
B Limitations of the Benchmark
21
Data Collection Guidelines
22
C.1 Mathematical Reasoning Definition
22
C.2 Mathematical Reasoning Examples
23
C.3 Visual Context Types
24
C.4 Source Dataset Summary
27
D Data Collection Details
28
D.I Automatic Selection of Mathematical Problems
28
D.2 Human Labeling of Mathematical Problems
................................
28
D.3 Annotating Three New Datasets
29
D.4 Human Labeling of Mathematical Reasoning
E More Dataset Analysis
30
More Details on the Setup
33
F.1 Frequent Guess
33
F2 Prompt for Answer Extraction
33
F3 Prompts for Response Generation
34
F.4
Prompt for Caption Generation
34
F55
Model Hyperparameters
34
F6
Human Performance
34
F7
Multimodal Bard Assessment Task
35
G More Experimental Results
36
G.1 Results on the Test Set
36
G.2 Scores for Math Reasoning Types
36
G.3 Scores for Various Visual Contexts
37
G.4 Scores Across Different Grade Levels
G.5 Ablation Study for LLMs
38
G.6 LLMs with Different Shots
39
G.7 LMMs with Different Shots
39
G.8 Hallucinations in Model Explanations
40
G.9 More Examples for Multimodal Bard
G.10 Comparisons of Different Models
47
18

---

Published as a conference paper at ICLR 2024
H A Comparative Study of GPT-4V, Bard, and Other Models
54
H.1
GPT-4V Playground for Manual Evaluation
54
H.2
Leaderboard Scores
55
H.3
Abilities in Mathematical Reasoning
56
H.3.1
Algebraic Reasoning
56
H.3.2
Arithmetic Reasoning
59
H.3.3
Geometry Reasoning
61
H.3.4
Logical Reasoning
63
H.3.5
Numeric Commonsense Reasoning
66
H.3.6
Scientific Reasoning
69
H.3.7
Statistical Reasoning
72
H.4
Abilities Across Visual Contexts
74
H.4.1
Abstract Scene
74
H.4.2
Bar Chart
76
H.4.3
Function Plot
77
H.4.4
Geometry Diagram
79
H.4.5
Line Plot
H.4.6
Natural Image
83
H.4.7
Puzzle Test
85
H.4.8
Scatter Plot
87
H.4.9
Scientific Scene
89
H.4.10
Synthetic Scene
92
H.4.11
Table
94
H.4.12 Other Visual Contexts
96
H.5
Self- Verification in GPT-4V
97
H.6
Self-Consistency for GPT-VV
103
H.7 GPT-4V for Multi-Turn Human-Al Al Interaction.
109
19

---

Published as a conference paper at ICLR 2024
A RELATED WORK
Mathematical reasoning benchmarks. Recently, numerous benchmarks (Amini et al., 2019; Cobbe et al., 2021; Mishra et al., 2022; Frieder et al., 2023) have been proposed to evaluate the math- ematical reasoning capabilities of Large Language Models (LLMs). However, most of these are tex- tual only (Lu et al., 2023c), despite a substantial amount of mathematical information and reasoning being encapsulated in visual modalities. Meanwhile, some datasets exhibit performance saturation; for instance. GPT-4 achieves 92.0% accuracy on GSMSK (Cobbe et al., 2021), a dataset of grade- school mathematics questions. On the other hand, the recent rapid advancement of Large Multi- modal Models (LMMs) necessitates the establishment of robust multimodal benchmarks. However, current multimodal reasoning benchmarks provide limited coverage of rigorous and scientific do- mains (Antol et al., 2015; Kembhavi et al., 2016; Kahou et al., 2017; Mathew et al., 2022), which are key components for creating general-purpor Al assistants. To bridge this gap, it is crucial to develop a robust math reasoning dataset that integrates visual contexts.
Vision-language reasoning benchmarks. High-quality evaluation datasets and benchmarks are a cornerstone for assessing the progress of machine learning models to solve real-world tasks Liao et al. (2021). Prior studies such as VQA (Antol et al., 2015; Goyal et al., 2017), VizWiz (Gurari et al., 2018), and Pars PPrs-QQsCaas (Mobasher al., 2022) assess the general-purpose visual question answering abilities of the LMMs, with or without task-specific training. on open-ended questions about images. In addition, there are several works that focus on evaluating specific skills of the LMMs beyond natural scenes, such as abstract scenes and shapes) (Antol et al., 2015; Lu et al., 202 lb; Ji ct al., 2022), geometry diagrams (Sco et al., 2015; Lu et al., 2021a; Chen et al., 2022a; Cao & Xiao, 2022), figures and charts (Methani et al., 2020; Masry et al., 2022; Kahou et al., 2017; Chang et al., 2022; Kafle et al., 2018), documents (text in images) (Singh et al., 2019; Mathew synthetic images (Dahlgren Lindström & Abraham, 2022; Li et al., 2023c; Bitton-Guette et al., 2023). Besides, there has been significant progress on developing datasets judge LMMs skills that require external knowledge (Schwenk et al., 2022: Shah et al., 2019), common sense reasoning (Zellers et al., 2019; Yin et al., 2021), scientifi--knowlddge (Lu al., 2022; Kembhavi et al., 2017: 2016), medical understanding (Zhang et al., 2023b; Lau al., 2018). In this work, we create new datasets (IQTest, FunctionQA, PaperQA) and subsequently design benchmark for holistic evaluation of the math reasoning capabilities of the LMMs.
et al., 2022: Liu et al., 2023c),
Generative foundation models and their evaluation. Recently, there has been a surge of genera- tive foundation models (Bommasani et al., 2021) that are trained on web-scale data, such as GPT-3, ChatGPT, GPT-4, Claude, LLaMA, LLaMA-Adapter (Brown et al., 2020; OpenAl, 2022; 2023a; Anthropic, 2023; Touvron et al., 2023; Zhang et al., 2024), with the ability to solve a wide range of downstream tasks (Wei et al., 2022a) without any task-specific finetuning. Prior work has focused on evaluating their abilities to respond to the queries from various disciplines, grounded in text, such as QA, math, medicine, coding and science (Bubeck et al., 2023; Nori et al., 2023; Chen et al., 2021; Fu et al., 2023; Sun et al., 2023; Wang et al., 2023b; Huang et al., 2024; 2022: Liu et al., 2023a; Zhang et al., 2024: Gu et al., 2024). Prior work, such as PixStruct (Lee et al., 2023), MatCha (Liu al., 2022), and UniChart (Masry et al., 2023), has focused developing specialized pretraining recipe for improved math and chart reasoning in visual contexts.
On the vision- side, there are several generative foundation models such as LLaVA, miniGPT4, Instruct Flamingo, LLaMA-Adapter V2, Multimodal Bard, Gemini, SPHINX-X, Claude 3 (Liu al., 2024a; Zhu et al., 2023a; Dai al., 2023; Alayrac et al., 2022: Awadalla al., 2023; Gao et al., 2023; Google, 2023; Team et al., 2023; Gao et al., 2024; Anthropic, 2024) that are trained on vast amount of paired (Schuhmann et al., 2022; Sharma et al., 2018; Lin et al., 2014) and interleaved image-text data (Zhu et al., 2023b). In addition, there has been recent development on specialized versions of these LMMs for document understanding where visual contexts require text recognition, math understanding being one of them (Zhang et al., 2023c; Ye et al., 2023). In recent times, there have been several works, such as Visit-Bench, LVLM-eHub, MMBench (Bitton et al., 2023; Yu et al., 2023; Liu et al., 2023b; Xu et al., 2023; Shao et al., 2023), that assess their cnlrucction-oolooinng and reasoning capabilities. As the generative foundation models become more relevant to real-world applications, unlike prior work, we propose MATHVISTA to benchmark their capabilities of math reasoning (logical, arithmetic, statistical) on a diverse set of visual contexts (word problems in images, natural scenes, geometrical shapes, and plots).
20

---

Published as a conference paper at ICLR 2024
Recent work of LLM prompting and GPT-4V. We have witnessed the remarkable abilities of large language models (LLMs), and their reasoning capabilities are further enhanced by promoting approaches such chain-of-thought (CoT) (Wei et al., 2022b), program-of- thouuhhtttthtttthh (PoT) (Chen et al., 2022b), and inductive reasoning (Wang et al., 2023a; Tan & Motani, 2023). For example, the feasibility of using L.MMs solve the Abstraction and Reasoning Corpus (ARC) challenge has been verified using zero-shot, few-shot, and context-groundee prompting (Tan & Motani, 2023). In this paper, we evaluate LLMs using zero-shot, few-shot, CoT prompting, PoT prompting, well as aoo--augentee prompting, to explore their potential in solving mathematical reasoning in visual contexts MATHVISTA. moggam--aiee methods are widely used for mathematical reasoning due to their advancements in precise logical reasoning and arithmetic calculations (Drori & Verma, 2021; Tang et al., 2022; Drori al., 2022). In this work, we have developed the LLM baselines with PoT.
Recently, OpenAl released GPT-4V, the multimodal version of GPT-4, which shows promising per- formance in vision-language reasoning. However, the fine-grained study of its strengths and limi- tations still remains underexplored. The recent work (Zhang et al., 2023a) contributes pioneering efforts in this field, studying whether large multimodal models (LMMs), like GPT-4V, execute vi- sion and language tasks consistently independently. As concurrent work, our paper provides, for the first time, a comprehensive quantitative and qualitative study of GPT-4V and other LLMs in mathematical reasoning within visual contexts.
B LIMITATIONS OF THE BENCHMARK
Our benchmark, VITHVISTA, makes significant contributions by combining mathematical and vi- sual tasks, a domain where existing models like GPT-4V have shown promise but also face chal- lenges, especially in complex figure understanding and rigorous reasoning. While we have made strides in evaluating model performance, we acknowledge several limitations.
One limitation is the dataset coverage. While MATIIVISTA encompasses a broad spectrum of tasks and visual contexts, there may be gaps in the representation of certain types of mathematical prob- lems and visuals. Furthermore, the dataset's focus on mathematical reasoning within visual contexts, spanning specific domains like science and college-leve math, necessitates a more labor-intensive process for collecting high-quality data compared to textual-only or general-purpose datasets. Thus, the scalability and generalizability of our benchmark to other domains remain concern. Anno- tations were sourced from original data providers, resulting in only 85.6% of examples (Table 1) having annotations. Due to the heterogeneity of these sources, annotations lack unified format and structure. For example, the annotations could be logic forms of the problem parsing from Geome- try3K (Lu et al., a0) natural language solutions from TabMWP (Lu et al., 2023b), and theorems from TheoremQA (Chen et al., 2023). Given the rapid development in foundation models, our study focused exclusively on the most recent and prominent models.
In future iterations, our benchmark will be beneficial to encompass a broader array of problems and visual contexts, while also providing unified and comprehensive annotations. Our benchmark is part of an ongoing research process, and we are committed to maintaining the datasets, such as refining the potential data noise, response the community feedback. Also, are committed evolving the leaderboard in response to new models.
In conclusion, while there are limitations to our current approach, MATH VISTA represents signif- icant step forward in the field. We are dedicated to continuously improving our benchmark to better understand and enhance the capabilities of Al in mathematical and visual reasoning.
21

---

Published as a conference paper at ICLR 2024
C DATA COLLECTION GUIDELINES
C.I MATHEMATICAL REASONING DEFINITION
Seven mathematical reasoning types are defined in Table
Math Reasoning
Description
It covers the fundamenial operations such addition, subtraction, multiplication, di- vision, and understanding of namber propertiex. may also include the ability interpret numerical data in different forms.
Arithmetic Reasoning
(34.1%)
It focuses on data interpretation and analyvis, including measures (mean, median, mode), dispersion metrics (standard deviation, range), probability concepts, regres- sion, correlation, and data inferences. It also identifies trends, outliers, and patterns.
Statistical Reasoning (30.5%)
It encompasses understanding variables, equations, and the manipulation of expres- sions with polynomials and exponents. It also covers solving simple to complex equa- tions, and grasping functions, their properties, and graphical depictions.
Algebraic Reasoning (28,5%)
It emphasizes spatial understanding, analysis of 2D and 3D figures, and reasoning about their shapes, sizes, and relationships. It includes symmetry, congruency, simi- larity, area, volume, and transformations.
Geometry Reasoning (23,3%)
Numeric common scnse (14.0%)
It involves intuitive understanding of daily numerical concepts, including understand- ing time differences, numerical judgment, and estimates. covers temporal reasoning, spatial numeric assessments, and practical uses like budgeting and time reading.
It deals with the application of mathematical concepts in scientific contexts. This includes scientific notations, formula use, understanding rates, proportions, and per- centages in practical situations, and problem-solving in scientific inquiries.
Scientific Reasoning (10.7%)
It focuses on critical thinking and deduction from provided information. including pattern recognition, sequence understanding, predictions, and statement evaluation. Key components include premises, conclusions, and the of abstract reasoning.
Logical Reasoning (3.8%)
Table 3: Definitions and proportions of seven mathematical reasoning categories in MATHVVSST.
22

---

Published as a conference paper at ICLR 2024
C.2
MATHEMATICAL REASONING EXAMPLES
Math
Examples
Question:
Karen bought 4 pounds of silk scraps and pounds of canvas scraps. How much did she spend?? (Unit: $)
silk scraps
$9.08/bb
denim scraps
$8.47/lb
Solution:
carvva scraps
$8.17/lb
Find the cost of the silk scraps. Multiply: $9.08 x 4 $36.32
ARI
felt scraps
$7.29/lb
Find the cost of the canvas scraps. Multiply: $8.17 x 4 $32.68
Now find the total cost by adding: $36.32 + $32.68 S69
faux fur scraps
$11.79/bb
She spent $69.
lace scraps
$6.37/lb
Answer: 69
6.20
Seglnh huloned Mandarts Muntia Fetowedd by Begins
Question: How many sequences have nega- tive Influence Scores? Answer: 2
0.33
STA
8.80
0.02
Sequesoos
Question: The derivative of at is that 8.
Choices: (A) larger (B) equal (C) smaller than
long Ea)
Answer: (A) larger than
ALG
Question: How many zeros does this function have?
Answer:
Question: What is the value of y at x 1? Answer: 0
Question: AB a diameter, AC = inches, and BC 15 inches. Find the radius of the circle.
Diagram logic forms:
FointLiesOnLine (D, Line (B, A)
GEO A
B
Foint (B, Circle radius)
Point Li esonči Lissnnii rele (A, Circle (D, radius)
PointLiesOnCine (C, Circle (D, radius)) Answer: (C) 8.5
Question: What is the age gap between these two people in image? (unit: years)
Named entities: Winston Churchill, Charles de Gaulle
NUM
Wiki caption: Winston Churchill and General de Gaulle Marrakesh, January 1944
Answer: 16
(mg: L
Question: The graph of the concentration function c(t) shown after a 7-mg injection dye into heart. Simpson's 's Rule to estimate the cardiac output. Answer: 5.77
SCI
10
14
(scconds)
Brain Teaser IQ Test
Question: Find the value of the square the figure.
Solution:
Circle Square 5, Triangle + Triangle
Triangle =
Circle + Triangle = 7, Circle
Therefore Square = 2
Answer:
Table 4: Examples of seven mathematical reasoning categories in MATUVVSSA.
23

---

Published as a conference paper at ICLR 2024
C.3
VISUAL CONTEXT TYPES
3y
cm
F
13
49
2x
20cm
5ga
Figure 7: Examples the visual context for the geometry diagram type.
Figure 8: Examples the visual context for the synthetic scene type.
Acmunary difiement algorittime
espect of U.S-Gemean relatiosshp
leaf
Afsidseed
15
damain See
wiarchy 138
Ocon-spurted
Sent Eogims inceren cuer LE
AgertBench Ovenal Scoee
AaciiBenc's ucrow
Dakkel lino
Figure 9: Examples of the visual context for the bar chart type.
100
Figure 10: Examples of the visual context for the natural image type.
R
V.
H
OH
Palwwatis Tasf
Phonylalanine
Faeeth
W
Lesf
Figure 11: Examples of the visual context for the scientific figure type.
24

---

Published as a conference paper at ICLR 2024
foxd collected
ollanr
$3.18 per kilogram
Skite Spylem
Settings
Craned
parsley
$3.10 per kilagram
Wil
Wio Lox
29.371
0379
6.093
resemary
$3.52 per kilogram
wil Plane Consitency
90
8915
6.058
Vocel
318
0330
wio Foeward. Neerral Ray
31.108
B921
6/132
oregano
por itlogram
win joint Optimization
67.01
03775
0.186
mint
$1.06 per illogram
11.0
Pull Model
52.622
6933
00377
Picuree
chamronille
$2.04 per Kilogram
Neyaaa
400
145
Table 3: We quantitatively analyse our medel design and training schemes the synthetic bedroom.
Figure 12: Examples of the visual context for the table type.
kg
Figure 13: Examples of the visual context for the function plot type.
8
4
5
Figure 14: Examples of the visual context for the abstract scene type.
22
35
32
12
Figure 15: Examples of the visual context for the puzzle test type.
Figure 16: Examples of the visual context for the scatter plot type.
Coraee India Ria Apals
Hulbee
Switching Anabyls
O pping Anclpp's
HusFrsOA (100
13 24
241
Tam und cheneeeeeiiee pofferanc powesex ber vaccon
Figure 17: Examples of the visual context for the line plot type.
25

---

Published as a conference paper at ICLR 2024
Seyon
online socil
Republicane Democrate say they agree
wan (12)
alte chagge met
Figure 18: Examples of the visual context for the pie chart type.
Fitted (AM DOOPATS
Figure 19: Examples of the visual context for the document image type.
Figure 20: Examples of the visual context for the medical image type.
Opensting
10100n
24 1155
LDVEB
Figure 21: Examples of the visual context for other types, including word cloud, map chart, radar chart, violin plot, and heatmap chart.
26

---

Published as a conference paper at ICLR 2024
C.4
SOURCE DATASET SUMMARY
The source datasets are summarized in Table 5.
Dataset
Category
Task
Context
Math Skill
IQTest (Ours)
Marh-Targeted
FQA
Puzzle Test
Logical, Arithmetic Scientific
PaperQA (Ours)
Marg-Targeted
FQA
Charts and Plots
Function (Ours)
Marh-Targeted
TQA
Function Plot
Algebraic
GeometryåK (2021a)
Math-Targettd
GPS
Geometry Diagram
Geometry, Algebraic
GeoQA+ (2022)
Math-Targeted
GPS
Geometry Diagram
Geometry, Algebraic
GEOS (2015)
Math-Targeted
GPS
Geometry Diagram
Geometry, Algebraic
UniGeo (2022a)
Targeted
Geometry Diagram
Geometry, Algebraic
CatVR-Math (2022)
Math-Targeted
MWP
Synthetic Scene
Arithmetic
IconQA (2021b)
Targ-Targeted
MWP
Abstract Scene
Arithmetic
TabMWP (2023b)
Math-Targe har-TTargeted
MWP
Table
Statistical, Arithmetic
SciBench (2023h)
Math- Targeted
TQA
Scientific Figure
Scientific
TheoremQA (2023)
Math-Targee Targeted
TQA
Scientific Figure
Scientific
ChartQA (2022)
General VQA
FQA
Charts and Plots
Statistical
FigureQA (2017)
General VQA
FQA
Charts and Plots
Statistical
DVQA (2018)
General VQA
FQA
Bar Chart
Statistical
MapQA (2022)
General VQA
FOA
Map Chart
Statistical
PlotQA (2020)
General VQA
FQA
Scatter Plot
Statistical
DocVQA (2022)
General VQA
FQA
Document Image
Statistical
A12D (2016)
General VQA
TQA
Scientific Figure
Scientific
ScienceQA (2022)
General VQA
TQA
Scientific Figure
Scientific
TQA (2017)
General VQA
TQA
Scientific Figure
Scientific
A-OKVQA (2022)
General VQA
VQA
Natural Image
Arithmetie, Numeric
KVQA (2019)
General VQA
VQA
Natural Image
Arithmetic, Numeric
ParsVOA-Caps (2022)
General VQA
VQA
Natural Image
Arithmetic, Numeric
TextVQA (2019)
General VQA
VQA
Natural Image
Arithmetic, Numeric
VizWiz (2018)
General VQA
VQA
Natural Image
Arithmetic, Numeric
VQA2.0 (2017)
General VQA
VQA
Natural Image
Arithmetie, Numeric
PMC-VQA (2023b)
General VQA
VQA
Medical Image
Scientific
VQA-RAD (2018)
General VQA
VQA
Medical Image
Scientific
Super-CLEVR (2023c)
General VQA
VQA
Synthetic Seene
Arithmetic
VQA-AS (2015)
General VQA
VQA
Abstract Scene
Arithmetic
Table 5: Summary of the 31 different source datasets in MATHVISTA. Among these, FunctionQA, IQTest, and PaperQA are our newly annotated datasets. The table provides details on their category, task, visual context, and primary mathematical reasoning skill types.
27

---

Published as a conference paper at ICLR 2024
D
DATA COLLECTION DETAILS
D.1 AUTOMATIC SELECTION OF MATHEMATICAL PROBLEMS
most, least, fewest more, less, fewer, largest, smallest, greatest, larger, smaller, greater, highest, lowest, higher, lower, increase, decrease, minimum, maximum, max, min, mean, average, median, total, sum, add, subtract, difference, quotient, gap. half, double, twice, triple, square, cube, root, approximate, approximation, triangle, rectangle, circle, square, cube, sphere, cylinder, cone, pyra- mid, multiply, divide, percentage, percent, ratio, proportion, fraction, rate
Table 6: Dictionary of quantity words used for the automatic selection of questions likely to involve mathematical reasoning.
D.2 HUMAN LABELING OF MATIEEMATICAL PROBLEMS
Home
Welcome! You are editing the A-OKVQA dataseti (problem 8, uss 7 7/94)
Problem Diagram
Choices
Vogetarian
ketoçemic
Vogetarian
Comment
roblem Text
person followngg whatiine of diet luust Bou
problem involes mnttematt
Netm
Figure 22: GUI for labeling if problem involves mathematical reasoning.
We are compiling dataset that incorporates image context and involves mathematical reasoning (MathQA in visual contexts). We have gathered set of examples in which some involve mathe- matical reasoning, while others do not.
In our task, question can be classified mathematical problem if
Involves numbers symbols in the question text or the image context, AND requires further operations or transformations to be performed on them to reach a solution.
Involves more complex forms of mathematical reasoning, including logical reasoning, abstract thought, and understanding of patterns.
Based on the definition above, problem classified as a negative example (NOT involving math- ematical reasoning) if it:
Does not involve any numbers or quantity words, OR
Involves only counting, reading, or recognizing numbers, OR
Relics solely on factual information, such as recalling years and dates.
Table 7: Instructions for human annotators to identify if a problem involves mathematical reasoning.
We developed an annotation tool, as illustrated in Figure 22, to enable expert annotators to label problems that involve mathematical reasoning. Annotators were trained using detailed instructions,
28

---

Published as a conference paper at ICLR 2024
shown in Table 7, along with variety of examples positive that involve mathematical reasoning and negative ones that do not. We provided three labeling options:
Yes This indicates that the problem involves mathematical reasoning.
No - This indicates that the problem does not involve mathematical reasoning.
Unsure This option should be selected if it is uncertain whether the problem involves mathematical reasoning. (Annotators are advised
this option sparingly.)
They may leave comments if they find anything incorrect or offensive for removal at later stage.
In our study, we employed the Fleiss Kappa score to conduct an aneer-nnntttor agreement analysis among three annotators tasked with labeling examples based on mathematical reasoning. The Fleiss Kappa score statistical measure used to evaluate the reliability agreement between multiple raters, providing quantifiable metric to assess the consistency across different annotators. A score of indicates perfect agreement, while score of 0 suggests no agreement beyond what would be expected by chance. Our analysis yielded Fleiss Kappa score of 0.775, indicating substantial level of consistency among the annotators. This high degree of agreement underscores the reliability of our annotation process and affirms the quality of the labeled data generated for our study.
D.3 ANNOTATING THREE NEW DATASETS
Protdern Imagg
Prbblen Test
2
6
?
which iu mboo kimkseegg
Choices (Optionel)
Option
3
3
Anewver
2
4
2
3
sigif
Submit
Figure 23: GUI for annotating our new source datasets.
D.4 HUMAN LABELING OF MATHEMATICAL REASONING
Home
Welcome! You labeling the mathematical reasoning skills! (prablem 46
Previcus
Next
Problem Diagram
Choices
SPIDER LIFE CYCLE
Ess sBc
Adult spider population would remain the
Adutt
Adult spider population would double.
Adiuts spider population would decneese
Easr
Adult spider papulittion
Answer
Spidertings
Boby spidentnng
spider population
of following mathematical skills problem involve?
What would happon the adult spiders predotor the spi der agps?
Logical
Scientific
Commonsense
Geometry
Algebraic
Statistical
Arithmetic
Figure 24: GUI for labeling mathematical reasoning skills.
29

---

Published as a conference paper at ICLR 2024
E MORE DATASET ANALYSIS
Question distribution. Apart from English questions, MATHVISTA contains 6.57% non-English questions, including languages such Chinese and Persian. The multilingual feature necessitates that models be capable of understanding and processing multiple languages to ensure accurate results the dataset. As illustrated Table the average number of words in English questions within MATHVISTE is 15.58, while the maximum number of words in question reaches 213.
Figure 25 further elucidates the distribution of word counts, highlighting the diverse patterns of questions. MATH VISTA features two types of questions: multiple-choiee questions and free-form questions. For multiple-choice questions, the average number of choices is 3.4, while the maximum number of choices is 8. In the case of free-form questions, answers can be integers, floating-point point numbers, or lists, which can be converted into a standard format. The standard settings in question and answer types facilitate consistent accuracy evaluation for existing models.
Distribution Number Question Words
Mean 15.58
Median
Percentage (96)
10
20
30
40
50
60
Question Length
Figure 25: The distribution of the number of words per question MATUVISTA. Questions with length greater than 60 are categorized as 61 for visualization simplicity.
Dataset category and task type. Source datasets in MATHVISTA can be categorized into two types: math-targeted VQA datasets, which are originally proposed for assessing mathematical rea- soning, and general VQA datasets, which address visual reasoning in everyday scenarios. The dis- tribution proportions these two categories (55,4% vs. 44.6%, as illustrated in Figure 26) within MATH enable balanced examination of mathematical reasoning in both domain-specii general-purge purpose applications. The distribution of the five tasks contained within MATH VISTA is vi- sualized in Figure 27, The relatively balanced distribution of these tasks enhances the benchmarking robustness that our dataset provides.
and
Math-targeted VQA VQA
44.6%
2,739
55.
3,402
Figure 26: Category distribution of problems within MATHVISTA.
Grade level. The datasets within MATIVVISTA categorized into four distinct grade levels: el- ementary school. high school, college, and not applicable, each representing a different level of reasoning complexity and contextual application. The elementary school category aligns with the typical mathematical curriculum elementary education, introducing basic topics such as arith- metic operations and introductory geometry. High school level questions delve into more complex
30

---

Published as a conference paper at ICLR 2024
Figure question answering
15.3% 937
Geometry etry problem solving
Math word problem
1,647
8%
Visual question answering
Textbook question answering
16.9% 1,038
19.5%
1,200
Figure 27: Task type distribution of problems within MATHVISTA.
mathematical concepts such as algebra, geometry, and introductory calculus. The college category encapsulates the highest level of complexity, featuring questions on advanced mathematical and sci- entific concepts like calculus, linear algebra, and physics. Questions without specific grade levels are categorized as not applicable.
The distribution of questions across these grade levels is visualized in Figure 28. This structured categorization enriches the diversity of MATHVISTA, providing a meaningful framework for evalu- ating and benchmarking the mathematical and visual reasoning capabilities of various models across different educational contexts, thereby assessing their practical utility and educational relevance.
Not applicable
10. 8%
Elementary school
High school
College
2,313
30 9% 1,895
20.7% 1,272
Figure 28: Distribution of questions across different grade levels within MATHVISTA.
Visual context. The datasets within MATHVVSTA encompass over 10 different visual contexts (with the distribution shown in Figure 29), crucial for evaluating models' ability to interpret and reason across diverse visual information. Common visual contexts include geometry diagrams, syn- thetic scenes, bar charts, natural images, and scientific figures illustrated in Figure to Figure 19. Less frequent, yet equally important visual contexts such as medical images, word clouds, map charts, radar charts, violin plots, and heatmap charts depicted in Figure 20 and Figure 21. These visual contexts, ranging from common to specialized representations. challenge the models to de- code and reason with varying visual information, contributing more robust and comprehensive evaluation. The diversity in visual contexts enriches MATIIVISTA enhancing the benchmarking ro- bustness and providing solid foundation for understanding the practical utility and comain-speccio performance of various models across different domains and applications.
Mathematical reasoning ability. The datasets within MATHVISTA encompass spectrum of seven distinct mathematical reasoning types, facilitating thorough evaluation of models' mathe- matical reasoning capabilities. Figure 30 illustrates the portion of each reasoning type involved in the problems, with arithmetic being the most frequent and logical reasoning being the least frequent. This distribution reflects the varying degrees of mathematical reasoning required across different problems. Figure 31 further delineates the distribution of reasoning types, showcasing mean of
31

---

Published as a conference paper at ICLR 2024
Geomeery diagram
2
22.3 2..3%
Synthetic scene
7%
Bar chart
6.1% %
Natural image
Scientific figure
6.5%
Table
Function plot
7.3%
13.0%
13.
Abstract scene
Puzzle test
8.4%
Scatter plot
12.7% %
Line plot
10.0%
Pie chart
Others
Figure 29: Visual context distribution within MATHVISTA.
1.45. The sparse distribution observed aids the precise analysis of each type's performance by the models, providing nuanced understanding of their strengths and weaknesses across different mathematical reasoning domains. This structured representation of mathematical reasoning types within MATH not only enriches the dataset but also significantly contributes to more robust and comprehensive evaluation of models, aiding in the identification of areas for improvement and the development of more proficient mathematical reasoning models.
Logical reasoning
na
Scientific reasoning
Numeric commonsense
14.0%
Geometry reasoning
23.3%
Algebraic reasoning
28.5%
Statistical reasoning
Arithmetic reasoning
0%
5%
10%
15%
20%
25%
30%
35%
40%
Figure 30: Portion of each mathematical reasoning type involved in the problems of MATHVISTA.
Distribution of Number of Mathematical Reasoning Classes
Mean 1.45
Median 1.00
3000
2500
Frequency
1500
1000
500
0
3
Number of Skills
Figure 31: Distribution of the number of mathematical reasoning types within MATH VISTA.
32

---

Published as a conference paper at ICLR 2024
F MORE DETAILS ON THE SETUP
F. FREQUENT GUESS
We employ strategy where the most frequent answers in the lestmini set utilized as predictions for various question and answer types. For multiple-choice questions, the most frequent option is selected based on the number of available options. For instance, option B chosen for questions with two options, aligning with the answer distribution in festmini. Similarly, for questions requir- ing an answer type of integer, a floating number with one decimal place, a floating number with two decimal places, list, we use 2, 1.2, 0.21, and [0, 2. 0, 1, 7, 0, 3, 0, 6) respectively, in accordance with the answer distribution observed in festmini.
F.2
PROMPT FOR ANSWER EXTRACTION
The prompt used instruct GPT-4 for answer extraction illustrated in Table 8.
Element
Prompt
Please read the following example. Then extract the answer from the model response and type it the end of the prompt.
Task description
Hint: Please answer the question requiring an integer answer and provide the final value, c.g., 2, 3, at the end.
Question: Which number is missing?
Example
Model response: The number missing the sequence
Extracted answer: 14
Hint: Please answer the question requiring a floating-point number with decimal place and provide final value, e.g., 1.3, 1.4, the end. Question: What is the fraction of females facing the camera?
Example
Model response: The fraction of females facing the camera is 0.6, which means that six out of females the group are facing the camera.
Extracted answer: 0.6
Hint: Please answer the question requiring a floating-pon point number with two decimal
places and provide the final value, c.g., 1.23, 1.34, 1.45, at the end.
Question: How much money does Luca need to buy apple candy and butter- scotch candy? (Unit:
Example
Model response: Luca needs $1.45 to buy sour apple candy and butterscotch candy.
Extracted answer: 1.45
Hint: Please answer the question requiring Python list an answer and provide the final list, e.g., 21 3]. [1.2, 1.3, 1.4), the end. Question: Between which two years does the line graph saw its maximum peak?
Example
Model response: The line graph saw its maximum peak between 2007 and 2008.
Extracted answer: [2007, 2008]
Hint: Please answer the question and provide the correct option letter, e.g., A. B. C.
D. at end,
Question: What fraction of the shape is blue? Choices: (A) 3/11 (B) 8/11 (C) 6/11 (D) 3/5
Example
Model response: The correct answer (B) 8/11.
Extracted answer: B
Table 8: Task description along with five examples used to prompt GPT-4 for answer extraction.
33

---

Published as a conference paper at ICLR 2024
F.3
PROMPTS FOR RESPONSE GENERATION
Question type
Answer type Task instruction
Please answer the question and provide the correct option letter, c.g., A, B, C, D, at the end.
multiple-choice
Text
Please answer the question requiring an integer answer and provide the final value, c.g. 1, 2, 3, at the end.
Free-form
Integer
Please answer the question requiring floating-point number with one dec- imal place and provide the final value, e-g., 1.2, 1.3, 1.4, at the end.
Free-form
Float (1)
Please answer the question requiring floating-poin number with two dec- imal places and provide the final value, e.g., 1.23, 1.34, 1.45, the end.
Free-form
Float (2)
Please answer the question requiring Python list as an answer and provide the final list, e.g., [1, 2, 31, [1.2, 1.3, 1.4), at the end.
Free-form
List
Table 9: The task instructions for different question and answer types in answer extraction. Here, Float (1) refers to floating-poit number with one decimal place, and Float (2) refers to a floating- point number with two decimal places.
F.4 PROMPT FOR CAPTION GENERATION
We instruct Multimodal Bard to generate detailed description for an input image, aiming aug- ment current LLMs with visual understanding capabilities. The prompt is shown in Table 10.
Describe the fine- grained content of the image or figure, including scenes, objects, relationships, and any text present.
Table 10: Prompt for instructing Multimodal Bard to generate a detailed caption for an input image.
F.5 MODEL HYPERPARAMETERS
The hyperparameters for the experiments in $3.2 are set their default values unless specified otherwise. Table 11 and Table 12 detail specific generation parameters for the various large language models (LLMs)) and large multimodal models (LMMs) evaluated, respectively.
Model
Generation Setup
Claude-2
model = c aude 2, temperature = O, max tokens = 1024
ChatGPT
model = gpt - 3 5 turbo, temperature 0, max tokens = 1024
GPT-4
model = cpt-4-06133 temperature = 0, wwx_mmaaeeeentette 1024
Table 11: Generating parameters for various LMMs.
F.6 HUMAN PERFORMANCE
We conducted study to evaluate human performance on the testmini subset of the MATHVISTA, utilizing Amazon Mechanical Turk (AMT). Each question from the testmini subset was assigned to five annotators, all of whom have history of completing more than 5,000 HIT tasks and boast an acceptance score higher than 0.99, to ensure the quality of the results. The study comprised five test questions and two qualification questions, which were to be answered within 20-minute timeframe. The qualification questions consisted of elementary math word problems requiring basic arithmetic operations (e.g., addition and subtraction). Only annotators who successfully answered the qualification questions were deemed eligible for the study, and their responses were included in the final analysis. Additionally, annotators were requested to provide information regarding their
34

---

Published as a conference paper at ICLR 2024
Model
Generation Setup
IDEFICS-9B-Instree
wel_new_tokeen 256, temperature 1.0
mPLUCGOO//LLaMA-TB
do do.sample= True, top-k 5, max length 512
num_beams = 1, temperature 1.0, max max_new new_eenens = 300, max max_length
minIPPT4LLaMA-2278
1000
ALaMA-Adapter-V--77
max gen.len = 256, temperature = 0.1, top.pt 0.75
LLaVAR
do sample True, temperature = 0.2, max new tokens 1024
do_sample = False, num num_heams = max_ = 256, min_length 1, top-p 0.9, reperition penalty = 1.0, temperature =1
InstructLLIP-iccum
LLaVA-LLaMA-2-13B
do_sample = True, temperature = 0.2, max.new.tokens = 1024
Chatbot URL: https: / /tt///add//////////////////aaa.t www.g...oool.eea.ooooooooooo google com, evaluation dates range
Multimodal Bard
from Sep 8, 2023 to Sep 10, 2023
Chatbot URL: https: : htsp:::::hhh/e /tthtttcccc//cccccccc openai. com, evaluation dates range
GPT-4V (Playground)
from Oct 7, 2023 to Oct 15, 2023
Table 12: Generating parameters for various LMMs.
highest level of educational attainment. We retained the results exclusively from annotators who had achieved high school diploma or higher, as 30.9% of the problems in MATH are of high-school level difficulty and 10.8% correspond to college-level curricula.
F.7 MULTIMODAL BARD ASSESSMENT TASK
screenshot of our AMT worker interface, utilized for the Multimodal Bard assessment task, is provided in Figure 32. The workers were compensated at rate of $18 per hour.
Groand
Model Peggciee
14
19
25
32
?
Figure 32: Screenshot of the Multimodal Bard assessment task interface.
35

---

Published as a conference paper at ICLR 2024
G MORE EXPERIMENTAL RESULTS
G.1 RESULTS ON THE TEST SET
Table 13 reports the accuracy scores of two heuristic baselines, leading augmented LLMs (CoT GPT-4, PoT GPT-4), and one leading LMM (LLaVA-LLaMA-2-13B) on the test subset. The minor differences between scores on the subset and the testmini subset, shown in Table suggest that testmini effectively mirrors the test subset, serving as valuable evaluation subset for model development, especially for those who have limited computing resources.
Model
Input
ALL
FQA
GPS
MWP
TQA
VQA
ALG
ARI
GEO
LOG
NUM
SC1
STA
Random chance
17.86
15.46
24.12
4.54
23.36
24.33
25.84
13.85
22.69
13.40
8.82
15.76
14.28
Frequent guess
23.48
20.97
27.18
16.27
26.06
28.87
28.29
20.86
25.71
11.86
19.61
20.45
20.08
2-shot CoT GPT-4
Q. Q Ta T
30.50
27.21
35.91
21.30
43.13
28.17
35.72
25.17
35.80
24.74
15.41
47.28
31.29
2-shol PoT GPT-4
31.74
27.58
37.35
23.87
43.00
30.27
33.15
27.93
37.48
22.68
15.83
44.47
31.87
LLaVA-LLaMA-2-13B
Q.
25.40
22.86
24.57
18.15 35.82 29.69 26.93 22.47 24.45 19.07 19.05 34.71 21.61
Table 13: Accuracy scores on the test subset of MATIVVISTA. Input: Q: question, 1: image, Le: image caption, 4: OCR texts detected from the image. ALL: overall accuracy. Task types: FQA: figure question answering, GPS: geometry problem solving, MWP: math word problem, TQA: text- book question answering, VQA: visual question answering. Mathematical reasoning types: ALG: algebraic reasoning. ARI: arithmetic reasoning. GEO: geometry reasoning, LOG: logical reasoning, NUM: numeric common sense, SCI: scientific reasoning, STA: statistical reasoning.
G.2 SCORES FOR MATH REASONING TYPES
The accuracy scores across seven mathematical reasoning categories are reported in Table with primary baselines highlighted in Figures and 33. GPT-4V outperforms other baseline models most mathematical reasoning categories, except for logical reasoning and numeric commonsense reasoning. Multimodal Bard achieves comparable performance with GPT-4V in geometry reasoning (47.8% vs. 51.0%) and algebraic reasoning (46.5% vs. 53.0%), highlighting enhanced abilities in comprehending geometry diagrams and performing algebraic calculations.
Random chance
LLavA
PoT GPT-4
GPT-4V
LLaMA-Adapter V2
CoT GPT-4
Muitimodal Bard
Human
60
Accuracy Score (%)
30
Algebraic
Arithmetic
Geometry
Logical
Numeric
Scientific
Statistical
Figure 33: Accuracy scores of baselines across mathematical reasoning types in MATH VISTA.
Among open-source LMMs (ranging from IDEFICS to LLaVA), LLaVA achieves the best overall accuracy MATUVVSSA and the highest fine-grained scores for problems in geometry reasoning, logical reasoning, and statistical reasoning. However, these scores still substantially lag behind GPT-4V and Multimodal Bard, indicating gap in the overall effectiveness of these open-source models compared to more advanced proprictary systems. Despite this, LLaMA-Adapter- V2, tied with LLaVA, outperforms GPT-4V by 2.7% in logical reasoning, and InstructBLLIP beats GPT-4V
36

---

Published as a conference paper at ICLR 2024
by 0.3% in numeric commonsense, suggesting that specific enhancements in open-source models can lead to superior performance certain niches. LLaVAR, being on par with Multimodal Bard, which is specifically designed to enhance capabilities in detecting OCR texts and symbols from various forms, including scientific domains, further illustrates the potential of targeted improvements in open-source LMMs to achieve competencies that rival counterparts in specialized areas.
exceed those of their proprietary
CoT GPT-4, augmented with OCR texts and Bard captions, performs well in scientific reasoning, achieving a gain of 26.2% over random chance, showcasing its superiority in domain-specifie knowl- edge. This performance suggests a significant trend (Shen et al., 2023; Lu et al., 2023a) where the integration of specialized functionalities, such as OCR text recognition and advanced captioning, into L.MSs enhances their applicability and aceuracy in specific domains. PoT GPT-4 outperforms Multimodal Bard in categories such arithmetic reasoning, logical reasoning, numeric common- sense reasoning, and statistical reasoning. This superior performance attributed its ability to generate high-quality codes for precise mathematical reasoning, illustrating the effectiveness of in- tegrating advanced coding capabilities into language models for complex problem-solving tasks.
G.3 SCORES VARIOUS VISUAL CONTEXTS
Figure 34 illustrates the accuracy scores of leading baselines on MATIIVISTA across diverse range of visual contexts. Remarkably, GPT-4V outperforms human performance visual contexts of function plots, geometry diagrams, scatter plots, tables, and other types, which aligns with its periority in terms of related mathematical reasoning types. Other foundation models trail behind humans in visual perception and reasoning across most visual context categories. Multimodal Bard demonstrates comparable performance to humans in questions with a visual context of geometry diagrams, showcasing its promising capabilities in recognizing geometric shapes and relationships. On the other hand, PoT GPT-4, augmented by Bard captions, achieves significant performance ad- vantage over other baselines, exhibiting strong abilities in discerning structural information in tables and generating symbolic codes for precise statistical reasoning.
Random
LLaVA
Pol GPT-4
GPT-4V
LLaMA-Adapter V2
CoT GPT-4
Multimodal Bard
Human
80
Accuracy Score (%)
60-
50
40
30
10
0
Abstract Scene
Bar Chart
Function Plat
Geometry Diagram Line Plat
Natural Image
Puzzle Test
Scatter Plot
Scientific Figure
Synthetic Scene Table
Other
Figure 34: Accuracy scores of leading baselines across various visual contexts in MATHVISTA.
G.4 SCORES ACROSS DIFFERENT GRADE LEVELS
Figure 35 displays the average accuracy scores across different grade levels (elementary school, high school, and college) for the leading foundation models, as well as random chance and human performance, Humans exhibit the highest performance on questions at the elementary school level (70.4%), while they fare the worst college-level questions (52.6%) within MATHVISTA Foun- dation model baselines exhibit varying performance behaviors: they achieve better accuracy scores on high school level questions compared to the other categories.
37

---

Published as a conference paper at ICLR 2024
In addressing elementary school problems. the performance gap between human performance and the best-performin model. GPT-4V, is notably the largest when compared to other grade levels. This gap could potentially be attributed to the limited availability of age-specific training data that accurately captures the unique learning styles (i.e., rich with abstract scenes) elementary school students. On the other hand, GPT-4V demonstrates Bard, the second-best performing model in this category. This improvement suggests that while GPT-4V still lags behind human performance, its ability tackle elementary-level problems in visually intensive settings has been significantly enhanced.
improvement of 20.9% over the Multimodal
For high school problems, GPT-4V, with a score 61.8%, outperforms human performance, which stands at 58.2%. Additionally, the second-best performing model, Multimodal Bard, with score of 50.3%, on par with human performance. This disparity might be attributed the training regimen of the models, which perhaps aligns well with the high school curriculum.
In the context of college curriculum, the performance of various baselines varies dramatically. GPT- 4V demonstrates performance comparable to that of humans. The GPT-4 model, when augmented with vision inputs (CoT GPT-4V), outperforms the Multimodal Bard. Among the best open-source Large Multimodal Models (LMMs) on MATHVISTA, LLaMA achieves only a negligible gain over random chance. This suggests that while advanced models like GPT-4V and CoT GPT-4V show promise in higher education settings, there remains significant room for improvement in the devel- opment of LMMs to effectively address the complex and diverse nature of college-level content.
Random chance
LLaVA
PoT GPT-4
GPT-4V
LLaM/-Adapter
Multimodal Bard
Accuracy
Elementary School
High School
College
Figure 35: Average accuracy scores across different grade levels for primary baselines.
G.5 ABLATION STUDY FOR LLMs
Table 36 presents an ablation study conducted on LLMs, examining their performance under varying visual information inputs.
N/A
OCR Text
Caption
Caption OCR Text
Accuracy Score
24
22
20
ChatGPT
GPT-4
ChatGPT
PoT
Figure 36: Average accuracy scores of LLM baselines under various visual inputs.
38

---

Published as a conference paper at ICLR 2024
G.6 LLMS WITH DIFFERENT SHOTS
We explored whether LLMs and Augmented LLMs can benefit from larger numbers of few-shot examples MATHVISTA, with results reported in Figure 37. In the question-only input setting (a), both Claude-2 and ChatGPT suffer from a performance drop, suggesting that they are more sensitive the bias in demonstrations, especially in the absence of visual inputs. There is marginal improvement of 1.4% when the shot number increases from 2 to 4 for GPT-4. A similar phenomenon observed when L.MMs are augmented with external OCR texts and image captions with CoT prompting (b); notably, there is a significant drop of 3.4% when the shot number increases from to for CoT Claude-2. With PoT prompting (c), LLMs like ChatGPT and GPT-4 can obtain gains of 3.4% and 1.4%, respectively, with the shot number increasing from to 4. Overall, while there might marginal improvements, larger numbers few-shot examples do not necessarily benefit the LLMs MATHVISTH In some settings, LLMs suffer from unstable performance drops. This further indicates that the quality of the augmented information plays a more important role for augmented L.MMs.
34
Claude-2
38
CoT Claude-2
PoT ChatGPT
ChatGPT GPT-4
CoT ChatGPT
PoT GPT-4
Accuracy Score 69
32
Accuracy Score (%)
CoT GPT.4
30
acy Score
36
28
34
32
30
24
22
20
28
24
Shot Number
Shat Number
Shot Number
(a) Q Only.
(b) + OCR + Caption.
(c) Q OCR Caption.
Figure 37: Performance comparison of LLM models across different shots.
G.7 LMMs WITH DIFFERENT SHOTS
We conducted an initial study the few-shot learning ability of the Large Multimodal Model (LMM), specifically IDEFICS (Laurençon et al., 2023), on MATUVISTA. As shown Figure 38, there is modest improvement with increased shot numbers, suggesting potential benefits of few-shot learning for LMMs on MATHVISTH
However, recent studies highlight the instability of LMMs in few-shot settings. For instance, significant accuracy drop was observed in models like BLII-2 (Li et al., 2023a) and InstructBLIP (Dai et al., 2023) when applying 4-shot in-context learning in common et al., 2023b). These variations may stem from the specific training techniques or the nature of few- shot examples used, impacting the in-context learning performance LMMs. Given the rapidly evolving landscape of LMMs, the consistent benefits of few-shot learning remain an open question.
reasoning tasks (Li
Accuracy Score (%)
24
20
Shot Number
Figure 38: Performance comparison of the LMM model in the few-shot setting.
39

---

Published as a conference paper at ICLR 2024
G.8
HALLUCINATIONS IN MODEL EXPLANATIONS
0L385
200
Question:
What is the highest amount this class measures?
Response from Bard: The highest amount the beaker the image measures is 600 ml.
The markings on the side of the beaker are in milliliters (ml). The largest marking is 600 ml, which is the maximum amount of liquid that the beaker can hold.
The other markings on the beaker are 300 ml, 200 ml, 100 ml, -400 ml. The negative markings indicate the amount of liquid that is below the zero mark.
Wrong output: 600
Ground truth: 400
Figure 39; An example illustrating hallucinated information not depicted in the image.
A
D
Question: As shown the figure, if CB 4.0, DB 7.0, and D is the midpoint of AC, then the length ACC O.
Choices: (A) 6cm (B) Tem (C) 4cm (D) 5cm
Response from Bard: Since the midpoint of AC, then AD DC = (AC/2).
We know that CBB = 4.0 and DB = 7.0, and that AD = DC. So we can write the equation:
AD DC 4.0 + 7.0
2AD = 11.0
AD 5.5
AC = 2AD = 11.0cm
Therefore, the length of AC is 11.0 em, which is answer choice (D).
Wrong output: (D) 5em
Ground truth: (A) 6em
Figure 40: An example illustrating hallucinated information not depicted in the question.
40

---

Published as a conference paper at ICLR 2024
G.9
MORE EXAMPLES FOR MULTIMODAL BARD
Question: As shown in the figure: In RIAABC, 2C= 90.0, =C= 8.0, AB = 10.0, then the value sin is equal to O)
Choices: (A) 1 (B)
1 (
Response from Bard:
In a right triangle, the sine of an angle is equal to the ratio of the opposite side to the hypotenuse. In this case, the opposite side is AC = and the hypotenuse is AB 10. Therefore, sin = 8/10 = 3/55.
Here the detailed solution:
Given:
AAA BC right triangle with right angle at C
ACC 8
AB 10
find: sin
sin = opposite/hypoteeuse = AC/AB 8/10 3/55
Wrong output: (A)
once
Ground truth: (B)
Figure 41: A failure example from Multimodal Bard: incorrect answer with wrong calculation.
Question: What is age gap hetween these two people in image? (unit: years)
Response from Bard:
Sorry, I can't 't help with images of people yet.
Wrong output: N/A
Ground truth:
Figure 42: A failure example Multimodal Bard: restricted in responding to images people.
41

---

Published as a conference paper at ICLR 2024
Accuracy of different algorithms
globe
Accuracy
Question:
What is the accuracy of the algorithm with lowest accuracy?
Response from Bard: The algorithm with the lowest accuracy the graph is heel. accuracy
The other algorithms are desert, pine, and globe. Their accuracies are 2, 4, and 6, respectively.
Wrong output:
Ground truth:
Figure 43: A failure example from Multimodal Bard: incorrect answer with hallucination caused by visual perception.
Most Dissatisfied With Country Direction
Dissatisfied 78%
Satisfied 20%
Question: What is the age gap between these people in image? (unit: years)
Response from Bard:
The dissatisfied people are 78%, which 1.2 times more than the satisfied people (20%). So the answer is 1.2.
Wrong output: 1.2
Ground truth: 3,9
Figure 44: failure example from Multimodal Bard: incorrect answer with correct detection but wrong calculation.
42

---

Published as a conference paper at ICLR 2024
Question: Is the number of cars in front of the tiny metal thing less than the number of large matte things in front of the cyan rubher road bike? Choice: (A) Yes (B) No
Response from Bard: The answer is (B), No.
There
in front of the tiny metal thing (the motorcycle). There large matte things in front the eyan rubber road bike (the van and bicycle).
So the number of large matte things in front of the cyan rubber road bike greater than the number of cars in front of the tiny metal thing.
Wrong output: (B) No
Ground truth: (A) Yes
Figure 45: A failure example from Multimodal Bard: incorrect answer with correct explanation.
10
12
Question: What is the missing number in the picture?
Choice: (A) (B) (C) 10 (D)
Response from Bard: The missing number in the picture is (B) 8.
The numbers in triangles sequence where each number is more than previous num- ber.
2
10
/12
So, the missing number is 8.
Correct output: (B)
Figure 46: A successful example from Multimodal Bard: correct answer with hallucination (20 is present in the image) and wrong reasoning.
43

---

Published as a conference paper at ICLR 2024
y4
Question: The magnitude of the acceleration vector 10 cm/ss. Use the figure estimate the normal components of a.
Response from Bard:
From the figure, we can see that the acceleration vector is making an angle of 30 with the radius vector. The normal component of the acceleration is given
10v3
cos = 10 cm/s² / cos 30
cm/s²
2
The normal component of the acceleration is approximately 7.07 cm/s".
Here is breakdown of the steps involved in finding the normal component of the acceleration:
1. Find the angle between the acceleration vector and the radius vector.
2. Use the cosine function to find the cosine of the angle.
3. Multiply the magnitude of the acceleration vector by the cosine of the angle to find the normal component
of the acceleration.
Wrong output: 7.07
Ground truth:
Figure 47: A failure example from Multimodal Bard: the question is incomplete (or requires implicit measuring ability), but the model hallucinates the angle (the actual angle is 25%). Here, a, 10cm/ s cos 25 9cmm/s2
44

---

Published as a conference paper at ICLR 2024
0
log,
-4
Question: The derivative of y 6 is that 8
Chocies: (A) larger than (B) equal to (C) smaller than
Response from Bard:
The correct answer is (C). The derivative of
6 is smaller than that at 8.
The graph of the function y logg x concave up curve. This means that the derivative of the function is increasing as a increases. So, the derivative the function at x = 6, which a smaller value x, will be smaller than the derivative of the function at 8, which is larger value of z.
Here is table of the values of the derivative different values of
Jr dy/dr
+
6 0.5
0.77
8
As you can see, the value of dy/ds do increases from 0.5 to 0.7 to 1 as ær increases from 6 to 7 to This confirms that the derivative of y at x = 6 is smaller than that at 8.
Wrong output: (C) smaller than
Ground truth: (A) larger than
Figure 48: A failure example from Multimodal Bard: incorrect reasoning and wrong calculations.
45

---

Published as a conference paper at ICLR 2024
10
fto
2x
31+1
10
Question:
The derivative of [xx 2 Chocies: (A) larger than (B) equal to (C) smaller than
that
Response from Bard: The correct answer is (C). The derivative of f(z) at 2 is smaller than that at 5.
The derivative of a function at a point gives slope that point. The steeper the line, the larger the derivative. In the graph, the line tangent to the graph of f(x) at 2 is less steep than the line tangent to the graph at x = 5. Therefore, the derivative of f(x) at 2 is smaller than that al x 5.
line tangent the function's graph at
In mathematical terms, the derivative of f(z) is given by the expression f'(x) 2x 3. Plugging in x and we get (2) = 2(2) and (5) = 2(5) Therefore, f'(2) < (5).
Wrong output: (C) smaller than
Ground truth: (B) equal to
Figure 49: A failure example from Multimodal Bard: arguing based on incorrect facts in the context of the image.
46

---

Published as a conference paper at ICLR 2024
G.10
COMPARISONS OF DIFFERENT MODELS
Question: Which number is missing?
Ground truth: 40
10
14
19
Solution:
1 1 1 = 5,5 =
7
(2==11))
10 4 14, 5
19 (5 4
32
25 + 7 32,32
+ 40 (8 (8= 7+1)
IDEFICS:
The answer 1.
mPLUG-Owl: There are 13 missing numbers in the image. The missing number can be 3.
miniGPT4: The number missing in the image is 4.
[LaMA-Adapter-V? V2: The missing number is 13.
LLaVAR: The number number 25 is missing.
InstructBLIP: N/A
LLaVA: The number missing in the sequence is 14.
Multimodal Bard:
The missing number in the sequence is 28.
The sequence progresses by adding consecutive numbers to each number. The first number 4, and we add to get 5. We then add to get 7, and so on. The missing number is therefore the next number after 32, which is 28.
Here the complete sequence:
4 4+1=
5+3=7 =7
7+
10 14
14+9
19+11=20 25
25 32
32 + 15 = 28
GPT-4V: 4, 10, 14, 19, 25, 32, 5,
Answer: The missing number is 40.
This sequence follows pattern where the difference hetween each numher increases by one. For example, the difference between 4 and 5 is 1, the difference between 5 and is 2, the difference between 7 and 10 is 3, and missing number 40.
So, the difference between 32 and the missing number should be 8, making the
Correct output: 40
Figure 50: Among all LMM baselines, only GPT-4V accurately predicts the correct answer to this logical reasoning question, demonstrating correct visual perception and textual reasoning. Multi- modal Bard generates well-organized reasoming steps; however, incorrect calculations are present.
47

---

Published as a conference paper at ICLR 2024
Question: Which number is missing?
5
Ground truth: 40
Solution:
1 1 5.5
10 + 4 14
(5
4 +
25
32
25 + 7 32, 32
+8
40
7+1
Augmented Caption from Bard:
The image shows set numbers in circles on a white background. The numbers are 4, 5, 10, 14, 19, 25, and 32. There is question mark in the last circle. The numbers are arranged in increasing or- der, with each number being more than the previous number. Therefore, the missing number is 19 25.
The image is simple math problem, and it is likely that it is being used as part of IQ test other kind of intelligence test. The image is also consistent with webpages that you mentioned, which are all related to IQ lests.
Augmented OCR Text from EasyOCR: ([300, 80),
(183, 281J. 10), ([285, 281J. 14), ([491, 279], 19), (181, 485), 25), ([285, 485). 32), ([491, 467]. 2)
CoT GPT-4:
The pattern in the numbers is that each number is more than the previous number plus the position of the number in the sequence. For example, the second number (5) 1 more than the first number (4) plus the position the number the sequence (2). The third number (7) more than the second number (5) plus the position
number the sequence (3).
Following this pattern, the missing number should be more than the last number (32) plus the position of the number in the sequence (9).
32 +
42
So, the missing number is 42.
PoT GPT-4:
5, 7, 10, 14, 19, 25, 321 diffs [numbers [1, 2, 3, 4, 5, 6, 7]
numbers
numbers for in range (len (numbers) -1)
missing_numbo - numbers [-1] diffs[-1]
print (missing_number.
Output: 39
Figure 51: The LLM baselines augmented with Bard image captions and OCR also fail on this question. CoT GPT-4 generates explanations with incorrect reasoning. PoT GPT-4 also fails to understand the correct pattern behind the sequence of numbers.
48

---

Published as a conference paper at ICLR 2024
Question: What fraction of the shape is blue?
Choices: (A) 3/11 (B) 8/11 (C) 6/11 (D) 3/5
Ground truth: (A) 3/11
IDEFICS: The answer is (B) 8/11.
mPLUG-Ow:: B: 8/11
miniGPT4: The fraction of the shape that is blue is (B) 8/11.
I.MMA-Adapter-V2. V2: The correct answer (C) 6/11.
LLaVAR: The fraction of the shape that blue is I/3,
InstructBLIP: N/A
LLaVA: The correct answer is (B) 8/11.
Multimodal Bard:
The correct answer is (C) 6/11.
The shape is divided fraction of the shape that blue is 6/11.
equal parts. blue part takes up these parts. Therefore, the
The other choices are incorrect. Option (A). 3/11, is the fraction of the shape that is white. Option (B), 8/11, is the fraction of the shape that is not blue. Option (D), 3/5, is not fraction of the shape.
GPT-4V: The correct answer (B) 8/11.
Figure 52: All LMM baselines fail to answer this statistical reasoning question, indicating the un- satisfactory abilities in understanding abstract scenes and performing statistical reasoning.
49

---

Published as a conference paper at ICLR 2024
Question: What fraction of shape is blue?
Choices: (A) 3/11 (B) 8/11 (C) 6/11 (D) 3/5
Ground truth: (A) 3/11
Augmented Caption from Bard:
The image shows a blue circle with a white stripe in the middle. The cirele is divided four parts, each different color. colors are blue, green, yellow, and red. There no text present in image.
The image could be used to represent variety of concepts, such as fractions, colors, the four elements. It could also be used decorative element symbol.
COT GPTT4:
The image description does not provide enough information to determine what fraction of the shape is blue. Please provide more detailed description of the image.
PoT GPT4:
def fraction_of Lfcloo__oof___ blue (description, chcices)
parts =
'luu' 1,
'gree''; : 1,
"yellow 1, 'red':
total parts commmaprt.vaaueee ()) blue_fraction parts('blue's / tetal_parts
for choice in choices:
fraction choi cec.c...... splitp ('/)
if int (fraction [0]) int (fraction(1)) blue_fraction
return choice
return ""
choices 3'3/11' 18/11', '6/11', "3/5"1
description "The image shows blue circle with white stripe in the middle. The circle is divided into four parts, each with a different coler. The colors are blue, green, yellow, and red. There is no text present in the image."
print (fraction (fraction_o__bal blue (description choices))
Output: u*
Figure 53: Two LLM baselines also fail to answer this question. The incorrect facts in the image captions generated by Multimodal Bard further verify its weakness understanding the abstract concepts in this figure.
50

---

Published as a conference paper at ICLR 2024
Question: you add the two visible numbers, on the jerseys, what is the total sum?
Ground truth: 3
Solution: 2 =
IDEFICS: The answer 10.
mPLUG-Owl: The total sum the two numbers on the players' jerseys is 24.
miniGPT4: The numbers on the jerseys are and 15. If you add them together. the total sum is 22.
LLaMA-Adapter-V2: The total sum of the two visible numbers the jerseys 12.
LLaVAR: The two visible numbers on the jerseys add up 21.
InstructBLIP: 21
LLaVA: The two visible numbers on the jerseys are 2 and 1. If we add them together, the total sum is Correct output: 3
Multomodal Bard: Sorry, can't 'tnnt help with images of people yet.
GPT-4V: The total sum the two visible numbers on the jerseys + 8).
Augmented OCR Text from EasyOCR: ([110, 154), 21)
CoT GPT-4:
The image only shows one visible number on the jerseys, which 21.
Thus, the total sum is 21.
Augmented OCR Text from EasyOCR: (1110, 154], 21)
PoT GPT-4:
visible numbers [21]
total sum ) print (total_sum)
Execution output: 21
Figure 54: Among all LMM and LLM baselines, only LLaVA generates the correct answer this instance. Models like LLaVAR, Augmented CoT GPT-4, and Augmented PoT GPT-4 identify the correct number(s) in the image but fail to resolve the ambiguity within the contexts of the input question and image.
51

---

Published as a conference paper at ICLR 2024
Question: How much money does Luca need to buy sour apple candy and
sour apple candy
S0.06
butterscotch candy? (Unit: $)
piece of
S0.07
gummy worm
S0.09
Ground truth: 0.13
lemon drop
S0.05
Solution: Add the price of a sour apple candy and the price of a but-
piece of licorice
S0.07
terscotch candy: $0.06 $0.07 = $0.13. Luca needs $0.13.
butterscotch candy
$0.07
IDEFICS:
The answer 1.50.
mPLUG-Owl: The price of each candy not provided, so it not possible calculate the total price for buying both candy items.
miniGPT4: The answer to the question is:
Luca needs $0.75 to buy sour apple candy a butterscotch candy.
Explanation:
The price of the sour apple candy is $0.50.
The price of the butterscotch candy is $0.25.
Therefore, Luca needs total of $0.75 buy both candies.
IlMMA-Adapter-V22 V2: Luca needs $1.23 to buy sour apple candy and hutterscotch candy.
LLaVAR: Luca needs to earn $1.23 to be able buy apple candy and butterscotch candy.
InstructBLIP: 1,23
LLaVA: Luca needs $1.45 buy apple candy and butterscotch candy.
Multimodal Bard:
The price of sour apple candy is $0.06 and the price of a butterscotch candy is $0.07. So, Luca needs $0.06 + $0.07 $0.13.
In two decimal places, amount 0.13. Correct output: 0.13
GPT-4V:
The total cost for apple candy and butterscotch candy is $0.13. Luca would need at least $0.13 purchase both items.
Correct output: 0.13
Figure 55: Among all LMM baselines, only Multimodal Bard and GPT-4V accurately detect the text information in the image, and perform the correct calculation deduce the right answer for the arithmetic question within the visual context of table.
52

---

Published as a conference paper at ICLR 2024
Question: How much money does Luca need to buy sour apple candy and
sour apple candy
S0.06
butterscotch candy? (Unit: $)
piece of gum
S0.07
gummy worm
S0.09
Ground truth: 0.13
lemon drop
S0.05
Solution: Add the price of a sour apple candy and the price of a but-
piece of licorice
S0.07
terscotch candy: $0.06 $0.07 $0.13. Luca needs $0.13.
butterscotch candy
S0.07
Augmented Caption from Bard:
The image shows table with the prices of different types of candy. The table is on pink background. The candy is all different shapes and sizes, and the prices are all different. The candy is all different colors, and the prices different.
The following text is present in the image:
Sour apple candy $0.06
Piece of gum $0.07
* Gummy worm $0.09
Lemon drop $0.05
Piece of licorice $0.07
Butterscotch candy $0.07
Augmented OCR Text from EasyOCR:
(19, 71, sour apple candy), ([195, 5), S0.06),
([7, 391, piece of gum), ([195, 37], $0.07).
(19, 73), gummy worm), ([195, 69], $0.09).
([7, 105], lemon drop). ([195, 101], $0.05),
(16, 132], piece of licorice), ([195, 133], $0.07),
(19, 167], butterscotch candy), ([195, 165], $0.07)
CoT GPT-4: Find the total cost of sour apple candy and butterscotch candy.
50.06 + $0.07 $0.13
Luca needs $0.13.
Correct output: 0.13
PoT GPT-4:
ppur_apple_cand__pii 0.06
caaettehhhhaa____eeeee____ 0.07
cotal_prics = sour sorr_eple_candy_prrie + butterscot.ch_andyy.eee print (total_price)
Execution output: 0.13
Correct output: 0.13
Figure 56: Using the correct image captions and OCR text augmented inputs, both CoT GPT-4 and PoT GPT-4 predict the correct answer.
53

---

Published as a conference paper at ICLR 2024
H A COMPARATIVE STUDY OF GPT-4V, BARD, AND OTHER MODELS
GPT-4 with vision (GPT-4V) is the multimodal version of GPT-4 that is instructed to understand multiple modalities such as texts and images. Due its remarkable improvements over other Al models ($3.3 and $3.4), we have conducted a comprehensive evaluation to understand its capa- bilities, strengths, and areas for improvement. Our findings only validate GPT-4V's various problem-sol skills but also shed light on developing general general-purpose multimodal Al agents.
H.1 GPT-4V PLAYGROUND FOR MANUAL EVALUATION
Given that GPT-4V does not offer API access, we have performed manual evaluations using the playground platform' For fair comparison, used the same input queries those for all the other LMMs and recorded the responses a single round chat without additional feedback (Figure 57).
peanut butter cookie dough
per
double chocolate cookie dough
$3 per lb
chocolate chip cookie dough
per
oatmeal raisin cookie dough
$7 per lb
gingersnap cookie dough
per lb
snickerdoodle cookie dough
$8 per lb
Hint: Please answer the question requiring floating-point number with one decimal place and provide the final value, 1.2, 1.3, 1.4, the end.
Question: Linda sho spand? (Unit: $)
buy 0.9 pounds of double chocolate cookie clough. much will
To find much Linda spend 0.9 pounds of double chocoliaee cookie dough:
Price per pound $3
Amount she wants buy pounds
Total cost 0.9 pounds $3/pound $2.7
Linda will spend $27
Figure 57: The GPT-4V playground for manual evaluation.
Phtsps://chat..peen ccmm
54

---

Published as a conference paper at ICLR 2024
H.2 LEADERBOARD SCORES
The leaderboard in Figure 58 highlights GPT-4V's substantial advancements over the current LLM and LMM baselines. Notably, there is 15.1% % improvement the second-best performing Mul- timodal Bard model. However, significant gap of 10.4% still exists between GPT--4V and human performance, indicating plenty of room for further improvement by developing new LMMs and tool-augmented LLMs.
Accuracy on the testmini set Vista (1,000 examples)
60.3
GPT-4V (Playground)
49.9
Multimodal Bard
34.8
PoT GPT-4 (Caption+OCR)
33.9
GPT-4 (Caption+OCR)
33.2
CoT ChatGPT (Caption+DCR)
33.2
CoT Claude-2 (Caption+OCR)
33.2
PoT ChatGPT (Caption+OCR)
26.8
LLavA (LLaMA-2-13B)
26.1
InstructBUIP (Vicuna-78)
25.3
LLaVAR
25.2
LLaMA-Adapter-2 (7B)
23.9
miniGPT4 (LLaMA-2-78)
23.1
MPLUG-OWl (BLaMA-7B)
22.2
IDEFICS (9B-Instruct)
19.8
Random chance
17.9
0%
10%
20%
30%
40%
50%
60%
Figure 58: Accuracy scores of primary baselines on the testmini subset (1,000 examples) of MATH- VISTA. Both CoT GPT-4 and PoT GPT-4 are augmented with Bard captions and OCR text.
55

---

Published as a conference paper at ICLR 2024
H.3 ABILITIES IN MATHEMATICAL REASONING
This section compares the mathematical reasoning ability of GPT-4V with that of other LLMs on MATHVISTA, including LLaMA-AAaptee--V--2 -V277 (LLaMA-Adapter-2) for simplification), LLaVA- LLaMA-2-13B (LLaVA for simplification), and Multimodal Bard.
H.3.1 ALGEBRAIC REASONING
Algebraic reasoning problems on MATH VISTA require understanding the function plots from figures and inferring their properties. As shown in Figure GPT-4V demonstrates outstanding capabilities in algebraic reasoning, surpassing all competing models and even humans. For instance, GPT-4V accurately identifies the function plot by equation and subsequently infers its correct properties (Figure 59). However, both GPT-4V and the other LLMs face challenges in comprehending low- resolution figures (Figure 60) and those that depict multiple functions (Figure 61).
ID: 926
((x)
(X 3)
Question: What is the size of the shaded area under the curve? Round the answer to 2 decimal places.
3
Ground truth: 7.07
LLaMA-Addpteer- V2: The size of the shaded area under the curve 0.00.
LLaVA: The size of the shaded arca under the curve approximately 1.34.
Multimodal Bard: The given curve is circle with center (3, 0) and radius 3. The shaded area quarter of the cirele's 's area.
Area of circle .ग
3.14 3
28. 28.26.
28.26
Area of the shaded region
7.065.
Rounding to two decimal places, the area is 7.07.
Correct output: 7.07
GPT-4V:
To find the size of the shaded area under the curve f(x) / (x 3)3, need integrate the function over the given interval.
The given function represents the upper half of circle with radius of and centered at the point (3,0). The shaded region corresponds to the area of quarter circle with radius of 3. The formula for the of a circle is Tr². For quarter of the circle, the area is xr².
Given: r 3
1
Area
(3?2)
4
Area (9)
9
Area 4"
Area 7.07 (rounded 2 decimal places)
So, the size of the shaded area under the curve approximately 7.07 square units. Correct output: 7.07
Figure 59: Both Multimodal Bard and GPT-4V accurately determine the function depicted in the image and compute the answer to the specified decimal places.
56

---

Published as a conference paper at ICLR 2024
ID: 346
Question: What is the y-intereept of this function?
Ground truth:
LLaMA-Addapter-Vz The y-intercept of this function is O.
LLaVA: The y-intercept function is 2.
Multimodal Bard:
The y-intercept of the function is the point where the graph crosses the y-axis. Since the y-axis is also the line 0, the x-coordinate of the y-inlercept will always be 0. In the image, the graph crosses the y-axis at point (0,3), 3). Therefore, y-intercept of function 3.
Here are the steps on how to find the y-intercept of a function:
1. Set the z-coordinate O. 2. Solve for the y-coordinate.
In this case, we set x = 0 and get = 3. Therefore, the y-intercept is
GPT-4V:
The y-intercept of function is the point where the graph of the function crosses the y-axis.
From the provided graph, the curve intersects the y-axis at y = 4.
Thus, the y-intercept is 4.
Figure 60: The plot represents function that intersects the y-axis at y However, large multi- modal models, including GPT-4V, struggle to grasp the algebraic relationship depicted in this plot.
57

---

Published as a conference paper at ICLR 2024
ID: 963
Question: For what does f reach its local maximum?
Ground truth:
22
LLaMA-Adapter-Y
The local maximum of function ((x) occurs at ir
LLaVA: The function f reaches its local maximum x = 2.
Multimodal Bard:
The local maximum the function is point where graph function changes from increasing decreasing. the given graph, this point is at 3. So the answer 3.
Here is more detailed explanation:
local maximum point of function point on the graph of function where the function value higher than all the function values a small neighborhood the point.
In the given graph, the function value is increasing for all values of æ less than 3. However, for values of greater than the function value starts decrease. This means that the function value x higher all the function values in the interval (3. 4).
Therefore, the local maximum point of the function is at = 3.
Correct output: 3
GPT-4V:
The function f(z) reaches its local maximum where its first derivative, f((x), changes from positive to negative.
Looking the graph, f2( crosses the æ-axis (becoming at
So, the answer is
Figure 61: The graph depicts two functions, f(x) and f'(x). While Multimodal Bard produces the correct answer, provides inaccurate information about the properties f(x). In contrast, GPT-4V confuses f(x) with (z(
58

---

Published as a conference paper at ICLR 2024
H.3.2 ARITHMETIC REASONING
Arithmetic reasoning problems in MATIVVISTA require accurate fundamental operations in con- junction with understanding diverse visual contexts. As illustrated in Figure 1, GPT-4V exhibits significant improvement in arithmetic reasoning compared to existing models. For instance, some LLMs struggle with basic arithmetic tasks, such as determining the difference between two values bar chart (Figure 62) or computing the probability based on simple statistical data (Figure 63).
ID:
Share popalation naing at loast basic drinking water sourec, 2000 Areseel pasil my mus si aravio, aaoeeiaeea sour 3immmaae
Question: What is the difference of largest and smallest bar?
Ground truth: 47.6
Hungan
Centodt
2244
0%%
20%
40%
90%
80%
1100
Serer swor siCe Meniaing
then feninatoon
DC
LLaMA-Adapter-VZ. V2: The difference the largest and smallest bar 0.1.
LLaVA: The difference between the largest and smallest bar in the graph is 1.4.
Multimodal Bard:
The largest bar image is Iceland with 100%, and the smallest bar Cambodia with 52.4%. So the difference between the largest and smallest bar is 100 52.4 47.6%.
However, the question asks for the answer in one decimal place. Rounding 47.6% to decimal place gives 47.7%.
Therefore, the difference between the largest and smallest bar is 47.7%.
GPT-4V:
From the graph:
Largest bar (Iceland) = 100%
Smallest bar (Cambodia) 52.4%
Difference 100% 52.4% 47.6%
Correct output: 47.6
The problem involves performing a straightforward arithmetical operation after identi- fying two numbers from the bar chart. However, LLaMAA-Aapterr-V V2 and LLaVA are unable to interpret the figure correctly, while Multimodal Bard fails to round 47.6% to one decimal place.
Figure 62:
59

---

Published as a conference paper at ICLR 2024
ID: 718
Velvet
Cotton
Zigzags
Question: Jeffrey is the proud owner of an eclectic bow tie collection. He keeps track of how many how ties he has, and organizes them by pattern and material. What is the probability that randomly selected bow tie designed with swirls and is made of velvet? Simplity any fractions.
Swirts
3
5
Ground truth: 0.21
LLaMA-Adapter-V2
The probability that randomly selected bow designed with swirls and is made of velvet 0.00.
LLaVA:
To determine the probability of randomly selected bow tie being designed with swirls and made of velvet, need to divide the total number of bow ties with the specitic characteristics (swirls and velvet) by the total number of bow ties in Jeffrey's collection.
In the image, there are three bow ties with swirls and two bow ties made of velvet. The total number of bow ties in the collection is 10.
So. the probability of randomly selected bow tie being designed with swirls and made of velvet is:
(3 bow tics with swirls and velvet) (10 total bow tics) = 3/10 = 0.3 or 30%.
Multimodal Bard:
The probability that randomly selected bow tic is designed with swirls and is made of velvet is:
(Number of swirls velvet how ties)/(Total number of bow ties)
We can find the number of swirls and velvet bow ties by adding the values in the "Swirls" and "Velvet" columns. This gives 3 5 = bow ties.
We can find the total number of bow ties by adding values all three columns. This gives + 3++ 1 14 bow ties.
Therefore, the probability that randomly selected bow tie is designed with swirls and is made of velvet is:
(8)/(14) 0.57 54.14%
To two decimal places, the probability is 0.,774, or 57.14%.
GPT-4V:
randomly selected tie designed with swirls made velvet, we first need to find the total number bow ties and then determine how many of these fit the criteria.
find the probability
Total number of bow ties:
Zigzags (Velvet) 5
Zigzags (Cotton) 1
Swirls (Velvet) 3
Swirls (Cotton)
Total 5 + - 14
Number bow ties that are designed with swirls and are made of velvet=
Probability Number of desired outcomes Total number of possible outcomes Probability 3/14
Now, converting the fraction decimal (rounded to two decimal places): 3 : 0.21
Correct output: 0.21
The problem requires arithmetic reasoning combined with an understanding of a basic statistical table and fundamental knowledge of probability. However, only GPT-4V predicts the correct answer.
Figure 63:
60

---

Published as a conference paper at ICLR 2024
H.3.3 GEOMETRY REASONING
In geometry reasoning, the performance of GPT-4V is comparable to that of humans on MATII VISTA, as demonstrated in Figure 1. Figure 64 and Figure 65, respectively, present two geometry reasoning problems: one an elementary level and the other a college level. For both problems, GPT-4V produces the correct answers accompanied by detailed explanations.
3
ID: 606
Question: What is perimeter of the rectangle?
2 cm
2 cm
Ground truth:
3 cm
LLaMA-Adapter-Y The perimeter of the rectangle 10 units.
Correct output:
10
LLaVA: The perimeter of the rectangle is 10 cm.
Correct output: 10
Multimodal Bard:
see the image. The rectangle has width 3 cm and length of 2 cm. So, the perimeter of the rectangle is + 2 8 cm.
GPT-4V:
The perimeter of a rectangle calculated by adding up all its sides.
Using the provided dimensions:
Perimeter 2 x (length width)
Perimeter 2 x (3 cm + 2 cm)
Perimeter 2 x 5 cm
Perimeter 10 cm
The perimeter the rectangle 10 em. Correct output: 10
Figure 64: This problem poses elementary geometry question, calculating the perimeter of rectangle. While LLaMA-Adapter-V and LLaVA produce the correct answer, they do not offer explanations to confirm the correctness of their reasoning steps. Conversely, Multimodal Bard com- mits calculation error when summing up the lengths of the four sides.
61

---

Published as a conference paper at ICLR 2024
W
ID: 640
Question: Find WN if AIWN is equilateral and WN 5
N Choices: (A) π A (C) 5 (D)
Ground truth: (B)
LLMMA-Adapter-N V2: The correct answer (D) 10m.
LLaVA:
Given that AIWN is equilateral, we can use the cosine rule to find the length of the side opposite the angle ZWN.
ZWN
where and are the lengths of the adjacent sides of the equilateral triangle.
5
5
Substituting the values, get:
5 =
cos LWN
So. length of side opposite the angle ZWN is equal to length of the adjacent sides, which Therefore, the correct answer is: (D) 10
Multimodal Bard: The correct answer is (B).
Since AIWN N is equilateral, WIN- WN1 60%. Therefore, WN = 60
Correct output: (B)
GPT-4V:
For an equilateral triangle, each interior angle 60%.
The are corresponding to central angle in a circle is given by: Are length = central angle in radians x radius
Given that WI IN NW (since A/W N is equilateral), the radius of the cirele is 5,
Using formula above: WN 60%
T
x
x x
5m 3
The correct option is: (B) 3..
Correct output: (B) 3
Figure 65: This is a high school-level geometry problem requiring an understanding of geometric symbols and relations, the application of the correct theorem, and accurate calculations. While Multimodal Bard generates the correct answer, it applies the wrong theorem. GPT-4V outperforms other baselines, demonstrating its remarkable ability to reason geometrically.
62

---

Published as a conference paper at ICLR 2024
H.3.4 LOGICAL REASONING
Logical reasoning problems represent different type of question in MATIIVISTA. Solving these problems requires abstract thinking to deduce the underlying patterns of numbers or shapes from figures. Current foundation models struggle effectively tackle logical reasoning problems: GPT- 4V achieves only 21.6% accuracy in logical reasoning, which is modest improvement of 8. over random chance, as shown in Table 2. The challenges that logical reasoning problems present to current LMMs are further highlighted in Figures 66, 67, and 68.
Brain Teaser for IQ Test
ID: 98
Question: Find the value the square in figure.
8
Ground truth:
LLMMA-Adapter-VZ. The value of the square the figure is 16.
LLaVA: The square in the figure has value of 1.
Multimodal Bard:
The value of the square in the figure 7.
Here's 's how to solve it:
1. 8 13
2. 13 +
3. 20/2 =
The answer is the final number, which is 10.
GPT-4V:
From the given figure:
Circle + Square =
Circle = - Square (8)
Triangle + Triangle 8
Triangle 4 ... (ii)
Circle Triangle
Using (ii). Circle 4
Circle = (i01)
Using (i) and (iii),
Square
Square = 2.
Correct output: 2
Figure 66:
The question assesses abstract thinking through a puzzle test problem. Only GPT-4V
grasps the abstract concepts and employs step-by-ster reasoning to deduce the final answer.
63

---

Published as a conference paper at ICLR 2024
ID: 778
?
Question: Can you find the missing shape in this picture puzzle?
Choices: (A) (B) (D)
Ground truth: (A) A
LLMMA-Adapter-VP V2: The missing shape in this picture puzzle is hexagon.
LLaVA: The missing shape in the picture puzzle square.
Multimodal Bard:
The missing shape in the picture puzzle is (D).
The shapes in puzzle are all regular polygons, and the number sides increases by as you move from left to right. The first shape a pentagon, the second shape is hexagon, the third shape is an octagon, and the fourth shape is decagon. The fifth shape should be dodecagon.
The choice A is pentagon, which is already the puzzle. The answer choice B is heptagon, which has 7 sides. The answer choice dodecagon, which has 12 sides.
nonagon, which has sides. The answer choice D
Therefore, the missing shape is the dodecagon.
GPT-4V:
The pattern seems to involve alternating filled and unfilled shapes.
Following the pattern:
'The first shape is filled hexagon.
The second shape is unfilled hexagon.
Thus, the missing shape should a filled trapezoid.
The correct option is: (D) D.
Figure 67: All baselines struggle discern the shapes presented in the problem and the options, and are unable to identify the correct pattern.
64

---

Published as a conference paper at ICLR 2024
ID: 750
12
10
Question: Which number is missing?
2
Ground truth:
5
Solution: 5 x 2 = 10, 6 x 2 10, x 2 = 18
LLMMA-Adapter-VV V2: The missing number is 4.
LLaVA: The number missing in the image is 3.
Multimodal Bard:
The missing number the puzzle is
The numbers in the puzzle are arranged in circle, and each number is surrounded by the sum of the two
numbers next it. For example, the number 12 is surrounded by 9 and 10, because 9 10 19.
The missing number must surrounded by and 5, and only number that satisfies this condition Therefore, the missing number 7.
GPT-4V:
From the given image, the number in the center, 2, the result of the difference between the number directly opposite in each pair:
10 2 9 5 = 1
6-?=2
Given that the difference between and the missing number is 2, the missing number is: 6 6-2 2-
So, the missing number is
Figure 68: All baselines fail to deduce the pattern inherent the logical reasoning problem.
65

---

Published as a conference paper at ICLR 2024
H.3.5 NUMERIC COMMONSENSE REASONING
Problems involving numeric commonsense reasoning on MATIIVISTA require commonsense knowl- edge about daily objects and celebrities to answer visual questions. However, these problems present significant challenges to existing foundation models, including GPT-4V, depicted in Figure 1. For instance, Multimodal Bard struggles to understand the optical illusion in an image (Figure 69) and infer the age gap between two celebrities from another image (Figure 70), Figure 71 poses question about the maximum volume beaker can measure. However, GPT-4V lacks commonsense knowledge regarding the use of beaker, resulting in an incorrect prediction.
ID: 693
Question: Is the orange larger than the car?
Choices: (A) Yes (B) No
Ground truth: (B) No
I.A.AAAdaaptr-- V2: No, the orange is not larger than the car.
Correct output: (B) No
LLaVA: The orange larger than the car.
Multimodal Bard:
The answer is (B), No. The orange is not larger than the car. The car is elearly much larger than the orange. Even if is large orange, is still much smaller than the car. The orange is probably about the size of basketball, while the car is least the size of small SUV.
I can't t see the orange in the image you sent, but can still answer the question based on the information you provided. The question is asking if the orange is larger than the car in general, not in the specific image. So. even if the orange in the image is larger than the car in the image, the answer would still be no, because oranges are not generally larger than cars. Correct output: (B) No
GPT-4V:
The orange is not larger than the car; it's just closer to the camera, creating optical illusion. Correct output: (B) No
The question pertains to everyday common sense and the identification of ob- jects speecificci an orange and car-in the image. GPT-4V produces the correct answer by discerning an optical illusion present in the image.
Figure 69:
66

---

Published as a conference paper at ICLR 2024
*
ID: 29
Question: What is the age gap between these two peo- ple in image? (unit: years)
Ground truth: 4
LLaMA-Adaatee--2: The age gap between these people in the image is 10 years.
L.a.V:: The age gap between the two people in image 20 years.
Multimodal Bard:
Based on the image you sent, the age gap between the king and queen is 3 years.
The king Richard III England, 1456. So, Richard was years older than Anne when they married in 1472.
was born 1452. The queen Anne Neville, was born
The stained glass window is likely depicting the couple after their marriage, so the gap between them would still be 3 years.
GPT-4V:
The two figures depicted in the stained glass are King Richard III and Anne Neville, Richard III was born October 2, 1452, and Anne Neville was born on June 11, 1456.
Calculating the age gap:
1456
So, the age gap between Richard III and Anne Neville is 4 years. Correct output: 4
Figure 70: In this instance, only GPT-4V correctly recognizes the celebrities along with their profiles to calculate the age gap.
67

---

Published as a conference paper at ICLR 2024
ID: 19
Question: What is the highest amount this glass measures?
400
ALTIA
Ground truth: 400
CROSILICATE GLASS
600ml
2000
100
LLaMA-Adapter-V2. The highest amount this glass measures 400 ml.
Correct output: 400
LLaVA: The highest amount this glass measures is 3000 ml.
Multimodal Bard:
The highest amount the beaker the image measures is 600 ml.
The markings the side of the beaker are in milliliters (ml). The largest marking is 600 ml, which is the maximum amount of liquid that the beaker can hold.
The other markings on the beaker are 300 ml, 200 ml, 100 ml, and -400 ml. The negative markings indicate the amount of liquid that is below the zero mark.
GPT-4V:
The highest amount this glass measures is 600 ml.
Figure 71: Most AI models without tool augmentation fall short in this example. A model not only needs to recognize the text and numbers accurately but also possess numerical common sense to determine the highest amount that the beaker can measure.
68

---

Published as a conference paper at ICLR 2024
H.3.6 SCIENTIFIC REASONING
Scientific reasoning represents distinct mathematical reasoning ability within our MATHVISTA. To lackle problems in this area, a model must not only accurately interpret domain-specific information from figures, but also possess the necessary in-domain knowledge reason rigorously on scientific topics. Figure shows that GPT-4V substantially outperforms the other foundation models. This superiority further illustrated by the examples in Figures 72 and 73. However, the failure of GPT-4V, as shown in Figure 74, indicates that there is considerable room for improvement.
ID: 807
acuminate
oblique
Question: Which of the following leaf shapes would have the least amount of wind resistance and water loss?
acute
sagittate
Choices:
rounded
hastate
(A) Truncate Acuminate
(C) Rounded (D) Sagittate
cordate
truncate
Ground truth: (B) Acuminate
LaMMA-Adaterr-V: The correct answer (D) Sagittate..
The leaf shape with the least amount of wind resistance and water loss would be the truncate leaf. This is because truncate leaves have more streamlined shape, which reduces air resistance and minimizes water loss due evaporation.
LLaVA:
Multimodal Bard:
The answer is (C). Rounded leaves have the least amount of wind resistance and water loss. This is because they have smooth, curved surface that does not catch the wind as much other lcaf shapes. Rounded leaves also have smaller surface area than other leaf shapes, which means that they losc less water through evaporation.
The other Icaf shapes in the choices are not as aerodynamic as rounded leaves. Truncate leaves have flat end, which can catch the wind. Acuminate leaves have long, pointed tip, which can also catch the wind. Sagittate leaves have long, arrow-shaped base, which also catch the wind.
GPT-4V:
The leaf shape that would have the least amount of wind resistance and water loss would be the one that is most slender and pointed. This is because slender and pointed shape would allow wind to flow around it with the least resistance, and narrow surface would lead to less water loss through transpiration.
Among the options, "acuminate" leaf shape the most slender and pointed.
Correct output: (B) Acuminate
Figure 72:
The question requires grasping biology concepts depicted in the image. LLaMA- Adapter- V2, LLaVA, and Multimodal Bard demonstrate a deficiency in domain-specific knowledge and struggle interpret the illustration.
69

---

Published as a conference paper at ICLR 2024
ID: 278
Table View interpolation results (PSNR) for NyDifFRIc reconstructions the NeRF synthetic dataset, using either DMTEE FLEXICUBES for topelogy step. The image metric scores are arithmetic means over all test images. We also include Chamfer dislanices (CD) compuled on visible trian- gles (the triangles visible least test using, 2.5
Question: Among the following objects. which one has the best PSNR score?
Lower scures indicate beiter geumetric fidelity.
Choices: Lego (B)Mats (C) Mic (D)Ship
PSNR (dB)
Chair
Druma
Ficus
Hotdog
Lego
Mats
Mic
Ship
Ground truth: (C) Mic
DMIEr
31.8
24.6
30.9
33.2
29.0
27.0
30.7
26.0
PurxeCunrss
31.8
24.7
30.9
33.4
28.8
26.7
30.8
25.9
CD (10 1
Chair
Drums
Ficus
Hotdog
Lego
Mats
Mic
Ship
DMTET
4.51
3.98
0.30
2.67
2.41
0.41
1.20
55.8
FLEXiCURES
0.45
2.27
0.37
1.44
1.60
0.53
1.51
10.5
HaMMA-Adapter-V.
The correct option letter for the best PSNR is "C," which the PSNR for the mic.
LLaVA: The correct answer (D) Ship.
Multimodal Bard:
The answer is (D) Ship.
The PSNR score
ship is 33.4 cB, which higher than PSNR scores for other objects. PSNR score measures the amount distortion between the original image and the reconstructed image. A higher PSNR score indicates less distortion and better quality reconstruction.
The other objects have the following PSNR scores:
* Lego: 29.0 dB
* Mats: 27.0 dB
Mic: 30.7 dB
Therefore, the ship has the best PSNR score among the given objects.
GPT-4V:
Based on the table for PSNR (dB) scores:
Lego:
DMTe:: 29.0
FLeXiCubes: 28.8
Mats: DMTet: 27.0 FLeXiCubes: 26.7
Mic: DMTet: 30.7 FLeXiCubes: 30.8
Ship:
DMTe:: 26.0
FLeXiCubes: 25.9
Among the choices, "Mic" has the highest PSNR scores with both DMTet and FlexiCubes, so the answer is (C) Mic.
Correct output: (C) Mic
The problem necessitates a unified comprehension of the table and its corresponding caption within academic domains. Exclusively, GPT-4V accurately recognizes the OCR text from related columns and deduces the correct answer.
Figure 73:
70

---

Published as a conference paper at ICLR 2024
ID: 725
Frame moves past frame while both observe P.
Question: Figure, suppose that Barbara's velocity rela- tive Alex is constant UBa 52 km/h h and car P is moving the negative direction of the x axis. If Alex measures a constant UPA = -88 km/h for car P, what velacity erB will Barbara measure?
Frame
Frame B
P
Ground truth: -130
A4
Xpg
*pa Kpg
XBA
LLaMA-Adapter- V2: Barbara will measure velocity Up -52 km/h relative to
LLaVA:
If Alex measures constant - 78 km/h for car P. Barbara will measure a velocity of upa 26 km/ relative car P.
Multimodal Bard:
The velocity of car P relative to Barbara is given by:
UPB UPA UBA =78 km/h 52 km/h 1.30 km/h
Therefore, Barbara will measure a velocity -130 km/h for car P.
Here is the explanation:
The velocity of P relative Alex, vr is measured by Alex, who is stationary in frame A. velocity Barbara relative to Alex, Uga, measured by Alex, who also stationary frame The velocity of car / relative to Barbara, UPB, measured by Barbara, who is moving in frame B with a velocity of UBA-
According the relative velocity addition rule, the velocity of J' relative Barbara is equal to the velocity of car relative Alex minus the velocity of Barbara relative Alex.
In this case, the velocity of car P relative to Alex is -78 km / and the velocity of Barbara relative to Alex is 52 km/h. Therefore, the velocity of car P relative to Barbara is - -130 km/h.
Correct output: -130
GPT-4V:
Given:
овΛ = 52 km/h /h (velocity of Barbara relative to Alex)
-78 km/l / (velocity of car P as measured by Alex)
We need to find ere (velocity of car measured by Barbara).
Using the Galilean transformation for velocities in one dimension:
Upg UPA UBA UPD 78 km /h + km/h
UPB km/h
Figure 74: This college-level question evaluates the capacity understand scientific figure, com- mand domain-specific knowledge in science, and execute rigorous reasoning. While GPT-4V mis- applies its domain-specific knowledge resulting in incorrect response, Multimodal Bard delivers the correct answer supported by a logical explanation.
71

---

Published as a conference paper at ICLR 2024
H.3.7 STATISTICAL REASONING
In IATIVISTA, problems encompass a variety of charts, plots, and graphs designed assess the statistical reasoning capabilities of foundation models. As demonstrated in Figure I, GPT-4V shows strong statistical reasoning ability. For instance, GPT-4V produces accurate answers for the format- rich table in Figure 75 and the data analysis table in Figure 76.
ID: 649
Most Americans say reports of sexual misconduct reflect societal problems harassment
Question: How many values are below 30 Mainly are incidents of individual misconduct?
say recent allegations and assaulll...
Mainly Incidents of individual misconduct
Mainly reflect widespread problems in society
Ground truth: 4
Men
34
60
Women
22
71
College gred
ge
31
60
Reg/Lean Rep
34
61
Dem/Lean Dem
70
Note: responses
Source: Survey of adult conducted 2017.
ILaMA-Adapter-V2 V2: There 2 values below 30 in Mainly incidents of individual misconduct.
LLaVA: There are two values below 30 in Mainly are incidents of individual misconduct.
Multimodal Bard:
There is only value below 30 in the "Mainly incidents of individual misconduct" column. This value is 22, which is the percentage of women who bclieve that recent allegations of sexual harassment and assault are mainly incidents individual misconduct.
The other values in this column are 34, 31, 33, and 25, All of these values are greater than 30.
So the answer to your question
GPT-4V:
Based on the "Mainly are incidents of individual misconduct" column, the following values are below 30:
-28 (Total)
222 (Women)
199 (College grad+)
25 (Dem/Lean Dem)
That's total of 4 values.
So. answer Correct output: 4
Figure 75: The question with a richly formatted statistical figure presents challenges for prominent multimodal models such as ALLMA--Aaapee V2, LLaVA, and Multimodal Bard. However, GPT-4V successfully extracts essential information from the figure, producing the correct answer.
72

---

Published as a conference paper at ICLR 2024
Price
Quantity demanded
Quantity supplied
ID: 389
$270
20,200
10,600
Question: Look at the table. Then answer the question. At a price of $320, there a shortage or a surplus?
$320
20,000
12,500
$370
19,800
14,400
Choices: (A) shortage (B) surplus
$420
19,600
16,300
$470
19,400
18,200
Ground truth: (A) shortage
LLaMA-Adapter-VZ. The table shows surplus 1000 units at price of $320.
LLaVA: At a price of $320, there is surplus of 12,000 units.
Multimodal Bard:
The correct answer is (B), surplus.
At a price $320, the quantity supplied is 12,500 units, while the quantity demanded is 20,000 units. This means that there are 7,500 more units supplied than demanded. So the answer is surplus.
Here table summarizing the supply and demand each price:
Price
Quantity Supplied
Quantity Demanded
Surplus/Shortage
$270
10,600
20,200
Shortage 9,600
$320
12,500
20,000
Surplus of 7,500
$370
14,400
19,800
Surplus of 5,400
$420
19,600
19,400
No surplus or shortage
$470
18,200
16,300
Shortage of 1,900
GPT-4V:
At price of $320:
Quantity demanded = 20.000
Quantity supplied 12,500
Since the quantity demanded (20,000) is greater than the quantity supplied (12,500), there shortage. Correct output: (A) shortage
Figure 76: LLaMA-Adapter- V2, LLaVA, and Multimodal Bard struggle to discern the structured information within the table, leading to incorrect answers.
73

---

Published as a conference paper at ICLR 2024
H.4 ABILITIES ACROSS VISUAL CONTEXTS
This section compares the reasoning abilities of GPT-4V with other large multimodal models (LLMs) MATHVISTA, considering various types of visual contexts. Models used for compar- ison include LLaMA-Adapter- (simplified as LLaMA-Adapter-V2) LLaVA-LLaMA-2-138 (simplified LLa VA), and Multimodal Bard.
H.4.1 ABSTRACT SCENE
Based on Figure 1, current foundation models lag behind human performance in mathematical rea- soning in abstract scenes by a substantial margin. Consider the problems in Figures 77 and 78 that derived from math word problems found in elementary school eurricula. Despite their advanced capabilities, foundation models such as Multimodal Bard and GPT-4V fail to produce the correct responses.
ID: 742
Move the ruler to measure the length of the twig to the nearest inch. The twig is about () inches long.
Question:
Ground truth:
LLaMA-Adapter-Va The twig about inches long.
LLaVA: The twig is about inches long.
Multimodal Bard:
According to the image, inch mark of ruler is aligned with the base the twig. The tip the twig is just past the 5 inch mark. Therefore, the twig about inches long.
GPT-4V: The twig is about 4 inches long.
Figure 77: Large multimodal models like GPT-4V struggle to answer this elementary- question with the abstract scene as the context.
74

---

Published as a conference paper at ICLR 2024
ID: 7
12
11
Question: What time is shown? Answer by typing time word, not ber. is () after eight.
10
2
Choices: (A) half (B) quarter (C) o' clockk (D) quarter to (E) quarter past
7
6
Ground truth: (B) quarter
L.aMA-Adapter-Va V2: The time shown is "half past eight."
LLaVA: The correct answer (C) o' o'clock.
Multimodal Bard:
The image you sent shows a clock with the hands pointing to 8:15. So the time is quarter past eight. (E)
The hour hand is between 8 and which means is past 8 c'oloc. The minute hand is pointing the 3, which 15 minutes past. Therefore, the time is quarter past eight.
GPT-4V:
The minute hand is pointing to the 9, which indicates that it is 45 minutes past the hour. The hour hand is pointing a little past the 8, which means it's after 8 o' o'clock.
minutes past hour also described as "quarter to" next
So, the time shown is "quarter to nine." The correct answer is (D) quarter to.
Figure 78:
Large multimodal models face challenges in answering this elementary-leve question which entails interpreting the time from an abstract clock diagram.
75

---

Published as a conference paper at ICLR 2024
H.4.2 BAR CHART
As shown in Figure foundation models, including GPT-4V, significantly underperform humans in mathematical reasoning when bar charts serve as the visual context. Neither Multimodal Bard nor GPT-4V can solve the problems depicted in Figures 79 and 80, which do not need complex understanding and reasoning.
Title
ID: 497
Question: What the value the largest bar?
Ground truth: 9
Values
rinety
f1eze
curb
[.MMA-Adapter-Va V2: The value of the largest bar is 10.
LLaVA: The value of the largest bar is 90,
Multimodal Bard: The largest bar in the image the one for the title "ninety". has a value of 10.
GPT-4V: The value of the largest bar is 10.
Figure 79: Large Multimodal Models such Multimodal Bard and GPT-4 fail to extract the correct value for the tallest bar in this basic bar chart.
ID: 629
Marz proferred cojects
Question: What percentage of people prefer the object hut?
Ground truth:
9ink
SUTTIT
Incker
10
Pertent People
LLaMA-Adapter--V V2: 10% of people prefer the object hut..
LLaVA: Based on the image, 30% of people prefer the object hut..
Multimodal Bard: Based the image, 100% of people prefer the object hut. The line graph shows that the percentage of people who prefer the hut is 100%, which is the maximum value the graph.
GPT-4V: The percentage people who prefer the object "hut" is approximately 40%.
Figure 80: To address this question, the model simply needs to identify the "hut" and determine its value on x-axis. However, current foundation models such as GPT-4V struggle to produce correct answer.
76

---

Published as a conference paper at ICLR 2024
H.4.3 FUNCTION PLOT
GPT-4V outperforms other baselines on problems related to function plots and even exceeds human performance, Figures 81 and 82 show questions with digital and hand-drawn function plots, respec- tively. both cases, GPT-4V accurately identifies their functions and infers the correct properties.
y
ID: 767
x²
25
Question: What is the maximum value of y?
Ground truth:
(3, -)
[.MMA-Adapter-V. V2:
The maximum value of is 3.
LLaVA: The maximum value of y is
Multimodal Bard:
The maximum value of is 3.
The given equation is n² x This the equation of circle with center (0, and radius /2 The point (3, -4) lies the circle, the maximum value of is 3.
GPT-4V:
The equation given is for circle: x²² + 25.
The radius of this circle is the square root of 25, which is 5,
The maximum value of y corresponds to highest point of the circle, which is equal to the radius.
So. the maximum value of y is 5. Correct output: 5
The figure illustrates a function plot accompanied by its equation. GPT-4V correctly identifies the equation, leading to the correct answer.
Figure 81:
77

---

Published as a conference paper at ICLR 2024
y
Question: Is this function odd or even?
y=h² a>1
Choices: (A) odd (B) even
Ground truth: (A) odd
0
LLaMA-Adapter- The function odd. Correct output: (A) odd
LLaVA:
The function in the image is odd. Correct output: (A) odd
Multimodal Bard: The function in the image odd.
An even function is function where f(-1) f(x). An odd function is function where f(-z) = f(x).
In the case the function y 022 01 if we substitute =2, we get:
a2 2 -
u2 + d1
This equal to f(x), the function is odd.
The correct choice is (A). Correct output: (A) odd
GPT-4V:
The given function y a where >
To determine if the function is odd or even, we can use the following definitions:
function f(x) is even if f(-x) fxx) for all in the domain of f. function f(x) is odd if x)=x)=== = fx()) for all x the domain
For the given function = a", when we plug - for a::
y
is not equal to and not equal to a" either.
Thus, the function is neither odd nor even. However, since the given choices only include "odd" and "even". the best answer is neither. But since I must choose one of the provided options (A) odd.
Correct output: (A) odd
The figure showcases a hand-drawn function plot. GPT-4V identifies the function's expression and accurately determines its property.
Figure 82:
78

---

Published as a conference paper at ICLR 2024
H.4.4 GEOMETRY DIAGRAM
Geometry diagrams distinct type of visual context in MATUIVISTA. To answer questions involv- ing these diagrams, model must comprehend the fine-grained details, including symbols, variables, and relations from the figures. Additionally, it should apply associated theorems before executing calculations to produce final responses. GPT-4V surpasses other models and even humans due to its superior capabilities in geometry recognition and reasoning. the examples shown Figures 83 and 84, GPT-4V delivers the correct results through the application of relevant theorems and subsequent calculations.
ID: 478
Question: As shown in the figure, in the diamond ABCD, two diagonal lines AC 12.0, BD 16.0, then the edge length of this diamond is ()
8
Choices: (A) 10 (B) 8 (D)
Ground truth: 10
[aMMA-Adapter-V V2: The edge length of this diamond is
LLaVA: Based on the given information, the edge length the diamond ABCD is 8.0.
Multimodal Bard:
The answer is (A). 10.
rhombus, diagonals bisect each other at right angles. So, we can draw two right triangles within the rhombus, each with a leg of length and hypotenuse of length 10.
By the Pythagorean Theorem, the length of each side of the rhombus is 6
6
V22 = 10.
answer is (A). Correct output: 10
GPT-4V:
Given that AC 12.0 and BD 16.0 in the rhombus (diamond) ABCD.
Since sides of rhombus are congruent, the diagonals of rhombus are perpendicular bisectors each other. This means that each half of the diagonal BD 8.0 and each half of diagonal AC is 6.0.
Using the Pythagorean theorem for one of the right triangles (formed by the diagonals):
Side (Half of BD) + (Half of AC)
Side 8² + 6²
Side 64 +
Side 100
Side 10
Correct output: 10
Figure 83: For this geometry problem, GPT-4V correctly applies the appropriate theorem and arrives at the right answer. In contrast, Multimodal Bard commits calculation errors during its reasoning process.
79

---

Published as a conference paper at ICLR 2024
410
Question: Find the perimeter of ADEF, F, if ADEF ~ ACBF, perimeter of ACBF = 27, DF 6, and FC = 8.
B
Choices: (A) 20.25 (B) (C) 27 (D)
Ground truth: (A) 20.25
LLMMA-Adapter-V V2: The perimeter of ADEF is 20.25, as shown in the image.
Correct output: 20.25
LLaVA:
The perimeter ADEF can calculated using the similarity theorem, which states that if two triangles are similar, their corresponding sides are proportional. In this case, we know that ADEF ~ ACBF and the perimeter of ACBF is 27.
Since the ratio of the corresponding sides of similar triangles is the same, can use the perimeter of ACBF to find perimeter ADEF. The perimeter of ADEF is proportional ACBF, so we can multiply the perimeter of ACBF by the ratio of the corresponding sides.
perimeter
The ratio of the corresponding sides of ADEF and ACBF is 6:8, which means that the perimeter of ADEF is 8 48.
Therefore, the correct answer is (D) 48.
Multimodal Bard:
Since ADFF ACBF, we have
DE CB
DF
DE Since DE 6 and FC = 8, have
PC
CB
The perimeter ACBF is 27, so CB BF
Substituting
for CB BF
get BF + BF 27.
Solving for BF, we get BF 12.
The perimeter of ADEF is DE + EF DF = 6 EF + 6 EF 12. Substituting 3 for EF BF
we
get EF + 12 EF. Solving for EF, we get EF = 9.
Therefore, the perimeter of LDEF is EF DE DF ++0+0 21
GPT-4V: Given ADEF ACBF and DF and FC 8, we can find the scale factor of the two similar triangles.
Scale factor
DF FC
Given the perimeter of ACBF 27, can find the perimeter of 4DEF by multiplying by the scale factor.
Perimeter of ADEF= 27 x 20.25.
Correct output: 20.25
Figure 84: For this geometry problem, GPT-4V correctly applies the appropriate theorem and arrives at the right answer. contrast, Multimodal Bard commits calculation errors during its reasoning process.
80

---

Published as a conference paper at ICLR 2024
H.4.5 LINE PLOT
As evidenced by Figure current models such GPT-4V do not perform as well humans in mathematical reasoning involving line plots. We speculate that the low performance is mainly due to the difficulty in detecting OCR text in the figures and accurately grounding the values, as illustrated by the examples in Figures 85 and 86.
ID: 981
70W
Question: What the sum 2002, 2003 and 2004?
Shan mn
30.75
Ground truth: 70.4
506
39.8
606
25.15
19.4%
206
12.7%
106
LLaMA-Adapter- V2: The of 2002, 2003 and 2004 is 6.6.
LLaVA: The sum of 2002, 2003, and 2004 6007.
Multimodal Bard:
see the image. The sum of 2002, 2003 and 2004 is 1.40.
The year 2002 has share of 67.4%, 2003 has share of 64.3%, and 2004 has share of 62%. When add these three values, we get 1.937%. Rounding decimal place, the sum is 1.40.
GPT-4V:
The sum for the years 2002, 2003, and 2004 from the graph are:
2002: 19.4%
2003: 23.2%
2004: 25.1%
Sum 19.4 23.2 25.1 67.7%
Figure 85: The figure displays low-resolution line plot with numerous points. Current foundation models, including GPT-4V, appear to struggle with accurately interpreting the plot.
81

---

Published as a conference paper at ICLR 2024
ID: 209
30.1
200
Question: What is the high- value in black line chart?
28.2
25.5
Ground truth: 28.3
27A
275
26.8 26.9 26.9
27.1
27.1
27.
26.6
253
25 25.1251255
1896
2010
2012
2013
2014
2015
2017
2018
2019
Men Women
LLaMA-Addapte--Zz V2: The highest value in the black line chart is 1.4.
LLaVA: The highest value in the black line chart is 1.4.
Multimodal Bard: The highest value black line chart 1.4.
GPT-4V: The highest value in the black line chart, which represents "Women," is 30.1.
Figure 86: The figure displays two low-resolution line plots comprising numerous points. Current foundation models, including GPT-4V, seem to struggle with interpreting the plots accurately.
82

---

Published as a conference paper at ICLR 2024
H.4.6 NATURAL IMAGE
VIST includes questions that require numeric and spatial reasoning based on text and objects in natural images. If models have limited abilities to recognize text (OCR), as shown in Figure 87, or identify visual objects, as in Figure 88, they are unlikely generate correct answers visual questions.
ID: 980
Question: What the highest number shown?
Ground truth: 12
LLaMA-Adapter-22 V2: The highest number shown is 12.
Correct output: 12
LLaVA: The highest number shown on the clock tower is 3.
Multimodal Bard: I see the image. The highest number shown
GPT-4V: The highest number shown on the clock is 12.
Correct output: 12
Figure 87: LLaVA and Multimodal Bard are unable to identify the highest number displayed on the clock panel in the image.
83

---

Published as a conference paper at ICLR 2024
ID: 884
Question: What is the blue kite in the lower right corner shaped like?
Choices: (A) ferret (B) cat (C) cloud (D) octopus
Ground fruth: (D) octopus
LLaMA-Adapter-Va. V2: The blue kite in the lower right corner shaped like cloud.
LLaVA: The blue kite in lower right corner shaped like cloud.
Multimodal Bard: Sorry, can't 't help with images of people yet.
GPT-4V:
The blue kite in the lower right corner is shaped like (D) octopus. Correct output: (D) octopus
Figure 88:
The question demands spatial reasoning ability from the model to pinpoint the object within the image. In this context, only GPT--V delivers correct prediction.
84

---

Published as a conference paper at ICLR 2024
H.4.7 PUZZLE TEST
Math reasoning with puzzle text figures is challenging for current AI foundation models because interpreting these figures requires discerning underlying patterns from sets of shapes, as illustrated in Figure 89, and numbers, as Figure 90. There is plenty room for improvement.
ID: 123
Question: The diagram represents successive rotations, starting from the top down. Which shape comes next?
Choices: (A) (B) ( (E)
Ground truth: (D) D
LLaMA-Adapter- V2:
The diagram represents successive rotations, starting from the top down. The next shape that comes next is letter D.
Correct output: (D) D
LLaVA: The correct answer is (E) E.
Multimodal Bard: The answer is (C).
The diagram shows 3D cube being rotated 90 degrees each axis. Starting from the the first shape is square. The second shape is cube with square missing from the top face. The third shape cube with square missing from front face. The fourth shape cube with square missing from the left face.
To find the next shape, we need to rotate the cube 90 degrees on the z-axis. This will bring the square that is missing from the left face to the front face. So the next shape is the one shown in option (C).
GPT-4V:
The diagram shows a pattern rotating the shape a specific sequence. To determine the next shape:
Starting with the first shape, when we rotate 90 degrees counterclackwise, we get the second shape.
Rotating the second shape 90 degrees counterclockwise gives the third shape.
Thus, rotating the third shape 90 degrees counterclockwise will give us the next shape.
From the choices provided, the shape that matches this rotation is (E) E.
Figure 89: This puzzle test figure demands intricate skills in abstract thinking and spatial reasoning, which prove challenging for current foundation models. Although GPT-4V correctly discerns the transformation pattern of the shapes, errs in spatially distinguishing between shapes (D) and (E).
85

---

Published as a conference paper at ICLR 2024
ID: 941
Question: Which is the next number in the series?
3968
Ground truth: 2
63
Solution: (3968 + 1) = 63
(63 + 1) 8²
(8 + 1) =
1) 2
LLaMA-Adapter-VV V2: The next number in the series is 3,
LLaVA: The next number in the series is
Multimodal Bard:
The number the series is
The series is based on the square root of the number plus 1. So, the first number is 3968, and its square root is 63, Then, 63 + 64, and its square root is 8. Continuing the pattern, 8 + 1 and its square root is 3.
Therefore, the next number the series is 9.
GPT-4V:
The pattern in the ladder involves dividing number by the step number to get next number.
Step 1: 3968 + 3968
Step 2: 3968 + 1984 but have 63, which doesn't 't follow this pattern.
Step 3: 63
21 hut have 8. have
Step 4: 8
Step 5: 3 5 0.6
The given sequence doesn't 't seem have consistent pattern based the values presented.
However, if assume the ladder sequence based dividing by consistent factor (rather than step number), we can attempt to find common factor:
3968 63 63
63 8= 7.875 which isn't round number.
2.6666 again not round number.
Without clear and consistent pattern, it's challenging to determine the next number the sequence.
The question demands advanced abstract thinking discern the consistent pattern within a sequence of four numbers. Despite their capabilities, current foundation models including Multi- modal Bard and GPT-4V show limited abilities of logical reasoning.
Figure 90:
86

---

Published as a conference paper at ICLR 2024
H.4.8 SCATTER PLOT
scatter plot graphical representation of data points two-dimensional axis, where each point represents a value from the dataset. MATIIVISTA includes the reasoning task that requires comprehending scatter plots taken from daily life and academic papers, shown in Figures 91 and 92. Although GPT-4V outperforms other LMMs, such as Multimodal Bard, and even humans in overall accuracy (Figure I), it often fails in the cases where fine-grained understanding is required, as in Figure 92.
ID: 105
Lith
Question: Does Dark Violet have the minimum area under the curve?
Dark Vieiet
Pink
Choices: (A) yes
Medium Mint
Cyun
Ground truth: (A) yes
Yellow
Xusis labe!
LLaMA-Adapter-VZ V2: No, Dark Violet does not have the minimum area under the curve.
LLaVA:
Based on the image, Dark Violet does not have the minimum area under the curve. The area under the curve the highest for the color Green.
Multimodal Bard:
The correct answer is (B), no. Dark Violet does not have the minimum area under the curve. The color with the minimum area under the curve is Yellow Green.
The under the curve is calculated by summing the of all the rectangles under the The rectangle for Dark Violet has height of 92 a width of 20, its area 1840. The rectangle for Yellow Green has a height of 88 and a width of 40, so its area is 3520. Therefore, the area under the curve for Yellow Green is larger than the area under the curve for Dark Violet.
GPT-4V:
Based the scatter plot provided, "area under the curve" is not represented since there is shown. However, by "area under the curve" we mean the overall distribution and position the points "Dark Violet" compared to others, we can analyze the positions of the points.
From the scatter plot, the color "Dark Violet" consistently appears to be the lowest the y-axis across the entire range of the x-axis.
Given the choices: (A) yes (B) no
The correct answer is: (A) yes.
Correct output: (A) yes
Figure 91: The seatter plot presented in this problem is cemplate-generaae While models such as LLaMA-Adapter V2, LLaVA. and Multimodal Bard encounter difficulties in discerning quantitative relationships between different plot lines, GPT-4V successfully discerns the correct relationships and provides accurate answer.
87

---

Published as a conference paper at ICLR 2024
Model Performance
ID: 542
ag--
Question:
How many
models in the figure achieve an Acc score greater than 60?
80
LLa05-70
11a8592-78
LLoM12
Ground truth:
LLak5h535
LLakWW655
LLa0192-708
Acc
GPT-
LuM 201
Uing
organe
2.0
1.5
1.8 Pretrain Loss
1.4
1.2
Figure The performance of SPT (blue lines) and ICL (red lines) settings on GSM8K, GPT-4 states they part between SFT and ICI.
pre training, suggest others consider performance
L.MMA-Adapter- There are 10 models in the figure that achieve an Ace score greater than 60..
LLaVA: There are two models in the figure that achieve an Acc score greater than 60.
Correct output: 2
Multimodal Bard:
There are models in the figure that achieve an Acc score greater than 60, These models are:
GPT-4
GPT3-175B
LLLMA-7B
LLAMA2-7B
LLAMA- -1BB
LLAMA2-13B
LLAMA-33B
LLAMA-65B
LLAMA2-70B
The Ace score is the accuracy the model, which is the percentage of test cases that the model correctly classifies. An Acc score of 60 means that the model correctly classifies 60% of the test cases.
GPT-4V: There 6 models in the figure that achieve Acc score greater than 60.
The example showcases a seatter plot sourced from an academic paper. Current models, including GPT-4V, struggle to discern the fine-grained relationships depicted in the figure.
Figure 92:
88

---

Published as a conference paper at ICLR 2024
H.4.9 SCIENTIFIC SCENE
Answering questions based on scientific scenes poses challenge aligning the scientific concepts present in the question text and those in the accompanying figures. GPT-4V demonstrates its superior ability reason about scientific scenes compared other models, as evidenced in Figure In the example of Figure 93, GPT-4V adeptly identifies two organisms in the food web and elucidates their relationship. In another instance, shown Figures 94 and 95, both Multimodal Bard and GPT-4V are able to use knowledge in the physical domain to effectively ground the symbols and variables depicted the figure.
ID: 685
Sun
Question: What would happen to organ- ism organism increased?
Choices: (A) decrease (B) increase (C) can't predict (D) stay same
Organiem A
Organiam B
Organiam C
Organiam D
(Not drawn scale)
Ground truth: (B) increase
LaaMA-Adapter-V:
If organism b increases, organism would decrease. This is because organism c is dependent on organism b for its food supply, and an increase in organism b would lead decrease in organism
LLaVA:
If organism b increased, organism e would likely decrease. This is because the growth of organism b would lead to increased competition for resources, such as food and space. As result, organism e would have to compete more fiercely for these resources, which could lead to decrease in its population.
Multimodal Bard:
The answer is (A). organism increases, organism will decrease.
In the food chain, organism B is the predator of organism C. This means that organism B eats organism C. If there are more organism B, then there will be more predation on organism C. This will lead to decrease in the population of organism C.
So the answer is (A).
GPT-4V:
If Organism B (which appears be some kind of insect) increased in population, it would mean that there is available food for Organism (the fish).
Therefore, the population Organism would likely: (B) increase.
Correct output: (B) increase
Figure 93: Models such LLaMA-AAapter-Ve V2, LLaVA, and Multimodal Bard seem struggle with discerning the relationships between organisms depicted in the food chain. Instead, GPT-4V recognizes organisms in the food chain and possesses the knowledge to infer their relations.
89

---

Published as a conference paper at ICLR 2024
8n
Water pool
Launch
D
ID: 862
Question: One of the most dramatic videos the web (but entirely fictitious) supposedly shows sliding along long water slide and then being launched into the air land in water pool. Let's attach some reasonable numbers to
flight calculate the velocity with which the man would have water. Figure indicates the launch and landing sites includes superimposed coordinate system with its origin conveniently located at the launch site. From the video we take the horizontal flight distance as D 20.0 the flight time as / 2.50 and the launch angle as 0g 40.0% Find the magnitude of the velocity launch and landing.
Ground truth: 10.44
Multimodal Bard:
Given the following information:
* Horizontal flight distance, 20.0 m Flight time, t 2.50 s Launch angle, 0u = 40.0*
We find the magnitude of the velocity at launch using the following equation:
Uo / / vBy
where the the horizontal component of the launch velocity and Bay is the vertical component of the launch velocity.
The horizontal component of the launch velocity can be found using the following equation:
D 20.0 m 8.00 m/s
Vor t 2.50 s
The vertical component of the launch velocity can be found using the following equation:
Boy 2 0g 8.00 m/s sin 40.0 5.83 m/s
Therefore, the magnitude of the velocity launch is:
Vo U
v0 (88.00 m/s)3 (5.83 m/s)2 10.44 m/s z 10.4 m/s
The magnitude velocity at landing will be the same magnitude the velocity at launch, since there no air resistance in this case.
So the answer is 10.4 rounded to two decimal places.
Figure 94: This college-level question. derived from the physics curriculum, requires understand- ing the intricate scenario depicted in the scientific figure, possessing domain-specific knowledge, and performing multiple reasoning steps to obtain conclusive answer. Multimodal Bard correctly processes the reasoning but failed to present the answer in the appropriate format.
90

---

Published as a conference paper at ICLR 2024
8g
Water
Launch
pool
D
ID: 862
Question: One of the most dramatic videos the web (but entirely fictitious) supposedly shows sliding along long water slide and then being launched into the air land in water pool. Let's attach reasonable numbers such flight calculate the velocity with which the would have water. Figure indicates the launch and landing sites includes superimposed coordinate system with its origin conveniently located at the launch site. From the video we take the horizontal flight distance as D = 20.0 the flight time as / 2.50 and the launch angle as 0u 40.0% Find the magnitude of the velocity launch and landing.
Ground truth: 10.44
GPT-4V:
Given:
D 20.0
- 2.50
0b 40.0"
Using kinematic equations, the horizontal motion is described as:
D Voa
Where Voz is the horizontal component of the initial velocity.
Vo. Vo cos(fo)
From the first equation:
Vo x cos(fu)
Rearranging for Vo:
D
Vo
x cos
Substitute the given values get Vo:
20.0 m
VO 2.50 x cos (0.00))
Using a calculator:
20.0
V/ 2.50 x 0.766
20.0
V. 1.915
Vo 10.44 m/s
Thus, the magnitude of the launch velocity (which will be the same as the landing velocity, ignoring air resistance) is 10.44 m/s. Correct output: 10.44
Figure 95: Continuing from Figure 94, GPT-4V generates the correct answer and the associated reasoning steps for this scientific problem.
91

---

Published as a conference paper at ICLR 2024
H.4.10 SYNTHETIC SCENE
Problems involving synthetic scenes require a nuanced understanding of visual objects, such as the numbers, attributes, and positions of these objects, as shown in Figures 96 and 97. Although GPT- 4V demonstrates notable advancements over other models, such as Multimodal Bard, it still falls significantly short of human performance, as shown in Figure 1.
ID: 52
Question: Subtract big metallic spheres. Subtract all big brown matte things. How many objects are left?
Ground truth: 6
LLLMA-Adapter-V V2:
The final value is 1, as there is only one object left after subtracting all big red metallic spheres and all big brown matte things.
LLaVA:
After subtracting all big red metallic spheres and big brown matte things, there are 3 objects left.
Multimodal Bard:
Given the image, there are 4 objects:
I big red metallic sphere
big brown matte cube
small red metallic sphere small brown matte sphere
If subtract all big red metallic spheres, there will be 3 objects left::
big brown matte cube
small red metallic sphere
small brown matte sphere
If subtract all big brown matte things, there will be objects left: small red metallic sphere
small brown matte sphere
Therefore, if we subtract all hig red metallic spheres and big brown matte things, there will be objects left.
GPT-4V:
There are no big metallic spheres in image. There is big brown matte object (the cube).
Total objects to be subtracted = 0 +
Total objects in the image
Objects left = 7
There are objects left. Correct output: 6
Figure 96: In this example, GPT-4V able to recognize the objects with specific attributes from the synthetic figure and then perform arithmetic reasoning get the correct answer.
92

---

Published as a conference paper at ICLR 2024
ID: 177
Question: Is the number of brown tandem bikes that the the small blue matte car greater than the number tiny blue biplanes?
Choices: (A) Yes (B) No
Ground truth: (A) Yes
LLaMA- LLMMA-Adapter-Vz: the number brown tandem bikes are to the left of the small blue matte car greater than the number of tiny blue biplanes.
Correct output: (A) Yes
LLaVA:
The correct answer is (B) No. There are more tiny blue biplancs to the left of the small blue than brown tandem bikes.
Multimodal Bard:
The answer (B). There are two brown tandem bikes to the of the small blue matte car, and there only one tiny blue biplane. So the number brown tandem bikes is not greater than the number of tiny blue biplanes.
GPT-4V:
Looking the image, there brown tandem bike the left of the small blue matte car. There are no tiny blue biplanes in the image.
Thus, the number of brown tandem bikes that are to the left of the small blue matte car is (A) Yes greater than the number of tiny blue biplanes. Correct output: (A) Yes
The question in this example requires the models to accurately extract the attributes of different objects from the synthetic figure. GPT-4V infers the correct answer based its generated detailed description of the figure.
Figure 97:
93

---

Published as a conference paper at ICLR 2024
H.4.11 TABLE
Tables serve a powerful tool to present and summarize large amounts of data in comprehensible manner. In particular, GPT-4V has shown significant advancements over other foundation models and even surpasses human performance on table-related reasoning tasks, as shown Figure The example in Figure 98 shows a complex table taken from an academic paper. GPT-4V can accurately pinpoint the target cells among numerous rows and columns. Figure 99 shows QA task which the answer needs to be derived from the table regarding the push-up competition. GPT-4V is the only model that can produce the correct answer.
Table MM Vet evaluation results on various LMMs regarding each core VL cxpability. For each column, the highest, the second, and the third highest figures are highlighted by green, orange and blue backgronnds, All the numbers presented in and the full
ID: 923
100t..
Question: Which model achieves the highest score in terms of Rec?
Model
Rec
OCR
Know
Gen
Spar
Math
Tocal
Trunsformers Agent (GPT-4)
18.2
222
3.2
12.4
13.4=0.3
ILaMA-Adapter v2-7B [28]
3.0
13.6=0.2
OpeaFlamingo988 6]
24.6
14.4
13.0
12.3
18.0
15.0
21.8=0.1
Choices:
BLIP-2-12B
MiniGFT-4-8B [84]
27.4
15.0
12.8
13.9
20.3
22.1=0.1
(A) Transformers Agent (GPT-4)
LLaVA-78 [46]
28.0
17.1
16.3
18.9
21.2
11.5
23.8-0.6
(B) LLaMA-Adapter v2-7B v2-7B
MinJGPT-44114B [84]
29.9
16.1
20.4
22.1
22.2
3.8
24.40044
(C) LLaVA-7B
Instru:iBLIP-14B [19]
30.8
16.0
9.8
9.0
21.1
25.6:00.3
(D) Otter-9B
IestructBLIPP8B [19]
72.4
14.6
16.5
18.2
18.6
7.7
26.2::0.2
(E) R-M-RaAct-
MM--RAca-GP1:335510
LLaVA-138
24.2
30.9
31.5
23.5
21.5
20.7
26.4
24.3
32.3
26
27.9-011
26.4-0.1
(F) LLaVA-13B (LLaMA-2)
LLaVA-7B (LLaMA-2) [46]
32.9
20.1
19.0
20.1
25.7
522
28.1=04
(G) MM-ReAcI-GPT44
LLaVA 13B (V133, 336px) [46]
38.1
22.3
25.2
25.8
31.3
11.
32.510.1
(LLaMA-2) [46]
39.2
26.5
MM-ReAc--GPT4 [76]
33.1
65.7
29.0
35.0
568
69.2
44.6=20.
Ground truth:
(F) LLBVA-13B (LLaMA-2)
[lMMA-Adapter-V2: The correct answer (G) MM-ReAct-GPT-4
LLaVA:
The correct answer is (B) LIAMAAAAdppt v2-7B, which achieves the highest score in terms of Rec.
Multimodal Bard:
The answer is (G) MM-ReAcu-GPT-4
MM- MM-ReAct Mil-RRectGPP-44 GPT-4 achieves the highest score of 33.1 in terms of Rec, which is the ability recognize and understand text. other models have the following scores terms of Rec:
Transformers Agent (GPT-4): 18.2
LLaMA-Adapter v2-7B: 16.8
VLaVA-7B: 28.0
Otter-9B: 28.4
8 MM-ReAct-GPT.3.5: 24.2
LLLVAA13B (LLaMA-2): 32.9
GPT-4V:
The model that achieves the highest score in terms of Rec is "LLaVA-13B (LLaMA-2)" with score of 39.2%. The correct option is: (FF LLBVA-13B (LLaMA-2) Correct output: (F) LLa VA-13B (LLaMA-2)
Figure 98: The problem presents complex table from an academic paper, complete with caption
and multiple rows and columns. This format poses challenges models like LLaMA-Adapter-V LLaVA, and Multimodal Bard. However, GPT-4V demonstrates its exceptional reasoning abilities within tabular contexts.
94

---

Published as a conference paper at ICLR 2024
Push-up competition (number push-ups) ID: 265
Stem
Leaf
Question: Justine's P.E. class participated in push-up competition, and Justine wrote down how many push-ups cach person could do. How many people did least 60 push-ups? (Unit: people)
4
8
3677
7
1335778
Ground truth: 11
LLaMA-Adapter-VZ
Justine's P.E. class participated in push-up competition, and Justine wrote down how many push-ups cach person could There were people the competition, and least 60 push-ups were done by 3 people. The final value is
LLaVA:
In the image, there are total of people participating the push-up competition. Among them, there are two people who did at least 60 push-ups.
Multimodal Bard:
There are people who did least 60 push-ups.
The person with the highest number push-ups did 778, so the only people who did least 60 push-ups are the ones who did 60, 77, 78, and 778.
There is one person who did 60 push-ups, person who did 77 push-ups, one person who did 78 push-ups, and one person who did 778 push-ups. So there total of 1 + people who did at least 60 push-ups.
GPT-4V: For the stem-and-leaf plot provided:
represents tens value, and digits 3, 6, 7, and represent the ones value. So, the number push-ups represented by the row are 63, 66, 67, and 67.
7 represents the tens value, and the digits 1, 3, 7, and represent the ones value. So, the number of push-ups represented by the 7 row are 71, 73, 73, 75, 77, 77, and 78.
Counting these numbers: from the row 7 from row 11.
Correct output: 11
Figure 99: This figure features a stem-and-leaf plot used for statistical analysis. While LLaMA- Adapter- V2, LLaVA, and Multimodal Bard struggle to comprehend this domain-specific table, GPT- 4V excels in understanding its content.
95

---

Published as a conference paper at ICLR 2024
H.4.12 OTHER VISUAL CONTEXTS
On the reasoning tasks using other visual contexts, GPT-4V achieves higher overall accuracy than all the other models, as depicted in Figure I. For instance, GPT-4V is the only model that is capable of generating the correct answer to the question regarding violin plot, as shown Figure 100.
Audio-Audio Similarity
Text-Audio Similarit
1.0
0.5
0.0
MusicL.DM DM (mix-sp)
Muxicl DM (vrigima))
MuseLDM (BLM)
MusicLDM (BAM)
MuBERT
Figure 3: The violin plot the audio-audio similarity, and the text-to-audio similarity.
ID: 895
Question:
Which model has lowest Audio-Audio Similarity
Text-Audio Similarity scores
overall?
Choices:
(A) MusicLDM (mix-up)
(B) MusicLDM (original)
(C) MMsicI.DM (BLM)
(D) MusicLDM (BAM)
(E) MuBERT
LLaMA-Adapter-V
The model with the lowest Audio-Audio Similarity Text-Audio Similarity scores overall MusicI.DM (BLM).
LLaVA:
Based the image, the model with the lowest Audio-Audio Similarity and Text-Audio Similarity scores overall is Musicl. (mix-up).
Multimodal Bard: The answer is (C), MusicLDM (BLM) has the lowest Audio-Aude Similarity and Text-Audio Similarity scores overall.
The violin plot shows the distribution audio- audio similarity scores for MusicLDM (BLM) the most spread out, indicating that there is variation scores. distribution of text-audio similarity scores for MusicI.DM, .DM (BLM) is also the most spread out. This suggests that MusicLDM (BLM) not very consistent in its ability to capture the similarity between audio and text.
GPT-4V:
The model with the lowest Audio- Audio Similarity and Text-Audio Similarity scores overall is MuBERT.
Correct output: (E) MuBERT
Figure 100: Violin plots are widely used in academic papers visualize the distribution of data and probability density. In this example, only GPT-4V is able to generate the correct output from the figure.
96

---

Published as a conference paper at ICLR 2024
H.5 SELF- VERIFICATION GPT-4V
Self-verification is a social psychological theory asserting that people desire others to perceive them they see themselves. Consequently, individuals will take active measures view them in ways that confirm their stable self-concepts (Talaifar & Swann, 2020).
that others
Interestingly, our experiments, GPT-4V demonstrates an ability similar self-verification. The model can inspect its own behaviors during the course of reasoning and can take active actions to correct its mistakes. Note that self-verification we discuss here differs from several recent works on improving the model's outputs based on external feedback (Peng et al., 2023) or additional gener- ations (Yang et al., 2023b). The examples in Figures 101 and 103 show that GPT-4V, on own, can inspect set of candidate answers and identify the one that is valid and meets all the given con- straints. The multi-step reasoning example Figure 102 shows that GPT-4V can verify the validity of (the result of) each reasoning step, and explore alternative approaches if any invalid (intermediate) result detected (e.g., a negative value for length).
Although self-verification does not guarantee an accurate response after multiple tries, espe- cially when applying GPT-4V to visual perception or mathematical reasoning in intricate scenarios (see Figure 104), it is instrumental in improving the model performance MATH We also found that GPT-4V's verifiiirii weaker for non-English tasks, such as Mandarin, as shown in Figure 105. It is also worth noting that self-verification does emerge in other foundation mod- els we studied, or at least is not as robust as that of GPT-4V. As shown in Figure 106, Multimodal Bard first attempts a natural language solution, followed by program- -assisted one for verification. However, the
solution leads to different and incorrect prediction.
The emergent ability of self-verification highlights GPT-4V's potential in solving rigorous reasoning and theorem- provinn tasks. One of the most exciting research topics for future work is to develop mechanism that allows the model activate verfiiciiinntoonn consistently at the right time and to use set of alternative approaches that maximize the success rate of task completion.
ID: 615
12
Question: two sides of triangle measure and 7, which the following cannot be the perimeter the triangle?
Choices: (A) 29 (B) 34 (C) 37 (D) 38
Ground truth: (D) 38
GPT-4V:
The sum of the lengths of any two sides of a triangle must be greater than the length of the third side. Let be the length of the third side.
From the triangle inequality:
12+7
Similarly:
7+ > >>>
And:
12+ -5 (This will always be true since the length cannot be negative)
So,
5< 19
Now, let's check the given options:
(9) 29 29
12
10 (Possible)
- (B)
34
12
15 (Possible)
(C) = 37
12 -
18 (Possible)
(D) 38 => x 38 - 12 - 7 = 19 (Not Possible)
Correct output: (D) 38
Figure 101: Given several possible values for the length of the third side, GPT-4V verifies each candidate answer to ensure that it meets the given constraints (highlighted in blue).
97

---

Published as a conference paper at ICLR 2024
ID:
D
C
Question: As shown in the figure, if CB 4.0, DB 7.0, and D is the midpoint of AC, then the length of AC is O.
Choices: (A) 6 cm (B) 7 cm (C) 4 cm (D) cm
Ground truth: (A)
GPT-4V:
From the figure, we can deduce:
AD DC (since D the midpoint of AC)
Now, using the entire line segment AB:
AB AD DB
AB DC + DB
Given DB em, we can rearrange to find:
AD (or DC) = AB
AD = AB - 7.0
Now, let's find AB: From the given information,
AB AC CB
= AC 4.0
Using the previously deduced equation:
AD = AC + 4.0 - 7.0
AD = AC - 3.0
Since D is the midpoint of AC, AD will be half the length of AC:
AC 2(AD)
From the above:
AC 3.0 = 2(AD)
Given that AD and DC equal, represent DC as:
DC = AC/2
Using the relation CB 4.0 and DB 7.0
AC/2 7.0 7.0
AC/2 = -300
This is not possible as lengths cannot be negative. Let's re-evaluate our approach.
From the original figure:
AB AD+DB AB = DC + DB
Given DB 7.0 cm, we have:
AB 7.0 AD AC/2 7.0
Using the relation AB = AC CB:
AC/2+7.00 = AC 440
Subtracting AC/2 from both sides:
7.0 = AC/2+44
Subtracting 4 from both sides:
AC/2 = 3.0
Multiplying both sides by
AC 6.0 cm
Correct output: (A) 6 em
Figure 102: GPT-4V generates the result AC/2 = -300. It verifies the validity of the value and concludes it cannot negative. The model then re-evaluates its approach and adopts a new method to obtain the correct answer, highlighted in blue.
98

---

Published as a conference paper at ICLR 2024
ID: 28
Question: AB is diameter, AC 8 inches, and BC 15 inches. Find the radius of the circle.
B
Choices: (A) 7.5 (B) (C) (D)
Ground truth: (C) 8.5
GPT-4V:
Given that AB is diameter and AC 8 inches and BC 15 inches, use the Pythagorcan thcorem to determine the length of AB,
The right triangle is formed by A, the center of the circle, and C' makes AC the shorter leg, the radius the longer leg, and BC' the hypotenuse.
Let be the radius of the circle. Using the Pythagorean theorem:
AC + r² BC
Substituting the given lengths:
8 n 152
64 + 3 225
2 161
Taking the square root of both sides:
v161
However, is not any of the choices. We need to find the diameter, which is twice the radius.
AB = 2r = 2 I61
since the diameter just twice the radius and the radius is one of choices, the radius
/161 /2 8.5 (approximately)
Correct output: (C) 8.5
Figure 103: Starting by computing the initial radius length as = V161, GPT-4V performs two verifications resulting in different answers: r = 2V161 and r v161/2. then selects the matching option from the choices as its final prediction, highlighted in blue.
99

---

Published as a conference paper at ICLR 2024
ID: 644
Question: What is the missing number in the picture?
10
Choices: (A) (B) (C) (D)
Ground truth: (B) 8
Solution: 8 - 2 10, + = 12 - =
GPT-4V:
Let's 's analyze the pattern in the given triangles:
1. For the first triangle:
(8 -) 4) =
2. For the second triangle:
(3 2 2) 3
Applying same pattern the third triangle:
(12 - 1) x =
However, since the answer in single digits, seems take only the unit's place of the result.
So, the missing number is: (unit's place of 55)
This is not in the given options. Therefore, let's explore another possible patterm:
For first triangle:
(8 + 4)/2
2. For the second triangle:
(3 2)/3 5/3 1.67 (This does not match the number inside)
But let's continue with this pattern for the third triangle:
(12 + 1)/5 13/5 2.6
Again, this does not match any of the options.
Considering other possible operations:
1. For the first triangle:
(8 4)/2 = 16
2. For the second triangle:
(3 x 2)/3
3. For the third triangle:
(12 x 1)/5 5- 12/5 2.4
Again, this does not match any of options.
Given the information and the options provided, 1 am unable to find consistent pattern that applies to all the triangles.
Therefore, cannot determine the missing number from the given choices.
In this logical reasoning question, GPT-4V verifies its generated solution and retries twice. However, still fails to produce an answer that matches any of the options. Consequently. claims that it cannot determine the missing number from the given choices (highlighted in blue).
Figure 104:
100

---

Published as a conference paper at ICLR 2024
ID:
D
Question: 10 1 # ABCD 4, AB = AC, LCAB = -400 [U| LD M ( (In the figure. within ABCD, AB AC. and LCAB 40", then the measure of / is 0.)
Tyeo COR
Choices: (A) 40 (B) 50" (C) 60 (D) 70"
Ground truth: (D) 70
GPT-4V:
क AB AC AABC E T ZCAB = 40 B LCBA 1 40
# - A $
(Gven AB AC, AABC is isosceles triangle. Since ZCAB 40", LCBA is also 40 (because in
an isosceles triangle, the two base angles are equal).)
# ABCD. K 360 E LCAB = 10" LCBA = 100, #R LBCA 4y E म
LD
(Considering the quadritateral ABCD, its interior angles up to 360". We know that LCAB 40' /CBA= 40', and / /CAA and /D supplementary.)
# B LBCA M 1 ;:: (We can find the angle of LBCA A as:)
/BCA- 180 40
-40
-100
#T LBCA LD A RIff:
(As LBCA and LD supplementary, have:)
LD 180 100 80
DI :
(But this isn't given option. let's recheck.)
| ABCA _ ルル
ट 4 Bi (Notice that LBCA is an exterior angle. which equals the sum of two non non- adjacent interior angles. So:)
LBCA= LCAB + LCBA
LBCA 10 + 40 = 80
1F LD) 180 80 = 100 1
(Thus. 2D> = 180 80 = 100". But this isn't an option either.)
$ #
(Wait, seem to have thought wrong.)
LBCA $ LBDA ft 7 ABCD L
1 # 180
(LBCA and LBDA are supplementary because ABCD is quadrilateral, so their sum is 180%.)
ji,, LBDA= 180 80"
100",
f LD E/ LBDA मा LCBA $
(Therefore, /BDA = 180 80 = 100%, hut /D is difference hetween /BDA and /CBA)
MTC (So)
/D 100
40
60
Figure 105: The question is presented in Chinese, with the translated English provided in paren- theses after the Chinese text. GPT-4V is capable of performing self-verification until produces a solution that matches one of the given options (highlighted in blue).
101

---

Published as a conference paper at ICLR 2024
ID: 44
oval beads
$2/kilogram
rectangular beads
$3/kllogram
Question:
Chase wants to buy 4 kilograms of oval beads
and kilograms of star-shaped heads. How much will he spend? (Unic: $)
star-shaped beads
$2/kilogram
spherical beads
$2/kilogram
heart-shaped beads
$3/kilogram
Ground truth: 18
square beads
$2/kllogram
Solution: 4 2 10, 3 12
flower-shaped beads
$2/kllogram
Multimodal Bard:
The cost of kilograms of oval beads is 4 2 $8.
The cost of kilograms star-shaped beads 5 2 810.
So the total cost is 88 + $10 = 818.
Therefore, Chase will spend $18.
also wrote a Python code to calculate the total cost and got the same answer 35. def calculate_come (oval_bsa Loval_beads, star_shaped_beadss
oval_bead__om
star_ star_shapea_beat cost 2
total_cost oval oval_beads oval_bead_cost star_shaped_bead
star_shtee Shar_shaped__ htt__shhaedeeedddd_eeee bead_
return total_cost
print (calculate_cmm cost 5))
35
Figure 106: Multimodal Bard seems incapable of verifying its solutions and revisiting them if found invalid. For instance, in this example, after initially producing a natural language solution, Mul- timodal Bard subsequently generates Python code. However, the executed result of the code (35) differs from its initial solution (18).
102

---

Published as a conference paper at ICLR 2024
6.6 SELF-CONSISTENCE FOR GPT-4V
Self-consistency (Wang et al., 2022) is a decoding strategy for chain-of-thought prompting (Wei al., 2022b). diverse set of reasoning paths is sampled, and the most consistent answer lected as the final prediction. Moving beyond vanilla greedy decoding, this method resorts to the inherent coherence and reliability of multiple reasoning trajectories to produce more trustworthy conclusion. Sclf-consistency has been widely employed in LLMs for complex reasoning tasks, such as math word problems and commonsense reasoning.
In our experiments, we validated the effectiveness of using self-consistency for GPT-4V on MATH- VISTA. Given a question and context, we ran GPT-4V multiple times to obtain set of different reasoning paths and then selected the most frequent answer the final prediction. We found that self-consistency is instrumental in rectifying visual perception errors (Figure 107), correcting cal- culation mistakes (Figure 108), and mitigating hallucinations (Figure 109). In comparison, self- consistency less effective when GPT-4V has difficulties in interpreting complex visual contexts (Figures 110, 111) extracting salient information from images (Figure 112).
on WwW.SCOOOOOOO.ooo
ID: 648
Question: What the minimum value this function?
35
Ground truth: -1
-3
2
2
GPT-4V (Reasoning Path 1):
The minimum value the function is observed where the graph is at its lowest point along the y-axis.
From the graph, the minimum value lics on the y-axis between -0.5 and 0. Based on the visual representa- tion, it appears that the minimum value of y is: -0.5.
So. the answer -055 Wrong output: -0.5
GPT-4V (Reasoning Path 2):
The graph shown represents a function with circular data points plotted across the z-axis. To determine the minimum value of the function, need to locate the lowest point the graph.
From the graph, it evident that the lowest point (or the minimum value the function) is at cocordiieate of approximately -1.
Therefore, the minimum value of the function is -1.
Correct output:
GPT-4V (Reasoning Path 3):
The graph depicts the function using circular points. The minimum value of the function can be observed by identifying the lowest point on the graph.
From the graph, the lowest point seems at y = -1.
Therefore, the minimum value of this function is -1. Correct output: -1
Figure 107:
In this algebraic reasoning task, GPT-4V displays inconsistency recognizing the value range of the function. After reevaluating GPT-4V three times, three distinct reasoning paths emerge. Utilizing caff-consistency minimizes variation in the generated results, converging on the most frequent prediction (-1) as the final answer.
103

---

Published as a conference paper at ICLR 2024
ID: 331
Web pages visited
Person
Number of Web pages
Several people compared how many Web pages they had visited. What is the mean of the numbers?
Question:
Bonnie
84
Winston
56
Ground truth: 64
Percy
70
Derek
68
Jake
Carter
Kurt
52
GPT-4V (Reasoning Path I):
To find the mean of the numbers, you add all the numbers and then divide by the quantity of numbers.
Mean (84 56 + 70 68 + 47 52) 8
162 8
57.75
However, since the question requires integer answer, the mean number of web pages visited is: 58.
Wrong output: 58
GPT-4V (Reasoning Path 2):
To find the mean number of web pages visited:
First, sum the number of web pages for all the individuals:
Bonnie: 84, Winston: 56, Perey: 70, Derek: 68, Jake: 60, Finn: 47, Carter: 75, Kurt: 52
Total = 84 + 56 + 70 + 68 + 60 + 47 75 52 = 512
Next, divide by the number of individuals (8):
Mean 512 : 8 = 64
The number of web pages visited is 64.
Correct output: 64
GPT-4V (Reasoning Path 3):
First, I'll list the number of web pages each person visited:
Bonnie: 84, Winston: 56, Percy: 70, Derek: 68, Jake: 60, Finn: 47, Carter: 75, Kurt: 52
Next, I'll group some numbers together for casier addition:
(84 + 56) 140
(70 + 68) 138
(60 + 47) 107
(75 + 52) 127
Now, I'll sum these results:
140 + 138 + 107
512
Now, I'll divide the sum by the number individuals to get the mean:
512 8 64
So, the mean number web pages visited is 64. Correct output: 64
Figure 108: For this arithmetic reasoning question, GPT-4V arrives at the answers of 58, 64, and 64 through three distinct reasoning paths. Applying self-consistency would confirm the correct answer be 64.
104

---

Published as a conference paper at ICLR 2024
10
Question: The derivative of f(x) 0 is that at z
Choices: (A) larger (B) equal (C) smaller than
Ground truth: (C) smaller than
10
GPT-4V (Reasoning Path I):
The graph represents the function (xx) 12x 3|
1.
The slope (or the derivative) of f(x) is represented by the inclination of the line.
At x 0, the slope is negative (the line is declining), and at i 5. the slope is positive (the line is inclining). The magnitude of the slopes appears to be the same; only their directions differ.
However, when considering the magnitude of the derivative, the value is negative, and the value x is positive, but they have the same absolute value. Wrong output: (B) equal
GPT-4V (Reasoning Path 2): The given function f(x = 22x 3| + 1.
The absolute value function piecewise defined, so derivative have different values depending on where you take
For 2x 3 > O, which is æ > or 1.5. f(x) 2x 3. The derivative is f'(x) 2.
For 2z < O, which < or 1.5, f(x) -2x + The derivative is fa(x) =22.
At = 0, the derivative f'(x) = -2.
At the derivative fz(x) =2
Therefore, derivative f() at 0 smaller than that at J.
Correct output: (C) smaller than
GPT-4V (Reasoning Path 3): The function provided f(z 12r
The absolute value function has a V-shaped graph, as shown in image. The slope (or derivative) of the line to the left of the vertex is negative, and the slope of the line to the right of the vertex is positive.
The derivative of fxx negative (downward slope). The derivative f(x) side vertex, which is positive (upward slope).
0 corresponds the slope of the line the left side of the vertex, which is corresponds the slope the line on the right
Since a positive value is always larger than negative value, the derivative of f(x) at 0 is smaller than that at 5.
Correct output: (C) smaller than
Figure 109: This is another instance where the application of self-consistency across three reasoning paths results in correct final prediction.
105

---

Published as a conference paper at ICLR 2024
ID: 157
Question: What number is shown?
Ground truth: 9335
GPT-4V (Reasoning Path 1):
Let's break down number shown using base-ten blocks:
There are 6 large cubes, each representing 1, 000. So, x 1, 000 6, 000.
There are 7 flats, each representing 100. So. 7 x 100 = 700.
There are 4 rods, each representing 10. So, 4 10 40.
There are 3 units, each representing 1. So, 3
3.
Adding all these values together:
6. 000 + 700 + 40 + 3 6,773
The number shown is 6, 743. Wrong output: 6743
GPT-4V (Reasoning Path 2):
The image displays series of cubes that appear represent numbers in base 10 using place value.
From left to right:
1. There are six large cubes, which represent value of 6 x 1000 = 6000.
2. There are three medium-sized cubes, which represent value of 3 100 300. small cubes, which represent value of 2 10 20.
3. There
4. The vertical stack single cubes shows nine cubes, which represent a value 9.
Adding up these values:
6000 + 300 20 + 6329
The number shown is 6329.
Wrong output: 6329
GPT-4V (Reasoning Path 3):
The image displays series of cubes representing numbers in base 10 using place value.
From left to right:
1. There are seven large cubes, which represent value of x 1000 7000.
2. There are two medium-sized cubes, which represent value of 2 100 200.
3. There nine small cubes, which represent value of
90.
4. The vertical stack single cubes shows eight cubes, which represent a value 8.
Adding up these values:
7000 + 200 90 + 8 7298
The number shown is 7298. Wrong output: 7298
Figure 110:
In this arithmetic reasoning task, GPT-4V struggles with comprehending the abstract scene, leading to its inability to calculate the correct answer through multiplication. Multiple at- tempts yield varying incorrect solutions. Consequently, utilizing selffconssstency does not verify the definitive right answer.
106

---

Published as a conference paper at ICLR 2024
ID: 551
316
Question: What is the difference be- tween the highest tattoos in male and the least in female?
2/76
Ground truth: 14 (%)
10%
Solution:
27% (male in 2015)
13% (female in 2008)
14%
Male
Tesealo
GPT-4V (Reasoning Path I):
The highest percentage of tattoos in males is 31 % (in 2015), and the least percentage of tattoos in females is 15% (in 2008).
Difference = 31% 15% = 16%
Wrong output: 16%
GPT-4V (Reasoning Path
The highest percentage of males with least one tattoo is in 2015 31%.
The lowest percentage of females with least one tattoo in 2003, 2008, and 2012, all at 15%.
The difference between these values is: 31% 15% 16%
Wrong output: 16%
GPT-4V (Reasoning Path 3):
To determine the difference between the highest tattoos in males and the least in females:
Highest tattoos in males: 31% (in 2015)
Least tattoos females: 15% (in 2003 and 2008)
Difference: 31% 15% 16%
Wrong output: 16%
59%
399
20N
20W
1.%
100
2005
Prompt: What is the percentage of tattons in females in 2008?
GPT-4V: The percentage of tattoos in females 2008 is 15%.
Prompt: What is the percentage tattoos in males 2015?
GPT-4V: The percentage of tattoos in males in 2015 31%.
Figure 111: In this instance, GPT-4V persistently produces incorrect predictions across multiple evaluations. When prompted to detail the essential elements of the image, GPT-4V provides inac- curate descriptions. This suggests that if GPT-4V misinterprets vital visual cues, relying on self- consistency may not necessarily guide towards the correct conclusion.
107

---

Published as a conference paper at ICLR 2024
Question:
What's the lowest number yard
line that you see?
Ground truth:
GPT-4V (Reasoning Path 1):
The lowest number yard line visible in the image the 10-yard line.
GPT-4V (Reasoning Path 2):
From the image provided, the lowest number yard line that is clearly visible the "10" yard line.
GPT-4V (Reasoning Path 3):
From the image provided, the lowest number yard line that be is the 10-yard line.
GPT-4V (Reasoning Path 4):
From the image, the lowest number yard line that is visible is the "I" yard line.
GPT-4V (Reasoning Path 5)
From the image, the lowest number yard line that is visible is the 20-yard line.
Prompt: List the yard line numbers that visible in the image.
GPT-4V: The visible yard line numbers in the image are 10, 20, 30.
Prompt: Can you see the yard numbers 40 and 50 in the image?
GPT-4V; No, the yard numbers 40 and 50 are not visible in the image.
Figure 112: This example demonstrates that if GPT-4V misinterprets key information from an im- age, the application of self-consistency may not aid in producing the correct final prediction. We evaluated GPT-4V five times for the question "What's the lowest number yard line that you can see?" and received consistently incorrect responses. Further, when prompted to describe the yard line numbers present in the image, GPT-4V provided incorrect descriptions, indicating its difficulty in accurately recognizing them.
108

---

Published as a conference paper at ICLR 2024
H.7 GPT-4V FOR MULTI- HUMAN-AI INTERACTION
This section investigates the use of GPT-4V for multi-turn human-Al interaction on MATIVVSSA, exemplified in the goal-directed dialog in Figure 113.
We found that GPT-4V is effective engaging multi-turn goal-directed conversations with users. In particular, GPT-4V can make good use of hints (e.g., user feedback responses) to guide the conversion to generate desirable results. For instance, it can (I) rectify visual perception errors based on hints (Figure 114), (2) reassess reasoning steps and calculations (Figure 115), (3) correct misinformation using user-provided domain-spectien knowledge (Figure 1 116), and (4) aggregate intricate contexts over multiple turns in human-Al conversation (Figures 117 and 118).
We also observed failure cases in our evaluation. For instance, GPT-4V struggles to generate correct responses when questions and user hints are ambiguous (Figure 119), or when the model fails to understand abstract shapes and concepts visually (Figure 120). These failures motivate the develop- ment of more powerful, conversational foundation models.
sour apple candy
$0.06
piece of gum
$0.07
gummy worm
$0.09
lemon drop
$0.05
piece licorice
butterscotch candy
$0.07
How much money dloe Luca need buy a sour apple candy and butterscotch candy? (Unit: $)
The total cost for sour apple candy and butterscotch candy is $0.13.
Luca would need at least $0.13 to purchase both items.
Why?
The cost for each item $0.06 for the sour apple candy and $0.07 for the butterscotch candy. Adding the two together gives us total cost of $0.13. Hence, Luca would need at least $0.13 to purchase both items.
Figure 113: A multi-turn dialog between human user and GPT-4V.
109

---

Published as a conference paper at ICLR 2024
How many baseballs are there?
Let's count baseballs each bucket:
First bucket: baseballs
2. Second bucket: baseballs
3. Third bucket: baseballs
4.
Fourth bucket: 6 baseballs
Total baseballs: 5 6 6 = 23 baseballs.
Answer: 23.
Your solution is wrong. You counted the baseballs in each bucket incorrectly.
apologize for the oversight. Let's try counting again:
1. First bucket: 5 baseballs
2. Second bucket: baseballs
3. Third bucket: baseballs
4. Fourth bucket: 5 baseballs
Total baseballs: 5 + 5 5 20 baseballs.
Answer: 20.
Figure 114: GPT-4V initially produces incorrect solution due errors in visually perceiving the number of baseballs. After receiving user feedback, it recalibrates and generates the accurate answer.
110

---

Published as a conference paper at ICLR 2024
13 57 911 1315 1719
Which number comes next?
The pattern seems to involve prime numbers.
Let's break down:
1 3 are the first second prime numbers.
5 are the third and fourth prime numbers.
9 (not a prime) but followed by which fifth prime number. the sixth prime number and (not prime) followed by which seventh prime number. the eighth prime number.
Following this pattern,
prime numbers are and 29.
So, the next number sequence would be 2329.
Your solution is incorrect. The pattern does not involve prime numbers.
apologize for the oversight. Let's re-examine the pattern.
It seems the pattern is derived by concatenating consecutive numbers:
1 and 3 give 13.
5 and 7 give 57.
9 and give 911.
13 and 15 give 1315. and 19 give 1719.
Following this pattern, the next numbers to be concatenated would be 21 and 23.
So, the next number sequence would be 2123.
Figure 115: GPT-4V correctly identifies the sequence of numbers in the images but struggles to discern the underlying pattern. With feedback from the user, GPT-4V re-evaluates the pattern and produces the correct response to the logical reasoning question.
111

---

Published as a conference paper at ICLR 2024
Sample
Sample
Mass of each perticle: 44
Mass of each particle: 46
Average particle speed: 1,400 mis The diagrams below show two pure samples of gas in identical closed, rigid containers. Each colored ball represents one gas particle. Both samples have the same number of particles. Compare the average kinetic energies of the particles each sample. Which sample has the higher temperature?
Average particle speed: 1,400 m/s
Choices: sample (B) sample
neither; the samples
temperature
The average kinelic energy ideal gas particle is directly proportional the temperature of the gas and is given by the equation:
KE
2
Where:
KEang the average kinetic energy per molecule.
the Boltzmann constant.
i' the absolute temperature.
For gases at the same temperature and under the same conditions, the average kinetic energy is the same, irrespective of the mass of the particles.
Given that the average particle speed in both Sample A and Sample is 1,400 m/s, and since they are in identical closed, rigid containers, implies that both samples have same average kinetic energy. Therefore, they also have same temperature.
The correct
(C) neither: samples have the temperature.
Your solution is incorrect. am providing retrieved knowledge to help you:
The temperature substance depends on average kinetic energy of the particles in substance. The higher the average kinetic energy of particles, the higher the temperature of the substance. The kinetic energy particle determined by its mass and speed. For pure substance, the greater the of each particle in the substance and the higher the average speed of the particles, the higher their average kinetic energy.
apologize for the oversight. Let's reconsider the question.
The kinetic energy (KE) of a particle is given by the equation:
KF
mu
Where:
m is the mass of the particle.
is the velocity (or speed) of the particle.
Given:
Both Sample A Sample have particles with same average speed (1.400 m/s).
The of each particle in Sample A 44 and Sample B 46
Given that the speeds the same but the mass of the particles in Sample is greater than that of Sample A, the particles in Sample will have higher kinetic energy than those in Sample A. As the temperature is directly related to the average kinetic energy of the particles, Sample will have higher temperature than Sample A.
Therefore, the correct answer is: (B) sample
Figure 116: Initially, GPT-4V applies an incorrect theorem to address the scientific reasoning query. However, after receiving the appropriate domain knowledge hint from the user, it utilizes the correct theorem, combined with key visual cues from the images, to produce the right answer.
112

---

Published as a conference paper at ICLR 2024
Table AgentBench evaluates API-based open-sourced L.Mss A.-M-as-Agent challenges.
Size
Version
Creator
gpt-4 [56]
undisclosed undiscloned
api
0613
gpt-3. 3pt-3.6-turbo [55]
0613
OpenA1
text Sost-ddvinci--00 (58)
undisclosed
coxt-daviaci-002 [58)
undisclosed
claudo-instant [6]
undisclosed undisclowed undisclosed
api
api
Anthropic
text-biaons-01 [3]
api
Google
chatglm2 [100: 21
undisclosed 6B
v0.2 vI.1
chatglmm6b [100: 21]
Tsinghua & Zhipa
codegeex2-Gt-chat [102]
open
vicuna-7b [14]
vicuma-13b [14]
13B
7B
open
open
v1.1
v1.1
LMSYS
11ama2 cha2--cht-78 [80]
11ama2-chats -chat-1bb [80]
138
7B
Meta
openchat-81.92-1 [85]
openchat--133 [85]
open
open
Tsinghua
vizardin-30t [93]
30B
v1.0
kizardcader- 15b [48]
15B
open
v1.0
Microsoft
vizardIn-130b [93]
13B
v1.0
koala-13b [26]
open
UC Berkeley
interoln-chat-7b [79]
7B
opon
InterLMM Team
baichuan- 13b-chhtt [36]
13B
Baichuan Inc.
baichuan-7b [36]
dolly-12b [16]
open
Databricks
casst-sfte --t---ft-4---- 12b [42]
12B
LAION
How many models table have model size larger than 10B?
From the table:
Models with larger than 10B:
1. chatgim2 [100; 21]
2. Aicuna-13b [14]
3. vicuna-7b [14]
4. 11ama2-chat-13b [80]
5. openchat-8192-138 [85]
6. openchat-13b [85]
7. wizardim-30b
8. wizardlm-133 [93]
9. koala-13b [26]
10. baichuan- 13b-chat Baichuan-13b-chat [36]
Total: 10 models.
Figure 117: Answering this question requires comprehending a complex table regarding Al models (such as identifying relevant and columns), and up-to-date knowledge of Al research, and the ability to perform advanced arithmetic reasoning. Without human in the loop, GPT-4V fails to produce the correct answer.
113

---

Published as a conference paper at ICLR 2024
You've located the wrong column. Please refer to the "Model Size" column to get the correct answer.
apologize for the oversight. Let's try again referring to the "Model Size* column:
From the table:
Models with size larger than 10B:
vicuna-13b [14]
t1ama2-chat-13b [80]
openchat-81 92-133 [85]
4. openchat-138 [85]
5. wizardim-30b [93]
6. mizardim-133 [93]
koala-13b [26]
8. baichuan- 130-chhtt [36]
Total: models.
No, the "Model Size' column the with values like "undisclosed" '6B", "13B".
apologize for the oversight. Let's count again by focusing the "Model Size" column:
From the table:
Models with size larger than 10B:
1. vicuna-13b [14]: 13B
11ama2-chat-133 [80]:
openchat-81118 192-119 [85]:
4. openchat-133 [85]: 13B
5. wizardim-30b [93]: 30B
6. wizardim-13b [93]: 13B
7. koala-13b [26]: 13B
8. baichuan- baichuan-13b-chat 13b-chat [36]: 13B
Total: 8 models.
You're almost there. You missed some models, like "wizardcoder-1 15b", and two others at the bottom of the table. Also, correct name for "11ama2" should be "Ilama2", pronounced as "L" "one".
apologize for the oversight. Let's count again:
From the table:
Models with size larger than 10B:
1. vicuna-13b [14]:
llama2-chat-133 13b [80]: 13B
openchat-81 8192-1331333 13b [85]: 13B
4. openchat-13b [85]: 13B
5. wizardlm-30b [93]: 30B
6. wizardcoder-155 (48]: 15B
7. wizardIm-135 [93]: 13B
koala-13b [26]: 138
baichuan-13b-chat [36): 13B
dolly-126 [16]: 12B
oasst-ft-4-12P [42): 12B
Total: 11 models.
Figure 11: (Continued) In this multi-turn human-AI conversation. the user acts as a tutor to help GPT-4V generate the correct answer. (1) Initially, the user directs GPT-4V the "Model Size" col- umn after observing its misdirection. Despite the guidance, GPT-4V still struggles. (2) Recognizing the model's 's diffieulty in identifying the "Model Size" column, the provides structions. This time, GPT-4V on the right track but still makes some error its response. (3) With further information from the user, GPT-4V corrects the remaining errors and generate the cor- rect answer.
detailed in-
114

---

Published as a conference paper at ICLR 2024
you add the visible numbers, on the jerseys, what is the total sum?
total sum of two visible numbers on jerseys is 29 (21 8).
Where number
apologize, can only the number 21 clearly the jerseys. best guess is that the other number is 8, but I cannot be certain due to the image quality.
If you add the two visible numbers, on the jerseys, what is the total sum?
Based image quality, my best answer that total sum the two visible numbers the jerseys 29 (21 8). But cannot certain about the number due the image quality.
Since you certain about let's ignore it. this case, if you add visible numbers, the jerseys, what is the total sum?
My best answer, excluding the uncertain number 8, the total sum of visible numbers the jerseys
Figure 119; A failure example from GPT-4V in an interactive chatbot environment. Utiiizing the visual from the image resolve the question's ambiguity (distinguishing between numbers, 2 and 1, instead of the single number 21), the correct answer is 3, obtained by summing 2 and I.
115

---

Published as a conference paper at ICLR 2024
How many Triangles do you see in the picture?
triangles picture.
Please think step by step.
Sure, let's break it down step by step:
Larger Triangles:
The entire figure itself triangle. that's triangle.
2. Medium-sized Triangles:
The three triangles formed by the division of the larger triangle into four parts. So that's triangles.
Smaller Triangles:
The triangles inside each medium-sized triangles. There are three medium-sizze sizeed triangles, and each of them contains smaller triangles. So that's 3 x 9 triangles.
Adding them up:
(larger) (medium-sized) (smaller) 13 triangles total.
So. there are triangles the picture.
Your solution is incorrect. There are four categories of triangles: triangles containing one, triangles containing two, triangles containing three, the largest one.
apologize for the oversight. Let's re-evaluate the triangles step by step based on your guidance:
Triangles containing only triangle:
These the smallest triangles. There of them.
2. Triangles containing two of the smallest triangles:
These are formed by combining two adjacent smallest triangles. There are of them (2 for each of the medium-sized triangles).
Triangles containing three the smallest triangles:
These are the medium-sized triangles. There are of them.
The largest triangle:
The entire figure itself. So that's triangle.
Adding them up:
9 (smalless) 6 (containing two) (containing three) (largest) 19 triangles.
So. there are 19 triangles in the picture.
Figure 120:
In this example, even after the user offers helpful hints, GPT-4V struggles to decipher the number of triangles in the given geometric shape. The task is challenging for Al models, includ- ing GPT-4V, with limited ability to understand abstract shapes visually.
116


Reviews:
[{"review_id": "joAYq7Bm9p", "reviewer": "Reviewer_juci", "summary": "This work introduces MathVista, a benchmark for evaluating the mathematical reasoning abilities of large language models (LLMs) and large multimodal models (LMMs) within visual contexts. The work uses data from a broad range of existing math and visual question-answering datasets and constructs three novel datasets: IQTest, FunctionQA, and PaperQA. The work evaluates nearly a dozen models with MathVista and finds that Multimodal Bard, the best-performing model, achieves 58% of human performance.", "questions": "While the paper covers a broad range of models,\n \nit is missing an analysis of why specific models perform better than others?\n\nand what features of each model contribute to performance? \n\nThis would be a valuable contribution.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Curating the datasets from existing sources and developing three new datasets contribute to comprehensive and diverse testing.\n\n2. Evaluating nearly a dozen models and comparing their performances with a human benchmark provides insight into current performance.\n\n3. Mathematical reasoning within visual contexts is an important field.", "weaknesses": "1. The results have been completely superseded by GPT-4 Vision, which together with GPT-4 are SotA in vision-language models.\n\n2. The methods have been superseded by recent prompting methods.\n\n3. The related work and references are lacking:\n\na. Prompting:\n+ Phenomenal yet puzzling: Testing inductive reasoning capabilities of language models with hypothesis refinement\nL. Qiu, L. Jiang, X. Lu, M. Sclar, V. Pyatkin, C. Bhagavatula, B. Wang, Y. Kim, Y. Choi, N. Dziri, X. Ren, 2023.\n+ Hypothesis search: Inductive reasoning with language models, R. Wang, E. Zelikman, G. Poesia, Y. Pu, N. Haber, N. D. Goodman, 2023.\n+ Large language model (LLM) as a system of multiple expert agents: An approach to solve the abstraction and reasoning corpus (ARC) Challenge, J. T. Min, M. Motani, 2023.\n\nb. GPT-4V:\n+ Lost in translation: When GPT-4V(ision) can’t see eye to eye with text, a vision-language-consistency analysis of VLLMs and beyond,\nX. Zhang, S. Li, Z. Wu, N. Shi, 2023.\n\nc. PoT:\nSolving Linear Algebra by program synthesis, I. Drori, N. Verma, November 2021.\nSolving Probability and Statistics problems by probabilistic program synthesis at human level and predicting solvability\nL. Tang, E. Ke, N. Singh, B. Feng, D. Austin, N. Verma, I. Drori, AIED, 2022.\nA neural network solves, explains, and generates university math problems by program synthesis and few-shot learning at human level\nI. Drori, S. Zhang, R. Shuttleworth, L. Tang, A. Lu, E. Ke, K. Liu, L. Chen, S. Tran, N. Cheng, R. Wang, N. Singh, T. L. Patti, J. Lynch, A. Shporer, N. Verma, E. Wu, G. Strang, PNAS, 2022.\n\n4. The paper lacks a comprehensive discussion of the limitations of the benchmark.\n\n5. The work is missing an analysis of why specific models perform better than others.\n\nI expect the author response to include GPT-4 Vision results, missing related work and references, and updated methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6XUiOrIzyU", "reviewer": "Reviewer_2daf", "summary": "The paper proposes a mathematical and visual reasoning benchmark to evaluate various visual-based mathematical skills of large-language models (LLMs) and large-multimodal models (LMMs). The authors introduce a taxonomy for mathematical visual reasoning which involves seven different types of mathematical reasoning scenarios and five associated tasks including figure-based question answering, math world problems and geometry problem solving. The visual scenarios are targeted to be diverse involving real images, synthetic scenes, charts and plots, scientific figures, etc. While majority of the benchmark is formed through 28 existing publicly-available datasets, authors form three new datasets targeted to fill gap for mathematical scenarios not covered by existing datasets. \n\nThe benchmark is relatively small and meant to be a zero-or few-shot evaluation benchmark divided into a \"testmini\" (1000 examples) and \"test\" (5141 examples). Evaluation of several model types (including LLMs, LLMs with visual context augmentation and large m) under different setups (including chain-of-thought, program-of-thought, few-shot) is performed and results indicate that current models perform poorly in comparison to humans. A brief error/success analysis and qualitative examples are provided for multimodal bard and context-augmented GPT-4.", "questions": "Please see the weaknesses section above (primarily points 1,2 and 3).", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The construction of a visual-based mathematical reasoning benchmark to evaluate current LLMs and LMMs is well motivated and relatively novel. The method for construction and statistics of the final benchmark are adequately described with appropriate references to source datasets and prior works.\n\n2. The identified taxonomy covers a broad range of mathematical reasoning scenarios and tasks, and diversity is also maintained in the visual contexts. Further, the 3 newly collected datasets consider new tasks not covered by past works. \n\n3. Evaluation is performed on prominent LLMs (such as GPT-4, ChatGPT, Claude) and LMMs (mPLUG-Owl, InstructBLIP, LLaVa, Multimodal-Bard). LLMs are evaluated in zero-shot, few-shot, chain-of-thought and program-of-thought settings and also when they are augmented with visual contexts. Further, human performance is computed and qualitative analysis and fine-grained result comparisons are performed to better highlight capabilities and limitations of existing models. \n\n4. Paper is generally well written with appropriate figures and details illustrating the dataset examples and analysis, performance breakdown, qualitative examples, model prompts/settings and annotation methods.", "weaknesses": "1. For data collection of the 3 new datasets, it is not clear if inter-annotation consistency checks were conducted and how the mentioned \"rigorous review process\" was conducted (details are missing).  \n\n2. Few-shot performance is computed only for LLMs and not LMMs. Given LMMs such as Multimodal-Bard, Flamingo/Open-Flamingo and mPLUG-Owl also support few-shot learning, these can also be evaluated few-shot to better evaluate the benchmark challenges. \n\n3. Further, LLMs can also be evaluated on a broader range of K-shot settings (currently only 2-shot is evaluated). Evaluation over {2,4,8,16,32} could provide better evidence of whether mathematical reasoning capabilities can be learned in a few-shot manner.\n\nRelatively minor:\n\n4. Benchmark is relatively small (6141 examples) and meant as an evaluation benchmark primarily drawn from existing datasets (5405 examples) with no finetuning subset which could be useful for improving mathematical reasoning capabilities of current models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FOzqvIkgNI", "reviewer": "Reviewer_Vb5K", "summary": "The paper proposes a benchmark to evaluate the ability of math reasoning with visual contexts. The test-only benchmark contains a relatively large number of examples (6k), featuring a wide coverage of math reasoning types (7), data sources (28), and task types (5). On the benchmark, representative multi-modal LLMs are evaluated. The results show that Bard is the best performing model among all the evaluated ones, getting 35% accuracy, which indicates that the current models still suffer from math reasoning.", "questions": "See weakness.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The dataset is well-designed and relatively large. It contains 6k examples, coming from various (28) sources including existing ones and newly collected ones. The datasets also evaluate various types of reasoning abilities and tasks. The images contain synthetic, real and text-heavy images, which is a good coverage of different types.\n2. The paper focuses on a specific topic and defines the problem well, which is important for LLM evaluation.\n3. Several SoTA multimodal LLMs, including miniGPT-4, LLaVA(R), Bard, Instruct-BLIP, etc. are evaluated on the benchmark, and show reasonable results.", "weaknesses": "1. How to disentangle the visual understanding ability or text understanding ability with the math reasoning ability? For example, if a model incorrectly answers “how many cars are to the left of the tree”, it could be the error in spatial understanding (failing to find the cars on the left), or error in counting, or the model cannot understand this question. In the current form of the dataset, there is no disentangled evaluation of the visual/text understanding versus math reasoning. A possible way to address this questions could be having the annotations for **rationales** (results of intermediate reasoning steps or sub-questions), and evaluate the model’s performance against the intermediate steps.\n2. Why is the human accuracy only 60.3% on the dataset? Does this suggest that the dataset is noisy, containing ambiguous/uncertain cases, or simply the task is very difficult thus humans are not good at it as well? An analysis on the 40% of the data where humans cannot answer correctly will be preferred. \n3. While it is good that the paper defines 7 types of fine-grained reasoning ability, what are the take-away messages (of Tab-2) that can be derived with the differences in the 7 types? It would be nice if the expertise of different models can be reflected using the fine-grained types, besides that Bard is the best model\n4. Results of GPT-4V? I understand that the model is not released by the submission deadline, but it would be good to have the results in later versions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nfmGh37qXR", "reviewer": "Reviewer_iJ4P", "summary": "The paper focuses on the evaluation of mathematical reasoning based on visual inputs. This work presents a review of existing work on the topic, a new benchmark (MathVista) made of existing datasets plus three new ones, and a large evaluation of existing large pretrained models on this benchmark.", "questions": "Please comment on W2 above.\n\nMinor question regarding the sampling of data for \"testmini\", the text mentions the following:\n\"The KL Divergence and Total Variation (TV) distance between the testmini set and the entire set are 0.008 and 0.035\"\nWhat is being compared with KL and TV distances? Distributions of what?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Thorough review of the existing work on the topic, with a proposed taxonomy that clarifies and organizes the capabilities and settings relevant to visual/mathematical reasoning.\n\n- Consolidation of existing datasets into a comprehensive benchmark.\n\n- Large evaluation of existing models, under various settings (zero-shot, few-shot ICL, with various prompting strategies).", "weaknesses": "W1. A potential downside of the chosen tasks is that they \"amalgamate\" mathematical and visual reasoning (as stated in the abstract). This does not seem desirable since one would usually also wants to understand the capabilities of a model for these two steps (visual understanding and reasoning) independently. The argument that there exists other benchmarks that do look at these individual capabilities means that this is however not a critical issue.\n\n------\n\nW2. The low human performance on the benchmark (~60% accuracy) is concerning. Could this indicate an issue with data quality of annotation noise? (rather than intrinsic task difficulty)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (oral)