Title: MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts
Benchmark: MathVista
Summary: The paper introduces MathVista, a benchmark for mathematical reasoning in visual contexts. This includes algebraic/arithmetic/geometric reasoning as well as interpreting functional plots and chart data. MathVista combines math questions from 28 existing multimodal datasets, plus 3 new datasets hand-annotated from internet sources.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Mathematical reasoning in visual contexts
Phenomenon defined by authors: Yes
Phenomenon definition: "We propose a task taxonomy to guide the development of MathVista: (1) we identify seven mathematical reasoning types: algebraic reasoning, arithmetic reasoning, geometry reasoning, logical reasoning, numeric common sense, scientific reasoning, and statistical reasoning... and (3) we encompass a diverse array of visual contexts, including natural images, geometry diagrams, abstract scenes, synthetic scenes, as well as various figures, charts, and plots" (2)
Definition scope: Comprehensive
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Widely-agreed
Purpose notes: The phenomenon and corresponding tasks are explicitly laid out in the introduction and very well-motivated, including detailed task definitions in the appendix.
Task definition: "we focus on five primary tasks: figure question answering (FQA), geometry problem solving (GPS), math word problem (MWP), textbook question answering (TQA), and visual question answering (VQA)" (2)
Task item definition: The core problem is an image (e.g. a functional chart or table), a question in text, optionally some number of multiple choice options, and a solution. Metadata includes the original benchmark category, task, type of visual context, grade level, and math skill tested.
Task definition detail: Explicit examples including their metadata are given in the main body text, which is very nice.
Task source: Human exam questions (e.g. GRE questions), Modified from another benchmark (e.g. translation into another language)
Task ecology: Representative task (e.g. answering medical licensing exam questions)
Dataset size: 5,141
Dataset size detail: Validation is called "testmini" and test itself is not released publicly
Dataset size notes: 1,000
Dataset metadata available: Yes
Dataset metadata detail: original benchmark category, task, type of visual context, grade level difficulty, type of mathematical skill
Sampling method: Random sample (creators defined a task space and sampled from it), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test, Validation
Response format: Multiple choice, Short free response (e.g. single word or number)
Metric definition: Exact Match (accuracy, F1, precision, recall), LLM post-processing (extracting answers, reformatting for automated scoring)
Metric aggregation: Simple Mean
Metric statistics: simple mean/sum
Metric subscores: Yes
Metric subscores detail: Task type
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: Yes
Authors discuss validity: Yes
Author validity detail: "One limitation is the dataset coverage. While MATHVISTA encompasses a broad spectrum of tasks and visual contexts, there may be gaps in the representation of certain types of mathematical prob- lems and visuals." (21) (presented in the appendix)
Taxonomy: Reasoning / Mathematics
New taxonomy: General Capability / Mathematical Reasoning