IsoBench: Benchmarking Multimodal Foundation Models on Isomorphic Representations
Authors: Deqing Fu
Ruohao Guo
Ghazal Khalighinejad
Ollie Liu
Bhuwan Dhingra
Dani Yogatama
Robin Jia
Willie Neiswanger
Venue: COLM (2024)
Keywords: large language models, vision language models, evaluation, isomorphism
TL;DR: We propose IsoBench to measure multimodal foundation models' performance on various problems with isomorphic representations

Abstract: Current foundation models exhibit impressive capabilities when prompted either with text only or with both image and text inputs. But do their capabilities change depending on the input modality? In this work, we propose **IsoBench**, a benchmark dataset containing problems from four major areas: math, science, algorithms, and games. Each example is presented with multiple **isomorphic representations** of inputs, such as visual, textual, and mathematical presentations. IsoBench provides fine-grained feedback to diagnose performance gaps caused by the form of the representation. Across various foundation models, we observe that on the same problem, models have a consistent preference towards textual representations. Most prominently, when evaluated on all IsoBench problems, Claude-3 Opus performs 28.66 points worse when provided with images instead of text; similarly, GPT-4 Turbo is 18.71 points worse and Gemini Pro is 14.87 points worse. Finally, we present two prompting techniques, *IsoCombination* and *IsoScratchPad*, which improve model performance by considering combinations of, and translations between, different input representations.

Published as a conference paper at COLM 2024
IsoBench: Benchmarking Multimodal Foundation Models on Isomorphic Representations
Deqing Fu
Ruohao Guo
Ghazal Khalighinejad
Ollie Liu
Bhuwan Dhingra Dani Yogatama
Robin Jia Willie Neiswanger
*USC
'Georgia Tech ADuke
Abstract
Current foundation models exhibit impressive capabilities when prompted either with text only or with both image and text inputs. But do their capa- bilities change depending the input modality? In this work, propose IsoBench, a benchmark dataset containing problems from four major areas: math, science, algorithms, and games. Each example is presented with multiple isomorphic representations inputs, such as visual, textual, and mathematical presentations. IsoBench provides fine-grained feedback to di- agnose performance gaps caused by the form of the representation. Across various foundation models, we observe that on the same problem, models have consistent preference towards textual representations, Most promi- nently, when evaluated on all IsoBench problems, Claude-3 Opus performs 28.7 points worse when provided with images instead of text; similarly, GPT-4 Turbo is 18.7 points worse and Gemini Pro is 14.9 points worse. Fi- nally, we present two prompting techniques, IsoCombination and IsoScratch- Pad, which improve model performance by considering combinations of, and translations between, different input representations. Our dataset and evaluation code can be found at https: //wobbecch..ccc/.oo/ooo.e. gi github.io/. io/.
Prompts
Responses
IsoBench
Instruction about Check Function
Math
Delintion of Funation
The plot the function flx) shows symmatry about the y-axis. This symmetry indicates that fix) f(ex) all the domain of the function, which is the definition of even function Therefore, based on the plat, fixt is an even function.
Image Representation
Substituting into the original function,
Representation
we get
Here the expression Axx):
A-d:
Si-r a 6 11.3
Curse
Nath
This expression not equal to the original function f(x), indicating that the function is not
fit 3
Algi thn
+x2+++++++++
Additionally. the the negative of the original function. indicating that the function not add. Thersfore, fiA) neither even nor odd.
Irstrustion Output Format
Do multimodal foundation models treat every modality equally? In this example, a model provided with either image representation text representation isomorphic to the image, where the instructions are kept identical. Surprisingly, multimodal models often give different responses for these isomorphic inputs (e.g., in the figure above, only the response to the text representation correct). In IsoBench, we scale such examples into four domains (Math, Science, Algorithmss, Games) and find consistent preference towards text across many popular multimodal foundation models.
Figure 1:
1 Introduction
On the heels of the large language model (LLM) revolution, we are currently witnessing second revolution of multimodal foundation models. Exemplified by models like GPT-
"Equal Contribution, alphabetcall order. [deqingfu, yogatama, robinjia, nei swang ( khalighinejjd, bhuwan. dhinaaradingaaa.e (hinggra@@duke.. edu, rguo488gatech. edu, me@ollien moollielin com
1

---

Published as a conference paper at COLM 2024
Mathematical Functions
Science Questions
Tasks Hecakpoints
Functioo Parity
Tasks Function Cumeeslt
Task:
Faskee Beemistry
Repeocesice
Repeocttake
Reproceniiin
Gumdey
light
cablo ansueliee secishatin L.C. Sond
Irage Reprosettttinn
brage Represerraion
Imoge Ropresorratioe
Iesuge Reprisettttion
Ireage Repecsoraation
Antwer
Tlnnckes
Answers Concive Punccios
Antwerdiit laepe
Asswaratt strrggee
Graph Algorithms
Chess Games
Task:
Connectivity
Graph bommophisan
Meentifictioon
Yous Sepee
Sep oop.occctio
Tou Repessmraooo
Text Reprost madic
www.elttaaataaaaoo.oooooo bxigkik kijiy
Auwer Beiu
Irage Reprosttatios
Insage Regrnccraatoonn
Aeomnz Cormatted
Noi Lonnoopiii
Auannan Black
Figure 2: IsoBench contains four major domains: Mathematical Functions, Science Questions, Graph Algorithms, and Chess Games. For each domain, there are two or three subtasks. All examples within IsoBench are provided with one image representation and several textual representations that are isomorphic to each other.
4V (OpenAI, 2023b), Claude (Anthropic, 2024), and Gemini (Google, 2023), these models combine large language model backbone with vision encoder that enables them to accept either pure text inputs or a combination of text and images. The rapid rise of these multimodal models necessitates new benchmarks that accurately assess their capabilities.
In this work, we study whether these models exhibit the same capabilities when processing text and image inputs. Our key insight is that many problems consisting of both image and text prompts can be equivalently formulated with text-only prompts. By studying how well models do on both representations of the same problem, we can evaluate whether the multimodal fusion components of these models truly empower the model with the same capabilities when reasoning about images and text. While many benchmarks already exist for testing multimodal foundation models (Yue et al., 2023; Lu et al., 2024; Zhang et al., 2024a; Fan et al., 2024; Zhang et al., 2024b), none of these measure models' performance discrepancies between semantically equivalent inputs in different modalities.
We create IsoBench, a broad-coverage benchmark for evaluating multimodal foundation models on 10 different tasks with multiple isomorphic representations. Isoßench includes four major subjects: mathematical functions, algorithmic problems, science questions, and chess games. For each test example, IsoBench provides one image representation and one or more alternative textual representations. We evaluate a suite of multimodal foundation models, including GPT-41, Gemini, and Claude-3, on IsoBench, and find that all tested models perform substantially better when given textual representations compared with visual representations. This bias in favor of textual representations runs counter to human cognition: humans are known to exhibit a picture superiority effect (Defeyter al., 2009), preference towards visual representations, and humans generate visual images internally regardless of whether the required task is visual or verbal (Amit al., 2017).
Finally, we design two simple mechanisms to further study the performance of multimodal foundation models on tasks with isomorphic representations: IsoCombination (IsoCB), which feeds multiple isomorphic representations to the model, and IsoScratchPad (IsoSP), which
'Throughout this paper, use GPT-4 general term when referring to both the GPT-4 large language model, and the GPT-4V multimodal foundation model.
2

---

Published as a conference paper at COLM 2024
GPT-4 Turbo
Gemini 1.0
Claude-3 Opus
Mixtral- 8x7B
Random Guess
Subject
Representation
Pro
Image
72.0
66.7
71.3
Science
38.3
Text
86.7
69.3
89.3
68.0
Image Text
46.9
36.4
41.7
Mathematics
44.4
76.6
73.9
85.8
66.1
Algorithms
Image
54.2
37.0
39.1
34.7
Text
69.5
43.8
73.7
49.5
Image
27.6
22.9
21.0
Games
18.1
Text
42.8
34.9
33.4
35.5
Image
50.2
40.7
43.3
Average
Text
68.9
55.6
72.0
57.2
33,9
Delta (Text - Image)
+18.7
+14.9
+28.7
Table 1: IsoBench results. Scores are evaluated as accuracy. Within each subject, the best scores in processing image representations are highlighted in blue, and the best text representations are in red. Overall, GPT-4 Turbo is the best for images and Claude-3 Opus the best for text. Across four subjects within IsoBench, multimodal foundation models have strong preference for text modalities. The gap in accuracy between the best text representation and its isomorphic image representation can be as large as 28.7%
uses scratchpad to translate between visual and text representations. We find that both methods improve aspects of multimodal models, with IsoCB improving model perfor- mance on graph algorithm problems by up to 9.4 points compared with the best single representation, and IsoSP improving performance on science problems by up to 14.4 points.
In full, we summarize our contributions follows:
We introduce IsoBench, test dataset consisting of 1,887 samples spanning diverse domains such as discrete and applied mathematics, physics, chemistry, and chess. For each sample, we evaluate multiple isomorphic input representations containing the same information-one with visual representation and others with domain-specifie textual representations-to facilitate multimodal performance assessments.
We benchmark eight popular foundation models, and find that each of the tested multimodal models perform substantially better on text-only prompts than image- based prompts, in contrast with known human preferences for images over text.
To bridge performance discrepancies between different input modalities, we introduce IsoComiination (IsoCB) and IsoScratchP ad (IsoSP), two procedures that, respectively, fuse input modalities and transform visual input into textual representation at inference time. In certain settings, we find that IsoCB and IsoSP can improve the performance of multimodal foundation models by nearly 10 percentage points.
IsoBench
IsoBench collection of problems of the form (P,RRi.... RRm H), where P a natural language instruction of task, and R, are different representations of the sample. Ideally, these representations are isomorphic to each other, i.e. for any pairs of input R; R,, there exists a bijective function & such that (RR,) Ry. As illustrated in Figure 1, the prompts to foundation models share the same instructions except for the problem descriptions (where one of the representations R, is selected). We now detail the construction of IsoBench.
currently consists of 1, 1,887 examples across four domains, though the construction larger dataset is currently underway.
2The IsoBench dataset (www. huggingf seginnggace.... ace. hwtpt/sttttssssescc///sssa
3

---

Published as a conference paper at COLM 2024
GPT-4 Turbo
GPT-3.5 Turbo
Mixtral Mixtral-xx7B 8x7B
LLaMa-2-70B
LL 5-133B
Randcmm Guess
Gamin Pro
PaLM-2
Claude 3
Image
56.3
47.9
43.2
24.7
Even/Odd Function
77.1
36.7
49.0
35.2
73.7
36,2
33.3
33.3
Code
67.7
45.1
48.2
35.2
77.6
41.7
33.3
Image
74.8
54.3
49.6
35.2
Convex/ Convex/Concave Concave
65.6
85.9
85.2
63.7
91.4
67.2
67.6
50.0
Function
Code
66.02
89.1
87.9
67.6
93.0
69.1
74.2
Image
5.9
1.2
31.7
50.0
Count Breakpoints
86.7
100.0
100.0
100.0
98.4
100.0
0.0
50.0
Code
96.1
100.0
100.0
100.0
90.2
99.6
0.0
Table 2: Benchmark results Mathematicss problems. We report accuracy scores. For each model, the best score of each task is highlighted in bold.
2.1 Mathematics
Our first evaluation suite consists of problems in continuous mathematics. Foundation mod- els are increasingly deployed as assistants for data science applications (Cheng et al., 2023), and they must perform competently on plot understanding to provide helpful insights. We evaluate this capability by prompting foundation models to extract mathematical properties from continuous functions.
Formats. We consider three input formats: Image, Text with BTEX, and Text with Code. We first sample candidate functions and convert them to their IsTpXand sympy representations. Then, we use matplotlib to plot the functions. In practice, it can be difficult to exactly map these images back to their corresponding functions, so the authors manually filtered data to ensure that the properties to be tested are clearly visible in the images.
Tasks. We consider three tasks: classifying parity (i.e. eoen/odd/neithhr) functions, con- vex/concave functions, as well as counting the number of breakpoints in piecewise linear functions (either 2 or 3). These problems designed to be visually trivial for humans to solve, but requires some mathematical maturity to answer analytically. We sample 128 samples for each class, culminating 896 total samples. We refer readers Appendix A.1 for additional details and Appendix for sample prompts and responses.
2.2 Games
A critical area for enhancing the strategic capabilities of foundation models lies in under- standing game strategies. We focus on two tasks in the game of chess: winner identification and chess puzzle solving. For both tasks, we use data from the Lichess database?.
Formats. For chess game dataset, consider four representations: Graphical Board (Image), Algebraic Notation Layout (ANL), Portable Game Notation (PGN), and Forsyth- Edwards Notation (FEN). The Graphical Board is an image of the chessboard, whereas ANL, PGN, and FEN are all text-based representations of the game state. For additional details, please see Appendix A.3.
Tasks. We consider two tasks: winner identification from final board states and chess puzzles. In the winner identification task, the objective is to analyze a given representation of a chess game and determine the outcome: whether the game resulted in win for White, win for Black, or a draw. Our dataset for this task includes 257 games that lasted more than nine rounds in February 2024. In the chess puzzles task, a chess puzzle is given and the objective is to find the best first move. The dataset for this task comprises 200 puzzles.
parttss//lichhss.

---

Published as a conference paper at COLM 2024
GPT-4 Turbo
GPT-3.5 Turbo
Mixtral Bx77 Mi77
LLaMa-2-70B
LLaVa-15-130
Random Gures
Gemini Pro
PaLM-2
Claude
Image
54.3
41.5
Winner
ANL
50.4
45.7
27.5
44.6
60.1
40.7
25.4
Identification
33.3
PGN
85.7
52.3
69.8
65.1
66.3
70.5
53.7
FEN
50.4
31.4
8.1
8.9
74.4
7.4
12.4
Image
1.0
0.0
0.5
0.0
Chess Puzzles
ANI.
1.0
0.0
0.0
0.5
1.0
0.0
0.5
~2.
PGN
0.0
0.0
0.0
0.5
0.5
0.0
0.5
FEN
1.0
0.0
0.5
Table 3: Benchmark results chess Games problems. We report accuracy scores. For each model, the best score each task is highlighted in bold.
2.3 Algorithms
To provide concrete planning advice, foundation models need to reason algorithmically. We focus on graph algorithms, which are relevant in many realistic scenarios. For example, if asked to purchase flight tickets from Los Angeles Vienna, foundation models need to (1) enumerate all possible choices with possible connection stops in between, and (2) check the lowest ticket prices among all choices. Fundamentally, the first task is a graph connectivity problem and the second is weighted shorfest path problem.
In this section, we investigate three main graph algorithm problems with increasing com- plexity: graph connectivity, maximum flow, and graph isomorphism.
Formats. For graph algorithms, we consider three representations: 1. Image, where graphs visualized with networkx package with random styles; 2. Text with ETEX, where the adjacency matrix is chosen to be the mathematical representation of graphs; 3. Text with story or description, where the graph problems are described as a story-telling version of the scenario (e.g,, formulating the graph connectivity problem as determining whether it is possible to drive from one city to another).
Tasks. We consider three tasks. Graph Connectivity requires deciding whether two query nodes are connected in an undirected graph. Maximum Flow requires computing the maxi- mum flow one send from given source node to given sink node, within weighed directed graph. Finally, Graph Isomorphism requires determining whether two provided graphs with the same number of nodes are isomorphic (i.e., whether there exists a bijection between nodes that preserves all edges). Each task contains 128 problems with details in Appendix A.2.
2.4 Science
Formats. We consider two input formats for science questions: Image and Text. For the Image representation, each sample includes textual question and multiple choices, along with figure that provides additional context. To get the corresponding isomorphic Text inputs, one author manually wrote descriptions for each figure.' The annotator avoided introducing any extra reasoning or information beyond what is depicted in the figures. Isomorphic examples of image and text representation are provided in Appendix Table 7.
Tasks. We compiled dataset consisting of 75 Chemistry and 75 Physics questions. We first selected questions from the ScienceQA (Lu et al., 2022) test set, choosing only questions
*In preliminary experiments, we leveraged GPT-4 to generate image descriptions. However, we observed that these descriptions often include reasoning and additional knowledge, leading to biased "improved" performance.

---

Published as a conference paper at COLM 2024
GPT-4 Turbo
GPT-3.5 Turbo
Mixtral-Baa -ByBB
LLaMa-2-70B
LLa
Randon Gaess
Gemini Pro
PaLM-2
Claude
Image
75.8
50.8
53.1
50.0
Graph Connectivity
Math
80.5
53.1
50.0
53.9
82.0
62.5
50.0
50.0
Text
95.1
78.1
75.0
85.9
94.5
84.4
50.0
Image
36.7
13.3
12.5
2.3
Maximum Flow
Math
32.8
25.8
15.6
14.1
56.3
8.6
18.0
4-40
Text
20.3
19.5
45.3
73.4
Image
30.0
46.9
51.6
50.0
Graph
Isomorphism
Math
62.5
49.2
47.7
50.8
50.0
50.0
50.0
Text
57.0
50.0
36.7
50.0
53.1
50.0
50.0
Table 4: Benchmark results on graph Algorithuss problems. For connectivity and isomor- phism, binary classification accuracy is reported. For Maxflow, an exact match accuracy of total flows reported. For each model, the best score of each task highlighted in bold.
GPT-4 Turbo
GPT-3.5 Turbo
Mixtral-8xDh 8x7B
LLaMa-2-70B
LLaVa-1.5-13B
Random Guess
Genini Pro
PaLM-2
Claude
Image
74.7
61.3
70.7
42.7
Physics
38.3
Text
86.7
84.0
69.3
76.0
89.3
68.0
61.3
Chemistry
Image 69.3
72.0
72.0
45.3
38.3
Text
94.7
76.0
85.3
84.0
90.7
65.3
73.3
Table 5: Benchmark results on Science problems. We report accuracy scores. For each model, the best score of each task is highlighted in bold.
whose answers rely on understanding the corresponding figures. This process resulted in total of 50 chemistry and 60 physics questions. To enhance the diversity of the science dataset, we manually added 25 new chemistry questions and 15 new physics questions, introducing three new categories distinct from those in the ScienceQA dataset. The category distribution of our entire science dataset is detailed in Appendix Figure 4, illustrating the wide range of science questions covered.
3 Performance Analysis
We evaluate IsoBench on APl-access multimodal foundation models, such GPT-4V (gpt-4-0125-preview and spt-4-vision-previeww Claude 3 (claude-3-opu--2222222)) and Gemini Pro 1omminii110-prrr and gemini-pro-vision)) We also benchmark API- access single-modal large language models for reference, such as GPT-3.5 Turbo (OpenAI, 2023a, gpt-3. 3pt-3.5-turtee 5-turbo-0125) and PaLM-2 (Anil al., 2023, text-bison-001), We evaluted representative open-source models such as Mixtral 8x7B (Jiang et al., 2024), LLaMa-2 70B (Touvron al., 2023), and LLaVA-1.5 7B (Liu et al., 2023a) as well. All models we evaluate are instruction-tuned models and are zero-shot prompted. We present fine-grained results on these tasks Tables 2 to and provide sample responses in Figures 17.
Overview. We observe that, across all models that admit multimodal representations, language-only prompts almost always outperform vision-language prompts. On language prompts, APi-access models exhibit strong reasoning capabilities (e.g. Figure 13). They generally outperform open-sourced models, which exhibit tendency to generate defaul!

---

Published as a conference paper at COLM 2024
answer regardless of input samples. We now delineate several intriguing observations. We present aggregated IsoBench results in Table 1 with details on more models in Table 8.
Vision models may be insufficient. In tasks that require explicit enumeration of objects such as breakpoint and chemistry (e.g. Figures 12 and 17), we observe significant perfor- mance degradation due to counting errors, which has been a long-standing problem for vision models (Liu et al., 2022). In fact, leveraging a sub-optimal feature extractor for one modality may hurt the performance of a multimodal model; and the design of an optimal model requires nuanced understanding of the interaction effects between modalities (Liang et at. 2024). Thomason et al. (2018) also showed unimodal models can even outperform their multimodal counterparts, and it suggests that models can often learn shortcut only relying on language model part and perform weaker when forced to use only images. Overall, visual recognition errors, such as Figures 6, 8 and 12, are prevalent in failure cases, and often misdirect natural language generation towards incorrect conclusions.
Models cannot utilize low-level visual features for generation. Perhaps surprisingly, vision-language foundation models are far from perfect for convexity problems, which mostly feature simple, smooth curves, even though curve detectors (Olah et al., 2020) are among the most observed and well-studied phenomena in the mechanistic understanding of vision models. This inconsistency suggests that, even if performing certain tasks is within a vision model's capabilities, the current multimodal fusion scheme is insufficient for eliciting them. We posit that popular approaches such as image-tokenization only offer coarse-grained representation that summarizes high-level features, and may be unsuitable for detailed analysis such as plot and chart understanding.
LLM backbones may not be exempt from blame. Recent works (Gonen et al., 2022; Razeghi et al., 2022; Han & Tsvetkov, 2022) have reported language models to prefer textual formats that are more common in the pre-training data, which echo our observation in performance variability between different language-only inputs. While formats do not fully explain this discrepancy, we hypothesize that the imbalance between visual and input data, and in general the sub-optimal robustness of LLMs in the face of less common distribution can contribute to the performance gap between visual and language inputs.
Lastly, we observe generated responses of vision-language prompts to be cursory compared to those of language-onil prompts; and they sometimes contain direct answer instead of a reasoning process, despite being instructed to perform chain-of-thought reasoning. To mitigate this difference, we discuss two simple strategies that aims to augment vision- language prompt with better reasoning capabilitiess of language prompts, presented next.
4 IsoCombination and IsoScratchPad
In many scientific fields, the images are complicated, researchers usually provide extra text inputs to help the model. This is also a common practice in scientific fields, where figures and tables in most papers are coupled with captions (for example, this paper), providing descriptions and takeaways. Motivated by these and based on the observations made above on IsoBench, we can design two simple and deliberately contrived mechanisms to further study the performance of multimodal foundation models on tasks with isomorphic representations: IsoCombination (IsoCB) and IsoScratchPad (IsoSP). IsoCB feeds multiple the multimodal model simultaneously in order to see the combined effect of using multiple representations on performance. Alternatively, IsoSP uses a scratchpad to explicitly translate from visual to text representations (and then uses this text representation to solve task), taking advantage of the empirically text representations for visual data. In particular, for IsoSP we employ a two-step prompting strategy: initially, the model receives prompt featuring a visual representation, which it is tasked to translate into a textual format; then, the model is prompted with this generated text representation to predict the output. We illustrate both methods in Figure 3.
isomorphic representations
We show results for IsoCB and IsoSP in Table 6, along with results for image-only and text-only representations. Interestingly, we find that IsoSP, which converts from image to
7

---

Published as a conference paper at COLM 2024
IsoCombination
IsoScratchPad
Instruction about the Task: Graph homorpphim
1 Instruction: Describe the Image
Model Scratch Pad
Definition of Graph hooerppism
Image Representation
The semvic light aotor
Image Representation
0
0
19
Text Representation
Inttaction about the Teak: Gruph bonorphiom
Model Final Response
Definition of Graph Isomorphism
M--
9
Model Scratch Pad
Chick
Text Representation 2
Saca dag ofd
Decepptionl O
There as diricias prigh.
Descriptic' Cooph
soleral
with plur
iemerraie day peroniffeee
Vode Di cornected Mede
and Morie
Node la coneead to Mode
Node readd Mode
Mock and Morke
Node con recid to Mode and
lapal
Instruction Output
Instruction Output
Figure Illustration of IsoCombination (IsoCB) and IsoScratchPad (IsoSP). IsoCB combines all representations provided by user and constructs one unified prompt for a foundation first describes an
model. IsoSP is two-step prompting method, where a foundation image and then uses the textual description the sole representation for given task.
Task
GPT-4 Turbo
Gemini 1.0 Pro
Claude-3 Opus
Image
36.7
13.3
12.5
Text (Best)
56.3 (+19.5)
19.5 (+6.3)
73.4 (+60.9)
Maxflow
IsoCB
65.6 (+28.9)
21.9 (+8.6)
75.0 (+-2..5
IsOSP
52.3 (+15.6)
19.5 (+6.3)
22.7 (+10.2)
Image
75.8
50.8
53.1
Text
95.1 (+19.3)
75.0 (+25.8)
94.5 (+41.4)
Connectivity
TsoCB
83.6 (47.8)
52.3 (+1.6)
85.2 (+32.0)
IsoSP
82.0 (+6.3)
51.6 (+0.8)
63.3 (+10.1)
Image
74.7
61.3
70.7
Text
86.7 (+12.0)
69.3 (+8.0)
89.3 (+18.7)
Physics QA
IsoCB
88.0 (+13.3)
70.7 (+9.3)
88.0 (+17.3)
IsoSP
84.0 (+9.3)
76.0 (+14.7)
78.7 (+8.0)
Image
69.3
72.0
72.0
Text
94.7 (+25.3)
85.3 (+13.3)
90.7 (+18.8)
Chemistry QA
IsoCB
92.0 (+22.7)
82.7 (+20.7)
89.3 (+27.3)
IsoSP
88.0 (+18.7)
84.0 (+12.00)
73.3 (+1.3)
Table 6: IsoCombination and IsoScratchPad results. Best methods are highlighted in red and improvements over image-only prompts are in (green). We find that both methods improve performance in comparison with image representations, and for certain domains, IsoCombination additionally improves performance relative to text representations
text, typically outperforms image-only representations (and thus it could potentially be used improve performance given only image representations for some tasks). However, compared to direct text representations, IsoSP tends to fall short. Our manual review of IsoSP and text results provides a possible explanation: the model has difficulty understanding images, especially in tasks that require counting items comparing numbers visually. The larger the performance gap between IsoSP and Text representations, the harder becomes for the model to understand images. Another possible reason for IsoSP's underperformance relative to Text representations could be that, the model-genert redee-ggeerrteed interpretation of images tends to miss critical information for reasoning. For example, to determine which pair of
8

---

Published as a conference paper at COLM 2024
magnets in the given image has larger magnetic force in physics question, it is essential to consider both the distance between the magnets and their sizes. Yet, the descriptions generated by models like Claude-3 recognize only the distance, neglecting the magnet sizes, and thus leading to an incorrect answer. Additionally, we find that IsoCB, which combines representations, typically outperforms image-only representations, and even text-only representations on certain tasks.
5 Related Works
Vision-Language Foundation Models. There has been a plethora of recent developments in vision-language foundation models; they can be broadly categorized by their methods for representing visual modalities. A representative approach involves tokenizing visual inputs to be jointly trained with language imputs (Yu al., 2023; Google, 2023; McKinzie et al., 2024; Chen et al., 2022; 2023; Team, 2024, inter alia). Another line of work processes continuous visual features by directly projecting them to the language embedding space via a learnable function (Liu et al., 2023b;a; 2024a; Bavishi et al., 2023). While the design choices of some APT-access models (OpenAI, 2023b; Anthropic, 2024; Google, 2023; Reka, 2024) remain largely unknown, most performant models use early fusion, the practice of integrating visual and language features at the input level of an autoregressive model. At the core of these design choices is the hardness in representing visual features, which has been reported by several early studies (McKinzie et al., 2024) to be the key bottleneck towards performant vision-language foundation models. IsoBench aims to supplement prior works, which have mostly focused on analyzing modeling methods, with a data-centric view by measuring performance discrepancies incurred by multimodal representations.
Multimodal Datasets. Many lision-language datasets have recently been curated to assess the capabilities of multimodal foundation models (Yue et al., 2023; Lu al., 2024; Zhang et al., 2024a; Fan et al., 2024; Zhang et al., 2024b). They often consist of large collection of problems with varying levels of difficulty and usage of the visual inputs. As a result, they are suitable for assessing VLMs as a whole, but cannot attribute these performances to capabilities of their LLM backbones and improvements from downstream vision-language training. While several contemporaneous works (Fan et al., 2024; Zhang et al., 2024a;b) introduce language-only subset of their vision-language prompts, they focus primarily on assessing certain areas of foundation models, such as deductive reasoning and problems of reyoond-polyoomial complexity. IsoBench is the first of its kind that offers a holistic and fine-grainer analysis of the performance gap induced by variations in input modalities. For instance, contemporaneous work, MathVerse (Zhang et al., 2024a), studies redundant, but not necessarily equivalent input representations on math problems whilst IsoBench studies the impact of isomorphic representations on a broader set of tasks.
Sensitivity of Language Model to Perturbations. Similar to what IsoBench observes on multimodal foundation models' sensitivity to input modalities, uni-modal language models also show sensitivity to input perturbation. For example, LLMs are shown to be sensitive to subtle changes in zero-shot and fhw-show settings (Sclar et al., 2023; Chang & Jia, 2023). Meanwhile, the sensitivity of Transformers, the backbone of foundation models, is also broadly studied (Bombari & Mondelli, 2024; Bhattamishra et al., 2023; Vasudeva et al., 2024; Kong et al., 2024). To address the sensitivity issues, methods including prompt design (Yoo et al., 2021; Mishra et al., 2021; Le al., 2023; Liu et al., 2023c) and promptingbbase training strategies (Jain et al., 2023; Guo et al., 2023) have been employed. Isoßench is one of the pioneers to study model sensitivity input representations and modalities.
Foundation Models for Algorithmic Reasoning Recent work has shown that Transform- ers, the backbone of multimodal foundation models, are capable of implementing various algorithms after fine-tuning on particular tasks (Khalighinejad et al., 2023; Garg et al., 2022; von Oswald et al., 2022; Hanna et al., 2023; Fu et al., 2023; Lee et al., 2023; Zhou et al., 2024) or with proper prompting schemes (Wei et al., 2023; Yao et al., 2023; Liu et al., 2024b). However, the extent to which foundation models are capable of solving problems involving complex reasoning, algorithms, scientific awareness remains unclear.

---

Published as a conference paper at COLM 2024
6 Conclusion
In this paper, we developed IsoBench, a broad-coverage benchmark for evaluating multi- modal foundation models on varied set of quantitative tasks, where each example input is provided with multiple isomorphic representations. IsoBench includes four major subjects (mathematical functions, algorithmic problems, problem,,prooblmmm, science questions, and chess games) dataset consisting of over 1,630 examples. Within each domain, we evaluated multiple input representations -ne with visual input and others with domain-specific, isomorphic textual inputs-to to facilitate multimodal performance assessments, We found that all tested models perform substantially better on text-only prompts than image-based prompts, in contrast with known human preferences for images over text. To bridge performance dis- crepancies between different input modalities, we also introduced IsoCombination and IsoScratchPad, two procedures that, respectively, fuse input modalities and transforms visual input to textual representation at inference time. In full, we hope that IsoBench can provide fine-grained feedback to diagnose performance gaps caused by the form of given input representation, and help us better understand how the capabilities of multimodal foundation models change depending on the input modality that is provided.
Acknowledgement
We would like to thank the USC NLP, Duke NLP Group, and an Open Philanthropy University Organizer Fellowship for providing computing resources. DF and RJ were supported by an Open Philanthropy research grant and a Google Research Scholar Award. DF would like to thank Vatsal Sharan and Qilin Ye for their discussions graphs. OL would like to thank Xinyan Velocity Yu, Bill Yuchen Lin, and Corby Rosset for their helpful discussions.
10

---

Published as a conference paper at COLM 2024
References
Elinor Amit, Caitlyn Hoeflin, Nada Hamzah, and Evelina Fedorenko. An asymmetrical relationship between verbal and visual thinking: Converging evidence from behavior and fmri. Neurolmage, 152:619-627, 2017. ISSN 1053-8119. doi: http://///ooooo/11.10000000 neuroiimage:201..05.22222 URL httpps:/www/ http///ww..iieeeeerretttttc com//sciiemm////tiicllpppp $1853811917702379. 2
Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexan- dre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H. Clark, Laurent El Shafey, Yanping Huang, Kathy Meier-Hellstern, Gaurav Mishra, Erica Moreira, Mark Omernick, Kevin Robinson, Sebastian Ruder, Yi Tay, Kefan Xiao, Yuanzhong Xu, Yujing Zhang, Gustavo Hernandez Abrego, Junwhan Ahn, Jacob Austin, Paul Barham, Jan Botha, James Bradbury, Siddhartha Brahma, Kevin Brooks, Michele Catasta, Yong Cheng, Colin Cherry, Christopher A. Choquette-Choo, Aakanksha Chowdhery, Clément Crepy, Shachi Dave, Mostafa Dehghani, Sunipa Dev, Jacob Devlin, Mark Díaz, Nan Du, Ethan Dyer, Vlad Feinberg, Fangxiaoyu Feng, Vlad Fienber, Markus Freitag, Xavier Garcia, Sebastian Gehrmann, Lucas Gonzalez, Guy Gur-Ari, Steven Hand, Hadi Hashemi, Le Hou, Joshua Howland, Andrea Hu, Jeffrey Hui, Jeremy Hurwitz, Michael Isard, Abe Ittycheriah, Matthew Jagielski, Wenhao Jia, Kathleen Kenealy, Maxim Krikun, Sneha Kudugunta, Chang Lan, Katherine Lee, Benjamin Lee, Eric Li, Music Li, Wei Li, YaGuang Li, Jian Li, Hyeontaek Lim, Hanzhao Lin, Zhongtao Liu, Frederick Liu, Marcello Maggioni, Aroma Mahendru, Joshua Maynez, Vedant Misra, Maysam Moussalem, Zachary Nado, John Nham, Eric Ni, Andrew Nystrom, Alicia Parrish, Marie Pellat, Martin Polacek, Alex Polozov, Reiner Pope, Siyuan Qiao, Emily Reif, Bryan Richter, Parker Riley, Alex Castro Ros, Aurko Roy, Brennan Saeta, Rajkumar Samuel, Renee Shelby, Ambrose Slone, Daniel Smilkov, David R. So, Daniel Sohn, Simon Tokumine, Dasha Valter, Vijay Vasudevan, Kiran Vodrahalli, Xuezhi Wang. Pidong Wang, Zirui Wang, Tao Wang, John Wieting, Yuhuai Wu, Kelvin Xu, Yunhan Xu, Linting Xue, Pengcheng Yin, Jiahui Yu, Qiao Zhang, Steven Zheng, Ce Zheng, Weikang Zhou, Denny Zhou, Slav Petrov, and Yonghui Wu. Palm 2 technical report, 2023. 6
Anthropic. The claude model family: Opus, sonnet, haiku. 2024. URL https://www-cch. nthropic com/ de8ba9b0 01ta00/00000003000000000000000/0 2,9
pdf.
Rohan Bavishi, Erich Elsen, Curtis Hawthorne, Maxwell Nye, Augustus Odena, Arushi Somani, and Sagnak Taşrlar. Introducing our multimodal models, 2023. URL https: //www. adept.. http//llog/fuuurre
Satwik Bhattamishra, Arkil Patel, Varun Kanade, and Phil Blunsom. Simplicity bias in transformers and their ability learn sparse Boolean functions. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics, 2023. 9
Simone Bombari and Marco Mondelli. Towards understanding the word sensitivity of attention layers: A study via random features, 2024. 9
Yung-Yun Chang and Robin Jia. Data curation alone can stabilize in-context learning. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annul Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.
1143- Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/vi /v1 a202..acll-oong-- 452. URL httpp:://aclae httpat/so///..aaooooooo..o////// www.oogg..///.... aon- 452. 9
Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, et al. Pali: A jointly- scaled multilingual language-image model. arXiv preprint ar arXio:2209.06794, 2022. 9
Xi Chen, Josip Djolonga, Piotr Padlewski, Basil Mustafa, Soravit Changpinyo, Jialin Wu, Carlos Riquelme Ruiz, Sebastian Goodman, Xiao Wang, Yi Tay, et al. Pali-x: On scaling up a multilinuual vision and language model. ar arx preprint ar Xix::23055.185 2023. 9
Liying Cheng, Xingxuan Li, and Lidong Bing. Is gpt-4 a good data analyst? ar preprint ar arXim:2305.15038 2023. 4
11

---

Published as a conference paper at COLM 2024
Margaret Anne Defeyter, Riccardo Russo, and Pamela Louise McPartlin. The picture superiority effect in recognition memory: A developmental study using the response signal procedure. Cognitive Development, 24(3):265-273, 2009. ISSN 0885-2014. doi:
https://ooioorr//..00///oooooooo 200 URL https:: piatts://Nne /ttp://ccienneiirrcccc/ htt.eee/eretcccco/eeeeeeeeeeeeee itnte//artccee//1119999551999919
Lizhou Fan, Wenyue Hua, Xiang Li, Kaijie Zhu, Mingyu Jin, Lingyao Li, Haoyang Ling, Jinkui Chi, Jindong Wang, Xin Ma, et al. Nphardeval4v: A dynamic reasoning benchmark of multimodal large language models. arXiv preprint arXiv:2403.01777, 2024. 2,
Deqing Fu, Tian-Qi Chen, Robin Jia, and Vatsal Sharan. Transformers learn higher-order optimization methods for in-context learning: A study with linear models, 2023.
Shivam Garg, Dimitris Tsipras, Percy Liang, and Gregory Valiant. What can transformers learn in-context? case study of simple function classes. ArXiv, abs/ 2t////208.00000 2022.
Hila Gonen, Srini Iyer, Terra Blevins, Noah Smith, and Luke Zettlemoyer. Demystifying prompts in language models via perplexity estimation. arviv preprint arXi::2212.04007,, 2022. 7
Gemini Team Google. Gemini: A family of highly capable multimodal models, 2023. 2,
Ruohao Guo, Wei Xu, and Alan Ritter. Instruction tuning with lexicons for zero-shot style classification. Xiv preprint drXiv:2305.14592, 2023. 9
Xiaochuang Han and Yulia Tsvetkov. Orca: Interpreting prompted language models via locating supporting data evidence in the ocean of pretraining data. ar preprint axXiv:2205.12600, 2022.
Michael Hanna, Ollie Liu, and Alexandre Variengien. How does GPT-2 compute greater- than?: Interpreting mathematical abilities in a pre-trained language model. In Thirty- seventh Conference on Neural Information Processing Systems, 2023. URL https://openrew: ew. www.ee/forunn meet/forumm/d-ppPcckNssss
Neel Jain, Ping-yeh Chiang, Yuxin Wen, John Kirchenbauer, Hong-Min Chu, Gowthami Somepalli, Brian R Bartoldson, Bhavya Kailkhura, Avi Schwarzschild, Aniruddha Saha, et al. Neftune: Noisy embeddings improve instruction finetuning. arXiv preprint axXiv:2310.05914, 2023. 9
Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arxiv preprint ar Xii::2401.0408 2024. 6
Ghazal Khalighinejad, Ollie Liu, and Sam Wiseman. Approximating cky with transformers. arXiv preprint xrXiv:2305.02388 2023. 9
Xianghao Kong, Ollie Liu, Han Li, Dani Yogatama, and Greg Ver Steeg. Interpretable diffusion via information decomposition. In The Twelfth International Conference on Learning Representations, 2024. URL
www.ne/eooorumnmxxxttNNNaa 9
Duong Minh Le, Ruohao Guo, Wei Xu, and Alan Ritter. Improved instruction ordering in
recipe-grounded conversation. ar Arv preprint ar arXii:2305.17280 2023. 9
Nayoung Lee, Kartik K. Sreenivasan, Jason D. Lee, Kangwook Lee, and Dimitris Papail- iopoulos. Teaching arithmetic to small transformers. ArXiv, abs/ 2200///33..03,,,,, 2023. 9
Paul Pu Liang Yun Cheng, Xiang Fan, Chun Kai Ling, Suzanne Nie, Richard Chen, Zihao Deng, Nicholas Allen, Randy Auerbach, Faisal Mahmood, et al. Quantifying & modeling multimodal interactions: An information decomposition framework. Advances in Neural Information Processing Systems, 36, 2024. 7
Chang Liu, Yujie Zhong, Andrew Zisserman, and Weidi Xie. Countr: -rasef generalised visual counting. Xiv preprint Xrx:::2208.11 13721, 2022." 7
12

---

Published as a conference paper at COLM 2024
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, 2023a. 6,9
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023b. 9
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024a. URL https://llava-vL gi thub. wttpl/ooooo/////oo 2024-012222000 01 30- 10---00-1llaa- llava-next/.
Ollie Liu, Deqing Fu, Dani Yogatama, and Willie Neiswanger. DeLLMa: A framework for decision making under uncertainty with large language models, 2024b.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre--trin, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9):1-35, 2023c.
Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song- Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In The 36th Conference on Neural Information Processing Systems (NeurIPS), 2022.
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. MathVista: Evaluating mathe- matical reasoning of foundation models in visual contexts, 2024. 2,9
Brandon McKinzie, Zhe Gan, Jean-Philippe Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xianzhi Du, Futang Peng, Floris Weers, et al. Mml: Methods, analysis & insights from multimodal I1m pre-training. arXiv preprint ar XrXo:24003.06611, 2024. 9
Swaroop Mishra, Daniel Khashabi, Chitta Baral, Yejin Choi, and Hannaneh Hajishirzi. Reframing instructional prompts to gptk's language. Xiv preprint ar drXio:2109.07830, 2021.
Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction to circuits. Distill, 2020. doi: 10.23915/ ditiiii000000002.001...
OpenAL. Gpt 3.5 turbo. openai.com, 2023a. URL http:::/eelp..ommaa.cc//////l/// wttp://aa...mm////aireesss//////
35514-gpt-33 (itt-3-5-tuue 2--5-turbo-uudates... 6
OpenAl. Gpt-4 technical report, 2023b. 2,9
Yasaman Razeghi, Robert L. Logan IV, Matt Gardner, and Sameer Singh. Impact of pre- training term frequencies on few-shot reasoning. ar preprint arXte:2202.07206 2022.
Reka. Reka flash: An efficient and capable multimodal language modeagemooee.......oo.ooooooi... model. 2024. URL https: //rekaa.i/reke.. http//eekaalaaaaaaaar -etchha-n-fffieett--eer- aaaeeii--e---ddd---aalaae----lla
langinodaalllgggaege
Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. Quantifying language models' sensitivity to spurious features in prompt design or: How i learned to start worrying about prompt formatting, ar preprint ar Xio:2330.11334, 2023. 9
Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models. ar preprint
xxXio:2405.09818, 2024. 9
Jesse Thomason, Daniel Gordon, and Yonatan Bisk. Shifting the baseline: Single modality performance on visual navigation & qa. North American Chapter of the Association for Computational Linguistics, abs abs/1181000613 2018.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama Open foundation and fine-tuned chat models. Xrv preprint Xiv::2300 009808070998..808 2023. 6
13

---

Published as a conference paper at COLM 2024
Bhavya Vasudeva, Deqing Fu, Tianyi Zhou, Elliott Kau, Youqi Huang, and Vatsal Sharan. Simplicity bias of transformers to learn low sensitivity functions, 2024.
Johannes von Oswald, Eyvind Niklasson, E. Randazzo, João Sacramento, Alexander Mord- vintsev, Andrey Zhmoginov, and Max Vladymyrov. Transformers learn in-context by gradient descent. In International Conference on Machine Learning, 2022. 9
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large lan- guage models, 2023. 9
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models, 2023. 9
Kang Min Yoo, Dongju Park, Jaewook Kang, Sang-Woo Lee, and Woomyeong Park. Gpt3mix: Leveraging large-scale language models for text augmentation. ar preprint arXiv:2104.08826, 2021. 9
Lili Yu, Bowen Shi, Ramakanth Pasunuru, Benjamin Muller, Olga Golovneva, Tianlu Wang, Arun Babu, Binh Tang, Brian Karrer, Shelly Sheynin, et al. Scaling autoregressive multi- modal models: Pretraining and instruction tuning. Xrviv preprint ar Xiv:2309.002591, 2023.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu liang, Weiming Ren, Yuxuan Sun, et al. MMMU: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. arXiv preprint arXiv:2311.1650,, 2023. 2,9
Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai- Wei Chang, Peng Gao, et al. Mathverse: Does your multi-modal Ilm truly see the diagrams in visual math problems? arXiv preprint arXio:2403.1462, 2024a. 2,
Yizhe Zhang, He Bai, Ruixiang Zhang, Jiatao Gu, Shuangfei Zhai, Josh Susskind, and Navdeep Jaitly. How far are from intelligent visual deductive reasoning? arXiv preprint ar Xri::2203.00433, 2024b. 9,9
Tianyi Zhou, Deqing Fu, Vatsal Sharan, and Robin Jia. Pre-trained large language models use fourier features to compute addition, 2024. URL https: wttpv/.orgg/bb///0//// 03445.
14

---

Published as a conference paper at COLM 2024
Appendix
A Dataset Generation
We detail procedures to curate the IsoBench dataset. Please refer to the supplementary materials for concrete samples and further implementation details.
A.1 Mathematics
1. Image: the model is prompted with (textual) problem statement and the image visualization of the function. We generate these plots with matplotlib.
2. Text (BTEX): in addition to the problem statement, we provide L-TpXdefinition of the function as textual inputs.
3. Text (Code): we replace the ITI'definiiton function in the previous representation with its sympy definition. All other parts of the prompt are kept identical.
Even & Odd Function. We sample a dataset of rational functions - the quotients of two polynomials and extract their even and odd parts samples. Each prompt formulates 3-way classification problem for the model to choose among even, odd, and neither. Each class consists of 128 samples.
Convex & Concave Function. We sample convex (resp. concave) functions by sampling operations from [x",|x), |x|,- log x, exp x} (resp. the negative of these operations) as well as multiplicative weights. Each class consists of 128 samples; and prompt the model with their appropriate domains to ensure correctness of either of the properties.
Counting Breakpoints. We sample piecewise linear functions with 2 or 3 breakpoints, ie.. sudden changes in slope at intersections of linear functions. We manually audit to ensure that the changes in slope are visible. Each class consists of 128 samples.
Images for math problems are generated with 300 DPI in JPEG format.
A.2 Algorithms
1. Image:
the model is prompted with an instruction of the problem to solve and the image visualization of the graph. For graphs, we use networkx visualize.
2. Text with ITę:: the model is prompted with an instruction of the problem solve and the mathematical expression of the graphs, and here we choose to use the adjacency matrix as the mathematical representation.
3. Text with story or description: the model is prompted with an instruction of the prob- lem to solve and the story-telling description of the scenario. For example, formulate the graph connectivity problem as the possibility of driving from one city to another.
Graph Connectivity. We sample 128 undirected graphs using Erdos-Renyi random graph generation method. Each pair of nodes has probability of be connected. The query nodes are also sampled at random and we balance the benchmark to have 50% of the sample having query nodes connected and the rest disconnected.
Maximum Flow. We sample 128 weighted directed graphs. Each edge has random integer flow uniformly sampled from to and a fairly sampled direction. Maxflow fairly hard problem
restrict the graphs to have 3, 4 or 5 nodes.
Graph Isomorphism. We sample 128 pairs of graphs, with 64 pairs isomorphic and 64 of them not isomorphic. In generating isomorphic pairs G and H, we sample a random per- mutation matrix II, and the adjacency matrix corresponding to graph is Aн = IIAGII where Ag is the adjacency matrix of graph G. In generating non-isomorphic pairs, Aç and
15

---

Published as a conference paper at COLM 2024
Ан sampled individually. To avoid models using simple shortcuts such counting the number of nodes, graphs G and in each pair are guaranteed to have the same number of nodes and the same number of edges.
Images for algorithm problems are generated with 300 DPI in JPEG format.
A.3 Games
1. Graphical Board: the model is prompted with an instruction of the problem solve and the visual representation of the chess board and its pieces in graphical format (PNG) is given.
2. Algebraic Notation Layout (ANL): this refers to a text-based representation of the current position of pieces the chess board, using algebraic notation for the squares.
3. Portable Game Notation (PGN)): this is a textual representation of the chess game's moves in standard algebraic notation. PGN is widely used for the recording of games, allowing for both humans and computers read the game history. captures the entire sequence of moves from the beginning to the current state or the end of the game.
4. Persyth-Edwarde Notation (FEN): this is compact way to describe the current position of a game of chess in textual format.
Winner Identification from Checkmate The task is to analyze a given chess game repre- sentation and identify the winner: whether the game resulted in win for White or Black, or it ended without a winner. Our dataset for this task contains the games that lasted more than nine rounds in February 2024.
Chess Puzzles For this task, consider chess puzzles that consist of chess position, along with a sequence of optimal moves necessary to resolve the puzzle. In our evaluation, the primary objective is to predict the initial move.
Images for Chess are the same as Lichess, and are converted to JPEG formats.
A.4 Sciences
Physics
Chemistry
Matee
Solutions
motoon, energy
13.3%
Chemical equations
12.0%
Forow, molio
33.3%
Organic Compound
16.0%
Thermal energy
Chemical reactions
Magraix
16.0%
States mater
13.3%
Bllctric Graxt
8.0%
33.3%
Figure 4: Distribution of science problem categories. We introduced three new categories that are absent from the ScienceQA dataset: Electric Circuit, Organic Compound, and Chemical equations.
1. Image: Each sample contains the textual questions, choices, and accompanying figure providing additional context to the question.
2. Text:
Instead of prompting LLMs to describe the image content, we asked one author manually write descriptions for each figure. The annotator was instructed to avoid including additional reasoning or information beyond what is present in the figure, focusing solely on describing the content.
Images for Chess are the same ScienceQA, and converted to JPEG formats.
16

---

Published as a conference paper at COLM 2024
B Sample Responses for Math, Algorithms, Games, and Science Problems
We show sample responses for Math, Algorithms, Games, and Science Problems in the figures below, for multiple multimodal foundation models.
PROMPT
are given real-valued, scalar function fxx)
YOUR TASK is to determine whether f(x) is an even function, an odd function, or neither.
Definition an even function: function such that
(x))= f(-x)
where the value remains unchanged the sign of the independent variable reversed.
Definition of an odd function: A function such that
x-xx)=-(xx)
where the sign is reversed but the absolute value remains the same if the sign of the independent variable is reversed. odd if it does not satisfy either condition.
function is neither
Here is the expression of f(x):
7x +x 21.76
f(x) 2x + 4x + +3x2 11.34
Respond with 'even', 'odd', 'heither' first whether the function f(x) is even, odd, neither, based the definitions your observation of function. Please parse it to JSON format. Put the result (even, odd, or neither) under key 'answer', and put the explanations and reasonings under key 'explanation" (Do not summarize, paraphrase or change meanings).
GPT-4 RESPONSE
to determine whether the function
7x 3x x² 21.76
f(x) 2x + + + 3x + 11.34
is even, odd, or nei thit, we substitute for and compare the resulting expression original function. for an even function, we should find that
f(x)=f(-x)
and an function, we should find that
f(-x) = f(x)
substituting into the function, get
7( 3(-x))-((xx)-)) (-x)
7x + 3x +2 21.76
f(-x) =
2( -(-x)
2x + 4x4 + + 3x +1.34
this expression not equal the original function f(x), indicating that the function expression is the negative of the original function, indicating that the function is not odd. the presence of the x term, the other terms,
not even. additionally,
which changes sign when is replaced with -x, but does not satisfy the condition for an odd function is key factor in determining that the function is neither even nor odd.
Figure 5: Sample response from GPT-4 to parity problem in IsoBench. GPT-4 was able to analyze the parity of the rational function with the correct reasoning and computation.
17

---

Published as a conference paper at COLM 2024
VISUALTZATION
PROMPT
You are given plot of real-valued, scalar function f(x). YOUR TASK determine whether f(x) even function, an odd function, or neither.
Definition of an even function: A function such that
f(-x=
where the value remains unchanged if the sign of the independent variable is reversed.
Definition of an odd function: function such that
(fxx)=-(xx)
where the sign is reversed but the absolute value remains the if the sign of the independent variable is reversed
A function is neither even nor odd if it does not satisfy either definitions.
Respond with 'even", 'odd', 'heither' first whether the function f(x) is even, odd, neither, based the definition above. For any your conclusions (even, odd, or neither), please explain details based on your observation of the plot. Answer (start with 'even', 'odd', or 'neither'):
GPT-4 RESPONSE
The plot of the function f(x) shows symmetry about the y-axis. This means that for every point (x, fxx) on the right side of the y-axis, there is a corresponding point f(x)) on the left side of the y-axis with the same function value. This symmetry indicates that
f(x)
for all x in the domain of the function, which is the definition of an even function. Therefore, based on the plot, f(x) is an even function.
Figure 6: Sample response from GPT-4V to the same parity problem as in Figure 5. GPT-4V cannot distinguish the nuanced asymmetry at the top-rigth and top-left corners of the plot. We observe the same failure mode for Gemini-Pro and LLaVA.
18

---

Published as a conference paper at COLM 2024
PROMPT
You given real-valued, scalar function f(x)-
YOUR TASK is to determine whether f(x) is an even function, an odd function, or neither.
Definition an even function: function such that
f(x)=((xx)
where the value remains unchanged if the sign of the independent variable reversed. Definition of an odd function: A function such that
f(-x)=-f(x)
where the sign is reversed but the absolute value remains the same if the sign of the independent variable is reversed. odd if it does not satisfy either condition.
function is neither
Here is the expression of f(x):
4x (x² (22-1 1)
f(x) 16x10 8x 10x2 13.98
Respond with 'even', "odd', 'neither' first on whether the function f(x) is even, odd, neither, based on the definitions function.
your observation
Please parse it to JSON format. Put the result (even, odd, or neither) under key "answer", and put the explanations and reasonings under key 'explanation' (Do not summarize, paraphrase or change meanings).
GPT-4 RESPONSE
to determine whether the function
4x (x2 1)
f(x) 16x10 8x 10x2 13.98
is even, add, or neither, we substitute -x for and compare the result to the original function. substituting -x into the function gives
4( 4(-x)3 ((-))2 1)
4x (x2 1)
f(-x) 16(-x)"++(<x3>>> + 10( 10(-x)2 13.98 16x + 8x 10x + 13.98
which simplifies -(x). this that the function satisfies the condition for being odd function, which is
f( x) = f(x) the function does satisfy the condition for being
function, which would require
f(x) = f(-x)
therefore, the function is odd.
Figure Sample response from GPT-4 parity problem in IsoBench. GPT-4 was able to analyze the parity of the rational function with the correct reasoning and computation.
19

---

Published as a conference paper at COLM 2024
VISUALIZATION
009
000
001
000
-001
002
-08
PROMPT
You are given plot of a real-valued, scalar function f(x). YOUR TASK is to determine whether f(x) is an even function, an odd function, or neither. Definition an even function: function such that
(xx==(->>)
where the value remains unchanged the sign of the independent variable reversed.
Definition of an odd function: A function such that
x--x)=-(xx
where the sign is reversed but the absolute value remains the same if the sign of the independent variable is reversed function is neither
odd if does not satisfy either definitions.
Respond with "even", 'odd', 'neither' first on whether the function f(x) is even, odd, or neither, based on the definition above. For any of your conclusions (even, odd, or neither), please explain in details based on your observation of the plot. Answer (start with 'even', 'odd', or 'neither'):
GPT-4 RESPONSE
neither
The function f(x) depicted in the plot does not satisfy conditions for being an even function nor an function. a function to be even, the graph of the function should be symmetric with respect to the y-axis, meaning that the part of the graph in the positive x-region should be mirror image of the part in the negative x-region. This function does not exhibit such symmetry; the shapes and values of the function either side of the y-axis are not mirror images of each other.
Similarly, for function to be odd, the graph should have rotational symmetry about the origin, which means that rotating the graph 180 degrees around the origin should result the same graph. This is not the case for the given function either, as the positive and negative sides of the graph do not match when one side is rotated 180 degrees.
Since the function does not meet the criteria for being even or odd, we conclude that the function is neither even nor odd.
Figure 8: Sample response from GPT-4V to the same parity problem in Figure 7. GPT-4V cannot reason with the symmetry at origin. We observe the same failure mode for all other vision models.
20

---

Published as a conference paper at COLM 2024
PROMPT
You are given real-valued, scalar function f(x)
YOUR TASK is to determine whether f(x) is an convex function or concave function
Definition of a convex function: function such that for all x, y, and <=
f(tx +
(1 f)y) tf(x) f)
Definition of concave function: A function such that for all y, and <=
f(tx (1 f)y) 2 fx(x (1 f)
Here is the expression of f(x), defined for all x 0:
f(x) 0.31x 1.73 (x) 1.13 |x| 0.63
Respond with 'convex' or 'concave' first on whether the function f(x) is convex or concave, based on the definitions and your observation of the function.
Please parse it ISON format. Put the result (convex concave) under key 'answer', and put the explanations and reasonings under key 'explanation' (Do not summarize, paraphrase or change meanings).
Claude RESPONSE
The given function f(x)
function. This be determined by examining the individual terms in the expression:
1. The term 0.31x is linear function, which is both convex and concave.
2. The term -1.33 log(x) is concave function, as the logarithm function is concave for x > 0.
3. The term 1.13/x| is convex function, the absolute value function is convex.
4. The constant term 63.63 does not affect the convexity or concavity of the function.
Since the sum consider the dominant term. In this case, the convex term 1.13/x| grows faster than the concave term -1.73 lo.73log(x) as x increases. Therefore, for sufficiently large values of the convex term will dominate, making the overall function f(x) convex.
convex function and concave function is not guaranteed to either convex concave, we need to
Figure 9: Sample response from Claude 3 to convexity problem in IsoBench. Claude 3 was able to analyze the convexity of the function with the correct reasoning and computation.
21

---

Published as a conference paper at COLM 2024
VISUALIZATION
PROMPT
You are given plot of real-valued, scalar function f(x). YOUR TASK is to determine whether f(x) is convex function
function
Definition of convex function: function such that for all x, y, and <= <=
f(tx + (1 - t)y) s tf(x) + (1 - t)f(y)
Definition of a concave function: A function such that for all x, y, and <=
f(1x + (1 f)y) 2 fx(x (1 t)f(y)
Respond with 'convex', 'concave' first whether the function f(x) is convex concave based the definition above. For any your conclusions (convex or concave), please explain in details based on your observation plot. Answer (start with "cnvexx" or 'concave'):
Claude RESPONSE concave
Based on the plot, the function f(x) appears to be concave. The key observations are:
1. The curve is continuously increasing, but the rate of increase slowing down as x gets larger. This can seen from the curve "flattening out" and having a decreasing slope as x increases.
you draw line segment connecting any two points on curve, that line segment lies entirely below the curve. This is a key property of concave functions.
3. Intuitively, concave function curves inward like the interior of circle or sphere. The plot shows this inward curving shape characteristic of
functions.
4. In contrast, convex function would curve outward with an increasing slope, and line segments between points would lie above the curve.
So in summary, the decreasing slope, the inward curving shape, and the fact that line segments between points lie below the curve all clearly indicate that the plotted function f(x) is concave, not convex.
Figure 10: Sample response from Claude 3 the same convexity problem as in Figure 9.
Claude 3 cannot distinguish the orientation of the curve at all.
22

---

Published as a conference paper at COLM 2024
PROMPT
are given real-valued, scalar function f(x). YOUR TASK is to count the number of breakpoints in the plot of f(x). A breakpoint refers to a point on the function's 's domain at which the function changes its slope. Here is the expression of f(x):
28.88x + 375.32 for 14.9 < x -9.53
-18.18x
for 9.53 x 1.47
f(x)
54.18x
179.55
for 1.47 x 5.16
277.33 - 34.43x for 5.16 < 10.95
Respond with the number of breakpoints (in Arab digits) first on how many breakpoints the function f(x) contains based on the definition and your observation of the function. You should IGNORE the left and right end point of the domain, i.e. if the function is defined on [a, b], you should only consider the domain (a, b).
Please parse it to JSON format. Put the result (in Arab digits) under key 'answer', and put the explanations reasonings under key 'explanation' (Do not summarize, paraphrase or change meanings).
Claude RESPONSE
The function f(x) is piecewise function defined over 4 intervals. The endpoints of these intervals are -14.9, -9.53, 1,47, 1., and 10.95. Ignoring the leftmost and rightmost endpoints as per the instructions, we consider the points -9.53, 1.47, and 1. At each of these points, the function changes its slope (i.e., the coefficient of changes). Therefore, these three points breakpoints of the function f(x). Thus, the function f(x) has breakpoints within its domain, excluding the endpoints.
Sample response from Claude 3 to a breakpoint problem IsoBench. Claude was able to identify the exact locations of the breakpoints. We observe similar behaviors for all API-access foundation models.
Figure 11:
23

---

Published as a conference paper at COLM 2024
VISUALIZATION
AA
75
23
-22
-12
PROMPT
You given plot of real-valued, scalar function f(x). YOUR TASK at which the function changes its slope. You should IGNORE the left and right end point of the domain, i.e. if the function is defined on b,, you should only consider the domain (a,
count number breakpoints the plot f(x). breakpoint refers a point on function's domain
Respond with the number of breakpoints (in Arab digits) first how many breakpoints the function f(x) contains based the definition above. Answer (in Arab digits):
any your result, please explain details based on your observation of the plot.
Claude RESPONSE
Answer (in Arab digits):
Explanation: Based on the plot of the function f(x), observe two breakpoints within the interior of the domain, excluding the endpoints.
The first breakpoint occurs at = 0. As approaches 0 from the negative side, the function has positive slope. However, as and becames positive, the function changes to negative slope. This change in slope direction a breakpoint.
indicates
The second breakpoint is located at approximately == 5. From x = 0 to x = 5, the function has a negative slope. But as exceeds 5, the function's slope becomes positive again. This transition from negative to positive slope around x = signifies another breakpoint.
The left right endpoints of the function's domain are not considered breakpoints according to given definition. Therefore, focusing only on the interior the domain, the function f(x) contains exactly breakpoints based on the changes in slope observed in the provided plot.
Sample response from Claude 3 to the same breakpoint problem as in Figure 11. The behavior of Claude 3 is combination of failing to account for all the breakpoints and/or reason with respect to the boundary points. We observe similar behaviors for all multimodal foundation models.
Figure 12:
24

---

Published as a conference paper at COLM 2024
VISUALIZATION
PROMPT
are given an adjacency matrix graph and two query nodes. (one source node one sink node). The source node node where the flow starts and the sink node is the node where the flow ends.
YOUR TASK is to solve the maxflow problem given the weighted directed graph. Definition of Maxflow problem:
In the max flow problem, we have directed graph with source node and sink node t, and each edge has capacity (integer valued, colored in green) that represents maxinum amount of flow that can be sent through The goal is to find the maximum amount of flow that can be sent from s to t, while respecting the capacity constraints on the edges.
Query Example:
adjacency matrix:
[e 2]
[0 7]
[0 4]
[0 6]
[0
Source node (zero-indexed):
Sink node (zero-indexed). 4
In the query example, the nodes are zero-indexed. Compute the maximum flow from the source node the sink node.
CLAUDE-3 RESPONSE
To solve the maximum flow problem using the given adjacency matrix, we use the Ford-Fulkerson algorithm with the Edmonds-Karp implementation. The algorithm finds the maximum flow by repeatedly finding augmenting paths from the to the sink and updating the residual graph until no more augmenting paths can be found.
Given:
Adjacency matrix:
[e
2]
[e
[0 4]
[0 6]
[0 8]
Source node:
Sink node:
Step 1: Initialize the flow matrix with all values set to 0.
Step 2: Find an augmenting path from the source to the sink using BFS. Augmenting path 1: -> ->
Residual capacity: min(7, 7) 7
Update the flow matrix and residual graph.
Step 3: Find another augmenting path. Augmenting path 2: 0 -> 2 Residual capacity: min(4,
Update the flow matrix and residual graph.
Step 4: Find another augmenting path. Augmenting path 3: 0 -> -> Residual capacity: min(3, 6) =
Update the flow matrix and residual graph.
Step 5: Find another augmenting path.
Augmenting path 4: 0
Residual capacity: Update the flow matrix and residual graph.
Step 6: more augmenting paths can be found. The maximum is the sum of the flows on the edges from the source to sink.
Maximum flow = 7 2
Figure 13: Sample response from Claude-3-Opus to a maxflow problem in IsoBench.
25

---

Published as a conference paper at COLM 2024
PROMPT
You given a FEN representation a chess puzzle for which sequence of unique best moves determinable (e. g. sequences moves leading forced checkmate).
Definition of the Chess Puzzle: checkmate, starting from the given position.
chess puzzle, you are required make series of optimal moves leading
YOUR TASK is to predict THE FIRST MOVE that should be played given this board setup.
Your answer should first specify the move in Algebraic Coordinate Notation (e.g., "d2d1", "e5al", "c4f4"), Afterward, provide your reasoning for why you chose that as your first move.
FEN http://pppppppmmaaaa//////2222//
Answer (start the first move Algebraic Coordinate Notation like "e2e4" or "d7d5"):
GPT-4 RESPONSE
First Move: "a2a3"
Reasoning: this position, given by the FEN "r20k2rr/pp2pppp2266/811 http://///n/aeeiem/////////////p White's turn to move. The board setup reveals significant tension in the center and around the kingside, particularly with both bishops (the Black bishop on g4 and the White bishop on g5) exerting pressure.
KQkq 6 8",
The immediate threat to address is the pin of the White knight on f3 by the Black bishop g4. Moving the h3 to attack the bishop would tempting to relieve the pin on the knight; however, it weakens White's kingside structure can be met by from Black, securing the bishop's position.
A crucial observation here is the Black bishop on b4, which is currently pinning the white knight on e4, preventing it from moving wi thout losing the queen. The first move, "a2a3", is aimed at challenging this bishop. This move forces the Black bishop to make a decision; it either retreat be captured if it remains the b4 square. Retreating the bishop to a5 or c5 does not alleviate Black's positional difficulties significantly, and capturing on would lead bxc3, strengthening White's s center and opening lines for the rook and bishop.
This move is selected over other aggressive or defensive options because it immediately addresses piece that contributes to pinning and pressure wiohh compramising White's position or losing material. By initiating this challenge, White aims to disrupt Black's coordination and gain tempo to improve their position, possibly by preparing castle, unpin the knight on f3, or increase control in the center. This move also preserves the option of castling kingside for White, keeping the king's safety as a priority.
Figure 14: Sample response from GPT-4 to a chess puzzle problem in IsoBench.
26

---

Published as a conference paper at COLM 2024
VISUALTZATION
A
A
W
PROMPT
You are given visual representation of a chess puzzle for which sequence of unique best moves is determinable (e.g. sequences of moves leading to forced checkmate).
Definition of the Chess Puzzle: In a chess puzzle, you are required to make a series of optimal moves leading to checkmate, starting from the given position.
YOUR TASK is to predict THE FIRST MOVE that should be played given this board setup.
Your answer should first specify the move in Algebraic Coordinate Notation (e.g., "d2d1", "e5al", "c4f4"), Afterward, provide your reasoning for why you chose that as your first move.
Answer (start with the first move in Algebraic Coordinate Notation like "e2e4" or "d7d5"):
GPT-4 RESPONSE
Answer: Bg5f6
The best move for white to play in this position is Bg5xf6. The reason behind this move is that by capturing the knight on f6 with the white bishop, white is achieving multiple objectives:
White removing defender
the square, which could potentially an entry point for the white queen or other pieces if the knight on c6 moves. The g7 pawn is pinned by the white bishop on b2, and cannot recapture on f6 without losing material since the rook on h8 would be left undefended. This move also opens the e-file for the white queen and rook to become more active, possibly leading to attack the black king, who is still in the center of the board and unable to castle due to the king's ss move earlier. Thus, 8g5xf6 appears be the strongest move the position, setting up threats and potential for further penetration into black's s position.
Figure 15: Sample response from GPT-4V to the same chess puzzle problem as in Figure 14.
27

---

Published as a conference paper at COLM 2024
PROMPT
You are given multiple-choice chemistry question.
YOUR TASK is to read the question and select the correct answer from the provided options.
In Solution A and Solution B, the green particles represent the solute. The volume of the solvent in both containers are 20 mL. Solution A has three green particles, and Solution B has three green particles.
Which solution has higher concentration green particles?
Solution
B. neither; their concentrations are the same
C. Solution
GPT-4 RESPONSE
If Solution A and Solution have the same number of green particles (solute) and the volume of the solvent in both containers is equal, then their concentrations of green particles the same. Therefore, the correct answer is: neither; their concentrations are the same
Figure 16: Sample response from GPT-4 to a multiple-choice chemistry question in IsoBench.
VISUALIZATION
Solvent volume: 20 mL
Solvent volume: 20 mL
Solution
Solution
PROMPT
You are given mul miltieleee chemistry question.
YOUR TASK is to read the question and select the correct answer from the provided options.
which solution has higher concentration of green particles?
Solution
neither; their concentrations are the same
Solution
GPT-4 RESPONSE
Solution has higher concentration green particles. There are more green particles in Solution A than Solution B, and both solutions have the same volume of solvent (20 mL).
Figure 17: Sample response from GPT-4V to the same multiple-choice chemistry question as in Figure 16.
28

---

Published as a conference paper at COLM 2024
Image Representation
Text Representation
in Seliaion A and Solation B, the thee part clss reprosent the ackin The velumre of 40mi
blee partices.
higher
Which ast conconaraiio of partiima??
Solusion SolutoonA
their concentration
atlutio il Accidiin weither thei cont
This it cheeicss my equantin represmnting nin the wactin behoeen sold todkum 7400 and load water (F20) to korn aqueces sociun tydioooe (huDke anc rydtoggen gan 0.9% seadton too makes of modiame rest oow molee bydhogen gaa.
moles socker lydrooide and ono
the figar readiion belwein soaiaan waker ('o))), Which bost
Which best eup lano why the equarion talanced??
explana coundin
D. There to much aodium the mectents.
0 There to too much sockum in the resccents
There not enough water reacaara.
much
There is too much yotrggnn gae the productss
3. There is not enough sodium hydroodde the sroducts
many motec fomulla CBH7CI.
thene in givon organic mogaric molecula??
given organic
1.1 22 3.
The ditance between the maggers Pair 1.5 while the distacce between the magness in 2.
Think about the magnetic lorce beoween the magnet each pant Which the 120 tollowing steaeennts trua?
magreti ragges cach pak
The magnitute
boi
magiia mageess grease
magnitade magnetio greater
magrita maggelic forze greate
The magnitude the magnetic force is greater Pair
W-
20
W-
in pain throug locabin when the stucent dooee wenthees stexdd statent
There mann
V. ars tourch
elcotial brandh oa
consent watttmn otnm and zamne)
swdiches W,,
7:0
The sannch with the Y owich combaioo medertos O7 ohnn and chiet connectells series
switen consans single (7
A student wonto the loggoal poso life ourreet to sado Bhoouhh location Q wirce
1.X
Table 7: Examples of isomorphic representations on Science problems.
29

---

Published as a conference paper at COLM 2024
C
Full IsoBench Results
GPT-4 Turbo
GPT-3.5 5 Turbo
Mixtral-x77B
LLaMa-2-70B
LLaVa-1.5-138
Random Guess
Gemini Pro
Claude 3
PaLM-2
Image
72.0
66.7
71.3
44.0
Science
38.3
Text
86.7
69.3
89.3
84.0
76.0
68.0
61.3
Image
46.9
36.4
41.7
34.9
Mathematics
Text (IXTEX)
76.6
73.9
85.8
68.9
61.8
63.3
33.6
44.4
Text (SymPy)
75.3
73.9
85.6
73.3
63.0
66.1
35.5
Image
54.2
37.0
39.1
34.1
Algorithms
Text (LTTEX)
58.6
37.8
62.8
42.7
42.2
40.4
39.3
34.7
Text (Story)
69.5
43.8
73.7
49.5
55.2
49.5
35.4
Image
27.6
22.9
21.0
0.00
Text (ANL)
25.7
13.8
30.5
22.9
22.5
20.4
12.9
Games
18.1
Text (FEN)
27.4
4.7
38.2
15.7
4.6
3.7
6.5
Text (PGN)
42.8
34.9
33.4
26.2
32.8
35.3
27.1
Image
50.2
40.7
43.3
28.3
Average
Text (Best)
68.9
55.6
71.9
58.2
56.8
57.2
39.8
33.9
Gap
18.7
14.9
28.7
Table 8: IsoBench results with all API-access models and open-source models. All tasks of the four major domains are included.
30


Reviews:
[{"review_id": "5I2nFa3QQK", "reviewer": "Reviewer_CR6d", "summary": "The paper introduces a new benchmark dataset, IsoBench, which aims to evaluate multimodal foundation models on problems with isomorphic representations. The authors have put effort into constructing the dataset and conducting evaluations on various models. The paper is generally well-structured and easy to follow, with the use of tables, figures, and examples to illustrate the findings.\n\nWhile the work addresses an interesting research problem and provides some insights into the performance of multimodal foundation models, there are several concerns that limit its overall significance and originality. The task setting may not fully reflect real-world applications, and there are issues with data annotation and missing details that could affect the reliability of the benchmark. The proposed techniques, IsoCombination and IsoScratchPad, are relatively simple and do not provide substantial novel insights. Addressing these concerns and expanding the benchmark to include more challenging and diverse figures could strengthen the contribution of this work to the field of multimodal understanding and reasoning.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the feedback. We are grateful that you appreciate our efforts in curating the dataset, and find our analyses comprehensive and presentation clear. \n\n### **Task settings may not reflect real applications**\n\nWe designed IsoBench for two main purposes:\n\n- **Our tasks are helpful predicates to real-world applications.**\nVLMs need to understand the node ordering (graphs) to analyze flowcharts; they need to capture curve patterns (math plots) to offer insights on data visualization. Our science tasks also serve as a first step for more complex real-world scenarios.\n\n- **A diagnostic tool for studying the difference across modalities is valuable to the community.**\nMost multimodal datasets require strong reasoning capabilities or knowledge to solve. It is challenging to understand why models perform poorly on these datasets. Simple tasks in IsoBench can help pinpoint failure modes on the visual component.\n\n### **Data annotation concerns**\nWe acknowledge the potential bias in using a single annotator. To address this, they were instructed to follow the guidelines emphasizing consistency and objectivity in description generation. Further, the authors conducted internal reviews of the annotations. We will include these detailed guidelines in the revision. Additionally, the annotator is a STEM PhD student with scientific domain expertise relevant to accurately annotating question.\n\n### **Data construction details**\nWe will add image resolution details to Appendix A, which contains details on dataset construction. We will open-source our datasets. Both graph and math problems are generated with 300 DPI in PNG. Chess and Science images are the same as Lichess and ScienceQA, respectively.\n\n### **Naive baselines**\nIsoCB and IsoSP are designed to be simple to show existing VLMs’ strong bias towards “language priors” [2].\nFurther, our main goal is to introduce IsoBench as a diagnostic tool for VLMs. We left developing sophisticated methods as future work. \n\n### **Related work**\nWe kindly want to note that we consider MathVerse [1] to be contemporaneous work, as it was released on arXiv on 03/21, and COLM submission deadline was 03/29.\n\nNonetheless, we will include a more thorough discussion of MathVerse, which studies redundant, but not necessarily equivalent input representations on math problems. IsoBench studies the impact of isomorphic representations on a broader set of tasks.\n\n[1] https://arxiv.org/abs/2403.14624\n\n[2] https://arxiv.org/abs/2306.01879", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper addresses an important research question in evaluating multimodal foundation models and provides valuable insights for future work.\n\nThe comprehensive evaluations and analysis reveal consistent preferences for textual representations and identify specific issues, such as visual recognition errors and insufficient fusion schemes.\n\nThe paper is well-organized, clear, and easy to follow, making it accessible to readers.", "reasons_to_reject": "### (1) Task settings may not reflect real applications\n\nFor example, the science questions featured in IsoBench are relatively easy to describe in natural language. However, in real-world applications, many scientific figures are very complex and difficult to capture fully in natural language descriptions. These figures often serve as complementary information alongside the text. The benchmark may not adequately represent the challenges faced in real-world multimodal understanding tasks.\n\n### (2) Data annotation concerns and missing details\n\nThe authors mention that a single author manually wrote descriptions for each figure in the science questions, with instructions to avoid introducing extra reasoning or information. However, there are several issues with this approach:\n- Relying on a single annotator may introduce bias and inconsistencies in the data.\n- Annotating high-quality descriptions for scientific images can be challenging and requires domain expertise. The paper does not provide information about the annotator's qualifications or background.\n- There is no mention of any quality control measures or validation processes to ensure the accuracy and consistency of the annotations.\n\n### (3) Missing data construction details\n\nThe paper lacks important details regarding the construction of the benchmark dataset. For example, the resolution of the image representations is not specified. Low-resolution images may result in poor understanding by current VLMs. Without this information, it is difficult to assess the validity and reliability of the benchmark.\n\n### (4) Naive baselines \n\nThe two baselines proposed in the paper, IsoCombination and IsoScratchPad, are relatively simple and do not provide significant insights. It would be more valuable to see novel and insightful models that are inspired by the findings from the evaluations and analysis. The authors could explore more sophisticated approaches to address the performance discrepancies between different input modalities.\n\n### (5) Related work\n\nA superficial discussion on a very closely related work: MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?", "questions_to_authors": "The paper introduces IsoBench, an innovative benchmark dataset that sheds light on the performance of multimodal foundation models across various isomorphic representations. The authors' efforts in creating this dataset and conducting comprehensive evaluations are commendable and provide valuable insights for the research community.\n\nHowever, to further strengthen the paper, I would encourage the authors to consider expanding the benchmark to include more challenging and diverse figures that better reflect real-world applications. Providing additional details on the data annotation process, such as the background of the annotators and quality control measures, would also help bolster confidence in the dataset's reliability.\n\nMoreover, while the proposed IsoCombination and IsoScratchPad techniques offer interesting avenues for exploration, developing more advanced models that build upon the findings of this study could yield even more impactful insights.", "extra_scores": {}}, {"review_id": "pAkFVISB9e", "reviewer": "Reviewer_yceX", "summary": "This paper proposes an interesting benchmark called IsoBench, designed to evaluate multimodal foundation models on tasks with isomorphic representation. It provides key observations on existing models and introduces prompting techniques (IsoCombination and IsoScratchPad) to integrate input modalities effectively. \n\nThe experiments offer empirical findings of the capabilities of existing models when processing different modal inputs.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive feedback and suggestions. We are grateful that you recognize our efforts in curating the dataset, and find our analyses comprehensive and presentation clear. We are also glad that you find IsoBench useful to the broader multimodal community.\n\n### **Preference for text representation is not inherently negative or positive**\nThis is a great question! To clarify, we are not claiming that preferences toward a visual or textual representation are good or bad. The main observation is that multimodal foundation models exhibit preferences that are different from humans.\n\nNevertheless, we believe multimodal models should possess strong visual reasoning knowledge capabilities, even if they cannot match pure textual inputs on certain tasks. Multimodal models can unlock a broad spectrum of applications in cases where visual inputs are critical and/or natural. Some examples include agents that navigate computer environments [1] and multimodal planning [2]. \n\n### **Root cause of the observed model biases**\n\nTo our knowledge, this remains an open question, but one popular hypothesis is that multimodal models are endowed with a “language prior” [3] bias as a consequence of their training strategies.\n\nWe would also like to use our observations from IsoBench to improve open-source multimodal models. One possible approach is to use mechanistic interpretability (e.g. [4]) to pinpoint the root cause (e.g. poor modal fusion, insufficient training FLOPs, etc) of the performance discrepancies. This would hopefully motivate better architecture design and/or training strategies. \n\n[1] OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments, https://arxiv.org/abs/2404.07972\n\n[2] Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learning, https://arxiv.org/abs/2405.10292\n\n[3] Revisiting the Role of Language Priors in Vision-Language Models https://arxiv.org/abs/2306.01879\n\n[4] Interpreting CLIP's Image Representation via Text-Based Decomposition, https://arxiv.org/abs/2310.05916", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper thoroughly discusses and examines existing multimodal foundation models, enhancing our understanding of their capabilities in various modality input scenarios. The experiments are solid. The findings in the paper are valuable for future research.\n\n- The proposed benchmark could serve as a great resource for performance evaluation.\n\n- It introduces two prompting methods: IsoCB, which assesses the combined effect of using multiple representations, and IsoSP, which translates visuals to text to solve tasks with text-only inputs. Both methods have shown improved performance.\n\n- The paper is well-written and easy to understand.", "reasons_to_reject": "- The paper assumes that multimodal foundation models should possess strong visual capabilities, akin to human cognition. However, the path toward Artificial General Intelligence (AGI) remains undefined regarding this aspect. That is, emulating human visual cognition could be advantageous but may also be limited by human capabilities. These models might surpass human cognition in certain areas that are beyond human perception. Thus, a preference for text representation is not inherently negative or positive. The focus should be on whether the models function effectively, irrespective of the representation type.\n\n- While evaluating state-of-the-art commercial models like GPT-4V and Gemini is great, their performance may be influenced by several factors in the process chain, such as API calls, data transmission, prompt/image filtering for responsible AI, model inference, and output filtering. Each company's unique design behind these API calls makes it challenging to pinpoint the root cause of observed biases (i.e., preference for text inputs). The paper struggles to provide conclusive answers when considering the entire system design, which could limit its findings.", "questions_to_authors": "How do we pinpoint the root cause of the observed model biases, considering the complex system design behind these API?", "extra_scores": {}}, {"review_id": "Yoyux1hBp2", "reviewer": "Reviewer_aqMK", "summary": "The paper proposes a benchmark, IsoBench, to assess the performance of the large multimodal models on the isomorphic representations of the input. For example, difference in the performance of the models on a task with image input versus the image caption (text) as an input. The paper finds that the LMMs perform way better for text representations than image representations. In addition, the paper proposes two methods IsoCombination and IsoScratchPad to improve the performance on their dataset.\n\nOverall, I believe that the paper is not well-motivated and the main contribution is not interesting or surprising for the practitioners. \n\n1. Primarily, I feel that the authors fail to understand that you make the task easier for a model when you represent an input as text instead of the image. Specifically, when you want to solve a task over an image input, the LMM has to correctly “perceive” objects, “read” text, “understand” layout and then perform relevant reasoning. For example, if an image has an embedded text “1+2=?” then the model needs to read the embedded text perfectly and then solve the problem of summation. Whereas, if you provide the text directly “1+2=?” as a prompt to the model, it just has to perform the reasoning part. Hence, it is expected that the model will be better on the text representation of the input instead of the visual representation of the input. \n2. The introduction never justifies why their assessment is useful for the practitioners or for any relevant downstream task. “Testing whether the multimodal fusion components of these models truly empower the model with the same capabilities when reasoning about images and text.” is the effect of the dataset, but the motivation to study this remains unclear.\n3. I do not like the connection with picture superiority effect. The way LMMs are designed (internet-scale data) is not the same as how humans function (limited knowledge but a lot of visual feedback in our daily lives).\n4. I like the fact that the authors collected the dataset for a large range of domains (game playing and science) and did a good job at identifying various kinds of tasks within each domain. \n5. IsoCombination is a contrived method which would not work in practice. Since their dataset already has isomorphic representations (either hand-written for science domain or using existing tools), it was easy for them to prompt GPT-4V with them. However, a user would not create isomorphic representations for the model. \n6. Image captioning is a very popular VL task. I do not like the fact that the authors repackage it as something novel and call it IsoScratchPad. This method is popularly known as augmented LLM setup (see MathVista) where image captions are provided to the LLM. It is well known that it does not work when the generated captions are not accurate (e.g., MathVista). As a result, this approach is quite worse than providing the model with the entire image itself on complex reasoning tasks.", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your comments and suggestions, and we are glad to see the you appreciate that IsoBench covers a large range of domains and exemplary tasks. \n\n### **On implicit bias (and level of difficulty) of different modalities**\nThe implicit bias is indeed what we aim to show. Even though different modalities are all processed from their tokenizations, in most multimodal LLMs, there is still a sensible performance gap between them—as IsoBench shows. We agree that the current pretraining paradigm may favor textual representations, and this is exactly the point we’re trying to make, and IsoBench is designed to showcase this phenomenon and serve as a benchmark for future improvements. \n\nWe respectfully disagree that tasks are easier with text inputs than with image inputs. The LLM must understand token relationships in text, similar to layout understanding in images. For instance, a chess board image and its Algebraic Notation Language (ANL) text representation both require the LLM to comprehend piece positions and relationships, involving similar reasoning processes.\n\n### **Why IsoBench is useful for practitioners**\nThe performance gap of visual inputs from that of text may point to a direction of improvement. Matching capabilities from visual representation to those from textual representation is a direction that can be pursued in the future by practitioners. \n\n### **Connection with Pictorial Superiority effect**\nTo clarify, we do not claim that VLMs should exhibit the same preference as humans. Nevertheless, (dis)analogies between neural models and cognitive science is common in ML literature (e.g. [1]), and we hoped that this example could help provide intuition.\n\n### **IsoCombination and IsoScratchPad are trivial and contrived**\nWe agree that IsoSP and IsoCB are simple strategies. This was our intention, and these methods are purposefully designed to be minimal to demonstrate possibilities for closing the vision-text gap. We also hope that others could use these initial strategies to develop more-sophisticated methods in future work.\n\nAs Reviewer CR6d pointed out, in many scientific fields, the images are complicated, researchers usually provide extra text inputs to help the model. This is also a common practice in scientific fields, where figures and tables in most papers are coupled with captions, providing descriptions and takeaways. We will make such motivation clearer in our next revision.\n\n[1] https://arxiv.org/abs/1710.09829", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Mentioned in the summary", "reasons_to_reject": "Mentioned in the summary", "questions_to_authors": "Mentioned in the summary", "extra_scores": {}}, {"review_id": "KZhuE1NFiP", "reviewer": "Reviewer_R3GK", "summary": "The contribution outlined in the passage is centered around exploring the impact of input modality on the capabilities of current foundation models. The authors introduce IsoBench, a benchmark dataset comprising problems from various domains like math, science, algorithms, and games. Each problem in IsoBench is presented with multiple representations, including visual, textual, and mathematical formats. This allows for a thorough examination of how different input modalities affect model performance. Their findings reveal that across various foundation models, there's a consistent preference for textual representations over visual ones. For instance, when evaluated on IsoBench problems, models like Claude-3 Opus, GPT-4 Turbo, and Gemini Pro perform significantly worse when provided with images instead of text. Claude-3 Opus shows a decrease of 28.66 points, GPT-4 Turbo 18.71 points, and Gemini Pro 14.87 points. To address these performance gaps, the authors propose two prompting techniques: IsoCombination and IsoScratchPad. These techniques aim to improve model performance by considering combinations of different input representations and facilitating translations between them. Overall, this work sheds light on the importance of input modality in influencing the capabilities of foundation models and offers strategies to enhance their performance in diverse problem-solving scenarios.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive feedback and comments. We are glad that you  find that IsoBench sheds light on the importance of input modalities and offers simple strategies to enhance their performance on image modalities. We are also grateful you find our study thorough, and the conclusions solid. \n\n### **Number of questions in IsoBench**\nFor the validation split of IsoBench, we have 1,887 samples in total. We will continue to enrich the diversity and scale of our datasets, and will open-source our dataset and code for data synthesis. We will further collect a test split and make it public soon.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The study is thorough, the conlcusion obtained is solid to me, which includes: Visual representations may be insufficient, Models cannot utilize low-level visual features for generation, Representation matters even for text. Both IsoCombination and IsoScratchPad are considered.", "reasons_to_reject": "I think the overall quality is good, I didn't find a concrete reason to reject. I will listen to other reviewers opinion.", "questions_to_authors": "How many question do you have in total for the benchmark? Currently it is not clear from the paper.", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper introduces IsoBench to evaluate multimodal foundation models on problems with isomorphic representations. 
Pro: 
- The paper is well-written and clear
- The paper focuses on important research questions and provides comprehensive evaluation and benchmark 

Cons: 
- Most issues of missing details are relatively minor and can be handled in revision. 
- There are some concerns about if the setting and assumptions are realistic and if there is new insight that can be learned from the findings. I would suggest the authors to address the comments in the revision.
Decision: Accept