IsoBench: Benchmarking Multimodal Foundation Models on Isomorphic Representations

Authors: deqing  fu
ruohao  guo
ghazal  khalighinejad
ollie  liu
  bhuwan dhingra
dani  yogatama
robin  jia
willie  neiswanger
georgia  tech λ duke
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-06 00:02:30

Abstract: Current foundation models exhibit impressive capabilities when prompted either with text only or with both image and text inputs. But do their capabilities change depending on the input modality? In this work, we propose **IsoBench**, a benchmark dataset containing problems from four major areas: math, science, algorithms, and games. Each example is presented with multiple **isomorphic representations** of inputs, such as visual, textual, and mathematical presentations. IsoBench provides fine-grained feedback to diagnose performance gaps caused by the form of the representation. Across various foundation models, we observe that on the same problem, models have a consistent preference towards textual representations. Most prominently, when evaluated on all IsoBench problems, Claude-3 Opus performs 28.66 points worse when provided with images instead of text; similarly, GPT-4 Turbo is 18.71 points worse and Gemini Pro is 14.87 points worse. Finally, we present two prompting techniques, *IsoCombination* and *IsoScratchPad*, which improve model performance by considering combinations of, and translations between, different input representations.

Main paper: 
 introduction: 
 On the heels of the large language model (LLM) revolution, we are currently witnessing a second revolution of multimodal foundation models. Exemplified by models like GPT-Text RepresentationTask: Maximum FlowImage RepresentationAdjacency Matrix:. Query Nodes: 4 and 8.0 1 0 0 0 0 0 0 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 1 0 1 0 0 1 0 0 0 0 0 0 0 0Adjacency Matrix:Source: ; Sink: 0 6 6 1 7 0 0 0 0 6 0 5 0 0 9 0 3 2 0 7 0 0 0 0 0 0 4Text RepresentationAnswer: 3 Answer: Odd Function -26.25x -220.97, -10.31 < x ≤ -4.61 75.84x + 249.44, -4 .61 < x ≤ -1.97 61. 92 -19.33x, -1.97 < x ≤ 8.36 32.12x -368.48, 8.36 < x ≤ 14.58 Task: Function Parity G = H = 0 1 0 1 0 1 0 0 1 0 0 0 0 0 1 0 0 0 0 1 1 1 0 0 1 0 1 0 0 0 0 0 0 0 1 0 0 0 0 0 1 0 1 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 0 0 1 1 0 0 0 0 0 1 0 1 0 0 0 0 0 1 1 0 0 0 0 0 0 0 0 0 0 0 1 1 0 0 0 0 0 0 0 0 1 0 0 0 1 0 0 0 1 0 0 0 1 0 0 0 Science Questions 4V (OpenAI, 2023b) , Claude (Anthropic, 2024) , and Gemini (Google, 2023), these models combine a large language model backbone with a vision encoder that enables them to accept either pure text inputs or a combination of text and images. The rapid rise of these multimodal models necessitates new benchmarks that accurately assess their capabilities.f (x) =In this work, we study whether these models exhibit the same capabilities when processing text and image inputs. Our key insight is that many problems consisting of both image and text prompts can be equivalently formulated with text-only prompts. By studying how well models do on both representations of the same problem, we can evaluate whether the multimodal fusion components of these models truly empower the model with the same capabilities when reasoning about images and text. While many benchmarks already exist for testing multimodal foundation models (Yue et al., 2023; Lu et al., 2024; Zhang et al., 2024a; Fan et al., 2024; Zhang et al., 2024b) , none of these measure models' performance discrepancies between semantically equivalent inputs in different modalities.We create IsoBench, a broad-coverage benchmark for evaluating multimodal foundation models on 10 different tasks with multiple isomorphic representations. IsoBench includes four major subjects: mathematical functions, algorithmic problems, science questions, and chess games. For each test example, IsoBench provides one image representation and one or more alternative textual representations. We evaluate a suite of multimodal foundation models, including GPT-41 , Gemini, and Claude-3, on IsoBench, and find that all tested models perform substantially better when given textual representations compared with visual representations. This bias in favor of textual representations runs counter to human cognition: humans are known to exhibit a picture superiority effect (Defeyter et al., 2009) , a preference towards visual representations, and humans generate visual images internally regardless of whether the required task is visual or verbal (Amit et al., 2017) .Finally, we design two simple mechanisms to further study the performance of multimodal foundation models on tasks with isomorphic representations: IsoCombination (IsoCB), which feeds multiple isomorphic representations to the model, and IsoScratchPad (IsoSP), which uses a scratchpad to translate between visual and text representations. We find that both methods improve aspects of multimodal models, with IsoCB improving model performance on graph algorithm problems by up to 9.4 points compared with the best single representation, and IsoSP improving performance on science problems by up to 14.4 points.In full, we summarize our contributions as follows:• We introduce IsoBench, a test dataset consisting of 1,887 samples2 spanning diverse domains such as discrete and applied mathematics, physics, chemistry, and chess. For each sample, we evaluate multiple isomorphic input representations containing the same information-one with a visual representation and others with domain-specific textual representations-to facilitate multimodal performance assessments.• We benchmark eight popular foundation models, and find that each of the tested multimodal models perform substantially better on text-only prompts than imagebased prompts, in contrast with known human preferences for images over text.• To bridge performance discrepancies between different input modalities, we introduce IsoCombination (IsoCB) and IsoScratchPad (IsoSP), two procedures that, respectively, fuse input modalities and transform visual input into textual representation at inference time. In certain settings, we find that IsoCB and IsoSP can improve the performance of multimodal foundation models by nearly 10 percentage points. 
isobench: 
 IsoBench is a collection of problems of the form (P,{R 1 , • • • , R m }), where P is a natural language instruction of task, and R i 's are different representations of the sample. Ideally, these representations are isomorphic to each other, i.e. for any pairs of input R i , R j , there exists a bijective function ϕ such that ϕ(R i ) = R j . As illustrated in Figure 1 , the prompts to foundation models share the same instructions except for the problem descriptions (where one of the representations R j is selected). We now detail the construction of IsoBench. Our first evaluation suite consists of problems in continuous mathematics. Foundation models are increasingly deployed as assistants for data science applications (Cheng et al., 2023) , and they must perform competently on plot understanding to provide helpful insights. We evaluate this capability by prompting foundation models to extract mathematical properties from continuous functions.Formats. We consider three input formats: Image, Text with L A T E X, and Text with Code. We first sample candidate functions and convert them to their L A T E Xand sympy representations. Then, we use matplotlib to plot the functions. In practice, it can be difficult to exactly map these images back to their corresponding functions, so the authors manually filtered data to ensure that the properties to be tested are clearly visible in the images.Tasks. We consider three tasks: classifying parity (i.e. even/odd/neither) functions, convex/concave functions, as well as counting the number of breakpoints in piecewise linear functions (either 2 or 3). These problems are designed to be visually trivial for humans to solve, but requires some mathematical maturity to answer analytically. We sample 128 samples for each class, culminating to 896 total samples. We refer readers to Appendix A.1 for additional details and Appendix B for sample prompts and responses.A critical area for enhancing the strategic capabilities of foundation models lies in understanding game strategies. We focus on two tasks in the game of chess: winner identification and chess puzzle solving. For both tasks, we use data from the Lichess database3 .Formats. For our chess game dataset, we consider four representations: Graphical Board (Image), Algebraic Notation Layout (ANL), Portable Game Notation (PGN), and Forsyth-Edwards Notation (FEN). The Graphical Board is an image of the chessboard, whereas ANL, PGN, and FEN are all text-based representations of the game state. For additional details, please see Appendix A.3.Tasks. We consider two tasks: winner identification from final board states and chess puzzles.In the winner identification task, the objective is to analyze a given representation of a chess game and determine the outcome: whether the game resulted in a win for White, a win for Black, or a draw. Our dataset for this task includes 257 games that lasted more than nine rounds in February 2024. In the chess puzzles task, a chess puzzle is given and the objective is to find the best first move. The dataset for this task comprises 200 puzzles. To provide concrete planning advice, foundation models need to reason algorithmically. We focus on graph algorithms, which are relevant in many realistic scenarios. For example, if asked to purchase flight tickets from Los Angeles to Vienna, foundation models need to (1) enumerate all possible choices with possible connection stops in between, and (2) check the lowest ticket prices among all choices. Fundamentally, the first task is a graph connectivity problem and the second is a weighted shortest path problem.In this section, we investigate three main graph algorithm problems with increasing complexity: graph connectivity, maximum flow, and graph isomorphism.For graph algorithms, we consider three representations: 1. Image, where graphs are visualized with networkx package with random styles; 2. Text with L A T E X, where the adjacency matrix is chosen to be the mathematical representation of graphs; 3. Text with story or description, where the graph problems are described as a story-telling version of the scenario (e.g., formulating the graph connectivity problem as determining whether it is possible to drive from one city to another).Tasks. We consider three tasks. Graph Connectivity requires deciding whether two query nodes are connected in an undirected graph. Maximum Flow requires computing the maximum flow one can send from a given source node to a given sink node, within a weighed directed graph. Finally, Graph Isomorphism requires determining whether two provided graphs with the same number of nodes are isomorphic (i.e., whether there exists a bijection between nodes that preserves all edges). Each task contains 128 problems with details in Appendix A.2.Formats. We consider two input formats for science questions: Image and Text. For the Image representation, each sample includes a textual question and multiple choices, along with a figure that provides additional context. To get the corresponding isomorphic Text inputs, one author manually wrote descriptions for each figure. 4 The annotator avoided introducing any extra reasoning or information beyond what is depicted in the figures. Isomorphic examples of image and text representation are provided in Appendix Table 7 .Tasks. We compiled a dataset consisting of 75 Chemistry and 75 Physics questions. We first selected questions from the ScienceQA (Lu et al., 2022) test set, choosing only questions whose answers rely on understanding the corresponding figures. This process resulted in a total of 50 chemistry and 60 physics questions. To enhance the diversity of the science dataset, we manually added 25 new chemistry questions and 15 new physics questions, introducing three new categories distinct from those in the ScienceQA dataset. The category distribution of our entire science dataset is detailed in Appendix Figure 4 , illustrating the wide range of science questions covered.G P T -4 T u r b o G P T -3 . 5 T u r b o G e m i n i P r o P a L M -2 C l a u d e 3 M i x t r a l -8 x 7 B L L a M a -2 -7 0 B L L a V a -1 . 5 -1 3 B R a n d o m G u eG P T -4 T u r b o G P T -3 . 5 T u r b o G e m i n i P r o P a L M -2 C l a u d e 3 M i x t r a l -8 x 7 B L L a M a -2 -7 0 B L L a V a -1 . 5 -1 3 B R a n d o m G u e 
performance analysis: 
 We evaluate IsoBench on API-access multimodal foundation models, such as GPT-4V (gpt-4-0125-preview and gpt-4-vision-preview), Claude 3 (claude-3-opus-20240229), and Gemini Pro (gemini-1.0-pro and gemini-pro-vision). We also benchmark on APIaccess single-modal large language models for reference, such as GPT-3.5 Turbo (OpenAI, 2023a, gpt-3.5-turbo-0125) and PaLM-2 (Anil et al., 2023, text-bison-001) .We evaluted representative open-source models such as Mixtral 8x7B (Jiang et al., 2024) , LLaMa-2 70B (Touvron et al., 2023) , and LLaVA-1.5 7B (Liu et al., 2023a) as well. All models we evaluate are instruction-tuned models and are zero-shot prompted. We present fine-grained results on these tasks in Tables 2 to 5 , and provide sample responses in Figures 5 to 17.Overview. We observe that, across all models that admit multimodal representations, language-only prompts almost always outperform vision-language prompts. On language prompts, API-access models exhibit strong reasoning capabilities (e.g. Figure 13 ). They generally outperform open-sourced models, which exhibit a tendency to generate a default answer regardless of input samples. We now delineate several intriguing observations. We present aggregated IsoBench results in Table 1 with details on more models in Table 8 .Vision models may be insufficient. In tasks that require explicit enumeration of objects such as breakpoint and chemistry (e.g. Figures 12 and 17 ), we observe significant performance degradation due to counting errors, which has been a long-standing problem for vision models (Liu et al., 2022) . In fact, leveraging a sub-optimal feature extractor for one modality may hurt the performance of a multimodal model; and the design of an optimal model requires nuanced understanding of the interaction effects between modalities (Liang et al., 2024) . Thomason et al. (2018) also showed unimodal models can even outperform their multimodal counterparts, and it suggests that models can often learn a shortcut only relying on language model part and perform weaker when forced to use only images. Overall, visual recognition errors, such as Figures 6, 8 and 12, are prevalent in failure cases, and often misdirect natural language generation towards incorrect conclusions.Perhaps surprisingly, vision-language foundation models are far from perfect for convexity problems, which mostly feature simple, smooth curves, even though curve detectors (Olah et al., 2020) are among the most observed and well-studied phenomena in the mechanistic understanding of vision models. This inconsistency suggests that, even if performing certain tasks is within a vision model's capabilities, the current multimodal fusion scheme is insufficient for eliciting them. We posit that popular approaches such as image-tokenization only offer a coarse-grained representation that summarizes high-level features, and may be unsuitable for detailed analysis such as plot and chart understanding.LLM backbones may not be exempt from blame. Recent works (Gonen et al., 2022; Razeghi et al., 2022; Han & Tsvetkov, 2022) have reported language models to prefer textual formats that are more common in the pre-training data, which echo our observation in performance variability between different language-only inputs. While formats do not fully explain this discrepancy, we hypothesize that the imbalance between visual and input data, and in general the sub-optimal robustness of LLMs in the face of less common distribution can contribute to the performance gap between visual and language inputs.Lastly, we observe generated responses of vision-language prompts to be cursory compared to those of language-only prompts; and they sometimes contain a direct answer instead of a reasoning process, despite being instructed to perform chain-of-thought reasoning. To mitigate this difference, we discuss two simple strategies that aims to augment a visionlanguage prompt with better reasoning capabilities of language prompts, presented next.this is a textual representation of the chess game's moves in standard algebraic notation. PGN is widely used for the recording of games, allowing for both humans and computers to read the game history. It captures the entire sequence of moves from the beginning to the current state or the end of the game. 
isocombination and isoscratchpad: 
 In many scientific fields, the images are complicated, researchers usually provide extra text inputs to help the model. This is also a common practice in scientific fields, where figures and tables in most papers are coupled with captions (for example, this paper), providing descriptions and takeaways. Motivated by these and based on the observations made above on IsoBench, we can design two simple and deliberately contrived mechanisms to further study the performance of multimodal foundation models on tasks with isomorphic representations: IsoCombination (IsoCB) and IsoScratchPad (IsoSP). IsoCB feeds multiple isomorphic representations to the multimodal model simultaneously in order to see the combined effect of using multiple representations on performance. Alternatively, IsoSP uses a scratchpad to explicitly translate from visual to text representations (and then uses this text representation to solve a task), taking advantage of the empirically higher-performing text representations for visual data. In particular, for IsoSP we employ a two-step prompting strategy: initially, the model receives a prompt featuring a visual representation, which it is tasked to translate into a textual format; then, the model is prompted with this generated text representation to predict the output. We illustrate both methods in Figure 3 .We show results for IsoCB and IsoSP in Table 6 , along with results for image-only and text-only representations. Interestingly, we find that IsoSP, which converts from image to G = H = 0 1 0 1 0 1 0 0 1 0 0 0 0 0 1 0 0 0 0 1 1 1 0 0 1 0 1 0 0 0 0 0 0 0 1 0 0 0 0 0 1 0 1 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 0 0 1 1 0 0 0 0 0 1 0 1 0 0 0 0 0 1 1 0 0 0 0 0 0 0 0 0 0 0 1 1 0 0 0 0 0 0 0 0 1 0 0 0 1 0 0 0 1 0 0 0 1 0 0 0 The image displays two graphs.The Left Graph consists of eight nodes numbered 0 to 7, with the following connections:-0 is connected to 1, 3, and 5.-1 is connected to 6.-2 is connected to 3, 4 and 5.-7 is an isolated node.The Right Graph consists of eight nodes numbered 0 to 7, with the following connections:-0 is connected to 6 and 7.-1 is connected to 2 and 3.-2 is connected to 3.-4 is connected to 6, 7.-5 is an isolated node.Instruction about the Task: Graph IsomorphismModel Scratch PadThe image displays two graphs.The Left Graph consists of eight nodes numbered 0 to 7, with the following connections:-0 is connected to 1, 3, and 5.-1 is connected to 6.-2 is connected to 3, 4 and 5.-7 is an isolated node.The Right Graph consists of eight nodes numbered 0 to 7, with the following connections: -0 is connected to 6 and 7.-1 is connected to 2 and 3.-2 is connected to 3.-4 is connected to 6, 7.-5 is an isolated node.No, the two graphs G (Left Graph) and H (Right Graph) are not isomorphic.Explanation:In graph G, there are two nodes (0 and 2) that each have a degree of 3, meaning they are connected to three other nodes. However, in graph H, there is no node with a degree of 3. The nodes in graph H have the following degrees:-Nodes 0, 1, 2, 3, 4, 6, 7 have a degree of 2 -Node 5 has a degree of 0For two graphs to be isomorphic, they must have the same number of nodes with each degree. Since graph G has two nodes with a degree of 3 and graph H has no nodes with a degree of 3, the two graphs cannot be isomorphic.Furthermore, graph H has a node (Node 3) that is connected to two nodes (Node 1 and Node 2) which are also connected to each other, forming a triangle. In graph G, there is no such triangle structure.These structural differences between the two graphs prove that they are not isomorphic, as there is no bijection that can preserve the edge relationships between the nodes while mapping one graph to the other. text, typically outperforms image-only representations (and thus it could potentially be used to improve performance given only image representations for some tasks). However, compared to direct text representations, IsoSP tends to fall short. Our manual review of IsoSP and text results provides a possible explanation: the model has difficulty understanding images, especially in tasks that require counting items or comparing numbers visually. The larger the performance gap between IsoSP and Text representations, the harder it becomes for the model to understand images. Another possible reason for IsoSP's underperformance relative to Text representations could be that, the model-generated interpretation of images tends to miss critical information for reasoning. For example, to determine which pair of magnets in the given image has a larger magnetic force in a physics question, it is essential to consider both the distance between the magnets and their sizes. Yet, the descriptions generated by models like Claude-3 recognize only the distance, neglecting the magnet sizes, and thus leading to an incorrect answer. Additionally, we find that IsoCB, which combines representations, typically outperforms image-only representations, and even text-only representations on certain tasks.this is a compact way to describe the current position of a game of chess in textual format. 
related works: 
 Vision-Language Foundation Models. There has been a plethora of recent developments in vision-language foundation models; they can be broadly categorized by their methods for representing visual modalities. A representative approach involves tokenizing visual inputs to be jointly trained with language inputs (Yu et al., 2023; Google, 2023; McKinzie et al., 2024; Chen et al., 2022; 2023; Team, 2024, inter alia) . Another line of work processes continuous visual features by directly projecting them to the language embedding space via a learnable function (Liu et al., 2023b; a; 2024a; Bavishi et al., 2023) . While the design choices of some API-access models (OpenAI, 2023b; Anthropic, 2024; Google, 2023; Reka, 2024) remain largely unknown, most performant models use early fusion, the practice of integrating visual and language features at the input level of an autoregressive model. At the core of these design choices is the hardness in representing visual features, which has been reported by several early studies (McKinzie et al., 2024) to be the key bottleneck towards performant vision-language foundation models. IsoBench aims to supplement prior works, which have mostly focused on analyzing modeling methods, with a data-centric view by measuring performance discrepancies incurred by multimodal representations.Multimodal Datasets. Many vision-language datasets have recently been curated to assess the capabilities of multimodal foundation models (Yue et al., 2023; Lu et al., 2024; Zhang et al., 2024a; Fan et al., 2024; Zhang et al., 2024b) . They often consist of a large collection of problems with varying levels of difficulty and usage of the visual inputs. As a result, they are suitable for assessing VLMs as a whole, but cannot attribute these performances to capabilities of their LLM backbones and improvements from downstream vision-language training. While several contemporaneous works (Fan et al., 2024; Zhang et al., 2024a; b) introduce a language-only subset of their vision-language prompts, they focus primarily on assessing certain areas of foundation models, such as deductive reasoning and problems of beyond-polynomial complexity. IsoBench is the first of its kind that offers a holistic and fine-grained analysis of the performance gap induced by variations in input modalities. For instance, a contemporaneous work, MathVerse (Zhang et al., 2024a) , studies redundant, but not necessarily equivalent input representations on math problems whilst IsoBench studies the impact of isomorphic representations on a broader set of tasks.Similar to what IsoBench observes on multimodal foundation models' sensitivity to input modalities, uni-modal language models also show sensitivity to input perturbation. For example, LLMs are shown to be sensitive to subtle changes in zero-shot and few-show settings (Sclar et al., 2023; Chang & Jia, 2023) . Meanwhile, the sensitivity of Transformers, the backbone of foundation models, is also broadly studied (Bombari & Mondelli, 2024; Bhattamishra et al., 2023; Vasudeva et al., 2024; Kong et al., 2024) . To address the sensitivity issues, methods including prompt design (Yoo et al., 2021; Mishra et al., 2021; Le et al., 2023; Liu et al., 2023c) and prompting-based training strategies (Jain et al., 2023; Guo et al., 2023) have been employed. IsoBench is one of the pioneers to study model sensitivity to input representations and modalities.Foundation Models for Algorithmic Reasoning Recent work has shown that Transformers, the backbone of multimodal foundation models, are capable of implementing various algorithms after fine-tuning on particular tasks (Khalighinejad et al., 2023; Garg et al., 2022; von Oswald et al., 2022; Hanna et al., 2023; Fu et al., 2023; Lee et al., 2023; Zhou et al., 2024) or with proper prompting schemes (Wei et al., 2023; Yao et al., 2023; Liu et al., 2024b) . However, the extent to which foundation models are capable of solving problems involving complex reasoning, algorithms, or scientific awareness remains unclear. 
conclusion: 
 In this paper, we developed IsoBench, a broad-coverage benchmark for evaluating multimodal foundation models on a varied set of quantitative tasks, where each example input is provided with multiple isomorphic representations. IsoBench includes four major subjects (mathematical functions, algorithmic problems, science questions, and chess games) in a dataset consisting of over 1,630 examples. Within each domain, we evaluated multiple input representations-one with visual input and others with domain-specific, isomorphic textual inputs-to facilitate multimodal performance assessments. We found that all tested models perform substantially better on text-only prompts than image-based prompts, in contrast with known human preferences for images over text. To bridge performance discrepancies between different input modalities, we also introduced IsoCombination and IsoScratchPad, two procedures that, respectively, fuse input modalities and transforms visual input to textual representation at inference time. In full, we hope that IsoBench can provide fine-grained feedback to diagnose performance gaps caused by the form of a given input representation, and help us better understand how the capabilities of multimodal foundation models change depending on the input modality that is provided.A H are sampled individually. To avoid models using simple shortcuts such as counting the number of nodes, graphs G and H in each pair are guaranteed to have the same number of nodes and the same number of edges.Images for algorithm problems are generated with 300 DPI in JPEG format. 
: 
 This expression is not equal to the original function , indicating that the function is not even. Additionally, the expression is not the negative of the original function, indicating that the function is not odd. Therefore, is neither even nor odd.-x f(-x) = 7(-x) 6 -3(-x) 5 + (-x) 2 -21.76 2(-x) 6 + 4(-x) 4 + 3(-x) 2 + 11.34 = 7x 6 + 3x 5 + x 2 -21.76 2x 6 + 4x 4 + 3x 2 + 11.34  
 
 Appendix: 1. Graphical Board: the model is prompted with an instruction of the problem to solve and the visual representation of the chess board and its pieces in a graphical format (PNG) is given.2. Algebraic Notation Layout (ANL): this refers to a text-based representation of the current position of pieces on the chess board, using algebraic notation for the squares.The task is to analyze a given chess game representation and identify the winner: whether the game resulted in a win for White or Black, or if it ended without a winner. Our dataset for this task contains the games that lasted more than nine rounds in February 2024.For this task, we consider chess puzzles that consist of a chess position, along with a sequence of optimal moves necessary to resolve the puzzle. In our evaluation, the primary objective is to predict the initial move.Images for Chess are the same as Lichess, and are converted to JPEG formats. Images for Chess are the same as ScienceQA, and are converted to JPEG formats.We show sample responses for Math, Algorithms, Games, and Science Problems in the figures below, for multiple multimodal foundation models.You are given a real-valued, scalar function f (x).YOUR TASK is to determine whether f (x) is an even function, an odd function, or neither.• Definition of an even function: A function such thatf (x) = f (-x)where the value remains unchanged if the sign of the independent variable is reversed.• Definition of an odd function: A function such thatf (-x) = -f (x)where the sign is reversed but the absolute value remains the same if the sign of the independent variable is reversed.• A function is neither even nor odd if it does not satisfy either condition.Here is the expression of f (x):f (x) = 7x 6 -3x 5 + x 2 -21.76 2x 6 + 4x 4 + 3x 2 + 11.34Respond with 'even', 'odd', 'neither' first on whether the function f(x) is even, odd, or neither, based on the definitions and your observation of the function. Please parse it to JSON format. Put the result (even, odd, or neither) under key 'answer', and put the explanations and reasonings under key 'explanation' (Do not summarize, paraphrase or change meanings).to determine whether the function f (x) = 7x 6 -3x 5 + x 2 -21.76 2x 6 + 4x 4 + 3x 2 + 11.34 is even, odd, or neither, we substitute -x for x and compare the resulting expression to the original function. for an even function, we should find that f (x) = f (-x)and for an odd function, we should find that f (-x) =f (x)substituting -x into the function, we getf (-x) = 7(-x) 6 -3(-x) 5 + (-x) 2 -21.76 2(-x) 6 + 4(-x) 4 + 3(-x) 2 + 11.34 = 7x 6 + 3x 5 + x 2 -21.76 2x 6 + 4x 4 + 3x 2 + 11.34this expression is not equal to the original function f (x), indicating that the function is not even. additionally, the expression is not the negative of the original function, indicating that the function is not odd. the presence of the x 5 term, which changes sign when x is replaced with -x, but does not satisfy the condition for an odd function due to the other terms, is a key factor in determining that the function is neither even nor odd. YOUR TASK is to determine whether f(x) is an even function, an odd function, or neither.• Definition of an even function: A function such thatf (x) = f (-x)where the value remains unchanged if the sign of the independent variable is reversed.• Definition of an odd function: A function such thatf (-x) = -f (x)where the sign is reversed but the absolute value remains the same if the sign of the independent variable is reversed• A function is neither even nor odd if it does not satisfy either definitions.Respond with 'even', 'odd', 'neither' first on whether the function f(x) is even, odd, or neither, based on the definition above. For any of your conclusions (even, odd, or neither), please explain in details based on your observation of the plot. Answer (start with 'even', 'odd', or 'neither'):The plot of the function f (x) shows symmetry about the y-axis. This means that for every point (x, f (x)) on the right side of the y-axis, there is a corresponding point (-x, f (x)) on the left side of the y-axis with the same function value. This symmetry indicates thatf (x) = f (-x)for all x in the domain of the function, which is the definition of an even function. Therefore, based on the plot, f (x) is an even function.Figure 6 : Sample response from GPT-4V to the same parity problem as in Figure 5 . GPT-4V cannot distinguish the nuanced asymmetry at the top-rigth and top-left corners of the plot. We observe the same failure mode for Gemini-Pro and LLaVA.You are given a real-valued, scalar function f (x).YOUR TASK is to determine whether f (x) is an even function, an odd function, or neither.• Definition of an even function: A function such thatf (x) = f (-x)where the value remains unchanged if the sign of the independent variable is reversed.• Definition of an odd function: A function such thatf (-x) = -f (x)where the sign is reversed but the absolute value remains the same if the sign of the independent variable is reversed.• A function is neither even nor odd if it does not satisfy either condition.Here is the expression of f (x):f (x) = 4x 3 x 2 -1 16x 10 + 8x 8 + 10x 2 + 13.98Respond with 'even', 'odd', 'neither' first on whether the function f(x) is even, odd, or neither, based on the definitions and your observation of the function. Please parse it to JSON format. Put the result (even, odd, or neither) under key 'answer', and put the explanations and reasonings under key 'explanation' (Do not summarize, paraphrase or change meanings).to determine whether the function f (x) = 4x 3 x 2 -1 16x 10 + 8x 8 + 10x 2 + 13.98 is even, odd, or neither, we substitute -x for x and compare the result to the original function. substituting -x into the function givesf (-x) = 4(-x) 3 (-x) 2 -1 16(-x) 10 + 8(-x) 8 + 10(-x) 2 + 13.98 = -4x 3 x 2 -1 16x 10 + 8x 8 + 10x 2 + 13.98, which simplifies tof (x). this means that the function satisfies the condition for being an odd function, which isf (-x) = -f (x). the function does not satisfy the condition for being an even function, which would requiref (x) = f (-x). therefore, the function is odd. YOUR TASK is to determine whether f(x) is an even function, an odd function, or neither.• Definition of an even function: A function such thatf (x) = f (-x)where the value remains unchanged if the sign of the independent variable is reversed.• Definition of an odd function: A function such thatf (-x) = -f (x)where the sign is reversed but the absolute value remains the same if the sign of the independent variable is reversed • A function is neither even nor odd if it does not satisfy either definitions.Respond with 'even', 'odd', 'neither' first on whether the function f(x) is even, odd, or neither, based on the definition above. For any of your conclusions (even, odd, or neither), please explain in details based on your observation of the plot. Answer (start with 'even', 'odd', or 'neither'):GPT-4 RESPONSE neither The function f (x) depicted in the plot does not satisfy the conditions for being an even function nor an odd function. For a function to be even, the graph of the function should be symmetric with respect to the y-axis, meaning that the part of the graph in the positive x-region should be a mirror image of the part in the negative x-region. This function does not exhibit such symmetry; the shapes and values of the function on either side of the y-axis are not mirror images of each other. Similarly, for a function to be odd, the graph should have rotational symmetry about the origin, which means that rotating the graph 180 degrees around the origin should result in the same graph. This is not the case for the given function either, as the positive and negative sides of the graph do not match when one side is rotated 180 degrees. Since the function does not meet the criteria for being even or odd, we conclude that the function is neither even nor odd.Figure 8 : Sample response from GPT-4V to the same parity problem as in Figure 7 . GPT-4V cannot reason with the symmetry at origin. We observe the same failure mode for all other vision models.You are given a real-valued, scalar function f(x). YOUR TASK is to determine whether f(x) is an convex function or an concave function• Definition of a convex function: A function such that for all x, y, and 0 <= t <= 1f (tx + (1 -t)y) ≤ t f (x) + (1 -t) f (y)• Definition of a concave function: A function such that for all x, y, and 0 <= t <= 1f (tx + (1 -t)y) ≥ t f (x) + (1 -t) f (y)Here is the expression of f (x), defined for all x > 0:f (x) = 0.31x -1.73 log (x) + 1.13 |x| -0.63Respond with 'convex' or 'concave' first on whether the function f (x) is convex or concave, based on the definitions and your observation of the function. Please parse it to JSON format. Put the result (convex or concave) under key 'answer', and put the explanations and reasonings under key 'explanation' (Do not summarize, paraphrase or change meanings).The given function f (x) is a convex function. This can be determined by examining the individual terms in the expression:1. The term 0.31x is a linear function, which is both convex and concave.2. The term -1.73 log(x) is a concave function, as the logarithm function is concave for x > 0. 3. The term 1.13|x| is a convex function, as the absolute value function is convex.4. The constant term -0.63 does not affect the convexity or concavity of the function.Since the sum of a convex function and a concave function is not guaranteed to be either convex or concave, we need to consider the dominant term. In this case, the convex term 1.13|x| grows faster than the concave term -1.73 log(x) as x increases. Therefore, for sufficiently large values of x, the convex term will dominate, making the overall function f(x) convex. YOUR TASK is to count the number of breakpoints in the plot of f (x). A breakpoint refers to a point on the function's domain at which the function changes its slope.Here is the expression of f (x):f (x) =         28.88x + 375.32 for -14.9 < x ≤ -9.53 -18.18x -73.3 for -9.53 < x ≤ 1.47 54.18x -179.55 for 1.47 < x ≤ 5.16 277.33 -34.43x for 5.16 < x ≤ 10.95 Please parse it to JSON format. Put the result (in Arab digits) under key 'answer', and put the explanations and reasonings under key 'explanation' (Do not summarize, paraphrase or change meanings).The function f (x) is a piecewise function defined over 4 intervals. The endpoints of these intervals are -14. 9, -9.53, 1.47, 5.16, and 10.95 . Ignoring the leftmost and rightmost endpoints as per the instructions, we consider the points -9.53, 1.47, and 5.16. At each of these points, the function changes its slope (i.e., the coefficient of x changes). Therefore, these three points are breakpoints of the function f (x). Thus, the function f (x) has 3 breakpoints within its domain, excluding the endpoints.Figure 11 : Sample response from Claude 3 to a breakpoint problem in IsoBench. Claude 3 was able to identify the exact locations of the breakpoints. We observe similar behaviors for all API-access foundation models. The first breakpoint occurs at x = 0. As x approaches 0 from the negative side, the function has a positive slope. However, as x crosses 0 and becomes positive, the function changes to a negative slope. This change in slope direction at x = 0 indicates a breakpoint. The second breakpoint is located at approximately x = 5. From x = 0 to x = 5, the function has a negative slope. But as x exceeds 5, the function's slope becomes positive again. This transition from a negative to a positive slope around x = 5 signifies another breakpoint. The left and right endpoints of the function's domain are not considered breakpoints according to the given definition. Therefore, focusing only on the interior of the domain, the function f (x) contains exactly 2 breakpoints based on the changes in slope observed in the provided plot. To solve the maximum flow problem using the given adjacency matrix, we can use the Ford-Fulkerson algorithm with the Edmonds-Karp implementation. The algorithm finds the maximum flow by repeatedly finding augmenting paths from the source to the sink and updating the residual graph until no more augmenting paths can be found. Given: -Adjacency matrix: [0 7 4 3 2] [0 0 0 0 7] [0 6 0 0 4] [0 5 6 0 6] [0 0 0 0 0] -Source node: 0 -Sink node: 4Step 1: Initialize the flow matrix with all values set to 0.Step 2: Find an augmenting path from the source to the sink using BFS.-Augmenting path 1: 0 -> 1 -> 4 Residual capacity: min(7, 7) = 7 Update the flow matrix and residual graph. You are given a FEN representation of a chess puzzle for which a sequence of unique best moves is determinable (e.g. sequences of moves leading to a forced checkmate).• Definition of the Chess Puzzle: In a chess puzzle, you are required to make a series of optimal moves leading to checkmate, starting from the given position.• YOUR TASK is to predict THE FIRST MOVE that should be played given this board setup.• Your answer should first specify the move in Algebraic Coordinate Notation (e.g., "d2d1", "e5a1", "c4f4"). Afterward, provide your reasoning for why you chose that as your first move.FEN : r2qk2r/ppp2ppp/2n2n2/6B1/1b2N1b1/5N2/PPP1PPPP/R2QKB1RwKQkq -68 Answer (start with the first move in Algebraic Coordinate Notation like "e2e4" or "d7d5"):GPT-4 RESPONSE First Move: "a2a3" Reasoning: In this position, given by the FEN "r2qk2r/ppp2ppp/2n6/6B1/1b2N1b1/5N2/PPP1PPPP/R2QKB1R w KQkq -6 8", it is White's turn to move. The board setup reveals a significant tension in the center and around the kingside, particularly with both bishops (the Black bishop on g4 and the White bishop on g5) exerting pressure. The immediate threat to address is the pin of the White knight on f3 by the Black bishop on g4. Moving the h3 pawn to attack the bishop would be tempting to relieve the pin on the knight; however, it weakens White's kingside structure and can be met by h5 from Black, securing the bishop's position. A crucial observation here is the Black bishop on b4, which is currently pinning the White knight on e4, preventing it from moving without losing the queen. The first move, "a2a3", is aimed at challenging this bishop. This move forces the Black bishop to make a decision; it can either retreat or be captured if it remains on the b4 square. Retreating the bishop to a5 or c5 does not alleviate Black's positional difficulties significantly, and capturing on c3 would lead to bxc3, strengthening White's center and opening lines for the rook and bishop. This move is selected over other aggressive or defensive options because it immediately addresses a piece that contributes to pinning and pressure without compromising White's position or losing material. By initiating this challenge, White aims to disrupt Black's coordination and gain a tempo to improve their position, possibly by preparing to castle, unpin the knight on f3, or increase control in the center. This move also preserves the option of castling kingside for White, keeping the king's safety as a priority. You are given a visual representation of a chess puzzle for which a sequence of unique best moves is determinable (e.g. sequences of moves leading to a forced checkmate).• Definition of the Chess Puzzle: In a chess puzzle, you are required to make a series of optimal moves leading to checkmate, starting from the given position.• YOUR TASK is to predict THE FIRST MOVE that should be played given this board setup.• Your answer should first specify the move in Algebraic Coordinate Notation (e.g., "d2d1", "e5a1", "c4f4"). Afterward, provide your reasoning for why you chose that as your first move.Answer (start with the first move in Algebraic Coordinate Notation like "e2e4" or "d7d5"):The best move for white to play in this position is Bg5xf6. The reason behind this move is that by capturing the knight on f6 with the white bishop, white is achieving multiple objectives: White is removing a defender of the d7 square, which could potentially be an entry point for the white queen or other pieces if the knight on c6 moves. The g7 pawn is pinned by the white bishop on b2, and cannot recapture on f6 without losing material since the rook on h8 would be left undefended. This move also opens up the e-file for the white queen and rook to become more active, possibly leading to an attack on the black king, who is still in the center of the board and unable to castle due to the king's move earlier. Thus, Bg5xf6 appears to be the strongest move in the position, setting up threats and potential for further penetration into black's position. Throughout this paper, we use GPT-4 as a general term when referring to both the GPT-4 large language model, and the GPT-4V multimodal foundation model.The IsoBench dataset (www.huggingface.co/datasets/isobench/IsoBench) currently consists of 1,887 examples across four domains, though the construction of a larger dataset is currently underway.https://lichess.orgIn preliminary experiments, we leveraged GPT-4 to generate image descriptions. However, we observed that these descriptions often include reasoning and additional knowledge, leading to biased "improved" performance. 


Decision: The paper introduces IsoBench to evaluate multimodal foundation models on problems with isomorphic representations. 
Pro: 
- The paper is well-written and clear
- The paper focuses on important research questions and provides comprehensive evaluation and benchmark 

Cons: 
- Most issues of missing details are relatively minor and can be handled in revision. 
- There are some concerns about if the setting and assumptions are realistic and if there is new insight that can be learned from the findings. I would suggest the authors to address the comments in the revision.

Reviews: review: paper_summary: paper_summary: The paper introduces a new benchmark dataset, IsoBench, which aims to evaluate multimodal foundation models on problems with isomorphic representations. The authors have put effort into constructing the dataset and conducting evaluations on various models. The paper is generally well-structured and easy to follow, with the use of tables, figures, and examples to illustrate the findings.

While the work addresses an interesting research problem and provides some insights into the performance of multimodal foundation models, there are several concerns that limit its overall significance and originality. The task setting may not fully reflect real-world applications, and there are issues with data annotation and missing details that could affect the reliability of the benchmark. The proposed techniques, IsoCombination and IsoScratchPad, are relatively simple and do not provide substantial novel insights. Addressing these concerns and expanding the benchmark to include more challenging and diverse figures could strengthen the contribution of this work to the field of multimodal understanding and reasoning.
strength_weakness: strength_weakness: ### (1) Task settings may not reflect real applications

For example, the science questions featured in IsoBench are relatively easy to describe in natural language. However, in real-world applications, many scientific figures are very complex and difficult to capture fully in natural language descriptions. These figures often serve as complementary information alongside the text. The benchmark may not adequately represent the challenges faced in real-world multimodal understanding tasks.

### (2) Data annotation concerns and missing details

The authors mention that a single author manually wrote descriptions for each figure in the science questions, with instructions to avoid introducing extra reasoning or information. However, there are several issues with this approach:
- Relying on a single annotator may introduce bias and inconsistencies in the data.
- Annotating high-quality descriptions for scientific images can be challenging and requires domain expertise. The paper does not provide information about the annotator's qualifications or background.
- There is no mention of any quality control measures or validation processes to ensure the accuracy and consistency of the annotations.

### (3) Missing data construction details

The paper lacks important details regarding the construction of the benchmark dataset. For example, the resolution of the image representations is not specified. Low-resolution images may result in poor understanding by current VLMs. Without this information, it is difficult to assess the validity and reliability of the benchmark.

### (4) Naive baselines 

The two baselines proposed in the paper, IsoCombination and IsoScratchPad, are relatively simple and do not provide significant insights. It would be more valuable to see novel and insightful models that are inspired by the findings from the evaluations and analysis. The authors could explore more sophisticated approaches to address the performance discrepancies between different input modalities.

### (5) Related work

A superficial discussion on a very closely related work: MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?
questions: questions: The paper introduces IsoBench, an innovative benchmark dataset that sheds light on the performance of multimodal foundation models across various isomorphic representations. The authors' efforts in creating this dataset and conducting comprehensive evaluations are commendable and provide valuable insights for the research community.

However, to further strengthen the paper, I would encourage the authors to consider expanding the benchmark to include more challenging and diverse figures that better reflect real-world applications. Providing additional details on the data annotation process, such as the background of the annotators and quality control measures, would also help bolster confidence in the dataset's reliability.

Moreover, while the proposed IsoCombination and IsoScratchPad techniques offer interesting avenues for exploration, developing more advanced models that build upon the findings of this study could yield even more impactful insights.
paper_summary: paper_summary: This paper proposes an interesting benchmark called IsoBench, designed to evaluate multimodal foundation models on tasks with isomorphic representation. It provides key observations on existing models and introduces prompting techniques (IsoCombination and IsoScratchPad) to integrate input modalities effectively. 

The experiments offer empirical findings of the capabilities of existing models when processing different modal inputs.
strength_weakness: strength_weakness: - The paper assumes that multimodal foundation models should possess strong visual capabilities, akin to human cognition. However, the path toward Artificial General Intelligence (AGI) remains undefined regarding this aspect. That is, emulating human visual cognition could be advantageous but may also be limited by human capabilities. These models might surpass human cognition in certain areas that are beyond human perception. Thus, a preference for text representation is not inherently negative or positive. The focus should be on whether the models function effectively, irrespective of the representation type.

- While evaluating state-of-the-art commercial models like GPT-4V and Gemini is great, their performance may be influenced by several factors in the process chain, such as API calls, data transmission, prompt/image filtering for responsible AI, model inference, and output filtering. Each company's unique design behind these API calls makes it challenging to pinpoint the root cause of observed biases (i.e., preference for text inputs). The paper struggles to provide conclusive answers when considering the entire system design, which could limit its findings.
questions: questions: How do we pinpoint the root cause of the observed model biases, considering the complex system design behind these API?
paper_summary: paper_summary: The paper proposes a benchmark, IsoBench, to assess the performance of the large multimodal models on the isomorphic representations of the input. For example, difference in the performance of the models on a task with image input versus the image caption (text) as an input. The paper finds that the LMMs perform way better for text representations than image representations. In addition, the paper proposes two methods IsoCombination and IsoScratchPad to improve the performance on their dataset.

Overall, I believe that the paper is not well-motivated and the main contribution is not interesting or surprising for the practitioners. 

1. Primarily, I feel that the authors fail to understand that you make the task easier for a model when you represent an input as text instead of the image. Specifically, when you want to solve a task over an image input, the LMM has to correctly “perceive” objects, “read” text, “understand” layout and then perform relevant reasoning. For example, if an image has an embedded text “1+2=?” then the model needs to read the embedded text perfectly and then solve the problem of summation. Whereas, if you provide the text directly “1+2=?” as a prompt to the model, it just has to perform the reasoning part. Hence, it is expected that the model will be better on the text representation of the input instead of the visual representation of the input. 
2. The introduction never justifies why their assessment is useful for the practitioners or for any relevant downstream task. “Testing whether the multimodal fusion components of these models truly empower the model with the same capabilities when reasoning about images and text.” is the effect of the dataset, but the motivation to study this remains unclear.
3. I do not like the connection with picture superiority effect. The way LMMs are designed (internet-scale data) is not the same as how humans function (limited knowledge but a lot of visual feedback in our daily lives).
4. I like the fact that the authors collected the dataset for a large range of domains (game playing and science) and did a good job at identifying various kinds of tasks within each domain. 
5. IsoCombination is a contrived method which would not work in practice. Since their dataset already has isomorphic representations (either hand-written for science domain or using existing tools), it was easy for them to prompt GPT-4V with them. However, a user would not create isomorphic representations for the model. 
6. Image captioning is a very popular VL task. I do not like the fact that the authors repackage it as something novel and call it IsoScratchPad. This method is popularly known as augmented LLM setup (see MathVista) where image captions are provided to the LLM. It is well known that it does not work when the generated captions are not accurate (e.g., MathVista). As a result, this approach is quite worse than providing the model with the entire image itself on complex reasoning tasks.
strength_weakness: strength_weakness: Mentioned in the summary
questions: questions: Mentioned in the summary
paper_summary: paper_summary: The contribution outlined in the passage is centered around exploring the impact of input modality on the capabilities of current foundation models. The authors introduce IsoBench, a benchmark dataset comprising problems from various domains like math, science, algorithms, and games. Each problem in IsoBench is presented with multiple representations, including visual, textual, and mathematical formats. This allows for a thorough examination of how different input modalities affect model performance. Their findings reveal that across various foundation models, there's a consistent preference for textual representations over visual ones. For instance, when evaluated on IsoBench problems, models like Claude-3 Opus, GPT-4 Turbo, and Gemini Pro perform significantly worse when provided with images instead of text. Claude-3 Opus shows a decrease of 28.66 points, GPT-4 Turbo 18.71 points, and Gemini Pro 14.87 points. To address these performance gaps, the authors propose two prompting techniques: IsoCombination and IsoScratchPad. These techniques aim to improve model performance by considering combinations of different input representations and facilitating translations between them. Overall, this work sheds light on the importance of input modality in influencing the capabilities of foundation models and offers strategies to enhance their performance in diverse problem-solving scenarios.
strength_weakness: strength_weakness: I think the overall quality is good, I didn't find a concrete reason to reject. I will listen to other reviewers opinion.
questions: questions: How many question do you have in total for the benchmark? Currently it is not clear from the paper.

Comments: title: Thank you for your comments
New response
Reply to Rebuttal
comment: Thank you for engaging in an in-depth discussion with us during the rebuttal period!

## **On implicit bias (and level of difficulty) of different modalities**

> I still feel that the task of comprehending the piece positions and relationships is easier in the text modality. Assume that the location of the rook in Figure 2 is d8 – this location is explicitly mentioned in the text representation by three dots before ‘r’. The text position encodings can easily tell that the location of the rook is 4th from the left. In addition, the text representation explicitly mentions the character ‘r’ that disambiguates the presence of the rook itself.

We just want to point out an interesting observation that we think gives some evidence against this point. In our experiments, the best-performing text representation was, in fact, PGN, which does not describe the board state nor locations of pieces (as in the other representations), but instead records a trajectory of moves from the start of the game. As an example, here’s the **PGN for the board in Fig. 2**: 1. e4 e5 2. Nf3 Nc6 $\cdots \cdots \cdots$ 27. Ke3 Qf3#. This representation is **harder than just getting the current board state** (as in ANL and image representations) as it must do state-tracking to infer the current position. This gives an example of how a more-explicit representation of the current board state, which seems intuitively easy to us, is not always the easiest for these models.

## **re: usefulness for practitioners**

Thank you for this suggestion! We will add to our paper an in-depth discussion on the usefulness of IsoBench for practitioners. We share a curtailed version below.

Regarding the graphs, there are many natural cases where a practitioner (e.g., a data scientist) starts with a symbolic (tabular) representation, and then may find it natural to convert it to a visual plot. In these cases, they have a choice of which representation to feed to the model, and our results tell them which is likely to be superior.

Similarly, for finance users who need to work with Markov Chains – even though visualization as chains/graphs are more interpretable to humans – our results tell them it’s better to provide LLMs with transition matrices of the Markov Chains instead of their graphical visualizations. 

## **re: no experts can write accurate text representations**

We agree that there are tasks that are more naturally grounded in image than text, e.g. the case of analyzing scientific plots presented in papers. Our math tasks are deliberately designed to assess necessary properties for analysis automation, such as counting, curvature, and symmetry. The models’ inability to reliably perform on these tasks serves as a cautionary tale to warn practitioners from overtrusting VLMs to carry out these analyses without expert insight.

## **re: same model for image captioning**

If an expert themselves is using the LLM to help automate tasks, they could add text descriptions as needed. It's valuable for them to know that this is likely to be helpful.

On the curious case of the effectiveness of IsoSP, our hypothesis is that VLMs are better at *perceiving* than *reasoning* with visual modalities. High-quality, textual reasoning dataset are abundant (e.g. [1]), but their VL counterparts (e.g. [2]) are scarce. Thus, reasoning with imperfect captions may outperform reasoning with complete visual input, due to the lack of quality training for VL reasoning. We aim to investigate this hypothesis in future works.

[1] Platypus: Quick, Cheap, and Powerful Refinement of LLMs, https://platypus-llm.github.io/

[2] LLaVA Visual Instruct 150K, https://huggingface.co/datasets/liuhaotian/LLaVA-Instruct-150K
I have no further questions now, and I am still positive about this paper. I would like to keep my initial rating.
Thank you very much for your responses. You have addressed some of my concerns. I am pleased to increase the score.
I don't have additional questions at the moment and will adjust my score according to discussion with other reviewers.
I thank the authors for their rebuttal. I have increased the score to 4 since the authors have done a decent job in arguing the contribution of the work. Specifically, I understand that the paper identifies the gap in the model’s understanding of an identical input when grounded in the image representation or the text representation. However, I believe that the paper and the rebuttal have not been able to justify the usefulness of the knowledge about this gap. In addition, I am not against the simplicity of the method, but I feel that the paper sufficiently shows its applicability for practical tasks. 


### **On implicit bias (and level of difficulty) of different modalities**

I agree that a chess board image or an ANL representation require LLM to comprehend piece positions and relationships, involving similar reasoning processes. However, I still feel that the task of comprehending the piece positions and relationships is easier in the text modality. Assume that the location of the rook in Figure 2 is d8 – this location is explicitly mentioned in the text representation by three dots before ‘r’. The text position encodings can easily tell that the location of the rook is 4th from the left. In addition, the text representation explicitly mentions the character ‘r’ that disambiguates the presence of the rook itself. 

### **Why IsoBench is useful for practitioners**

While the benchmark shows that there is a gap between image and text representations, we should also acknowledge that the tasks studied in the paper are more commonly grounded in the “image representation” instead of the “text representation”. For instance, I will not know the underlying mathematical function for the graphs shown in Figure 2, and will not be able to write the text representation (function form) to solve my queries. 

### **IsoCombination and IsoScratchPad are trivial and contrived**
I am fine with the triviality of the approach – it is okay if simple things work. However, the paper does not provide any guidelines on the scalability of the approach. The paper or the rebuttal does not tell me how to solve questions in SciBench or MathVista using IsoCombination. In practice, I will not be able to find an expert to write accurate text representation of the images.
In addition, I think image captioning is repackaged as IsoScratchPad. Infact, Section 3 argues that the “models only offer a coarse-grained representation that only summarizes high-level features, and may be unsuitable for detailed analysis such as plot and chart understanding.” and then ends up using the same models for image captioning (IsoScratchPad).