SEAM: Semantically Equivalent Across Modalities Benchmark for Vision-Language Models
Authors: Zhenwei Tang
Difan Jiao
Blair Yang
Ashton Anderson
Venue: COLM 2025 (2025)
Keywords: Vision-Language Models
Benchmark
Modality Imbalance
TL;DR: We created a benchmark testing VLMs' ability to reason consistently across different modalities, revealing limitations in how models process semantically equivalent information based on format rather than meaning.

Abstract: Evaluating whether vision–language models (VLMs) reason consistently across representations is challenging because modality comparisons are typically confounded by task differences and asymmetric information. We introduce SEAM, a benchmark that pairs semantically equivalent inputs across four domains that have existing standardized textual and visual notations. By employing distinct notation systems across modalities, in contrast to OCR-based image-text pairing, SEAM provides a rigorous comparative assessment of the textual-symbolic and visual-spatial reasoning capabilities of VLMs. Across 21 contemporary models, we observe systematic modality imbalance: vision frequently lags language in overall performance, despite the problems containing semantically equivalent information, and cross-modal agreement is relatively low. Our error analysis reveals two main drivers: textual perception failures from tokenization in domain notation and visual perception failures that induce hallucinations. We also show that our results are largely robust to visual transformations. SEAM establishes a controlled, semantically equivalent setting for measuring and improving modality-agnostic reasoning.

Published as a conference paper at COLM 2025
SEAM: Semantically Equivalent Across Modalities Benchmark for Vision-Language Models
Zhenwei Tang', Difan Jiao', Blair Yang & Ashton Anderson'
1 Department of Computer Science, University of Toronto
2 Coolwei AT Lab {josephtang, di fanjiao, blair, asston),sssn))sss toronto.
Abstract
Evaluating whether vision-language models (VLMs) reason consistently across representations is challenging because modality comparisons are typically confounded by task differences and asymmetric information. We introduce SEAM, a benchmark that pairs semantically equivalent inputs across four domains that have existing standardized textual and visual nota- tions. By employing distinct notation systems across modalities, in contrast OCR-based image-text pairing, SEAM provides rigorous comparative assessment of the textual-symbolic and visual-spatial reasoning capabilities of VLMs. Across 21 contemporary models, we observe systematic modality imbalance: vision frequently lags language in overall performance, despite the problems containing semantically equivalent information, and cross- modal agreement is relatively low. Our error analysis reveals two main drivers: textual perception failures from tokenization in domain notation and visual perception failures that induce hallucinations. We also show that our results are largely robust to visual transformations. SEAM establishes a controlled, semantically equivalent setting for measuring and improving modality-agnostic reasoning. We publicly release the code, dataset, and leaderboard to foster further research.
1 Introduction
Vision-Language Models (VLMs) have made rapid progress in understanding and gener- ating content that spans visual and textual domains, making tangible steps towards more general artificial intelligence (Li et al., 2023; Liu et al., 2023c; OpenAl, 2024a). As these models are deployed more broadly, however, it becomes necessary to measure whether they reason consistently across representations, and assess whether multimodal models have general, integrated understanding or approach problems with narrow, modality processing. The ability to solve tasks across multiple modalities not, by itself, evidence of unified reasoning, as performance could depend on how information is represented.
fundamental obstacle with comparing VLM's ability across modalities confounding by modality: comparisons between vision and language typically vary both the representation and the task, making it unclear whether observed performance differences reflect genuine reasoning gaps or variable task difficulty (Lu et al., 2023; Yue et al., 2024a). Even for fixed concept, textual and visual instances rarely have matched semantics and difficulty, and asymmetric information content further obscures what is being measured. This lack of standardization across modalities makes it difficult to isolate and measure core multimodal processing capabilities. Existing approaches either lack rigorous cross-modal alignment or introduce biases through asymmetric information content, leaving the field without principled way to measure modality-agnose reasoning.
We introduce SEAM, short for Semantically Equivalent Across Modalities, a benchmark designed to rigorously assess modality-agnostic reasoning in VLMs by holding semantics constant while varying only representation. SEAM leverages domains with standardized notation systems in both language and vision modalities cchess (FEN notation vs. board

---

Published as a conference paper at COLM 2025
images), chemistry (SMILES strings vs. structural diagrams), music (ABC notation vs. sheet music), and graph theory (adjacency matrices vs. node-edge diagrams) to ensure semantic equivalence: property where representations preserve identical meaning despite being presented in different modalities. Each task is self-contained within a single modality, eliminating confounding factors from joint inference and enabling clean language-only, vision-only, and language-vision evaluation. The benchmark comprises 16 tasks (four per domain), and 200 items per task (3,200 total), formatted as multiple-choice questions with carefully constructed distractor answers to calibrate difficulty.
Our evaluation of 21 state-of-the-art VLMs reveals systematic modality imbalance: all models exhibit significant gaps between vision and language performance. Additionally, cross-modal answer agreement relatively low, often not far from a random baseline, sug- gesting that models differ substantially in how they process information across modalities, and have substantial room to improve in integrating reasoning and leveraging abilities across representations. Furthermore, we observe that modality imbalances vary significantly across domains. Finally, our error analysis highlights two recurring failure modes: (i) textual perception failures in tokenizing strings in textual inputs (e.g., SMILES, FEN), and (ii) visual perception failures that induce hallucinations. We also perform robustness checks that show our results are not sensitive to common visual transformations.
Our contributions are threefold. First, we introduce SEAM, the first benchmark to sys- tematically control for semantic equivalence across modalities, enabling fair evaluation of cross-modal reasoning. Second, we conduct a comprehensive empirical study across 21 models, measuring controlled cross-modal imbalances for the first time. Third, we analyze errors and discrepancies across tasks and models, and pinpoint perception-driven failure modes in modern VLMs that lower agreement rate across modalities, providing actionable insights for future research. SEAM provides principled framework for measuring progress toward more robust and genuinely intelligent VLMs.
2 Related Work
Vision-language models. Early large VLMs separately process visual and textual inputs with two-stream architectures (Lu et al., 2019; Tan & Bansal, 2019; Chen al., 2020), followed by unified models for both understanding and generation (Zhou et al., 2020; Zhang et al., 2021; Li et al., 2020). The Flamingo and BLIP families of models (Alayrac et al., 2022; Awadalla et al., 2023; Li et al., 2022; 2023; Dai et al., 2023) shift to bridging pre-trained vision and language models. The integration of LLMs into VLMs has led to powerful improvements such as the MiniGPT family (Zhu et al., 2023; Chen et al., 2023), LLaMA- Adapter family (Zhang et al., 2023b; Gao et al., 2023), and the LLaVA family (Liu et al., 2023cb;; 2024a; Li et al., 2024c;a), which have progressively enhanced visual instruction- following capabilities. Among proprietary models, the GPT series (OpenAI, 2023; 2024a;b; 2025a; 2024c; 202 the Claude series (Anthropic, 2024; 2025a;b), and the Gemini series (DeepMind, 2023; 2024a; 2025a; Comanici et al., 2025) have demonstrated state-of- the-art multimodal reasoning. Strong open-source alternatives have emerged including notable models such as the LLaMA series (Touvron et al., 2023a;b; Grattafiori et al., 2024), the Gemma series (DeepMind, 2024b;c; 2025b), the Intern family (Chen et al., 2024c;;;;; Zhu et al., 2025), the Qwen family (Bai et al., 2023; Yang et al., 2024a;b; Wang et al., 2024b; Bai et al., 2025; Xu et al., 2025), and Pixtral (Agrawal et al., 2024).
VLM benchmarks. Early benchmarks such as VQA (Antol et al., 2015; Goyal et al., 2017), OK- VQA (Marino et al., 2019), and MSCOCO (Lin al., 2014) were instrumental in evaluating basic visual understanding. Recent benchmarks have expanded the scope to cover more complex capabilities (Yin et al., 2023; Xu et al., 2024; Li et al., 2024b; Liu et al., 2024b; Tong et al., 2024; Yu et al., 2023; Jiang et al., 2024; Ying et al., 2024; Fu et al., 2024), including hallucination detection (Cui et al., 2023; Liu et al., 2023a) and advanced reasoning (Lu et al., 2023; Zhang et al., 2024a). The MMMU series (Yue et al., 2024a;b) introduces college- level multimodal questions and addresses shortcut issues by enforcing vision-dependent evaluation. EMMA (Hao et al., 2025) further emphasizes complex, multi-step reasoning
2

---

Published as a conference paper at COLM 2025
Chess
Chemistry
Benchmark Construction
RE
Text:
Text:
Sample data from existing datases/ools
C19H29000H
JPFFaFa11R5R
Licheus
ChamDA
Intetrran
Network
Pork Duesuatonn Vhhct the kolvei by pircse taking
Cartasn Dountin wiack ofree trrwwig the coneet
2. Sample Raw Text Representation
Cantpawe Evel Wrnch the io felvinng la cornct
Hystagen Courting Wann the is oving i ponee
PDN
SIILES
ABC
Ausso dceaig foor of mu tilluinn nowe the hou
Weger Colcurtion while the living cono
Deterriie Ground Truth Answers
Legal tovv daveiicatio Waich bolowing
Maiccing With diverption co cor
Pupder 4853 Forks
Certonar
agal in chhasa
Notee 5
Petti jI ta
Music
Graph Theory
Text:
[0.
1,
0,
Text:
0.
1
Incorrect Choices
X CcDd
11
10 1.0
0.
1.
1:
01
Mamericll Choic
Categoreal Chotee
Deserprive Charee
10
knoee lu (iie sbdi
of and we Lsing
17
Countingt Now many ndtid al
Pash Cocnting: Whic' toliwitng tt's sarvec
5. Convert to Standard Image Representations
(nikuding
picce?
Messuve Clanding Whith of the felow tu cerved
Purk dettoinng We of tuboe node late
Fove savnticatinn Wiich of bellving best dasetbas
Stortout Parh Aivesitation Which che toioving
Rtytive saveticatinn Whish of bilowing mesaurss
DPS Fravoval Which of folowing lata vi apresnni
python- chees
RDKR
Music21
NetworkX
Figure 1: SEAM includes 16 tasks in chess, chemistry, music, and graph theory domains with paired visual-spatial and textual-symbolic representations that are semantically equivalent.
across STEM fields. However, assessing the modality imbalance of VLMs remains difficult due to the lack of benchmarks with semantically equivalent inputs across modalities. While previous efforts have attempted benchmark modality imbalance, they exhibit significant limitations. Yue et al. (2024b) and Zhang et al. (2024b) introduced OCK-derived semantically equivalent image-text pairs, e-g,, screenshots or photos of textual questions, but these merely transform text into images without leveraging distinct notation systems in different modalities. Thus, OCR-based benchmarks mainly evaluate symbol recognition rather than modality-agnostic reasoning capabilities. A robust evaluation should contrast visual- spatial representations (encoding through spatial relationships and visual patterns) with tymbual-symbolic representations (encoding through abstract symbols and formal notation) to effectively assess modality imbalance. Further discussions on modality imbalance and domain-specific VLMs are in Appendix A.
3 SEAM: Semantically Equivalent Across Modalities Benchmark
In this section, we describe how we designed and instantiated the SEAM benchmark. We begin with our design principles that ensure semantic equivalence across modalities: stan- dardized notations, faithful conversions, and single-modality self-containment. Guided by these principles, we first select four domains that satisfy them (chess, chemistry, music, and graph theory) and then define tasks within each domain, carefully constructing questions for each task. The result is a 16-task, 3,200-item benchmark for attributing performance differences to input modality rather than task confounds.
3.1 Design Principles
Standardized notation in both modalities. We selected benchmark domains that have well-established, standardized notation systems that can represent semantically equivalent
3

---

Published as a conference paper at COLM 2025
information across both visual and textual modalities. In chess, positions can be precisely encoded as visual 2-D chessboards or as textual Forsyth-Edwards Notation (FEN) (Ed- wards, 1994) strings, with perfect semantic equivalence between these representations. Chemical compounds offer a similar duality through structural diagrams that visually represent molecular bonds and atoms versus SMILES (Simplified Molecular Input Line Entry System) (Weininger, 1988) strings that encode the same information textually. Musical compositions can be expressed through traditional visual staff notation with notes and measures or through ABC notation (Walshaw, 2004), which captures musical elements in plain text. Graph theory provides another ideal domain with node-edge diagrams for visual representation versus adjacency lists matrices for textual encoding, This bidi- rectional mapping property ensures that information content remains consistent across modalities, enabling rigorous evaluation of mrosl reasoning. While these notations provide strong semantic alignment, we acknowledge that achieving absolute equivalence impervious to subtle, low-level visual rendering variations (e.g,, specific line styles, exact node positioning) is a theoretical ideal; our focus is on ensuring that the essential symbolic information is rigorously preserved across modalities. We later verify that our results are robust to rendering changes.
Tool availability and real-world prevalence. Our domain selection is also driven by the availability of robust tools that support faithful conversion between standardized textual and visual representations, prerequisite for creating semantically equivalent cross-modal pairs. The python-chess library (Fiekas, 2025) provides conversion between FEN notation and visual chessboards; RDKit's capabilities (RDKit Development Team, 2025) allows trans- formation between SMILES strings and molecular structure diagrams; Music21 (Cuthbert & Ariza, 2010) converts between ABC notation and standard staff notation images; and the NetworkX (Hagberg et al., 2008) library handles both adjacency matrices and node-edge diagrams. These tools not only support cross-modal conversions but also enable automated generation of synthetic questions and answer options across modalities.
We also prioritize domains with widespread real-world use in both visual and textual forms. Chess practitioners routinely alternate between board visualizations and FEN notation on platforms such as Chess. com (Chess.com) 2025) and Lichess (Lichess Team, 2025b). In chemistry, researchers alternate between structural diagrams and SMILES strings, with databases (Kim et al., 2025) providing both. Music analysis involves translation between staff notation and formats like ABC notation (Roland, 2002), and graph theory researchers employ both visualizations and adjacency matrices (Leskovec & Sosič, 2016). This -tullmodality prevalence ensures that SEAM evaluates reasoning capabilities that directly transfer to real-world applications.
Self-contained in each modality. A key criterion is that every problem should be fully solv- able using only its textual representation or only its visual representation. By crafting tasks so that all necessary information is self-contained within each modality, we remove any need for joint inference or reliance on secondary cues. This approach enforces a strict evaluation of cross-modal reasoning and precludes models from exploiting superficial patterns that might appear exclusively in one modality. Consequently, performance in our benchmark reflects model's true ability to handle semantically equivalent information across different representations, mirroring real-world scenarios where experts freely alternate between visual and textual formats without losing crucial information.
3.2 Benchmark Construction
Guided by these principles, we construct 16 tasks (four per domain), formatted as 4-way multiple-choice questions, following MMMU (Yue et al., 2024a). Each task contains 200 items (3,200 total). Detailed construction procedures and hyperparameters for each task appear in Appendix C.
Text representations and ground truth. We first sample raw textual representations from datasets or generate them with domain-specifie tools. For chess, we extract FEN strings from the Lichess Open Database Puzzles (Lichess Team, 2025c) and Evals (Lichess Team,
4

---

Published as a conference paper at COLM 2025
2025a). For chemistry, we use SMILES strings from ChemBench (Guo et al., 2023) and ChemQQ (Zimmermann et al., 2024; Zhu et al., 2024). For music, we collect ABC notation from the Irishman dataset (Wu et al., 2023) and the Music Theory dataset (Seeker38, 2025).
For graphs, we generate adjacency matrices with NetworkX.
Some ground-truth labels cannot be extracted automatically. For example, free-text captions for chemical compounds require human annotation, so we sample them from existing datasets. By contrast, when ground truth is deterministically derivable from the textual representation, such as breadth-firss search traversal order in graphs, we generate it auto- matically with domain-specific tools.
Calibrated task difficulty. We calibrate task difficulty when constructing distractors (incor- rect options) to enable meaningful cross-modal comparisons. Tasks that are too easy result in near-perfect performance across all modalities, obscuring any potential gaps, while overly difficult tasks reduce performance to random guessing levels, also making it difficult to distinguish between modalities. Maintaining moderate difficulty exposes genuine modality imbalances that can be properly measured. SEAM includes three option categorical, and descriptive- priately challenging distractors. For numerical tasks, we perturb the correct answer by task-specific amounts (e.g., +330 centipawns for chess evaluations) to generate alternatives that remain within plausible range while avoiding near-duplicates. For categorical tasks (e.g., selecting music forms or legal chess moves), distractors are sampled uniformly from the label set, excluding the correct label. For descriptive tasks involving rich text (e.g., model (Wang
with tailored strategies to produce plausible, appro-
chemical compound captions), we the Gaalilingual-EE-lare----i et al., 2024a) to retrieve semantically similar alternatives, as random negatives are typically insufficiently confounding.
Text image conversion. For domains with highly asymmetric layouts, such as music staff notation, we pad the images with white space to produce square images. This preserves visual clarity and ensures that structural patterns remain recognizable to VLMs even after automatic resizing. All images are rendered at standardized resolution of 400x400 pixels to balance detail and computational efficiency. We adopt the default settings of each conversion tool described in Section 3.1 to preserve semantic equivalence with the original text representations, ensuring that neither modality introduces additional information. Importantly, instead of rendering text as images (e.g,, via screenshots), we use distinct, standardized notation systems for language and vision modalities. This design targets VLMs' ability to understand and reason over semantically equivalent content across modalities, rather than relying on OCR-style recognition.
Since question-answer pairs are generated with domain-specific tools or sampled from large-scale datasets (e.g., the Lichess Puzzle dataset with over 4.8 million entries), this design enables continual expansion and regular updates, reducing the risk of benchmark leakage from pretraining exposure (data contamination). As models improve, our distractor- generation procedure supports flexible retuning of task difficulty, enabling the creation of more challenging benchmark versions in future releases.
4 Experiments
4.1 Experimental Settings
We evaluate total of 21 state-of-the-art vision-language models (VLMs) under zero-shot thain-of-thought prompting (Kojima al., 2022), using the latest publicly available release of each model series at the time of evaluation. We use the vLLM framework (Kwon et al., 2023) on A100 GPUs to run inference for open-source models, adopting each model's default prompt format, system prompts, and generation hyperparameters when applicable, as detailed in Appendix C We follow the OpenCompass (Contributors, 2023) protocol to extract final answers from model outputs using external LLM when rule-based extraction fails. As final answer extraction is relatively straightforward, we use the smaller and open- source 1eree22557B--nnseee model for this task, given its strong performance and efficiency.
5

---

Published as a conference paper at COLM 2025
Table 1: Leaderboard of proprictary and open-source VLMs across language (L), vision (V), and vision-language (VL. modalities. Models are sorted by agreement between language and vision modalities. Bold and underlined values indicate best and second-best perfor- mance within each category, respectively.
Accuracy
Agreement
Model
VI.
Avg
L- L-
L-VI.
V-VL.
All
Proprietary Models
GPT-5-mini
0.787
0.653
0.830
0.756
0.630
0.846
0.653
0.584
GPT-5
0.804
0.632
0.857
0.765
0.627
0.876
0.657
0.596
Claude-3.7-Sonnet
0.743
0.591
0.679
0.671
0.594
0.715
0.624
0.506
Claude-41-OOpus
0.827
0.578
0.814
0.740
0.575
0.844
0.580
0.523
Claude-4-Sonnet
0.808
0.545
0.803
0.719
0.569
0,834
0.566
0.508
Claude-3.5-onneet
0,665
0,560
0.514
0.580
0.537
0.549
0,508
0,378
GPT-4o
0.635
0.482
0.627
0.581
0.503
0.686
0.532
0.410
GPT-5-nano
0.699
0.510
0.753
0.654
0.500
0.771
0.516
0.432
GPT-4o-mini
0.555
0.411
0.529
0.498
0.480
0.650
0.518
0.379
Claude-3.--aiku
0.530
0.433
0.496
0.486
0.479
0.556
0.534
0.346
Open-Source Models
Qwen2.5-VL-72B--nnste
0.547
0.475
0.519
0.514
0.447
0.504
0.532
0.318
InternV13-78BB
0.525
0.427
0.482
0.478
0.447
0.498
0.487
0.293
gemma-3-27b-it
0.516
0.428
0.450
0.465
0.447
0.497
0.575
0.325
gemma-3-12b-it
0,458
0.401
0,429
0.429
0.419
0,474
0.543
0.297
IntemVV-22.5-7BB
0.448
0.414
0.459
0.440
0.415
0.485
0.523
0.309
InternV1.3-8B
0.382
0.357
0.386
0.375
0.388
0.425
0.456
0.229
1aama--.2-00--Viiioo--ssstt
0.434
0.384
0.439
0.419
0.384
0.460
0.443
0.253
Qwen2.5-Omni-7B
0.363
0.354
0.364
0.360
0.353
0.375
0.375
0.183
Owenn.5-yL-7--Inntruuut
0.303
0.350
0.359
0.337
0.347
0.389
0.437
0.216
InternVL-2.5-8B
0.324
0.337
0.334
0.332
0.324
0.340
0.436
0.196
Llama-2-111D-Visioo-nssnn
0.289
0.330
0.323
0.314
0.287
0.303
0,401
0.152
As shown in Table 2 in Appendix B, the extraction results are highly consistent with those obtained using different LLMss
4.2 Results
General performance comparison. As shown in Table 1, proprietary models significantly outperform open-source alternatives across all modalities in general, with GITT-5 demon- strating superior accuracy (0.765) compared to the highest-performing open-source model, Qwenn..--VL-72--nnstiii (0.514). Performance consistently follows scaling laws across both proprietary and open-source model families, with larger variants outperforming their smaller counterparts (GPT-5 at 0.756 vs. GPT-5-nano at 0.654; Gemma-3 27B 0.465 vs. 12B at 0.429), highlighting the impact of model scale on multimodal reasoning capabilities.
Vision-language modality imbalance. In addition to accuracy, we calculate the agreement rate between extracted final answers across semantically equivalent questions presented in different modalities. For each sample, the agreement between modalities is binary (either for disagree or for agree), and these binary values are then averaged across all samples to obtain the overall agreement rate.
In principle, an intelligent reasoner, either human artificial, could (and perhaps should) exhibit consistent performance when presented with semantically equivalent information in different modalities. For example, molecule shown as structural diagram or described through a SMILES string contains the same information. human expert chemist might prefer the visual structural diagram for intuitive reasoning, but when given SMILES string, they can still identify the compound and its properties, possibly by first mentally or physically sketching the visual structure. The challenge lies in the molecular complexity

---

Published as a conference paper at COLM 2025
80%
70%
GPT-S-mini, GPT-5
Language-vision Agreement
60%
Claude.3.7
Claude-4.1-Opus
Claude-3 3.5-Sonnet
Claude-4-ooneet
GPT-S-nano
50%
GPT-4e
Claude-3.5-Halle
GPT-4o-mini
InterVLL3-788
Gemma3-276
Qwen2.5-VL-728
Gemma3-128
Random Baseline
InternVL2.5-78B
40%
InternVL3-88
Llama3.2-908
Qwen2.5-0mni-78.
Qwen2.5-VL
InternVL2.5-60
30%
Llama3.2-118
30%
40%
50%
60%
70% 80%
Average Accuracy
Figure 2: Correlation between final answer agreement rates with language and vision inputs and average accuracy across various VLMs. Models color-coded by family. Random Baseline denotes how often two models with identical accuracy would agree by random chance, which can be thought of as lower bound for cross-modal agreement of real multimodal models.
and properties, not in the modality of its presentation. truly intelligent VLM could behave analogously, and either naturally treat different input representations similarly, learn to translate between modalities when it is advantageous to do so so-and thus have high agreement between modalities when presented with semantically equivalent information across different modalities.
However, this is not what observe. The results in Table reveal substantial performance variation (low agreement) across modalities for all evaluated models. Proprietary and larger open-source models consistently achieve higher accuracy with language inputs and lower accuracy with vision inputs. Because VLMs typically build upon extensively pre-trained LLMs, they inherit their reasoning capabilities from base LLM. Textual inputs fed directly into the LLM, while visual inputs must first pass through visual encoder and projector before being integrated into the LLM, which is noisy process.
Note that VLM could theoretically achieve near- pear--erfect agreement across vision and lan- guage on our semantically equivalent tasks, even at low accuracy levels. One conceptually simple method would be to learn how to translate between modalities, which is always possible by construction for SEAM tasks, attempting to solve problems both modalities, and picking one of the solutions. More ideally, a generally intelligent model could learn how to leverage the strengths of each modality, and in which situations it is wise to prefer one modality over another. But none of the VLMs we tested have high agreement, meaning there is substantial progress to made in attaining fluid and general intelligence that adapts across modalities.
7

---

Published as a conference paper at COLM 2025
Correlation between agreement and accuracy. We observe strong positive correlation between cross-modal agreement and overall accuracy across most model series in Fig 5, suggesting an association between modality alignment and performance. This raises important question about the direction of causality: Does higher accuracy lead to higher agreement, does higher agreement drive better accuracy?
Note that as models become more accurate, they naturally converge as they increasingly agree on correct answers across modalities, artifactually increasing agreement. In the theoretical limit of perfect accuracy, cross-modal agreement would necessarily reach as all modalities output identical correct answers. However, evidence from Zhang et al. (2024b) challenges the notion that accuracy improvements are the sole driver of higher agreement. Their findings demonstrate that accuracy and agreement can vary independently within practical performance ranges: VLMs can exhibit high agreement despite poor performance, or achieve high accuracy with relatively low agreement between modalities. This pattern is clearly visible in Fig 5, particularly among models in the GPTT and Claude series. This decoupling suggests that the strong positive correlation we observe cannot be explained solely by accuracy driving agreement.
To measure the artifactual relationship relattionii baseline that represents how often two models with identical accuracy p would agree by random chance, which can be thought of as a lower bound for cross-modal agreement of real multimodal models. We simulate two independent "modalities" with identical accuracy but no genuine coordination. For a multiple-choice question with 4 options where one is correct, each modality independently selects the correct answer with probability p and each incorrect answer with probability I-P, The agreement rate between the two modalities is -3" P= P + -2 +2 (1t)) = p2 --pp" where P1 is the probability both select the correct answer and P2 is the probability both select the same incorrect answer. By varying p, generate the random baseline curve shown in Fig 5.
between agreement and accuracy, we define a random
Comparing this random baseline with our measured results of contemporary multimodal models suggests that current models do not achieve significant cross-modal alignment beyond the simple agreement achieved by task correctness. Most models cluster much closer to this random baseline than to the ideal upper bound of perfect agreement, suggesting that what appears to be improved cross-modal alignment as models scale may simply reflect the mathematical constraint that higher accuracy necessitates higher agreement rates, rather than representing true progress toward unified multimodal understanding.
Domain-specific modality imbalance. As illustrated in Fig 3, the degree of modality imbalance is highly domain-specific. In chess and chemistry, models frequently demonstrate comparable slightly superior performance with vision inputs compared with language inputs. However, this pattern reverses in music, where language inputs generally yield better results than vision inputs. This imbalance becomes even more significant in graph- related tasks, where the performance gap between language and vision inputs substantially widens. These domaii--peccffic asymmetries in input modalities suggest that cross-modal consistency varies significantly depending on the reasoning domain. Such observations motivate us to investigate the underlying mechanisms behind domain-specific performance disparities.
Textual perception error. We examine the tokenization processes of open-source models to understand how they parse domain-specific text inputs. In chemistry tasks, we find that SMILES strings such are often incorrectly segmented into semantically meaningless subwords like "OC", "cc", and "([". Such tokenization errors are particularly problematic, where structural elements like parentheses (indicating branches) and square brackets (denoting charged atoms) carry precise molecular semantics that, when incorrectly parsed, lead to fundamentally different chemical interpretations. Similarly, we also identified severe tokenization errors in chess FEN notation examples. However, music ABC notation exhibits fewer issues, likely due to its abundant punctuation markers that provide clearer tokenization boundaries. Interestingly, graph-related tasks, where adjacency matrices primarily consist of commas, Os, and 1s,
8

---

Published as a conference paper at COLM 2025
Chess
Chemistry
0.746
GPT-5-mini
0.776
0.786
0.779
6.834
GPT nano
0.743
0.629
0.731
0.738
0.528
0.744
GPT-40
0.625
0.644
0.624
0.530
0.573
0,653
Claude-4,1-0ppe
0.718
0.946
Cloude Sonnet
0,799
0.734
0.784
0.891
8: F04
0,906
Cleude 3.7-Sonnet
6.706
0.656
0.675
0.888
0.003
0.070
6.655
Claude-3 S-Haiku
6.623
0.549
0.635
0.574
0.529
0.589
Oven2.5-VL-72--20 5-1--t2-ii
0.542
0.586
0.571
0.575
0.559
0.609
Qwen7.5WV (iin2.55ii-B-irstrue
5-0mni-7
0.512
0.307
0.338
InternVL3-78B
0.549
0.549
0.560
0.468
0.446
0.505
IntemiVL3-88
0,420
0.431
0.454
0.372
0.289
0.364
Int77V-2.55-88B
InternVL 2.5 BB
0.466
0.329
0.300
Uamma 2.2...2.900 2-B itsto--iiii
0.495
0.538
0.535
0.501
0.363
0.551
Ulama 3.2 118 Vision Instruct
0.312
0.446
0.415
0.304
0.294
0.340
gemma-3 3-27b-it
0.454
0.490
0.476
0.446
0.509
Language
Vision
Vision Language
Language
Vision
Vision Language
Music
Graph Theory
GPT-5
GPT-5-miri
0.736
0.348
0.730
0.788
0.599
0.996
CPT-S-rano
0.548
0.328
0.549
0.768
0.525
0.990
GPT-4o-mini
0.688
Clude-4.1-Opus
0.581
0.348
0.580
0.674
0.394
0.936
Cloude 4 Sonnat
0.569
0.328
0.555
0.574
0.414
0.971
Sannat
0.365
0.446
0.473
Claude-3 Halkku
0,361
0.301
0.290
0.561
0.353
0.471
Owen2.5 VL-T2B Instruct
0.425
0.341
0.373
0.647
0.414
0.524
0.307
Owen2 5-mni-5-0
0.318
0.364
IntennVL3-700
0.411
0.285
0.361
0.574
0.430
0,502
IntemVL3-88
0.302
0.289
0.278
0.434
0.419
0.668
IntemvV--2.5-88
0,245
0.220
0.221
0.331
0.336
0.365
Llama-3 2-909--iiioo-instruu
0.331
0.269
0.276
0.409
0.367
0.303
LJama-3 2atee-3.-11B--iiiirrrro
2-118--iiioonirsirrt
0.335
gemma-3-27b-it
0.354
0.339
gemma-3-125 3-b2b
0.395
0.280
0.316
0.521
0.386
0.416
Language
Vision
Language
Language
Vision
Language
Figure 3: Comparison of model accuracy across different modalities in each domain.
demonstrate minimal tokenization errors. The severity of such textual perception limitation negatively correlates with the advantage of language inputs.
To further investigate tokenization error, rerun the chess tasks with "gold-standard" tokenization, which separates the tokens in FEN according to human common sense. For example, if there two pawns next to each other, the FEN will include "PP" which should intuitively be tokenized into two "P" tokens. But VLMs often suboptimally regard "PP" as single token. We expect the models perform better with the modified tokenizer. performs almost the same (original 0.542 vs. gold-standard 0.540 accuracy). We hypothesize that two contradictory factors combined to result in such performance: although the tokenization becomes intuitively better (positive factor), the models were not trained with such tokenization (negative factor). For example, "PP" might occur lot in the training data, and the model always uses the single "PP" token to understand FEN. When we simply replace the "PP" token with two "P" tokens, the semantics learned in the embedding of "PP" could not be directly used, while the semantics of two adjacent "P"s were not as well trained. Such observations suggest the importance of designing task-specific tokenizers and training domain-specift VLMss.
However, we found that Diven2.5-VL-72B-Innfff

---

Published as a conference paper at COLM 2025
Visual perception error. Furthermore, results in Fig 3 suggest parallel constraints in visual perception. In particular, the lack of synergistic improvement when combining vision and language input indicates that the vision modality suffers from its own perception challenges. In domains where language inputs face known tokenization difficulties, such as chemistry and chess, the failure of vision inputs to compensate for these limitations is evident across most VLMs. In domains with minimal tokenization problems, most VLMs yield better results with language than vision-language. This indicates that vision inputs are contributing negatively, which points to fundamental constraints in visual perception capabilities. In particular, the process of cutting each image into patches to feed ViTs is problematic. We found severe VLM hallucinations when vision inputs for graph theory tasks are cut near the intersection of edges. The hallucinations are particularly related to the edges and nodes involved in these intersections. A detailed example is shown in Fig 7 in the Appendix, Note that textual inputs of the graph theory tasks are presented as adjacency matrices, in which numbers were well-separated by commas, and thus there are no known tokenization issues. This could potentially explain why the vision performance is much worse than language in Fig 3.
Discussion
Semantic equivalence. Although we generate and curate SEAM with standardized tools to maximize cross-modal semantic equivalence, achieving perfect equivalence remains theoretical ideal. Subtle rendering details (e.g,, line thickness, fonts, layout-driven node placement) can introduce low-level perceptual differences relative to purely symbolic text, potentially interacting with model perception. Nevertheless, results in Fig 4 (Appendix B) demonstrate practical robustness to substantial visual alterations that preserve core sym- bolic information. Specifically, applying resolution changes, grayscale conversion, and 180 rotation yields only minimal performance variation (at most 1.66, 13.1%, and i 1.9%, respectively), suggesting that models respond primarily to underlying content rather than superficial visual characteristics. Additionally, some domains admit minor semantic mis- matches between modalities. For example, FEN includes details (e.g,, castling rights, en passant availability) that are not always recoverable from board image alone. In practice, these differences affect task performance in only very small minority of cases.
Limitations. We focus exclusively on images without evaluating video or multi-image sequences, though such extensions are possible and important future work. For example, comparing chess move sequences in PGN notation with sequences of board states could assess cross-modal temporal reasoning. Our domain coverage, though carefully selected, covers only subset of domains meeting our selection criteria; additional domains such as circuit diagrams versus SPICE netlists (Vungarala et al., 2025) could be incorporated. Our evaluation includes a limited number of models due to budget constraints for proprietary models and computational constraints for open-source models. SEAM will be made pub- licly available with a leaderboard, enabling broader community contribution and ongoing evaluation of emerging models. Future research could explore qualitative methods like report cards (Yang al., 2024c) to address the limitations of current VLM benchmarks.
6 Conclusion
Our SEAM benchmark reveals significant modality imbalance in VLMs, which struggle to reason consistently across semantically equivalent visual and textual representations in distinct notation systems. This fundamental limitation highlights the gap between current capabilities and truly modality-agnoset AI. Moving forward, crucial aspect of VLM development is the ability to process information regardless of its representation format. SEAM provides a principled framework for measuring progress toward more robust and genuinely intelligent VLMs.
10

---

Published as a conference paper at COLM 2025
References
Pravesh Agrawal, Szymon Antoniak, Emma Bou Hanna, Baptiste Bout, Devendra Chaplot, Jessica Chudnovsky, Diogo Costa, Baudouin De Monicault, Saurabh Garg, Theophile Gervet, et al. Pixtral 12b. arviv preprint ar arXi::2410.07073, 2024.
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. Advances in neural information processing systemss, 1123716-237366 2022.
Anthropic. The claude 3 model family: Opus, sonnet, haiku, 2024. URL https://assets: Boaiidd----Moos 3-Model-Card pdf.
anthropic. com/m/ httpr//eedfffffffff///rrrgaaaaaa
Anthoopic. Claude 3.7 sonnet system card, 2025a. URL https://anthropic.com/
claude-3-74 --lee-------ontt--sy- systemt-yssm--m--e card.
Anthropic. System card: Claude opus 4 & claude sonnet 4, 2025b. URL http:/cccccom2233339433 hwtprppt...ooeee//00000666003333
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pp. 2425-2433, 2015.
Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, et al. Openflamingo: An open- source framework for training large autoregressive vision-language models. Xiviv preprint ar (eXiv:2308.01390 2023.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint ar (rXi:2230911609, 2023.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun lang, al. Qwen2. 5-vl technical report. arviv preprint drXiv:2502.13922 2025.
Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Krishnamoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny. Minigpt-v2: large language model as a unified interface for vision-language multi-task learning. Xrxiv preprint ar Xie::2310.09478 2023.
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In European conference on computer vision, pp. 104-120. Springer, 2020.
Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling, ar preprint (rXiv:2412.05271, 2024a.
Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. Science China Information Sciences, 2112222 2024b.
Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the I.F.F//VF conference on computer vision and pattern recognition, pp. 24185-24198, 2024c.
Chess.com. Chess.com, 2025. URL httphs:/www.cchss..coom com/. Accessed: March 2025.
11

---

Published as a conference paper at COLM 2025
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arv preprint ar arXio:2507.0626) 2025.
OpenCompass Contributors. Opencompass: A universal evaluation platform for foundation models. https: ht thwb. http:///ww...oooooooomcooppppaas httpp/necccccpppppppeppppppppssp 2023.
Chenhang Cui, Yiyang Zhou, Xinyu Yang, Shirley Wu, Linjun Zhang, James Zou, and Huaxiu Yao. Holistic analysis of hallucination in gpt-4v (ision): Bias and interference challenges. arXiv preprint ar ArXii:223110333 2023.
Michael Scott Cuthbert and Christopher Ariza. music21: A toolkit for computer-aided musicology and symbolic music analysis. In 11th International Society for Music Information Retrieoal Conference, Pp. 637-642, 2010.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning, 2023. URL https://aaxin or orggbbs///// 2305.06500
Google DeepMind. Gemini: a family of highly capable multimodal models. arXiv preprint ar arXiv:2312.11805, 2023.
Google DeepMind. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint drXiv:2403.05530, 2024a.
Google DeepMind. Gemma: Open models based on gemini research and technology. arviv preprint xrXi::2433.08295, 2024b.
Google DeepMind. Gemma 2: Improving open language models at practical size. arXio preprint ar Xix::2408.00118, 2024c.
Google DeepMind. tic era, 2025a. moogle-geminis
Introducing gemini 2.0: our new ai model for the agen- URL. https: //blog google/technologee het.loogg/ggoglleeeemnnldl/ waw--geemii-a-uddatteeeaeeeaeeee suudate-december--e
Google Deep Gemma 3 technical report, 2025b. URL httptpp:///storgggggggleeee com/
Steven J Edwards. Portable game notation specification and implementation guide, 1994.
Chess programming standard.
Yunfeng Fan, Wenchao Xu, Haozhao Wang, Junxiao Wang, and Song Guo. Pmr: Prototypical modal rebalance for multimodal learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20029-20038, 2023.
Weixi Feng, Wanrong Zhu, Tsu-jui Fu, Varun Jampani, Arjun Akula, Xuehai He, Sugato Basu, Xin Eric Wang, and William Yang Wang. Layoutgpt: Compositional visual planning and generation with large language models. Advances in Neural Information Processing Systems, 36:18225-18250 2023.
Niklas Fiekas. python-chess: A chess library for python, 2025. URL htpphtts////ython-cctt readthedocs. io/. Accessed: March 2025.
Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A Smith, Wei-Chiu Ma, and Ranjay Krishna. Blink: Multimodal large language models can see but not perceive. In European Conference on Computer Vision, pp. 148-166. Springer, 2024.
Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, et al. Llama-adapter v2: Parameter-efficienn visual instruction model. ar Xix preprint ar Xitt:2304.10110 2023.
12

---

Published as a conference paper at COLM 2025
Yash Goyal, Tejas Khot, Douglas Summers- Stay,, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 6904-6913, 2017.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The Ilama 3 herd of models. arXiv preprint Xrv::22407222 217773,7,, 2024.
Taicheng Guo, Bozhao Nan, Zhenwen Liang, Zhichun Guo, Nitesh Chawla, Olaf Wiest, Xiangliang Zhang, et al. What can large language models do in chemistry? a compre- hensive benchmark on eight tasks. Advances in Neural Information Processing Systems, 36: 59662-59688, 2023.
Aric Hagberg, Daniel Schult, and Pieter Swart. Exploring network structure, dynamics, and function using networkx. In Proceedings of the 7th Python in Science Conference, pp. 11-15, 2008.
Yucheng Han, Chi Zhang, Xin Chen, Xu Yang, Zhibin Wang, Gang Yu, Bin Fu, and Hanwang Zhang. Chartllama: A multimodal 11m for chart understanding and generation. arXiv preprint ar wix::2311116456 2023.
Yunzhuo Hao, Jiawei Gu, Huichen Will Wang, Linjie Li, Zhengyuan Yang, Lijuan Wang, and Yu Cheng. Can milms reason in multimodality? emma: An enhanced multimodal reasoning benchmark. ar Xie preprint ar 0ate22500004444, 2025.
Qidong Huang, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Jiaqi Wang, Dahua Lin, Weiming Zhang, and Nenghai Yu. Deciphering cross-modal alignment in large vision-language models with modality integration rate. arviv preprint arXio:2410.0716. 2024.
Yu Huang, Junyang Lin, Chang Zhou, Hongxia Yang, and Longbo Huang. Modality compe- tition: What makes joint training of multi-modal network fail in deep Garning?(provabll) In International conference on machine learning, pp. 9226-9259. PMLR, 2022.
Dongfu Jiang, Xuan He, Huaye Zeng, Cong Wei, Max Ku, Qian Liu, and Wenhu Chen. Mantis: Interleaved multi-image instruction tuning. arviv preprint ar arXiv:2405.01488 2024.
Difan Jiao, Yilun Liu, Zhenwei Tang, Daniel Matter, Jürgen Pfeffer, and Ashton Anderson. Spin: Sparsifying and integrating internal neurons in large language models for text S classification. In Findings of the Association for Computational Linguistics ACL 2024, pp- 4666-4682, 2024.
Sunghwan Kim, Jie Chen, Tiejun Cheng, Asta Gindulyte, Jia He, Siqian He, Qingliang Li, Benjamin Shoemaker, Paul A Thiessen, Bo Yu, et al. Pubchem 2025 update. Nucleic Acids Research, 53(D1):D1516-D11 -1555,,, 2025.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199-222222 2201222 2022.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009."
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and lon Stoica. Efficient memory management for large language model serving with pagedattention. Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Jure Leskovec and Rok Sosič. Snap: A general-purpose network analysis and graph-mining library. ACM Transactions on Intelligent Systems and Technology (TIST), 8(1):1-20, 2016.
13

---

Published as a conference paper at COLM 2025
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. ar preprint ar Xio:2408. 033266 2024a.
Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. Seed-bench: Benchmarking multimodal large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13299-13309) 2024b.
Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. Llava-nex-iinterleave Tackling multi-image, video, and 3d in large multimodal models. ar Xiv preprint ar Xiiv2247.007895 2024c
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre- pre-training for unified vision-language understanding and generation. In International conference on machine learning, pp. 12888-12900. PMLR, 2022.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. International conference machine learning, pp. 19730-19742. PMLR, 2023.
Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, al. Oscar: Object-semantics aligned pre-training for vision-language tasks. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXX 16, pp. 121-137. Springerr 2020.
Lichess Team. Lichess evaluation database, 2025a. URL https:://attaaaaaeeeeee .erreee.cchhee lichess. org/ #evals. Accessed: March 2025.
Lichess Team. Lichess: Free online chess, 2025b. URL https:///ichhsss.orr Accessed:
March 2025.
Lichess Team.
Lichess puzzle database, 2025c. URL https://dataaasee
#puzzles. Accessed: March 2025.
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer vision-ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part 13, pp. 740-755. Springer, 2014.
Fuxiao Liu, Tianrui Guan, Zongxia Li, Lichang Chen, Yaser Yacoob, Dinesh Manocha, and Tianyi Zhou. Hallusionbench: You see what you think? or you think what you see? an image-context reasoning benchmark challenging for gpt-4v (ision), llava-1.5, and other multi-modality models. arXiv preprint arXiv:2310.14566, 2(3):9, 2023a.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, 2023b.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023c.
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024a. URL https://llava-v/ gi thub. wtt.l/o//lll//// 200p://22440000 -1024--0--3---1aa-neee- llava-next/.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? In European conference on computer vision, pp. 216-233. Springer, 2024b.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visi- olinguistic representations for vision-and- -nd-aaguagee tasks. Advances in neural information processing systems, 32, 2019.
14

---

Published as a conference paper at COLM 2025
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathe- matical reasoning of foundation models in visual contexts. arXiv preprint arXi::2310.02255, 2023.
Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEF/cof conference on computer vision and paltern recognition, pp. 3195-3204, 2019.
OpenAl. Gpt-Av(ision) system card, 2023. URL https: https://ccch.opeaaa.coo openal cow cotpppae...c//aaper////PVP System.Card, pdf.
OpenAl. Gpt-4o system card. arXiv preprint arXfo:2410.21276, 2024a.
OpenAl. Gpt-4o mini: Advancing cost-efficient intelligence, 2024b. URL https://openai.
com/index/gpa ttteee//pt-o--mnnii 40---ooi-iiiiiiee miii-advancing cost- terffiiieen-inellignnee
OpenAl. Openai ol system card, December 2024c.
URL https://ccd.oopnaat
01 wwstts--ccrd-2eae0 225. .
OpenAI. Gpt-4.5 system card, 2025a. Apt-4-5-system-carde 2ararrd222222..... pdf.
URL https://ccd..openai.
OpenAL. Gpt-5 system card, 2025b. 210088/80000000--022222222223330
URL https://cdnt http://ccdoooppaii..cco com/pdf/
card-aug7.pdf.
OpenAl. Openai 03-mini system card, February 2025c. URL
com/
03- ---mini-ysttmmcarddfeeeee
OpenAl. Openai 03 and o4-mini system card, 2025d. URL http:: http:/cc..ppaaa..coooooooo//./
Rohan Pandey, Rulin Shao, Paul Pu Liang, Ruslan Salakhutdinov, and Louis-Philippe Morency. Cross-modal attention congruence regularization for vision-language relation alignment. arviv preprint ar xro::2212..00549 2022.
Simon Park, Abhishek Panigrahi, Yun Cheng, Dingli Yu, Anirudh Goyal, and Sanjeev Arora. Generalizing from simple to hard visual reasoning: Can we mitigate modality imbalance in vlms? xixiv preprini ar Xiv:25501 256.00000.,.6 2025.
Xiaokang Peng, Yake Wei, Andong Deng, Dong Wang, and Di Hu. Balanced multimodal learning via on-the-fly gradient modulation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8238-8247, 2022.
RDKit Development Team. Rdkit: Open-source cheminformatics, 2025. URL https://www.
rdkit.org/. Accessed: March 2025.
Perry Roland. The music encoding initiative (mei). In Proceedings of the First International Conference on Musical Applications Using XML, volume 1060, pp. 55-59, Citeseer, 2002.
Seeker38. Music abc notation with music theory dataset, 2025. URL hnttps://hugaa/ wothuggingfaan co/dataaetss/Sekerr33 http///eeeeeeeei........ooootttt wtt.wwithhhuuuu th.music.theory. Accessed: March 2025.
Hao Tan and Mohit Bansal. Lxmert: Learning cross-modality encoder representations from transformers. arxiv preprint arXit:1908.07490, 2019.
Mingni Tang, Jiajia Li, Lu Yang, Zhiqiang Zhang, Jinghao Tian, Zuchao Li, Lefei Zhang, and Ping Wang. Nota: Multimodal music notation understanding for visual large language model. ar Xiviv preprint Xix:2502.1144 14893,,11..,, 2025.
Peter Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Adithya Jairam Vedagiri IYER, Sai Charitha Akula, Shusheng Yang, Jihan Yang, Manoj Middepogu, Ziteng Wang, et al. Cambrian-1: A fully open, vision-centric exploration of multimodal lmss. Advances in Neural Information Processing Systems, 37:87310-8773 87356,, 2024.
15

---

Published as a conference paper at COLM 2025
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint urXio:2302.13971, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, et al. Llama Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09233 009888 2023b.
Tao Tu, Shekoofeh Azizi, Danny Driess, Mike Schaekermann, Mohamed Amin, Pi-Chuan Chang, Andrew Carroll, Charles Lau, Ryutaro Tanno, Ira Ktena, et al. Towards generalist biomedical ai. Nejm Ai, 11)a.A1o2300113,, 2024.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(1),, 2008.
Deepak Vungarala, Md Hasibul Amin, Pietro Mercati, Arnob Ghosh, Arman Roohi, Ramtin Zand, and Shaahin Angizi. Limca: Llm for automating analog in-memory computing architecture design exploration. arXiv preprint ar (xxt:2503.13301 2025.
Chris Walshaw. Abc notation standard, 2004. URL http://abcnntation.ccm notation standard.
Music
Jiayu Wang, Yifei Ming, Zhenmei Shi, Vibhav Vineet, Xin Wang, Sharon Li, and Neel Joshi. Is picture worth thousand words? delving into spatial reasoning for vision language models. Advances in Neural Information Processing Systems, 37:75392-75421, 2025.
Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. Multilingual e5 text embeddings: A technical report. arXiv preprint 2rXiv:2402.05672, 2024a.
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint arv: 12191, 2024b.
David Weininger. Smiles, chemical language and information system. 1. introduction to methodology and encoding rules. Journal of Chemical Information and Computer Sciences, 28 (1):31-36, 1988.
Shangda Wu, Xiaobing Li, Feng Yu, and Maosong Sun. Tunesformer: Forming irish tunes with control codes by bar patching. In Lorenzo Porcaro, Roser Batlle-Roca, and Emilia Gómez (eds.), Proceedings of the 2nd Workshop 2023 co-located with the 24th International Society for Music Information Retrieval Conference (ISMIR 2023), Milan, Italy, November 10, 2023, volume 3528 of CEUR Workshop Proceedings. CEUR-WS.org, 2023. URL
Human-CCntrrn Music Information Retrieval
oro/Vol- 3528/ htte//pperrppppm .df.
Siwei Wu, Yizhi Li, Kang Zhu, Ge Zhang, Yiming Liang, Kaijing Ma, Chenghao Xiao, Haoran Zhang, Bohao Yang, Wenhu Chen, et al. Scimmir: Benchmarking scientific multi-modal information retrieval. arXiv preprint ftfo:2401.137777 2024.
Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, et al. Qwen2. 5-omni technical report. arXiv preprint ar arXiv:2503.20215, 2025.
Peng Xu, Wenqi Shao, Kaipeng Zhang, Peng Gao, Shuo Liu, Meng Lei, Fanqing Meng, Siyuan Huang, Yu Qiao, and Ping Luo. Lvlm-ehub: A comprehensive evaluation bench- mark for large vision-language models. IEEE Transactions Patlern Analysis and Machine Intelligence, 2024.
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Cheng- peng Li, Chengyuan Li, Dayiheng Liu, et al. Qwen2 technical report. arxio preprint arXi::2407.1067 2024a.
16

---

Published as a conference paper at COLM 2025
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. technical report. ari preprint mriiv2442.115113 2024b.
Blair Yang, Fuyang Cui, Keiran Paster, Jimmy Ba, Pashootan Vaezipoor, Silviu Pitis, and Michael R. Zhang. Report cards: Qualitative evaluation of language models using natural language summaries, 2024c. URL http:s//aaarrv.orr//b///222// 00844.
Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Anwen Hu, Haowei Liu, Qi Qian, Ji Zhang, and Fei Huang. mplug-ow12: Revolutionizing multi-modal large language model with modality collaboration. In Proceedings of the ieee/cof conference on computer vision and paltern recognition, pp. 13040-13051, 2024.
Zhenfei Yin, Jiong Wang, Jianjian Cao, Zhelun Shi, Dingning Liu, Mukai Li, Xiaoshui Huang, Zhiyong Wang, Lu Sheng, Lei Bai, et al. Lamm: Language-assisted multi-modal instruction-tuning dataset, framework, and benchmark. Advances in Neural Information Processing Systems, 66-26650-26685, 2023.
Kaining Ying, Fanqing Meng, Jin Wang, Zhiqian Li, Han Lin, Yue Yang, Hao Zhang, Wenbo Zhang, Yuqi Lin, Shuo Liu, et al. Mmt-bench: A comprehensive multimodal bench- mark for evaluating large vision-language models towards multitask agi. arXiv preprint arXiv:2404.1 16006, 2024.
Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. Xiv preprint ar Xiv:23008002490 2023.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of CVPR, 2024a.
Xiang Yue, Tianyu Zheng, Yuansheng Ni, Yubo Wang, Kai Zhang, Shengbang Tong, Yuxuan Sun, Botao Yu, Ge Zhang, Huan Sun, et al. Mmmu-pro: A more robust multi-discipline multimodal understanding benchmark. Xiv preprint ar #rXiv2449.008113 2024b.
Chunhui Zhang, Xin Sun, Yiqian Yang, Li Liu, Qiong Liu, Xi Zhou, and Yanfeng Wang. All in one: Exploring unified vision-language tracking with multi-modal alignment. In Proceedings of the 31st ACM International Conference on Multimedia, pp. 5552-55611 2023a.
Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao. Vinvl: Revisiting visual representations in vision-language models. In Proceedings of the JEEE/CVF conference on computer vision and pattern recognition, pp. 5579-5588, 2021.
Renrui Zhang, Jiaming Han, Chris Liu, Peng Gao, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, and Yu Qiao. Llama-adapter: Efficient fine-tuning of language models with zero-init attention. ar Xiv preprint ar 2rX1::2303.16199 2023b.
Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Yu Qiao, et al. Mathverse: Does your multi-modal Ilm truly see the diagrams in visual math problems? In European Conference on Computer Vision, pp. 169-186. Springer, 2024a.
Xiang Zhang, Senyu Li, Zijun Wu, and Ning Shi. Lost in translation: When gpt-4v (ision) can't see eye to eye with text. vistor-lnngugg--oonisseee analysis of vllms and beyond. arv preprint ar Xiv:233... 125200, 2023c.
Xiang Zhang, Senyu Li, Ning Shi, Bradley Hauer, Zijun Wu, Grzegorz Kondrak, Muhammad Mbduu-Maaged, and Laks VS Lakshmanan. Cross-modal consistency in multimodal large language models. ar Xiv preprint ar Xrx::2411100221 2024b.
17

---

Published as a conference paper at COLM 2025
Xiaoman Zhang, Chaoyi Wu, Ziheng Zhao, Weixiong Lin, Ya Zhang, Yanfeng Wang, and Weidi Xie. Pmc-vqa: Visual instruction tuning for medical visual question answering. arXiv preprint ar arXi::2305.10415, 2023d.
Luowei Zhou, Hamid Palangi, Lei Zhang, Houdong Hu, Jason Corso, and Jianfeng Gao. Unified vision-language pre-training for image captioning and vqa. In Proceedings of the AAAI conference on artificial intelligence, volume 34, PP. 13041-13049, 2020.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. at Xiv preprint ar Xiv:2304. 10592, 2023.
Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, et al. Internv13: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint ar Xiv:2504.10479 2025.
Shang Zhu, Xuefeng Liu, and Ghazal Khalighinejad. Chemqa: a multimodal question- and-answering dataset on chemistry reasoning. httos::/huuginggffcccc//// co/dttsees/ shangzhu/ChemQ4, 2024.
Yoel Zimmermann, Adib Bazgir, Zartashia Afzal, Fariha Agbere, Qianxiang Ai, Nawaf Alampara, Alexander Al-Feghali, Mehrad Ansari, Dmytro Antypov, and Others. Reflec- tions from the 2024 large language model (Ilm) hackathon for applications in materials science and chemistry, 2024. URL httpss///axiiv wtw.aa.ww......////////aa// 15221 15221.
18

---

Published as a conference paper at COLM 2025
A Extended Related Work
Modality imbalance. Despite the integration of visual and textual understanding in modern VLMs, an inherent challenge persists in achieving balanced performance across different modalities. Recent studies have uncovered significant inconsistencies between vision and language capabilitties in multimodal models, showing that even state-of-the-arr systems demonstrate varying levels of reliability depending on input modality (Zhang et al., 2024b; 2023c). This imbalance becomes particularly evident in complex spatial reasoning tasks, where VLMs surprisingly underperform compared to their text-only counterparts, often neglecting visual information when textual clues are available (Wang al., 2025). The un- derlying mechanisms driving this phenomenon have been theoretically explored, revealing a competitive dynamic between modalities during joint training where gradient-based opti- mization leads to only subset of modalities being effectively learned (Huang et al., 2022). Various approaches have emerged to address this challenge, including adaptive gradient modulation techniques that dynamically adjust the optimization pace for different modal- ities (Peng et al., 2022), protolype-based methods that specifically boost slower-learning modalities without interference from dominant ones (Fan et al., 2023), and explicit image-to- text conversion strategies that bridge the gap between simple and complex visual reasoning tasks (Park et al., 2025). However, existing approaches to evaluate modality imbalance lack rigorous cross-modal semantic equivalence, leaving the field without principled bench- marks isolate and measure modality- reasoning capabilities. Further discussions on related works about domain-specific VLMs are presented in Appendix A.
Domain-Specific VLMs. While general-purpose VLMs have shown remarkable capabilities across diverse tasks, a growing trend has emerged developing domain-speccaalize vision- language models that focus on particular fields requiring expert knowledge. NOTA (Tang et al., 2025) represents one such effort in the music domain, bridging the gap between two-dimensional score images and one-dimensional symbolic notation through a dedicated multimodal music notation dataset. In the medical domain, MedVInT (Zhang et al., 2023d) and Med-PaLM (Tu et al., 2024) leverage specialized medical visual-textual pretraining to enhance clinical reasoning and diagnosis. For scientific literature, SciMMIR (Wu et al., 2024) focuses on scientific diagram understanding and multimodal information retrieval. Other domain-specific efforts include ChartLlama (Han et al., 2023) for chart interpretation, Mplug-Doc -ocOwl (Ye al., 2024) for document understanding, and LayoulGPT (Feng et al., 2023) for graphic design. These specialized models often demonstrate superior performance in their target domains compared to general-purpon VLMs, highlighting the importance of domain-specific knowledge and training data in addressing specialized visual reasoning
tasks.
B Additional Results
Modality
Extraction Model
Average Accuracy
Agreement
Qwenn.5-7B-Instrrut
0.635
Language
0.997
Qwen22.-77D-IInstruc
0.635
Qwen2.5-7B-Instrruct
0.482
Vision
0.998
Qwen22.-72l-Innstruct
0.481
Qwen2.5-7B-Instrruct Qwen2..-72B-Instrruct
0.627
Vision-Language
0.998
0.627
Table GPT-4o results with different final answer extractors.
B.1 Semantic Equivalence
Admittedly, achieving perfect semantic equivalence across modalities presents theoretical challenges, our benchmark design implements rigorous controls to ensure comparable
19

---

Published as a conference paper at COLM 2025
Effect of Resolution Changes (Chess)
Effect of Black White Conversion (Chess)
Effect of Rotation (Chemistry)
0.8
0rgg Ha
mm Slace & While
Organ Nataree
Mod fell Nesolutan
0.6
3,238
00220
0.5
10000
0.024
2.001
.0.03
6.3
6.2
728
728
Figure 4: Effect of performance from visual transformations on Qwen2.5-VL models.
information content between visual and textual representations. To validate the robustness of our approach, we conduct experiments examining how visual transformations affect model performance, as shown in Figure 4. We systematically tested three common image transformations: resolution changes in chess diagrams, black-and-white conversion of chess boards, and 180-degree rotation of molecular structures Qwen 2.5 models.
Our results demonstrate minimal performance variations across these modifications. For resolution changes, observe differences of no more than +1.6% in accuracy. Similarly, black-and-whind conversion produces maximum deviations of #3.1%, while molecular rota- tion yields changes within +1.9%. The stability of performance across these transformations supports our claim that models are responding to the underlying semantic content rather than superficial visual properties. This is particularly notable given that these modifications substantially alter the pixel-level representation while preserving the symbolic information content.
B.2 Cross-model Agreement
The cross-model agreement analysis on semantically equivalent tasks reveals striking modality-dependent variations model consistency. When processing the same prob- lems in language (left panel), models demonstrate high within-family agreement (GPT variants: 0.79-0.86, Claude variants: 0.77-0.82) and moderate cross-family agreement (0.64- 0.75), suggesting that textual-symbolic reasoning elicits relatively consistent responses across models. However, this consistency deteriorates markedly in the vision modality (center panel), where agreement scores drop substantially, indicating that visual-spatial processing of the same semantic content produces more divergent model behaviors. This aligns with our finding that vision frequently underperforms language despite semantically equivalent information being provided.
B.3
Error Analysis
Our analysis revealed two distinct failure patterns in text versus vision modalities. These examples demonstrate fundamental limitations in how each modality handles structured information.
Carbon counting:
The text-only system incorrectly tokenized the SMILES notation as:
SMILES representation.
(cTncec((c22ccc))cc222233
Tokenized sequence.
20

---

Published as a conference paper at COLM 2025
0P13
1,000
0044
0.937
2.740
0.733
0.990
OPT-
0.000
0.004
0:307
3.503
0.313
2.44
GPY5
1,00
0.911
0.014
01222
0,770
3.79
0.79
P50
0..44
0.900
0:009
6.999
2.144
0.373
1:993
6.00
0.71
0.77%
5.16
6,09
3:796
1,033
0.664
0.586
2,000
2550
0.931
5.20
0.634
0.807
037
0.710
2.22
Nomat
9726
660
2.522
0.020
1000
Languaee
Vilan
Velen- Largugge
Figure 5: Cross-model agreement on semantically equivalent tasks presented in different modalities. Correlation matrices show pairwise agreement scores between GPT-5, GPT-5- mini, GPT-5-nano, Claude-4.1-OPp Claude-4-Sonned and Claude-3.7-Sonne models when solving the same problems in language (left), vision (center), and vision-language (right) formats, with darker blue indicating higher agreement.
Question: Which the following the correct number carbon atome in this compound? Ansmer 11
Question: Which tollowing pornect number unique simple paths from 3 5 the graph? Answer:
Net ()
Q.
1]
O,
Menntiying cerbon atoms: SMLEE
Only
Latt count the disnadara stinng
Lafk trace the posaible pattcc
intood has
The second "DchtsiFladd"" has t'' atomss The limal pant
and simeady counted.
1o 7- B D- Thix ls uaildd path.
sample becszen sgain.
Summinng up the carben atome:
andt ring: arome
> This docan' lood S.
doaent
Tooal 3 = atomaa, X
2 > o-8 Not single.
> > 2- Not single
Vision Only
Thanskon, Sanu ure snlque dir patha from ao
The berueen (CaH-5) carton
Vision Only
The pyrdine ring (C34 15h0 haw carbon atome.
any There ocditional
Let' anagaa gast
unique from
3 > - This aimmpe puth.
l This simplic
Sum the cartonn atoms:
stoma beroven ring:
to 1 7 & This is stmple path.
pyriiie
Figure 6: Case-study of text-based versus vision-based processing in VLMs, showing two incorrect counting of carbon atoms in chemical
error cases: (left) Qwen2.5-VL-72B--nntt structure due to SMILES tokenization, and (right) gemma-3-i22 1-1bbbitt hallucinated graph paths in a simple path counting problem. Both examples show discrepancies between text-only and vision-only representatinos.
['N', 'c', "1' 'n', 'cee', '(0', 'c', '2', "cc', 'F',, '), "cce', '2', 'F', ")c', ' "I'j
This results in counting only 7 carbon atoms instead of the correct 11. This error stems from text systems struggling with context-dependent character interpretation in specialized notations where 'c' signifies aromatic carbon. In contrast, the vision-only approach correctly identified 11 carbon atoms by properly recognizing the benzene ring (6 carbons) and pyridine ring carbons) in the 2D molecular structure.
21

---

Published as a conference paper at COLM 2025
QUESTION
MODEL OUTPUT
nambbe from is the grigh??
determine the number of unique simpll node mode graph, identify all possible putls that not reviait mnds.
A.1
pachs step-by-step:
1. Palk I:
C.4
Start node
Move
thak trottp to ncewer you MUST onn Car
guorien.
Path
Start node J.
Move to node
Move mode
WALLDIIN
Move node
Pathi
INVALID
3. Park
Shert ol made
Move to mode
CORRECT ANSWER
Move mode
Move node
Moviz node
Actuall palls from to
Path:
IWVALID
Only 2 valdd paths esist. The model hallocnnred connechore from 2.
unique simple putts
patts.
Mote: Edjer 2-7 snd 5-0 strrect stmally consectel at interaction point
option
Figure 7: Visual perception error example.
Unique simple path counting: When analyzing simple undirected graph, the text-only approach correctly identified exactly unique simple paths from node node through path enumeration. However, the vision-only approach erroneously counted 4 paths by hallucinating edges that don't exist the graph (0 + 5, 0 - 1, 1,1- - 7). This suggests vision systems may struggle with precise spatial reasoning in graphs, potentially interpolating connections based on spatial proximity rather than actual edge connections.
These examples highlight complementary strengths and weaknesses across input modalities. Text-based systems excel at following explicit rules but may fail with specialized notation requiring contextual understanding. Vision-based systems can interpret visual structures holistically but may hallucinate relationships based on visual proximity. Our findings suggest potential benefits from multimodal approaches that cross-validate across modalities to mitigate mopality-specific errors.
B.4 Final Answer Extractor
While VLMs often generate verbose rationales when prompted for multi-choice answers, need to extract their definitive selection of option A, B, C, or D robustly. We evaluate two
22

---

Published as a conference paper at COLM 2025
Table 3: Hyperparameter settings for benchmark creation and model inference.
Category
Parameter
Value
Node Limits
Min: Max: 9
Edge Limits
Min: 5, Max: 20
Graph Tasks
Path Count Offset
= 0.1 x edge count +1 1 from correct count
Image Size
400 400 px
Samples
200 per task
Carbon Offset
+3 from correct count
Hydrogen Count Offset
#3 from correct count
Chemistry Tasks
Weight Count Offset
220% of correct molecular weight
Image Size
400 400 px
Samples
200 per task
Max Puzzle Rating
1200
CP Offset
#300 centipawns from correct evaluation
Chess Tasks
Image Size
400 400 px
Samples
200 per task
Measure Limits
Min: 24, Max: 48 measures
Note Count Offset
from correct count
Music Tasks
Measure Count Offset
#3 from correct count
Image Size Samples
600 600 px
200 per task
Temperature Inference
Model Default
Temperature Extraction
0
Model Inference
Max New Tokens
8192
Max Model Length
16384
potential extractors: Qwen2.5-7B-Instruue and Qwen2..-72B-Instrruc with vast difference in the scale of parameters, comparing their extraction reliability on GPT--o outputs across all modalities. As shown in Table 2, both extractors achieved near-identical results with exceptionally high agreement across all modalities. The consistency between the 7B and 72B models indicates that answer extraction a relatively straightforward task that hardly benefit from additional model capacity. Therefore, we adopt the more computationally efficient 2e-n2.5-7BInnsruue as our standard extractor for all experiments, assigning special token Z in cases where no valid option could be extracted, rather than making random assignments that might artificially inflate performance metrics.
23

---

Published as a conference paper at COLM 2025
C Reproducibility
C.1 Hyperparameter Settings
Our benchmarking framework involves carefully designed hyperparameters across dif- ferent task categories and model inference settings. Table 3 presents the comprehensive hyperparameter configuration used in our experiments for reproducibility purposes.
C.2 Prompt Templates
Language-only Prompt Template.
(notation name): (notation)
(instructions)
A. {option_a}
{option_b] }
{option.c
{option.d}
Let's think step-by-step to answer the above question. One and only option is correct. If you are unsure, provide your best guess. If you believe none of the options are correct, select closest one. You MUST conclude with: The best option is [the_option_leteer where the [the_option_letter] MUST be one of A, B, C or D.
Vision-only Prompt Template.
(image}
(instructions)
(option_a }
(option_b b
C. (option_c)
D. (option.d)
Let's think step-by-step to answer above question.
One and only one option is correct. If you are unsure, provide your best guess. If you believe none of the options are correct, select the closest one. You MUST conclude with: The best option is tthe_option__etter where the [the_option] iite_ootton__etterr MUST be of A, B,
Language & Vision Prompt Template.
(image)
(notation.name):
(notation)
(instructions)
A. (option.a)
(option.bl
{option.cs )
{option.d}
Let's think step-by-step to answer the above question. One and only option is correct. If you unsure, provide your best guess.
24

---

Published as a conference paper at COLM 2025
If you believe of the options correct, select the closest one. You MUST conclude with: The best option is per__potio__eeterr. where the [che__optoon_eeter]] MUST one of A, B, C or D.
Prompt Template for Final Answer Extraction.
Here is the complete predicted answer for multiple-choice question)
***(prediction)***
Your task: Extract the final answer (the best option) from the text above. Ignore the reasoning process and any inconsistence in the above complete predicted answer.
It is usually in the format 'The best option is [letter]* at the end of the complete predicted answer.
If found, reply with the letter 'A', 'B', 'C', or 'D'. Otherwise, reply with 'Z'
25

---

Published as a conference paper at COLM 2025
C.3 Task Design Details
Our task designs focus on creating specialized problems that test the ability of models to understand information presented across different modalities. Each domain requires careful construction of semantically equivalent representations while ensuring appropriate difficulty levels and clear evaluation metrics.
C.3.1
Chess
Chess offers an ideal domain for testing modality equivalence due its well-established symbolic notation systems and visual representations via standard tools. We developed four chess-based tasks that assess different aspects of chess understanding and reasoning, ensuring that questions can be answered through either visual board inspection formal chess notation analysis.
Fork Detection. The fork detection task evaluates model's ability to identify pieces creating tactical forks on the chess board. A fork occurs when single piece simultaneously attacks two more opponent pieces. We constructed this task using the python-chess library to analyze positions from the Lichess puzzle database. For each position, we load the FEN -dorsysh-Edwards Notation) string and analyze all pieces on the board to identify those attacking multiple opponent pieces simultaneously using board.attacks() to count the number of opponent pieces under attack from each piece from python-chess library. A random piece creating a fork is selected as the correct answer, while three random non-forking pieces are sampled as incorrect options. All incorrect options reference legitimate pieces of the appropriate color on the board, making the task require genuine tactical understanding rather than simple pattern matching.
Legal Move Discrimination. The legal move discrimination task evaluates whether models can distinguish legal from illegal moves in complex positions. Chess positions frequently contain moves that appear valid but are illegal due to pins, checks, or other tactical con- straints. We constructed this task by loading chess positions from FENs and identifying legal moves using the board.legal laaad.egallmoove movee generator from the python-chess library. We then generate plausible-looking illegal moves from appropriate pieces but invalid destinations. These illegal moves fall into several categories: moves that would leave the king in check, moves of the correct piece color but to illegal destinations, and moves that appear geometri- cally plausible but violate piece movement rules. For each position, we select one random legal move and three illegal moves, presenting the position and candidate moves across all modalities.
Puzzle Solving. The puzzle solving task challenges models to find the best move in tactical positions drawn from the Lichess puzzle database. We carefully selected puzzles with maximum ELO rating to ensure appropriate difficulty for current models. For each puzzle, we parse the FEN position and the first move from the puzzle sequence, then apply this move to reach the position where the model must find the best continuation. The best move, provided by the dataset, is presented alongside three randomly selected legal but suboptimal moves as multiple-choice options obtained from python-chess library.
Position Evaluation. The position evaluation task assesses a model's ability to judge the relative advantage in a chess position. Professional chess engines like Stockfish quantify advantage using centipawn values, where positive values indicate an advantage for White and negative values for Black. We selected positions from a pre-evaluated Lichess database where positions had moderate advantages to avoid trivial cases, For each position, we created four evaluation options by taking the correct centipawn value and applying offsets to the correct answer. This task is particularly challenging as it requires holistic assessment of multiple factors including material balance, piece activity, king safety, and pawn structure that transcend simple piece counting to correctly evaluate positions.
All chess tasks utilize python-chess library's SVG rendering capabilities through brass.stg.bbardl) to generate visualizations, which are then converted to PNG images using
26

---

Published as a conference paper at COLM 2025
Cairosvg. Positions presented with appropriate board orientation, with board flipped when Black to move and with coordinate notations to ensure complete information is avail- able in the visual modality. For each task, carefully selected 200 samples to create balanced and challenging dataset.
C.3.2 Chemistry
Molecular chemistry provides another appropriate test domain for modality equivalence due to its dual representation systems: visual structural diagrams and symbolic SMILES notation. We developed four chemistry-based tasks that evaluate understanding of molecu- lar properties across different modalities, ensuring each task can be solved through either visual inspection of molecular diagrams or analysis of SMILES notation.
Carbon Counting. The carbon counting task tests model's ability to accurately count the number of carbon atoms in organic molecules. We selected molecules from the ChemQA dataset with substantial carbon content to create appropriately challenging problems. Using RDKit, we parse the SMILES notation of each molecule and programmatically identify the exact carbon atom count to yield the correct answer, with incorrect options differing by multiples of offset.
Hydrogen Counting, The hydrogen counting task is similar to the carbon counting one. Using RDKit's GelTolalNumHHs() method, we calculate the total hydrogen count for each molecule, including both explicit and implicit hydrogens. This task requires understanding of organic chemistry valence rules correctly infer hydrogen positions that may not be explicitly shown in the molecular diagram or directly encoded in the SMILES string.
Molecular Weight Calculation. The molecular weight calculation task assesses model's ability to estimate the molecular mass of chemical compounds. We use RDKit's Descrip- Mols.)) function to calculate the exact molecular weight of each compound based its atomic composition. For each molecule, we generate four options where the correct answer the actual molecular weight and incorrect options differ by a percentage- based offset.
Molecular Caption Matching. The molecular caption matching task evaluates semantic un- derstanding of molecules by requiring models to identify the most accurate description for given molecule. We curated a set of expert- molecule descriptions from the ChemQA dataset, where each description explains molecule's structure, function, biological activ- ity. For each target molecule, we present its structure along with four possible descriptions, only one of which correctly describes the molecule. To ensure challenging distractors, we embedding model to identify semantically similar but
use the aaatiiinnual-----r---nn-e incorrect captions based on cosine similarity.
The visual representations of all chemistry tasks are generated using RDKit's MoloaMoo/ollmaee() function with consistent size parameters. For each task, we carefully selected 200 samples to create balanced and challenging dataset.
C.3.3 Music
Musical notation provides a complex test domain for modality equivalence, as it combines visual symbols with structured temporal patterns. We developed four music-based tasks that evaluate understanding of musical compositions across different modalities, ensuring each task can be solved through either visual analysis of sheet music or interpretation of the symbolic ABC notation format.
Note Counting. The note counting task assesses a model's ability to accurately count specific note occurrences in musical compositions. Using the music21 library, we parse ABC notation from the Irishman dataset and systematically count occurrences of each note type from to using the score,flat, BeattgeeeeleeentBCC..... Note') method. For each piece, we randomly select target note that appears in the composition and create multiple-choice
27

---

Published as a conference paper at COLM 2025
options where the correct answer represents the actual note count, with incorrect options differing by an offset that scales based on the correct count.
Measure Counting. The measure counting task tests model's ability to identify the total number of measures in a musical composition. We use music21's parsing count measures from the ABC notation, applying the part.get werr-gfflleeenttByCCee (eaasurrer method to identify distinct measures in the score. As with the note counting task, the offset for incorrect options scales with the distance from the correct answer, making the task more difficult for pieces with ambiguous measure counts or complex repeat structures that might confuse measure counting.
Musical Form Identification. The musical form identification task evaluates a model's understanding of compositional structure. Using curated dataset of musical pieces with labeled forms from the music theory dataset', we present models with sheet music ABC notation and ask them to identify the correct musical form from options including Only One Section, Through Composed, Compound Binary, Compound Ternary, and American Popular forms. This task requires understanding of how musical sections relate to each other and recognition of structural patterns that define different compositional forms, testing higher-level music theory knowledge.
Rhythm Pattern Detection. The rhythm pattern detection task examines model's ability to identify specific rhythmic patterns within composition. Using custom pattern matching regular expressions that analyze ABC notation, identify measures containing specific dotted note patterns (including dotted sixteenth, dotted eighth, dotted quarter, or dotted half notes). For each composition, we select a measure containing the target rhythm pattern and create options with three measures that don't contain the pattern to make up the false answers.
All music tasks are presented in both standard five-line staff format generated using mu- sic21's rendering capabilities and symbolic ABC notation format. The sheet music images are standardized to 600x600 pixel square images, ensuring consistent visual representation. We carefully filtered and balanced each task category to create a comprehensive benchmark of 200 samples for evaluating music understanding across modalities.
C.3.4 Graph
We present a series of stochastic algorithms for generating graph-related tasks. Each algo- rithm dynamically creates suitable graphs with specific constraints to ensure appropriate difficulty levels and clear conceptual focus.
The framework follows a general pattern: (1) Generate candidate graphs with controlled parameters, (2) Filter graphs to ensure they satisfy task-specific constraints, (3) Generate correct solutions, plausible distractors, and incorrect options, and (4) Format and return the complete task with answer options.
Each specific task generator (Cycle Detection, Path Counting, Path Existence, BFS Traversal) implements this framework with task-specific constraints and choice generation logic.
abc notation with music theory
28

---

Published as a conference paper at COLM 2025
Algorithm 1: Generate Cycle Detection Task
Initialize ma..ttemmpss <- 50
for attempt to max_attempts do
Generate random directed graph D with few nodes
Convert D to undirected graph G if D not connected then
| continue end
if D has no cycles then
continue end
directed cyycles all simple cycles in D undirected cycles no new cycles undirected graph then continue end
-cclllsseedcyclesseeeeeeeeeeeee t all simple cycles in G
Design answer choices
Correct + random cycle from directed cycles Confusion + cycle in G but not in D b Generate incorrect answers
Incorrect two paths that:
Are not part of any cycle
Cannot form cycle
-re distinct
failed generate any answer then continue
end
return D, G, Correct, Confusion, Incorrect
end
return failure
29

---

Published as a conference paper at COLM 2025
Algorithm 2: Generate Path Counting Task
begin
Initialize max_attempts <- 50
for aftempt to max_affemmth do
Generate random undirected graph G with few nodes G not connected then continue end if too few nodes in G then
continue
end
Create and shuffle all possible source-target pairs foreach (source, target) in shuffled pairs do if no path exists from source to target then continue end
Count all simple paths from source to target
if path_count < 10 then
b Generate answer options
correct.answer + path_count
of <- [0.1 x edge_count + 1] options + empty list
for delta offt ofee do if delta + and path count + delta > 0 then
Add path count delta to options end end while loptions| < do of offse < offt + Add viable options using new offset end
if loptions| > 3 then
Sort options by distance from path_count
Keep only closest options
end
return G, source, target, path options
end
end
end
return failure
end
30

---

Published as a conference paper at COLM 2025
Algorithm 3: Generate Path Existence Task
begin
Initialize max max_attempts t- 50
for aftempt to max_aftempis do
Generate random directed graph D with few nodes Convert D to undirected graph if D not connected or too few nodes then
continue end
Create and shuffle all possible source-target pairs foreach (source, target) in shuffled pairs do if no path exists from source to target in D then continue
end
Get directed paths in D and undirected paths in G if too few or too many paths then
continue
end
if any undirected path differs from all directed paths then
source_node +- source
target_node target
b Generate answers
correct t- random directed path from source to target
tricky +- undirected path not in directed paths
Generate incorrect answers
incorrect <- empty list for up to 50 attempts do if lincorrect] = 2 then break end
D. Try different strategies generate invalid paths non-path one of:
1. Path with invalid edges
2. Path with repeated nodes
3. Random sequence of nodes if mon_pptt is valid and not duplicate then Add non path to incorrect end end
if correct + None and tricky + None and lincorrect] = 2 then return D, G, correct, tricky, incorrect end
end end
end
return failure end
31

---

end Published as a conference paper at COLM 2025
Algorithm 4: Generate BFS Traversal Task
begin
b Generate sui table graph and BFS levels for attempt = to max_atiempts do Generate random undirected graph G not connected or has too few nodes then continue
end
foreach start_node in shuffleed nodes do grouussroups t- BFS levels from start_node
fewer than 3 levels unsuitable level sizes then continue end
correct cereect_orerrreee + flattened level level-grouu groups (nodes sorted within levels) return G, start_node, level groups, covrect_order end end
b Generate answer options con confusing-distraccto <- None for seceral attempts do
Copy level grouups distractor_level Swap nodes between non-root levels distractor order +- flattened distractor levels if distractor _order + correct then con fusing distractor +- distractor Lordrr
break
end
incorrect opttionss + empty list for several attempts do cecoorrect_optione break
then
end
Try shuffling level structure Copy and shuffle non-root levels while keeping root fixed incorrect order <- flattened shuffled levels
incorrect order is valid and unique then Add to incorrect opptions
end end
Format and return results
Format all traversal options with level grouping Combine into option_lis and shuffle
correct iidx + index of correct option return G, option list, correct idx, level sizes
end
32

---

Published as a conference paper at COLM 2025
Qwen2.5-VL-78-instrrcl CIFAR-100
2wwn22.5WWL772--Insrruud SEAM
Language
SEAM
CIFAR-100
Vision Languagerittion
Figure 8: T-SNE visualization of mopality- embeddings from Qwen models.
D Internal Representation Alignment
SEAM also enables to investigate how VLMs process semantically equivalent information across different input formats internally. A fundamental capability of robust VLMs is to create unified representations across modalities that align on semantic content rather than cluster by input format (Pandey et al., 2022; Zhang et al., 2023a). Following insights from Huang al. (2024) that effective VLMs should narrow the modality gap in deeper layers, examine the final layer representations of both ween.5-VV-7B-Insst
and Qwen2.5- VL-72B-Instruct models to investigate whether current VLMs achieve this capability on our challenging benchmark.
We include CIFAR-100 (Krizhevsky et al., 2009) as a comparison baseline because repre- sents simple and well-established dataset, serving as approximate upper bound for cross-modal alignment. We extract embeddings from three input modalities: language-only, vision-only, and multimodal (language plus vision) for both datasets. These embeddings are then projected into owo-dimensional space using T-SNE (Van der Maaten & Hinton, 2008) for visualization and analysis. Fig presents these T-SNE visualizations from both model scales. For CIFAR-100, we observe that the models demonstrate integrations of embeddings across modalities, indicating fairly successful cross-modal alignment for simple categorical information. In contrast, our SEAM benchmark reveals persistent modality gaps at the final layer in both models, with embeddings from all three modalities forming distinct clusters with minimal overlap. This disparity in separation magnitude suggests the issue extends beyond any natural tendency to preserve modality-specific information for output purposes, both datasets would show similar patterns if that were the only factor. Instead, this separation demonstrates that despite impressive performance on standard benchmarks, current VLMs struggle to form unified representations when reasoning over semantically equivalent information presented in fundamentally different notation systems.
D.1 Embedding Extraction
For the Owen2.5-VLL-7B-Ine embeddings from each transformer layer E 2,., L} in the language decoder part, where represents the total number of layers. Given an input in modality m E (language(L), rision(VV, vision-language( L)), we extract the hidden states Ii E IR where Sm is the sequence length for modality m and d is the hidden dimension.
and
models analyzed, we extract
To obtain a single representative embedding for each input-modality pair, we apply mean pooling across all tokens:
5m
(1)
i,m
im.
Sm
This approach captures comprehensive representations for both image and text inputs, rather than relying solely on special tokens like [CLS] or [IMG], which might not fully encapsulate modality-specific information in intermediate layers (Jiao et al., 2024).
33

---

Published as a conference paper at COLM 2025
D.2 Cross-Modal Representation Analysis
D.2.1 T-SNE Visualization
We run T-SNE on the mean-pooled embeddings to visualize the embedding spaces across modalities. For the models involved, we extract embeddings from the final layer for inputs from CIFAR-100 and our SEAM benchmark in three modalities: language, vision, and language+vision. For the CIFAR-100 dataset, we use class labels as the language input. For our SEAM benchmark, we use the pure symbolic representation without further guidance prompts as the language input. The T-SNE algorithm is then applied with perplexity of 30 and 1,000 iterations to obtain two-dimensional projection that preserves local neighborhood relationships.
34


Reviews:
[{"review_id": "gN6vTIcdx0", "reviewer": "Reviewer_hEvb", "summary": "The paper presents a new benchmark for measuring the gap in visual vs. language capabilities in vision language models across a number of different domains (chess, chemistry, graph theory and music). The data is collected or generated such that all information to answer the question is available in either visual or language form, which previous benchmarks did not guarantee. Moreover, the visual representation is not simply an image of a text transcription, like previous OCR-based benchmarks. The authors conduct an evaluation of existing models on the benchmark and show that there is indeed a gap between visual and language capabilities and present some early hypotheses as to why.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The problem seems well motivated. The benchmark design is done carefully and rigorously. From what I understand, it does nicely fit into the previous literature.", "reasons_to_reject": "The empirical evaluation seems a bit thin after the presentation the benchmark. Table 1, Figure 2 and Figure 3 are all versions of the exact same data with different granularity. In addition, there are a number of claims in Section 4.2 that I don't think are well substantiated. I'll speak about them individually below:\n\n> \"One might hypothesize that as models become more accurate, they naturally converge... However, in the practical performance range of current models, evidence from Zhang et al. 2024b challenges this assumption... these results indicate that improving cross-modal alignment actively contributes to general reasoning capabilities\"\n\nI understand this claim as: the source of agreement cannot be attributed to accuracy improvements alone, where the support is drawn from a different paper entirely, not any additional experiments of this paper. I don't think Zhang et al. 2024b is great evidence for this particular claim because their evaluation used exactly the kind of text -> OCR benchmark that this paper avoids. I'd also want a specific pointer for which exact experiment of Zhang et al. 2024b the authors are referring to. Moreover, the last sentence in the quoted section above is a claim that is difficult to make precise. If it were true, one would expect that improving cross-modal alignment alone would result in, for example, stronger language reasoning abilities. But there is no such experiment in the text.\n\n> \"The severity of such textual perception limitation negatively correlates to the advantage of language inputs.\"\n\nI believe the way this claim is written is a bit tautological. What I understand is that tokenization errors are negatively correlated to language-only performance. If this is the case, I would want to at least see a plot attempting to quantify this, but there are just a few qualitative case-studies in Appendix C.2 along with this paragraph. Additionally I would want at least some indication that this kind of issue is true for the vast majority of tokenizers for the models studied.\n\n> \"In domains where language inputs face known severe tokenization difficulties, such as chemistry and chess, the failure of vision inputs to compensate for these limitations is evident across most VLMs.\"\n\nI interpret this claim as saying for cases where vision-alone is better than language-alone due to tokenization issues, vision-language fails to do better than vision-alone. This does not seem to be the case in Figure 2 (looking at Chemistry for Qwen2.5-VL-7B, InternVL-2.5-8B and Chess for Qwen2.5 72B, Qwen2.5 7B, InternVL 78B). These are the only cases where there's such a performance discrepancy. Many other models are still better using language-alone on Chess and Chemistry.\n\n> \"Such observations indicate that vision inputs are giving negative contributions, which empirically points to fundamental constraints in visual perception capabilities.\"\n\nI am not sure that these \"negative contributions\" are in any way \"fundamental.\" That would suggest that visual perception could not be improved, which is not a position I would take.", "questions_to_authors": "See Reasons to Reject above for the critical issues. I think there are a number of empirical results which are needed to support the claims made in the paper. \n\nOn a minor note, I'd recommend passing the paper through a grammar aide and spell-checker. There are a large number of minor grammatical issues that can be easily fixed.", "extra_scores": {}}, {"review_id": "6ilzgBaa8w", "reviewer": "Reviewer_EVRg", "summary": "This work presents a new benchmark, SEAM, designed to test vision-language model capabilities on semantically equivalent tasks that are presented in different modalities. Notably, SEAM focuses on tasks where an instance can be independently represented in the vision and text modality with modality-specific notation, e.g. an image of a chess board and a text representation of the chess pieces/locations. They selected four tasks, including chess boards, chemical compounds, music compositions and graphs. In these tasks, every example includes text and an image that are semantically equivalent and used for a QA task.\n\nUsing SEAM, this paper evaluates different VLMs on the language only input, the vision only input and the vision+language input (assuming I’m correctly understanding Table 1 and Figure 3; I don’t see this explicitly described). Model performance tends differ between modalities across all models. Better performance in the visual domain may be due to poor language tokenization, such as for chemistry inputs, whereas better performance language performance may be due to weaknesses in visual perception. Overall, SEAM demonstrates that all models show a modality imbalance.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- SEAM allows for probing of the language capabilities and visual capabilities independently. This differs from most vision-language benchmarks where, even when seeking to probe each modality, the examples inherently require both inputs. I also like that every example is deterministically bidirectional.\n- In addition to the benchmark, this paper also introduces a framework to continuously generate new examples. I appreciate this as we’re now in an era where evaluation datasets potentially leak into training data. Even though the tasks themselves are quite different, the authors chose easily available codebases for translating (text $\\leftarrow$$\\rightarrow$ vision) for each domain.", "reasons_to_reject": "- The tasks here (chess, chemistry, music and graph) are fairly domain/knowledge specific. It’s tough to discern whether failures are due to poor (text-/visual-) reasoning or whether it’s due to a knowledge gap. For instance, the Music results in Figure 3 are fairly low across the board for every setup. One could ask – *is the modality gap here actually significant or is this just a challenging task?*\n     - While this is touched on in L248-253, strong evidence isn’t really provided.", "questions_to_authors": "*Questions*\n- It’s not immediately clear to me what the “Vision-Language” results are in Table 1 and Figure 3. Is the model given both modalities as input for these experiments?\n\n*Minor typos*\n- L102: “ensure semantically equivalence” → “ensure semantic equivalence”\n- L257: “a molecular shown” → “a molecule shown”", "extra_scores": {}}, {"review_id": "RZRG48n2RR", "reviewer": "Reviewer_yu5A", "summary": "This paper introduces SEAM, a novel benchmark for evaluating vision-language models (VLMs) on their ability to reason consistently across semantically equivalent representations in different modalities. Unlike existing benchmarks that use OCR-derived image-text pairs, SEAM leverages domains with standardized notation systems in both visual and textual formats: chess (FEN vs. board images), chemistry (SMILES vs. structural diagrams), music (ABC notation vs. sheet music), and graph theory (adjacency matrices vs. node-edge diagrams).\nThe authors evaluate 13 state-of-the-art VLMs on this benchmark and discover significant modality imbalances across all models. They find a strong correlation between cross-modal answer agreement and overall accuracy, suggesting that models with better modality alignment tend to perform better generally.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Novel evaluation paradigm: SEAM provides a fundamentally new way to evaluate VLMs by testing their reasoning across distinct notation systems rather than just OCR-derived text, addressing a significant gap in existing benchmarks.\n\nRigorous benchmark design: The authors carefully select domains with standardized notation systems, ensuring semantic equivalence while providing challenging reasoning tasks that require domain-specific understanding.\n\nImportant empirical findings: The discovery that cross-modal answer agreement strongly correlates with overall accuracy provides actionable insights for future VLM development.\n\nThorough model evaluation: The paper evaluates a comprehensive set of 13 VLMs spanning both proprietary and open-source models of various sizes, providing valuable comparative analysis.\n\nInsightful error analysis: The detailed investigation of modality-specific failures (e.g., tokenization errors, visual perception limitations) helps identify specific areas for improvement in current VLMs.\n\nInternal representation analysis: The paper goes beyond surface-level performance metrics to examine how VLMs internally process semantically equivalent information across modalities.", "reasons_to_reject": "Limited domain coverage: While the four selected domains are well-justified, they represent only a subset of possible domains with dual representation systems, potentially limiting generalizability.\n\nTask diversity limitations: Each domain has only 4 tasks with 200 samples each, which may not capture the full range of reasoning capabilities required in those domains.\n\nComplexity control concerns: The paper doesn't fully address how well the benchmark controls for inherent complexity differences between modalities (a textual representation might inherently be more/less complex than its visual counterpart).\n\nLimited investigation of improvement strategies: While the paper identifies limitations in current VLMs, it offers relatively few concrete suggestions for how to address modality imbalance issues.", "questions_to_authors": "Have you considered extending SEAM to include other domains with standardized notation systems, such as circuit diagrams vs. SPICE netlists (as mentioned in your limitations)?\n\nFor the robustness tests of semantic equivalence (Fig. 5), have you considered more comprehensive transformations beyond resolution changes, B&W conversion, and rotation?\n\nThe paper mentions that cross-modal answer agreement correlates with accuracy. Have you investigated whether explicitly optimizing for cross-modal agreement during training could improve VLM performance?\n\nYou mention that tokenization errors are a significant issue, particularly for SMILES strings. Have you considered experimenting with specialized tokenizers for domain-specific notations?\n\nThe internal representation analysis shows persistent modality gaps in embedding space. Have you investigated at which layer in the model architecture this separation begins to emerge?", "extra_scores": {}}, {"review_id": "WUT42u3W2G", "reviewer": "Reviewer_NLx2", "summary": "This paper investigates the performance inconsistency of vision-language models (VLMs) when identical semantic content is presented in different input modalities, such as visual versus textual. The study reveals that current VLMs suffer a significant performance degradation when a piece of information is provided in visual modality as opposed to text, even when the semantic content remains the same. This raises notable questions about multi-modal understanding capabilities of these models—particularly their robustness and consistency across modalities.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper tackles a timely and important question: how reliable are VLMs when the modality changes but semantic content remains unchanged?\n- It highlights a notable performance gap when switching to the vision modality, which raises critical concerns around robustness and modality invariance in VLMs.", "reasons_to_reject": "- The paper lacks a quantitative error analysis to determine why vision modality leads to weaker performance. Is it a result of perception issues or reasoning limitations? Potential TODOs:\n    - Test whether vision inputs could be reliably transformed into equivalent textual representations using existing VLM capabilities—an analysis that could significantly complement the core findings.\n    - Test two-stage VQA solving architectures (e.g., VLM description followed by LLM reasoning [1]), which is increasingly common in practice and may show a different performance profile.\n- There is no comparison of  VLMs & their corresponding LLMs on the textual version of SEAM benchmark, leaving the cause of modality-induced degradation unclear.\n\n[1] Prism: A Framework for Decoupling and Assessing the Capabilities of VLMs", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Four reviewers initially found the paper to be above or just around (above and below) the acceptance threshold.

Reviewers pointed out the following strengths:
  + “The problem seems well motivated. The benchmark design is done carefully and rigorously.”
  + “SEAM allows for probing of the language capabilities and visual capabilities independently. This differs from most vision-language benchmarks where, even when seeking to probe each modality, the examples inherently require both inputs.”
  + “Novel evaluation paradigm: SEAM provides a fundamentally new way to evaluate VLMs by testing their reasoning across distinct notation systems rather than just OCR-derived text, addressing a significant gap in existing benchmarks.”

In contrast, reviewers pointed out the following main weaknesses:
  - "The empirical evaluation seems a bit thin”
  - "a number of empirical results are needed to support the claims made in the paper.”
  - "Limited domain coverage"
  - "Limited investigation of improvement strategies"

After a lively discussion with the authors, many revisions were suggested and accepted by the authors that will make the presentation of the paper stronger. The main remaining questions revolve around whether this benchmark paper points out precise issues for further improvement that were difficult to measure previously, and whether the results are a bit too thin for a full paper.

After discussion, all reviewers recommend to accept the paper (scores: 6,6,7,6).

Having reviewed the paper, the reviews, and the discussion, the AC tends to agree that the paper is above the acceptance threshold and encourages the authors to incorporate the new insights from the discussion into their final manuscript.
Decision: Accept