Visual Graph Arena: Evaluating Visual Conceptualization of Vision and Multimodal Large Language Models
Authors: Zahra Babaiee
Peyman Kiasari
Daniela Rus
Radu Grosu
Venue: ICML 2025 poster (2025)
Keywords: Graph
Visual Reasoning
Reasoning
LLM
Multimodal LLM
Primary area: applications->computer_vision

Abstract: Recent advancements in multimodal large language models have driven breakthroughs in visual question answering. Yet, a critical gap persists, `conceptualization'—the ability to recognize and reason about the same concept despite variations in visual form, a basic ability of human reasoning. To address this challenge, we introduce the Visual Graph Arena (VGA), a dataset featuring six graph-based tasks designed to evaluate and improve AI systems’ capacity for visual abstraction. VGA uses diverse graph layouts (e.g., Kamada-Kawai vs. planar) to test reasoning independent of visual form. Experiments with state-of-the-art vision models and multimodal LLMs reveal a striking divide: humans achieved near-perfect accuracy across tasks, while models totally failed on isomorphism detection and showed limited success in path/cycle tasks. We further identify behavioral anomalies suggesting pseudo-intelligent pattern matching rather than genuine understanding. These findings underscore fundamental limitations in current AI models for visual understanding. By isolating the challenge of representation-invariant reasoning, the VGA provides a framework to drive progress toward human-like conceptualization in AI visual models. The Visual Graph Arena is available at: \href{https://vga.csail.mit.edu/}{vga.csail.mit.edu}.

Visual Graph Arena:
Evaluating Visual Conceptualization of Vision and Multimodal Large Language Models
Zahra Babaiee
Peyman M. Kiasari *1
Daniela Rus
Radu Grosu
Abstract
between current AI capabilities and the goal of achieving artificial general intelligence (AGI). One critical aspect of this gap is the ability of Al models to engage in deep rea- soning and understand fundamental concepts from visual inputs (Gupta & Kembhavi, 2023), which is essential for reaching human- cognitive abilities (Han al., 2019). This gap is particularly evident in what we term 'conceptual- ization' the ability to recognize and reason about concepts across different representations, even when their visual form varies significantly.
Recent advancements in multimodal large lan- guage models have driven breakthroughs in visual question answering. Yet, a critical gap persists, 'conceptualization" - the ability to recognize and reason about the same concept despite variations in visual form, basic ability of human reason- ing. To address this challenge, we introduce the Visual Graph Arena (VGA), a dataset featuring six graph-based tasks designed to evaluate and im- prove AI systems' capacity for visual abstraction. VGA uses diverse graph layouts (e.g., Kamada- Kawai vs. planar) test reasoning independent of visual form. Experiments with state-of-the- art vision models and multimodal LLMs reveal a striking divide: humans achieved near- near-perfect ac- curacy across tasks, while models totally failed on isomorphism detection and showed limited suc- cess path/cycle tasks. We further identify be- havioral anomalies suggesting pseudo-intelligent pattern matching rather than genuine understand- ing. These findings underscore fundamental limi- tations in current Al models for visual understand- ing. By isolating the challenge of representation- invariant reasoning, the VGA provides a frame- work to drive progress toward human-like concep- tualization in Al visual models. The Visual Graph Arena available at: pea.csail.mit.ede
Humans have a long history of developing various methods of visual representation to simplify complex information and enhance understanding. These representations span across diverse domains, such as chemical structures, logic circuits, flowcharts, and statistical data, each with its unique set of visual conventions and abstractions. Figure show- cases the diversity and complexity of visual representations that humans regularly encounter and interpret. In Figure 1-a, two different visualizations of the same logic circuit are presented, illustrating how identical functional relationships can be conveyed through varied graphical styles. Similarly, Figure 1-b depicts two representations of the same chemical structure, highlighting the challenge of recognizing under- lying molecular configurations when presented in different visual forms. Figure 1-c features a graph drawn in two dis- tinct layouts, both representing the same underlying data structure and relationships. These examples underscore a crucial aspect of visual reasoning and highlight the core challenge of 'conceptualization" the ability to consistently understand different visual representations of the same con- cept, task that is typically intuitive for humans but remains challenging for AL.
1. Introduction
In recent years, the fields of object recognition (Girshick et al., 2015) and visual question answering (Antol et al., 2015; Goyal et al., 2017: Yang et al., 2022) have witnessed remarkable progress, pushing the boundaries of artificial intelligence. Despite these advancements, a gap remains
Defining conceptualization: generalize beyond superficial patterns lies at the heart of human-like reasoning. While out-of-distribution generaliza- tion encompasses various challenges, we focus specifically on models' ability to understand different representations of the same underlying concept. To precisely characterize this aspect, we use the term "Conceptualization".
Undoubtedly, the ability to
EEqual contribution *CPS. TU Wien, Vienna, Austria *CSAIL, MIT. Correspondence to: Peyman Kisari <pey- man man kwn.kiiiiiiiiin @aassaritwwiin.ccc.....
Conceptualization is a specific subset of aur-of-distribution (OOD) generalization that examines a model's ability to recognize and reason about OOD data whose underlying
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Visual Graph Arena
OH
(a) Two representations of logic circuit.
(b) Two depictions of a chemical structure. (c) A graph drawn in two different layouts.
Figure I: Examples of different visual representations of identical concepts across different scientific and technical domains. Each pair of images illustrates the challenge of recognizing identical relationships or structures when presented in varied visual formats.
concept aligns with the training distribution, but whose representation diverges in form, structure, or manifestation.
small enough such that is easy to visually inspect their prop- erties, but still large enough to create enough samples for the datasets.
Unlike broader OOD generalization challenges-whicl may involve shifts in spurious correlations, contextual biases, or surface-level ceatures--coneepuuuliaaioo isolates the capac- ity to abstract and transfer core conceptual invariants across representational variations. (Figure 1)
The tasks in the Visual Graph Arena revolve around three main concepts: isomorphism, path, and cycle, with two sub- tasks for each concept. The training sets are drawn using specific layouts (KamadaKKawll or random, depending on the task), while the test sets employ different layouts to assess whether models can learn and apply the concepts, re- gardless of their visual presentation, similarly to the human capabilities.
For instance, while understanding different accents in speech recognition represents OOD challenge, it falls outside the scope of conceptualization, the core linguistic constructs remain consistent. Indeed, we would not attribute an American's 's difficulty understanding a Scottish accent to lack of conceptual understanding of English.
We evaluated several leading vision models, including ViT (Dosovitskiy et al., 2020), Swin Transformers (Liu et al., 2021), and ConvNeXt (Liu et al., 2022), on the Visual Graph Arena datasets. The results were very revealing: All models failed the graph isomorphism detection tasks. Additionally, these models underperformed relative to the human capabilities on the other tasks, when tested with graphs drawn differently than the layouts in their training sets. Our investigation also revealed a fundamental reason- ing gap in current state-of-the-ard vision and multimodal language models. When we tested advanced AI models, such as GPT-ol of OpenAl and CJaude-3.5-Sonne of An- thropic, tasks that humans handle intuitively based on visual information alone, the results were striking. In graph isomorphism detection task, where pairs of graphs were drawn in different layouts (Kamada-Kate et al., 1989) and planar), all tested models failed to recog- nize the identical underlying structures, while human sub- jects achieved more than 90% accuracy. This stark contrast between human and AI performance highlights critical challenge in AI conceptualization.
Why Graphs? Among the various visual representations, graphs, consisting of nodes and edges, offer an ideal test case for studying conceptualization in AI systems. While these graphical representations are easily interpreted by humans, particularly when the graphs are simple and have a limited number of nodes, they present a challenge for Al models the ability to abstract the underlying structure regardless of visual layout.
is worth noting that while computational algorithms exist for solving graph problems through matrix operations or adjacency list traversal, our focus here is fundamentally dif- ferent. The challenge lies not in developing grepe-th solvers, we aim to evaluate whether Al models can solve these problems using only visual reasoning and conceptual abstraction, without relying on algorithmic computation. This distinction is crucial align more closely with human- like reasoning.
(Kamada
This insight led to the creation of the "Visual Graph Arena", a dataset and benchmark consisting of six graph-based tasks. These tasks are designed to evaluate and enhance the visual reasoning and conceptualization capabilities of AI models through various challenges. The datasets contain diverse range of training samples, from 27,000 150,000, ensuring robust training environments. The graphs have 8-9 nodes,
The Visual Graph Arena dataset helps identify Al systems' limitations in processing conceptual visual data while pro- viding a foundation for improvement. By testing models' conceptual reasoning abilities, we work bridge the gap be- tween human and machine interpretation of abstract graph- ics.
2

---

Visual Graph Arena
(a) Random Layout
(b) HamaddaKawaz Layout
(c) Planar Layout
Figure 2: Illustration of single graph displayed in three distinct visualization layouts to highlight the impact of graphical representation on perceived structure and complexity.
2. Related Work
of-thought reasoning (Visual CoT), compositional robotic reasoning (ClevrSkills), mathematical Olympiad problems (Children's Math Olympiads evaluation), commonsense riddles (Visual Riddles), and multimodal math reasoning (MATH-Vision)
Graph Datasets. Graph datasets are cornerstone in numer- research domains (Hu al., 2020), ranging from social network analysis to protein micro molecules (Morris et al., 2020). These datasets are typically divided into two main types: those that represent real-world information in graph form, like social media connections or road networks, and synthetic graph datasets crafted for specific computational experiments.
Vision) (Gong et al., 2024; Shao et al., 2024; Guetta et al., 2024; Wang et al., 2024; Ma al., 2024: Cherian et al., 2024; Haresh et al., 2024). A recent study introduces Vision Graph, benchmark for visual reasoning in graph theory problems (Li et al., 2024), with two key limitations: its circular graph layouts become visually ambiguous with in- creasing nodes, and its dataset size is insufficient for training dlta--ntenssie models.
Among synthetic datasets, the variety is broad. For instance, various works have introduced datasets aimed at tasks like counting substructures within graphs (Chen et al., 2020) or distinguishing between different families of non-isomorphic graphs (Wang& & Zhang, 2023; Abboud et al., 2020; Balcilar al., 2021; Murphy et al.. 2019). These synthetic datasets primarily designed to benchmark and improve graph neural networks (GNNs). They often include data of nodes, edges, and additional features facilitate computational processing and analysis tailored to GNN architectures.
Our work introduces the "Visual Graph Arena" graph- based dataset designed not only to evaluate Al model ca- pabilities in solving graph theory problems but also, higher level, to assess their conceptualization ability, This is achieved by systematically varying graph layouts during both training and testing, thereby avoiding over- fitting specific patterns and isolating reasoning chal- lenges from perceptual artifacts. By emphasizing scalability (27 (275000-55,,78 samples per task) and layout diversity, ensure that all samples remain visually solvable while prob- ing deeper cognitive processes. This focus on practical solv- ability-validatee through human ieailaatioondssiiggguuee our benchmark as robust tool for advancing visual graph understanding and fostering models capable of generalizing across different visual contexts.
However, a gap exists in these offerings concerning visual interpretation. Most existing graph datasets are structured for algorithmic GNN consumption and do not address the challenge of visual graph reasoning directly. They opti- mized for models that process graphs textual or abstract data structures rather than as visual entities.
Visual Reasoning Datasets. Recent advancements in ma- chine learning have seen burgeoning interest in tasks that interrogate visual scenes through
3. Visual Graph Arena
(QA)
formats (Johnson al., 2017; Antol et al., 2015).
This section introduces the Visual Graph Arena Dataset, meticulously crafted to challenge and evaluate the visual reasoning capabilities of Al models through a series of graph-basee tasks. The dataset is structured around three primary concepts, each divided into two tasks, resulting in total of six distinct datasets. These datasets are designed to rigorously test various aspects of graph understanding and
Visual QA tasks have driven progress in multimodal reason- ing, with datasets like CLEVV (spatial/relational reasoning) and FigureQA (scientific plots) using synthetic data to iso- late challenges (Johnson et al., 2017: Kahou et al., 2017).
Recent benchmarks address diverse aspects: long-context document understanding (MMLONGBENCH.-OOG chain-
3

---

Visual Graph Arena
Visual Graph Arena
Graph Isomorphism Tasks
Graph Path
Graph Cycle
Finding Tasks
Finding Tasks
Task#1: Easy Isomorphism
Task#1: Hamiltonian Path
Task#1: Hamiltonian Cycle
Yes-No question on whether two given graphs are isomorphic
Yes-No question
Yes-No question
whether the given graph has a Hamiltonian path
whether the given graph has a Hamiltonian cycle
In false labels the graphs are chosen randomly
Task#2: Shortest Path
Task#2: Biggest Chordless
Cycle
Find shortest path length between two marked nodes
Task#2:
Find the biggest chordless cycle in
Hard Isomorphism
the given graph
Yes-No question on whether two given graphs are isomorphic
The question has four choices: 1, 2, or
The question has four choices: 5, or 6
false labels the graphs are degree equivalent
Figure 3: Overview of the 'Visual Graph Arena Benchmark' detailing the division of tasks into categories of Graph Isomorphism, Graph Path Finding, and Graph Cycle Finding. Each category further breaks down into two subtasks.
reasoning.
3.1. Graph-Isomorphism Tasks
In order to ensure that the graphs are both manageable and visually interpretable, we have constrained the number of nodes to range between 8 and 9. This node count strikes balance between maintaining visual simplicity and provid- ing sufficient complexity for generating robust training and test sets, which are suitable for deep learning applications.
This task focuses on graph isomorphism, utilizing graphs consisting of 8-9 nodes, to test the models' capabilities in recognizing isomorphic structures. Graph isomorphism refers the concept where two distinet graph representa- tions share the same number of vertices, edges, and connec- tivity patterns, essentially making them indistinguishable in graph-theoretic terms despite their different visual layouts. To assess this, the task presents pairs of graphs drawn in different layouts, one Kamada-Kawal and the other in planar, and models must determine whether the graphs isomorphic.
The tasks are devised around core graph concepts that are intuitively recognizable by humans when interacting with small graphs:
1. Graph Isomorphism: Tests the ability to determine the isomorphic nature of graph pairs.
The task is structured to ensure a balanced approach: each set contains an equal number of isomorphic and non- isomorphic graph pairs. This balance is crucial to prevent models from learning and exploiting label distribution biases instead of mastering the intended isomorphic detection.
2. Path Finding: Focuses on the capability to identify various path types within the graphs.
3. Cycle Finding: Challenges the models to detect differ- ent types of cycles.
Difficulty Levels:
The graph-isomorphism-rroooo reecogigiiinoooiiiiinoiioo task demands quite sophisticated reasoning from the models, as superficial attributes like edge count or node degree are in- sufficient for determining isomorphism. The models must engage in deeper analytical processes to also discern under- lying structural equivalences or differences.
Figure 3 shows an overview of the Visual- Seeual-Grap-AAree Benchmark tasks and datasets, and Table shows the break- down of the tasks with the number of samples in the training and test set for each task.
Easy: In the easier version of this task, the second graph in each pair is selected randomly from the set of all possi- ble graphs with 8-9 nodes. Given the vast diversity within
In the subsequent subsections, we will describe each task in detail.

---

Visual Graph Arena
this family, non-isomorphic pairs often exhibit clear, eas- ily discernible differences, such as varying node degrees, simplifying the detection of non-isomorphism.
nodes are specifically marked. The goal is to find the length of the shortest path between these nodes, with potential answers of 1, 2, 3, 4. This setup frames the task as classification problem with four classes. The training dataset contains graphs drawn from the random layout. The test set samples are drawn from random, Gamada-Kawal and planar layouts, forming three testing variations.
3.3. Graph-Cycle-Finnn Tasks
Figure 4:
Two non non-isomorphic degree- equivalent graphs with the degree sequence of (3, 3, 2, 2, 1, 1}.
These tasks and associated datasets test the ability of a model to understand the notion of cycles within graphs, where each graph consists of 8-9 nodes. As before, these tasks also feature two levels of difficulty, to assess different aspects of the models' capabilities.
The hard category intensifies the challenge by se- lecting graph pairs that are degree-equivalee These pairs visually different but share identical numbers of edges and identical node degrees. Figure 4 shows example of non-isomorphic degree- degree-equivalent equivalent pair of graphs.
Hard:
Hamiltonian Cycle: the graph contains a Hamiltonian cycle, that is, a cycle which visits each vertex exactly once. The dataset is bal- anced between graphs that do and do-not contain a Hamil- tonian cycle. To evaluate the models' ability to generalize their learning, similar to the Hamiltonian path task, the train- ing set graphs are visualized in the Kamada- Kawai layout, whereas the test set is drawn in planar layout.
The goal here is to determine whether
These varied levels of difficulty are designed to progres- sively train and evaluate the Al's depth of understanding and its ability to perform complex visual reasoning, mir- roring the cognitive processes humans use interpreting graphical data.
3.2. Beahh-Path-Finndiie Tasks
The Graph-Path Sauph-Path-Findiing Findinn datasets explore the model's ability to understand the concept of path within graph, where each graph contains 8-9 nodes. The task is divided into two subtasks:
Figure 6: example for the biggest-chordless-eclle prob- lem in graph with 10 nodes.
In this challenging scenario, the task involves identifying the size of the largest chordless cycle in graph. A chordless cycle is defined as one where no two vertices of the cycle are connected by an edge that is not part of the cycle itself. Figure 19 shows an example of the biggest chordless cycle
Biggest Chordless Cycle:
graph with 10 nodes. The possible answers are 3, 4, 5, or 6, effectively making this classification task with four options. Similar the previous task, the training and testing samples drawn in the Kanaa and planar layouts, respectively.
Figure 5: Example of Hamiltonian path, a path that visits each node a single time.
The objective is determine whether graph contains Hamiltonian path-a path that visits each vertex exactly once. Figure 5 shows an example of such a path in a graph. This is essentially a binary classification problem, with labels indicating Hamiltonian path. To maintain balance, the dataset includes an equal number of graphs with and without such paths. The training-set graphs are drawn in Hamada-Kawal layout, and the test-set graphs are drawn in planar layout. This variation in the training and testing layouts helps to assess the conceptualization ability of the models.
Haminitonian Path:
The detailed process of the creation of the Visual-Graph- Arena datasets and example samples of each of the tasks they include presented in the Appendix.
presence or absence of
4. Experiments
In order to rigorously evaluate the effectiveness of Visual Graph Arena, we conducted a series of experiments, aimed at benchmarking both human and Al performance on the graph- based tasks. Initially, we established a baseline by testing human subjects, providing a comparative standard
Shortest Path: This task presents graphs in which two
5

---

Visual Graph Arena
Table I: Number of samples in the training and test sets of the Visual Graph Arena tasks.
Isomorphism
Path
Cycle
Easy
Hard
Hamiltonian
Shortest
Hamiltonian
Biggest Chordless
Training Samples
140,000
127,374
25,000
80,000
69,935
80,000
Testing Samples 15,718
14,298
2,480
8,672
7,740
6,484
Table 2: Comparative performance of humans, vision models, and MLLMs Visual Graph Arena tasks. Human participants achieved near-pen cear-perfect accuracy (88.2-100%), excelling in shortest path identification. Vision models failed isomorphism tasks with CenvNeXt outperforming transformer-ban based models in non-failing tasks. MLLMs showed near-random performance except GPT-ol, which achieved limited success in shortest path (55%) and Hamiltonian cycle (66.65 6%)) tasks.
Isomorphism
Path
Cycle
easy
hard
hamiltonian'
shortest path
hamiltonian'
chordless
random
kawai
planar
human
95.0
91.6
96.6
98.3
100
100
93.3
88.2
Vit Base
FAIL3
FAIL
57
67.4
66.7
64.4
67.8
32.3
Swin-T Base
FAIL
FAIL
65.8
68.6
65.7
65.4
71.1
34.4
ConvNext Base
FAIL
FAIL
72.9
73.3
82,4
73.3
74.9
36.3
SigLIP Base
54.4
FAIL
59.5
25.2
63.5
28.0
DINov2 Base
FAIL
FAIL
56.8
36.4
FAIL
31.1
GPT-ol
FAIL
FAIL
FAIL
55.0
66.0
FAIL
GPT-4o
FAIL
FAIL
FAIL
FAIL
FAIL
FAIL
Claude 3.5 Sonnet
FAIL
FAIL
FAIL
FAIL
FAIL
FAIL
1 These tasks are trained kawai layout and tested planar layout.
2 Shortest path task is trained on random layout, and tested on random, kawai and planar layouts.
3 We use FAIL to indicate near-random performance, borrowing the term from (Tay et al., 2021).
for assessing AI capabilities in similar conditions. Follow- ing this, we moved on to evaluate advanced multimodal- including GPT-ol and Claude 3.5 Sonnet, to test their profi- ciency in visual reasoning tasks that demand an integration of visual and textual analysis. Subsequently, we focused on the performance of leading vision models, which were specifically trained on our datasets and then tested the designated test sets containing a different graph layout.
Vision Models:
We tested three categories of vision models:
Supervised Pre-trained Models: VIT, ConvNeXt, and Swin-T Base, each pre- pre-trained on the ImageNet dataset using supervised learning with labeled data.
Multimodal Vision-I Laguage Model: SigL.P Base, pre- trained on image-text pairs.
Self-Supervised Vision Model: DINOv2 Base, trained on unlabeled image data.
We detail each experiment, and discuss the methodologies employed, the models tested, and the results obtained.
All models were fine-tuned on our tasks across 10 epochs. Details regarding the training parameters, settings, and addi- tional considerations are provided the Appendix.
Humans: To establish baseline of human performance on the VGA tasks, we conducted experiments with subjects with total of 24 questions, distributing four questions per task. Participants were drawn from the student and staff population of the engineering faculty, possessing general understanding of graph concepts, though not specifically in graph isomorphism, Hamiltonian paths and cycles, or chordless cycles-men most requested definition and clarity of these concepts. More detail is provided in the Appendix.
We evaluated GPT-4o and ol, Claude 3 Opus and 3.5 Sonnet, and Google Gemini on 100 random samples per task to determine their ability to handle graph-based vi- sual reasoning. Unfortunately, all models performed poorly across tasks, with ol being the only exception, showing higher than random performance in shortest path and Hamil- tonian cycle problems.
MLLMs:
6

---

Visual Graph Arena
4.1. Results
Hamitonian Path
leomorphiom
Random
Hard
Human
Table Confusion matrix for GPT-ol performance on the shortest path task. Rows represent ground-truth shortest path lengths (1-4), columns show model predictions. Val- ues indicate prediction percentages (diagonal = correct clas- sifications). The model performs best on length 2 (69% accuracy) but frequently confuses lengths 1 1-2 (67%), sug- gesting difficulty distinguishing adjacent node path lengths.)
Rose
Swhh-T Base
ComoNolt Bose
Shortest Path
Isomorphiem
Fawa
Easy
Shorlest Path Planar
Ragent Chardless Cyew
Hamilonian Cyde
Output
Label
4
Figure 7: Breakdown of the performance comparison of the VIT, Swin, and ConvNeXt models to humans the Visual Graph Arena tasks. The central area displays the performance of a random agent.
Length 1
28.0
67.0
5.0
0,0
Length2 2
18.0
69.0
13.0
0.0
Length 3
0.0
40.0
60.0
0.0
Length 4
0.0
5.0
40.0
55.0
Figure 7 illustrates the performance comparison of the trained models on the tasks of the VGA benchmark. The performance of random agent is in gray. On the graph iso- morphism tasks all models performed like random agent.
Table 2 presents the results of the experiments conducted on human subjects, vision models, and MLLMs for the Visaul Graph Arena tasks. The human participants demon- strated strong performance across all tasks, achieving 100% accuracy in finding the shortest path length and over 90% accuracy the other tasks. Finding the length of the biggest chordless cycle proved to be the most challenging for human subjects, with an accuracy of 88.2%, which was their lowest performance among all tasks.
Furthermore, we evaluated the performance of advanced MLLMS on the Visual Graph Arena tasks. Quite surpris- ingly, most tested MILMs, including GPT-4o, Claude 3.5 Sonnet, 3 Opus, and Google Gemini, failed solve any of the given tasks. However, GPT-ol demonstrated partial success, achieving non-random accuracy on two specific tasks: shortest path (55% accuracy) and Hamiltonian cy- cle (67% accuracy). Our analysis reveals that GPT-ol's above-random performance on the Hamiltonian cycle task is primarily attributed to the model's ability to identify graphs with leaf nodes, which trivially cannot contain Hamiltonian cycles. When cases involving leaf nodes are excluded, GPT- ol's 'ss performance drops to near-random levels, which may also explain its failure on the related Hamiltonian path task. This suggests that the model relies on specific structural cues rather than a deeper understanding of graph theory.
For the vision models, we report the best validation accuracy achieved during the training epochs. Notably, all models except SigLIP failed in the isomorphism tasks, with SigLIP achieving 54.4% accuracy on the easy isomorphism task while still failing on the hard variant. This indicates a sig- nificant limitation in most models' ability to reason about graph isomorphism from visual inputs. In the other tasks, the models exhibited relatively poor performance, consid- ering that the labels were either binary or limited to four choices. The chordless cycle task emerged as the most chal- lenging for all vision models, with the ConvNeXt model achieving the highest accuracy of 36.3%. An interesting observation is that the Xo model consistently outper- formed the Vision Transformer (VIT) and Swin Transformer Tw models on all tasks where the models did not com- pletely fail. This performance gap was most significant in the task of finding the shortest path length in the Kawai layout, where ConvNeXt achieved an accuracy of 82.4%, surpassing ViT and Swin-T by 17.7 and 16.7 percentage points, respectively. The newer models, SigLIP and DINov2, showed mixed results, with SigLIP demonstrating some ca- pability in isomorphism detection but both models generally underperforming compared to ConvNeXt. This finding sug- gests that convolutional architectures may be more effective than transformer-bart based models in capturing and reasoning about visual concepts in graph representations.
For the shortest path task, GPT-ol exhibited varying per- formance across different path lengths, shown in Ta- ble 3. The confusion matrix highlights that the model performs best when the shortest path length is 2 (69% ac- curacy) but frequently misclassifies adjacent nodes, such as confusing lengths 1 and 2. These observations under- score the limitations of current MLLMs in handling basic tasks, particularly when visual reasoning on graph-structured data is required.
The results of our experiments reveal significant challenges and limitations in the current state of the art vision mod- els and MLLMs when it comes to reasoning about graph- structured visual data. While human participants exhibit strong performance on the Visual Graph Arena benchmark tasks, the AI models struggle to match human-level rea-
7

---

Visual Graph Arena
soning capabilities. These findings emphasize the need for further research and development of Al systems that can effectively understand and reason about abstract graphical representations, bridging the gap between human-like vi- sual understanding and current Al capabilities. Our Visual Graph Arena may help to overcome these limitations.
its above-chance success rate while failing demonstrate the conceptualization of genuine intelligence.
This limitation becomes particularly significant given the scale of contemporary training. As noted by Sutskever, current models have been trained on nearly the entire inter- net (Sutskever, 2024), and our investigations confirm sub- stantial exposure to graph-related related content. This brings into question whether current probabilistic training approaches are inherently incapable of achieving conceptualization. Be- ing said, other training methods like reinforcement learning, due to their foundations in biological learning processes, may serve as a path forward for solving this challenge.
5. Conceptualization Analysis in GPT-ol
In our evaluation, we observe behavioral patterns in MI.LMs, particularly o1, that fundamentally diverge from mature human intelligence, that questions the capability of these models to conceptualize. Specifically, we identified two distinet anomalies.
6. Conclusions
5.1. The Middle-Score Anomaly
In this paper, we introduced the Visual Graph Arena, col- lection of six datasets designed designed to evaluate Al models' capacity for visual reasoning and conceptual un- derstanding through graph-based tasks. By focusing on iso- morphism detection, path finding, and cycle analysis across diverse visual layouts, our work isolates the challenge of conceptualization the ability to recognize invariant con- ceptual properties despite variations in representation.
The Middle-Score Anomaly emerges in atomic-task as- sessments of intelligent agents where only extreme perfor- mances (perfect or minimal) are expected, while intermedi- ate performance is anomalous.
Consider identifying numbers ending in zero atomic task. For human, we expect performance to be near either 100% or near 50% (indicating guessing, not having task knowledge). An 80% score would be anomalous because this task cannot be partially known-one either knows to check the last digit or doesn't.
Our experiments revealed performance gap between hu- mans and wate-of-the-art Al models. Humans achieved near-perfect accuracy (88-100%) across tasks, while vision models struggled significantly, especially when presented with graphs drawn using different layouts. MLLMs (GPT- ol, Claude 3.5 Sonnet) struggled to perform even basic graph reasoning tasks often performing no better than ran- dom chance. This finding highlights the need for further research of AI systems that can effectively conceptualize and reason about abstract graphical representations.
5.2. The Easier-Worse Anomaly
The Easier Worse Anomaly emerges in atomic-task assess- ments of intelligent agents where performance on simpler instances is unexpectedly inferior to performance on more complex variants of the same task. This pattern contradicts fundamental assumptions about cognitive capabilities.
The Visual Graph Arena serves as a valuable resource for the Al research community, providing a foundation for ad- vancing the field of visual reasoning on graph-structured data. By addressing the challenges posed by this dataset, researchers can work towards developing more robust and flexible Al systems that can bridge the gap between human- like conceptual understanding and reasoning and machine interpretation of abstract graphical information.
Consider finding the shortest path between two nodes in a graph. For genuine intelligence, performing considerably better finding shortest paths with or 4 length than finding shortest path between adjacent nodes would be anomalous. Such inverted performance within single atomic task can- not be reconciled with genuine understanding.
5.3. From Anomalies to Pseudo- Intellgnntel
Future Work. Expanding visual conceptualization bench- marks domains like chemical structures and logic circuits, where reasoning over structural invariants is critical, rep- resents a vital next step. While developing such datasets demands domain expertise, precise annotation, and rigorous validation to ensure fidelity, these efforts will deepen our understanding of Al's conceptualization capabilities. Such datasets will accelerate progress toward Al systems capable of human-like generalization across representations.
Our experiments with GPT-ol strongly demonstrate both anomalies. In Table 3, when tasked with finding shortest paths in graphs, the model achieves 29% accuracy for adja- cent nodes (path length I) compared to 70% accuracy for paths of length 2, a clear case of the Easier-Worse Anomaly.
The presence of both anomalies defies explanation in terms of low intelligence, but rather suggests the existence of pseudo-intelligence (likely result of probabive-robaabilstiic reasoning) form of capability that mimics intelligence in
8

---

Visual Graph Arena
Impact Statement
knowledge protocol with multimodal knowledge graph datasets for reasoning and vision-language pre-training. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. URL https: /pppreeiiwww net/ forum? id-LdRZ9SFBkU.
This paper presents work whose goal to advance the field of Machine Learning. There are many potential societal consequences of our work, none which feel must be specifically highlighted here.
Acknowledgments
Goyal, Y., Khot, T., Summers-Stay, D., Batra, D., and Parikh, D. Making the V in vqa matter: Elevating the role of image understanding in visual question answer- ing. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6904-69999999 6913, 2017.
Zahra Babaice and Radu Grosu are supported by the Aus- trian Science Fund (FWF) project MATTO-GBM 6605. Peyman M. Kiasari is supported by the TU Wien TrustACPS PhD School program that supported by TTTech Auto and B&C Privatstiftung.
Guetta, N. B., Slobodkin, A., Maimon, A., Habba, E., Rassin, R., Bitton, Y., Szpektor, I., Globerson, A., and Elovici, Y. Visual riddles: a commonsense and world knowledge challenge for large vision and language mod- els. In The Thirty-eight Conference on Neural Informa- tion Processing Systems Dalasets and Benchmarks Track, 2024. URL https: wpw.neeviwwwweeeeeeeepeeee..e.we .nt/ forum? Id=hwbRjjsRR5
References
Abboud, R., Ceylan, I. I., Grohe, M., and Lukasiewicz, T. The surprising power of graph neural networks with ran- dom node initialization. arXiv preprint arXiv:2010.01199, 2020.
Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L.. and Parikh, D. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pp. 2425 2223,223,,22 2015.
Gupta, and Kembhavi, A. Visual programming: Compo- sitional visual reasoning without training. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 14953-14962, 2023.
Balcilar, M., Héroux, P., Gauzere, B., Vasseur, P., Adam, S., and Honeine, P. Breaking the limits of message passing graph neural networks. In International Conference Machine Learning, pp. 599-608. PMLR, 2021.
Han, C., Mao, J., Gan, C., Tenenbaum, J., and Wu, J. Vi- sual concept-metaconcept learning. Advances Neural Information Processing Systems, 32, 2019.
Chen, Z. Chen, L., Villar, S., and Bruna, J. Can graph neural networks count substructures? Advances in neural information processing systems, 33:1038: 13:10333-10995 2020.
Haresh, S., Dijkman, D., Bhattacharyya, A., and Memisevic, Clevrskills: Compositional language and visual reason- ing in roboties. In The Thirty-eight Conference on Neu- ral Information Processing Systems Datasets and Bench- marks Truck, 2024. URL https:/ / openreview. net / Forummm idd 6 54s2tF20tss S0h6.
Cherian, A., Peng, K.-C,, Lohit, S., Matthiesen, J., Smith, K. A., and Tenenbaum, J. B. Evaluating large vision-and- language models on children's mathematical olympiads. In The Tiiht Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. URL https://////ttttttttttt www.r/veeee..eeeeeeeeeeee. wttpiiiwwww.eeeeeeeeeeeeeeeeeeee net forum? id id=YMAU2KJgZX.
Hu, W., Fey, M., Zitnik, M., Dong, Y., Ren, H., Liu, B., Catasta, M., and Leskovee, J. Open graph benchmark: Datasets for machine learning graphs. Advances in neural information processing systems, 33:221 33:22118-22133 2020.
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani. M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale, 2020.
Johnson, J., Hariharan, B., Van Der Maaten, L., Fei-Fei, L., Lawrence Zitnick, C., and Girshick, R. Clevr: A diagnostic dataset for compositional language and ele- mentary visual reasoning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2901 29000, 2017.
Girshick, R., Donahue, J., Darrell, T., and Malik, J. Region- based convolutional networks for accurate object detec- tion and segmentation. IEEE transactions pattern analysis and machine intelligence, 38(1:) 3821): 2015.
Kahou, S. E., Michalski, V., Atkinson, A., Kádár, Á., Trischler, A., and Bengio, Y. Figureqa: An anno- tated figure dataset for visual reasoning. arXiv preprint arXiv:1710.073000 2017.
Gong, B., Tan, S., Feng Y., Xie, X., Li, Y., Chen, C., Zheng, K., Shen, Y., and Zhao, D. UKnow: unified

---

Visual Graph Arena
Kamada, T., Kawai, S., et al. An algorithm for drawing general undirected graphs. Information processing letters, 31(1):7-15, 1989.
International Conference on Learning Representations, 2021. URL https::/p///.....t. httppp/////peeeeeeee...o net/ id=gVyew-grC2k.
For.rrr/iii.nnee/oommm forum?
Li, Y. Hu, B., Shi, H., Wang, W., Wang, L., and Zhang, M. Visiongraph: Leveraging large multimodal models for graph theory problems in visual context. arXiv preprint arXiv:2405.044 049500 2024.
Wang. K., Pan, J., Shi, W., Lu, Z.,, Ren, H., Zhou, A., Zhan, M., and Li, H. Measuring multimodal mathematical reasoning with MATH-vision dataset. In The Thirty-eight Conference Neural Information Processing Systems Datasets and Benchmarks Track, 2024. URL https: / //openreview net/ forum? Peerumiid--Nee ddQQTCcxMMPPA.
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B. Swin transformer: Hierarchical vision transformer using shifted windows. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 10012-10022. October 2021.
Wang, Y. and Zhang, M. Towards better evaluation of gnn expressiveness with bree dataset. arXiv preprint arXiv:2304.07702, 2023.
Liu, Z., Mao, H., Wu, C.-Y., Feichtenhofer, C.,, Darrell, T., and Xie, S. A convnet for the 2020s. Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Yang, Z., Gan, Z., Wang, J., Hu, X., Lu, Y., Liu, Z., and Wang, L. An empirical study of gpt-3 for few-shot knowledge- based vqa. In Proceedings of the AAAI Confer- ence on Artificial Intelligence, volume 36, pp. 3081 108999 2022.
Ma. Y., Zang, Y., Chen, L.. Chen, M. Jiao, Y., Li, X., Lu, X., Liu, Z., Ma, Y., Dong, X., Zhang. P., Pan, L., Jiang, Y-G., Wang, J., Cao, Y., and Sun, A. MMLONGBENCH-DOG Benchmarking long- context document understanding with visualizations. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. URL https:://///a/ct http:///ppeereeeeeeeemmmm hwtps/wwwwwweeeeeeeeeeeeeeeeeeee met/foount id=loJMlacwzf.
Morris, C., Kriege, M., Bause, F., Kersting, K., Mutzel, P. and Neumann, M. Tudataset: A collection of bench- mark datasets for learning with graphs. arXiv preprint nrXiv:2007.08663, 2020.
Murphy, R., Srinivasan, B., Rao, V., and Ribeiro, B. Re- lational pooling for graph representations. Interna- tional Conference on Machine Learning, pp. 4663 4673. PMLR, 2019.
Shao, H., Qian, S., Xiao, H., Song, G., Zong, Z., Wang, L.,, Liu, Y., and Li, H. Visual cot: Advancing multi-modal language models with comprehensive dataset and bench- mark for chain-of-the thouthtttt reasoning. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. URL https: //openreview.ro wet.ee.weeeeeeeeeeeee
forum? id htwp:/rrttooutimieoeeoo/o a0eiaiMeei..F.....
Sutskever, I.
Sequence to sequence learning with neural networks: What a decade. Test of Time Award talk at the 38th Conference on Neural Information Processing Systems (NeurIPS 2024), 2024. Available at https:
//www. www.yyyouubbeccom com hattom//wtcci wh 2v=H1Gi--4 4v1002.4.......
Tay, Y., Dehghani, M., Abnar, S., Shen, Y., Bahri, D., Pham, P., Rao, J., Yang. L., Ruder, S., and Metzler, D. Long range arena: benchmark for efficient transformers.
10

---

Visual Graph Arena
A. Dataset Description
The Visual Graph Arena (VGA) Benchmark is a collection of six datasets designed to evaluate and enhance the visual reasoning capabilities of Al models on graph-basee based tasks. The benchmark aims bridge the gap between human-like understanding and machine interpretation of abstract graphical information. The datasets are intended for research purposes in the field of artificial intelligence, specifically focusing on advancing visual reasoning on graph-structured data.
The benchmark datasets are hosted on the Visual Graph Arena wehsite at visualgrapharena.com. The website provides access to the datasets, along with documentation and usage guidelines.
Figure 8 illustrates the directory structure of the Visual Graph Arena (VGA) dataset. The VGA dataset is structured to facilitate easy access to various graph analysis tasks including isomorphism, path problems, and cycle detection. Each category is further subdivided into training and testing datasets with detailed sample counts and label files. For each input data, 'image_ pnng' corresponds to the i-th row of it's 'label.txt'. For more information and to access the dataset, please visit the VGA dataset website at visualgrapharena.com
Author Statement: We, the authors, take full responsibility for any potential violation of rights related to the Visual Graph Arena dataset and its materials. We confirm that the dataset is our original work and grant the hosting platform permission to distribute it under the CC BY 4.0 license. We will ensure proper access and maintenance of the dataset.
B. Dataset Creation
To create the datasets, we utilized the collection of connected graphs with 8 and 9 nodes from https: http:://userrs ceas. anu edu. au/-bdm/ data/ graphs...... seraphhshtml.. This collection comprises 1 11,117 connected graphs with 8 nodes and 261,080 connected graphs with 9 nodes. The Python library NetworkX was employed to visualize the graphs in plots for the datasets. Each sample was plotted within a square canvas of 700 x 700 pixels. The following details the specific steps involved creating each task:
B.1. Isomorphism Tasks
For the easy isomorphism tasks, we plotted two graphs in a single plot, with the graph on the right using the Kamada-Kave layout and the graph on the left using the planar layout. As not all graphs are planar, we first identified and selected all planar graphs from the set of connected graphs. We then split these planar graphs into separate test and train sets, ensuring that graph from the training appeared in the test set. The true and false samples were created as follows:
Easy Isomorphism:
True Labels: For each true label, we took a graph and plotted it using both the Kamada Kawai and planar layouts.
False Labels: For each false label, we plotted one graph using the Kamada- Kawai layout and then plotted another randomly selected graph from the set using the planar layout.
To maintain a balanced dataset, we ensured an equal number of true and false labels in both the training and test sets. Figure 9 presents examples of the easy isomorphism task with both true and false labels.
Hard Isomorphism: For this task, we proceeded similarly the previous one, but with a key difference in the selection of graphs for false labels. Instead of picking a random graph from the set, we searched for graphs within the set that are degree-equivalent the first graph. From these degree-cquivale planar layout. In cases where no engree-equivalent graphs were found for the first graph within the set, we discarded that sample. As a result, the number of samples in this task is smaller compared to the easy isomorphism task. Figure 10 presents examples of the hard isomorphism task with both true and false labels.
graphs, randomly chose one and plotted it using the
B.2. Path Finding Tasks
Shortest Path:
For the shortest path task, we created a balanced set of graphs with shortest paths of lengths 1, 2, 3, and 4 between two marked nodes. We opted not to include graphs with shortest paths of length 5 or longer, as the number of such graphs within the 8-9 node graph set was too small to maintain a balanced dataset with an equal number of samples for each path length. We began by separating the planar graphs for the test set. From the remaining non-planar graphs, which were

---

Visual Graph Arena
used for the training set, we identified pairs of nodes with shortest paths of lengths 1, 2. 3, and 4. These nodes were marked with squares in the plot, while the rest of the nodes were represented as circles. The training set graphs were plotted using the Kawai layout. For each path length, we plotted 20,000 samples, resulting in total of 80,000 samples for the training set. For the test set, we created similarly balanced set of graphs with different path lengths and plotted them using the planar layout. Figure 11 shows examples of the shortest path task.
For this task, we first identified the graphs that contain a Hamiltonian path from the entire set of 8-9 connected graphs. To create balanced dataset with an equal number of true and false labels, we then randomly selected the same number of graphs from the remaining non-Hamiltonian graphs as the number of Hamiltonian graphs found. This step was necessary since Hamiltonian graphs are fewer in count compared to non- oan-Hamilnnn graphs. After creating the balanced dataset, we split the set into train and test subsets. For the training set, we plotted the graphs using the Hamada-Kawel layout, while for the test set, we used the planar layout. Figure 12 shows examples of the Hamiltonian path task.
Hamiltonian Paths:
B.3. Cycle Finding Tasks
The task of finding Hamiltonian cycles was approached in similar manner to the Hamiltonian path task. We first identified graphs that contain Hamiltonian cycle from the entire set of 8-9 connected graphs. To create a balanced dataset with an equal number of true and false labels, we randomly selected the same number of graphs from the remaining non- non-Hamiltonian cycle graphs as the number of Hamiltonian cycle graphs found. After creating the balanced dataset, we split the set into train and test subsets. For the training set, we plotted the graphs using the Kamada- Kawai layout, while for the test set, we used the planar layout. Figure 13 presents examples of the Hamiltonian cycle task.
Hamiltonian Cycles:
Chordless Cycles: The chordless cycle task was similar to the shortest path task, but instead focused on the largest chordless cycle lengths of 3, 4, 5, and 6. We chose not to include graphs with no chordless cycles (trees) because the number of such graphs was too small to maintain balanced set.
We first separated the planar graphs for the test set. From the remaining non non-planar graphs, which were used for the training set, we identified the largest chordless cycles of lengths 3, 4, 5, and For each chordless cycle length, we plotted an equal number of samples, resulting in a balanced dataset. The training set graphs were plotted using the Kamada-Kawa layout.
For the test set, we created similarly balanced set of graphs with different chordless cycle lengths and plotted them using the planar layout. Figure 19 shows examples of the chordless cycles task.
C. Training the Vision Models
For training the vision models on the Visual Graph Arena benchmark tasks, we utilized three state-of-the-art architectures: ConvNeXt base, Swin Transformer base, and ViT (Vision Transformer) base. All models were initially pre-trained on the ImageNet dataset and then fine-tuned on our specific graph-based tasks. The training process was implemented using PyTorch, with the Adam optimizer and a learning rate of Le 4. The batch size was set to 32, and the models were trained for 10 epochs on each task. The LrossEntropyLoss was used as the loss function, as the tasks were formulated as either binary or multi-class classification problems. We used 2 NVIDIA TITAN RTX GPUs for training the models.
The input images were preprocessed using transformations from the PyTorch trans forms module. The images were resized to 384 384 pixels, and the pre-trained models were trained on the same size of images on ImageNet.
To handle the different output requirements of the tasks, the number of output classes in the final layer of the models was adjusted accordingly. For the isomorphism tasks and the Hamiltonian path/cycle tasks, the number of output classes was set to 2 (binary classification). For the shortest path and chordless cycle tasks, the number of output classes was set to 4, corresponding to the different path lengths or cycle sizes.
During training, the model's s performance was evaluated on the test set at the end of each epoch. The test set consisted of graphs drawn using different layouts than those used in the training set, allowing us to assess the model's ability to generalize to different visual representations. After the training process, the perfforern model on the test set was selected for each task and architecture combination. The code for training the vision models, written in Python using PyTorch, is provided in the supplementary materials and will be publicly available in GitHub repository.
12

---

Visual Graph Arena
train
140,000 samples
inputs
test
15,718 samples
easy
train
labels.txi
labels
test
labels.txt
isomorphism
train
127,374 samples
inputs test
14,298 samples
hard
train
labels.txt
labels
test
labels.txt
train
25,000 samples
inputs
kawai
2,480 samples
hamiltonian test
planar
2,480 samples
labels train
labels.txt
test
labels.txt
path
train
80,000 samples
Graph Arena
inputs
random
8,672 samples
test
kawai
8,672 samples
shortest
planar
8,672 samples
train
labels.txt
labels
test
labels.txt
train
69,935 samples
inputs
kawai
7,740 samples
hamiltonian test
planar
7,740 samples
labels train
labels.txt
test
labels.txt
cycle
train
80,000 samples
inputs
kawai
6,484 samples
test
cordless
planar
6,48 6,484 samples
train
labels.xxt
labels
labels.txt
Figure 8: Overview of the directory structure of the Visual Graph Arena benchmark.
13

---

Visual Graph Arena
(a) True
(b) False
Figure 9: Examples of the easy isomorphism task with true and false labels.
(a) True
(b) False
Figure 10: Examples of the easy isomorphism task with and false labels.
14

---

Visual Graph Arena
(a) The shortest path is of length 3.
The shortest path length
Figure 11: Examples of the shortest path task between marked nodes.
(a) True
(b) False
Figure 12: Examples of the Hamiltonian path task with true and false labels.
15

---

Visual Graph Arena
True
(b) False
Figure 13: Examples of the Hamiltonian cycle task with true and false labels.
(a) The biggest chordless cycle is of length
The biggest chordless cycle length
Figure 14: Examples of the biggest chordless cycle task.
16

---

Visual Graph Arena
D. MLLMs adjacency matrix creation
In this section, we evaluate performance of GPT-4o and ol, Claude Opus, and Claude 3.5 Sonnet in transcribing graph structures. We presented these models with three graph images and provided the prompt: "The image presented contains graph. Please transcribe this graph's adjacency matrix." We then analyzed their ability generate accurate adjacency matrices from the visual input. Our results reveal that all models struggle to correctly transcribe the graphs, with varying degrees of accuracy in extracting basic graph properties. GPT-4 comes close to accurately identifying the number of nodes but struggles with edge counts. Claude 3.5 Sonnet shows the best performance, closely approximating both the number of nodes and edges. In contrast, Claude 3 Opus performs poorly across all metrics. We visualized the graphs transcribed by these models to assess their accuracy. Although Claude 3.5 Sonnet was not available at the time of our initial submission, we have incorporated its responses into our analysis, because demonstrates superior performance compared to both GPT-4 and Claude Opus, showing a marked improvement over Opus in particular.
prompt: The image presented contains a graph. Please transcribe this graph's 's adjacency matrix.
(a) Graph
(b) Graph
(c) Graph
Figure 15: The three graphs we asked the LLMs to generate the adjacency matrix.
(a) Graph
Graph
(c) Graph c
Figure 16: reconstructed graphs from the generated adjacency matrix by GPT-4o.
E. LLMs Prompts and Responses Samples
In this section, we present one example of each task, along with the prompts used and the responses from GPT-o1, GPT-4, Claude Opus, and Claude 3.5 Sonnet. We highlight correct final answers in green and incorrect ones in red. The results demonstrate that these models can analyze graphs, provide chains of thought, and attempt track nodes and edges. However,
17

---

Visual Graph Arena
(a) Graph
(b) Graph
(c) Graph c
Figure 17: reconstructed graphs from the generated adjacency matrix by Claude 3.5 Sonnet.
(a) Graph
(b) Graph
(c) Graph
Figure 18: reconstructed graphs from the generated adjacency matrix by Claude Opus.
they often fail to solve the problems accurately. In the example in three cases, while the final yes/no responses are correct, closer inspection the analysis reveals obvious mistakes, such as incorrectly counting the number of nodes. The correct responses are not surprising given the yes/no nature of some tasks. Notably, in the shortest-path task, all models unexpectedly fail to detect that the two marked nodes are connected by single edge.
E.1. Isomorphism Easy
prompt: The image presented contains two side-by-side graphs. Without using code, please answer whether the two given graphs are isomorphic. Please report your final answer as 'Yes" or 'No" as your very last word.
18

---

Visual Graph Arena
(a) Graph
Graph
(c) Graph c
Figure 19: reconstructed graphs from the generated adjacency matrix by GPT ol.
GPT-4o Output:
To determine if the two graphs are isomorphic, we need to check whether there is a one-to-one correspondence between the vertices of the two graphs that preserves the adjacency relationships. Looking the graphs in the image:
1. Number of vertices: Both graphs have the same number of vertices.
2. Number of edges: Both graphs also have the same number of edges.
3. Vertex degrees: The degrees of corresponding vertices (number of edges connected to each vertex) should be the same.
However, upon visual inspection, the structure of the graphs seems to differ significantly. The graph on right has more symmetrical structure, while the graph on the left appears more irregular. Specifically, the arrangement of connections and the overall shape of the graphs suggest that their connectivity patterns differ. This implies that there isn't one-to-one correspondence that preserves the adjacency relationships between the vertices of the two graphs. Therefore, the final answer is No.
19

---

Visual Graph Arena
Figure 20: The input image, the answer is 'Yes"
Claude 3.5 Sonnet Output:
To determine if these graphs are isomorphic, I'll analyze their structures: Left graph:
9 vertices
10 edges
2, Degree sequence: 3, 2, 2, 2,
Right graph:
vertices
10 edges
Degree sequence: 4, 2, 2,
While both graphs have the same number of vertices and edges, their degree sequences are different. The left graph has a maximum degree 3, while the right graph has a vertex with degree 4.
Isomorphic graphs must have identical degree sequences. Since these graphs have different degree sequences, they cannot isomorphic.
No
20

---

Visual Graph Arena
Claude 3 Opus Output:
To determine if the two graphs are isomorphic, I'll check if there exists bijective mapping between their vertex sets that preserves adjacency.
Both graphs have vertices. Let's label the vertices of the left graph L1 to L5 from top bottom, and the vertices of the right graph RI to R5 from top to bottom.
Comparing vertex degrees:
LI, L2, L4, L5 have degree 3. L3 has degree 4.
RI, R3, R5 have degree 3. R2, R4 have degree
Since the degree sequences don't 't match, there cannot an adjjacncy--preerviig bijection between the vertex sets. Therefore, the two graphs No
isomorphic.
GPT-ol Output:
Thought about isomorphic graphs for 3m 21s
A quick way (the topmost point) clearly has degree 5 (it has edges to five other vertices). In the left-hand graph, no vertex has that many incident edges (each has degree at most 3 or 4). Since an isomorphism would require a one-to-one match of vertices with the same degree, the two graphs cannot be isomorphic. No
they cannot match is by comparing vertex- degree. In the right-hand graph, the "apex" vertex
E.2. Isomorphism Hard
prompt: The image presented contains two side-by-side graphs. Without using code, please answer whether the two given graphs are isomorphic. Please report your final answer as 'Yes' or 'No' your very last word.
21

---

Visual Graph Arena
MA
Figure 21: The input image, the answer 'No'
GPT-4o Output:
To determine if the two graphs are isomorphic, I will analyze the structural properties of each graph, such as the number of vertices, edges, and the degree of each vertex, and compare them. Step 1: Count the vertices and edges.
Lefft graph:
Number of vertices: 10
Number of edges: 13
Right graph:
Number of vertices: 10
Number of edges: 13
Step 2:
Compare the degree of each vertex.
Both graphs should have the same degree sequence if they are isomorphic. The degree of each vertex is the number of edges connected to it.
Step 3: Visual inspection.
Compare the overall structure and connectivity to see if a one-to-one correspondence exists between the vertices and edges of the two graphs.
Upon close inspection, can conclude that both graphs have the same number of vertices and edges, and they appear to have similar degree sequence and connectivity structure. Given these observations, the graphs seem to be isomorphic.
Final answer: Yes
22

---

Visual Graph Arena
Claude 3,5 Sonnet Output:
To determine the two graphs are isomorphic, analyze their structures: Left graph:
11 vertices
15 edges
Complex structure with multiple triangles and intersecting lines
Right graph:
vertices
15 edges
More elongated structure, fewer triangles, more "tree-like" in parts
While both graphs have the same number of vertices and edges, their structures appear quite different:
1. The left graph is more densely intercomnected, with multiple triangles and a more compact shape.
2. The right graph has more stretched out, less symmetrical structure.
3. The degree distribution (number of edges connected to each vertex) seems different between the two graphs.
4. The left graph appears to have more cycles than the right graph.
These structural differences suggest that there no one-to-one mapping that could transform one graph into the other while preserving connections. Given these observations, my conclusion is that these graphs are isomorphic.
23

---

Visual Graph Arena
Claude 3 Opus Output:
Okay, let's carefully analyze the two graphs to determine they are isomorphic. isomorphic, the two graphs must have:
1. The same number of vertices
2. The same number of edges
3. Equivalent connectivity between corresponding pairs vertices
Comparing the two given graphs:
Number of vertices:
Both graphs appear to have vertices each. So this condition is satisfied.
2. Number of edges: count matches.
count 9 edges in the graph the left, and 9 edges in the graph the right as well. Edge
3. Vertex connectivity: vertices of the two graphs, such that the edge connections preserved.
This is trickier to verify visually. We'd need to find one-to-one mapping between the
One potential mapping: Left graph vertex (from top) + 1, 2 + 3 - 6, 4 5,5 + 4, + 3 on right graph This mapping appears preserve all the edge connections between corresponding vertex pairs. Since there exists valid mapping, and the graphs have equal number of vertices and edges, the graphs isomorphic. Therefore, my final answer is Yes
GPT-ol Output:
Thought about graph isomorphism for 3m 45s A quick way to see they differ is by looking at how each drawing "uses" its outer boundary. In the right-hand graph, all vertices lie on or inside single large triangular boundary; in the left-hand graph, the outer boundary clearly has more than three vertices. That alone forces a mismatch in adjacency that no relabeling can fix.
Hence, the two graphs are not isomorphic. No
E.3. Cycle Hamiltonian
prompt:
The image presented contains a graph. Without using code, please answer whether the graph has Hamiltonian cycle. Please report your final answer as 'Yes" or 'No' as your very last word.
24

---

Visual Graph Arena
A
Figure 22: The input image, the answer is 'Yes'
GPT-40 Output:
To determine whether the graph has a Hamiltonian cycle (a cycle that visits every vertex exactly once and returns to starting vertex), we need to consider the structure of the graph and see such cycle can traced. Upon inspecting the graph visually, possible way visit every vertex exactly once and return to the starting point without either missing vertices or retracing edges.
clear that the graph does not include Hamiltonian cycle because there is
The final answer is No.
25

---

Visual Graph Arena
Claude 3,5 Sonnet Output:
To determine if this graph has a Hamiltonian cycle, we'll examine if there's path that visits each vertex exactly once and returns to the starting vertex. The graph has 9 vertices connected by lines.
A Hamiltonian cycle would need to:
Visit all vertices
2. Use only existing edges
3. Return to the starting point
4. Visit each vertex only once (except the start/end)
Looking the graph structure:
There are several vertices with only two connections, which limits path options.
The graph is not fully connected it has "pinched" middle section.
There's s no obvious circular path that covers all vertices.
Attempting to trace potential paths:
Starting from any outer vertex, it's impossible to reach all other vertices and return without revisiting some.
The central "pinched" area forces any potential cycle to cross over itself, which isn't allowed in Hamiltonian cycle.
Given these observations, it's not possible create path that meets all the criteria for Hamiltonian cycle in this graph.
No
Claude Opus Output:
To determine if the given graph has Hamiltonian cycle, need to check if there exists cycle that visits each vertex exactly once and returns to the starting vertex. Looking at the graph, can observe the following:
1. The graph has 6 vertices.
2. Each vertex has degree (number of edges connected to it) of at least 2.
3. It appears that we can trace path that visits each vertex exactly once and returns the starting point.
One possible Hamiltonian cycle in this graph could be:
Start at any vertex, move to an adjacent vertex, then continue moving to unvisited adjacent vertices until all vertices have been visited once, and finally return the starting vertex.
Therefore, based on the observation that cycle visiting each vertex exactly once exists in the given graph, my final answer Yes
26

---

Visual Graph Arena
GPT-ol Output:
Thought about Hamiltonian cycle in graph for 3m 47s
From purely visual/graph-thhereei inspection (tracking each vertex and checking that we can "thread" single eycle through all of them exactly once), finds that there is indeed way visit all vertices in cycle. In other words, the graph is Hamiltonian. Yes
E.4. Cycle Biggest Chordless
prompt: The image presented contains graph. Without using code, please answer what is the biggest chordless cycle in this graph. Please report your final answer number in your very last word.
Figure 23; The input image, the answer '3'
27

---

Visual Graph Arena
GPTT-4o Output:
To find the biggest chordless cycle the given graph, let's analyze connections between the vertices step by step. A chordless cycle, also known as an "induced cycle," a cycle graph that does not have any chords, where chord is edge that is not part of the cycle but connects two vertices of the cycle.
Steps to Analyze the Graph:
1. Identify cycles: We need to identify all the possible cycles in the graph.
2. Determine if they are chordless: For each eycle, check if there are any extra edges (chords) connecting non-consecutive vertices in the cycle.
Observation from the Image: Looking the graph, few potential cycles stand out:
Cycle 1: Going through five points that seem to form pentagon shape without any internal edges connecting non-adjacent vertices.
Cycle 2: A cycle involving four vertices.
Upon inspecting the graph:
The cycle involving the five vertices appears to be the largest cycle.
2. There are no additional edges (chords) connecting non-consecutive vertices in this eycle, making it chordless cycle.
Conclusion:
The biggest chordless cycle in this graph consists of vertices.
Thus, the final answer is
Claude 3.5 Sonnet Output:
To find the biggest chordless cycle in this graph, I'll examine the structure carefully:
1. First, I'll identify all cycles in the graph.
2. Then, I'll check each cycle if it has any chords (edges connecting non-adjacent vertices within the cycle).
3. Among the chordless cycles, I'll determine the largest one.
After analyzing the graph:
There are several triangles (3-cycles), but these are too small to be the answer.
can see a few 4-cycles, but upon closer inspection, they all have chords.
The largest cycle can find without chords is 5-cycle.
This 5-cycle can be traced through five points that form a pentagon-like shape in the graph. Any larger cycles try to form end up having chords connecting vertices within the cycle. Therefore, the biggest chordless cycle in this graph contains vertices.
To express my final answer as requested: 5
28

---

Visual Graph Arena
Claude Opus Output:
To find the largest chordless cycle in the given graph, I'll analyze the graph visually. The graph consists of 6 vertices connected by edges in a triangular pattern. can see several cycles:
The outermost cycle connecting all 6 vertices
Two smaller cycles, each connecting 4 vertices, sharing edge of the outer cycle
Four cycles connecting vertices each (the smallest cycles possible)
The 3-vertex cycles all have chords (edges connecting non-adjacent vertices in the cycle), they are not chordless. The 4-vertex cycles don't 't have any chords. These are the largest chordless cycles in the graph. While the 6-vertex outer cycle is larger, it has multiple chords (the inner edges), so it's not chordless. Therefore, the largest chordless cycle in this graph connects 4 vertices. 4
GPT-o1 Output:
Thought about chordless cycle size for 2m 30s
A succinct way to spot the largest chord-free ("induced") cycle is to look around the "outer rim" of the drawing and check whether any diagonals cut across those vertices. By inspection, you will see that all 6 vertices forming the perimeter loop in the lower and middle/top portions can be traversed in cycle without encountering an internal chord. Any attempt include the interior edges or to form 7-cycle will run into an extra cross-edge (a chord). Hence the longest chordless cycle in this graph has length 6
29

---

Visual Graph Arena
E.5. Path Hamiltonian
prompt:
The image presented contains graph. Without using code, please answer whether the graph has Hamiltonian path. Please report your final answer as 'Yes' or "No" as your very last word.
Figure 24: The input image, the answer is 'Yes'
GPT-4o Output:
To determine if the given graph has Hamiltonian path, will visually inspect the graph. A Hamiltonian path is path in a graph that visits each vertex exactly once.
Upon inspecting the graph:
1. I identified the vertices and observed the connections between them.
2. checked if there's a path that visits every vertex exactly once.
Given the structure of the graph, it appears that there is such path that visits every vertex exactly without retracing any edge missing any vertex.
Final answer; No
30

---

Visual Graph Arena
Claude 3,5 Sonnet Output:
To determine if this graph has a Hamiltonian path, we'll examine whether there's path that visits each vertex exactly once.
The graph has vertices connected by edges forming network of triangles and lines.
Starting from any vertex, it's possible to traverse the graph visiting each vertex once:
1. Begin at the leftmost point
2. Move right the next point
3. Go up to the top of the leftmost triangle
4. Move right to the center point of the graph
5. Go down to the bottom point of the rightmost triangle
6. Move up to the right point of that triangle
7. Go to the top point of the rightmost triangle
8. End at the uppermost point of the graph
This path visits vertices exactly once, without repeating any vertex. Therefore, the graph does have Hamiltonian path.
Yes
Claude 3 Opus Output:
To determine whether the given graph has a Hamiltonian path, I'll analyze its structure and properties. The graph appears have vertices, connected by 8 edges in a triangular mesh pattern. Every vertex has a degree (number of edges connected to it) of either 2 or 4.
For graph have Hamiltonian path, must be possible to visit every vertex exactly once in continuous path. One necessary condition is that the graph must be connected, meaning there no isolated vertices separate components. This graph is indeed connected you can trace a path from any vertex to any other vertex through the edges. This satisfies the connectivity requirement.
However, the graph's s structure poses a challenge. The triangular arrangement means that certain vertices, especially the central ones with degree 4, will likely need to be visited more than once to cover all vertices in single path. After mentally tracing potential paths, haven't found a successful Hamiltonian path that visits each vertex exactly once. The graph's layout seems necessitate revisiting some vertices. Therefore, based on this analysis, my conclusion is that this particular graph does not have Hamiltonian path. No.
GPT-ol Output:
Thought about Hamiltonian path in graph for 2m 49s
From inspection, one can find a route that visits each vertex exactly once (in particular, the "lone" degree-1 vertex at the top can serve as one end of such path). Therefore the graph does indeed admit Hamiltonian path.
Yes
31

---

Visual Graph Arena
E.6. Path Shortest
The image presented contains graph. Two nodes marked by square. Without using code, please answer what is the shortest path between the two square nodes in this graph. Please report your final answer number in your very last word.
prompt:
Figure 25: The input image, the answer '1'
GPT-4o Output:
To find the shortest path between the two square nodes in the graph, we can visually inspect the connections between the nodes.
Looking at the image:
Start from the square node on the left side of the graph.
2. There is a direct connection to the node above it.
3. From this node, there a connection the square node on the right side.
Therefore, the shortest path consists of two edges.
The final answer is
32

---

Visual Graph Arena
Claude 3,5 Sonnet Output:
To find the shortest path between two square nodes in this graph, I'll count minimum number of edges that need to be traversed get from one square node to the other. Looking at the graph, I can see that there are two potential paths:
A direct path that goes through the center node the graph.
2. A path that follows the outer edges of the graph.
The direct path through the center is clearly shorter. It requires traversing only 2 edges:
1. From the first square node to the center node
2. From the center node the second square node
The path along the outer edges would require traversing at least edges. Therefore, the shortest path between the two square nodes in this graph is
Claude 3 Opus Output:
The image shows graph with 6 nodes, where nodes marked by squares. To find the shortest path between the two square nodes:
1. Starting from the top square node, there are two possible paths: down-left or down-right.
2. Following the down-left path:
Square Circle - Square
This path has length of edges.
3. Following the down-right path:
Square - Circle - Circle > Circle >> Square
This path has length of 4 edges.
Therefore, the shortest path between the two square nodes goes through the down-left path, with length of 2 edges. The shortest path between the two square nodes is
GPT-ol Output:
Thought about shortest path in graph for 3m 21s
From inspection of the diagram's 's edges, there is direct edge connecting the two square square-marke marked nodes; however, they lie exactly two edges apart (via single intermediate vertex).
33


Reviews:
[{"review_id": "EWcZw3Y3zJ", "reviewer": "Reviewer_qStX", "summary": "This paper presents visual graph arena (VGA), a multimodal dataset designed to evaluate and improve AI systems’ capacity for visual abstraction.\n\nAlthough being straightforward for humans, the authors find that VGA is very challenging for current MLLMs: they totally fail on some of the tasks in VGA and show limited success on other tasks.\n\nBy deeper analyais, the authors find the behavioral patterns of current MLLMs, especially o1, are significantly different with humans'. Such finding brings an open question about whether the current models are able to conceptualize as humans.\n\n## update after rebuttal\n\nThe distinction with previous benchmarks generally make sense to me. \nThe analysis provided in this paper is helpful for the community to understand MLLMs' ability in visual conceptualization and abstract reasoning.\nTherefore I will keep my rating to 3.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your positive assessment of our work and for helpful comments. Below, we address your questions and suggestions.\n\n**Distinction from Related Work**\nThe reviewer asked how VGA differs from the cited papers on visual reasoning. While all these works explore aspects of visual abstraction and reasoning, VGA offers several important distinctions:\n\n- VGA uniquely tests whether models can recognize the same graph concept across different visual layouts. By training on one layout (e.g., Kamada-Kawai) and testing on another (e.g., planar), we isolate the ability to abstract away from visual form to understand invariant properties.\n- Graph-theoretical foundation: Unlike more general visual reasoning datasets, VGA's tasks are grounded in well-defined graph theory problems with deterministic answers. This provides clear evaluation criteria *without subjective interpretation.*\n\n- With 27,000-150,000 training samples per task, VGA is substantially large, while maintaining careful separation between train and test sets with no graph repetition.\n\n**Other comments or suggestions**\n\nWe appreciate the reviewer's hypothesis about the Middle-Score and Easier-Worse anomalies. We're interested in exploring controlled experiments that could isolate memorization effects from genuine conceptualization, and agree that evaluating the memorization effect is a promising direction for future work. We used different layouts for training and testing sets to avoid memorization when trained on our dataset.\n\nTo test on MLLMs, one could design experiments with systematically varied graph structures that share superficial patterns but require different abstract reasoning. We would appreciate it very much if the reviewer has experiment suggestions in this direction for future work.", "claims_and_evidence": "The problem and evaluation scopes are well defined and motivated, and the experiments are well designed.\nClaims made by this paper are convincingly supported by the experimental evidence.", "methods_and_evaluation_criteria": "Yes, the choice of the evaluation design is well motivated and is suitable for evaluating MLLMs conceptualize ability.", "theoretical_claims": "There is no complex proofs for theoretical claim that need to be checked in this paper, it is an empirical study and all proposed methods are justified by experiments.", "experimental_designs_or_analyses": "Yes, please refer to paper summary.", "relation_to_broader_scientific_literature": "The scope of this paper is revelant to visual abstract reasoning.\nCompared with piror works, this work clearly connects its evaluation with the conceptualize ability of MLLMs, and make detailed analysis about current MLLMs behaviors.", "essential_references_not_discussed": "Since this paper is studying the conceptualization/congnitive behavior of MLLMs/vision models, the below list of papers explored and discussed similar topics and should be included in the discussion in this paper. \n\n[1] Jiang, Y., Sun, K., Sourati, Z., Ahrabian, K., Ma, K., Ilievski, F., & Pujara, J. (2024). Marvel: Multidimensional abstraction and reasoning through visual evaluation and learning. Advances in Neural Information Processing Systems, 37, 46567-46592.\n\n[2] Zhang, C., Gao, F., Jia, B., Zhu, Y., & Zhu, S. C. (2019). Raven: A dataset for relational and analogical visual reasoning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 5317-5327).\n\n[3] Campbell, D., Kumar, S., Giallanza, T., Griffiths, T. L., & Cohen, J. D. (2024). Human-like geometric abstraction in large pre-trained neural networks. arXiv preprint arXiv:2402.04203.\n\n[4] https://github.com/fchollet/ARC-AGI", "other_strengths_and_weaknesses": "No", "other_comments_or_suggestions": "My hypothesis on the Middle-Score anomaly and the easier-worse anomaly is that the models learns to memorize some specific patterns during training. (similar with 5.3)\nSo it would be good if there are some experiments that can fully isolate the memorization problem of the MLLMs.\nThis is a suggestion for the future extension not for the current work, so the authors don't have to add any additional experiments in response to this.", "questions_for_authors": "I would like to hear how VGA is different from the papers I mentioned above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ROUFJxceDa", "reviewer": "Reviewer_qrzb", "summary": "This paper investigates multimodal models' ‘conceptualization’—the ability to recognize and reason about the same concept despite variations in visual form, a basic ability of human reasoning. They introduce the Visual Graph Arena (VGA), a dataset featuring\nsix graph-based tasks designed to evaluate and improve AI systems’ capacity for visual abstraction. VGA uses graphs rendered in diverse layouts to test reasoning independent of visual form. Experiments with state-of-the-art vision models (ViT, Swin Transformers, ConvNeXt) and multimodal LLMs (GPTo1, Claude 3.5 Sonnet) reveal a striking divide: human participants achieved near-perfect accuracy (88–100%) across tasks, while models totally failed on isomorphism detection and showed limited success in path/cycle tasks.\n\n## update after rebuttal\nI appreciate authors' replies and the additional experiments. This paper is focused on evaluation and analysis, and my concerns about limited baseline mllm experiments still remains, which I think is very important to this evaluation paper, just to show people how different models behave. Therefore I'd remain my original score. But if this concern was addressed I would raise to 3.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your thoughtful and detailed feedback. We appreciate the positive reception to the core idea of VGA. Below, we address each of the points raised:\n\n**3D format**\n\n You raise an interesting point about testing graphs in 3D format. Our current focus was on testing visual conceptualization across different 2D layouts (Kamada-Kawai, planar, random) as these represent common ways graphs are visualized in practice. Testing 3D representations would be a valuable extension; however, it would present significant technical challenges. Current top multimodal LLMs like ChatGPT and Claude only support 2D image inputs, with no native capability to process or reason about true 3D representations. This limitation would make direct testing of 3D graph comprehension infeasible.\n\n**More Baselines**\n\nWe acknowledge your suggestion to include more baseline models. We agree this would strengthen our analysis and have already begun running these additional experiments. Our GPUs are currently processing. Given that our framework requires six distinct tasks per model and considering our computational constraints during the ICML rebuttal period, these experiments will take some time to complete. We will update you with the results during the discussion period as they become available. Thank you for valuable recommendation.\n\n**Table 2 Clarification**\n\nThank you for pointing this out, You're absolutely right that this needs clarification. In our study, all vision models (ViT, Swin, ConvNeXt) were fine-tuned on VGA's training sets, while the multimodal LLMs were evaluated in a zero-shot setting without fine-tuning. We will make this distinction explicit in the revised table, caption, and methodology section.\n\n**Sample Size for LLM testing**\n\nWe appreciate your concern about the sample size for LLM testing. Our 20-sample approach was sufficient to detect random-level performance in most cases; we expanded to 100 samples only where GPT-o1 showed promise, to obtain better precision. We adopted the \"FAIL\" designation from Tay et al. (2021) when performance was not better than random guessing, as precise values near the random baseline offer minimal scientific insight. It's worth noting that querying these models is not free—accessing models like GPT-4o, Claude, and Gemini involves substantial API costs. Testing across different layouts means each model required ~160 experiments, and we’ve conducted nearly 900-1000 evaluations including unreported models (Claude Opus, Gemini 1) and we have to update the existing models to Claude 3.7 and O3-mini. Expanding to 50+ samples would significantly increase costs with low returns in clearly failing conditions. However, if the reviewer believes that larger sample sizes are essential for the scientific contribution of this paper, we would absolutely prioritize and carry out additional testing accordingly.\n\n**Effect on model performance on natural images**\n\nThis is an insightful question. Since we start with pretrained models and finetune them on VGA tasks—replacing the classification layer with a binary one—it inevitably alters the underlying weights. As such, it's unclear how to meaningfully assess \"performance on natural images\" post-finetuning. In practice, a multitask setup would likely be needed to preserve general visual capabilities. If the reviewer has thoughts on how to evaluate this more effectively, we’d be very interested to hear them.\n\nWe greatly appreciate your feedback, which will help us strengthen the paper. We believe addressing these points will improve the work and hope you'll reconsider your assessment based on our planned revisions.", "claims_and_evidence": "I generally like the idea of VGA. It is interesting to see how multimodal models learn graphs. But I wonder if these graphs should be given in 3D format instead of in 2D, e.g. as according to Figure 2?", "methods_and_evaluation_criteria": "More baseline multimodal language models should be included, especially the ones designed for perceptual tasks. E.g. phi4 vision, llava-video, Gemini 2.0 Flash, MetaMorph etc. Also, testing on 20 examples seems a bit less convincing because the number is small and result number is replaced by FAIL. \nShould also add more vision model baselines, e.g. Dino, Siglip, clip, etc.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "See above. Also in Table 2, it's a bit confusing which models are finetuned on the training set and which are not. 20 testing number is not very convincing.", "relation_to_broader_scientific_literature": "It's related to testing Graphs understanding abilities of Multimodal language models.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Experiments are good. Since this dataset is synthetic by nature, I wonder if training on it would hurt a bit model performance on natural images?", "other_comments_or_suggestions": "See above.", "questions_for_authors": "See above. Happy to raise score if questions are addressed.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IxgdFV0FGO", "reviewer": "Reviewer_VePQ", "summary": "The paper proposes a new benchmark to evaluate whether visual models can understand underlying concepts with different visual appearances. The benchmark contains generated graphs with three layouts: Random, Kamada-Kawai, and Planar. The authors then train the visual model on trained on the Kawai layout, then ask them to determine whether two graphs are isomorphic, find the required path, and circle.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your thoughtful feedback and positive evaluation of our paper. We're happy that you found our benchmark sound and valuable for evaluating visual models' ability to understand concepts across different visual representations. Below we try to address your concerns.\n\n**Additional datasets to discuss**\n\nRegarding the datasets you mentioned (Geometry problems, Table QA, and Science Figure QA), we appreciate the recommendation. We have discussed and cited FigureQA in our related work section, ~L160 (Kahou et al., 2017). We will expand our discussion to include the other suggested datasets and clarify the key differences between our approach and these works. Our benchmark uniquely focuses on isolating the specific challenge of conceptualization across visual representations using graph-theoretic concepts as the test case.\n\n**Graphs described in natural language**\n\nYour question about how GPT-o1 would perform when given graphs described in natural language is insightful. While this is an interesting direction, our focus was specifically on the visual format. Graphs visually represent complex data and relationships in ways that are intuitive for humans to understand without textual descriptions. The visual aspect is critical to our research question about conceptualization across different visual layouts.\nThere is indeed existing work on textual graph representation for language models, such as Fatemi et al.'s \"Talk Like a Graph: Encoding Graphs for Large Language Models,\" ICLR2024, which explores different textual encoding strategies for graph reasoning tasks. However, our work deliberately targets the visual domain to evaluate a different capability - the ability to recognize the same underlying concept despite variations in visual form, which is fundamental to human visual reasoning. We hope to revise the paper to include this refrenece and clarify the focus.", "claims_and_evidence": "The paper claims the benchmark can evaluate visual models' ability of visual abstraction. The benchmark can mostly support the claim.\n- A model needs to understand the concept and logic behind the graph image to predict the correct answer.\n- The concepts are mostly related to logic and graphs instead of the image. For example, one can perform a similar evaluation by describing the graph in natural language and without images. The visual concepts are more related to color, material, and object relations in a natural image, whose details are hard to describe all within limited text.", "methods_and_evaluation_criteria": "The proposed benchmark evaluates models based on yes/no classification and multiple-choice accuracy. Additionally, all training and test samples have the same candidate choices. Thus, the evaluation criteria can correctly reflect the models' ability.", "theoretical_claims": "No theoretical claims or proofs.", "experimental_designs_or_analyses": "The experimental designs are sound. The paper evaluates several closed-source MLLMs, as well as open-source models that have been fine-tuned on the large provided training set. The results can show the new difficulties for current vision models.", "relation_to_broader_scientific_literature": "The proposed dataset can inspire future work on extracting logical information represented in different visual forms.", "essential_references_not_discussed": "The paper only cites graph datasets but neglects datasets such as Geometry problems, Table QA, and Science Figure QA that also require vision models to perceive underlying information. Authors may include some of these works and explain the differences.", "other_strengths_and_weaknesses": "The proposed dataset contains large numbers of training samples, which can evaluate the effectiveness of a training method for disentangling visual and logical information.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "How will GPT-o1 perform when given the graphs described in natural language?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KAGUFO21rN", "reviewer": "Reviewer_8ruU", "summary": "This paper introduced a new dataset, named **V**isual **G**raph **A**rena (**VGA**), specifically designed to evaluate and enhance deep models for visual graph analysis. VGA comprises six distinct tasks: Easy Isomorphism, Hard Isomorphism, Hamiltonian Path, Shortest Path, Hamiltonian Cycle, and Biggest Chordless Cycle. With its diverse visual layouts, VGA effectively challenges models to recognize invariant conceptual properties despite variations in graphical layouts.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for thoughtful comments and the time you spent for the detailed review. We hope to address your concerns below:\n\n**More Baselines**\n\nWe acknowledge your suggestion to include more baseline models. We agree this would strengthen our analysis and have already begun running these additional experiments. Our GPUs are currently processing. Given that our framework requires six distinct tasks per model and considering our computational constraints during the ICML rebuttal period, these experiments will take some time to complete. We will update you with the results during the discussion period as they become available. Thank you for valuable recommendation.\n\n**Real world domains:**\n\nRegarding our emphasizing of the significance of varying visual representations of identical concepts across different real-world domains, we would like to be more clear: The ability to understand concepts is a fundamental capability expected from an \"intelligent\" agent. And this makes investigating the capability of our models in this regard significant. The simplest and dataset-makeable instance of conceptualization that we could think of was graphs. Our work deliberately focuses on graphs as a clear, well-defined example of such visual representation challenges. Upon testing the existing models on these basic graph tasks, especially isomorphism, we saw them significantly struggling.\nJust to be more clear, we do not in any shape claim that our dataset directly enhances model conceptualization in other scientific domains like chemistry. We believe that these, like simple graphs, are other instances of conceptualization, but more complex. We hope this work will inspire future research to introduce diverse real-world representation datasets for conceptualization.\n\n**Regarding similarities with [1]:**\n\nWe thank the reviewer for bringing this work to our attention. We missed this reference as our creation of VGA took several months, and at the time we began, [1] was not yet published. We will certainly add a comprehensive discussion of it to our final paper.\n\nOur work is fundamentally distinct from [1] in several important ways:\n1) **Representation approach**: [1] inputs textual data of graphs node and edge lists in addition to images. In contrast, our work focuses exclusively on visual conceptualization without additional text information. More importantly, the edge list that they are giving as text information, is layout independent, and this is in striking contrast with the whole idea of our paper. \n\n\n2) **Purpose of layout variation**: We deliberately employ different layouts in training vs testing (and no two graphs being the same) to assess models' conceptualization abilities and to prevent pattern memorization between training and testing. Conversely, [1] uses different layouts as an **augmentation technique to expand training data**, while also providing layout-invariant textual representations. This is strongly the opposite of what we intend to do.\n\n\n3) **Human interpretability**: The essences of the plotted data and the way they are plotted are far apart. We carefully selected layouts and representations to ensure they remain fully understandable by human subjects, as confirmed by our human evaluation studies. This was critical since our approach relies solely on visual information. By contrast, [1] employs layouts and graphs that are often visually complex and challenging even for humans to interpret without supplementary textual information. For reference, examine this example of graph 100 in different layouts from [1]:\n- [Circo layout](https://huggingface.co/datasets/Yanbin99/GVLQA-AUGLY/blob/main/data/connectivity/graph_visual/layout/circo/hard/graph100.png)\n- [Dot layout](https://huggingface.co/datasets/Yanbin99/GVLQA-AUGLY/blob/main/data/connectivity/graph_visual/layout/dot/hard/graph100.png)\n- [FDP layout](https://huggingface.co/datasets/Yanbin99/GVLQA-AUGLY/blob/main/data/connectivity/graph_visual/layout/fdp/hard/graph100.png)\n- [Neato layout](https://huggingface.co/datasets/Yanbin99/GVLQA-AUGLY/blob/main/data/connectivity/graph_visual/layout/neato/hard/graph100.png)\n- [SFDP layout](https://huggingface.co/datasets/Yanbin99/GVLQA-AUGLY/blob/main/data/connectivity/graph_visual/layout/sfdp/hard/graph100.png)\n\n- ...\n\nReagding [2]: we have already cited it in L132. We will include [1] and [3] in our final version.\n\n**Question**\nRegarding layout selection, we carefully chose the current layouts from standard graph representation methods that remain human-understandable. For instance, we deliberately avoided circular layouts since they become visually challenging even for humans as the number of nodes increases. We are currently working on a carefully hand-crafted test set with special graph representations to publish on the dataset website in the future. However, since these are manually created, it takes time to develop a collection that is both rich enough and of high quality.", "claims_and_evidence": "Empirical evidence demonstrates that models consistently struggle with path and cycle tasks across diverse layouts, whereas human participants achieve near-perfect accuracy. This supports the claim that current multi-modal large language models face significant challenges in recognizing and reasoning about the same concept when presented in varying visual forms.", "methods_and_evaluation_criteria": "This paper does not propose a method.", "theoretical_claims": "This paper does not involves any theoretical claims.", "experimental_designs_or_analyses": "The authors compare the performance of ViT-based models, CNN-based models, and closed-source MLLMs on the proposed dataset. However, they do not fine-tune open-source MLLMs, such as LLaVA and Qwen-VL, which would have enabled a more comprehensive comparison. Incorporating such models could provide valuable insights into the adaptability of open-source alternatives.\n\nAdditionally, the authors emphasize the significance of varying visual representations of identical concepts across different real-world domains. However, they do not present any empirical evidence to support this claim. At a minimum, conducting preliminary experiments on real-world datasets would be feasible and could strengthen their argument.", "relation_to_broader_scientific_literature": "The proposed dataset could enhance AI models' ability to recognize and reason about visual representations of identical concepts across different scientific and technical domains, such as electrotechnics and chemistry.", "essential_references_not_discussed": "The core argument of this paper closely aligns with that of [1], which emphasizes the importance of layout augmentations in visual graphs. In [1], the authors identified layout augmentation as the most effective method for improving AI models' recognition and reasoning abilities on visual graph reasoning, despite variations in visual form. In other words, [1] has already demonstrated the significance of this concept. While the proposed dataset includes one additional task (i.e., Isomorphism) that was not covered in [1], it is recommended to cite [1] to acknowledge its contributions.\n\n[1] Wei, Yanbin, et al. GITA: Graph to visual and textual integration for vision-language graph reasoning. _NeurIPS 2024_.", "other_strengths_and_weaknesses": "Weaknesses:\n\nThe contribution of this paper appears marginal due to significant overlap with previously published work [1]. While I appreciate the authors’ effort in incorporating the performance of human participants, the novelty remains limited. Additionally, to enhance the potential value of the proposed VGP dataset, I recommend providing more empirical evidence across diverse domains to better differentiate it from prior works [1, 2, 3].\n\n[1] Wei, Yanbin, et al. GITA: Graph to visual and textual integration for vision-language graph reasoning. _NeurIPS 2024_.\n\n[2] Deng, Yihe, et al. GraphVis: Boosting LLMs with Visual Knowledge Graph Integration. _NeurIPS 2024_.\n\n[3] Hu, Baotian, et al. VisionGraph: Leveraging Large Multimodal Models for Graph Theory Problems in Visual Context. _ICML 2024_.", "other_comments_or_suggestions": "For Figure 2, it is recommended to provide additional explanations for each layout, as not all readers may be experts in this field.", "questions_for_authors": "Is it possible to introduce more different layouts for the same visual graph?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper received 1 weak reject and 3 weak accepts after the rebuttal. On the positive side, the reviewers recognized the value of the benchmark as it may inspire more future work in this direction. On the negative side, the reviewer qrzb had concerns about limited baselines and sample sizes. The authors had a couple of rounds of discussions with the reviewer, where results of both opne-source and closed-source models are reported. 

Overall, the AC found the positive side outwieghs the negative side and recommends an accpetance. The authors are highly encouraged to incorporate the new results reported in the rebuttal into the final version.
Decision: Accept (poster)