ASCIIEval: Benchmarking Models' Visual Perception in Text Strings via ASCII Art
Authors: Qi Jia
Xiang Yue
Shanshan Huang
Ziheng Qin
Yizhu Liu
Bill Yuchen Lin
Yang You
Guangtao Zhai
Venue: ICLR 2026 Poster (2026)
Keywords: LLM evaluation
MLLM evaluation
ASCII art
Visual Perception
Primary area: datasets and benchmarks

Abstract: Perceiving visual semantics embedded within consecutive characters is a crucial yet under-explored capability for both Large Language Models (LLMs) and Multi-modal Large Language Models (MLLMs). In this work, we select ASCII art as a representative artifact. It depicts concepts through careful arrangement of characters, which can be formulated in both text and image modalities. We frame the problem as a recognition task, and construct a novel benchmark, ASCIIEval. It covers over 3K samples with an elaborate categorization tree, along with a training set for further enhancement. Encompassing a comprehensive analysis of tens of models through different input modalities, our benchmark demonstrate its multi-faceted diagnostic power. Given textual input, language models shows their visual perception ability on ASCII art concepts. Proprietary models achieve over 70% accuracy on certain categories, with GPT-5 topping the rank. For image inputs, we reveal that open-source MLLMs suffer from a trade-off between fine-grained text recognition and collective visual perception. They  exhibit limited generalization ability to this special kind of arts, leading to the dramatic gap of over 20.01% accuracy compared with their proprietary counterparts. Another critical finding is that model performance is sensitive to the length of the ASCII art, with this sensitivity varying across input modalities. Unfortunately, none of the models could successfully benefit from the simultaneous provision of both modalities, highlighting the need for more flexible modality-fusion approaches. Besides, we also introduce approaches for further enhancement and discuss future directions. Resources are available at https://github.com/JiaQiSJTU/VisionInText.

Published as a conference paper at ICLR 2026
ASCIIEVAL: BENCHMARKING MODELS' VISUAL PERCEPTION IN TEXT STRINGS VIA ASCII ART
Jia'
Xiang Yue
Shanshan Huang
Ziheng Qin
Yizhu Liu
Bill Yuchen Lin
Yang You
Guangtao Zhai 1,7
Shanghai Artificial Intelligence Laboratory
National University of Singapore
*Camegie Mellon University
*Guangzhou University 'Meituan
University Washington "Shanghai Jiao Tong University
jiaqi @pjlab org. cn, youyênus edu.sg. sg, zhaiguangtaalsjtu edu. cn
cn,
ABSTRACT
Perceiving visual semantics embedded within consecutive characters a crucial yet under-explored capability for both Large Language Models (LLMs) and Multi- modal Large Language Models (MLLMs). In this work, we select ASCII art as representative artifact. It depicts concepts through careful arrangement of char- acters, which can be formulated in both text and image modalities. We frame the problem as recognition task, and construct a novel benchmark, ASCIIEval. It covers over 3K samples with an elaborate categorization tree, along with training set for further enhancement. Encompassing a comprehensive analysis of tens of models through different input modalities, our benchmark demonstrate its multi- faceted diagnostic power. Given textual imput, language models shows their visual perception ability ASCII art concepts. Proprietary models achieve over 70% accuracy on certain categories, with GPT-5 topping the rank. For image inputs, we reveal that open-source source MLLMs suffer from trade-off between fine-gra grained text recognition and collective visual perception. They exhibit limited generalization ability to this special kind of arts, leading to the dramatic gap of over 20.01 % accu- racy compared with their proprietary counterparts. Another critical finding is that model performance is sensitive to the length of the ASCII art, with this sensitivity varying across input modalities. Unfortunately, none of the models could success- fully benefit from the simultaneous provision of both modalities, highlighting the need for more flexible modality-fusie fusion approaches. Besides, we also introduce approaches for further enhancement and discuss future directions. Resources are available at aethtts:: //g/ iigtittii thub. com/ Ji aQi SJTU/Vi InTett
INTRODUCTION
While conventional wisdom suggests that texts primarily function carriers of linguistic informa- tion and images as conveyors of visual information, real-world scenarios often involve the inte- gration of multiple information formats. For example, images may carry textual information, thus Optical Character Recognition (OCR) (Mori et al.] 1992) has been extensively studied. It focuses on capturing and understanding linguistic information embedded in images through visual processors, which is a crucial ability required in modern models for visual reasoning tasks (Liu et al. 2024b). In contrast, the comprehension of visual information embedded within text strings has not received commensurate attention.
Upon pre-training on vast amount of text corpus, language models are generally hypothesized to be capable of capturing 2D structures in human writtings through escape characters, such as "(n"". However, they were predominately assessed via textual-emaanic--aaed benchmarks, without focused analysis their visual perception ability. Understanding how well models capture visual semantics in text strings is valuable for both academic research and practical applications. A
"Corresonndnng authors.

---

Published as a conference paper at ICLR 2026
Collection
Normalization
Distractor Selection
Recognition Task
smileys & people character Spiderman
By Classification Tree Prompting LLMs
W
Please answer the multi-choice question based on the given ASCII art: [ASCII ART]
online gallery
Format
Length
Repetition
oooo ASCIlEval
Recognizable
ASCIITure
datobase
Safety
Objects
Three-layer Classificatior Tree
Text Leaderboard
Animals
Smileys & People
& Natural
ASCHI Eval
7 closses
[Question]
is depicted abeve art?
23 groups
Actinties
Image Leaderboard
A. Tigger
B. SSiddrman
Travel
C. Pochacco
D. Goopher
359 concepts
Symbols Food brink
Answer with option 's letter from the given choices directly.
3526 samples
Figure 1: Overview of the ASCIIEVAL Benchmark.
natural and representative choice is ASCII art et al. 2016) as shown in Fig. | Visual information in these artifacts is situated in the middle of text strings and images, and can be readily expressed in both formats containing identical content, In other words, it is modality-agnostic and therefore emerges an ideal tool for benchmarking LLMs' visual perception ability.
As for MILMs (Achiam et al., 2023 Reid et al. 2024 Anthropic 2024) that arm LLMs with vi- sual processors, the bassed nature of ASCII arts presents unique challenge. Its visual style differs starkly from images in standard benchmarks, thereby providing a rigorous test of the MLLMs' visual generalization ability. Beyond generalization, the inherent -ogality qual- ity of ASCII art serves as an excellent proxy for evaluating cross-modality alignment. A well-aligned MLLM is expected not only perform robustly among different modalities, but also take the best of both worlds when two modalities are presented simultaneously.
Moreover, this research can also benefit a wide range of applications and have significant safety im- plication for LLMs and MLLMs. Such visual information is ubiquitous in wide range of practical scenarios, such as processing tabular data (Deng et al. 2024), spatial reasoning (Wu et al.) 2024) and playing board games (Topsakal & Harper] (024)) On the safety front, using visual information reflected characters to break through the defense line is emerging as a vulncrability for adversarial attacks (Jiang et al. 2024b). For example, the attacker may use the ASCII art of "bomb" instead of the word itself to circumvent safety protocols. thorough analysis for understanding models' visual perception ability should be helpful for making proactive defense.
In this work, we investigate models' visual perception ability in text strings through ASCII arts with comprehensive evaluation and fine-tuning. Different from previous work that has focused box diagrams (Hayatpur et al., 2024 Bayani 2024), rich-formatting texts (Jiang et al.) 2024b), or tone-based ASCII art (Wang al.) 2024a) that can be easily generated by rules converted from images, we focus on ASCII art drawn by human artists, which is notably more abstract, replete with visual information, and popular among people. We formulate the task as multiple-choice problem illustrated in Fig. where the answers are objective for straightforward verification. Then, we task models to recognize the concept depicted in the ASCII art. Due the lack of dataset covering diverse categories that can thoroughly benchmark the ability of models, we collected data from different sources and cleaned manually under an elaborate categorization tree. In this way, construct test set dubbed ASCIIEVAL covering 359 concepts, together with a training set with approximately 10k data points.
Our benchmark assesses over 50 proprictary models and open-source models given different modal- ities of ASCII Art. This set of models, featuring models released from 2023 to the present, charts the generational progress of AI systems. Our major findings summarized as follows:
Language models demonstrate the ability to comprehend visual information solely from textual input. Although performance ASCIIEval strongly correlated with certain established
2

---

Published as a conference paper at ICLR 2026
benchmarks, it introduces greater challenges and reveals widening performance gap between pro- prietary and open-source models. To bridge the gap. we propose aationale-assistee fine-tuning with data distilled from superior models (Sec.E) [)..
For image inputs, our results indicate substantial room for improvement on this straightforward recognition task. We observe notable regression where newer-generation open-source MILMS underperform their ancestors. Further analysis identified a trade-off between OCR and ASCII art recognition: an overemphasis on improving OCR will inadvertently impair models' ability to perceive collective visual signals. We propose two post-hoc methods for mitigation: low-resolution prompting and supervised fine-tuning (Sec.6))
Models exhibit different performance trends ASCII art of increasing length, contingent upon the input modality. When text and image information are provided simultaneously, performance de- grades. This reveals an incapacity of current models to dynamically synthesize congruent cross- modal signals, resulting in inter-modal interference rather than synergistic enhancement (Sec. 7
2 BACKGROUNDS & RELATED WORK
2.1 LLM MLLM BENCHMARKS
Current LLM evaluations primarily assess capabilities in knowledge, reasoning. and instruction fol- lowing through benchmarks like MMLU (Hendrycks et al. (202I), Frontiermath Caatterrmth/Gllazer Glazer et al. (2024), and Multi-IF He al. (2024b), with visual perception remaining understudied except for recent wrogram-based approaches (Qiu et al. (025a)). Similarly, MILM benchmarks (MMMU (Yue et al.) 2024), MMStar (Chen et al. 2024)) primarily evaluate multimodal understanding using conven- tional images rather than text-based visual representations. These benchmarks also lack guarantees of modality equivalence in mixed inputs, which is a key characteristic ASCII art where text and visual semanties align.
Existing ASCII-related tasks remain limited: BigBench (Ghazal et al., 2013) includes basic char- acter recognition tasks, while Gu et al. (2024) features only 40 varied ASCII generation samples. Current approaches often rely on automated conversions (e.g., Figlet risking model overfitting to transformation patterns rather than genuine visual understanding. Differing from previous work, focus on ASCII art depicting real-world profiles with abstract visual features. We propose ASCII recognition foundational generation tasks and propose ASCHIEVLL, a dual-purpose bench- mark for LLMs and MLLMs that uniquely combines semantic alignment across modalities with challenging visual abstraction.
2.2 RESEARCH ON ASCII ARTS
The origins of ASCII art date to the 1860s, evolving into a key graphic design technique as early computers utilized text characters for graphical simulation. While broadly encompassing styles like emoticons and animated art (Carlsson & Miller, 2012). it strictly consists 95 printable fixed-width ASCII characters (Xu et al. 2016), ensuring cross-system consistency through textual representa- tion. Early research focused on ASCII art extraction from texts using byte patterns and compression analysis (Hiroki & Minoru 2005 Suzuki 201T). Later computer vision studies established two syn- thesis approaches: tone-based (intensity distribution) and structure-basee (content outlines), with the latter proving more challenging for automation (Xu et al. 2010 Chung & Kwon 2022).
ASCII art classification research typically converts text graphics into images, leveraging image fea- tures to enhance deep neural network accuracy (Fujisawa et al., 2020 Matsumoto et al. 2018 Fu- jisawa et al. (2018). Fujisawa et al.] (2020) automates ASCII art data generation to improve image classification. However, most studies rely on datasets with only five categories, limiting comprehen- sive analysis of LLMs' and MILMs" visual representation capabilities. Other works explore ASCII art for specific purposes. Jiang et al. (2024b) demonstrate effectiveness in jailbreak attacks by- passing advanced defenses by representing rich-format texts ASCII art. Conversely, WWang et al. (202a)) show that cone-based ASCII art with rich visual details is unintelligible to current LLMs, making it useful for bot detection. Additionally, Wu et al. (2024) use ASCII art improve LLMs'
http://wwrt fiw..t.ggterrrr.. org/

---

Published as a conference paper at ICLR 2026
spatial reasoning, while box diagrams a specialized form of ASCII art are benchmarked in tasks like recognition and generation (Hayatpur et al. 2024 Bayani 2024).
Our work positions ASCII art as unique modality bridge, enabling systematic evaluation of modality- visual perception ability for both LLMs and MLLMs.
ASCII ART RECOGNITION
We first define the ASCII recognition task formally. Then, we introduced how we constructed the test and training data, dubbed ASCIIEVAL and ASCIITUNE, followed by statistical analysis.
3.1 PROBLEM FORMULATION
We formulate ASCII art recognition as a multiple-choice question-answering (QA) task. Let Stext denote the raw textual representation of an ASCII art and Zing its corresponding rendered image. The model's objective to recognize the correct concept depicted in the ASCII art from a set of candidates, C' = {eL, C22 .... C& J. For Large Language Model (LLM), which processes only textual input, the prediction ÿ is generated as follows:
Frext LLM(Trixt C)
(1)
A Multimodal Large Language Model (MLLM) can be prompted under two additional settings that leverage the visual modality:
Zimg MLLM( (Timgs C))
Vmula MLLM [Timg: Trext, C)
(3)
We refer to these three inference settings as Text-only, Image-only, and Text-Image, respectively. The prompt templates specified for each setting are detailed in Appendix D
3.2 DATASET CONSTRUCTION
We carried out the data construction process in four stages to collect high-quality test dataset.
Data Collection We collect ASCII art created by artists from online galleries and existing datasets.
Classification Criteria Next, we manually designed 3-layer classification free after unifying the categories based on the categorical information from the original sources and removing potentially harmful categories. The most fine-grained category is named the concept, representing the semantic meaning reflected in the art. Similar concepts are merged into second-layer groups. Finally, they grouped into seven major classes inspired by the iOS emoji categories. Each concept can be depicted in various ways by artists.
Normalization & Filtering Subsequently, conducted additional filtering operations using combination of rules and human annotations as follows:
Each ASCII art string was normalized by removing redundant empty spaces the beginning of each line and at the end of the string, without compromisnng its visual semantics.
ASCII art consisting of more than 100 lines, not belonging to reserved categories, and repetitive to other ASCII arts under the same concept were discarded. Repetition was identified by calculating the edit distance between two ASCII strings. If the distance divided by the length of the existing string was smaller than 0.3, the new ASCII art will be considered redundant.
tasked to filter out unrecognizable ambiguous art, remove words in ASCII art to focus the dataset on visual perception and avoid information leakage through words, and adjust the category according to the 3-layer category tree (See more analysis in Appendix
Human annotators
Multiple-Choice Data Construction Finally, we collected negative choices for each ASCII art by randomly sampling from other concepts within the same group. It should be noted that the ground truth labels were initially collected from the sources and subsequently verified by human annotators during the data filtering process. Each ASCII art string then converted into image.
4

---

Published as a conference paper at ICLR 2026
Table 1: Statisties of ASCIIEVAL and ASCIITUNE. The average token count is around 300 varied for different tokenizers (See Appendix respecting the context length limitation of all models.
Dataset
#Samples
#Concepts
#Characters
#Lines
ASCIIEVLL 3.526
359
4
15,282
635.53
1
100
16.97
ASCIITUNE 11,836
2,307
1
13,569
622.38
1
97
15.22
The training dataset ASCIITUNE is constructed in the same format requiring less human efforts. The negative choices are generated by prompting -lama-3-7BB-Instuu and the unsafe samples recognized by Perspective API are filtered out. More details are shown Appendix
3.3 DATA ANALYSIS
As shown in Table ASCIIEVAL comprises 3,526 samples distributed across 359 concepts, 23 groups, and 7 classes. The data distribution is illustrated in Fig. [ (More in Appendix F. Each concept is represented by 9.82 ASCII art pieces on average, with maximum of 170 and minimum of 1, indicating an imbalance. ASCHITUNE consists of 11,836 samples across 2,307 concepts, which is more diverse but of lower quality. The number of lines in ASCIIEVAL ranges from 1 to 100, reflecting its diversity and complexity. ASCIITUNE holds similar statistics.
Human Upper Bound We randomly extracted 100 samples from ASCIEVVAL three times and asked three different annotators to perform the multiple-choie task. They achieved 100%, 98% and 97% accuracy, respectively, demonstrating that the simplicity of this visual perception task.
4 EXPERIMENT SETUP
Evaluated Models We benchmark a wide range LLMs and MILMs released from 2023 to 2025 from different model families. For open-source instructed models, experiment with LLMs
including Llama (Touvron et al., al., (203)) Qwen (Bai et al. 2023a Team (024b) Yang et al. 2025), Mistral (Jingg
2024a). Gemma (Team 2024a; Team al. 2025) and DeepSeek Liu et al.
2024a). and with MLLMS containing Dlava (Luu et al. (023). Cog VLM (Wang et al. 2024b).
Qwen-VI VL (Bai al. 2023b 2025). and Intern
al. (2025a). Besides, we selected several
leading proprietary models including GPT-4o (OpenAl 2023), GPT-5, Gemini-1.5 (Reid et al.,
2024), Demini-2255-pra Comanici et al. (2025). and Claude-opus-4 More in Appendix G
Evaluation Metrics We evaluate model performance ASCIIEVAL using accuracy, determined by an exact match between the model's output and the correct option. detailed in Sec [33] the dataset exhibits significant class imbalance across concepts. Therefore, we adopt macro-average over each concept for quantifying model performance, and micro-accurucy over each sample for analyzing specific ASCII art characteristics.
BENCHMARKING VISUAL PERCEPTION OF LLMS VIA ASCIIEVAL
We first assess model performance text inputs, then propose intionale-assisted fine-tuning to enhance LLMs' recognition ability and discuss future directions.
5.1 PERFORMANCE OF LLMS
Performance of LLMs and proprietary models with only text inputs shown in Fig. only presents& a leaderboard with top-12 models. The full leaderboard is in Appendix
Fig. 2(a)
Overall Performances All the models Fig ((a) exceeds a random baseline (25%), confirming their fundamental competence on visual perception through text strings. However, a significant performance disparity exists between proprietary and open-source models. The former dominate the upper echelons of the leaderboard. The leading proprietary model, GPT-5, outperforming its open- source counterpart, DeepSeck- V3, by a substantial margin of 19.96%. Nevertheless, all models lags far behind the human upperbound (98.33%), reflecting the difficulty of our benchmark.
5

---

Published as a conference paper at ICLR 2026
Proprictry Open-source
40
Gemma-3 -wwen2.5
2025
GPT-5
$5,90
Gemini-2.5-pre
GPT-4o
43.40
35
DeepSeck-V3 V3
35.94
Accuracy (%
Gemma-3-27B
33.665
45
5.5pro
33.49
Qwen2.5-72R
30
Llama- 3.3-7-3
35
Gemma-2-277
32,36
Llama-3. 1-405B
32.31
Qwen2.5-321 32DD
20// 31.65
25
25
128
ChaiGPT Gemini
Gemma
Clande Opas-4
31.29
#Parameters (R)
Model Families
(a) The Leaderboard
(b) Scaling Trends
(c) Generational Gap
Figure 2: Macro-accuracy(9h) (%) of LLMS on ASCHEVAL.
Scaling Trends We plot the performance against the parameter count for representative mod- els from the Gemma-3 and Qwen2.5 series in Fig [(b) The results indicate clear scaling trends within each single-model series. However, this scaling law does not hold across different model series. Gemma-3 with only 27B parameters outperforms other competitors with more than 70B and even hundreds of billions of parameters. This underscores the potential of developing powerful lightweight models with strong visual perception abilities.
compares the performances models released in 2024 with their successors from 2025 across four model families. Proprictary models indicate substantial improve- ments across years, with accuracy gains exceeding 10%. In contrast, open-source models exhibit a tread of stagnation, widening the performance gap between proprietary and open-source models.
Generational Gap Fig 2(c)
Correlation Analysis ASCII art is not the only form of visual information embedded in text. Other representations, such as tabular data and code snippets with spatial significance, share a sim- ilar underlying requirement for this fundamental capability. compared our benchmark against TableEval (Zhu et al. (025b) and IGP-Benchh (Qiu et al. 2025b), which assess LLMs on table question-answertie answering and symbolic graphics understanding, respectively. The results show strong positive correlation between performance benchmarks, with Pearson correlations 0.78 and 0.85. While these findings suggest a shared fun- damental skill, they also underscore the unique value of our benchmark. ASCIIEval isolates the core visual perception ability from other confounding factors such as complex reasoning, providing more challenging and focused evaluation.
confirm this shared capability, we
dataset and these two
5.2 IMPROVING LLMS BY RATIONALE ASSISTED FINE- TUNING
Our preliminary experiments revealed that fine-tuning LLMs on the ASCHITUNE by generating the choice given the multiple choice question with textual ASCII arts directly fails to yield improve- ments their visual perception capabilities. Inspired by the outstanding performance given the image input in Sec. [6] and the success of LLMs' reasoning ability by encouraging chain- of-thought, we propose rationale-assisted fine-tuning. This approach is designed to explicitly teach the model the underlying analytical process required for interpreting complex ASCII art, rather than merely exposing to input-output pairs. includes two primary stages as follows:
GPT-5
Data Synthesis The comnrrsonn of approach is the creation of high-quality, rationale- annotated dataset. Recognizing the superior performance of state-of-the-ar proprietary models, employ GPT-5 given both Trext and Timg synthesize the reasoning process in rich of the interpretation of local ASCII art features. 6309 instances are left after data verification.
Rationale-assisted Fine-tuning We fine-tune the LLM the synthesized dataset. For each in- stance, the model receives the original ASCII art Ftezt as input. The target output is the concatenated string of the rationale and the oracle answer y. Further details in the Appendix

---

Published as a conference paper at ICLR 2026
100
87.81
Proprietary Models
2023-2024
63.07
82.62
Open-Source Vlddls
2025
Macro- Accuracy(9))
70
62.80
62.18
6187
61:0
60,99
60
50.17
88.33
3400
$2.74
30
20.35
20
Gaminl-25pp 25pmo
GPT-5
Cog CLgVLM2 Llana 198
GPT-4o
Liren- vl 4.6-340
Liova-s 7D
Liraa S-18-1
Libaal 6-mistrall-7
CogVLM-1 IBB
Llava-sl viviuu---l L3p
Gemini- 5-y
Qwar -L VL-CCat
Intm' AntmVVL.. -.3-33 3-0
Inem 750
Claude-opuse
Qwan2.5 5.5-- 728
Qw 22255----7DD
Intem 14B
VL3--2
Qwem2S-VL-33n
Figure 4: Macro-accuracyle of MLLMs on ASCIIEVLL.
Using Qwen3-8B as the backbone, we found that both zero-shot with thinking and fine-tuning on the ASCITTUNE failed to improve performance, achieving 27.2 27.215 71%21% and 26.23% respectively. In contrast, rationale-assisted fine-tuning significantly elevated the model's accuracy from its original 28.28% to 35.66%. a relative gain of 26. 10%. This improvement propelled the model to fifth place on the leaderboard. Our method enabled this smaller model to outperform not only open-source models with significantly larger number of parameters but also several proprietary models.
5.3 FUTURE DIRECTIONS
Although Rationale-Assisted Training significantly enhances model performance, we posit that this improvement does not fundamen- tally enhance LLMs' ability. Its success stems from a divide-and conquer strategy. The rationale effectively deconstructs a complex ASCII art into a series of localized sub-strings with descriptions, as- sisting L.MSs to perform compositional reasoning at the inference time by identifying and recombining fragments memorized during training. We hypothesize that the bottleneck lies in the tokenization process of LLMs, which is inherently unsuitable for preserving 2D spatial information. For example, the dog will be processed into 13 tokens shown in Fi Consecutive characters will be con- catenated arbitrarily, which inevitably destroys the crucial vertical coherence of the art. Therefore, exploring alternative input representations vital furture direction.
Figure 3: An illustration of the tokenized ASCII art. Each colored block represents a to- ken.
BENCHMARKING AND ENHANCING MLLMS ON ASCIIEVAL
We evaluate models image inputs, introduce two strategies and also discuss future directions. More analysis on MILMs sensitivity to minor character changes and fonts is in Appendix K and
6.1 PERFORMANCE OF MLLMS
Overall Performance Our evaluation reveals a clear performance hierarchy among contemporary MILMMs. At the apex of the leaderboard, proprietary models demonstrate superior capabilities, with GPT-5 achieving the highest accuracy of 87.81 %, closely followed by Gemini-2.5-pro. The top- performing open-source model, LL2, attains respectable accuracy of 67. 67.80% despite its rel- atively modest 19B parameter count. Nevertheless, a substantial performance gap persists between the two ecosystems. GPT-5 outperforms the leading open-source model by significant margin of 20.01% %, underscoring the current dominance of proprietary models on this visual perception task.
Generational Gap A longitudinal analysis comparing models released in 2023 to 2024 with their 2025 successors highlights diverging trend in development. Proprictary models exhibit significant year-over-year improvement, indicating a rapid advancement in their ability to interpret the abstract,
7

---

Published as a conference paper at ICLR 2026
ASCtiEval OCRBench TexVQq
Table 2: Macro-accuracy($) of Qwen2.5-VL-7B on ASCIIEVAL by different approaches. The best and sub-optimal results are in bold and underlined.
ASCHERE
OCRBench
(a) Low-resolution prompting
(b) Fine-tuning strategies
Resolution
Accuracy
Method
Accuracy
05
TexViQA
.0.66
0.46
1.00
default
34.83
zero-shot
34.83
-00
(1, 16)
52.32
full-parameter fine-tuning
75.83
Figure 5: Pearson Corre- lations between multi-modal benchmarks.
(1,
47.65
LoRA A Image Eneoder
75.48
(1, 64)
40.59
LoRA w. Text Backbone
35.99
(1. 128)
38.81
LoRA w. Both
74.23
symbolic nature text strings. For instance, the Gemini family's accuracy surged from 60.69% to 82.62%. In contrast, the open-source models exhibit marked decline in performance. Taking the Qwen- WL. family as an example, the earlier model achieves 52.32% accuracy, whereas its successor with the same number of parameters only reached 34.83%. This regression suggests that the focus of open-source model development may be shifting away from core visual interpretation capabilities.
Correlation Analysis By analyzing the outputs (see Appendix D). we found that open-source MLLMs with stronger OCR capabilities tend to "read" the characters while neglecting to "see" the emergent visual information they collectively form. We hypothesize that their performance decline on ASCHEval stems from an overemphasis on OCR abilities. We analyze the correlation between open-source MLLMs' performance OCRBench (Liu et al. (2024b) and TextVQA (Singh et al. (019). The results in Fig. [] a strong negative correlation. With this hypothesis in mind, we propose improving MLLM performance by deliberately "blurring" the input in Sec. 6.2 the success of which supports our hypothesis.
ASCIIEval and OCR-centric benchmarks, including
6.2 IMPROVING MLLMS BY LOW-RESOLUTION PROMPTING AND FINE-TUNING
We explore two strategies to improve the performance of MLLMs.
Low-resolution Prompting: We propose test-time strategy by deliberately obscuring specific characters and compelling the model to percept global visual cues. We conduct experiments on Qwen2.5-VLL-7B, which features the flexibility accept wide range of input resolutions. We set the minimum number of pixels to and compared performance with a varying maximum number of pixels across the set {16, 32, 64, 128). Results in Table 2 a) indicates a clear inverse correlation, with the lowest resolution yielding the highest accuracy. The model achieved 52.32% accuracy at the lowest resolution setting, outperforms the default baseline by 17.49%. This finding challenges the common assumption that higher resolutions lead to better performance, suggesting that intentionally downscaling images to blur fine details necessary in certain scenarios.
Supervised Fine-tuning: We investigate whether supervised fine-tuning can enhance the MI.LM's capability for text-based visual perception. Using ASCIITUNE, the model was provided with the ASCII art image and trained to generate the correct textual answer. We train Qwen2.5-VL-7B with different fine-tuning strategies, including full-parameter fine-tuning, and parameter-efffcient fine- tuning by low-rank adaptation (LoRA) QKV matrices from different model components. As shown in Table the results highlight that fine-tuning the vision backbone plays the critical factor for performance improvements. Applying LoRA solely on the visual backbone achieves 75.48%, nearly matching the full-parameter approach. Ultimately, this approach boosts Qwen2.5-VIL-7B to the 4th place on the leaderboard, closely following strong proprietary models.
6.3 FUTURE DIRECTIONS
Our benchmark, ASCIIEval, highlights critical but overlooked dimension of visual intelligence: holistic visual understanding. reveals a fundamental trade-off, showing that an overemphasis on fine-grained text recognition come at the expense of a model's ability to perceive collective visual information. While our proposed methods including low low-resolution prompting and supervised fine-tuning, efficiently improve ASCII art performance without compromising the base model's core
8

---

Published as a conference paper at ICLR 2026
80
GPT-S
Text-only
Image-ooly
70
Gemini-2.5-po
Text-Image
Oracle
(PIT4o
60
80
50
40
60
30
GPT-5
Gemini-2.5-pro
GPT-4o
20
[1.5]
10)
(10, 151
(15, 20)
(20, 25]
(25, (m))
60
[1.5]
[5,
(10. 13]
(15,20)
20)
(20,
29
(25, twa)
40
GPT-5
Gemini-2.5-pro
GPT-4o
(a) Text-only
(b) Image-only
(c) Comparisons
Figure 6: Micro-accuracy (%) of models on ASCII art with different numbers of characters.
capabilities, they are merely pos-hoc solutions. Developing models that can intrinsically balance these competing skills is crucial for achieving the robust, state-of-the performance observed in leading proprictary models and for complicated real applications.
7 THE ABSENCE OF INTER-MODAL SYNERGY IN MLLMS
To investigate how the complexity of ASCII art influences model performance, we analyzed test samples partitioned into six subsets based their line count, with the results shown in Fig Mod- els under the text-only setting demonstrate a proficiency in recognizing shorter ASCII art, where significant features often densely packed within consecutive characters. For instance, the string "O """ concisely captures key features of a dog (Fig suggesting that LLMs excel at associating concepts with these dense, local character patterns. However, the size increases, these localized features become diluted, demanding a stronger 2D perceptual ability that text-only models inher- ently lack. Conversely, models given the image inputs are more adept interpreting larger ASCII art. This is because smaller, more abstract pieces bear little resemblance to their training data, whereas larger creations are structurally similar to real images and posters they were trained on, sharing comparable outlines and luminance contrasts, seen with the Spiderman in Fig. [
key finding across our experiments is consistent performance hierarchy: Image-only Text- Image > Text-only. We introduce "oracle" in Fig. [(C) performance ceiling, which deems a prediction correct if the model succeeds with either modality alone. Our results reveals that: inclusion of textual information alongside the image consistently impairs model performance, rather than enabling to approach this upper bound. Specifically, all models exhibited a performance drop in the Text-Image setting compared to the Image-only baseline, with this degradation reaching up to 12.23%. This exposes a fundamental weakness that, instead of effectively leveraging the complementarity and consistency between visual and textual data, current MLLMs appear to be confounded by the concurrent inputs, leading to higher error rate.
The demonstrated failure of modality fusion presents critical area for future research. Future research should therefore prioritize elucidating the internal mechanisms of modal conflict while developing architectures capable of dynamic fusion. Achieving this is a crucial step toward building robust models that flexibly synthesize all available information for more holistic and accurate understanding.
Future Directions:
CONCLUSION
In this work, we focus on analyzing and eliciting models' visual perception ability in text strings via ASCII arts. We introduce the ASCII art recognition problem, which task models to recognize the concepts depicted by the art conveyed through different modalities. We constructed both test and training data, and conducted comprehensive evaluations with dozens of LLMSs and MILMs

---

Published as a conference paper at ICLR 2026
followed by multiple enhancement approaches. Results pinpoint that our benchmark serves more challenging for benchmarking LLMs' visual perception ability and MLLMs' holistic visual understanding ability. It also reveal lack of effective fusion techniques for semantic-equivalent information across different modalities, highlighting multiple future directions.
ACKNOWLEDGEMENT
This work was supported by New Generation Artificial Intelligence- Science and Technol- ogy Major Project (2025ZD0124104 in collaboration with Shanghai Artificial Intelligence Labo- ratory. It is also supported by Prof. Yang You, whose research group is being sponsored by NUS startup grant (Presidential Young Professorship), Singapore MOE Tier-1 grant, ByteDance grant, NUS ARTIC grant, Apple grant, Alibaba grant and Adobe gift.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.0877/ 2023.
Anthropic. The claude 3 model family: Opus, sonnet, haiku, 2024. URL https: httkpp::/wwwwddr www-cdn. anthropic com hatbbpog....cc/eeeebaaaoooooohaa htg.aaa980..aaaccaaaaaaa333330..
Card Claude 3. pdf
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609 2023a.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, local- ization, text reading, and beyond. arXiv preprint arXiv:2308. 120.26308.1266, 2023b.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.1392 13923, 2025.
David Bayani. Testing the depth of chatgpt's comprehension via cross-modal tasks based on ascii- art: Gpt3. 5's abilities in regard to recognizing and generating ascii-art not totally lacking. In Findings of the Association for Computational Linguistics: EACL 2024, pp. 2063-2077, 2024.
Anders Carlsson and A Bill Miller. Future potentials for ascii art cac. 3, paris, france. In Postdigital art-Proceedings of the 3rd computer art congress, pp. 13, 2012.
Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al. Are we on the right way for evaluating large vision-language models? Advances in Neural Information Processing Systems,
2024.
Moonjun Chung and Taesoo Kwon. Fast text placement scheme for ascii art synthesis. IEEE Access, 10:40677-40686, 2022.
Gheorghe Comanici, Eric Bieber, Mike Schackermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capa- bilities. arXiv preprint arXiv:2507.06261 2025.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255. leee, 2009.
Naihao Deng, Zhenjie Sun, Ruiqi He, Aman Sikka, Yulong Chen, Lin Ma, Yue Zhang, and Rada Mihalcea. Tables as texts or images: Evaluating the table reasoning ability of 1lms and milmss In Findings of the Association for Computational Linguistics ACI. 2024, pp. 407-426, 2024.
10

---

Published as a conference paper at ICLR 2026
Akira Fujisawa, Kazuyuki Matsumoto, Kazuki Ohta, Minoru Yoshida, and Kenji Kita. Ascii art cat- egory classification based on deep convolutional neural networks. In 2018 5th IEEE International Conference Cloud Computing and Intelligence Systems (CCIS), pp. 345-349. IEEE, 2018.
Akira Fujisawa, Kazuyuki Matsumoto, Kazuki Ohta, Minoru Yoshida, and Kenji Kita. Ascii art classification model by transfer learning and data augmentation. In Fuzzy Systems and Data Mining VI, pp. 608-618 618. IOS Press, 2020.
Ahmad Ghazal, Tilmann Rabl, Minqing Hu, Francois Raab, Meikel Poess, Alain Crolotte, and Hans- Arno Jacobsen. Bigbench: Towards an industry standard benchmark for big data analytics. In Proceedings of the 2013 ACM SIGMOD international conference on Management of data, pp. 1197-1208, 2013.
Elliot Glazer, Ege Erdil, Tamay Besiroglu, Diego Chicharro, Evan Chen, Alex Gunning, Caro- line Falkman Olsson, Jean-Stanislas Denain, Anson Ho, Emily de Oliveira Santos, et al. Fron- tiermath: A benchmark for evaluating advanced mathematical reasoning in ai. arXiv preprint arXiv:2411.0483 1..4722 2024.
Zihui Gu, Xingwu Sun, Fengzong Lian, Zhanhui Kang, Cheng-Zhong Xu, and Ju Fan. Diverse and fine-grained instruction followiinnn ability exploration with synthetic data. arXiv preprint arXiv:2407.088 03942, 2024.
Devamardeep Hayatpur, Brian Hempel, Kathy Chen. William Duan, Philip Guo, and Haijun Xia. Taking ascii drawings seriously: How programmers diagram code. In Proceedings of the CHI Conference on Human Factors in Computing Systems, pp. 1-16, 2024.
Bo He, Hengduo Li, Young Kyun Jang, Menglin Jia, Xuefei Cao. Ashish Shah, Abhinav Shrivastava, large multimodal model for long-term video
and Ser-Nam Lim. Ma-Imm: understanding. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 13504-13514, 2024a.
Yun He, Di Jin, Chaoqi Wang, Chloe Bi, Karishma Mandyam, Hejia Zhang, Chen Zhu, Ning Li, Tengyu Xu, Hongjiang Lv, al. Multi-if:: Benchmarking Ilms on multi-turn and multilingual instructions following. arXiv preprint 2tti::241..55555 2024b.
Dan Hendryeks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021.
T Hiroki and M Minoru. Ascii art pattern recognition using svm based on morphological analysis. Technical report, Technical report of IEICE. PRMU 104 (670), 25-30 (20050218), 2005.
Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 3d-llm: Injecting the 3d world into large language models. Advances in Neural Information Processing Systems, 18:20482-22494, 2023.
Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bam- ford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral experts. arXiv preprint arXiv:2401.04088. 2024a.
Fengqing Jiang, Zhangchen Xu, Luyao Niu, Zhen Xiang, Bhaskar Ramasubramanian, Bo Li, and Radha Poovendran. Artprompt: Ascii brt-based jailbreak attacks against aligned llms. In Pro- ceedings of the 62nd Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 15157-15173, 2024b.
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.194377 2024a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023.
Yuliang Liu, Zhang Li, Mingxin Huang, Biao Yang. Wenwen Yu, Chumyuan Li, Xu-Cheng Yin, Cheng-Lin Liu, Lianwen Jin, and Xiang Bai. Ocrbench: on the hidden mystery of ocr in large multimodal models. Science China Information Sciences, 6711))220010,, 2024b.

---

Published as a conference paper at ICLR 2026
Kazuyuki Matsumoto, Akira Fujisawa, Minoru Yoshida, and Kenji Kita. Ascii art classification based on deep neural networks using image feature of characters. J. Softw, 13(10):559-55 2018.
Shunji Mori, Ching Suen, and Kazuhiko Yamamoto, Historical review ocr research and devel- opment. Proceedings of the IEEE, 90(7):1029-1058, 1992.
OpenAl. Gpt-4. OpenAl Blog, 2023. URL http:p/://ppeaa..coo/eeeeeaaeaee
Zeju Qiu, Weiyang Liu, Haiwen Feng, Zhen Liu, Tim Z. Xiao, Katherine M. Collins, Joshua B. Tenenbaum, Adrian Weller, Michael J. Black, and Bernhard Schölkopf. Can large language mod- els understand symbolic graphics programs? In The Thirleenth International Conference on Learning Representations, 2025a.
Zeju Qiu, Weiyang Liu, Haiwen Feng, Zhen Liu, Tim Z. Xiao, Katherine M Collins, Joshua B Tenenbaum, Adrian Weller, Michael J Black, and Bernhard Schölkopf. Can large language models understand symbolic graphies programs? In The Thirteenth International Conference Learning Representations, 2025b.
Machel Reid, Nikolay Savinov, Denis Teplyashin. Dmitry Lepikhin, Timothy Lillicrap, Jean- baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. Gem- ini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024.
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8317-8326, 2019.
Tetsuya Suzuki. Text normalization on the text art extraction method using data compression rate. In Proceeding of the 17th of The Annual Meeting of the Association for Natural Language Pro- cessing, 2011.
Gemma Team. Gemma. 2024a. doi: 10.37740KAAGGLEMM/333 URL https://www.
kaggle. wtt.cccmmmmmm.em m300
Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. Gemma 3 technical report. urXiv preprini arXiv:2503. 19786, 2025.
Qwen Team. Qwen2 technical report. arXiv preprint
2024b.
Oguzhan Topsakal and Jackson B Harper. Benchmarking large language model (Ilm) performance for game playing via tic-tac-toe. Electronics, 13(8):1532, 2024.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307. 092888, 2023.
Hong Wang, Xuan Luo, Weizhi Wang, Melody Yu, and Xifeng Yan. Bot or human? detecting chatGPT imposters with single question. In First Conference on Language Modeling, 2024a.
Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong. Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang. Lei Zhao, Xixuan Song, et al. Cogvlm: visual expert for pretrained language models. In Pro- ceedings the 38th International Conference Neural Information Processing Systems, pp- 221475-12118 121499, 2024b.
Wenshan Wu, Shaoguang Mao, Yadong Zhang, Yan Xia, Li Dong, Lei Cui, and Furu Wei. Mind's eye of Ums: visualization-ol thothtt elicits spatial reasoning large language models. Advances in Neural Information Processing Systems, 37:90277-90317, 2024.
Xuemiao Xu, Linling Zhang, and Tien-Tsin Wong. Structure- based ascii art. In ACM SIGGRAPH 2010 papers, pp. 1-10, 2010.
12

---

Published as a conference paper at ICLR 2026
Xuemiao Xu, Linyuan Zhong, Minshan Xie, Xueting Liu, Jing Qin, and Tien-Tsin Wong. Ascii art synthesis from natural photographs. IEEE Transactions on Visualization and Computer Graphics, 23(8):1910-1923, 2016.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint 099388 2025.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multi- modal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 9556-9567, 2024.
Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, et al. Internv13: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504. 10479, 2025a.
Junnan Zhu, Jingyi Wang, Bohan Yu, Xiaoyu Wu, Junbo Li, Lei Wang, and Nan Xu. Tableeval: A real-world benchmark for complex, multilingual, and nuuti-struuttured table question answering. XrXiv preprint (rX.:i2506.03499 2025b.
13

---

Published as a conference paper at ICLR 2026
A THE USE OF LARGE LANGUAGE MODELS
Large language models, including Gemini-2.5-pro and Gemini-2.5-Flash were adopted to polish the writing this work. All of the models' outputs are verified and further modified by the authors. LLMs did not participate in any other stages of the research process.
B DATA LICENSE
We express our gratitude to ASCII artists from online galleries whose fantastic creations underpin our research. In order to assess the visual perception abilities of models, we made slight modifica- tions the original ASCII for the set ASCIIEVAL, to avoid information leakage through text hints. Meanwhile, we retained the original ASCII art and the URL to the data source. We follows the term of use guidelines from the original websites [ and datasets Data will be released and licensed under CC BY NC 4.0, which permits only non-commercial use and is intended exclusively for research purposes.
C FUTURE DIRECTIONS
Based on the results and analysis, we discuss more future directions as follows:
Constructing high-quality training data automatically. We randomly selected 100 samples from ASCHTTUNE for the quality check and the human annotator achieved only 70% accuracy. This indi- cates that ASCIITUNE is much noisier than ASCIIEVAL (98.33%), pointing out the importance of collecting more training data with higher quality. On the one hand, utilizing the ASCII art synthesis tools to convert image datasets into ASCII can be considered enlarge the size of the training data, under the awareness of the style differences between the converted ones and the ones created by artists. On the other hand, more strict filtering strategies should be incorporated, such as verifying the validity of ASCII art with strong MLLMs under the Image-only setting.
Improving the model architecture. All the tested LLMs and MLLMs show the inability to recognize information that can be fully represented in text. One potential reason is the lack of exposure to this type of data. It may be also a result of the structural limitation of current models. As for human beings, perceive text from the aspects of character sequences and their visual shapes at the same time, while these two aspects are conventionally distinguished into two modalities when being processed by neural models. More flexible processing techniques and architecture among modalities should not only benefit the models' visual perception ability in text strings, but also make the model closer to human beings with more efficient information processing abilities.
Incorporating more complicated scenarios. Currently, we only considered the basic type of ASCII made up of 95 printable fixed-width ASCII characters. Nevertheless, there also exist more fasci- nating ASCII arts, such as color ASCII art, 3D ASCII art, animated ASCII art, etc. These different kinds of ASCII are also valuable for understanding LLMs designed for video understanding (He) et al.
2024a) and 3D modeling (Hong et al. 2023).
D PROMPT TEMPLATE
We adopted the following three prompt templates for different input modes:
Prompt Template for Text-only Input
Please answer the multi-choics question based on the given ASCII art
(ASCII ART]
ascii
htttss//asciiartt.webbbs websittewwbbttt https//ssseee/pppeeeeao.o...o.o. httpes//mgginnffc....aaaaattteee
14

---

Published as a conference paper at ICLR 2026
[Question]
What is depicted in the above ASCII art? fchoices)
Answer with the option's s letter from the given choices directly.
Prompt Template for Image-only Input
Please answer the multi-choice question based on the given ASCII art image.
[ASCII ART]
<image>
[Question]
What is depicted in the above ASCII art? {choices}
Answer with the option's letter from the given choices directly. Prompt Template for Image-text Inpul
Please answer the multi-choien question based on the given ASCII art in both image and text formats.
[ASCII ART Image] <image>
[ASCTT ART Text] ascii.ar
[Question]
What is depicted in the above ASCII art? {choices)
Answer with the option's s letter from the gi given choices directly.
All of the models except Qwen VI. are evaluated based on these prompt templates with minor mod- ifications to adapt their default settings, especially for the position of the image.
Qwen- VL more sensitive to prompt templates according our experiments. Therefore, we adapted the above templates into Qwen- VL's original format, which is "Context: Question: Answer:".
E DATA COLLECTION FOR ASCHITUNE
To further elicit models' visual perception ability, the creation of a training set is essential.
intuitive solution is to leverage previous works ASCII art synthesis (Xu et al. 2016 2010) by converting existing image datasets, such as ImageNet (Deng et al. 2009). A public dataset in- dicates that after automatic tone-based synthesis, approximately 85% data samples
filtered out
due to poor quality. Furthermore, existing data conversion tools are inadequate for structure- ASCII art, which accounts for 94% of the data according to annotators' labels in ASCIIEVAL. Artists also frequently combine both lone-based and structure-based features in a single artifact.
Therefore, chose collect the training set in manner similar ASCHEVALL instead of relying on automatic conversion. Data sources include ASCII arts from another less organized website [ and the crawled content extracted into individual ASCII art pieces based rules derived from observations. We also included the unrecognized ASCII art that was withdrawn during the construc- tion of ASCIIEVAL. The normalized ASCII art is discarded if recognized as repetitive with samples in ASCIIEVAL or among each other.
https://ascii.co uk/rtt
15

---

Published as a conference paper at ICLR 2026
Table 3:
The number of samples under each category.
Classes
Groups
animals & natural (1,122)
animal (870), plant (130), nature (122)
objects (777)
object (451). electromics (192), clothing (81). furmiture (53)
smileys & people (644)
role (199), character (195), body (146), occupation (68), people (36)
activitics (473)
event (207). sport (126), activity (84), instrument (35), monument (21)
travel & places (406)
transportation (123), building (123), places (30)
food & drink (66)
food (66)
symbols (38)
logo (27). astrology (11)
Table 4: Statistics of token length by different tokenizers.
ASCIIEVAL
ASCHTTUNE
Min
Max
Avg
Min
Max
Avg
Llama-3 Tokenizer
71
2,192
262.72
69
3,673
215.10
Mistral-vO.J 1 Tokenizer
85
2,890
332.91
83
4,294
267.93
Qwen-2 Tokenizer
80
2,833
278.17
78
3,996
273.40
Due to the large amount of data with diverse concepts, carefully categorizing data for high-qualig, distractors is unfeasible. Instead, we prompted Llama-3-70B. --n--Innttuct to generate negative choices given the ground truth concept and utilized the Perspective API to filter out unsafe samples based on the concatenation of candidate choices. Samples with scores less than 0.2 across all six dimensions, ie., toxicity, severe toxicity, identity attack, insult, profanity and threat, are retained.
F DATA ANALYSIS AND STATISTICS
During the data filtering process, we recognized that some of the ASCII art have multiple interpre- tations, which be summarized into two types:
The ASCII art itself, as a kind of art form, is abstract and ambiguous. For instance, certain depictions of cats might resemble rats. Regarding these cases, asked human annotators to remove such unrecognizable and ambiguous art.
The ASCII is rich content, potentially allowing two interpretations from different aspects. For example, the third ASCII art in Fig. 15 can be interpreted as a beach scene, coconut tree, sunset, etc. Most of the ASCII art in ASCIIEVAL VAL only contains a single object, and we also tried to remove such ambiguities by carefully designing and adjusting the classification criterion. Ultimately, there are only less than 1.67% ambiguous cases in ASCITÉVAL, leading to the imperfect performance of human annotators.
Finally, the number of samples and the hierarchical relationship between classes and groups of ASCHEVAL illustrated are shown in Table 3
The token length of samples under the Text-only mode tokenized by three representative tokenizers is in Table 4 The ASCII art data used in our experiments respects the context length limitation of nowadays models.
G DETAILS ABOUT EVALUATED MODELS
For open-source instructed models, experiment with the following LLMs and MLLMs:
LLMs. Llama (Touvron et al. 2023) contains three collections of generative models with dif- ferent sizes, including Llama-2 Llama-3, Llama-3.1, and Llama-3.3; Qwen (Bai et al.] 2023a Yang et al. 2025) is another group of models with instructed verions, including Owen, Qwenl.3, Qwen2, Qwen2.5 and Qwen3 series; Mistral (Jiang et al., 2024a) includes dif- models, i.c., Cestral-7B-Instructt.
Team 2024b
v0.2 and v0.3. Be- which are pre- tre-trained generative Sparse Mixture of Experts also compared; Gemma (Team] 2024a: Team al. 2025) is fam-
ferent versions of instruction
sides, Mixtral-8x7B Inntttttttttt-.O and -aaallxx22----ttteee
16

---

Published as a conference paper at ICLR 2026
ily of lightweight text-to-text models with instruction-tuned variants. We considered Gemma-2 and Gemma-3 series; DeepSeek Liu et al. (2024a) is a series of open-source large language models, with DeepSeek- V3 being notable model in this tamily.
2023) augmented pre-trained LLM with pre-trained vision en- coder. The vision model's representations are projected into the LLM's representation space with a projection layer, and it is frozen during instruction tuning while the projector and the backbone LLM are updated; CogVLM (Wang et al.) 2024b) aims at retaining the original capabilities the LLM while adding visual understanding abilities. Representations from the pre-trained vision transformer encoder are passed through an MLP adapter as the input, and a group of trainable visual expert modules in the attention and FFN layers are introduced into the LLM. All of the parameters except the ones from the original LLM are tuned; Qwen-VL (Bai et al. 2023b) proposed a position-aware vision-language adapter for compressing image features. The model is trained through three stages, i.e., pre-trate mre-training, multi-task pre- treiiiiinng and supervised fine-tume tuning: Qwen2.5-VI. (Bai et al.) 2025) introduce dynamic resolution processing ad excelling in moni-document parsing, Intern V VL3 con- solidates language pre-training and multi-modal alignment training into unified pre-training stage with interleaving multi-modal data.
MLLMs. Llava (Liu et al.
For proprietary models, the specific versions we evaluated are GPTT-4--22240806 (OpenAl 2023 GPT-5-20250807, -laude--pus44-2255551444 Gemini-1.5 1.5-rro (Reid al. 2024) and Gemini-2.5. pro Comanici et al. (2025).
H ASCIIEVAL LEADERBOARD
We summarize the above models performance ASCIIEVAL given different input modalities in T The statistics used for calculating correlations in Sec. 5_1 and Sec. 6.1 were collected by ex- and tracting scores of the overlapped models covered both in Table and corresponding leaderboards
I DATA SYNTHESIS AND TRAINING DETAILS
Recognizing the superior performance of state-of-the-art proprictary models, we devised a multi- step data synthesis pipeline:
Data Curation: We first employ high-performing open-source model to filter the ASCIITune.
This initial pass serves remove low-quality or ambiguous samples, yielding 8925 samples.
Rationale Generation: For each curated data point, we provide the teacher model, GPT-5, with both Trext and Timg- The model is prompted to first generate detailed analytical process, i.e. rationale, that describes its the reasoning process for recognition in rich of the interpretation of local ASCII art features, with the answer at the end of the output.
Fidelity Verification: Only 6309 instances where GPT-5's final answer is correct are retained.
An example of the output distilled from GPT-5 is shown Fig. 7 Different colors marks the corresponding string in output text and the ASCII image. The output analysis explains the details of the model's perception process reasonably, but also contains minor errors. "V_" is a piece of hallucinated text string which not included in the original ASCII art. Employing more rigorous filtering strategies to remove such mistakes for high-quality data collection will be considered the future.
The LLMs in Sec. 5.2 is finetuned on the distilled data for epoch on full parameters, with batch size equaling 16 and learning rate equaling 2e-5. The MLLMs trained in Sec. 6.2 adopted the same batch size and learnin rate. We did fine-tuning with full parameters for epoch and with LoRA for 2 epochs.
17

---

Published as a conference paper at ICLR 2026
Table 5: ASCIIEVAL Leaderboard. The scores are macro accuracy (%) averaged among different concepts. Average refers to the mean among the three input settings if available. All of the models are "instruct" or "chat" versions. The best and sub-optimal results each group of models are in bold and underlined. The tvalues mark the 95% confidence interval for concept-level ASCII art recognition.
Model
Text-only
Image-only
Text-Image
Average
Proprietary Models
GPT-5
55,90
87.81 +5.54
86.40
76.70
GPT-4o
43, 43,40
82.62.55.11
75.41
67.14
Gemini-2.5-pp 2.0-pr
50.65
83.07 1.113
61.6446..1
71.79
Gemini--15-ppp pro
33, 33.49
60. (1.69922 :6.00
58.3347 .s
50.84
Claude-opus
31.29+7.04
40.41
36.68412.16
36.13
Open-source source LLMs
DeepSeek-V3
35.94
35.94
Qwen3-8B
28.28
28.28
Qwen3-14B
30.79
30.79
Qwen3-32B
30.1847.64
30.18
Qwen2.5-78 B-BB
27.57
27.57
Qwen2.5-14p
29.1447.17
29.14
Qwen2.5-322
66.65
31.65
Qwen2.5-728
33.20
33.20
Qwen2-77 -B
26.71
27.71
Qwen2-72B
30.73
30.73
Qwenl .5-7B
26.71
26.71
Qwen 5-110B
30.28.77200
30.28
Qwen-7B
23.30
23.30
Gemma-3-44
27.34+7.30
27.34
Gemma-3-128
29. 29.29
29.29
Gemma-3-27B
35.65 7.66
35.65
Gemma-2-98 2-9B
30.50+7.37
30.50
Gemma-2-222
32.30 32.36
32.36
Llama-3.3-700
32.74
32.74
Llama-3.1-888
27.22
27.22
Llama-3.1
31.27=5.44
31.27
Llama-3. 1--05B
32.31+7.43
32.31
Llama-3-88 B-BB
28.71+8.00
28.71
1lama-3-7005
30.42++4.46
30.42
Llama 2-7B
24.59+7.61
24.59
Llama 2-13B
25.93
25.93
Llama-2-70B
28.08
28.08
Mistral-7B-vO.1
26.88
26.88
Mistral-7B-v0.2
27.28
26.28
Mistral-7B-v..3
25.57+7.79
25.57
Mixtral-&x7B-v0.1
25.31
25.31
Mixtral-8x22B-v0.1
28.20 28.20+6.51
28.20
Open-source MLLMs
Qwen2.5-VL-BB
25.05
34.83
37.01
32.30
DwC32.5-VL-32BB
29.8248.48
29.35
32.07 48.79
30.41
Owen2.5-VL-72B
34.20 47.02
36.4247..57
37.82 37.82
36.15
Qwen-Vi VL
24.79
52.32.88.0
40.09
39.07
Intern VL3-8B
27.30+7.39
32.7414..0
33.5816
31.21
Intern VL3--4B
25.9116.78
33.25
31.5011..6
30.22
Intern VL3338B
32.10+7.60
50.27 #7.68
47.28
43.22
Intern
33.55 4.886
48.33
48, 48.5449..2 22
43.37
CogVILM2-Llama3-199
21.73+6..88
67.80+6..1
66.68
53,07
CogVLM-17B
21.25
61.00
57.5846.79
46.61
Slava-vl.--mistra
25.89+6.63
60.72448.0 03
59.02+*.m 03
48.54
Llava-v1.6 1ioaavl...-viuua--11
26.03
59.70 49.70
56.55.77.6 6.66
47.43
Llava-v1.5579
24.66
62.18 02.18447.40
61.52
49.45
Llava-v1.5-13B
26.00
6.1877
60.70.7.00
49.52
Llava-v1.6-44B
28.62
65.66
61.33
51.87
18

---

Published as a conference paper at ICLR 2026
Thee string uses slashes dashes to outline an aireraf). The sequence forms long liselage with suggesingg windows or body striping. The segmments V and subsequent // /f create angled wings. The curved ' the front suggests rounded nose, and the "E immeediately afler resembles pnupellerspprmet The buck side shows "-! 1-"( inndicattnng the assembly with diagonal stabilizers. small near the bottom reads like tail skid/gear. Altogether these parts depicr an airplane.
Figure 7: An example of distilled data for rationale-assisted fine-tuning.
Table 6: The number of samples with ASCII arts divided by different characteristics.
#Characters
[1, 50]
(50,100)
(100, 200]
(200, 400]
(400, 800]
(800, 1600]
(1600, +00)
#Samples
221
366
546
710
760
618
305
#innss
[1,5]
(5. 10]
(10, 15]
(15, 20]
(20, 25]
(25,+
#Samples
414
854
699
534
399
626
90
100
GPT-
Geminn-22.-ppr
GPT-4o
70
50
70
GPT-5
Gemini-2.55pp
30
GPT40
[1,
(50. 100]
(100, 200]
(200. 400]
(400, 800]
(800, 1600]
(1600, oz
[1,50]
(50. 100]
(100, 2007
(200, 400]
(400, s00]
(800. 1600]
(1600, +00)
(a) Text-only
(b) Image-only
Figure 8: Micro accuracy( %) of models on recognizing ASCII arts with different numbers of char- acters.
J ANALYSIS ON SAMPLES UNDER DIFFERENT ASCII ART SIZES
Based the length characteristics of different ASCII art, we divided the test set into various subsets, as shown in T 6
The performances of models on testing samples grouped by the number of lines contained in the ASCII art are shown in Fig. g The trends are similar to those grouped by the number of characters in Sec i.e., models favor smaller ASCII under the Text-only setting while they prefer larger ASCII art under the Image-only setting. Besides, when an ASCII art exceeds 800 characters, the model's 's performance tends to plateau even degrade, underscoring that recognizing large-scale ASCII art also remains challenging for MI.LMs.
K SENSITIVITY TO MINOR CHARACTER CHANGES
We randomly removed tokens (other than spaces, "\n" and "(t") from ASCII art and manually checked if the result remained recognizable. Two representative examples are illustrated in Fig. In both cases, the ASCII art remains recognizable when only few characters are removed. However, the first ASCII art becomes progressively indistinguishable more characters missing. Meanwhile,
https://sgp benchhhhppbbeh..oommm hithh.. http:///hgggggge.eooooo/o//oooon
io/
19

---

Published as a conference paper at ICLR 2026
Original
Remove 5 chars
Remove 10 chars
Remove 20 chars
Remove 50 chars
(p)p)
(
g1
7.
= t
t
=
)
Y nn
Y
Y
nn
nn"nn
n
nn
Figure 9: An illustration of removing characters in the ASCII art. "chars" is short for "characters".
Table 7:
The micro-accuracy (%) at different perturbation ratios. "PR" is short for "Perturbation
Ratio".
PR
Human
Text-only
Image-only
Text-Image
1%
99
94
96
96
99
91
10%
97
91
93
92
20%
94
84
87
83
the second one just gradually has some additional noise and remains recognizable. This suggests that the number of characters increases, the importance of each character diminishes as it carries less visual information.
We did more quantitative analysis by sampling 100 cases from ASCIIEVAL, among which Llava- v v1.6-34B provided correct answers under all three test settings. Next, we randomly replaced 1 1%, 5%, 10%, and 20% of tokens (other than spaces, "(n" and ((("") the original ASCII with spaces.
The computed micro-accuracy of lluvavvl.63344 under different test settings, as well as the human upper bound, are shown Table Changing the characters in ASCII art will make the recognition task more challenging both for humans and the model, while Human is relatively more robust than Llava-v1.66 1.6-44 under different settings.
L SENSITIVITY WITH DIFFERENT FONTS
In this work, we only considered the traditional ASCII art composed of 95 printable fixed-width ASCII characters. The semantic meaning remains unchanged long as is displayed with fixed- width font. In addition to the VDejaVu Sans Mono" font used in this work, examples of the same ASCII rendered with 4 different fonts are shown in Fig. TO All of the dogs are recognizable, with only minor differences. In other words, the multiple-choice questions for ASCII art recognition in ASCHIEVAL remain valid, regardless of the specific wiith font used.
Although humans have difficulty recognizing ASCII art rendered with different fonts, this raises the question of whether MLLMs are sensitive to these variations and show a preference to a specific font. We take lava-v11.6334.
example and evaluated its performance art under both Image-only and Text-Iman settings where the images are rendered using different fonts mentioned in Fig. 10 It should be noted that the textual ASCII art is unaffected by font
ASCII
20

---

Published as a conference paper at ICLR 2026
----\
VujaVu Sans Mono
Cascadia Code
Comic Mono
Courier
Fantasque Sans
Figure 10: An illustration of an ASCII art displayed different fixed-width fonts.
Table 8: Macro-accuracye of Llava-v1.6-22 under Image-only and Text-Image setting with ASCII art rendered by different fix-width fonts.
Mode
Vu Sans Mono
Cascadia Code
Comic Mono
Courier
Fantasque Sans
Image-only Text-Image
65.66
63.41
66.68 62.11
63.84 59.89
66.73
61.33
59.85
64.04
① activities
natural food drink (4) objects smileys people symbols (Z) trivel & places
80
80
80
80
60
60
60
12-3300 4 (7)
3 55
2 4 66
1 3
2
2
(1) GPT-5
(2) 2empin-2.5-pro
(3) GPT-4o
(4) DeepSock-V3
(5 (5XGemmaa-32278
(a) Text-only
100
100
100
SO
80
60
60
60
60
40
40
40
40
1/2 2 (Z)
1 2 3 4-
1 33 8
2 2 66
(1) GPT-5
(2)Gemini-2.5ppo
(3) GPT-4o
(4) Cog VLM-Llam--
(5) L.ava-v1.66344
(b) Image-only
Figure 11: Macro-accuracy (%) of models on recognizing ASCII arts under different classes.
variations, and Llava-v1.6-34BB performance under the Text-only setting is identical to the result in 5
According the results in Table X MILMs do face challenges in performing robustly among dif- ferent text fonts in ASCII art recognition and the performance varies. Nevertheless, its best per- formance in this table with 66.73% and 64.04% still lags far behind that of GPT-4o with 83.69% and 76.52% under both settings respectively. Moreover, the accuracy under the Text-Image setting consistently lower than that under the Image-only setting. These observations are same as the results in Sec. 7
On the one hand, how reduce this sensitivity and improve the MLLMs' robustness is important and worth further exploration. On the other hand, changing the fonts in rendered ASCII art can po- tentially useful data augmentation technique for boosting MILMS* performance on ASCIIEVAL.
M How DO MODELS PERFORM ON DIFFERENT CATEGORIES?
Models' performances across the 7 different classes are shown in Fig. II Models given text input perform better at recognizing ASCII arts belonging to the "objects" class. Models under Image-only mode show consistent improvement in recognizing "travel places" over Text-only mode compared to other classes relatively. Moreover, all models struggle with ASCII art referring "symbols", which comprise different logos and astrology symbols. MLLMs actually perform quite well at rec- ognizing well-known logos, such as Apple and Linux, where GPT-5 achieves 100% macro-accuracy
21

---

Published as a conference paper at ICLR 2026
Models
[0,0.1]
(0.1.0.2)
(0.2,0.3)
(0.3,0.4)
(0.4.0.5)
DeepSeek-V? V3
7
220
834
1249
908
Gemma-3-27B
2
171
812
1158
913
Qwen2.5-72B
10
242
923
1286
839
Models
(0.3,0.6)
(0.6.0.7)
(0.7.0.8)
(0.8.0.9)
(0.9,1.0]
DeepSeek-V3 V3
275
27
6
0
0
Gemma-3-27B
361
87
15
6
1
Qwen2.5-72B
191
26
6
3
Table 9: Number of samples in different compression ratio buckets.
and CogVL.M2-Llama--Chan 19B gets 91.16%. However, their performance drops dramatically on relatively niche astrology symbols. Nevertheless, it is simple for both LLMs and MLLMs to an- swer the question "Can you show me some astrology symbols?". Existing models tend to rare Unicode characters or emojis to explain the symbols, but cannot understand the visual semantics embedded in those symbols flexibly.
The models' performance under different groups is shown in Fig. I2 Overall, the performance of models under Image-only mode is more balanced across different categories, except for the drops in "astrology" and "character". Meanwhile, accuracy given images fluctuates among different groups, with "electronics", "monumnet" and "object" topping the rank.
N DOES TOKENIZER COMPRESSION RATIO AFFECT MODEL PERFORMANCE:
This section explores whether the compression ratio of model's tokenizer correlates with its accu- racy on ASCII art recognition. We define the compression ratio for each ASCII art as the number of tokens generated by a model's s tokenizer divided by the total number of characters in the string. A lower ratio indicates higher compression. To analyze this relationship, we group the test samples into ten buckets based on their compression ratio for top-3 open-source LLMs in Fig 2(a)
The results presented in Figure 13 reveals no single, consistent trend across all evaluated models. While two models, DeepSeek- V3 and Gemma-3-27B, exhibit U-shaped performance curve, this pattern is not universal. @wwen2.5-72B does not follow this trend. Furthermore, according to Table the apparent high performance the extreme ends of the ratio spectrum is based on sparse data set, making the results less reliable and prone to variance.
Given the lack of a unified performance trend across all models and the sparse data distribution at the extremes of the compression ratio, we conclude that there is no clear correlation between the tokenizer compression ratio and model performance on this task.
O CASE STUDY ON MODELS WITH A STRONG OCR ABILITY
MLLMs with strong OCR capabilities often struggle with ASCII art as shown in Sec. They tend to prioritize recognizing individual characters over perceiving the holistic image formed by them. An example is shown Fig. 14 The model Siven2.5-L-775 incorrectly classified an ASCII art of mouse as butterfly.
To understand this failure, we prompted the model with the question, "What depicted in the above ASCII art?" Its response reveals complete focus on character-level details instead of the overall shape:
The image appears to be stylized or abstract representation of mathematical or logical expression. It includes symbols and characters that resemble logical oper- ators and variables, such as "q", "p". "p","" "t", "t", and "nn". The symbols are arranged in a way that suggests a logical or mathematical formula, but the exact meaning context is not clear without additional information. The use of slashes, dots. and other symbols might be part of a specific notation or a creative interpretation of logical expressions.
22

---

Published as a conference paper at ICLR 2026
10
70
50
50
30
10
(1) GPT-5
(2) Gemini-2.5-pro
70
70
0
50
00
30
10
10
mi
(3) GPG.44
(4) DeepSeck-V3
70
70
50
50
0
30
10
10
Aid
(5) Gemma-3-27B
(6) Avemge
(a) Text-only
100
1100
80
180
100
400
20
200
Son
201
Aart Ale
mu
0 GPTS
(2) Gemini-2255.
100
100
80
800
60
00
all
20
20
Conurnan L
20 2000
bo
COCP (3)
SPT-46
(4) CogVLM-LLamn32119
1000
80
60
60
40
20
20
(5) 1.vavv
(6) Average
(b) Image-only
Figure 12: Micro accuracy(SR) (%) of models on recognizing ASCII arts in different groups Average is calculated as the mean of the top 5 models.
23

---

Published as a conference paper at ICLR 2026
100
40
Ve
o G-mma-3-770
Qwan25-7228
j0, (..
03]
(0.8..)
(..44)
(04 0.5)
(0.5. an]
(06.07)
(33,03)
69)
Figure 13: Micro-accuracy($) of LLMs performance under different compression ration with cor-
responding tokenizers.
nn-nn
nn
(-q_Tp)
What is depicted in the above ASCII art?
=-t/= ".nn.
A. monkey
B. fish
C. mouse
D. butterfly
nn-nn) ("nn- -nn)
(a) The original case from ASCIIEVAL.
(b) ASCIT art repeated by Qwen2.55-VI-777
Figure 14: Case study for Owen2.5-VL-722. The choices in red and in blue refer to the correct answer and the incorrect candidate chosen by the model, respectively.
This reply confirms that the model is not seeing mouse at all. Instead, its OCR abilities lead it to interpret the art as a "mathematical or logical expression," a clear failure holistic visual reasoning.
Furthermore, when asked the model simply list the characters it perceived, it successfully tracted many of them. However, their spatial arrangement in the output was completely disordered. This reinforces the conclusion that while the model excels identifying characters, it challenging to process their positional relationships.
P CASE STUDIES
We selected seven samples belonging to different classes from ASCIIEVAL and show the cases in Fig. 15 and Fig. 16
The correct answers marked in red.
24

---

Published as a conference paper at ICLR 2026
animals & natural animal + panda
objects electronics light bulb
Please onswer the multi-choice question based on given ASCII art: [ASCII ART]
Please answer the multi-chaice question based on given ASCII art: [ASCII ART]
[Question]
[Question]
What is depicted the abave A ASCII art? A, caterpillor B, panda
What is depicted the above A ASCII art? A. light bulb
8, CD
C. ent
lizard
C. mouse
phone Answer with the option's letter from the given choices directly.
Answer with the option's letter from the given choices directly.
travel places nature beach
food drink food ice
Please the multi-choice question based the given ASCII art: [ASCII ART]
Please the malti-choice question based the given ASCII ort: (ASCII ART]
[Question]
[Question]
depicted A, stor
art?
What depicted above B. strawberry
B. lightning
A. Gingerbread man
beach
D. sun
C, pie Answer with the option's 's letter from the given directly.
cream
Answer with the option's 's letter from the given directly.
Figure 15: Case studies (Part I).
25

---

Published as a conference paper at ICLR 2026
symbols astrology Aries
activity activity dancing
Please onswer multi-choice question based on given ASCII art: [ASCII ART]
Please onswer the multi-choiie question based on given ASCII art: [ASCII ART]
0
[Question]
[Question]
What depicted in the above ASCII ar1?
What is depicted the abeve ASCII art?
Aries
B. Cancer
A. ice-skating B. dancing
Leo
Pisces
C. c.mmpinggg rodeo
Answer with the option's letter from the given choices directly.
Answer with the option's 's letter from the given choices directly.
smileys people body hand
Please enswer multi-choice question based the given ASCII ort:
[ASCII ART]
[Question]
What depicted in the above ASCII art? skeleton
hand face foot Answer with the option's letter from the given choices directly.
Figure 16: Case studies (Part II).
26


Reviews:
[{"review_id": "DY6TgpqI7e", "reviewer": "Reviewer_p7db", "summary": "This work introduces a cross‑model benchmark for evaluating the ability of LLMs and multimodal models to interpret ASCII art presented in multiple formats. The authors curate approximately 3K samples covering 359 concepts across seven broad classes (e.g., Animals, Symbols), organized in a three‑level hierarchy, and evaluate a range of state‑of‑the‑art models, highlighting improvements in more recent systems. Alongside ASCIIEval, the authors release a larger but noisier training set, ASCIITune (~11.8K samples). They evaluate large number of models (both pure‑text LLMs and multimodal LLMs (MLLMs)) under three settings: text‑only input, image‑only input, and text‑plus‑image (to probe multimodal fusion).", "questions": "My questions to the authors:\n\n1. Have you analyzed correlations with broader vision-language or reasoning benchmarks to support its general relevance?\n2. Did you consider including an open-ended version of the task to test true recognition rather than choice elimination?\n3. Could you share how models performed on particularly ambiguous or confusable ASCII samples?\n4. Have you analyzed in detail (apart from Table 4), whether tokenizer compression ratio (e.g. number of tokens vs characters per ASCII sample) correlates with model performance?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The main strengths of the paper are as follows:\n\n1. A novel cross-modal benchmark that covers still underexpored domain (visual pattern recognition within text) is introduced. Prior research has heavily focused on reading text in images (OCR) or traditional image understanding, while ASCIIEval utilizes ASCII art as a modality-agnostic bridge between text and vision.\n2. The authors created the benchmark under the comprehensive methodology, e.g., 3-layer category hierarchy of the tasks that enables us to estimate the models performance in a more detailed manner. As reported, the resulting test set covers 359 distinct concepts (from pandas to light bulbs to “Aries” zodiac symbol) across 23 groups and 7 top-level classes. \n3. Comprehensive evaluation and analysis is remarkably thorough. The authors evaluate a wide range of models, including both closed-source models (GPT-4, GPT-5, Claude 3, Google Gemini, etc.) and open-source LLMs (LLaMA, Qwen, Mistral, etc.), and also open MLLMs. They report results in three conditions – text-only, image-only, and combined – allowing for insightful comparisons. The analysis surfaces several important findings backed by data: for instance, proprietary models outperform open models by a large margin.\n4. Interesting insight about trade-off of OCR training and ASCII comprehension is revealed.\n5. Despite the benchmarks and evaluation, the training recipes and ablations are conducted by the authors to improve models' ASCII art recognition.", "weaknesses": "The main weaknesses of the paper are as follows:\n\n1. While ASCII benchmark is interesting itself, it is still a little bit narrow. From the motivation of the benchmark creation, it is a bit unclear. what general insights ASCII art evaluation provides beyond this specific format. For example, how well this translates to broader model capabilities. Stronger correlations with general benchmarks would help clarify relevance. While, some notes about trade-off between the OCR performance and ASCII performance is visible, it would be interesting to evaluate the mistakes provided by the models with strong OCR abilities on the ASCII-based tasks.\n2. Training data quality is low. ASCIITune is noisy (70% human accuracy), which may limit model learning. Also, using Llama-3 to generate distractors could introduce bias or unintended cues.\n3. While I don't consider it as a main weakness, still, the multiple-choice format makes evaluation easier but may oversimplify the task. Adding a generative variant could better reflect real understanding.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Nq6PGkdqu1", "reviewer": "Reviewer_Mraa", "summary": "The  submission introduces ASCIIEval, a multiple-choice benchmark for recognizing visual concepts depicted in ASCII art across text-only, image-only, and text-image settings, comprising 3526 test samples over 359 concepts (organized into 7 classes and 23 groups) plus a larger training set (ASCIITune); evaluating 50+ models from 2023–2025, the authors analyze scaling and generational trends, sensitivity to length, fonts, and perturbations, and propose rationale-assisted fine-tuning for LLMs (via GPT-5-distilled rationales) and two post-hoc MLLM improvements—low-resolution prompting and supervised fine-tuning—finding that proprietary models dominate, open-source MLLMs face an OCR-versus-holistic perception trade-off, text favors short ASCII while images favor long, text+image often degrades performance relative to image-only, and the proposed methods yield sizable accuracy gains.", "questions": "(1) Dataset reliability:\n- What is the inter-annotator agreement (e.g., Cohen’s kappa) on concept labels and recognizability decisions? Can you share confusion matrices for top-confused concepts/groups?\n\n- How often do distractors share strong lexical overlap with ground truth (e.g., “cat” vs “kitten”)? Any adversarial distractor stress tests?\n\n(2) Prompt robustness:\n\n- Did you evaluate with multiple prompt templates per model and report mean/variance? Especially for models known to be prompt-sensitive (Qwen-VL)?\n- Any experiments with calibrated decoding (e.g., constrained output to options, logit-based choice selection) to reduce format errors?\n\n(3) OCR trade-off:\n\n- Can you show controlled training where you modulate OCR-heavy data vs ASCII-like data to quantify the trade-off? Are there models that buck the negative correlation trend?\n- In low-resolution prompting, do you measure OCR performance drop on OCRBench/TextVQA to confirm the intended trade?\n\n(4) Fusion failure:\n\n- What fusion architectures were used by the tested MLLMs (early/late)? Any evidence (attentions, gradient norms) that text tokens overshadow image features or vice versa?\n- Have you tried modality dropout or gating at inference (e.g., confidence-based selection) to approach the “oracle” bound?\n\n(5) Rationale distillation:\n- How sensitive are LLM gains to teacher choice (e.g., Gemini vs GPT-5) and rationale quality filters? Any comparisons across teachers or rationale lengths?\n\n- Do rationales generalize beyond seen concepts or primarily boost memorization of local patterns?\n\n(6) Tokenization alternatives:\n\n- Did you experiment with preserving 2D structure via newline-aware 2D positional encodings, blockwise tokenization, or line-by-line embeddings? Even small proof-of-concept results would be valuable.\n\n(7) Safety and misuse:\n\n- Since ASCII can be used to bypass safety (as you cite), did you measure whether improved perception correlates with better or worse adherence to safety policies in ASCII-mediated harmful content?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(1) Addresses an underexplored capability: visual perception in text strings; ASCII art is a strong, modality-agnostic testbed.\n\n(2) Carefully curated benchmark with taxonomy, safety filtering, human upper bound, and objective multiple-choice evaluation.\n\n(3) Broad, current evaluation with clear, actionable insights (OCR vs holistic trade-off; length effects; fusion failure) and simple, effective mitigations (low-res prompting; vision-backbone finetuning; rationale distillation).", "weaknesses": "(1) Ambiguity and label integrity: Although human filtering was applied, the paper acknowledges remaining ambiguity (<1.67%) and reports a relatively low accuracy (70%) for spot-checks in ASCIITune. More rigorous inter-annotator agreement (IAA), label adjudication protocols, and confusion analyses across similar concepts would strengthen trust in labels and distractors.\n\n(2) Potential source bias and reuse: The dataset draws heavily from online galleries with human-made ASCII, which is valuable but may contain stylistic and cultural biases. The paper could better quantify coverage (e.g., long-tail concepts, regional styles) and discuss domain shift to auto-generated ASCII or box diagrams.\n\n(3) Evaluation scope vs prompting sensitivity: The authors note that some models (e.g., Qwen-VL) are prompt-sensitive and adjust templates. It raises concerns about fairness across models. A more systematic prompt ablation or standardized multi-prompt evaluation could ensure robustness of the leaderboard.\n\n(4) Multiple-choice design and distractors: Distractors are within-group, but for ASCIITune they are LLM-generated (Llama-3-70B) and filtered by Perspective API. This may introduce distributional artifacts or lexical cues. Quantitative analysis of distractor hardness (e.g., human error with easy vs hard distractors, option entropy) is limited.\n\n(5) Correlation claims and causality: The negative correlation with OCR benchmarks is interesting but non-causal. Additional controlled experiments (e.g., training on OCR-heavy data vs balanced data; freezing vs finetuning OCR components) would better substantiate the trade-off claim.\n\n(6) Limited architectural exploration: The discussion rightly points to tokenization and 2D structure loss in text as a bottleneck, but no experiments probe alternative tokenizers or spatialized embeddings (e.g., 2D position encodings over monospace grids, rasterization-aware tokenization, or character-graph inputs). Even small-scale prototypes would strengthen the argument.\n\n(7) Text-image fusion analysis: The lack of synergy is a key finding, but the paper doesn’t deeply analyze where fusion fails (early vs late fusion, attention saliency across modalities, conflict diagnostics). Ablations with different fusion strategies or controlled noise in one modality would clarify the interference mechanism.\n\n(8) Reproducibility details: While appendices include prompts and some training details, more exact hyperparameters, seeds, and compute budgets for fine-tuning would help reproducibility. It’s unclear how many runs or confidence intervals were computed for key results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "B00OO1DGZB", "reviewer": "Reviewer_osj6", "summary": "This paper introduces ASCIIEval, a novel benchmark designed to systematically evaluate the visual perception capabilities of both Large Language Models and Multimodal Large Language Models. The core premise is that ASCII art represents a unique, modality-agnostic artifact where visual semantics are embedded within the 2D arrangement of text characters. The authors formulate the task as a multiple-choice question-answering problem for concept recognition. The benchmark is comprehensive, featuring a manually curated test set of over 3.5K samples across a detailed categorization tree, alongside a larger training set", "questions": "See Weakness", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.The paper is well written and easy to follow.\n2.The authors do extensice experiments and test a wide range of different LLMs and MLLMs.\n3.The problem of ASCII art for LLMs and MLLMs is interesting.\n4.The work is solid. It not only constructs a high-quality test set (ASCIIEval) with a multi-layer categorization system but also provides a training set (ASCIITune) for enhancing model capabilities. The exhaustive evaluation of over 50 mainstream models across three modalities makes its conclusions highly credible and representative of the current state of the art.", "weaknesses": "**1.limited techincal contribution**：While the research question is intriguing, the paper's technical contribution remains relatively modest. The proposed Rationale-Assisted Training approach essentially leverages GPT to construct chain-of-thought data, which can be viewed as a form of capability distillation from a more powerful model.\n**2.Failure to Fully Explore Model Robustness**: The paper only briefly touches upon font sensitivity and character perturbation analysis, which should have been a more critical evaluation dimension. For visual perception, robustness to variations in artistic style, character substitution, and local occlusions is a key metric. The lack of systematic robustness testing of this kind is a shortcoming of the evaluation framework.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "y9BLVxDjK1", "reviewer": "Reviewer_47dC", "summary": "This paper focuses on evaluating the model's ability to understand ASCII art. To systematically assess this capability, the authors introduce ASCII-Eval, a benchmark comprising over 3,000 samples. Their comprehensive analysis reveals that proprietary LLMs such as GPT-5 demonstrate strong performance when processing ASCII art as text. The open-source MLLMs face a trade-off between text recognition and visual perception. Additionally, they find that model performance is sensitive to the length of the ASCII art, with the degree of sensitivity varying across input modalities.", "questions": "See the weaknesses.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The idea of evaluating the performance of LLMs and MLLMs on ASCII art is interesting.\n\n2. After proposing this benchmark, the authors also provided directions for enhancing the model's performance on it. (Low-resolution Prompting and Supervised Fine-tuning)", "weaknesses": "1. The authors claim that understanding how well models can capture visual semantics in text strings is valuable for both academic research and practical applications, yet they do not provide specific explanations. For example, what exactly is its value for academic research? And how does it benefit practical applications?\n\n2. The paper would benefit from a more in-depth analysis of the experimental results, rather than merely listing them. For instance, while Qwen2.5-VL-72B outperforms Llava-v1.5-13B on most benchmarks, the latter scores higher on the benchmark proposed in this paper. Why does this reversal occur? Additionally, why is the performance under the \"Image-only\" settings better than under the \"Text-Image\" settings?\n\n3. The observation that Qwen2.5-VL-72B (the latest version of Qwen-VL-Chat) performs worse than Qwen-VL-Chat on this benchmark raises questions about the benchmark's generalizability and reliability in evaluating model capabilities.\n\n4. Typo: In Figure 4, the authors evaluated Qwen2.5-VL-7B, while in Section 6.2 it is referred to as Qwen2.5-VL-8B.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)