Texture or Semantics? Vision-Language Models Get Lost in Font Recognition
Authors: Zhecheng Li
Guoxian Song
Yujun Cai
Zhen Xiong
Junsong Yuan
Yiwei Wang
Venue: COLM 2025 (2025)
Keywords: vision language models
font recognition
texture or semantics
stroop effect
TL;DR: We introduce a special two-level benchmark to assess VLMs’ font recognition abilities. The results indicate that VLMs perform poorly on font recognition tasks and are easily influenced by visual cues rather than semantic understanding.

Abstract: Modern Vision-Language Models (VLMs) exhibit remarkable visual and linguistic capabilities, achieving impressive performance in various tasks such as image recognition and object localization. However, their effectiveness in fine-grained tasks remains an open question. In everyday scenarios, individuals encountering design materials, such as magazines, typography tutorials, research papers, or branding content, may wish to identify aesthetically pleasing fonts used in the text. Given their multimodal capabilities and free accessibility, many VLMs are often considered potential tools for font recognition. This raises a fundamental question: Do VLMs truly possess the capability to recognize fonts? To investigate this, we introduce the Font Recognition Benchmark (FRB), a compact and well-structured dataset comprising 15 commonly used fonts. FRB includes two versions: (i) an easy version, where 10 sentences are rendered in different fonts, and (ii) a hard version, where each text sample consists of the names of the 15 fonts themselves, introducing a stroop effect that challenges model perception. Through extensive evaluation of various VLMs on font recognition tasks, we arrive at the following key findings: (i) Current VLMs exhibit limited font recognition capabilities, with many state-of-the-art models failing to achieve satisfactory performance and being easily affected by the stroop effect introduced by textual information. (ii) Few-shot learning and Chain-of-Thought (CoT) prompting provide minimal benefits in improving font recognition accuracy across different VLMs. (iii) Attention analysis sheds light on the inherent limitations of VLMs in capturing semantic features.

Published as a conference paper at COLM 2025
Texture or Semantics? Vision-Language Models Get Lost in Font Recognition
Zhecheng Lit Guoxian Songl
Yujun Cai
Zhen Xiong
Junsong Yuan* Yiwei Wang
+ University of California, San Diego
ByteDance
$ The University of Queensland Uniiersity of Southern California
University at Buffalo & University of California, Merced
zh1 zh11868ucsd. edu https://gi thub. ctw wwtt///iihecceegg////MM0o..
Abstract
Modern Vision-Language Models (VLMs) exhibit remarkable visual and linguistic capabilities, achieving impressive performance in various tasks such as image recognition and object localization. However, their effective- ness in fine-grained tasks remains an open question. In everyday scenarios, individuals encountering design materials, such as magazines, typography tutorials, research papers, or branding content, may wish to identify aesthet- ically pleasing fonts used in the text. Given their multimodal capabilities and free accessibility, many VLMs are often considered potential tools for font recognition. This raises fundamental question: Do VLMs truly pos- sess the capability to recognize fonts? To investigate this, we introduce the Font Recognition Benchmark (FRB), a compact and well-structured dataset comprising 15 commonly used fonts. FRB includes two versions: (i) an easy version, where 10 sentences are rendered in different fonts, and (ii) a hard version, where each text sample consists of the names of the 15 fonts themselves, introducing stroop effect that challenges model perception. Through extensive evaluation of various VLMs font recognition tasks, we arrive at the following key findings: (i) Current VLMs exhibit limited font recognition capabilities, with many state-of-the-art models failing achieve satisfactory performance and being easily affected by the stroop effect introduced by textual information. (ii) Few-shot learning and Chain- of-Thought (CoTT) prompting provide minimal benefits in improving font recognition accuracy across different VLMs. (iii) Attention analysis sheds light on the inherent limitations of VLMs in capturing semantic features.
1 Introduction
Font recognition is fundamental visual task with applications across various domains. It is particularly relevant to graphic design, publishing, advertising, and digital content creation, where typography plays crucial role in visual communication (Zramdini & Ingold, 1998; Chen et al., 2014b; Zhu et al., 2001). For example, researchers may seek to identify aesthetically appealing fonts in academic papers or images, while designers sometimes encounter unique typefaces they wish to integrate into their projects. Statistics indicate that over 60% of websites utilize custom or non-system fonts, highlighting the significance of font identification in web design. Additionally, Google Fonts are used on more than 50 million websites, demonstrating the extensive adoption of diverse typefaces (Solomons, 2023). This widespread demand underscores the need for efficient and accurate font recognition tools.
Traditionally, Convolutional Neural Networks (CNNs) have been the predominant ar- chitecture for font recognition (Chen et al., 2021; Li et al., 2022b). While CNNs achieve high accuracy, they rely on large and well-annotated datasets for training (Wang & Zong, 2023; Zhang, 2023; Tonmoy et al., 2024). The process of data collection, annotation, and
1

---

Published as a conference paper at COLM 2025
curation is time-consuming and resource- intensive, limiting the scalability and adapt- ability of CNN-based systems in real-world applications. In contrast, VLMs, pre-trained large-scale multimodal datasets, have demonstrated remarkable capabilities in processing both visual and textual informa- tion (Zhang et al., 2024c; Bordes et al., 2024; OpenAI al., 2024b; OpenAI, 2024; Team et al., 2024; Google, 2025). Their ability to generalize across diverse visual tasks with- out extensive task-specific training makes them highly versatile. Additionally, many VLMs are open-weight and freely accessi- ble, positioning them as practical alterna- tives for everyday font recognition.
what font is used in the text of the inage?
ARIaIL
GroundTrutk:
The font used for the text is Monoton.
VLM's Response:
The font used for the text is Arial, which is exactly
content inage. x
Human Response:
The font used far the text of the image is definitely not Arial, it should be Monoton.
Although VLMs have the potential to aid in font recognition, there has been compre- hensive analysis or systematic evaluation of their capabilities in this domain. In this pa- per, we introduce a font recognition bench- mark comprising 15 commonly used fonts, with images containing textual content ren- dered in these typefaces. To ensure rig- orous evaluation, we construct two dataset versions: an easy version, where fonts are recognized from images containing sentences, and hard version, which introduces the stroop effect (Besner et al., 1997; MacLeod, 1991; Verhaeghen & De Meersman, 1998) by displaying font names in mismatched typefaces, adding an additional layer of difficulty. This benchmark provides systematic assessment of state-of- state-of-the-art VLMs' font recognition abilities.
Figure 1: An illustrative example where the image displays the word 'Arial' rendered in the 'Monoton' font. The vision-language model incorrectly predicts the font as 'Arial, indicating over-reliance textual content and limited sensitivity to font semantics.
We conduct extensive evaluations 13 open-weight and closed-source VLMs. All tested models are struggle to differentiate between font types and exhibit high sensitivity to the textual content within images, illustrated in Figure 1. The hest-performing model achieves only around 30% accuracy on the easy version of the benchmark, while performance on the hard version even more challenging, with accuracy dropping approximately 15%. Additionally, chain-of-thought (Col) prompting (Wei et al., 2023; Kojima et al., 2023) provides limited performance gains in this task. Furthermore, we incorporate single- letter images as demonstrations in few-shot learning scenarios to help VLMs identify font similarities. Interestingly, despite the widespread familiarity these fonts and the relatively low difficulty of the task for humans, where few-shot examples significantly aid in font recognition, current VLMs continue exhibit poor performance.
In summary, our contributions are threefold:
(i) We propose a font recognition benchmark with two versions to systematically evaluate the capabilities of VLMs and conduct a comprehensive assessment of 13 different models.
(ii) Our results reveal that current VLMs perform poorly in font recognition, with neither Chain-of-Thought prompting nor few-shot learning providing significant improvements.
(iii) We analyze attention matrices across image patches to gain deeper insights into their failure modes and susceptibility to texture-based perturbations.
2 Related Works
2.1 Vision-Language Models
With the foundation laid by Transformer and CLIP (Vaswani et al., 2023; Radford et al., 2021), Vision-Language Models (VLMs) have emerged as powerful multimodal systems that
2

---

Published as a conference paper at COLM 2025
seamlessly integrate both visual and linguistic modalities. These models bridges the gap between image and text understanding, enabling them to process and interpret complex multimodal data efficiently (OpenAI et al., 2024a; Bai et al., 2025; Team et al., 2024; Li et al., 2024; Liu et al., 2023; Caffagni et al., 2024; Zhang et al., 2024a; Yin et al., 2024; Team et al., 2025; Chen et al., 2024).
Recent advancements in large-scale image-text datasets and contrastive learning have significantly improved VLM performance. These models now exhibit strong general rep- resentation capabilities, effectively aligning visual and textual information and achieving state-of-the--a of-tth--art results across multiple benchmarks, establishing themselves as key technolo- gies in multimodal AI research (Zhang et al., 2024c; Bordes et al., 2024; Schuhmann et al., 2022; Li al., 2023; Chen et al., 2025; 2020).
The strong multimodal capabilities of VLMs make them valuable tools in real-world ap- plications. Models such as BLIP (Li et al., 2022a), Flamingo (Alayrac et al., 2022) and later GPT-40 (OpenAl et al., 2024b) exhibit impressive performance in visual question answer- ing, image captioning, and multimodal dialogue, enabling applications in interactive Al agents and assistive technologies for visually impaired individuals (Agrawal et al., 2016; Zhang et al., 2024b; Guo et al., 2024; Gao et al., 2024; Zhai et al., 2024). These advancements underscore the versatility of VLMs in handling complex multimodal tasks.
2.2 Font Recognition
Font recognition, a critical task for people such as researchers and designers, inherently combines visual and textual components. However, it has received limited attention within the Vision-Language Models (VLMs) research community. Traditionally, has been ap- proached a text-image classification problem, predominantly relying on CNN-based methods (Wang et al., 2015b; Tensmeyer et al., 2017; Qi et al., 2025; Wang & Zong, 2023; Mohammadian et al., 2022; Cui & Inoue, 2021). While these approaches achieve relatively high accuracy with their proposed CNN-based architectures on specific datasets, they typi- cally require extensive fine-tuning on large-scale datasets. This requirement makes them time-consuming and also limits their generalization capabilities.
VLMs have recently emerged versatile tools in visual tasks, offering strong adaptability through large-scale pretraining. They present a promising, convenient solution for font recognition. However, their capabilities in this task remain underexplored, motivating our study to provide systematic evaluation and identify potential limitations.
3 Font Recognition Benchmark
We propose the Font Recognition Benchmark (FRB) to assess VLMs' ability to recognize typographic styles from images. Unlike different versions or subsets of the VFR dataset, which contains more than 2,400 font classes and further categorizes the same font style into regular, bold, and other sub-types, the VFR dataset more suitable for CNN-based model training rather than for basic capability exploration of VLMs in line with our ob- jectives (Wang et al., 2015a; Chen et al., 2014a). In contrast, FRB follows three principles: practicality, using commonly encountered fonts; diversity, covering multiple font categories; and graduated challenge, testing different recognition scenarios. It includes two versions: an easy version, where sentences appear in various fonts, and a hard version, where font a names are displayed in mismatched typefaces, creating a stroop effect.
3.1 Font Selection
Fonts are generally categorized into three major types: Serif Fonts, Sans-Serif Fonts, and Script & Decorative Fonts. To ensure stylistic diversity in our experiments, we select 15 widely used fonts from these three categories, We choose these fonts based responses from GPT-40, CLAUDE-3.5-SONNEE and statistical data (as shown in Appendix A). The classification and corresponding font names are follows:
3

---

Published as a conference paper at COLM 2025
Strps: Foot Selaction
Step2: Bricwaart Contractioo
Steps: Model Pvaluction
FUSOrr JESTENAL
auSjoriic FAST NnN
font styles suanning arnoss three auger categories forts?
(ffeet)
drose Lhet heoded tow.
TTF
Times Mon Roman
Estteree Oheo fer toe
Flace stotte Dogee Ltre
Hinkarzila
Nettondar
to sllers puitte catamainy and locks
fart tian
Skit
Andi
Canora
shags, Dragetrcke Zaegerast, Texting
LoV
Seaderetk
Callan
Helveto
LOBSTER
Shub bedieard floss joblanlguk
Satiaty
LOaSTER
Tha
PSem-Sast Poets
Georgia
Helvetica
Lubet::
Grad
Chatle wibtty laai
Fex- Shet berrastratee
2'a confusselll
Racffoo
Saiay
colo unice
with
man beggy Aalo
C D E
F
9
H
Figure 2: An overview of the complete pipeline for constructing the font recognition bench- mark, including the two-tier evaluation design. TTF files and corresponding codes ensure visual consistency across images. The easy task evaluates the basic font recognition ability of VLMs, while the hard task introduces stroop effect to assess robustness under semantic- visual conflict.
Serif Fonts:
Baskerville SC, Georgia, Merriweather, Times New Roman, Roboto Slab.
Sans-Serif Fonts: Arial, Calibri, Consola, Helvetica, Monoton, Vera.
Script & Decorative Fonts: Great Vibes, Lobster, Pacifico, Satisfy.
3.2 Benchmark Image Construction
We generate 10 images per font, each containing distinct sentence designed to incorporate as many of the 26 English letters as possible. This results in a total of 15 x 10 = 150 images, forming the easy version of the benchmark. These images are created using a consistent Python script that ensures uniform line breaks, font size, and background, maintaining balanced aspect ratio aligned with the model's input requirements. This easy version allows VLMs to focus exclusively on font recognition without interference from textual content.
For the hard version of the benchmark, we generate 15 additional images per font, each displaying its respective name. The image generation process employs a unified Python script utilizing the ImageDraw and ImageFont packages, ensuring consistency in font size and background dimensions across all images. This approach eliminates discrepancies associated with alternative methods such as screenshots. The inclusion of font names introduces the stroop effect, increasing the benchmark's difficulty. In total, this results in 15 x 15 = 225 images, constituting the complete hard version of the benchmark.
Finally, to support few-shot learning experiments, we generate 52 additional images, each containing one uppercase or lowercase letter from the 26 English alphabets. These im- ages serve reference samples, allowing VLMs to leverage few-shot learning for feature comparison and adaptation in subsequent experiments.
3.3 Challenges
3.3.1 Font Recognition
When the text content in an image consists of sentences, this task simulates real-world scenarios where individuals capture screenshots of textual content to identify the font used. The primary objective is to assess the VLMs' ability to recognize fonts. The presence of greater number of letters enhances the models' capacity to capture distinctive font characteristics, while the relatively small text size further challenges its ability to perform fine-grained recognition.
4

---

Published as a conference paper at COLM 2025
EASY-VERSION
HARD-VERSION
Base Model
Zeno-Shot Zero-Shot CoT
Zero-Shot" Zero-Shot Coth Zero-Shot Zeto-Shot CoT Zeno-Shopl Zero-Shot CoTM
Closed-Source Models
GPT-4o
24.67
30.87
66.67
46.67
12.44
15.11
27.11
25.78
CLAUDE-3.5-SONNE
31.33
29.33
28.67
29.33
8.44
7.11
7.11
6.67
GRMINI-2.0- ASA--0001
18.67
9.78
GPT-IO-MINI
0,00
2,67
33,33
25.00
4.41
1.89
8.11
8.9
Open-Weight Models
9.33
12.67
11.33
11:33
8.44
9.78
9.33
7.11
LLAMA-332-11BB V-NI-1/B--VIONNNNNTTRR
18.67
9.33
16.67
14000
13.33
7.36
6.67
3.39
SHn-3-VISONN-226 Vit.oooNNNNKK///sueuue
0.67
4.77
0.30
2,00
7.33
733
4.44
4.44
6.67
7.56
OWEN2,5-VL-P2 1A--5---V7BB-SSTRUCC
12,00
5.33
17:33
12.00
5.33
5.33
6.67
8.00
OWEN2-VL-72B-INTTTUUT
IDEFIC33-B-LLAMMA3
1,33
1,33
7.33
7.53
4.89
0.89
6.67
6.2
IDENCS2-88
8.00
11.33
9.33
8.00
8.44
7.11
6.22
6.22
Table 1: The accuracy of 13 open-weight and closed-source vision-language models on both the easy and hard versions of the benchmark under different settings. M indicates the use of MCQ setting for inference. The highest accuracy in each setting highlighted in bold, while the second-best results are underlined.
3.3.2 Stroop Effect
When image text contains font names, a conflict between the textual meaning and the visual font triggers the stroop effect (MacLeod, 1991; Verhaeghen & De Meersman, 1998; Besner et al., 1997). For instance, the word "Times New Roman" rendered in Lobster font may mislead model to predict the text's meaning rather than the actual font. Even with clear visual differences, models often misclassify due to cognitive interference. This setup challenges VLMs to disentangle textual semantics from typographic appearance.
4 Experimental Setup
4.1 Vision Language Models
To effectively assess the capabilities of the latest VLMs, conduct experiments 13 widely recognized models. Detailed information about these models is provided in Appendix B.
Few-Shot Selection
Given the large number of tokens required for image input and the common real-world practice of identifying fonts by analyzing individual letter styles, we construct dataset comprising 52 images, each representing an uppercase or lowercase letter from the 26 English alphabets. To ensure consistency and fairness in selecting few-shot samples, we employ the CLIP model for image similarity retrieval. In this setup, the 52 single-letter images serve as the retrieval database, while the experimental images containing font names and sentences function as target samples. We design six experimental scenarios, ranging from 1-shot to 6-shot settings.
5 Experimental Results
We evaluate multiple VLMs using proposed font recognition benchmark. The inference methods include the standard zero-shot approach, as well as Chain-of- -haiu-of-Thought (CoT) (Wei et al., 2023; Kojima al., 2023) and few-shot methods, providing comprehensive assess-

---

Published as a conference paper at COLM 2025
ment of VLMs' font recognition capabilities. All experimental prompts are provided in Appendix F.
5.1 Vision-Language Models Fail on Font Recognition Task
Table 1 demonstrates that contemporary VLMs exhibit suboptimal performance on font recognition tasks, despite the selected 15 fonts being among the more commonly used ones across three major font categories. Under the easy version of the benchmark in zero- shot setting, the CLAUDDE-3.-SONNET achieves the highest accuracy among the evaluated models but only reaches approximately 31% while powerful GPT-4O-MINI achieves even 0% accuracy, Among open-weight models, performance remains notably poor, with only LLAMA-32-11B--VSSONN-N
surpassing 10%, achieving an accuracy of nearly 19%.
Performance further deteriorates under the hard version of the benchmark, which incor- porates the stroop effect. In this setting, almost all models fail to exceed 15% accuracy in a zero-shot scenario. The sole exception is GPT-40, which attains around 15% accuracy while using zero-shot COT. Even other powerful closed-source models fail to surpass 10% accuracy, regardless of whether CoT is applied. These findings underscore the fundamental limitations of existing VLMs in font recognition tasks.
5.2 Vision-Language Models Can Rely on Spurious Visual Cues
To further investigate whether contempo- rary VLMs possess font recognition capa- bilities, we conduct an experiment using multiple-choice questions (MCQ) under zero-shot setting, where models are explic- itly instructed to select the correct font cate- gory from the 15 given options. The results of this experiment are also presented in Ta- ble 1. We now compare these experimental results under the setting that does not use CoT prompting:
65.1
56.1
Accurary
40
2.45
172
For the easy version of the benchmark, we observe that among the four powerful closed-source models, MCQ does not lead to significant improvement for CLAUDE- 3.5-SONNET and GEMINI-2.0-FLASIH-O0 However, it significantly improves the accu- racy of the two GPT--eries models. Among all open-weight models, MCQ improves accuracy by 6-10 percentage points for IDRFicS3-BB-LIAAMA
ZenoSlot
Zero-Slet OVT
ZecoStoe
ZerSShoi Cur
Figure 3: Accuracy comparison on the hard version of the benchmark under two labeling conditions: whether the rendered content in the image conflicts with the actual font label. M denotes inference under a multiple-choice setting.
and PHI-3-VISION-12 VVV-V//SIIO-11KKKNNTTT INSTRUCT. However, this improvement remains marginal, as their original accuracy is close to 0%, indicating that even with constrained choice set, these models still perform poorly. For other models, the accuracy gains under the MCQ setting are smaller, remaining below percentage points.
DWNN2-VI--7-IINSTRUS
For the hard version of the benchmark, among closed-source models, only GPT-40 exhibits significant improvement in accuracy under the MCQ setting, while other models show little to no change. Among open-weight models, a notable finding is that after applying MCQ, all models achieve an accuracy close 6.67% - precisely the expected accuracy if model correctly identifies the font only when the font style matches the font name in the image. Figure's demonstrates that applying MCQ to the hard version the benchmark only further amplifies the influence of the stroop effect. However, it has minimal impact when the text in the image does not match its font type. This strongly suggests that these models lack the capability to discern the correct font under the stroop effect; instead, they are merely influenced by the textual content in the image. This result further underscores the susceptibility of VLMs to the stroop effect in this task. Given their inherent limitations in font

---

Published as a conference paper at COLM 2025
GPT-tomrin
6.2
0.5
420
5.0
8.
70
30
10
Slods
State
Quar2 (Vl--Bierrt
5.0
5.00
730
6.0
3.25
1.45
0.0
400
431
4
Figure 4: Few-shot accuracy (1-shot to 6-shot) on the hard benchmark setting, evaluated across six vision-language models from distinct model families.
recognition, even when provided with explicit answer choices, these models fail leverage the constraints meaningfully. Instead, they appear to be even more influenced by the textual content, as evidenced by the accuracy drop observed in LLAM--32-11--V/IONN--- on the hard version.
5.3 Few-shot Learning Provides Limited Improvements
In real-world scenarios, when given reference images of individual letters written in spe- cific font style, humans can carefully compare these samples with test image to determine the font type. This raises the question of whether few-shot learning can enhance VLMs' font recognition capabilities - specifically, whether VL.Ms can focus on the features provided in the demonstrations and leverage comparison-based reasoning to identify the correct font in the final test image.
To investigate this, we conduct few-shot experiments on six open-weight and closed-source VLMs under the setting described in Section 4.2. As shown in Figure 4, increasing the number of shots from 1 to 6 results in only marginal improvement in model accuracy, with gains of less than 4 percentage points across all models. Given the dataset size for the hard version of the benchmark evaluation, this level of improvement is not ideal. Furthermore, we observe that, with the exception of the either increases or remains stable as the number of shots increases, other models exhibit fluctuations in accuracy despite receiving more examples, which is counterintuitive. This instability further underscores the limited capabilities of VLMs font recognition under few-shot settings. Notably, this contrasts sharply with human performance in fine-grained recognition tasks, where individuals effectively leverage reference samples improve accuracy.
whose accuracy
Categories of Errors
To investigate the reasons behind the suboptimal performance of VLMs in font recogni- tion, conduct detailed analysis of model outputs to identify specific error patterns. Based on our observations, we classify the errors into four distinct categories and provide representative examples of each in Table 3.

---

Published as a conference paper at COLM 2025
Frror Type
Easy Version (%)
Hard Version (%)
Content-Biased Misattribution
0.00
70.11
Explicit Inability Acknowledgment
10.88
2.13
Indecisive Classification
64.87
14.46
Confidently Incorrect Prediction
24.25
13.30
Table 2: Error ratios for four distinct error categories, aggregated over all incorrect predic- tions by 13 vision-language models on both benchmark tiers (easy and hard).
Content- Biased Misattribution: Error occurs primarily in the hard version of the bench- mark, where the model mistakenly assumes that the font name appearing in the image corresponds to the actual font used for the text.
Explicit Inability Acknowledgment: The model explicitly admits it cannot determine the font, often stating "more information is needed" or "I can't determine the font", reflecting awareness of its limitations.
Indecisive Classification: The model analyzes font characteristics and distinguishes broad categories like Serif vs. Sans-Serif but fails to name specific font, instead listing multiple candidates.
Confidently Incorrect Prediction: The model gives a definitive but incorrect answer, some- times with font analysis, displaying unjustified confidence and leading to misleading results.
Table 2 presents the distribution of error types across two benchmark levels. Notably, in the easy version, where the image content consists of complete sentences, content-biasee misattribution does not occur. Although the textual content in this setting does not pose a stroop effect challenge, a considerable portion of the model responses indicate either explicit acknowledgment of their inability to perform the font recognition task the provision of only a set of possible candidates or a broader font category, without definitive prediction. This suggests that current VLMs lack the robustness required for accurate font recognition and often exhibit uncertainty when faced with this task.
In contrast, in the hard version, the presence of font names within the images introduces significant confusion. Even when utilizing Chain-of- Thought prompting, most models struggle to consistently analyze font features and instead rely heavily on the textual content. Only few sufficiently strong models demonstrate the ability to reason in the correct direction. These findings indicate that current VLMs lack robustness and are not yet capable of reliably handling font recognition tasks.
7 Do VLMs Truly Focus on Crucial Edge Regions?
Different VL.Ms utilize distinct mechanisms for processing image inputs. Considering both their popularity and open-weight availability, we analyze the attention matrices from two models:
and QWINN-VLL-7--NSTRUCY
Figure 5 presents the attention matrix of the final cross-attention layer for the LLAMA- model on incorrect predictions from the hard version of the benchmark. All images in this figure share a common characteristic: the displayed font name does not match the actual font used. If VLMs truly possess the ability to identify fonts, their predictions should not be heavily influenced by this discrepancy. For instance, human observer would easily recognize that the text displaying 'Baskervville SC' is not written in the Baskervville SC font, as the characteristics of the letters suggest a script-style font. However, as observed in the attention matrix, LLAMA-3.2 2- 111-BBBB-- VISION-INSTRUS fails to focus on the regions that capture distinctive font features, such as the edges of individual characters. Instead, across all six sub-images, the model exhibits misplaced attention, often focusing irrelevant background regions rather than the critical font-defining elements. This misalignment in attention leads the failure of the recognition task, highlighting the model's limitations in accurately distinguishing font styles.
8

---

Published as a conference paper at COLM 2025
Error Category
Label
Incorrect Model Prediction
Content-Biased Misattribution
Lobster
The text in the image is 'Georgia', This is font name, which is used describe the style and appearance text.
Cunsola
The image cuntains the text 'calibri'. The funt used in the text is Calibri.
Explicit Inability Acknowledgment
Calibri
The information provided is not sufficient for me. Therefore, is not possible to determine the font used in the text of the image. I'm not able to identify the font used in the image. If you have to cen-iideniiiitooon tools, they might provide more accurate result.
Arial
Indecisive Classification
Arial
The font is likely standard system font, such as Calibri or Hel- vetica, which are widely used in various applications docu- ments.
Great Vibes
Based on characteristics, seems to resemble ing Script or Pacifico, but it's hard to pinpoint the exact name.
Danc-
Confidently Incurrect Prediction
Arial
Based these observations the font resembles Geungia. Geurgia is known for its readability on screems and has a classic serif style with noticeable stroke contrast.
Helvetica
Based on these observations, the font resembles Times New Ro- man. This font widely used for its readability and classic style.
Table 3: Examples of four different error types in both the easy and hard versions of the benchmark, displaying the incorrect model predictions alongside the corresponding ground- truth labels.
Label: Pacifiin
Label: Great Vibes
Label Atial
Times New Roman
Meiviweather
Baskervvlle Se
-88
Label: Times
Tabel: Callbri
Label: Roboio Slub
Georgia consola
Great Vibes
Figure 5: Attention heatmaps from the final cross-attention layer of LLAMA-3 1.0MMA--.2-111 1---1BBVISIOON INSTRUCT, illustrating how text tokens attend to image patches for six input images from the hard version of the benchmark, all of which the model fails to classify correctly. Attention weights are averaged across all heads.
Figure presents the attention matrix for QWEN2- of the benchmark. Compared to LLAMM-33.2-1 B- VISION- DWEN2-VL-7B. INSTRUCT employs a larger merged patch size. However, regardless of patch size, a similar issue persists: the model's attention does not sufficiently focus on the edges of the letters in the image, Instead, only a few letters receive more attention than other parts, and in many cases, the model assigns higher attention to the white background rather than the textual content. However, when comparing the attention matrices in Figure 6 with those in Figure 5, we observe that in the easy version of the benchmark, QWWEN2-VL--7--IISSIRCCYY demonstrates a stronger tendency to focus on the textual regions. The red red-highlighted areas in the attention matrix cover most of the text, even though the highest attention weights are not fully concentrated on the letters. Nevertheless, this represents a notable improvement over LLAMA-3. 1-AAMA-----1BVVSSION-NN V-IIIVIIIIIIISUUCCCO 's performance on the hard version of the benchmark, where attention is more poorly distributed.
on the easy version
Based on our analysis of the above two VLMs, we infer that current VLMs remain inadequate for font recognition tasks, even for widely used font styles. This limitation likely arises from their inability to focus on the critical regions of the textual input that are essential for accurate font identification. We posit that this shortcoming stems from insufficient

---

Published as a conference paper at COLM 2025
Labe: Arial
Tabel: Corsola
Label: Great Vihes
Brianna typed her final exam in cozy library nook, quietly watching zebra crossing video and just sipping fizzy lemonade.
Gabriel watched an exciting quiz show
Leuaa goictly obtunged a pianic with Zor. bringing fresh fanits juzzz musie, extia. nophins, and sreton cefost blooted to nnjyy
drank fizzy soda while
munching on quick pizza slice for supper, after movie.
48
Arrontte Woight
Label: Roboto Sab
Label: Loister
Label: Times New Roman
Sylvia quickly typed her new quiz on a fancy laptop, while Jack fixed broken gaming mouse and drank zebra-striped coffee.
Priscilla drove to the zoo with Victor, examining quirky zebras. zebras.foxes, foxes, and jovial monkey who loved peanuts.
Yesterday, Max drove quite far grab zesty pizza slice, fresh juice, a weird box of quirky vitamins, then headed home
Figure 6: Attention heatmaps from the middle cross-attention layer (14 of QWEN2-VL- 7B-INSTRUCT, illustrating how text tokens attend to image patches for six input images from the easy version of the benchmark, all of which the model fails to classify correctly. Attention weights are averaged across all heads.
alignment between visual attention mechanisms and the subtle typographic features that define font identity. Consequently, advancing font recognition within VLMs demands more targeted modeling of local visual regions and a deeper integration of typographic priors
Conclusion
In this paper, we introduce a specially designed Font Recognition Benchmark (FRB) compris- ing 15 popular font styles, designed with both easy and hard versions to comprehensively assess the capabilities of contemporary VLMs in this fine-grained recognition task. We conduct extensive experiments with a range of VLMs and observe that these models con- sistently struggle with font recognition, particularly on the hard version of the benchmark, where the stroop effect substantially reduces accuracy. This finding highlights that VLMSs remain susceptible to misleading visual cues even when performing semantically grounded tasks. To further examine the capacity of VLMs in this setting, we evaluate various inference strategies, including CoT prompting, Multiple-Choii Learning. However, our results demonstrate that these methods offer only marginal im- provements in overall accuracy. Additionally, by VLMs into four distinct categories and analyze the attention patterns of two publicly available VLMs. Our analysis reveals that these models fail to adequately focus on the edge information of characters in the input images, which impairs their ability to correctly identify font styles and renders them vulnerable variations in texture content.
Question answering, and Few-Shot
categorize the error types exhibited
Limitation
In this paper, we propose a specially designed benchmark comprising two task versions to enable a comprehensive evaluation of VLMs' performance on font recognition and analyze the factors contributing to their suboptimal performance. However, our study still has the following limitations: (i) Due to budget constraints, we are unable to conduct few-shot inference on more powerful models such as CLAUDE-3.5-SONNE
and GPT-40. (ii) As certain high-performing models are closed-source, we are unable to analyze their attention matrices, limiting our understanding of their docision-making processes. (iii) Our experiments rely on straightforward prompts for inference. Designing more sophisticated prompts, such as guiding models to focus on key regions in the input image, may enhance their performance on this task. We plan to explore these directions in future work.
10

---

Published as a conference paper at COLM 2025
Acknowledgments
The work is partially supported by the NSF of the United States Grant CRII 2451683, an NVIDIA Academic Grants Program, University of California at Merced, and UC Merced Faculty Research Award. The views and conclusions are those of the authors and should not reflect the official policy or position the U.S. Government.
Ethics Statement
Ethical considerations play central role in this research. All models used in this study are ei- ther open-weight or widely adopted within the scientific community, ensuring transparency and reproducibility. The proposed benchmark and evaluation framework aim assess the capabilities of current VLMs on font recognition without introducing or reinforcing harmful biases. No personally identifiable information or sensitive data involved in this work. We are committed to responsible research practices, and we advocate for the transparent reporting and ethical deployment of Al technologies in ways that serve the broader interests of society.
References
Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Qin Cai, Vishrav Chaudhary, Dong Chen, Dongdong Chen, Weizhu Chen, Yen-Chun Chen, Yi-Ling Chen, Hao Cheng, Parul Chopra, Xiyang Dai, Matthew Dixon, Ronen Eldan, Victor Fragoso, Jianfeng Gao, Mei Gao, Min Gao, Amit Garg, Allie Del Giorno, Abhishek Goswami, Suriya Gunasekar, Emman Haider, Junheng Hao, Russell J. Hewett, Wenxiang Hu, Jamie Huynh, Dan Iter, Sam Ade Jacobs, Mojan Javaheripi, Xin Jin, Nikos Karampatziakis, Piero Kauffmann, Mahoud Khademi, Dongwoo Kim, Young Jin Kim, Lev Kurilenko, James R. Lee, Yin Tat Lee, Yuanzhi Li, Yunsheng Li, Chen Liang, Lars Liden, Xihui Lin, Zeqi Lin, Ce Liu, Liyuan Liu, Mengchen Liu, Weishung Liu, Xiaodong Liu, Chong Luo, Piyush Madan, Ali Mah- moudzadeh, David Majercak, Matt Mazzola, Caio César Teodoro Mendes, Arindam Mitra, Hardik Modi, Anh Nguyen, Brandon Norick, Barun Patra, Daniel Perez-Becker, Thomas Portet, Reid Pryzant, Heyang Qin, Marko Radmilac, Liliang Ren, Gustavo de Rosa, Corby Rosset, Sambudha Roy, Olatunji Ruwase, Olli Saarikivi, Amin Saied, Adil Salim, Michael Santacroce, Shital Shah, Ning Shang, Hiteshi Sharma, Yelong Shen, Swadheen Shukla, Xia Song, Masahiro Tanaka, Andrea Tupini, Praneetha Vaddamanu, Chunyu Wang, Guan- hua Wang, Lijuan Wang, Shuohang Wang, Xin Wang, Yu Wang, Rachel Ward, Wen Wen, Philipp Witte, Haiping Wu, Xiaoxia Wu, Michael Wyatt, Bin Xiao, Can Xu, Jiahang Xu, Wei- jian Xu, Jilong Xue, Sonali Yadav, Fan Yang, Jianwei Yang. Yifan Yang, Ziyi Yang, Donghan Yu, Lu Yuan, Chenruidong Zhang, Cyril Zhang, Jianwen Zhang, Li Lyna Zhang, Yi Zhang, Yue Zhang, Yunan Zhang, and Xiren Zhou. Phi-3 technical report: A highly capable language model locally on your phone, 2024. URL https: /ttp///axxvvv../ wttp://..rg//bb//////////
14219.
Aishwarya Agrawal, Jiasen Lu, Stanislaw Antol, Margaret Mitchell, C. Lawrence Zitnick, Dhruv Batra, and Devi Parikh. Vqa: Visual question answering, 2016. URL https: // //t.xiv.org/abs///// 00468.
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Has- son, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Mon- teiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. Flamingo: a visual language model for few-shot learning, 2022. URL https://aaxxv.orr//bbs///
14198.
Anthropic. Claude 3.5 sonnet. https:// http://artrroocccccc/////s/////s 3-t-idee------- 3----uudde------ntet----------,e June 2024. Accessed Jun 20, 2024.
11

---

Published as a conference paper at COLM 2025
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report, 2025. URL https: ottpkiii...r//a//552....
abs/2502. 13923.
Derek Besner, Jennifer A Stolz, and Clay Boutilier. The stroop effect and the myth of automaticity. Psychonomic bulletin & review, 4(2):221-22,, 1997.
Florian Bordes, Richard Yuanzhe Pang, Anurag Ajay, Alexander C. Li, Adrien Bardes, Suzanne Petryk, Oscar Mañaas, Zhiqiu Lin, Anas Mahmoud, Bargav Jayaraman, Mark Ibrahim, Melissa Hall, Yunyang Xiong, Jonathan Lebensold, Candace Ross, Srihari Jayakumar, Chuan Guo, Diane Bouchacourt, Haider Al-Tahan, Karthik Padthe, Vasu Sharma, Hu Xu, Xiaoqing Ellen Tan, Megan Richards, Samuel Lavoie, Pietro Astolfi, Rey- hane Askari Hemmat, Jun Chen, Kushal Tirumala, Rim Assouel, Mazda Moayeri, Arjang Talattof, Kamalika Chaudhuri, Zechun Liu, Xilun Chen, Quentin Garrido, Karen Ullrich, Aishwarya Agrawal, Kate Saenko, Asli Celikyilmaz, and Vikas Chandra. An introduction to vision-language modeling, 2024. URL https:: https::/arxxvv.rr//bb/////
17247.
Davide Caffagni, Federico Cocchi, Luca Barsellotti, Nicholas Moratelli, Sara Sarto, Lorenzo Baraldi, Lorenzo Baraldi, Marcella Cornia, and Rita Cucchiara. The revolution of multi- modal large language models: A survey, 2024. URL httpps:/aaxiv..et.
www.:/or/..bb///22.....// httt 12451.
Guang Chen, Jianchao Yang, Hailin Jin, Jonathan Brandt, Eli Shechtman, Aseem Agarwala, and Tony X. Han. Large-scale visual font recognition. In 2014 IEEE Conference on Computer Vision and Pattern Recognition, pp. 3598-3605, 2014a. 2014.460.
doi: 10.1109/CVPR.
Guang Chen, Jianchao Yang, Hailin Jin, Jonathan Brandt, Eli Shechtman, Aseem Agar- wala, and Tony X Han. Large-scale visual font recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3598-3605, 2014b.
Jingchao Chen, Shiyi Mu, Shugong Xu, and Youdong Ding. Henet: forcing a network to think more for font recognition. In Proceedings of the 3rd International Conference on Advanced Information Science and System, pp. 1-5, 2021.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple frame- work for contrastive learning of visual representations, 2020. URL https://arxiv... org/ 0bs//202..05009
Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and lifeng Dai. Internvl: Scaling up vision foundation models and aligning for generic visual- linguistic tasks, 2024. URL https:://a/:/iii wwwperwwwww..oraeaeeeeeeeeeo otg/abs//////aaaoooo/.oooeee ab/////s.22222..///// 14238.
Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, Lixin Gu, Xuehui Wang, Qingyun Li, Yimin Ren, Zixuan Chen, Jiapeng Luo, Jiahao Wang, Tan Jiang, Bo Wang, Conghui He, Botian Shi, Xingcheng Zhang, Han Lv, Yi Wang, Wenqi Shao, Pei Chu, Zhongying Tu, Tong He, Zhiyong Wu, Huipeng Deng, Jiaye Ge, Kai Chen, Kaipeng Zhang, Limin Wang, Min Dou, Lewei Lu, Xizhou Zhu, Tong Lu, Dahua Lin, Yu Qiao, Jifeng Dai, and Wenhai Wang. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling, 2025. URL https: / https::/aanvi..rr///bbbss abs//////a///////// 05271.
Wenyi Cui and Kohei Inoue. Chinese calligraphy recognition system based convolutional neural network. ICIC Express Letters, 15(11):1187-1115, 2021.
Zhi Gao, Bofei Zhang, Pengxiang Li, Xiaojian Ma, Tao Yuan, Yue Fan, Yuwei Wu, Yunde Jia, Song- Zhu, and Qing Li. Multi-modal agent tuning: Building vlm-driven agent for efficient tool usage. arXiv preprint arXiv:2412.1 15606, 2024.
Google. Gemini 2.0. https: / /ttp///deeelees...eeeee/e gow.eeelgggggbbboogoooooooo com/en wonggmini---ee femmini-2-faree 2- te-ffmll--expandee February 2025. Accessed FEB. 5, 2025.
12

---

Published as a conference paper at COLM 2025
Taicheng Guo, Xiuying Chen, Yaqi Wang, Ruidi Chang, Shichao Pei, Nitesh V Chawla, Olaf Wiest, and Xiangliang Zhang. Large language model based multi-agents: survey of progress and challenges. arXiv preprint arXiv:2402.01680, 2024.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners, 2023. URL httpps://arkiii wttpr/iiw.orgab////..../// 11916.
Jian Li, Weiheng Lu, Hao Fei, Meng Luo, Ming Dai, Min Xia, Yizhang Jin, Zhenye Gan, Ding Qi, Chaoyou Fu, et al. A survey benchmarks of multimodal large language models. arXiv preprint arXiv:2408.08632. 2024.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation, 2022a. URL 12086.
https://arxiv
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models, 2023. URL https://arxiv :ttps://arrl..ogg/////3101 12597.
Xudong Li, Jingyi Wang, Haiyang Zhang, Yongke Huang, and Huihui Huang. Swordnet: 2022b.
Chinese character font style recognition network. IEEE Access,
Haotian Liu, Chumyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023.
URL / httpss:///axxvvooraaa/ao www./oog//////a....ooo.. 2atbos//334.000 08485.
Colin M MacLeod. Half a century of research on the stroop effect: an integrative review.
Psychological bulletin, 109(2):163, 1991.
Meta AI. Llama 3.2: Revolutionizing edge ai and vision with open, cus- tomizable models. Meta AI Blog, 2024. URL https://ai.men. cot/blog/ 1lama-3- c-nnect- wow.eect-224-vviiiioe 1024-vision-edge-mee -ediieeeeeicess...
Mehrdad Mohammadian, Neda Maleki, Tobias Olsson, and Fredrik Ahlgren. Persis: A persian font recognition pipeline using convolutional neural networks. In 2022 12th International Conference on Computer and Knowledge Engineering (ICCKE), pp. 196- 204. IEEE, 2022.
OpenAL. 2024.
https://openaa http:/penai.ccmmaaaaa/ wtt.a/accii.ooo.eeeeeiieeeeeeeee July 2024. Accessed JUL. 18,
Gpt-4o-mini.
com/ index/
OpenAL, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, llge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapiro, Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brak- man, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Lukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick,
13

---

Published as a conference paper at COLM 2025
Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Lukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reiichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cullen O'Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Fran- cis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine B. Thomp- son, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sher- win Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojeiech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. Gpt-4 technical report, 2024a. URL https: http://.rggg///////ee/oe/ee 08774.
OpenAI, Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander Madry, Alex Baker-Whitcomb, Alex Beutel, Alex Borzunov, Alex Carney, Alex Chow, Alex Kirillov, Alex Nichol, Alex Paino, Alex Renzin, Alex Tachard Passos, Alexander Kirillov, Alexi Christakis, Alexis Conneau, Ali Kamali, Allan Jabri, Allison Moyer, Allison Tam, Amadou Crookes, Amin Tootoonchian, Ananya Kumar, Andrea Vallone, Andrej Karpathy, Andrew Braunstein, Andrew Cann, Andrew Codispoti, Andrew Galu, Andrew Kondrich, Andrew Tulloch, Andrey Mishchenko, Angela Baek, Angela Jiang, Antoine Pelisse, Antonia Woodford, Anuj Gosalia, Arka Dhar, Ashley Pantuliano, Avi Nayak, Avital Oliver, Barret Zoph, Behrooz Ghorbani, Ben Leimberger, Ben Rossen, Ben Sokolowsky, Ben Wang, Benjamin Zweig, Beth Hoover, Blake Samic, Bob McGrew, Bobby Spero, Bogo Giertler, Bowen Cheng, Brad Lightcap, Brandon Walkin, Brendan Quinn, Brian Guarraci, Brian Hsu, Bright Kellogg, Brydon Eastman, Camillo Lugaresi, Carroll Wainwright, Cary Bassin, Cary Hudson, Casey Chu, Chad Nelson, Chak Li, Chan Jun, Channing Conger, Charlotte Barette, Chelsea Voss, Chen Ding, Cheng Lu, Chong Zhang, Chris Beaumont, Chris Hallacy, Chris Koch, Christian Gibson, Christina Kim, Christine Choi, Christine McLeavey, Christopher Hesse, Claudia Fischer, Clemens Winter, Coley Czarnecki, Colin Jarvis, Colin Wei, Constantin Koumouzelis, Dane Sherbur,, Daniel Kappler, Daniel Levin, Daniel Levy, David Carr, David Farhi, David Mely, David Robinson, David Sasaki, Denny Jin, Dev Valladares, Dimitris Tsipras, Doug Li, Duc Phong Nguyen, Duncan Findlay, Edede Oiwoh, Edmund Wong, Ehsan Asdar, Elizabeth Proehl, Elizabeth Yang, Eric Antonow, Eric Kramer, Eric Peterson, Eric Sigler, Eric Wallace, Eugene Brevdo, Evan Mays, Farzad Khorasani, Felipe Petroski Such Zhang, Filippo Raso, Francis Zhang, Fred von Lohmann, Freddie Sulit, Gabriel Goh, Gene Oden, Geoff Salmon, Giulio Starace, Greg Brockman, Hadi Salman, Haiming Bao, Haitang Hu, Hannah Wong, Haoyu Wang, Heather Schmidt, Heather Whitney, Heewoo Jun, Hendrik Kirchner, Henrique Ponde de Oliveira Pinto, Hongyu Ren, Huiwen Chang, Hyung Won Chung, lan Kivlichan, Ian O'Connell, Ian
14

---

Published as a conference paper at COLM 2025
Osband, Ian Silber, Ian Sohl, Ibrahim Okuyucu, Ikai Lan, Ilya Kostrikov, Ilya Sutskever, Ingmar Kanitscheider, Ishaan Gulrajani, Jacob Coxon, Jacob Menick, Jakub Pachocki, James Aung, James Betker, James Crooks, James Lennon, Jamie Kiros, Jan Leike, Jane Park, Jason Kwon, Jason Phang, Jason Teplitz, Jason Wei, Jason Wolfe, Jay Chen, Jeff Harris, Jenia Varavva, Jessica Gan Lee, Jessica Shieh, Ji Lin, Jiahui Yu, Jiayi Weng, Jie Tang, Jieqi Yu, Joanne Jang, Joaquin Quinonero Candela, Joe Beutler, Joe Landers, Joel Parish,Jhhanre Heidecke, John Schulman, Jonathan Lachman, Jonathan McKay, Jonathan Uesato, Jonathan Ward, Jong Wook Kim, Joost Huizinga, Jordan Sitkin, Jos Kraaijeveld, Josh Gross, Josh Kaplan, Josh Snyder, Joshua Achiam, Joy Jiao, Joyce Lee, Juntang Zhuang, Justyn Harriman, Kai Fricke, Kai Hayashi, Karan Singhal, Katy Shi, Kavin Karthik, Kayla Wood, Kendra Rimbach, Kenny Hsu, Kenny Nguyen, Keren Gu-Lemberg, Kevin Button, Kevin Liu, Kiel Howe, Krithika Muthukumar, Kyle Luther, Lama Ahmad, Larry Kai, Lauren Itow, Lauren Workman, Leher Pathak, Leo Chen, Li Jing, Lia Guy, Liam Fedus, Liang Zhou, Lien Mamitsuka, Lilian Weng, Lindsay McCallum, Lindsey Held, Long Ouyang, Louis Feuvrier, Lu Zhang, Lukas Kondraciuk, Lukasz Kaiser, Luke Hewitt, Luke Metz, Lyric Doshi, Mada Aflak, Maddie Simens, Madelaine Boyd, Madeleine Thompson, Marat Dukhan, Mark Chen, Mark Gray, Mark Hudnall, Marvin Zhang, Marwan Aljubeh, Mateusz Litwin, Matthew Zeng, Max Johnson, Maya Shetty, Mayank Gupta, Meghan Shah, Mehmet Yatbaz, Meng Jia Yang, Mengchao Zhong, Mia Glaese, Mianna Chen, Michael Janner, Michael Lampe, Michael Petrov, Michael Wu, Michele Wang, Michelle Fradin, Michelle Pokrass, Miguel Castro, Miguel Oom Castro Temudo de, Mikhail Pavlov, Miles Brundage, Miles Wang, Minal Khan, Mira Murati, Mo Bavarian, Molly Lin, Murat Yesildal, Nacho Soto, Natalia Gimelshein, Natalie Cone, Natalie Staudacher, Natalie Summers, Natan LaFontaine, Neil Chowdhury, Nick Ryder, Nick Stathas, Nick Turley, Nik Tezak, Niko Felix, Nithanth Kudige, Nitish Keskar, Noah Deutsch, Noel Bundick, Nora Puckett, Ofir Nachum, Ola Okelola, Oleg Boiko, Oleg Murk, Oliver Jaffe, Olivia Watkins, Olivier Godement, Owen Campbell-Moore, Patrick Chao, Paul McMillan, Pavel Belov,Peng Peter Bak, Peter Bakkum, Peter Deng, Peter Dolan, Peter Hoeschele, Peter Welinder, Phil Tillet, Philip Pronin, Prafulla Dhariwal, Qiming Yuan, Rachel Dias, Rachel Lim, Rahul Arora, Rajan Troll, Randall Lin, Rapha Gontijo Lopes, Raul Puri, Reah Miyara, Reimar Leike, Renaud Gaubert, Reza Zamani, Ricky Wang, Rob Donnelly, Rob Honsby, Rocky Smith, Rohan Sahai, Rohit Ramchandani, Romain Huet, Rory Carmichael, Rowan Zellers, Roy Chen, Ruby Chen, Ruslan Nigmatullin, Ryan Cheu, Saachi Jain, Sam Altman, Sam Schoenholz, Sam Toizer, Samuel Miserendino, Sandhini Campbell, Sara Culver, Scott Ethersmith, Scott Gray, Sean Grove, Sean Metzger, Shamez Hermani, Shantanu Jain, Shengjia Zhao, Sherwin Wu, Shino Jomoto, Shirong Wu, Shuaiqi Xia, Sonia Phene, Spencer Papay, Srinivas Narayanan, Steve Coffey, Steve Lee, Stewart Hall, Suchir Balaji, Tal Broda, Tal Stramer, Tao Xu, Tarun Gogineni, Taya Christianson, Ted Sanders, Tejal Patwardhan, Thomas Cunninghman, Thomas Degry, Thomas Dimson, Thomas Raoux, Thomas Shadwell, Tianhao Zheng, Todd Underwood, Todor Markov, Toki Sherbakov, Tom Rubin, Tom Stasi, Tomer Kaftan, Tristan Heywood, Troy Peterson, Tyce Walters, Tyna Eloundou, Valerie Qi Veit Moeller, Vinnie Monaco, Vishal Kuo, Vlad Fomenko, Wayne Chang, Weiyi Zheng, Wenda Zhou, Will Sheu, Wojciech Zaremba, Yash Patil, Yilei Qian, Yongjik Kim, Youlong Cheng, Yu Zhang, Yuchen He, Yuchen Zhang, Yujia Jin, Yunxing Dai, and Yury Malkov. Gpt-4o system card. arXiv preprint, August 2024b. URL https://arxiv org/abs//2100 21276. Accessed June 22, 2025.
Hengnian Qi, Hao Yang, Zhaojiang Wang, Jiabin Ye, Qiuyi Xin, Chu Zhang, and Qing Lang. Ancientglyphnet: an advanced deep learning framework for detecting ancient chinese characters in complex scene. Artificial Intelligence Review, 58(3)888, 2025.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021. URL httpss///arxt
arrg/abs/2103 00020..
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in neural information processing systems, 04.25278-25294 2022.
15

---

Published as a conference paper at COLM 2025
Monique Solomons. 75 font statistics: Usage and market analysis. Linearity Blog, 2023.
URL hattps:/www..lieaariii wwtp////oootts//o/eeeeooooooooee statississsssssss..
Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Gar- rett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, Soroosh Mariooryad, Yifan Ding, Xinyang Geng, Fred Alcober, Roy Frostig, Mark Omernick, Lexi Walker, Cos- min Paduraru, Christina Sorokin, Andrea Tacchetti, Colin Gaffney, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, 2024. URL https://arxiv. http:///axxv..rr//bbaap.a orggabs//4g/ 05530.
Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penchev, Gaël Liu, Francesco Visin, Kathleen Ke- nealy, Lucas Beyer, Xiaohai Zhai, Anton Tsitsulin, Robert Busa-Fekete, Alex Feng, Noveen Sachdeva, Benjamin Coleman, Yi Gao, Basil Mustafa, lain Barr, Emilio Parisotto, David Tian, Matan Eyal, Colin Cherry, Jan-Thorsten Peter, Danila Sinopalnikov, Surya Bhupati- raju, Rishabh Agarwal, Mehran Kazemi, Dan Malkin, Ravin Kumar, David Vilar, Idan Brusilovsky, Jiaming Luo, Andreas Steiner, Abe Friesen, Abhanshu Sharma, Abheesht Sharma, Adi Mayrav Gilady, Adrian Goedeckemeyer, Alaa Saade, Alex Feng, Alexander Kolesnikov, Alexei Bendebury, Alvin Abdagic, Amit Vadi, András György, André Susano Pinto, Anil Das, Ankur Bapna, Antoine Miech, Antoine Yang, Antonia Paterson, Ashish Shenoy, Ayan Chakrabarti, Bilal Piot, Bo Wu, Bobak Shahriari, Bryce Petrini, Charlie Chen, Charline Le Lan, Christopher A. Choquette-Choo, CJ Carey, Cormac Brick, Daniel Deutsch, Danielle Eisenbud, Dee Cattle, Derek Cheng, Dimitris Paparas, Divyashree Shivakumar Sreepathihalli, Doug Reid, Dustin Tran, Dustin Zelle, Eric Noland, Erwin Huizenga, Eugene Kharitonov, Frederick Liu, Gagik Amirkhanyan, Glenn Cameron, Hadi Hashemi, Hanna Klimczak-Pluciisske Harman Singh, Harsh Mehta, Harshal Tushar Lehri, Hussein Hazimeh, Ian Ballantyne, Idan Szpektor, Ivan Nardini, Jean Pouget-Abadie, Jetha Chan, Joe Stanton, John Wieting, Jonathan Lai, Jordi Orbay, Joseph Fernandez, Josh Newlan, Ju yeong Ji, Jyotinder Singh, Kat Black, Kathy Yu, Kevin Hui, Kiran Vodrahalli, Klaus Greff, Linhai Qiu, Marcella Valentine, Marina Coelho, Marvin Ritter, Matt Hoffman, Matthew Watson, Mayank Chaturvedi, Michael Moynihan, Min Ma, Nabila Babar, Natasha Noy, Nathan Byrd, Nick Roy, Nikola Momchev, Nilay Chauhan, Noveen Sachdeva, Oskar Bunyan, Pankil Botarda, Paul Caron, Paul Kishan Rubenstein, Phil Culliton, Philipp Schmid, Pier Giuseppe Sessa, Pingmei Xu, Piotr Stanezyk, Pouya Tafti, Rakesh Shivanna, Renjie Wu, Renke Pan, Reza Rokni, Rob Willoughby, Rohith Vallu, Ryan Mullins, Sammy Jerome, Sara Smoot, Sertan Girgin, Shariq Iqbal, Shashir Reddy, Shruti Sheth, Siim Pöder, Sijal Bhatnagar, Sindhu Raghuram Panyam, Sivan Eiger, Susan Zhang, Tianqi Liu, Trevor Yacovone, Tyler Liechty, Uday Kalra, Útku Evci, Vedant Misra, Vincent Roseberry, Vlad Feinberg, Vlad Kolesnikov, Woohyun Han, Woosuk Kwon, Xi Chen, Yinlam Chow, Yu- vein Zhu, Zichuan Wei, Zoltan Egyed, Victor Cotruta, Minh Giang, Phoebe Kirk, Anand Rao, Kat Black, Nabila Babar, Jessica Lo, Erica Moreira, Luiz Gustavo Martins, Omar Sanseviero, Lucas Gonzalez, Zach Gleicher, Tris Warkentin, Vahab Mirrokni, Evan Senter, Eli Collins, Joelle Barral, Zoubin Ghahramani, Raia Hadsell, Yossi Matias, D. Sculley, Slav Petrov, Noah Fiedel, Noam Shazeer, Oriol Vinyals, Jeff Dean, Demis Hassabis, Koray Kavukcuoglu, Clement Farabet, Elena Buchatskaya, Jean-Baptiste Alayrac, Rohan Anil, Dmitry, Lepikhin, Sebastian Borgeaud, Olivier Bachem, Armand Joulin, Alek Andreev, Cassidy Hardin, Robert Dadashi, and Léonard Hussenot. Gemma 3 technical report, 2025. URL /ttpp:///axxiv org abs////s/////// 19786.
Chris Tensmeyer, Daniel Saunders, and Tony Martinez. Convolutional neural networks for font classification. In 2017 14th IAPR international conference on document analysis and recognition (ICDAR), volume 1, pp. 985-990. IEEE, 2017.
Moshiur Rahman Tonmoy, Akhtaruzzaman Adnan, Aloke Kumar Saha, MF Mridha, and Nilanjan Dey. Descriptor: Multilingual visual font recognition dataset (mvfr). IEEE Data Descriptions, 2024.
16

---

Published as a conference paper at COLM 2025
Léo Tronchon, Hugo Laurençon, and Victor Sanh. Introducing idefics2: A powerful 8b vision-language model for the community. Hugging Face Blog, 2024. URL https:// huggingface wtt.ee//ccoco/eee.eee.eeeoeooolt co/blog/idefics. idefics..
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, 2023. URL 03762.
http://axxiiv..ggbbbbb///
Paul Verhaeghen and Lieve De Meersman. Aging and the stroop effect: meta-analysis. Psychology and aging, 13(1):120, 1998.
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vision-language model's perception of the world any resolution, 2024. URL https: //arxiv.org/abbst wttp:/rg/bbb///////// 121911
Yixuan Wang and Yanyi Zong. Calligraphy font recognition algorithm based on im- proved densenet network. In 2023 Global Conference on Information Technologies and Communications (GCITC), pp. 1-5. IEEE, 2023.
Zhangyang Wang, Jianchao Yang, Hailin Jin, Eli Shechtman, Aseem Agarwala, Jonathan Brandt, and Thomas Huang. Deepfont: Identify your font from an image, 2015a. URL 03196.
ortp https://arx.v.orrra////11.....
Zhangyang Wang, Jianchao Yang, Hailin Jin, Eli Shechtman, Aseem Agarwala, Jonathan Brandt, and Thomas S Huang. Real-world font recognition using deep network and domain adaptation. arXiv preprint MANiv:11544000225 2015b.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, elicits reasoning in large lan-
Quoc Le, and Denny Zhou. Chain-of-thouugtt guage models, 2023. URL htt.:tss/aarriv.oog//////// 11903.
Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. A sur- vey on multimodal large language models. National Science Review, 11(12), November 2024. ISSN 2053-714X. doi: 10.1093/ (0009/////n// nsr/nwae403. URL http: http:://xx.ox...... wrg. 1093/ nsr/nwae403.
Simon Zhai, Hao Bai, Zipeng Lin, Jiayi Pan, Peter Tong, Yifei Zhou, Alane Suhr, Saining Xie, Yann LeCun, Yi Ma, et al. Fine-tuning large vision-language models as decision-makite agents via reinforcement learning. Advances in Neural Information Processing Systems, 37:110935-110971, 2024.
Duzhen Zhang, Yahan Yu, Jiahua Dong, Chenxing Li, Dan Su, Chenhui Chu, and Dong Yu. Mm-llms: Recent advances multimodal large language models, 2024a. URL 13601.
Jingyi Zhang, Jiaxing Huang, Sheng Jin, and Shijian Lu. Vlsion-language models for vision tasks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024b.
Jingyi Zhang, Jiaxing Huang, Sheng Jin, and Shijian Lu. Vision-language models for vision tasks: A survey, 2024c. URL httpp:://rxxvv http:///awwwvoooaa////// 22000/00885.........
Xiangyang Zhang. Design and implementation of the chinese character font recognition system based on binary convolutional encoding and decoding network. In 2023 IEEE 5th International Conference Power, Intelligent Computing and Systems (ICPICS), pp. 368-33 068-373.10 IEEE, 2023.
Yong Zhu, Tieniu Tan, and Yunhong Wang. Font recognition based on global texture analysis. IEEE Transactions on pattern analysis and machine intelligence, 23(10):1192-1200 2001.
Abdelwahab Zramdini and Rolf Ingold. Optical font recognition using typographical features. IEEE Transactions on pattern analysis and machine intelligence, 20(8)8877882, 1998.
17

---

Published as a conference paper at COLM 2025
A
Font Introduction
Fonts are generally categorized into three major types: Serif Fonts, Sans-Serif Fonts, and Script & Decorative Fonts.
Serif fonts are well-suited for print materials, formal documents, and books, as their serifs help guide the reader's ss eye, improving the readability of long passages of text. Sans-serif fonts are commonly used in web design, presentations, and modern typographic layouts due to their enhanced clarity on digital screens. Script and decorative fonts are typically employed in posters, logos, and invitations, where individuality and artistic expression are prioritized.
We provide a brief introduction to all fonts used this study, Additionally, Table 4 presents statistical data several popular fonts analyzed in our experiments (Solomons, 2023).
Font Name
Usage Percentage
Popular in Industry
Year Created
Helvetica
25%
Graphic Design
1957
Times New Roman
20%
Publishing
1932
Arial
15%
Business
1982
Roboto
10%
Web Design
2011
Georgia
5%
Digital Media
1993
Calibri
5%
Office Use
2007
Table 4: Some popular fonts used in the paper, along with their typical usage statistics.
A.1 Serif Fonts
Baskerville SC: A classic serif font with small caps styling, Baskerville SC exudes elegance and traditional sophistication. It is ideal for formal documents, book titles, and branding.
Georgia: Designed for readability on screens, Georgia is warm and robust serif font with a classic touch. Its sturdy letterforms make it suitable for both print and digital use.
Merriweather: A modern serif font with slightly condensed design, Merriweather offers pleasant reading experience. It is often used for digital content, thanks to its balanced contrast and legibility.
Times New Roman: A widely recognized serif font, Times New Roman is synonymous with formal and academic writing. Its narrow proportions make efficient for fitting text into limited spaces.
Roboto Slab: A geometric slab-serif font, Roboto Slab combines modern feel with sturdy, industrial aesthetic. It is commonly used for branding, web design, and contemporary editorial layouts.
A.2 Sans-Serif Fonts
Arial: One of the most widely used sans-serif fonts, Arial is known for its clean and simple design. It is commonly used in digital and print media due to its high legibility.
Calibri: A default font in Microsoft Office, Calibri features modern and rounded design. It provides a smooth reading experience and is often used in professional documents.
Consola: A monospaced sans-serif font, Consola is frequently used in coding environments. Its even spacing ensures clarity in programming and technical texts.
18

---

Published as a conference paper at COLM 2025
A timeless and versatile sans-serif font, Helvetica is known for its neutrality and balanced proportions. It is extensively used in corporate branding and signage.
Helvetica:
Monoton: A decorative sans-serif font, Monoton features a retro, futuristic design with bold, high-contrast strokes. It is best suited for eye-catching headlines and artistic typography.
Vera: A clean and modern sans-serif font, Vera provides excellent readability and is com- monly used user interfaces. Its simplicity makes it adaptable to various design applica- tions.
A.3 Script Decorative Fonts
Great Vibes: A flowing script font with elegant curves, Great Vibes is perfect for invitations and sophisticated branding. Its graceful letterforms add touch of luxury to any design.
Lobster: A bold and stylish script font, Lobster features connected letterforms with a casual yet artistic feel. is widely used in branding, menus, and creative projects.
Pacifico: Inspired by 1950s American surf culture, Pacifico is playful script font with relaxed, handwritten feel. It is often used for casual branding, logos, and fun designs.
Satisfy: A charming script font, Satisfy strikes a balance between casual and elegant lettering. It works well for branding, invitations, and creative typography.
B Vision-Language Models
To effectively assess the font recognition capabilities of the latest VLMs, we conduct experi- ments on a selection well-known models.
Closed-Source Models: We evaluate two multimodal models developed by OpenAI: GPT- 40 and GPT-4O-MIN (OpenAT et al., 2024b; OpenAI, 2024), along with two widely recog- nized models from Anthropic and Google: CLAUDE-3.5 5-SNNET and DEMINN--2.0-FLASE 001 (Anthropic, 2024; Google, 2025).
Open-Weight Models: Our experiments include diverse range of open-weight VLMs, spanning nine models across multiple series. Specifically, we test:
Three models from the Qwen multimodal series: DWENN- SwwEN25-VV--B-IINTRUU and QWEN2-VLL-72--INSTRUC (Wang et al., 2024; Bai et al., 2025).
Two models from the latest Llama 3.2 Vision series: LLAMA-3.2-90B-VE INSTRUCT and -LLMAA-.--1D--VVIIONNNSSSS
(Meta AI, 2024).
Two models from the Phi series developed by Microsoft: VIN.355-VISIONNNSTTRCCT and PHI-3-VISION- 12K INSTRUCT (Abdin et al., 2024).
Two models from the Idefics series: IDEFICS--B-LLAMAX and IDEFICS2-8B (Tron- chon et al., 2024).
Here provide brief introduction to all Vision-Language Models utilized in this study.
B.1 Closed-Source Models
Developed by OpenAL, GPT-40 is state-of-the-art multimodal model capable of processing text and images efficiently. It enhances vision-language understanding with improved reasoning, speed, and adaptability.
GPT-40:
19

---

Published as a conference paper at COLM 2025
GPT-4O-MINE: A compact version of GPT-40, GPT-4O-MINI is designed for faster inference while maintaining strong multimodal capabilities. optimized for applications requiring efficiency without sacrificing quality.
CLAUDE-3. S.:NENTTT: Developed by Anthropic, CLAUDE-3 1IDDE-3.--SNNEEE is highly capable vision-language model known for its alignment with human intent. It provides strong contextual understanding and excels in multimodal reasoning tasks.
GEMINI-2.0-FLASH- Created by Google, GEMINI-2.0 GEMINI-2.0-FLARR 0000-F--F--OO11 is optimized vision-language model focused on rapid image-text processing. is designed for applica- tions that demand fast and efficient multimodal comprehension.
B.2 Open-Weight Models
QWEN2-VL-77--NSTRRCYY Part of the Qwen multimodal series, QWEN2-VL-7B-INSTIN is an open-weight vision-language model by Alibaba. It provides balance between effi- ciency and accuracy for multimodal reasoning tasks.
QwEN2:5-VL--B--ISSTRUCC A refined version of its predecessor, QWEN2.5-VL-7B- INSTRUCT introduces improved visual understanding and stronger text-image alignment. It designed for tasks requiring high accuracy in multimodal analysis.
The most powerful model the Qwen2 series, QWEN2-VL- 72B-INSTRUCT features a larger architecture for superior vision-language comprehension. excels in complex tasks that require deep reasoning across modalities.
QWENZ--L--2B-INSTRRUU
LLAMA-32-900--VSIoNNNssnee
A high-capacity model from Meta's Llama series, DAAMA--32-00B-VIsIONNNNS--- is designed for advanced vision-language interaction. Its large-scale architecture enhances performance multimodal understanding.
LLAM--32--IB-VVSIO--NNTTTCCC A smaller variant of the Llama 3.2 Vision series, LLAM--32-11B-VVIIONNNSTTUU offers balance between efficiency and capability. It is ideal for resource-constrainee environments requiring vision-language processing.
PHI-3.5-VISION-INSTRUUCC Developed by Microsoft, SIII-..-VISION-INSS lightweight yet powerful model tailored for multimodal tasks. It is optimized for efficiency while maintaining strong vision-language reasoning.
is
PHI.3-ViSION-123K-IInRRRR
larger-scale version in the Phi series, PHI-3- VISION- 128K-INSTRUCT supports longer context lengths, making it suitable for detailed and com- plex multimodal tasks. It enhancess vision-language applications with broader contextual awareness.
DDEFICS3-SB-LLMMSS A vision-language model from the Idefics series, IDEFICS3-8B- LLAMA3 focuses open-ended multimodal reasoning. It is designed to handle diverse vision-language tasks with improved contextual understanding.
IDEFICS2-8B: Another model from the Idefics family, IDEFICS2-8B provides a strong founda- tion for vision-language applications. It is optimized for tasks requiring nuanced text-image comprehension.
C Implementation Details
C.1 Metric
We use the most straightforward metric: Accuracy. The model's performance is assessed by extracting the predicted labels from the text output and comparing them to the ground
20

---

Published as a conference paper at COLM 2025
truth labels using string matching. The accuracy of a model is calculated follows:
1 Accuracy = L
(1)
where I(-) is an indicator function that is if the condition is true and 0 otherwise, yi represents the true label, !i the predicted label, and N is the total number of samples.
C.2 Model Implementation
For all VLMs used in the experiments, we set the temperature to 0.0 and top to 0.95 to ensure faithfulness and minimize generation randomness, thereby enhancing reproducibility.
For open-weight models, we obtain the weights from HuggingFace and perform inference using h precision. All experiments involving these models are mainly conducted on setup with four NVIDIA A100 80GB GPUs.
For closed-source models, we access the official API for inference. All experiments are conducted between January 25, 2025, and March 25, 2025. To compute the similarity between images for selecting few-shot demonstrations, we employ the openaa/clip-vit-base--ateee model on HuggingFace.
D Capability Gaps Between Open-weight and Closed-source Models
The experimental results indicate that contemporary VLMs struggle with font recognition tasks, with a significant accuracy gap persisting between open-weight and closed-source models.
In the sentence-based font recognition task, which is more commonly encountered in real- world scenarios, three out of four closed-source VLMs achieve least 18% accuracy across various inference settings, with the best west-performing model reaching nearly 67% under the multiple-choice question (MCQ) setting. In contrast, among all open-weight VLMs evaluated in this study, only LLAMA-3 3..--11211 3.AMA-3.2-1118 --I--BBIOON--STTTIIIII
and QWFN2-VL--72 INSTRUCT attain approximately 18% accuracy, while the remaining models fall below 10%, with some approaching 0% accuracy. These findings highlight the substantial performance disparity between open-weight and closed-source VLMs in font recognition.
For the more challenging version of the task, overall performance remains low across both open-weight and closed-source models. The hard version introduces the stroop effect, adding additional cognitive challenge that negatively impacts nearly all VLMs. Under the MCQ setting, all closed-source models are able to achieve accuracy above 6.67%, indicating that, in some cases, they successfully identify font styles rather than being misled by the textual content in the image. In contrast, ween-w models fail to do so in almost all cases, with the only exception of LLAMA-3.22 10A-M---2-00B--VSOOO--N--
Comparative Analysis of Model Accuracy Across Different Fonts
To demonstrate the capability gap of VLMs in recognizing different font types, we evaluate 10 models across 15 distinct fonts and report the overall number of correct predictions under the four inference methods described in Table 1, on both the easy and hard versions of the benchmark.
From the results presented in Figure 11 to Figure 30, it is evident that each model tends to perform well on one or two specific font types, but struggles significantly with the remaining fonts. Notably, there is no single font or even a small subset fonts that all models consistently recognize with high accuracy. Among the tested fonts, Arial and Times New Roman appear to be relatively easier for many models, likely due to their widespread use and higher representation in training corpora. In contrast, other fonts yield highly variable 10MA-3.2-90BB 1-90---IIION---. VISION-INSTRUCT correctly identifies 23 instances of the Pacifico font in the easy version of the benchmark, whereas
performance across models. For instance,
21

---

Published as a conference paper at COLM 2025
despite belonging to the same model family, fails to recognize any instances under identical conditions. This highlights a critical limitation of current VLMs: they lack generalizable font recognition capabilities and exhibit substantial performance disparities across font types.
F
Experimental Prompts
We present the simple prompts employed in the experiments shown in Figures 7 through 10.
Zero-Shot Prompt
What font is used in the text of the image?
Figure 7: Zero-Shot prompt for all experiments in this paper.
Zero-Shot CoT Prompt
What font is used in the text of the image? You need think step by step.
Figure 8: Zero-Shot CoT prompt for all experiments in this paper.
Zero-Shot Multiple-Choice Questions Prompt
What font is used in the text of the image? You must choose one specific font name from the following list: [choices].
Figure 9: Zero-Shot Multiple-Choice Questions prompt for all experiments in this paper.
Zero-Shot CoT Multiple-Choic Questions Prompt
What font is used in the text of the image? You must choose one specific font name from the following list: [choices]. You need to think step by step.
Figure 10: Zero-Shot CoT Multiple-Choice Questions prompt for all experiments in this paper.
G Additional Attention Visualizations
Due to space constraints in the main part of the paper, we give additional visualizations for two ween-weight models to illustrate their attention patterns on the same input image across present attention matrices
different layers. For LLAMA-3.2. -.----...11B---- 2-11B-VISION-INSTRUUCE from all cross-attention layers, highlighting how attention evolves throughout the network. These visualizations reveal that, regardless of whether layer is shallow deep, the model consistently struggles to attend to the edge regions of characters in the image, which are critical for accurate font recognition.
22

---

Published as a conference paper at COLM 2025
Correct Predictions by CLAUDDE-..-SONNE for Each Foot on the Easy Benehnuarts
40
Nanbet Carrec, Predictios
-00
27
20
10
Robato slab
Merrinnaaar
Times New RomaD
Grest Vibee
Lotater
ralibe
Sanady
Gourle
Pasifeu
Bedkrvville SC
capasle
Helretion
Menotoo
Figure 11: Number of correct predictions made by CLAUDE-3.5-SONNE on the Easy Bench- mark for different fonts. The model performs best on Roboto Slab (39 correct predictions) and Merriweather (27 correct predictions), while struggling with fonts like Helevetica (0 correct predictions), Monoton (0 correct predictions), and Vera (0 correct predictions). The total number of experimental trials for each font is 40.
Correct Prodictions by CLAUDE-35-SONET for Each Font on the Hard Benchunark
10
Number of Conrect Prediction
Astal
talier
Georgi
Great Vilao
Labetor
Meriiectller bar
Mosuton
Pasifru
Robato Rab
Tinee New Roman
Settef
Vam
Budereeille S0
Figure 12: Number of correct predictions made by CLAUDF on the Hard Bench- mark for different fonts. The model performs best on Helvetica (11 correct predictions) and Arial (9 correct predictions), while struggling with fonts like Satisfy (2 correct predictions), Vera (2 correct predictions), and Baskerville SC (0 correct predictions). The total number of experimental trials for each font is 60.
23

---

Published as a conference paper at COLM 2025
Correct Prodictions by GPT-40 for Each Foot on the Easy Benchmark
40
40
Nanbet of Carrec, Prodiction
80
25
10
12
10
Arial
Times New Beatenn
callel
Panifio
Grvit Vilee
Goargin
Baskerevilee SC
Lobstir
Meniscenter
Safidd
Helvotinn
Robesto Slab
Viac
Figure 13: Number of correct predictions made by GPT-40 the Easy Benchmark for different fonts. The model performs best on Arial (40 correct predictions) and Times New Roman (40 correct predictions), while struggling with fonts like Helevetica (2 correct predictions), Roboto Slab (0 correct predictions), and Vera (0 correct predictions). The total number of experimental trials for each font is 40.
Correct Predictions by GPT-40 Each Font on Hard Benthmark
Number of Conrect Prediction
20
18
25
Grest Vilao
Times New Roman
Jaleter
Paciffu
Satidy
Astal
bonsala
Baskerevlle SC
Hehweteo
calfeel
Geongi
Merisesther har
Robeto Bab
Vica
Figure 14: Number of correct predictions made by GPT-40 on the Hard Benchmark for different fonts. The model performs best on Great Vibes (32 correct predictions) and Monoton (30 correct predictions), while struggling with fonts like Merriweather (4 correct predictions), Roboto Slab (4 correct predictions), and Vera (2 correct predictions). The total number of experimental trials for each font is 60.
24

---

Published as a conference paper at COLM 2025
Correct Predlcttions by GPT-40-MINT Each Fant on the Easy Benehmark
20
19
18
17.5
Namber of Carree Predictoos
7.5
5.
25
00
Graat Vilwe
Menstop
Goorge
Times New Ronsa
Badereville SC
califed
Helvetion
Labster
Menisranter
Pasifen
Robeto Stah
Safisdy
Ver
Figure 15: Number of correct predictions made by GPT-4O-MINI on the Easy Benchmark for different fonts. The model performs best on Great Vibes (21 correct predictions) and Arial (20 correct predictions), while struggling with fonts like Calibri (0 correct predictions), Lobster (0 correct predictions), and Pacifico (0 correct predictions). The total number of experimental trials for each font is 40.
Correct Predictions by GPT-40-MINI Each Font on the Hard Benchmark
12
10
Number of Conrect Predictions
Grost Vilao
Labetor
Arial
calieel
Georgia
Merrisnadher
Mondtap
Bobatt Bab
Satidf
Times New Roman
Paolfo
nattola
Vam
Budereeille SC
Figure 16: Number of correct predictions made by GPT-4O-MINI on the Hard Benchmark for different fonts. The model performs best on Great Vibes (12 correct predictions) and Lobster (5 correct predictions), while struggling with fonts like Consola (2 correct predictions), Vera (2 correct predictions), and Baskervville SC (0 correct predictions). The total number of experimental trials for each font is 60.
25

---

Published as a conference paper at COLM 2025
Correct Prodictions by GEMINI-220-FL.AE FLASS-O0 Each Fout: on the Easy Benchmarks
40
40
Nanbet of Carrec, Prodiction
80
25
1.
10
Arial
Times New Bration
Hobotts
Grest Vibee
Labsitr
Beskervvlle. SC
calibei
eomala
Gougle
Meriiseaa Bag
Monntop
Pasffro
Rolato Stab
Satisfy
Vicc
Figure 17: Number of correct predictions made by GEMINI-2.O-FLASHHE on the Easy Benchmark for different fonts. The model performs best on Arial (40 correct predictions) and Times New Roman (33 correct predictions), while struggling with fonts like Calibri (0 correct predictions), Pacifico (0 correct predictions), and Satisfy (0 correct predictions). The total number of experimental trials for each font is 40.
Correct Predictions by GEMINI-20 0 FLASH-001 Each Font ou Hard Benchanack
20
211
Nunber of Conrect Predictios
15
10
Asial
Finos New Roman
Great Filko
calibel
nonade
Georgie
Helestaa
Lobster
Merrimather
Monsten
Paciboo
Robelo Sadh
Bookrvelle lif 8G
Saiteds
Vica
Figure 18: Number of correct predictions made by GEMINI-2.0-F FL-SS-00000 FFVINN1-200FFLASSF-00
on the Hard
Benchmark for different fonts. The model performs best on Arial (24 correct predictions) and Times New Roman (20 correct predictions), while struggling with fonts like Baskervville SC (0 correct predictions), Satisfy (0 correct predictions), and Vera (0 correct predictions). The total number of experimental trials for each font is 60.
26

---

Published as a conference paper at COLM 2025
Correct Predictions by ALLAMA-12-9000 1.-.-A-------IIIIIIOX for Each Font ou the Easy Benchrurrk
Nambee of Carrec, Prediction
10
10
Penifoo
Timm New Beateen
Araa
Laboter
Hechetton
Grvet Vilee
Bakkreville SC
califel
couado
Goorgi
Meniseenthrr
Rolato Slab
Satisfy
Vitc
Figure 19: Number of correct predictions made by LLAMA-3.2 ----A-3.--90B- 2-B-- VISION-INSTTUSE the Easy Benchmark for different fonts. The model performs best on Pacifico (23 correct predictions) and Times New Roman (21 correct predictions), while struggling with fonts like Calibri (0 correct predictions), Consola (0 correct predictions), and Georgia (0 correct predictions). The total number of experimental trials for each font is 40.
Correct Predictions by LLAMA-3220U IISOON-ISTRUUTT Each Font on the Hard Benchmnak
Nunber of Conect Prediction
15.0
12.5
7.5
25
0.
Asial
Helvstha
Tutee New Roman
Peellea
calilel
soteads
Govat Vibee
Robato Stbb
Mostston
Georgl
Budervella SC
Leleter
Sattdy
Figure 20: Number of correct predictions made by L1AMA-3.2-990 --I00--ISSION-NNSTRRIIL the Hard Benchmark for different fonts. The model performs best on Arial (19 correct pre- dictions) and Helvetica (12 correct predictions), while struggling with fonts like Baskervville SC (0 correct predictions), Satisfy (0 correct predictions), and Lobster (0 correct predictions). The total number of experimental trials for each font is 60.
27

---

Published as a conference paper at COLM 2025
Correct Predictions by 1II.AA-----IB-IIIII-IINSSSS INSTRUCT for Each Font ou the Easy Benchrurrk
Namber of CCrrece Prediction
15
10
10
Tumes New Romni
Monnioo
Arial
Goungla
Labster
Hchvotia
Baskreville SC
caliiel
couado
Graat Vibes
Meniweenthr
Fasifio
Rolato Stab
Satisfy
Vitc
Figure 21: Number of correct predictions made by 11AMA-3.2-11- B--II--IN--SSTRRIOE the Easy Benchmark for different fonts. The model performs best on Times New Roman (24 correct predictions) and Monoton (18 correct predictions), while struggling with fonts like Calibri (0 correct predictions), Consola (0 correct predictions), and Vera (0 correct predictions). The total number of experimental trials for each font is 40.
Correct Predictions by
LLAMA-32 HIB Each Font on the Hard Benchmnar
15.0
15
Nunbee of Coneet Predictions
12.5
10.0
7.5
5.0
25
0.0
Astal
Times New Roman
Hebvskoa
balibel
nandd
Gmaa Vilter
Lototer
Ba
Pasifru
Goorgla
Menuton
Boowrellle 8C
Rolato Sah
Satlefy
Vem
Figure 22: Number of correct predictions made by on the Hard Benchmark for different fonts. The model performs best on Arial (18 correct predictions) and Times New Roman (15 correct predictions), while struggling with fonts like Roboto Slab (2 correct predictions), Satisfy (2 correct predictions), and Vera (2 correct predictions). The total number of experimental trials for each font is 60.
28

---

Published as a conference paper at COLM 2025
Correct Predictions by VL for Each Font on the Easy Benchmark
AF
40
Nunber of Cotrect Prediciion
90
20
15
10
Arial
Tioser New Reman
callbei
untoota
Georgia
Great Viben
Hokeetia
Lebates
Moze Morciwarlles
Monstau
Paeifso
Saakeff
Vira
Boderrville
Robsto
on the Easy Benchmark for different fonts. The model performs best on Arial (40 correct predictions) and Times New Roman (4 correct predictions), while struggling with all remaining fonts. The total number of experimental trials for each font is 40.
Figure 23: Number of correct predictions made by QWEN2-VL-72B-INSTEE
Coneec Predictions by -WEE--2-L-T2--INSTRICCC for Each Fout on the Hard Benchmack
Number Ccenec Prediccons
Artal
Helvetics
nadibal
da
Gesrpi
Lebator
Murriwattee
Mouston
Parilies
Rudesto Hlab
Touns New Ronino
Satiaf
Baseerwille 83
Great Viller
Viraa
Figure 24: Number of correct predictions made by QWEN2-VL-72D-INSTINE INSTRUCT the Hard Benchmark for different fonts. The model performs best on Arial (9 correct predictions) and Helvetica (8 correct predictions), while struggling with fonts like Baskervville SC (0 correct predictions), Great Vibes (0 correct predictions), and Vera (0 correct predictions). The total number of experimental trials for each font is 60.
29

---

Published as a conference paper at COLM 2025
Correct Prodictions by QWENZVL- for Each Font on the Easy Benchmark
14
12
Nunber of Cotrect Predictoon
Arial
Great Vibee
Timus Roman
Menrbnathhr
Bokervvlle fo?
Gallbal
catoela
Georgi
Hebrotia
Lobetos
Monstau
Paeifso
Saakefs
Vira
Robsto
Figure 25: Number of correct predictions made by QWEN2-VL-7B-INSTRER on the Easy Benchmark for different fonts. The model performs best on Arial (14 correct predictions) and Great Vibes (6 correct predictions), while struggling with fonts like Calibri (0 correct predictions), Consola (0 correct predictions), and Georgia (0 correct predictions). The total number of experimental trials for each font is 40.
Coerect Predictions QWEN2- WWWEN--VL-B-INSTRUU Each Font the Hard Benclmaak
Number of Cocrect Prelicsioss
valibari
cannaaa
Geargi
Helvetia
Meminattio
Rubato Stah
Timm New Roman
Great Vilan
Lobster
Munsio
Pacifka
Vena
Baservollle SC
Sutide
Figure 26: Number of correct predictions made by QWEN2-VL-77-LNSTTE on the Hard Benchmark for different fonts. The model performs best on Arial (5 correct predictions), while struggling with fonts like Baskervville SC (2 correct predictions), and Satisfy (2 correct predictions). The total number of experimental trials for each font is 60.
30

---

Published as a conference paper at COLM 2025
Correct Predictions by QWEN2.5 VI-TI--NSTRUCT for Each Font on the Easy Benchmark
Nambee of Carrec, Prediction
15
10
Arial
Panifis
Times New Roman
Ribesto Slab
Ginal Viles
Beskrvville SC
Menierather
Satidf
Goorgia
calfed
capadle
Lobaies
Menottoo
Vitc
on the Easy Benchmark for different fonts. The model performs best on Arial (23 correct predictions) and Pacifico (14 correct predictions), while struggling with fonts like Lobster (0 correct predictions), Monoton (0 correct predictions), and Vera (0 correct predictions). The total number of experimental trials for each font 40.
Figure 27: Number of correct predictions made by OWEE2.5-VLL-7-INSTTI
Correct Predictions by VI-TII-IRRR Each Font on Hard Benchmark
Number of Conrect Prediction
Arial
PrciBru
Labster
caliel
nattads
Georgia
Hebvstog
ber
Times Roman
Robeto Rab
Satidy
Vere
Badereeille SC
Gimaa Vileer
Monsten
on the Hard Benchmark for different fonts. The model performs best on Arial (7 correct predictions) and Pacifico (6 correct predictions), while struggling with fonts like Baskervville SC (2 correct predictions), Great Vibes (2 correct predictions), and Monoton (2 correct predictions). The total number of experimental trials for each font is 60.
Figure 28: Number of correct predictions made by QWWN..55-VL-7B-INNTTE
31

---

Published as a conference paper at COLM 2025
Correct Predictions by IDEFICSSS SB- LLAMA3 for Each Fout on the Easy Benchmark
14
12
Nunber of Cotrect Predictons
Tince New Romaia
calibei
Lobeite
Georeta
Artal
Budorvville SC
entoela
Great Vilos
Hebrotica
Moustau
Paeifto
Saakefs
Vira
Robsto
on the Easy Benchmark for different fonts. The model performs best on Times New Roman (14 correct predictions) and Calibri (7 correct predictions), while struggling with fonts like Consola (0 correct predictions), Monoton (0 correct predictions), and Vera (0 correct predictions). The total number of experimental trials for each font is 40.
Figure 29: Number of correct predictions made by IDEFICS3-BB-LL.S
Cotrect Predictions IDEFICS3-BB-LLMMA Each Fout Hard Benchmack
Number Coctect Prelicsioss
Thues New Roman
calibri
Hebsrka
Lolster
Arial
eanula
Georgi
Mentewutter
Rubato Slab
Gront Villo
Mansiau
Pacifka
Satady
Vera
Badeveville SC
Figure 30: Number of correct predictions made by IDEHICS3-SB-LLAMA. on the Hard Benchmark for different fonts. The model performs best Times New Roman (6 correct predictions), while struggling with fonts like Baskervville SC (0 correct predictions). The total number of experimental trials for each font is 60.
32

---

Published as a conference paper at COLM 2025
Model: 1.amm-3.2.11 Vision-Iustruct Ground Truth:
Lager Indes: Tayn Indle: 13
Layer Index: 18
Pacifico
Pacifico
Pacifico
10
066
Layer luckst
Layes Index
Layer Index:
Pacifico Pacifico
Pacifico
Figure 31: Visualization of attention weights from six cross-attention layers of LLAMA-3.22 3.2- IIB-VISION-INSTRUC (averaged across all attention heads), illustrating the progression of attention across the model's layers.
Model: Llana-3.2- Wrs.Viiion---eett Instriet Ground Trush: Bashreville
Lager Indere
Layre Fndlee: 13
Layer Indor: 18
10
MERRIWEATHER
MERRIWEATHER
MERRIWEATHER
11
Layer Indle 26 MERRIWEATHER MERRIWEATHER
Layer Incles:
Layer Inder: MERRIWEATHER
Figure 32: Visualization of attention weights from six cross-attention layers of LLAMA-3.2 2-
11B- VI-VISSION-NSTTRCC (averaged across all attention heads), illustrating the progression of attention across the model's layers.
Llama-3.2 11B. Vision- Instrrct &cl Ground
Roboto
Layer Index:
Layee Index 13
Layer Indox
Monoton
Monoton
Monoton
Layer Inckes
Layre Indlee 26
Layer Inder:
Monoton Monoton
Monoton
Figure 33: Visualization of attention weights from six cross-attention layers of LLAMA-3.2 2- LIB-VVISON-IINTRUCC (averaged across all attention heads), illustrating the progression of attention across the model's layers.
33

---

Published as a conference paper at COLM 2025
Model: Llama-3.22 2-1.-.2---1---1111 2--11--Viiooeeee Visson-lnsttrctt &ode Ground Truth: Satisdy
Lager Inder:
Layr Indle 13
Layer Index. 18
Great Vibes
Great Vibes
Great Vibes
16
Layer Inclex:
Layre Indle 26
Layer Index: 33
Great Vibes
Great Vibes
Great Vibes
Figure 34: Visualization of attention weights from six cross-attention layers of LLAMA-3.2- 11 B- V--VISION-INSTRUCT (averaged across all attention heads), illustrating the progression of attention across the model's layers.
Model: Llama-3.2- 11B- Visioninn-nnt & Ground Troth: Times New Roman
Layer
Layre Iidec 13
Layer Indor:
calibri
calibri
calibri
HPPWW 175 PPERROTI
Layer Indexe 23
Layee Indree 28
Loyer Indox: 33
calibri
calibri
calibri
Figure 35: Visualization of attention weights from six cross-altention layers of LLAMA-3.2 11B- VISION-INNSRUUCT (averaged across all attention heads), illustrating the progression of attention across the model's layers.
34

---

Published as a conference paper at COLM 2025
Model: Llamm-3.2-11 11B: Vision-Instruu &ck Ground Truthe Arial
Lager
Tayer
Layer
Yesterday, Max drove quite
Yesterday, Max drove quite
Yesterday, Max drove quite
far to grab zesty pizza
far to grab a zesty pizza
far to grab zesty pizza
10
slice, fresh juice, a weird
slice, fresh juice, a weird
slice, fresh juice, weird
box quirky vitamins, and then headed home.
box of quirky vitamins, and
box quirky vitamins, and
then headed home.
then headed home.
16
Layer Indest: 23
Layee Indrec: 26
Loyer Indox: 33
Yesterday, Max drove quite far to grab zesty pizza
Yesterday, Max drove quite
Yesterday, Max drove quite
far to grab a zesty pizza
far to grab zesty pizza
slice, fresh juice, a weird
slice, fresh juice, a weird
slice, fresh juice, weird
box of quirky vitamins, and then headed home.
box of quirky vitamins, and
box of quirky vitamins, and then headed home.
then headed home.
Figure 36: Visualization attention weights from six cross-attention layers of LLAMA-3.2- 11B- VISION-INNSTUUCT (averaged across all attention heads), illustrating the progression of attention across the model's layers.
Model: Llamn-3.22 3.2mm-3.2-11B-- Visson-Instruul Ground Truth: consola
Lager Index:
Laysr Todle 13
Layor Index: 18
Gabriel watched
Gabriel watched
Gabriel watched
exciting quiz show, then
exciting quiz show,
exciting quiz show, then
drank fizzy soda while
drank fizzy soda while
drank fizzy soda while
10
munching on quick pizza
munching on quick pizza
munching a quick pizza slice for supper, after
slice for supper, after
slice for supper, after
movie.
movie.
movie,
Indes:
Layie 28
Layer Index
0,4
Gabriel watched an exciting quiz show, then drank fizzy soda while
Gabriel watched an exciting quiz show, then drank fizzy soda while munching on quick pizza slice for supper, after
Gabriel watched an exciting quiz show, then drank fizzy soda while
munching on quick pizza slice for supper, after movie.
munching on a quick pizza slice for supper, after movie.
movie.
Figure 37: Visualization attention weights from six cross-attention layers of LLAMA-3.2- 11B- VISIOON-NNSTTUCCT (averaged across all attention heads), illustrating the progression of attention across the model's layers.
35

---

Published as a conference paper at COLM 2025
Mamm--322111 2....11-iiiiiio-nnsstttt Ground Truth: Georgia
Layer Index;
Lays Indox 13
Layer Index.
Priscilla drove to the zoo with Victor, examining
Priscilla drove to the zoo
Priscilla dröve to the 2oo
with Victor, examining
with Victor, examining quirky zebras, foxes, and jovial monkey who loved
quirky zebras, foxes, and a jovial monkey who loved peanuts.
quirky zebras, foxes, and jovial monkey who loved peanuts.
peanuts.
Aotonnio Weight
Lager Indlr: 23
Taysr Trclec 25
Layer Index: 33
Priscilla drove the zoo with Yietor, examining quirky zebras, foxes, and jovial monkey who loved peanuts.
Priscilla drove to the zoo
Priscilla drove the zoo
with Vietor, examining
with Vietor, examinning
quirky zebras, foxes, and
quirky zebras, foxes, and
jovial monkey who loved
jovial monkey who loved
peanuts.
peanuts.
Figure 38: Visualization of attention weights from six cross-attention layers of LLAMA-3.2 2- 11B- VIB-VISION-ISTTUUC (averaged across all attention heads), illustrating the progression of attention across the model's layers.
Model: Llama-3.2-11B- Viston-iin- & Ground Truth: Helvetica
Layer
Layes
Layer Index
Lucas quickly arranged
Lúcas quickly arranged
Lucas quickly arranged
picnic with Zoe, bringing
picnic with Zoe, bringing
picnic with Zoe, bringing
fresh fruits, jazz music,
fresh fruits, jazz music,
fresh fruits, jazz music, extra napkins, and a warm
extra napkins, and a warm
extra napkins, and warm
velvet blanket enjoy.
velvet blanket to enjoy.
velvet blanket to enjoy.
Layar Iudos 23
Layas Indox 25
Layer Indes. 3s
Lúcas quickly arranged a picnic with Zoe, bringing fresh fruits, jazz music, extra napkins, and a warm velvet blanket to enjoy.
Lucas quickly arranged picnic with Zoe, bringing fresh fruits, jazz music, extra napkins, and warm
Locas quickly arranged picnic with Zoe, bringing fresh fruits, jazz music, extra napkins, and a warm velvet blanket to enjoy.
160
velvet blanket to enjoy.
Figure 39: Visualization of attention weights from six cross-attention layers of LLAMA-3.2 2- 11B- VISION-INNSST (averaged across all attention heads), illustrating the progression of attention across the model's 's layers.
36

---

Published as a conference paper at COLM 2025
Model: Llama-3.2-11BB lastru- & Ground Truth: Vera
Layer Index;
Layer Index 13
Layer Index:
Quincy faxed wacky memo to his boss, just highlighting zero budget for extra zebra-themed pencils this week.
Quincy faxed wacky
Quincy faxed wacky
memo his boss, just highlighting zero budget for extra
memo to boss, just highlighting zero budget for extra vebra-themes pencils this week.
10
pencils this week.
14
Laper Index:
Layre Iadez: 26
Layer Indox
Quincy faxed a wacky
Quincy faxed wacky memo to his boss, Just highlighting zero budget for extra zebra-thee toora-themedd
Quincy faxed a wacky memo to his boss, Just highlighting zero budget for extra zebra-themed
memo to his boss, just
highlighting zero budget
for extra zebra-themed pencils this week.
pencils this week.
pencils this week.
Figure 40: Visualization of attention weights from six tross-attention layers of LLAMA-3.22 LIB-VVSSON-INSTRUC (averaged across all attention heads), illustrating the progression of attention across the model's layers.
37

---

Published as a conference paper at COLM 2025
Model: Qwen-2- Doorn-2-VL--TB... VT.-7TB-Tnstruet
Ground Truth:
Layer Indest 16
Layre Indeec: 17
Layer Index: 18
Brianna typed her final exam in cozy library nook, quietly watching zebra crossing video and just sipping fizzy lemonade.
Brianna typed her final exam a cozy library nook, quietly watching a zebra crossing video and just sipping fizzy lemonade.
Brianna typed her final exam in cozy library nook, quietly watching zebra crossing video and just sipping fizzy lemonade.
1.
Layes
Layer ludex:
088
Brianna typed her final exam in cozy library nook, quietly watching zebra crossing video and just sipping fizzy lemonade.
Brianna typed exam in cozy library nook, quietly watching a zebra crossing video and just sipping fizzy lemonade.
Brianna typed her final exam in cozy library nook, quietly watching zebra crossing video and just sipping fizzy lemonade.
04
Neighe Samaani
Layer Indes: 22
Layes Indese 23
Layer Indox: 24
Brianna typed her final exam in cozy library nook, quietly watching zebra crossing video and just sipping fizzy lemonade.
Brianna typed her final exam a cozy library nook, quietly watching a zebra crossing video and just sipping fizzy lemonade.
Brianna typed her final exam in cozy library nook, quietly watching zebra crossing video and just sipping fizzy lemonade.
Layer
Laya 26
Layer
Brianna typed her final exam in cozy library nook, quietly watching zebra crossing video and just sipping fizzy lemonade.
Brianna typed her final exam cozy library nook, quietly watching zebra crossing video and just sipping fizzy lemonade.
Brianna typed her final exam in cozy library nook, quietly watching zebra crossing video and just sipping fizzy lemonado.
Figure 41: Visualization of attention weights from the 17th layer to the final attention layer of DWWEN-VV--B-INSTRUCC (averaged across all attention heads), illustrating the progression of attention across the model's 's layers.
38

---

Published as a conference paper at COLM 2025
Model: Qwen-2-VL- VL-TTB-lsttttee
Ground Truth: consola
Laver Index
Layer Ixder 17
Layer Index
Felix quickly discovered hidden behind the wardrobe, just suggesting bizarre facts about quantum physics.
Felix quickly discovered hidden behind the wardrobe, just suggesting bizarre facts about quantum physics.
Felix quickly discovered hidden behind the wardrobe, just suggesting bizarre facts about quantum physics.
10
Layer lnder
Layrs Index 20
Layer Indox
Felix quickly discovered hidden note behind the wardrobe, just suggesting bizarre facts about quantum physies.
Felix quickly discovered a hidden note behind the wardrobe, just suggesting bizarre facts about quantum physics.
Felix quickly discovered hidden note behind the wardrobe, just suggesting bizarre facts about quantum physics.
Aitenationn Wedglt
Layer luckss
Laye Indes 23
Layer Index.
Felix quickly discovered hidden note behind the wardrobe, suggesting bizarre quantum physics.
quickly discavered hidden note behind the wardrobe, just suggesting bizarre facts about quantum physics.
Felix quickly discovered hidden note behind the wardrobe, just suggesting bizarre facts quantum physics.
Layar Indos 25
Lays Index 35
Layer Indes: 27
Felix quickly discovered hidden note behind the wardrobe, just suggesting bizarre facts about quantum physics.
Felix quickly discovered hidden note behind the wardrobe, just suggesting bizarre facts about quantum physics.
Felix quickly discovered hidden note behind the wardrobe, just suggesting bizarre facts about quantum physics.
Figure 42: Visualization attention weights from the 17th layer to the final attention layer of OWEN2-VL-7B-INSTIE (averaged across all attention heads), illustrating the progression of attention across the model's layers.
39

---

Published as a conference paper at COLM 2025
Model: Qwen-2-VL- -i----22--BBB-sttttrr &cls Ground Truth: Great Vibes
Indert
Layre 17
Layer Index:
Charta guod lankdd twoddy edtn mndlins for Toy's gord sole, heging awaze kungry lolts with verg joezy toppings.
Chartic quictly bolbat bo geftn aosfins for Iag's Groisggrrdd gord hoping omenze hangry folls with orry jaery teppings.
Chartic quickly botkal beodg erten maslins for Iieis goodd sale, hoging to amaze hoegry foltss with woth very, jazzyy topgings
10
Layer Indeer 19
Layer Indrer 20
Layer Index: 21
Charti quickly bolkad fwoayy oxha modion for Tais sole, heging amace hungry folts with vory juezy toppings.
Chartic quichly boload loonyy mofinssooon for Jag's gordd hoping to omanz hongry folts with oery jozzy teggings.
Charlic quiddly batead toody odton modfons for Iryis goodd sole, hoging to cmaze honggy folls with very joazyy topgings.
Attccntion Weight
Indees
Layes 23
Layer lndex.
Chartu quickly baked toody edhn mesfins for Ias's gord heging amace hungry lotts with very juzzyy toppings.
Chariic quichly balaed loody extin mesfos for Iag's goodd sate, hoping omazz hongly folts with orry joarg joeryy togpings.
Charli quiclyy bahad hoody eden maffins for Iaels gord loying to amaze hungry folts woth wothieery very juzzy togsings.
Layer Indls 25
Layes Todoc 25
Layer Indox.
Chastic guackly bolked loady exta madies for Ioi' sole. loginn smice hungry folts with wer jorny toypings.
Charti quichlly bodedd foody oxha maffons for Iay's good hoping omace hungly folbs srith oery jozny tegpings.
Chartic quickly baheed loody moffins for Ioe's good tade, loginn to amaze hongry folts mith juary toppings.
Figure 43: Visualization attention weights from the 17th layer to the final attention layer of MA-N2--VL-B-IITTERR (averaged across all attention heads), illustrating the progression of attention across the model's layers.
40

---

Published as a conference paper at COLM 2025
Model: Qwen-2-VLL VL-TB-Instrue & Ground Trath:
Helvetica
Layer Indert
Layer Indrec: 17
Loyer Indox: 18
Gabriel watched an exciting quiz show, then drank Tizzy soda while munching on a quick pizza slice for supper, after a movie.
Gabriel watched an exciting
Gabriel watched an exciting quiz show, then drank fizzy soda while munching on a quick pizza slice for supper, after a movie.
quiz show, then drank fizzy soda while munching on quick pizza slice for supper, after movie.
Lyye
Layer Gabriel watched an exciting quiz show, then drank fizzy soda while munching quick pizza slice for supper, after a movie.
Gabriel watched an exciting quiz show, then drank fizzy soda while munching quick pizza slice for supper, after movie.
Gabriel watched an exciting quiz show, then drank fizzy soda while munching quick pizza slice for supper, after movie.
Attontion Woight
Layer Indes: 22
Layer Indio 23
Layer Indox: 24
Gabriel watched an exciting quiz show, then drank fizzy soda while munching on a quick pizza slice for supper, after a movie.
Gabriel watched an exciting quiz show, then drank fizzy soda while munching on quick pizza slice for supper, after movie.
Gabriel watched an exciting quiz show, then drank fizzy soda while munching on a quick pizza slice for supper, after a movie.
Layes Gabriel watched an exciting quiz show, then drank fizzy soda while munching quick pizza slice for supper, after movie.
Layer Gabriel watched an exciting quiz show, then drank fizzy soda while munching quick pizza slice for supper, after a movie.
Gabriel watched an exciting quiz show, then drank fizzy soda while munching quick pizza slice for supper, after movie.
Figure 44: Visualization of attention weights from the 17th layer to the final attention layer of BEEE2-VLL-BB-NNTEER (averaged across all attention heads), illustrating the progression of attention across the model's layers.
41

---

Published as a conference paper at COLM 2025
Model: Quenn-2-V--TB-nnstrreet & Ground Teuth: Lobeter
Inclex:
Tayer Indec: 17
Layer Index:
"ucas quickly arranged picnic with Zoe, bringing
Lucas quickly arranged
Lucas quickly arranged
pienic with Zoe, bringing
pienic with Zoe, bringing
fresh fruils. jazz music,
fruitsfusssaarr music,
freshfruitt, fruits. jazz music, extra napkins, and warm velvet blanket to enjoy.
extra napkins, and warm velvet blanket to enjoy.
extra napkins, and warm velvet blanket to enjoy.
Layer Indes: 10
Layre Indre:: 20
Layer lndox: 21
Lucas quickly arranged a
"ucas quickly arranged a
Lucas quickly arranged a
pientc with Zoe, bringing
picnic with Zoe, bringing
pienic with Zoe, bringing
fruils, jazz music,
fruits, jazz music,
Sresh fruils, jazz music, extra napkins, and a warm
extra naphins, and warm
extra naphins, and warm
Atteaston Weight
velvet blanket to enjoy.
veluet blanket enjoy.
velvet blanket to enjoy.
Inder:
Layee
Layer lndex:
Lucas quickly arranged
Lucas quickly arranged
Lucas quickly arranged
HM
plenic with Zoe, bringing
picnic with Zoe, bringing
picnic with Zoe, bringing
fresh fruils, music,
fruitssjaaass music, extra napkins, and warm velvet blanket to enjoy.
fresh fruits, jazz music, extra napkins, and a warm velvet blankel to enjoy.
extra napkins, and warm
velvet blanket to enjoy.
Layer Index: 25
Layee lndrsc 26
Layer Indox: 27
Lucas quickly arranged a plentc with Zoe, bringing fresh fruils, jazz music, extra naphins, and warm velvel blanket to enjoy.
Tueas quickly arranged
Lucas quickly arranged a
plente with Zoe, bringing
plentc with Zoe, bringing
fruits, Jazz music,
fresh fruils, jazz music, extra napkins, and a warm velvel blanket to enjoy.
extra naphins, and warm
velvet blanket to enjoy.
Figure 45: Visualization of attention weights from the 17th layer to the final attention layer of OWEN2-VL--B-INNTTE (averaged across all attention heads), illustrating the progression of attention across the model's layers.
42

---

Published as a conference paper at COLM 2025
Model: Qwen-2-VT. -T---------tBtnnttreet Ground Truth: Merriweather
Layer Indes
Layes Indox 17
Layer Index. 18
Quincy faxed a wacky memo to his boss, just highlighting zero budget for extra zebra- theeedd pencils this week.
Quincy faxed a wacky
Quincy faxed a wacky
memo to his boss, just highlighting zero budget for extra sebra-thenee pencils this week.
memo to his boss, just highlighting zero budget for extra zebra -heemd pencils this week.
10
Lager Indls
Layer 20
Layer
Quincy faxed a wacky memo to his boss, just highlighting zero budget for extra zebra-thoe themed pencils this week.
Quincy faxed wacky memo to his boss, just highlighting zero budget for extra zebra- themed pencils this week.
Quincy faxed wacky memo to his boss, just highlighting zero budget for extra zebra- -eeemdd pencils this week.
Artensioon Weigle
Lager Incles: 22
Tayre Trdle 23
Layer Indox:
Quincy faxed a wacky memo his boss, just highlighting zero budget for extra zebra- themed pencils this week.
Quincy faxed wacky memo to his boss, just highlighting zero budget for extra zebra themed pencils this week.
Quincy faxed a wacky memo to his boss, just highlighting zero budget for extra zebra tetrr-- pencils this week.
Laper Indee:
Layre Indee 26
Layer Indor:
Quincy faxed a wacky memo to his boss, just highlighting zero budget for extra pencils this week.
Quincy faxed wacky memo to his boss, just highlighting zero budget for extra zebra-thee tthemed pencils this week.
Quincy faxed wacky memo to his boss, just highlighting zero budget for extra pencils this week.
Figure 46: Visualization of attention weights from the 17th layer to the final attention layer of OWEN2-VL-7B-INSTRE (averaged across all attention heads), illustrating the progression of attention across the model's layers.
43

---

Published as a conference paper at COLM 2025
Model: Qwen-2-VLas VL-TB-Instrucc &cde Ground Troth: Satisfy
Layer Indest
Layee Indier 17
Loger Index: 18
Priscilla dhove to the joo with
Priscille drove the zoo with
Priscilla drove To zoo with Victor, examining quisky zebras. fores, and joviall monkey who loved peanuts.
Victor. examining quisky zebras.
Victon, examining quirky zebras.
fores, andiaiovial monkey who loved peanats.
foxes, and jovial monkey who loved peanuts.
Indeet
Layee 20
Layer ludex:
Priscilla drove to the zoo with Victor examining quirky zebnas.
Priseillo drove to the joo with Victor examining quirky zebnas. foxes, and a jovial monkey who loved peanuts.
Priscilla drove to the gou with Vietor, examining quisky zebras. foxes, and jovial monkey who loved peanuts.
foxes, and andiajovial monkey who loved peanute.
Atroution Woight
Layer Indese 22
Layer Indioc 23
Layer Indox: 24
Priscilla dhove to the zoo with Victor, examining quirky gebnas. fores, and as jovial monkey whe loved peanuts.
Priseille drove the zoo with Vietor, examining quisky zebras. foxcs, and a jovial monkey whe loved peanuts.
Priscilla dreve to zoo with Vfetor, examining quisky zebras. foxes, and a jovial monkey whe loved peanuts.
1M
ludkes Priscilla drave to the zou with Victor, examining quirky gebnas, fores, jeviall monkey whe loved peanuts.
Layes 26
Layer Indox.
Paiseilla drove to the goo with Victor, examining quirky zebnas, foxes, and a jovial monkey whe loved peanuts.
Priscilla drove to the zoo with Vietor, examining quirky zebnas, fores, and jovial monkey whe loved peanuts.
Figure 47: Visualization of attention weights from the 17th layer to the final attention layer of OWEN2-VL-7BB -----VLL----NSTRCCC (averaged across all attention heads), illustrating the progression of attention across the model's layers.
44


Reviews:
[{"review_id": "ZWIoimgpSn", "reviewer": "Reviewer_A1JS", "summary": "This paper introduces a font recognition benchmark comprising 15 common fonts to evaluate the font recognition capabilities of 13 VLMs. The dataset is divided into two difficulty levels: in the easy version, sentences are rendered in the specified font; in the hard version, the font names are rendered in different font. Experimental results demonstrate that current VLMs perform poorly on this task, and that additional techniques such as chain-of-thought prompting or few-shot learning do not yield substantial improvements. VLM errors in font recognition are categorized into four types. Attention matrices of two open-source VLMs are analyzed.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) This paper systematically investigates VLMs’ performance on font recognition by proposing a benchmark with two levels of difficulty, exploring the Stroop Effect’s impact on model behavior.\n2)This paper evaluates 13 mainstream VLMs, including both open-source and closed-source models, offering a thorough performance comparison.\n3) This paper provides visualizations of the attention matrices for LLAMA-3.2-11B-VISION-INSTRUCT and QWEN2-VL-7B-INSTRUCT, shedding light on how these models attend to font cues.", "reasons_to_reject": "1) This paper requires VLMs to precisely distinguish certain fonts, which may be overly demanding due to visually near-identical typefaces (e.g., Helvetica vs. Arial). For the “Indecisive Classification” error category, it would be informative to know whether the incorrect predictions are visually similar to the target font.\n2) The paper lacks attention matrices visualizations for QWEN2-VL-7B-INSTRUCT on the hard version of the task, leaving that part of the study incomplete.\n3) The benchmark could be more comprehensive. The easy version uses full sentences, while the hard version uses single words. Sentence format can also be used to introduce the Stroop Effect. Additionally, font size is not controlled across the two versions, potentially confounding the results.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "q1tNiJNCVd", "reviewer": "Reviewer_FEXv", "summary": "This paper tests VLMs for their ability to recognise the font used in an image. For this purpose, the paper introduces a dataset that contains samples from a select set of fonts. As an additional twist, the paper also introduces a kind of stroop effect test, in that for each of the fonts there are images where the text set in this font is the name of a different font. The authors ran various experiments using frontier closed and open models, and testing single shot as well as few-shot and CoT prompting. The performance of even the biggest models is not great. An attention analysis shows that the models appear to focus on less relevant details.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- clear question, well executed", "reasons_to_reject": "- somewhat limited scope / special interest\n- no comparison to specialised models\n- discussion of wider impact (inability of model to reflect on the modality of information and resolve conflict) somewhat limited", "questions_to_authors": "Please let us not normalise moving essential sections of the paper like the discussion of related work into the appendix. (Line 85.) The related work section that is in the main body of the paper is actually ok, so it's not a problem here, but let's just not start with this.", "extra_scores": {}}, {"review_id": "DFrAWffO3w", "reviewer": "Reviewer_P7Qz", "summary": "This paper proposes two benchmarks to assess existing VLMs' ability in font recognition. The first is to predict one of 15 fonts from an image of a sentence, and the second is to predict a font from an image of a font. Since the word is about fonts, the second benchmark is more confusing. On these two benchmarks, this paper finds that existing VLMs perform worse and are easily deceived by the second benchmark. This paper further analyzes the categorization of errors and the attention map of cross-attention layers in VLMs.", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper identifies serious issues of VLMs' ability to recognize fonts.\n\n2. This paper is easy to follow.", "reasons_to_reject": "1. This paper does not compare previous work on font recognition, including benchmarks and specific models. Thus, I can't identify its unique and significant contribution. For example, is the previous benchmark suitable for evaluating LLMs?\n\n2. This paper only compares features of CoT and few-shot and does not consider fine-tuning or test-time scaling. Fine-tuning is a reasonable way to perform specific tasks.\n\n3. This paper exploits attention scores to explain why VLMs perform badly,  but even if the vlms perform good on one domain, their attention scores may not be distributed as you expect.", "questions_to_authors": "1. Which attention layer do you choose to print attention scores? How do you convert 2-dimensional matrices to 1-dimensional matrices?\n\n2. What is the use of font recognition? Have other papers in this domain directly used or fine-tuned VLMs?", "extra_scores": {}}, {"review_id": "EDCDtSFjKl", "reviewer": "Reviewer_A9ZJ", "summary": "The paper presents a new benchmark with 15 popular fonts, and shows the results from extensive benchmarking of visual language models in their ability to recognize the fonts. The practical need to recognize font is well motivated. The experiments are well executed and the paper is very clearly written. There is an effort to explain the poor performance of the models by analyzing their attention. This is the part I found least informative. The most interesting part is the hard condition, in which a font name is written in a different font. Results plummet in this condition.", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Well motivated, well executed experiments, described well.\nThe results are useful in documenting the weaknesses of current VLMs for the task.", "reasons_to_reject": "Not really a reason to reject, but a drawback is that there is no proposal of how to teach models font recognition. A comparison with supervised simpler models is also missing.", "questions_to_authors": "Is it really reasonable to expect that the ability to recognize fonts would emerge without any training on at somewhat related tasks? Is there a way to elicit the font names a model knows? I am not suggesting such experiments as condition for accepting the work, just wondering based on the presented content.\n\nCan the CNNs mentioned in the intro be benchmarked against the VLLMs? Mostly I am looking for some sort of a strong take away or recommendation, like \"include font-related task in the training mix\", \"it is not worth trying to coax VLLMs to do the task\", or something of the sort that can serve researchers who may be interested in the task.", "extra_scores": {}}]

Author rebuttal:

Meta-review: Reviewers are somewhat split between borderline and positive. The paper presents an interesting effect of fonts on the performance of VLMs, as noted by at least two reviewers. Reviewers who are more negative had concerns about experiments that were in the opinion of the AC, fully satisfied for reviewer 1 (A1JS) but this is not reflected in the review score. Given that this is a somewhat unconventional topic, the AC sides more on the positive side for this work despite some reviewers' reservations.

[Automatically added comment] At least one review was discounted during the decision process due to quality]
Decision: Accept