Textural or Textual: How Vision-Language Models Read Text in Images
Authors: Hanzhang Wang
Qingyuan Ma
Venue: ICML 2025 poster (2025)
Keywords: Typographic Attack
Vision-Language Models
Intrinsic Dimension
CLIP
Primary area: social_aspects->accountability_transparency_and_interpretability
TL;DR: This study explores how multimodal visual models represent textual semantics and the mechanisms through which text disrupts visual understanding.

Abstract: Typographic attacks are often attributed to the ability of multimodal pre-trained models to fuse textual semantics into visual representations, yet the mechanisms and locus of such interference remain unclear. We examine whether such models genuinely encode textual semantics or primarily rely on texture-based visual features. To disentangle orthographic form from meaning, we introduce the ToT dataset, which includes controlled word pairs that either share semantics with distinct appearances (synonyms) or share appearance with differing semantics (paronyms). A layer-wise analysis of Intrinsic Dimension (ID) reveals that early layers exhibit competing dynamics between orthographic and semantic representations. In later layers, semantic accuracy increases as ID decreases, but this improvement largely stems from orthographic disambiguation. Notably, clear semantic differentiation emerges only in the final block, challenging the common assumption that semantic understanding is progressively constructed across depth. These findings reveal how current vision-language models construct text representations through texture-dependent processes, prompting a reconsideration of the gap between visual perception and semantic understanding. The code is available at: https://github.com/Ovsia/Textural-or-Textual

Textural or Textual: How Vision-Language Models Read Text in Images
Hanzhang Wang" Qingyuan Ma
Abstract
given that textual elements, while structurally and sym- bolically distinct from visual objects, may nevertheless be encoded in ways that resemble general visual pattern recog- nition rather than language speuuiiieseeificccc processing. This echoes findings in cognitive neuroscience suggesting that the brain regions supporting visual word recognition (e.g., the visual word form area) are recycled from object recognition sys- tems and may initially process written words similarly to visual objects before being modulated by linguistic feedback (Carreiras et al., 2014). Such tendency raises concerns that these models may be aligning low-level textures rather than achieving a deeper, cross-modal semantic integration. Moreover, given the hierarchical nature of neural network processing, it remains unclear at what stage textual features begin to influence an image's semantic interpretation. These challenges motivate our study, which aims to disentangle textual and textural representations to better understand how vision-language models encode and form textual semantics.
Typographic attacks are often attributed to the ability of multimodal pre-trained models to fuse textual semantics into visual representations, yet the mechanisms and locus of such interference re- main unclear. We examine whether such models genuinely encode textual semantics or primarily rely texture-basse based visual features. To disen- tangle orthographic form from meaning, intro- duce the ToT dataset, which includes controlled word pairs that either share semantics with dis- tinct appearances (synonyms) or share appearance with differing semantics (paronyms). A layer- wise analysis of Intrinsic Dimension (ID) reveals that early layers exhibit competing dynamics be- tween orthographic and semantic representations. In later layers, semantic accuracy increases as ID decreases, but this improvement largely stems from orthographic disambiguation. Notably, clear semantic differentiation emerges only in the fi- nal block, challenging the common assumption that semantic understanding is progressively con- structed across depth. These findings reveal how current vision-language models construct text rep- resentations through rexture-depender processes. prompting a reconsideration of the gap between visual perception and semantic understanding. The code available at: https: thub. com/ Ovsia//ai Taaaaaeeeutraaaae or-oor---oo Textual -reettuua
One significant manifestation of these uncertainties is typo- graphic attacks (Goh et al., 2021), which highlight vulner- abilities in vision- rision-language models when interpreting text within images. These attacks embed misleading text into images, causing misclassification driven by the misalign- ment between textual and visual modalities. For instance, an image of dog overlaid with the word "laptop" may be mis- classified as an electronic device (Lemesle et al., 2022). As models like GPT-4v (Yang et al., 2023) become increasingly capable, these vulnerabilities raise pressing security con- cerns, including the risk of unintended command execution resembling model "jailbreaking" (Gong al., 2023; Robey et al., 2023; Wang et al., 2023). Addressing such vulnera- bilities requires a deeper understanding of how typographic attacks exploit the models' internal representations.
1. Introduction
While laiiunla models have demonstrated the ca- pacity to process textual content embedded within images, fundamental question remains: do they genuinely capture the semantic meaning of the text, merely treat it as other visual pattern? This distinction is especially pertinent
A Although typographic attacks may not conform to traditional definitions of adversarial perturbations, they nonetheless ex- pose how vision- lasiunnl models can conflate visual text with image content, reflecting fragile alignment between symbolic and perceptual representations. These models of- ten exhibit implicit associations between textual cues and corresponding visual concepts (Cao et al., 2023), suggesting a shared embedding space where semantically related modal- ities are co-located. For instance, model might position the visual features of cat near both the textual token "cat" and its associated conceptual representation. However, it
School of Computer Engineering and Science, Shanghai Uni- versity, Shanghai. China. Correspondence to: Hanzhang Wang <hanzhang, mon. wang @ @ooggggmailloom gomi.
Proceedings of the nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Textural or Textual: How Vision-Language Models Read Text in Images
remains unclear whether such proximity reflects genuine se- mantic understanding or arises from superficial correlations learned during pretraining. Disentangling these factors is essential for understanding the robustness and interpretabil- ity of vision-language representations under multimodal perturbations.
performance between the original image and the typo- graphic classification, achieving significant improve- ments across diverse defense scenarios.
2. Related Work
These observations raise an important question: to what ex- tent do language models distinguish between the vi- sual appearance of text and its semantic content? To explore this, we use Intrinsic Dimension (ID) to quantify the com- plexity of visual representations by measuring the degrees of freedom required to encode them accurately. Rooted in high-dimensional geometry, ID has been widely applied to the analysis of neural network behavior, particularly in the context of adversarial robustness (Amsaleg al., 2017: Ma et al., 2018). We extend this approach by applying ID to the ToT dataset, which introduces controlled variations in both orthographic form and semantic meaning. This al- lows us to examine how visual and conceptual sources of complexity affect the internal representations learned by lasion-language models.
2.1. Typographic Attacks
CLIP (Radford et al., 2021) is known for its ability to joint understanding of language and vision. Due to its large amount and spin of training images, many of which incor- porate both visual and textual features, it can read visually presented text, or scene-text (Materzyíska et al., 2022; Cao et al., 2023). A notable aspect of CLIP is its tendency, in certain instances, to rely predominantly on text for image classification. This reliance can lead to what's 's termed a ty- pographic attack (Goh el al., 2021), where misclassífication occurs due to overemphasis on text.
In response to such vulnerabilities, various defense strate- gies have been explored. (Materzyúska al., 2022) imple- ment linear transformation to bifurcate the model into two distinct streams: one dedicated to visual information and the other to textual data. (Azuma & Matsui, 2023) introduce the Dense-Prefix token in conjunction with prompt learning, placing it before class names to significantly enhance accu- racy against real-world typographic attack scenarios. PAINT (Iharcco et al., 2022) involves a method that interpolates be- tween a model's pre- and post-fine-tuinng weights, showing notable success in mitigating typographic attacks. (Cao et al., 2023) takes a different route by filtering out dataset samples containing text regions within images, leading to not only improved defense against typographic attacks but also heightened accuracy in other tasks.
We introduce the ToT (Textural or Textual) typographic attack dataset, which contains text overlays that seman- tically consistent, irrelevant, or nonsensical with respect to the underlying image. This setup allows us to examine how a pre-trained vision-language model responds to different types of text-image relationships. We further introduce 10 garonym-synonyae pairs designed to disentangle how the model represents orthographic similarity versus semantic relatedness. Our analysis shows non-linear progression in representational dynamics: earlier layers refine texture-level features that facilitate surface-leve text recognition, with semantic distinctions emerging only in the final block of the network. Specifically, the main contributions of this work are as follows:
2.2. Disentangling Visual and Textual Semantics in Vision-Language Models
We provide a systematic analysis of how vision- language models process typographic attacks, using intrinsic dimension (ID) to quantify representational complexity across network layers. Our results show that improvements in semantic decoding accuracy are largely attributable to orthographic recognition, which relies on texture-based features rather than abstract se- mantic understanding. This challenges the common assumption that semantic representations are gradually constructed across layers. Clear semantic differentia- tion emerges only at the final block.
Large vision-and-language pre-trained models like CLIP (Radford et al., 2021) showease their efficacy through exten- sive pre-training on diverse datasets, excelling in tasks such as image classification (Zhang al., 2022), visual question answering (VQA) (Song et al., 2022), and image captioning (Mokady et al., 2021). The treatment of visually presented text within these models sparks debate in the field. Some re- searchers recommend removing the language representation from the visual aspects of the model (Materzyúska et al., 2022: Cao al., 2023). In contrast, other researchers un- derscore the indispensable role of language comprehension in tasks like Text- VQA and Text-Captioning (Yang et al., 1021; Kil et al., 2023). They advocate for a harmonious in- tegration of visual and textual information, pointing out that such synergy is crucial for a more holistic understanding of images.
Building on these observations, we defend against ty- pographic attacks by simply fine-tuning only the fi- nal block of the model to better distinguish between textural and textual representations. Experimental re- sults show that our strategy effectively balances the
2

---

Textural or Textual: How Vision-Languaal Models Read Text in Images
GROp
daisy
fish
Cwd
goose
zebra
PrKqlU
ma DIE
lix
Original
Consistent
Irrelevant
Nonsense
Figure 1. Example images from Subset of the ToT dataset, illustrating consistent, irrelevant, and semantically nonsensical image-text pairs
In line with this debate, our research undertakes series of comparative experiments focusing CLIP's Vision Trans- former (Dosovitskiy et al., 2020). These experiments aim to unravel the intricate dynamics between scene-text recogni- tion and the multi-modal properties inherent in CLIP. Ad- dressing the complexities of multi-modal models, particu- larly their challenge in differentiating visual elements from textual semanties, our study seeks to fine-tune this delicate balance. We endeavor to enhance the model's capability to discern physical objects from scene text, thereby enrich- ing its understanding and interpretation of both visual and textual components a unified and coherent manner.
3. Method
3.1. ToT Datasets
3. SUBSET 1: SEMANTIC CONFUSION
We propose the ToT (Textural or Textual) dataset, derived from ImageNet- Ik, which features 100 categories of com- objects overlaid with texts of varying semantics. The dataset contains 50,000 images, with 500 randomly selected images per category. These categories represent frequently encountered real-world objects with short, distinct names and minimal semantic overlap, making the dataset highly relevant for studying typographic attack scenarios practi- cal contexts. Figure illustrates the three types of textual modifications applied to the images generate a diverse set of compositions.
2.3. Intrinsic Dimensions of Multimodal Representation
The Intrinsic Dimension (ID) is the minimum number of dimensions required to represent data effectively (Levina & Bickel, 2004). In neural networks, ID is derived from the model's representations, indicating the fewest parameters needed to capture specific features (Amsaleg al., 2015). Ansuini et al. (2019) demonstrated correlation between the final layer's 's ID and the model's accuracy, noting that ID typically follows a hunchback hunchback-shaped shaped curve across layers, reflecting the learning process (Ansuini et al., 2019). More- over, ID is crucial for interpreting learned representations and exploring its relationship with neural network training (Aghajanyan al., 2020; Pope et al., 2021). Basile al. (2025) introduced correlation-basee metric that uses local intrinsic dimensionality to reveal non-obvious connections between paired visual and textual features, which standard distance or similarity metrics fail to capture.
The Original subset consists of unaltered images from ImageNet-Ik, serving as a baseline. In the Consistent subset, each image is overlaid with its matched category label, preserving semantic alignment. The Irrelevant sub- set pairs each image with a randomly selected label from a mismatched ImageNet-1k category, ensuring the overlaid text is a valid label but semantically irrelevant to the image content. The Nonsense subset overlays each image with a string formed from random combinations of letters, averag- ing six characters in length and resembling the structure of category names, but carrying no semantic meaning.
1.2. SUBSET 2: PARONYMS VS. SYNONYMS CONFUSION
Since the form of a word is often intrinsically linked to its meaning, variations in word structure typically lead to words with distinct semantic differences. This suggests that neural networks may distinguish words based solely on superficial textural features, leading to what appears to be semantic- level comprehension. To explore this hypothesis, we design subset of 10-word pairs specifically aimed disentangling the relationship between word form and meaning.
Amsaleg et al. (2017) and Ma et al. (2018) used local ID to assess adversarial robustness, finding that LID increases with noise in adversarial perturbations. This connection em- phasizes how ID influences model's 's vulnerability. Tulchin- skii et al. (2024) further explored ID in textual data, reveal- ing that human-genaee -eneratee texts have an average ID of 7 to 9, while Al-generated texts often fall below 1.5. This distinc- tion enables classifiers to effectively differentiate between human and AI- Al-generated content.
This subset explores how models differentiate between words that are visually similar but semantically distinct,
3

---

Textural or Textual: How Vision-Language Models Read Text in Images
Algorithm Intrinsic Dimension Estimation Across Layers
goose
goose
Require: n2 number of images; A: number of layers;
model(-, A): layer-wise representation
Ensure: ID[A]: estimated intrinsic dimension at each layer <- randomly select v images
for À = 1 to do
moose
gander
Z [ < mS do Compute d1, d2: distances nearest neighbors of
for i
[
Paronyms Pair
Synonyms Pair
R + d1/d2
end for
Figure 2.
Image-text examples from Subset 2 of the ToT dataset with paronym and synonym word pairs, designed isolate ortho- graphic and semantic similarity.
Estimate ID[A] from the distribution of end for
return ID
lyzing the distances between nearest neighbors in dataset. Algorithm details the procedure applied to layers in pre-trained model. Specifically, model A) extracts the representation of layer À for the image set S, and ID[A] stores the estimated ID values for each layer.
as well as those that share semantic meaning but have dif- ferent visual forms. Each of the 10-word pairs consists base word (selected from the ToT dataset) and two related words: the Paronyms Pair, which refers words that are visually similar but differ in meaning. and the Synonyms Pair, which refers to words that have similar meanings but distinct spellings. All words are real-world entities and are commonly used. For example, as shown in Figure 2, 'goose' is paired with 'moose" as its Paronyms Pair and 'gander' as its Synonyms Pair. This dataset enables a detailed analysis of how models process both visual and semantic similarities in language.
The algorithm computes the distance ratios R[i de for each sample, where di and d2 are the distances the first and second nearest neighbors, respectively. For higher in- trinsic dimensions, the ratios R follow a Pareto distribution, denoted as Pa(d + 1). This relationship is captured by the likelihood function:
d2
3.2. Estimating the Intrinsic Dimensions of ToT Datasets
P(R(d) dN II
R 4-1
For images containing varying levels of semantic complex- ity, estimate the Intrinsic Dimension (ID) of their rep- resentations layer by layer. As Algorithm 1 shows, this process involves using the ID's magnitude as a metric to evaluate how specific layers the model articulate the tex- tual semantics embedded within the images.
where P(R d) represents the likelihood of observing the distance ratios R given an intrinsic dimension d. A linear fit on the fog-transformed distance ratios is then used to estimate the intrinsic dimension, maximizing the likelihood under the assumed Pareto distribution.
Our study focuses on ViT-based vision models, particularly CLIP ViT-B/16 (cli), given its widespread use in multi- modal pretraining, The results on additional architectures are presented in the appendix. CLIP ViT-B/16 is based on Vision Transformer (ViT) architecture with 12 transformer blocks, and our analysis examines the representations ex- tracted from each block. Within each block, we evaluate three key layers: Attn refers to the output of the attention mechanism after the linear transformation. c c_fc is feed- forward expansion layer that projects features from 768 to 3072 dimensions. proj is projection layer that reduces the dimensionality from 3072 back to 768.
4. ID Analysis of Textual and Textural Representations in Typographic Attacks
4.1. Layer Sensitivity to Typographic Attacks
To analyze how different layers of visual models respond to semantic variations, we first cluster image representa- tions with various textual overlays using t-SNE, offering a preliminary two-dimensional visualization. However, since t-SNE is limited to low-dimensional spaces, we further esti- mate the intrinsic dimensions (ID) of each layer to evaluate whether intermediate layers preserve semantic distinctions in nigher-dimensione spaces.
The TwoNN algorithm (Facco et al., 2017) estimates the intrinsic dimension (ID) of visual representations by ana-
Representation Clustering. We sample image represen-

---

Textural or Textual: How Vision-Language Models Read Text in Images
Img.a langen
Ing. ioddu Ordovant
Foxt, rceralienti
Img.- intraated
Ing. "ian Inrisvatii
Ing. Tor toro
trcosesst
0.2
0.4
0.6 Relative Depth Network
0.8
1.0
Figure 3. t-SNE visualization of representations from ViT-B/16 in different depths. Early layers show two clusters primarily based on image backgrounds, while the final layer shows cight clusters corresponding to all semantic categories.
tations from different network depths and visualize them using t-SNE (Van der Maaten & Hinton, 2008), shown in Figure 3. In the earlier layers, the representations form two clusters, likely reflecting variations in image background content. In contrast, the final layer reveals eight distinct clusters, capturing a combination image and text seman- tics. This pattern suggests that multi-modal visual models initially treat text as visual texture before gradually incor- porating semantic information in deeper layers. To explore this hypothesis further, we estimate the intrinsic dimensions (ID) of representations across layers in the next section.
35
Origin
Consistent
30
Irrelvannt
Nonsense
25-
20
10
10
15
20
25
30
35
Layer
Intrinsic Dimensionality Estimation. We randomly sam- pled 2,000 images from each subset of the ToT dataset to estimate the Intrinsic Dimension (ID) of representations in the CL.PP visual model. The results, shown in Figure 4, reveal a swell-shrink pattern across network layers: repre- sentation complexity initially increases before decreasing. This behavior, consistent with prior findings in CNN visual models (Ansuini et al., 2019; Muratore et al., 2022), also emerges in Transformer-base formation bottleneck theory (Shwartz-Ziv & Tishby, 2017), which describes an early fitting phase followed by compres- sion. Notably, while ID values fluctuate across layers, their ratios to the original image's ID remain stable. Typogra- phy is observed to consistently increase representational complexity by factor of 1.2 to 1.3 across most layers.
Figure 4. The Intrinsic Dimension (ID) variations of ViT-B/16 on the ToT datasets.
aspect of representations. Notably, when the text closely relates to the image content, it appears to reduce represen- tational complexity, as indicated by lower ID values in this layer.
models, aligning with the in-
4.2. Disentangling Textual and Textural Representations
To better understand the findings in Section 4.1, 1, we design two experiments to disentangle orthographic and seman- tic representations and analyze their effects on typographic attack. Semantic Constancy using the subset 1 of ToT to examines how variations in font size affect texture-level representations while keeping semantic content unchanged. Linear Probe leverages paronym paronym-synonym pairs (subset 2 of ToT ) to assess the model's progressive disentanglement of textual and textural features across layers. Despite their different implementations, both experiments aim to char- acterize the interaction between texture and semantics in vision-language models.
However, in the final block, nonsensical and irrelevant sub- sets show significantly higher IDs than the original, while consistent images exhibit notable decrease. This discrep- ancy, particularly pronounced in the last layer closest to the classification layer, suggests that the final block has significant impact on the semantic representation of the en- tire image. In addition to ViT-B/16, we also tested the IDs of multiple visual encoders with different architecture, the results are shown in Figure 9 in the appendix.
Semantic Constancy with Varying Font Size. Despite differences in visual appearance, images containing text of varying sizes are often perceived as semantically equiva- lent (Figure 8, appendix). This raises the question of how visual models encode such variability while preserving se-
Overall, typography increases the complexity of representa- tions in the intermediate layers, regardless of the semantic relationship between the image and text. However, in the final block, text overlay primarily influences the semantic
5

---

Textural or Textual: How Vision-Language Models Read Text in Images
Table Accuracy (%) on the ToT dataset for multimodal (CIII, MetaCIIP) and vision-only (ViT, DINOv2) models across different semantic contexts and font sizes. Numeric suffixes (e.g.. 80 or 120) indicate font size. Cons, Nons, and Irr refer to consistent, nonsensical. and irrelevant textual content, respectively. Examples are shown Figure (Appendix).
Orig
Cons 80
Nons 80
Irr 20
Irr 40
Irr 60
Irr 80
Irr 100
Irr 120
CLIP
86.6
98.4
80.4
78.8
60.7
49.2
42.9
40.5
38.9
MetaCLIP
84.7
94.6
79.3
83.3
79.6
75.2
72.1
70.0
68.4
ViT
91.1
86.8
86.3
91.0
89.9
88.9
87.1
85.9
84.2
DINOv2
82.8
82.4
82.0
82.7
82.6
82.4
81.8
80.7
80.2
mantic consistency. We compare multimodal models (CLIP (Radford al., 2021) and MetaCLIP (Xu et al., 2023)) with unimodal vision models (ViT-B/16 (Dosovitskiy al., 2022) and DINOv2 (Oquab et al., 2023)), evaluating their perfor- mance across variations in text size and semantic context. To ensure a fair comparison, both models share the same ViT-B/16 backhone architecture. The ViT-B/16 model, pre- trained on ImageNet- Ik (Russakovsky, 2015), serves as the baseline for vision-only models. Both models are evaluated on the ToT dataset, with results presented in Table 1.
Linear Probe across Different Blocks
1.00
Paronym Synonym
0.65
0.55
0.10
0.00
Multimodal models exhibit marked sensitivity to both the semantics and the visual appearance of overlaid text. For instance, CLIP achieves 98.4% accuracy in the Cons_80 condition, but its performance drops to 42.9 92.9% under Ir0, despite the font size remaining constant. This suggests that the model's internal representations are shaped not only by surface-level features, such as texture, but also by the semantic alignment between text and image.
Relative Depth
Figure 5. Linear probe results: Classification accuracy of each layer for paronym pairs (pink) and synonym lines represent individual pairs, while darker lines show the average accuracy for each set of 10 pairs.
pairs (orange). Lighter
ment for textual features is concentrated in the layers closer to the output.
In contrast, vision-only models show minimal performance variation across Cons_80, Nons_80, and Irr_80, regardless of semantic content. This contrast indicates that while mul- timodal models integrate textual meaning into visual un- derstanding, pure vision models primarily treat text texture, not language.
4.3. Tracking Semantic Emergence through Intrinsic Dimension
To investigate how visual models progressively form textual representations across layers, we analyze the relationship between intrinsic dimension (ID) and semantic decoding performance, as shown in Figure 6. The combined results reveal two-phase trajectory- initial representational ex- pansion followed by compression-whic which aligns with the information bottleneck theory (Saxe et al., 2019). Within this broader pattern, we identify four functionally distinct stages: initialization, texture dominance, compression, and semantic integration.
Linear Probe on Paronym-Synonym Pairs. We apply linear binary classifier probe to the final outputs (In_2 2 layer) of all 12 Residual Attention Blocks for both Synonyms and Paronyms pairs. For each pair in subset of the ToT dataset, we use 320 image samples for training and 80 for testing. Following the approach used in CLIP's linear probe experiments, employ logistic regression the classifier.
As shown in Figure 5, each lighter-colored line represents paronym pair (orthographically similar, pink) or a synonym pair (semantically similar, orange), the darker lines indicate the average accuracy of the corresponding 10 pairs. It is evident that all layers achieve higher accuracy when classi- fying based orthographic similarity. However, the layers with the steepest slopes for these curves show a distinct pat- tern: the significant improvement for texture features occurs primarily in the middle layers, whereas the notable enhance-
Random Initialization. the early layers, representational complexity increases gradually, but neither orthographic nor semantic features are reliably formed. Both paronym and synonym classification accuracies remain near chance.
Texture Dominance. Intrinsic dimension rises substantially, reflecting increasing representational complexity. Ortho- graphic decoding (i.c., paronym classification) improves rapidly, driven by heightened sensitivity to low-level visual
6

---

Textural or Textual: How Vision-Language Models Read Text in Images
Swell
Shrink
Random
Competition Collaboration (Téxture > Textual)
30
Origin
Consistent
0.95
Irrelvvnnt
25
Nonsense
Paronym
0.90
Linear Probe Acc
Synonym
0.85
20
0.80
15
0.75
0.70
10
0.65
0.0
0.2
0.4
0.6
0.8
1.0
Relative Depth
Figure 6. Combined visualization of intrinsic dimension (ID) and linear probe accuracy (from Figures 4 and 5), illustrating the progression of text representations across network layers.
features. During this stage, textural and semantic signals initially compete but gradually begin co-activate within shared representational space.
form.
(3) Late-Stage Semantic Differentiation. clear separa- tion based semantic content emerges only at the final block, marking shift from texture-level correlation to se- mantic abstraction.
Compression. In the mid-to mid-to-late layers, the model under- goes sharp compression, evidenced by steep drop in ID. Semantic decoding accuracy (e.g., synonym classification) increases markedly during this period. However, no signifi- cant divergence ID is observed across different semantic categories, suggesting that the learned representations re- main largely texture-dependent.
5. Defense Against Typographic Attacks through Fine-Tuning
Building on the findings in Section 4, different layers of the visual model encode text in distinct ways, influencing its vul- nerability to typographic attacks. This suggests targeted defense strategy: fine-tunin tne-tuning specific layers enhance ro- bustness. In this section, we explore two approaches, se- lectively fine-tuning individual blocks (Section 5.1) and fine-tuning only the last block (Section 5.2), to assess their effectiveness in mitigating typographic attacks.
Semantic Integration. In the final block, semantic accuracy peaks, and ID values diverge significantly across semantic categories. This indicates that semantic variation increas- ingly reflected structure of the learned representations.
Together, these observations suggest that while semantic accuracy increases gradually across layers, true semantic differentiation does not emerge progressively. Instead, it appears abruptly at the final block. We further highlight three key findings:
To verify this, we design three typographic attack tasks of varying difficulty: Easy: Recognizing image content while igmoring text, similar the setup in most typography attack work (Materzyúska al., 2022; Ilharco et al., 2022; Azuma & Matsui, 2023). Medium: Detecting the presence of text without understanding its meaning. Hard: Distinguishing the semantics of both text and image.
(1) Overlapping Features in Textual and Textural En- coding, Across most layers, textural and semantic features rely on partially shared representations. Semantic encod- ing consistently lags behind, indicating dependency lower-level visual patterns during early stages.
The progression from easy to hard illustrates the increas- ing complexity semantic understanding required each stage. Idcally, fine-tuning only the swell blocks should not effectively defend against any level of attack. In con- trast, fine-tuning the shrink and last in shrink blocks should
(2) Texture-Induced Accuracy Gains. Early improve- ments in semantic decoding may be misleading, as they reflect the model's reliance on texture-based patterns rather than semantic comprehension in its commonly assumed

---

Textural or Textual: How Vision-Language Models Read Text in Images
Table 2. Accuracy (%) comparison of defense performance when fine-tuning different parts of the CLIP visual encoder.
Fine-tuned Blocks
Easy Irr
Medium
Hard Irr
Orig
Cons
Nons
Cons
Irr
Nons
Cons
Nons
CLIP w/o ft
82.3
97.3
50.6
73.7
94.5
65.9
77.4
14.5
59.9
77.2
Swell
62.8
85.8
39.1
51,2
79.0
52.1
56.3
6.6
29.7
56.2
Shrink
82.6
98.6
43.0
77.0
97.8
68.9
81.3
32.6
70.8
81.1
Shrink Last
83.5
98.7
32.0
76.6
98.1
61.2
81.4
36.0
68.9
81.7
All
54.8
50.1
50.3
49.1
48.8
48.9
48.1
0.1
0.5
48.2
Last (Ours)
84.7
98.2
60.0
76.8
96.7
74.5
81.0
35.0
69.5
81.7
Defense Level
Image
Text
Table 3. Cross-evaluation of SOTA defense methods on handwrit- typographic datasets (easy level). Results reported
Easy
photo o ((Pod).
racy (%).
Method
Disentangle
PAINT
Prefix
Avg.
photo of word written over picture of a (Podd.
Medium
d Dragnrey
CLIP
43.3
50.0
47.2
46.8
A photo the word (dragonfly) written over a picture of (IPod).
Hard
Disentangle
77.8
55.5
57.6
63.6
PAINT
53.2
58.2
53.6
55.0
Prefix
71.9
63.6
58.0
64.5
Figure 7. Example image-text pairs from the dataset across easy, medium, and hard defense levels (ground truth).
Ours
73.3
68.2
67.0
69.5
requiring nuanced text-image understanding, with strong results in the Medium and Hard categories (70.8% in Hard- Imr). However, fine-tuning the Shrink-Last module provides balanced performance, almost matching Last in the most difficult tasks while still lagging slightly in simpler cases like Orig (83.5%). This suggests that while Shrink- captures some mid-layer texture refinement, is not as adept at final-stage semantic comprehension as Last alone.
provide varying levels of defense based on semantic compre- hension. For example, medium difficulty may only require recognition of word orthography, necessitating adjustments to the shrink blocks, while the hard level requires under- standing specific meanings, thereby requiring fine-tuning of the last block for effective defense.
of our experiments are conducted on a GeForce RTX 3090 GPU. We use a batch size of 512 and learning rate of 1 x 10-4, with weight decay of 0.2. The Adam optimizer is employed for fine-tuning.
5.2. Fine-Tuning the Last Block
5.2.1. 1. DEFENSE WITH IGNORING TYPOGRAPHY
Setup. Following standard practices, we fine-tune the model on subsets original and irrelevant images under the easy setting, simulating training scenario where textual con- tent is treated noise. To assess real-world performance, we evaluate our method three publicly available typo- graphic attack datasets: Disentangle (Materzyńska et al., 2022), PAINT (Ilharco et al., 2022), and Prefix (Azuma Matsui, 2023), which all feature handwritten text overlaid on notepads. We perform cross-dataset evaluations using the respective test sets provided by each method.
5.1. Block-Specific Fine- Fine-Tuning for Textual and Textural Control
We divide the CLIP encoder into three sections: Swell, Shrink-Last, and Last, as described in Section 4. We fine- tune each section on the hard-level task and evaluate their performance across easy, medium, and hard tasks. The results are shown Table 2.
Fine-tuning the Swell block alone yields suboptimal per- formance across all diffieulty levels, partieularly in tasks requiring semantic understanding. Fine-tuning the Last block proves most effective, particularly in handling higher complexity tasks like Hard-Nons (81.7%) and maintaining high Orig performance (84.7%).
Results. Table presents the cross-datase accuracy of various defense methods. Notably, the Prefix approach fine- tunes only the language model, whereas others fine-tune both vision and language components. Our method, which fine-tunes only the vision model, achieves the best overall performance across all datasets, with slight exception on
The Shrink strategy also performs well, especially tasks
8

---

Textural or Textual: How Vision-Language Models Read Text in Images
Table Comparison of defense accuracy (%) between our method and SOTA approaches (CLII, Prefix (Azuma Matsui, 2023), and
Disentangle (Materzyúska et al., 2022)) on medium and hard difficulty levels, which require semantic understanding of text rather than simply ignoring it, as in the easy-level setting.
Medium Irr
Hard Irr
Method
Orig
Cons
Nons
Cons
Nons
CLIP
82.3
94.5
66.0
77.4
14.5
59.9
77.2
Prefix
82.0
91.4
69.9
76.0
10.6
27.1
75.5
Disentangle
79.9
85.0
72.0
75.2
12.9
13.8
75.3
Ours_ Med
83.5
92.2
82.4
82.9
8.1
22.0
82.4
Ours_Har 84.7
96.7
74.5
81.0
35.0
69.5
81.7
the Disentangle dataset, where it is marginally outperformed by the original method. This cross-evaluation demonstrates that our method is not overly fitted to our ToT dataset but instead captures a generalizable defense strategy effective across diverse typographic attack scenarios.
we reveal that early layers of visual models primarily rely texture features rather than true semantic understanding. Only in the final block do models construct meaning- driven semantically understanding after significant compres- sion of textural information. Furthermore, we demonstrate an effective defense strategy by fine-tuning the final block, which enhances the model's ability to distinguish between textural and textual elements. This approach significantly improves performance across various defense scenarios, of- fering a practical solution to typographic attacks.
5.2.2. DEFENSE WITH PRESERVING THE TYPOGRAPITY SEMANTICS
Table 4 presents the evaluation results for medium and hard levels of defense, which require the recognition of the ab- sence of words and specific semantics, respectively. Our method outperforms other models across all difficulty levels. The Prefix and Disentangle methods, trained on datasets similar to those used for easy-level tasks, reveal limitations in recognizing character forms and semanties, as demon- strated by their performance in the hard-level results. In contrast, our model exhibits superior comprehension across various difficulty levels, particularly when the image-text relationship is semantically consistent.
A Acknowledgements
This work was supported by the National Natural Science Foundation of China (62206167, 82430060).
Impact Statement
This paper investigates how multimodal vision models pro- cess textual semantics, revealing that early layers rely more on visual textures than on textual meaning, which makes these models vulnerable to typographic attacks.
Training on datasets with higher difficulty levels presents challenges in balancing 'Cons' and 'Irr' image-text pairings in the medium scenarios. However, in hard scenarios, where understanding both textual and visual semantics is essential, performance can be improved simultancously. With the appropriate training data, our method effectively fine-tunes models to comprehend both textual and visual semantics.
By shedding light the internal mechanisms of these mod- els, this work advances the transparency and accountability of machine learning systems. It contributes to the broader effort to make AI more interpretable and aligned with so- cietal values, particularly as these systems are increasingly deployed in decision-makinn processes. From a societal perspective, the vulnerabilities exposed in this research have important implications for the safety and fairness of Al sys- tems in real-world applications. Typographic attacks could lead to unintended consequences in fields such as law en- forcement, healthcare, or finance, where misinterpretations could disproportionately affect vulnerable groups. Ensuring that models are both transparent and secure is crucial for their responsible integration into these high-impact areas.
Another advantage of our approach is its ability to balance adversarial tasks with the original task. As shown in the 'orig' column of Table 4, our methods outperform all other models, despite primarily being trained on typographic sam- ples. Notably, the "Ours_Hard" model demonstrates im- proved 'orig' accuracy, even when typographic semantics potentially conflict with original image classification.
6. Conclusion
We explore how visual models process textual semanties in the context of typographic attacks. By introducing the ToT dataset and applying Intrinsic Dimension (ID) analysis,
9

---

Textural or Textual: How Vision-Language Models Read Text in Images
References
Facco, E., d'Errico, M., Rodriguez, A., and Laio, A. Esti- mating the intrinsic dimension of datasets by minimal neighborhood information. Scientific reports, 7(1):12140, 2017.
Aghajanyan, A., Zettlemoyer, L., and Gupta, S. Intrin- sie dimensionality explains the effectiveness of language model fine-tuning. arXiv preprint arXiv:2012.13255. 2020.
Goh, G., Cammarata, N., Voss, C., Carter, S., Petrov, M.. Schubert, L., Radford, A., and Olah, C. Multimodal neurons in artificial neural networks. Distilll 6(3)e30, 2021.
Amsaleg, L.,, Chelly, O., Furon, T., Girard, S., Houle, M. E., Kawarabayashi, K.-i., and Nett, M. Estimating local in- trinsic dimensionality. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Dis- covery and Data Mining, pp. 29-38,2 2015.
Gong, Y., Ran, D., Liu, J., Wang, C., Cong. T., Wang, A.,
Duan, S., and Wang, X. Figstep: Jailbreaking large vision- language models via typographie visual prompts. arXiv preprint arXiv:2311.085.
2023.
Amsaleg, L., Bailey, J., Erfani, S., Furon, T., Houle, M. E., Radovanovic, M., and Vinh, N. X. The vulnerability of learning to adversarial perturbation increases with intrin- sic dimensionality. In 2017 IEEE Workshop on Informa- tion Forensics and Security (WIFS), pp. 1-6, 2017.
Ilharco, G., Wortsman, M., Gadre, Y., Song. S., Hajishirzi, H., Kornblith, S., Farhadi, A., and Schmidt, Patching open-vocabulary models by interpolating weights. Ad- vances in Neural Information Processing Systems, 35: 29262-29277, 2022.
Ansuini, A., Laio, A., Macke, J. H., and Zoccolan, D. In- trinsic dimension data representations in deep neural networks. Advances in Neurul Information Processing Systems, 32, 2019.
Kil, J., Changpinyo, S., Chen, X., Hu, H., Goodman, S., Chao, W.-L., and Soricut, R. Prestu: Pre-training for scene-text understanding. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 15270-15280, 2023.
Azuma, H. and Matsui, Y. Defense-prefix for pre- venting typographic attacks on clip. arXiv preprint arXiv:2330.04512..222 2023.
Lemesle, Y., Sawayama, M., Valle-Perez, G., Adolphe, M., Sauzéon, H., and Oudeyer, P.-Y. Language- image classification: evaluation based semantic representa- tions. arXiv preprint arXiv:2201. / 2022.
Basile, L., Acevedo, S., Bortolussi, L., Anselmi, F,, and Ro- driguez, A. Intrinsic dimension correlation: uncovering nonlinear connections in multimodal representations. In ICLR, 2025.
Levina, E. and Bickel, P. Maximum likelihood estimation of intrinsic dimension. Advances neural information processing systems, 17, 2004.
Cao, L.,, Zhang, B., Chen, C., Yang, Y., Du, X., Zhang, W., Lu, Z., and Zheng, Y. Less is more: Removing text- regions improves clip training efficiency and robustness. arXiv preprint arXiv:2305.05095 2023.
Ma, X., Li, B., Wang, Y., Erfani, S. M., Wijewickrema, S., Schoenebeck, G., Song, D., Houle, M. E., and Bailey, J. Characterizing adversarial subspaces using local intrinsic dimensionality. arXiv preprint arXiv: /801. 0226/0 2018.
Carreiras, M., Armstrong, B. C., Perea, M., and Frost, R. The what, when, where, and how of visual word recogni- tion. Trends in cognitive sciences, 18(2):90-95 2014.
Materzyúska, J., Torralba, A., and Bau, D. Disentangling visual and written concepts clip. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16410-16419, 2022.
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1 11929, 2020.
Mokady, R., Hertz, A., and Bermano, A. H. Clip- cap: Clip prefix for image captioning. arXiv preprint arXiv:2111.09734 2021.
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenbom, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly. S., Uszkoreit, J., and Houlsby, N. An image worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2022.
Muratore, P., Tafazoli, S., Piasini, E., Laio, A., and Zoc- colan, D. Prune and distill: similar reformatting of im- age information along rat visual cortex and deep neural networks. 36th Conference on Neural Information Processing Systems, 2022.
10

---

Textural or Textual: How Vision-Language Models Read Text in Images
Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., EI- Nouby, A., al. Dinov2: Learning robust visual features without supervision. arXiv preprint arX1v:2304.07799, 2023.
Xu, H., Xie, S., Tan, X. E., Huang, P.-Y,, Howes, R., Sharma, V., Li, S.-W., Ghosh, G., Zeutleoyerr L., and Feicht- enhofer, C. Demystifying clip data. arXiv preprint urXiv:2309.16771 2023.
Yang. Z., Lu, Y., Wang, J., Yin, X., Florencio, D., Wang, L., Zhang, C., Zhang, L.,, and Luo, J. Tap: Text-aware pre- training for text-vqa and text-caption. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8751-8761, 2021.
Pope, P., Zhu, C., Abdelkader, A., Goldblum, M., and Gold- stein, T. The intrinsic dimension of images and its impact on learning. arXiv preprint Axi1:2204.08890 2021.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763, PMLR, 2021.
Yang. Z., Li, L.,, Lin, K., Wang, J., Lin, C.-C., Liu, Z., and Wang. L. The dawn of Imms: Preliminary explorations with gpt-4v (ision). arXiv preprint arXiv:2309. /7421, 9 (1):1-116, 2023.
Zhang, R., Zhang, W., Fang. R., Gao, Li, K., Dai, Qiao, Y., and Li, H. Tip-adapter: Training-fre free adaption of clip for few-shot classification. In European Conference on Computer Vision, pp. 493-510. Springer, 2022.
Robey, A., Wong, E., Hassani, H., and Pappas, J. Smooth- IIm: Defending large language models against jailbreak- ing attacks. arXiv preprint arXiv:2310.03664 2023.
Russakovsky, O. a. Imagenet large scale visual recog- nition challenge. In International journal of computer vision, volume 15,, pp. 211-252. Springer, 2015.
Saxe, A. M., Bansal, Y., Dapello, J., Advani, M., Kolchinsky, A. Tracey, B. D., and Cox, D. D. On the information bottleneck theory of deep learning. Journal of Statistical Mechanics: Theory and Experiment, (019(12):124020) 2019.
Shwartz-Ziv, R. and Tishby, N. Opening the black box of deep neural networks via information. arXiv preprint arXiv:1703.000 2017.
Song, H., Dong, L., Zhang, W.-N., Liu, T., and Wei, F. Clip models are few-shot learners: Empirical studies on vqa and visual entailment. arXiv preprint 07X1:2203..71990 2022.
Spearman, C. The proof and measurement of association between two things. The American journal of psychology, 100(3/4):441-44 4711, 1987.
Tulchinskii, E., Kuznetsov, K., Kushnareva, I.., Cherni- avskii, D., Nikolenko, S., Burnaev, E., Barannikov, S., and Piontkovskaya, I. Intrinsic dimension estimation for robust detection of ai- ai-generated texts. Advances in Neural Information Processing Systems, 36, 2024.
Van der Maaten, L. and Hinton, G. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
Wang, X., Ji, Z., Ma, P., Li, Z., and Wang, S. In- structta: Instruction-tumed targeted attack for large vision- language models. arXiv preprint arXiv:2312.01886, 2023.

---

Textural or Textual: How Vision-Language Models Read Text in Images
A. Appendix
A.1. Details the ToT Datasets
To create the Textural or Textual (ToT) dataset, we follow the approach of PAINT and Prefix. We resize the images to 224 pixels in the shorter dimension using bicubic interpolation and crop 224x224 pixel area from the center, consistent with standard CLIP resizing and cropping techniques. The text is randomly overlaid at arbitrary positions on the images.
Font. We randomly select from Roman, Courier, and Times fonts and utilize cight colors: black, blue, cyan, green, magenta, red, white, and yellow. The text is outlined with 1-point shadow in contrasting color.
Font sizes. We use 80 points to generate images for the Consistent, Irrelevant, and Nonsense categories. Additionally, to further investigate the impact of font size on identification (ID), Irrelevant images are created in font sizes ranging from 20 120 points. The examples are shown in Figure
Irrelevant (Font Size 20)
Irrelevant (Font Size 40)
Irrelevant (Font Size 60)
abra
eebra
Zebra
Irrelevant (Font Size 80)
Irrelevant (Font Size 100)
Irrelevant (Font Size 120)
Figure 8, Examples of typography with different sizes.
Subset Categories. The 100 categories of the ToT datasets are peacock, goose, koala, jellyfish, snail, flamingo, sea lion, Chihuahua, tabby cat, lion, tiger, bee, dragonfly, zebra, pig, llama, panda, backpack, barrel, basketball, bikini, bottlecap, bow, broom, bucket, buckle, candle, cannon, cardigan, carton, coffee mug, coffeepot, crib, envelope, fountain, iPod, iron, jean, ladle, laptop, lighter, lipstick, lotion, mailbox, mask, microwave, mitten, mouse, nail, necklace, paddle, pajama, perfume, pillow, plastic bag, printer, projector, purse, radio, refrigerator, ruler, shovel, sock, stove, suit, sunglass, swing, switch, table lamp, teapot, television, toaster, tray, tub, umbrella, vacuum, vase, violin, wallet, whistle, ice cream, bagel, hotdog, cucumber, mushroom, Granny Smith, strawberry, orange, lemon, banana, hay, dough, pizza, potpie, red wine, espresso, cup, volcano, daisy, and corn.
Subset 2 Categories. The subset includes the following paronym-synonym pairs: Goose (n01855672): Moose, Gander; Bee (n02206856): Beef, Wasp; Pig (n02395406): Fig, Hog; Fountain (n03388043): Mountain, Spring: Mitten (n03775071) Kitten, Glove; Nail (n03804744); Mail, Spike; Hay (n07802026): Ray, Straw; Espresso (n07920052): Express, Coffee; Lemon (n07749582): Demon, Lime.
A.2. ID Variations of Different Architectures and Datasets
Other vision models. To further assess the generalizability of our findings, we examined the Intrinsic Dimension (ID) of visual encoders from several multimodal architectures, beyond the ViT-B/16 used in CLIP. Figure 9 shows the intrinsic dimension curves of these models. Except for DINOv2, which is a vision-only pretrained model, all other models exhibit clear divergence ID at the final block when different semantic texts are overlaid.
12

---

Textural or Textual: How Vision-Language Models Read Text in Images
140
Origin
Conexbent Nontense
140
Last
Imslevant
120
20
14
Layar
Lover
(a) ResNet-50x4
(b) DINOV2 (ViT-S/14)
Origih
Orgin
Consigert
Concelert
32
Insecent
Wonsense
Manserse
The
(c) MctaCLIP (VIT-B/32)
(d) Sigl.p2 (VIT-B/16)
Origin Comsisttnn Iridevvnt Nonsonse
Normaasse
33-
25
10
10-
20
30
20
Layyr
Layer
(c) CLIP (VIT-L/16)
( CLIP (VIT-H/14)
Origin
The Laxt
175
Condistent
Nomsemse
Inelevant
150
125
100
100
120
Layer
(g) ShareGPT-4v
Figure 9, Layer-wisc intrinsic dimensions (IDs) of visual encoders across different architectures.
13

---

Textural or Textual: How Vision-Language Models Read Text in Images
Other datasets. While our primary focus is the ToT dataset, observe that the behavior of the Intrinsic Dimension (ID) also holds across other datasets. Using the same methodology for the ToT dataset, applied text overlays the Caltech101 dataset. As shown in Figure 10, the resulting ID variations follow the similar trend as those observed with the ToT dataset.
Origin
22.5
Consistent
Block
Nonserse
20.0
15.0
1.0
15
20
25
30
Layer
Figure 10. IDs of ViT-B/16 on the Caltech101 Dataset.
A.3. Defense Performance on MetaCLIIP
Table 5. Performance comparison when fine-tuning different parts of the MetaCIII visual encoder:
Fine-tuned
Orig
Easy Irr
Medium Irr
Hard Irr
Blocks
Cons
Nons
Cons
Nons
Cons
Nons
Swell
19.3
17.5
16.7
15.7
16.8
15.9
0.4
16.0
Shrink
70.5
89.3
63.2
65.0
88.0
60.7
63.3
17.5
46,3
62.5
Shrink Last
73.1
87.5
64.8
66.4
86.7
62.4
65.1
22.8
44.8
64.0
Last (Ours)
83.6
91.3
79.0
80.0
90.3
77.4
78.2
40.3
54.7
78.3
A.4. Correlation Between ID and Accuracy
Table 6 shows the relationship between the ID of the last fully connected layer (ID Last), the maximum ID (ID Max), and classification accuracy for ViT and CLIP models. The Spearman correlation coefficient (Spearman, 1987) is used to measure the correlation between accuracy and ID values.
The correlations for ViT are p(ID Last, Acc) Acc) = -0.13 and p(ID @ID_Maxx, Acc) 70.73. Overall, an inverse correlation observed, suggesting that lower values correspond to higher classification accuracy. However, this trend is not consistent for the last layer ID, deviating from patterns typically found in standard image classification tasks (Ansuini al., 2019).
-0.98 and (ID Max, Ace) = -0.63, while for CLIP, they are
Table Correlation between classification accuracy and ID values for the last layer and maximum ID across layers, differentiated by typography type and size. The models compared are pre-trained via multimodal (CIIP) and pure vision (ViT) models.
Model
Orig
Cons 80
Nons 80
Irr 20
Irr.40
Irr 66
Irr 80
Irr 100
Irr.120
ViT
Acc.
91.1
86.3
91.0
89.9
88.9
85.9
84.2
Lastast
6.8
7.8
8.0
7.0
7.2
7.6
7.8
8.1
8.3
ID Max
89.9
92.4
95.4
95.6
89.1
95.2
100.0
99.1
102.5
CLIP
Acc. 86.6
98.4
80.4
78.8
60.7
49.2
42,9
40.5
38.9
Last
12.4
9.8
14.0
14.8
14.3
13.0
12.7
12.5
12.6
ID Max
26.2
29.4
29.6
29.2
29.5
29.4
29.5
29.8
29.8
For the VIT model, there clear correlation between accuracy and text size: as text size increases, accuracy decreases,
14

---

Textural or Textual: How Vision-Language Models Read Text in Images
which aligns with changes in ID ID_last values. This observation supports findings from (Ansuini al., 2019), where text size plays significant textural role in pure vision models.
In contrast, the CLIP model shows that text semantics significantly impact accuracy, even when size is controlled. The relationship between accuracy and ID metrics is more complex here; while no clear correlation exists with ID last for semantically irrelevant texts, there is a strong inverse correlation between accuracy and ID max as text size increases. This suggests that mam captures textural complexity, whereas ID_last reflects both textural and textual features. CLIP's representation of text involves complex interaction between these elements, with semantics heavily influencing accuracy, yet no single layer fully captures this correlation.
15


Reviews:
[{"review_id": "A3d2XLySo6", "reviewer": "Reviewer_d3BZ", "summary": "This paper explores how vision-language models (e.g., CLIP) process text in images via the ToT (Textural or Textual) dataset, showing that early layers rely on visual texture while semantic understanding emerges in the final blocks. Using Intrinsic Dimension (ID) analysis, the paper reveals changing representational complexity across layers and propose fine-tuning the last layer to counter typographic attacks, yielding substantial performance gains in various defense scenarios.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their close reading and thoughtful comments. We would like to clarify that this paper is primarily an **interpretability study** that analyzes how vision-language models process text in images, using Intrinsic Dimension (ID) as a lens to reveal the transition from textural perception to semantic abstraction across layers. Fine-tuning experiments are not our core contribution but serve to validate our representational insights. We intend our findings to serve not as a conclusion, but as a **foundation for deeper inquiries** into the representational dynamics of multimodal models.\n\nWe address the reviewer’s concerns point-by-point below. Additional experimental results are available at: https://anonymous.4open.science/r/TOT-078C/\n\n---\n\n**Q1. Generalizability beyond ViT-B/16**\n\nWe appreciate the concern. Results on **ResNet** (Figure 9) are included in the appendix. Additionally, we evaluate **ViT-L/14, ViT-H/14, SigLIP, and MetaCLIP** in Figures A2–A5. Across all these architectures, we observe that significant shifts in ID values **consistently emerge in the final block**, suggesting that the phenomenon is not specific to ViT-B/16 but generalizes across a broad range of visual encoders.\n\n**Q2. Why not use mutual information or other theoretical metrics to support the stage transitions?**\n\nWe understand the reviewer’s suggestion to explore more theoretical metrics such as mutual information for supporting the stage transitions. While MI is theoretically relevant, its estimation in high-dimensional, structured representations is **fundamentally limited by the bias-variance tradeoff of current estimators**. Methods like MINE or kNN often yield unstable or uninterpretable results across layers due to distributional shifts and the absence of ground-truth joint distributions. Given our focus on semantic emergence, ID offers a more stable and interpretable alternative.\n\nOur segmentation into representational \"four stages\" is not intended as a strict theoretical taxonomy, but rather as a **descriptive scaffold** that highlights where semantic abstraction emerges most distinctly. ID is used as a tractable and interpretable proxy for representational complexity, and it is especially well-suited to our goal of diagnosing typographic vulnerability. We view this as a methodological decision, **balancing interpretability with empirical rigor**.\n\n**Q3. Could the observed gains stem from data quality rather than the fine-tuning strategy?**\n\nTo isolate the effect of fine-tuning, we introduced several controls:\n\n1. In Appendix A.2, we replicate our ID analysis on a **distinct dataset** with matched structure. Results remain consistent, reinforcing that the observed gains are not tied to specific image content.\n2. In Table 3, we conduct **cross-evaluation**, where each model is tested on all datasets (not just the one it was trained on).\n3. All baselines in Table 4 were **fine-tuned on our datasets** to ensure fair comparisons under the same conditions.\n\nWe believe these controls confirm that performance improvements arise from the **fine-tuning strategy**, not data artifacts.\n\n**Q4. The conclusion that semantic understanding is delayed until the final layer lacks deeper theoretical explanation**\n\nWe believe the lack of a settled theory on when and how semantic understanding emerges in vision-language models reflects the field’s ongoing search for foundational insight, rather than a weakness.\n\nOur work contributes to this effort by introducing Intrinsic Dimension (ID) as a **structured, interpretable proxy** for representational complexity. We find that semantic fragility under typographic attacks coincides with sharp ID shifts in the final layers, suggesting a transition toward higher-order abstraction beyond surface features.\n\nWhile our interpretation remains exploratory, we argue that **such empirical signals are essential for building theoretical understanding**. By combining observations (Sec. 4), cross-model evidence (Sec. 5), and actionable metrics, we aim **not to close the question, but to make it visible and tractable**.\n\n**Q5. Clarify the fine-tuning loss**\n\nWe use the **standard CLIP contrastive loss** for all fine-tuning experiments. The only modification lies in the construction of **positive pairs**, which we adapt to reflect semantic similarity across different attack settings. We do not introduce new loss functions. Figure B2 provides explicit training details.\n\n---\n\nThank you again for helping improve this work. We hope these clarifications address your concerns and highlight both the rigor and potential impact of our contributions.", "claims_and_evidence": "The paper’s claim is clear and well-supported by experiments. Through the ToT dataset and ID analysis, the paper show how the model processes text and visuals across layers, culminating in stronger semantic understanding in the final layer.\n\n**Weakness:**\n1. The experiments were conducted only on ViT/B-16 and have not been verified on larger-scale ViT models (e.g., ViT/L-14, ViT/H-16) or other architecture (ResNet) of image encoders.\n\n2. The paper classifies stages solely by changes in ID and accuracy. Could metrics commonly used in information bottleneck theory—such as mutual information—be adopted to further support the four-phase explanation of VLM image processing?", "methods_and_evaluation_criteria": "The analysis and fine-tuning strategies fit the problem well. The ToT dataset carefully balances text semantics and visual forms, and metrics (accuracy, ID) effectively measure complexity and defense performance.", "theoretical_claims": "The paper lacks theoretical explanations, focusing on experimental results but not exploring why semantic understanding emerges late or how ID changes relate to it.", "experimental_designs_or_analyses": "The experimental design is solid and the analysis methods are effective. Through ID analysis and t-SNE, the authors show how representations evolve across layers and validate the defense strategy with fine-tuning.\n\n**Weaknesses:**\n\n1. The experimental could be expanded by including more model architectures to better validate the conclusions.\n2. It is unclear whether all comparison baselines were fine-tuned on the ToT dataset, potentially affecting the reliability of the results. Specifically, it remains unclear whether performance improvements stem from higher-quality data or from the new fine-tuning strategy.", "relation_to_broader_scientific_literature": "This paper uses ID analysis to reveal changes in representational complexity across different layers, and proposes a defensce strategy against typographic attacks. However, the defence strategy is relatively simple to implement and shares core ideas with some existing work, suggesting limited novelty.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "**Strengths:**\n\n1. The paper is well-written, with a clear motivation.\n2. The design of the ToT dataset contributes a valuable resource for subsequent research in this area.\n\n**Weaknesses:**\n\n1. The conclusion that semantic understanding is delayed until the final layer lacks deeper theoretical explanation, and the available experiments only cover ViT-B/16. It remains uncertain whether the same pattern holds for larger ViT models.\n2. There is no detailed training loss using in the fine-tuning.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "See above weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SVaDY6d7vn", "reviewer": "Reviewer_8NZn", "summary": "This paper investigates Typographic attacks in vision language models. They investigate whether these models encode textual semantics through representation complexity, and identify the mechanisms by which text disrupts visual understanding. \nTo decouple orthography from semantics, they introduce the ToT dataset, containing minimal pairs of words that either share semantics with distinct visual forms (synonyms) or match visual forms with conflicting semantics (paronyms). \nBy analyzing layer-wise Intrinsic Dimension (ID), they found that early layers exhibit competing dynamics between orthographic features and semantics, while later layers improve semantic accuracy primarily through orthographic disambiguation. Crucially, semantically driven representations emerge only in the final block, challenging the assumption of progressive semantic understanding.\n\n## update after rebuttal\n\nDue to this limited scope of experiment, and that the writing of this paper definitely needs to be improved, I'd like to remain my original score.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful and open review. This paper is primarily an **interpretability-driven study**: our goal is to understand how VLMs represent and process text in images across layers. Rather than proposing a new defense method, our core contribution lies in using Intrinsic Dimension (ID) to **trace where semantic abstraction emerges, and how it breaks** under typographic attacks. The fine-tuning experiment serves to validate our explanation in a practical setting.\n\nWe address your concerns point by point below. Additional experimental results can be found at: https://anonymous.4open.science/r/TOT-078C/\n\n---\n\n**Q1.  Could you extend to other models (DINOv2, SigLIP, MetaCLIP...)?**\n\nYes, while our main experiments focus on CLIP (a widely-used baseline), we have extended our analysis to **ViT-L/14, ViT-H/14 DINOv2, SigLIP, and MetaCLIP**. Please see Figures A1–A5 in the supplementary material.\n\nWe observe a **consistent semantic abstraction pattern across these models**: semantic distinctions between consistent, irrelevant, and nonsense text inputs remain indistinguishable in early layers, but diverge only in the final block (reflected by their ID separation). This suggests that delayed semantic resolution is a **shared property of VLMs, not limited to CLIP**.\n\nNotably, **DINOv2, a pure vision model** without language pretraining (similar to the ViT in Table 1), does not exhibit such ID bifurcation in the final block, also supporting our interpretation.\n\n**Q2. In Table 1, how should we read the “Irr_s” numbers in the first row?**\n\nThe first-row entries like “Irr_80” refer to **irrelevant text overlaid at a font size of 80**. Appendix Figure 8 shows examples. Font size modulates the visual salience of the text: larger fonts induce stronger texture-level interference, allowing us to evaluate robustness across texture intensity. \n\nWe will make this explicit in the table caption. \n\n**Q3. Section 3.2 mentions paronym vs. synonym confusion, but their comparison is not in Table 1. Where is it?**\n\nThis analysis is in **Figure 5**, via a linear probing experiment across residual blocks.\n\nWe trained logistic regression classifiers to distinguish **paronym pairs** (orthographically similar but semantically different) from **synonym pairs** (semantically similar but orthographically distinct). Each line represents a word pair; darker lines are averages over 10 pairs.\n\nThis reveals **how orthographic and semantic information evolve across depth**, and aligns with our main claim: semantic abstraction is delayed and fragile.\n\nWe will revise Section 3.2 to better connect with this figure and clarify the setup.\n\n**Q4. In Table 3, row and column names are the same. What exactly are the “SOTA methods”?**\n\nThank you for pointing this out, we see how it could be confusing. Table 3 is a **cross-dataset evaluation**: each row represents a defense method, and each column is the dataset it was trained on (each paper proposes a new dataset). The diagonal shows **in-domain robustness**, while off-diagonal cells reveal **cross-domain generalization**. \n\nWe deliberately adopted this structure to **decouple the effects of methods and datasets**, ensuring a more rigorous and fair comparison. Our method achieves the **best or second-best performance across all datasets**, demonstrating its broader applicability. We will clarify this in the table caption.\n\n**Q5. In Tables 2 and 4, under “Hard,” why are Irr and Nons scores higher than Cons? Shouldn't Cons be easier?**\n\nThis is a subtle but important point. In the “Hard” setting, the training set contains more Irr and Nons samples than Cons, introducing both a distributional shift and higher conflict complexity (see Fig. 7). As a result, models trained in this setting tend to perform better on the more frequent categories, even if Cons is conceptually easier.\n\nWhile Cons is generally easier for the **original CLIP**, once we fine-tune the model, it begins to **balance trade-offs across tasks** such as consistent vs. irrelevant recognition. This can lead to non-monotonic changes in relative difficulty, where Cons is no longer always the easiest class.\n\n**Q6. The writing can be improved. The fine-tuning method seems to lack novelty.**\n\nWe will revise the paper for clarity, especially in table structure, dataset presentation, and experimental details.\n\nAs for the fine-tuning method, it is not positioned as a core novelty, but rather as a **minimal but effective validation** of our main claim regarding layer-wise semantic abstraction. The method is **intentionally simple**: our goal is to show that **understanding** where semantic features emerge allows even lightweight interventions to yield significant robustness gains. This underscores the **practical value** of our interpretability-driven analysis.\n\n---\n\nWe appreciate your thoughtful engagement. We hope our responses have addressed your concerns and clarified our motivation. Thank you for considering a re-evaluation.", "claims_and_evidence": "First of all I'm not familiar with this area of Typographic attacks in vision language models. \nAfter reading the paper, I think most claims are supported with evidence. But there are some confusing parts. \n1. It seems that all the conclusions can only apply to CLIP, instead of general vision language models. Could you extend to other models, e.g.DINOv2, SigLIP, MetaCLIP...?\n2. In Table 1, how to read the numbers in the first row? It's hard to understand what the authors want to say about the Irr_s.\n3. Also,  Sec 3.2 mentions PARONYMS VS. SYNONYMS CONFUSION but their performance comparison is not presented in Table1? Where can we read their experiment results?\n4. The Table 3 row names and column names are the same, it's not clear what the \"sota methods\" are exactly. \n5. In Table 2 and 4, under \"Hard\", why are the Irr and Nons scores much higher than Cons score, opposite to the other columns? Shouldn't Cons always be easier than Irr and Nons?", "methods_and_evaluation_criteria": "There are some confusing points. See above.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "There are some confusing points. See above.", "relation_to_broader_scientific_literature": "It's related to safety issues for general vision language models.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "I think the presentation of the paper can be further improved. Also, the finetuning defense method seems a bit lack of novelty. More models and more explanations should be added.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "See above. Happy to raise my score if questions are addressed.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MHWw3Y6lJJ", "reviewer": "Reviewer_UJSz", "summary": "This paper investigates images with overlaid text, and how vision-language models process them. The paper analyzes representations throughout model layers using Intrinsic Dimension as a measure of complexity. It finds that early layers primarily encode textures while the last one encodes semantics. Through these insights, they are able to achieve significant improvements against typographic attacks by only fine-tuning the final layer.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the thoughtful and encouraging review. We're glad the main idea came through clearly: we aim to understand how vision-language models handle overlaid text, and how representational insights can guide robustness improvements. Your suggestions on Figure 5 and the evaluation setup are especially helpful. Below we respond point by point. Additional materials can be found at: https://anonymous.4open.science/r/TOT-078C/\n\n---\n\n**Q1. Figure 5 appears to contradict Table 1, and the setup is unclear**\n\nThese two experiments address different questions and use different evaluation metrics:\n\n- Figure 5 uses a **probing classifier** to test whether representations at **each layer** contain enough information to distinguish orthographic (paronym) vs. semantic (synonym) distractors.\n- Table 1 evaluates end-to-end **image-text matching accuracy (by the last layer)** under typographic attacks, focusing on task-level performance.\n\nThe confusion may stem from the similar word pairs appearing in both contexts. In Table 1, they’re assessed for semantic alignment; in Figure 5, they are diagnostic probes for how semantic and visual cues are encoded across layers. The lower probe accuracy on synonym distractors (e.g., goose-gander) indicates that semantic information is harder to linearly extract at intermediate layers, whereas orthographic differences remain easier to capture. We also provide a detailed explanation of both the evaluation and training setups in **Figures B1 and B2** in the appendix. We hope this helps clarify how the probe task (Figure 5) differs from the classification task (Table 1).\n\nRather than contradicting, this result **reinforces our central finding**: semantic abstraction emerges later than visual/textural cues in VLMs. We will revise the text to clarify the goals and setup of the probing experiment.\n\n**Q2. L044–045 of the Introduction: Is there prior work showing that \"textual elements [...] are often encoded similarly\"?**\n\nThank you for pointing this out. That sentence was intended as a motivating question rather than a factual claim. We agree that the phrase “are often encoded similarly” may overstate the point. In the final version, we will cite prior work that offers perspectives on both sides of this question, and present it with an objective and open stance.\n\n**Q3. Minor typos and naming / layout suggestions**\n\nThank you for these valuable presentation suggestions. We will revise the manuscript accordingly:\n\n- Clarify that **Figure 6 combines Figures 4 and 5**, and update the caption;\n- Stack Figures 4 and 5 on the same page for easier visual comparison;\n- Rename categories from **\"Consistent / Irrelevant\" → \"Matched / Mismatched\"**, and **\"Orthographic / Semantic\" → \"Paronym / Synonym\"**, to improve interpretability.\n\n---\n\nWe appreciate your engagement with both the technical and presentation aspects of this work. Your suggestions helped us clarify our framing and improve the accessibility of the paper. We believe the revised version reflects these improvements and presents a clearer, more complete contribution.", "claims_and_evidence": "- Claim 1. In early layers texture and semantics compete, while in late layers semantic accuracy improves.\n    - Sec 3 discusses the design of the evaluation set, which contains paronyms (which are texturally related) and synonyms (which are semantically related).\n    - Figure 4 shows that any typography increases ID in middle layers. In the last layer, there is a clear ordering where the most to least consistent typography has the smallest to biggest ID.\n    - Table 1 shows for a pure vision ViT is less sensitive to text semantics and more to size whereas multimodal CLIP is heavily sensitive to semantics.\n    - I don’t understand the result in Figure 5. It seems to imply that the model achieves higher accuracy on the paronym pairs, for example “moose” overlaid on a picture of a goose, which can also be considering to be a conflicting or “irrelevant” case. This result conflicts with Table 1, where “Cons_80” significantly outperforms “Irr_80” for CLIP. The setup of this experiment is also unclear: is every image duplicated four times, as shown in Figure 2, with different text overlaid, and the classification accuracy is computed averaged across all these duplicated images?\n- Claim 2. Building on these insights, fine-tuning only the final block of the model is sufficient to achieve state-of-the-art performance for ignoring typography.\n    - This result is supported by Tables 3 and 4. The prior work used as baselines focus on subspace discovery, weight interpolation, or prefix tuning, rather than direct fine-tuning as investigated by this work.", "methods_and_evaluation_criteria": "See “Claims and Evidence” above.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "See “Claims and Evidence” above.", "relation_to_broader_scientific_literature": "This work conducts a controlled evaluation of texture versus semantics in typographic attacks (Figure 4, Table 1), and shows that direct fine-tuning of the final block is sufficient to achieve state-of-the-art results (Table 3), compared with more hand-crafted methods in prior work.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths\n\n- I liked how the insight from Figure 4, where it is identified that most of the semantic separation occurs in the last block, is used to motivate the experiments in Sec. 5.\n\nWeaknesses\n\n- I don’t understand the experiment in Figure 5 / L313, where the result seems to conflict with the result in Table 1 (see “Claims and Evidence” above).\n\nOverall, I liked the premise of investigating texture versus semantics in typographic attacks, and the experiments reasonably supported this investigation. The result in Sec. 5 is also strong. However, the presentation is hard to parse, in particular keeping track of all the evaluation sets described in Sec 3.1.", "other_comments_or_suggestions": "Below are minor clarity and writing comments that do not affect my score, but are intended as constructive feedback.\n\n- I don’t understand L044-045 of the Introduction; is there a prior work that shows that “textual elements […] are often encoded similarly”?\n- Typo, in L063 “We” should be lowercased as “we”\n- I took me a long time to understand that Figure 6 was a combination of Figures 4 and 5; you should also reference these figures in the figure caption. It would be much easier to read if Figures 4 and 5 were stacked on the same page or in a single figure, rather than overlaid in Figure 6.\n- Typo, in L064 the citation for Cao et. al. should use \\citet\n- In L157, I would recommend renaming the categories [“Consistent”, “Irrelevant”] → [“Matched”, “Mismatched”] or some more symmetric naming.\n- In Figure 5 I would recommend renaming [”Orthographic”, “Semantic”] → [”Paronym”, “Synonym”] so it’s easier to understand what is being evaluated.", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9dNDltthHJ", "reviewer": "Reviewer_HHNW", "summary": "This paper systematically analyzes how the encoder-only vision-language models (i.e., CLIP) perceives the textual and semantic information. The paper uses ID and linear probe to measure the representation complexity and semantic perception ability of vision-language models. The analysis suggests that at earlier layers the model captures texture information while in the later layers, the model captures semantic information. Based on above findings, the paper proposes a method for against typographic attacks by solely fine-tuning the last layer and the experimental results show the improved performance.\n\n\n## update after rebuttal\nThanks for the effort, since the concerns are not well addressed, I decide to keep the score.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the detailed comments. We appreciate the focus on the underlying assumptions behind our interpretation, and we have carefully examined all concerns related to experimental validity and inference logic. We address each point below, and will incorporate clarifications, results, and new visualizations into the revised version. Additional materials can be found at: https://anonymous.4open.science/r/TOT-078C/\n\n---\n\n**Q1. Font size may introduce multiple confounds (texture vs. visibility vs. occlusion)**\n\nWe agree that font size may introduce occlusion or legibility effects. To control for this, we included a **vision-only ViT as a baseline** in Table 1. Since ViT lacks language input, its accuracy drop reflects purely visual factors. \n\nCLIP, with the same visual backbone, shows a different trend, performance degrades more when the overlaid text **conflicts semantically** with the image. This contrast suggests that CLIP’s behavior cannot be explained by occlusion alone, and that semantic interference plays a key role.\n\n**Q2. Small sample size in paronym-synonym probing; representativeness of word pairs**\n\nThe probe experiment was designed as a controlled interpretability analysis using **representative and well-matched word pairs**. Each pair was selected to reflect either semantic or orthographic similarity, while controlling for **word frequency, visual appearance, and word length**. All words used are **frequent and familiar**, ensuring they are easily interpretable by both humans and models.\n\nWhile the total number of pairs is small, they were chosen to **typify common semantic vs. paronym contrasts** (e.g., “goose–gander” vs. “goose–moose”). The key observation holds consistently across all pairs, supporting the robustness of the trend.\n\nWe will clarify the selection criteria and framing in the revision, and agree that expanding to a larger lexical set is a valuable direction for future work.\n\n**Q3. Increased accuracy in deep layers may not imply semantic abstraction**\n\nWe understand the concern that deeper layers may naturally yield higher separability. However, **if this were purely due to compression or feature refinement,both paronym and synonym distinctions should improve similarly**.\n\nInstead, we observe a clear asymmetry: orthographic separability is strong from early layers, while semantic separability emerges only in the final block. This pattern cannot be explained by general separability alone, and provides a strong structural signal of **delayed semantic abstraction**.\n\n**Q4. ID drop may reflect redundancy reduction, not semantic compression**\n\nWe agree that a decrease in ID can result from various factors such as redundancy reduction or noise suppression, and does not directly prove semantic abstraction. In our work, we treat ID as a proxy for representational complexity, not for semantic content itself.\n\nTo support our interpretation, we provide **multiple converging signals**:\n\n1. ID Analysis shows a **consistent drop** in the final block, particularly under semantic perturbations (e.g., synonym substitutions), suggesting a shift in representational dynamics.\n2. Linear Probing reveals that only the final block supports linear separation of semantic distractors (synonyms), whereas paronym separability appears **much earlier**.\n3. **Grad-CAM visualizations** (Figure C1) show that:  in the final block, **attention re-centers on the object only when the overlaid text is semantically aligned with the image**. In contrast, for irrelevant or nonsensical text, the model continues to attend to the text region, even in the deepest layers.\n\nTogether, these findings suggest that semantic abstraction is not merely a byproduct of general compression, but emerges **selectively and meaningfully** in the representational structure. We do not claim causal proof, but this triangulation offers strong evidence of **a structured link between ID reduction and semantic resolution**.\n\nWe will clarify this framing and add more explanation for Figure C1 in the revision.\n\n**Q5. Lack of full-model fine-tuning baseline in Section 5.1**\n\nFull-model fine-tuning (**Table A1**) underperforms our final-block-only strategy by **over 30%** across all splits, likely due to **overfitting and disruption of early representations**. Our analysis shows that semantic abstraction emerges in the final block, making targeted fine-tuning both more effective and efficient.\n\n---\nThank you for your thoughtful and precise feedback. We’ve carefully responded to the points raised to clarify assumptions, tighten the experimental framing, and better reflect the structure of our findings. We hope our explanations help strengthen the rigor and improve the clarity of the work.", "claims_and_evidence": "The two main experiments designed in Section 4.2 can not fully support the claims.\n\n1. Semantic Constancy with Varying Font Size: The author hypothesizes that increasing font size enhances texture complexity, thereby influencing the model's judgment. However, changes in font size may simultaneously affect semantic readability (e.g., smaller fonts may be harder to recognize), introducing confounding variables. For instance, the ViT model's accuracy decreases with larger fonts (Table 1), which the author attributes to texture interference. However, it is possible that the actual reason is that larger fonts obscure more image content, rather than solely due to changes in texture.\n\n2. ​Linear Probe on Paronym-Synonym Pairs​: The author conduct experiments on only 10 pairs of words, which constitutes a relatively small sample size. Furthermore, is the selection of word pairs balanced in terms of factors such as word frequency and visual similarity? For instance, do the visual and semantic differences between \"goose-moose\" and \"goose-gander\" adequately represent the broader spectrum of such comparisons? If there is bias in the selection of word pairs, the conclusions drawn may lack generalizability.\n\nThe conclusion drawn from observations are problematic:\n\n1. Figure 5 shows a significant improvement in the classification accuracy of synonyms in deeper layers, which the authors attribute to the delayed formation of semantic understanding. However, the increase in accuracy might merely reflect that deeper features are more linearly separable, rather than indicating the enhanced comprehension of semantic. For instance, deeper features could become easier to classify due to dimensionality compression, but this does not necessarily imply semantic encoding.\n\n2. The authors interpret the decrease in ID as semantic compression. However, the reduction in ID could also stem from decreased feature redundancy or noise suppression, which may not directly correspond to semantic abstraction. Additional evidence, such as feature visualization or intervention experiments, is needed to substantiate the causal relationship between changes in ID and semantic representation.", "methods_and_evaluation_criteria": "See claims and evidence", "theoretical_claims": "NA", "experimental_designs_or_analyses": "The experiment presented in Section 5.1 needs another baseline, i.e., fine-tuned the whole layers.", "relation_to_broader_scientific_literature": "This paper engages with the existing literature in the following aspects:\n\nIt proposes a defense strategy by solely fine-tuning the last layer, balancing efficiency and performance.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strengths: The paper is well-structured.\n\nWeakness: see claims and evidence.", "other_comments_or_suggestions": "See claims and evidence.", "questions_for_authors": "No question", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Even after discussion this paper has divergent reviews (2 weak accepts and 2 weak rejects). After reading the paper, reviews, and discussion, it is a very close decision. I tend to agree with the accepts that this task and analysis has inherent value and that the finding (in particular, that the last layer uniquely causes separation between texture and textual) is interesting. However, I also tend to agree with the rejects that the presentation of the paper could be improved and that there may be other possible explanations for the observations that would conflict with the conclusions being drawn.

While it's close, I advocate for acceptance. I think that the paper's task, dataset, and analysis make it useful to field and am sympathetic to the argument that in deep learning the best we can do is present multiple observations that are suggestive of a conclusion. I encourage the authors to integrate the reviewers suggestions on improving the readability and presentation of the paper.
Decision: Accept (poster)