Dyslexify: A Mechanistic Defense Against Typographic Attacks in CLIP
Authors: Lorenz Hufe
Constantin Venhoff
Erblina Purelku
Maximilian Dreyer
Sebastian Lapuschkin
Wojciech Samek
Venue: ICLR 2026 Poster (2026)
Keywords: Multimodality
Circuit analysis
Probing
AI Safety
Vision transformers
Primary area: interpretability and explainable AI
TL;DR: We present Dyslexify, a training-free defense for CLIP that ablates a causal attention circuit, yielding up to +22.1% robustness on ImageNet-100-Typo while keeping standard accuracy within 1% of the baseline.

Abstract: Typographic attacks exploit multi-modal systems by injecting text into images, leading to targeted misclassifications, malicious content generation and even Vision-Language Model jailbreaks.
In this work, we analyze how CLIP vision encoders behave under typographic attacks, locating specialized attention heads in the latter half of the model's layers that causally extract and transmit typographic information to the cls token.
Building on these insights, we introduce Dyslexify - a method to defend CLIP models against typographic attacks by selectively ablating a typographic circuit, consisting of attention heads. Without requiring finetuning, dyslexify improves performance by up to 22.06\% on a typographic variant of ImageNet-100, while reducing standard ImageNet-100 accuracy by less than 1\%, and demonstrate its utility in a medical foundation model for skin lesion diagnosis. Notably, our training-free approach remains competitive with current state-of-the-art typographic defenses that rely on finetuning. To this end, we release a family of dyslexic CLIP models which are significantly more robust against typographic attacks. These models serve as suitable drop-in replacements for a broad range of safety-critical applications, where the risks of text-based manipulation outweigh the utility of text recognition.

Published as a conference paper at ICLR 2026
DYSLEXIFY: A MECHANISTIC DEFENSE AGAINST TYPOGRAPHIC ATTACKS IN CLIP
Lorenz Hufe 1,2 Constantin Venhoff 2, Erblina Purelku' '
Maximilian Dreyer Sebastian Lapuschkin 1.3 Wojciech Samek 1.1
Fraunhofer Heinrich Hertz Institute, University of Oxford 3 Technological University Dublin, Technische Universität Berlin 1 lorenz. hwwe@hhi. fraunhofer.de
ABSTRACT
Typographic attacks exploit multi-modal systems by injecting text into images, leading to targeted misclassifications, malicious content generation and even Vision- Language Model jailbreaks. In this work, we analyze how CLIP vision encoders behave under typographic attacks, locating specialized attention heads in the latter half of the model's layers that causally extract and transmit typographic informa- tion to the cls token. Building on these insights, we introduce Dyslexify method to defend CLIP models against typographic attacks by selectively ablating typographic circuit, consisting of attention heads. Without requiring finetuning, Dyslexify improves performance by up to 22.06% on a typographic variant ImageNet-1000 while reducing standard ImageNet- accuracy by less than 1%, and demonstrate its utility in a medical foundation model for skin lesion diagnosis. Notably, our gradient- free approach remains competitive with current state-of-the- art typographic defenses that rely on finetuning. To this end, we release a family of dyslexie CLIP models which are significantly more robust against typographic attacks. These models serve as suitable drop-in replacements for broad range of safety-critical applications, where the risks of text-based manipulation outweigh the utility of text recognition.
Poject page:
Code:
INTRODUCTION
CLIP models are increasingly adopted as general-purpose vision laniunel representations, enabling applications in zero-shot classification, retrieval, diffusion-based generative models, and large-scale vision-language models (VLMs). Their versatility has further driven adoption in safety-relevant domains such healtheare (Yang et al., 2024; Wang et al., 2022; Eslami et al., 2023), remote sensing (Liu et al., 2024a; Vivanco et al., 2023; Li et al., 2023), and content moderation (Schuhmann et al., 2022; Liu al., 2025; Reyes et al., 2025). However, despite their widespread use, CLIP models remain vulnerable to typographic attacks: inserting text into an image can mislead classification, trigger malicious generations, or even jailbreak multi-modal systems (see Fig. 1).
Existing defenses against typographic attacks require gradient- gradient-based optimization. While effective to some extent, these methods require substantial computational resources and lack interpretability into the mechanisms underlying CLIP's behavior.
In this work, we introduce Dyslexify a gradient-free defense that directly targets model circuits responsible for the vulnerability to typographic attacks. By identifying and ablating set of attention heads with demonstrable causal effects, we construct dyslexic CLIP models that are substantially more robust to typographic attacks. Our method scales seamlessly to billion-paramete models, making it applicable to state-of-the-an multi-modi systems. Beyond improving robustness, our approach also enhances interpretability of CLIP models, enabling targeted intervention that computationally efficient and easily integrated into existing pipelines without additional overhead.
The contributions of this work are:

---

Published as a conference paper at ICLR 2026
a Typographic attacks
b Identifying typographic heads
&Robustffying CLIP
original
typographic dataset
identily lypography circuit
original
CLIP
Dyslexic-CLIP
O
O OOOO
pooo
V
Firearm
CLIP
Firearm
Sunrays trees
attacked
attacked
ablate circuit
BANANA
CLIP
with localization mask
00.00
EANANA
Dyslexic-CLIP
Dyslexic-
Banana
CLIP
Firearm
Figure I: Defending CLIP against typographic attacks with Dyslexify a) Adversarial text in images can dominate CLIP's representation and lead miselassification. b) We construct a circuit of attention heads responsible for transmitting typographic information. c) By suppresses the typographic circuit, we defend against typographic attacks without single gradient step.
Mechanistic Understanding: We present the Typographic Attention Score to locate spe- cialized typographic attention heads and demonstrate their causal role in typographic attacks within CLIP models through controlled interventions.
Gradient-Free Defense: We introduce a method that utilizes circuit ablation to effectively defend against typographic attacks, while maintaining general visual capabilities. Due to its gradient- free nature, Dyslexify seamlessly scales to billion- parimmeer models on consumer grade hardware.
Empirical Validation: We validate Dyslexify across diverse set of zero-shot classification tasks, demonstrating that our approach improves robustness to typographic attacks by up to 22.06% typographic version of Imagenet- 100 while maintaining high accuracy non-typographic benchmarks.
Medical Use Case: We show that typographic attacks pose a tangible risk to safety-critical medical foundation models, and demonstrate that Dyslexify substantially mitigate this vulnerability.
Model Release: To facilitate safer deployment, we release a family of dyslexic CLIP models with reduced typographic sensitivity, suitable for use in safety-critica applications.
Our approach provides practical, interpretable, and computationally efficient typographic defense, paving the way for safer multimodal systems without the need for fine-tuning.
RELATED WORK
CLIP models (Radford et al.. 2021) are pretrained on large-scale image-text datasets such as Laion-5b (Schuhmann et al., 2022), aligning global image features with textual descriptions for strong zero-shot transfer. This reliance on textual supervision also makes them vulnerable typographic attacks.
Typographic attacks (Goh et al., 2021) insert written text into an image to maliciously alter a model's behavior. Recent work demonstrates that typographic attacks can degrade model performance, bypass safety filters (jailbreaking) and hijack goals in Vision Language Models (VL.Ms) (Qraitem et al., 2024; Kimura et al., 2024; Gong et al., 2025; Cao et al., 2024; Westerhoff et al., 2025), trigger harmful content generation in image-to tongg-tto-mmgeeee pipelines (Cheng al., 2024), and cause targeted misclassification in object detection and zero-shot classification settings (Materzyńska et al., 2022; Ilharco al., 2022; Azuma & Matsui, 2023; Westerhoff al., 2025; Dreyer al., 2025).
Several defenses have been proposed, they rely either on fine-tuning the model (Ilharco et al., 2022), learning projection matrix (Materzyíska et al., 2022), incorporating learnable text-token called Defense-Pree Preffxx (Azuma & Matsui, 2023), or employing Sparse Autoencoders (Joseph et al., 2025). Crucially, none of these approaches offer gadient-free method for mitigating typographic attacks.
2

---

Published as a conference paper at ICLR 2026
In contrast, work introduces controllable intervention at inference time by directly locating and suppressing the components responsible for typographic sensitivity, without requiring gradients or fine-tuning.
3 MOTIVATION: LOCATING LAYERS OF TYPOGRAPHIC UNDERSTANDING
To better understand CLIP's vulnerability to typographic attacks and to motivate our method, we begin by investigating which layers and components are responsible for typographic understanding using linear probes.
Typographic datasets: We further construct typographic attack datasets from standard image classi- fication datasets D = {(,, y:) Hi by assigning each input example an additional typographic label & A Yi different from the original label ye, and overlaying a corresponding textual description of 2; onto the original image at random location as shown in Fig. 2a. We denote these modified datasets with the suffix "-typo". More information on the datasets is provided Appendix F.
Extending the ImageNet- 100 dataset into the Srageeeet-100-typee dataset, understanding by training linear probes on the cls token embedding at each layer of OpenCLIP models, ranging in scale from ViT-B ViT-bigG (Ilharco et al., 2021).
evaluate typographic
Formally, for a layer we define a linear probe Pt
(x()) = w h +b,
where heis E Id denotes the activation of the model at layer on the cls token for an input sample and wo and are the probe's weight vector and bias term, respectively. We train two types of probes: Ping,e, which predicts the object label and Piypo.,, which predicts the typographic label The accuracy of probe P is denoted by Acc(P). (P).
Fig. 2b shows that Acc( (spoo)) > 0.99 at the final layer for all tested models, indicating that CLIP models can distinguish the typographic classes. Furthermore it shows, that Aco(Prypo.e) is low in early layers, but exhibit sharp increase in the latter half of the model.
Fig. 2c shows that Acc( Pimg..) improves gradually over the layers, while Acc( Cec(Prypo.l c) exhibits a sharp performance rise around second half of the models layers.
Fig. 2d highlights the effect of the attention and the MLP blocks onto Acc(Himged and ACc(P e). Attention layers consistently improve accuracy indicating that they add linearly decodable information to the cls token. In contrast, the MLP layers tend to reduce accuracy. We show Appendix B, that the Intrinsic Dimensionality (ID) of the signal decreases after the MLP blocks, suggesting that the MLP compresses or discards information.
DYSLEXIFY: A DEFENSE AGAINST TYPOGRAPHIC ATTACKS
To defend CLIP against typographic attacks we present Dyslexify: framework for detecting and suppressing typographic circuits. Based on the finding in Section 3 that attention heads are responsible for adding typographic information, we only consider attention heads for the circuit construction. We define a circuit as subset C C Vi, where V denotes the set of attention heads in CLIP
V = {Hie
{O....,T) 7). L (O.....I.))
(2)
with 7 heads per layer and 7. layers in total, where H., denote the ith attention head in layer f.
To robustify a model M against typographic attacks, we conduct circuit-ablation circuit C. Circuit-ablation modifies here only the residual stream of the cls token, leaving all other computations intact. Specifically, the residual update of the cls token is given by
typographic
2 hits + MLP(N(1.)
(3)
h
cla
Σ Ha.t.ote
(4)
where </1 is the cls activation after the layer-& MLP block is applied to W and Hi (2 the contribution of head Hi,e to the els token.
3

---

Published as a conference paper at ICLR 2026
b) Probe performances show: typographic understanding emerges abruptly in the second half of the models layers
a) Imagenet100- is used for contrastive linear probes
Typo
VTT.B
1.0
Linear Probe
VIT-L
hognose snake
MITH
0.8
Probe Accuracy
VT-G
BIT
Layer n
hy i
Layer n+1
0.4
Linear Probe
0.0
Predict Object Label
25
50
100
Layer Position (%6
d) Attention Layers add information to the CLS tokens, while MLPs remove information from
object probes develops continuously
Layer Position
VIT B Layer
0
10
11
1
Typographic Probes
1.0
Object Probe
Accuracy
0.8
Object Probes
Typographic Probe
Attention adds Information
Probbe Accuracy
probes
0.6
objects early layers
0
0.4
0
MLP removes Information
0.0
Typographic understanding later and more abruptly.
MLP Block
Figure 2: Investigating where typographic understanding emerges in CLIP. a) We train two linear probes on all layers of CLIP models. Probe Pimg.e is used to predict the text label of each sample while P trained predict the typographic class. b) Plypo. shows a consistent pattern across all model sizes: typographic information emerges abruptly in the second half of the models layers. e) This trend not true for the object probes Ping.e. Object specific information builds gradually over the layers. Each line in the shaded area represents one CLIP model. d) While attention layers seem add linearly decodable information the cls token, MIP layers remove or obscure information.
We define ablation typographic circuit C as
Ht, (F(l)) - 0 for all Hi,e C,
(5)
while leaving all spatial contributions unchanged. We use Me to denote model M in which circuit is ablated.
4.1 TYPOGRAPHIC ATTENTION SCORE
Building on Hung et al. (2024) we introduce the Typographic Attention Score The to guide the circuit construction. Intuitively the score measures the amount of spatial attention a head H., dedicates to typographic content.
Given head Hi,e and an input x, we write (ze(z) E [o, 1|T+1 to denote the cls tokens attention pattern, where T' the number of spatial tokens and the additional entry corresponds the cls token. We further define the spatial cl s- s-ttentioon pattern (i,e(z) E [0, 1]7 which excludes the cls-to-cls entry (z), such that
Ar,((x) = (Ai,, Az))(
(6)
We write (z) to index the fth element of the pattern. Formally the score is given by:
ΣT T 1=1 1(t) (2)
T Σ
(7)
ΣT (Z)
ZED
4

---

Published as a conference paper at ICLR 2026
Where is a data point, D is the dataset, and 1 is indicator function so that (tt if the input patch associated with the token at index t corresponds to typographic content and is zero otherwise.
4.2 TYPOGRAPHIC CIRCUIT CONSTRUCTION
To defend against typographic attacks without degrading zero-shot classification, Dyslexify constructs a typographic circuit C. The circuit is built iteratively while monitoring the accuracy of the circuit- ablated model Me on non-yypographic benchmark Dimg and typographic benchmark Dyyo ensuring that the accuracy on Ding never decreases by more than threshold E R.
Our procedure consists of steps: rank all attention heads H4,e by their typographic score Ti,s (ii) add heads to C in descending order of Ti, evaluating accuracy after each addition.
Let (cc D) denote the accuracy of model M on dataset D. For each candidate head H. compute
AAciin Acc(M Ding) Acc Acc(MMune Ding), ACCCypp = Aco(Meun. Dyyo) - Acc(Me. DMyo)
(8)
(9)
where AAccing measures the accuracy drop on Damg relative to the base model, and AAcCypp measures the incremental gain on Diyoo from adding head H. the current circuit C.
If Acco & O, the head H. is skipped, as it does not improve robustmess to typographic attacks. If the head is not skipped and A ACCiig < c, the head added to C: otherwise, the algorithm terminates. In addition, if more than l E N heads are skipped consecutively, the algorithm also terminates.
We refer the final model Mc. equipped with the constructed eircuit as the dyslexic model.
Algorithm Dyslexify
Initialize circuit C 0, skip counter f-
2: Set hyperparameters: tolerance max skips k.
3: Rank heads Hi.t by score Ti
4: for head H., in descending order of T3 do
5: Compute AACoing AACCypp
6: if AAccypo < then
7: 8 + $ s 1;
8:
if k then break; end if
9:
continue;
end if
11: if AAccin 2 then break; end if
Add H4,e to C: +.0.
13: end for
Return final circuit C.
5 EXPERIMENTS
5.1 EVALUATING THE TYPOGRAPHIC ATTENTION SCORE
We construct localized typographic dataset consisting of 10,000 natural images from Unsplash originally containing minimal typographic content. To efficiently analyze the attention patterns, we synthetically introduce typographic content at the bottom center of the image, responding to the lowest two token rows in the spatial grid.
For each attention head Hi, we extract the attention pattern (x)(2) over this localized dataset. We use the known spatial bias to define the indicator mask 1(()) E (0, 1}, where (1() = at the typographic region and 1 [tt 0 elsewhere.
Fig. shows the resulting Typographic Attention Scores Ti, for VIT-B, A small subset of heads shows high scores of up to T,, 2 p(T') + (T)),, revealing strong spatial bias towards typography.

---

Published as a conference paper at ICLR 2026
Calculate Typogrephic Tttt Attention
small number of heads have high Ti,
The probe accuracy spikes after layer with high Ti.
38.5
29.0
247
387
Highly attendive to Tyoography
23.7
362
33.6
332
366
335
152
0.
33.0
9.
02.0
Hooees Tuy Euyee
313 186
34
225
186
12.3
22.1
312 327
23
230
12.0
High T, head
Siighlly attentive Tyoography 22.6%
38.
385
37.5
342
15.4
38.0
12.0
13.
13.
4.3
06.1
42.0
10
146 15.7
211
74
Typographic understandlin forms after high headss headss
I13
Not attentive Tyongraphy 12.8%
342 12.9 22.1
322
384
125
40
324 21.6 212
114 244 322 28.1
81
74
18.0
18:2
400
HG3
23.7 35.0 22. 25.0 22.7
Attention Patters
L4
0.5
Layer
Layer
Figure 3: Analysis of the Typographic Attention Score. a) For each head in the model we calculate the Typographic Attention Score e,, utilizing the spatial bias in the Unsplash-typo dataset. b) Depiction of ViT-B's T scores. While most attention heads do not show any spatial bias in their attention patterns, a few attention heads indicate significantly elevated scores, exceeding Tie 2 (TT) + 20 (T). Those heads only occur in the second half of the models layers e) Overlaying the linear probes with significantly elevated Tie scores, highlights an interesting correlation. Only after the attention heads with exceptionally high Ti, scores are passed the model the accuracy of Prypo. begins increase rapidly.
Furthermore we observe, that the spike in Acc(Popo,r) only occurs after layers with high Ti, heads. More results can be found in Appendix K.
5.2 EVALUATING THE TYPOGRAPHIC CIRCUITS
To construct the cireuits C, we set the tolerance to 0.01, set the maximum number of consecutive skips to A: = 10, use the ImageNet- 100 training split as Dimg- and Traageet--100-yppp as Dype- Table 5 shows that the resulting circuits are sparse, covering most 10.1% of T.
Fig. 4 plots Acc(Mc. Dimg) and Acc(Me; Drype) as heads are added. Dyslexify, improves accuracy on ImageNet- mmngeNee-100-typo'ss 's train by more than 20% across all evaluated models, while limiting the 100 drop in ImageNet- 100 accuracy to below 1%. More results can be found in Fig. 9.
VITH
50
Imageeen To0Tsps
30-
Image wwg.eaaaaee 1pp
50-
Imagoeer 103
13
15
20
26
10
23
o'Aclded meads
Warber Abluldd Houds
Number d'Abite
Figure 4: Tradeoff between general accuracy and typographic robustness as a function of the number of ablated heads. Ablations applied in decreasing order of Tie
5.2.1 DEMONSTRATING CAUSALITY OF TYPOGRAPHIC CIRCUITS
We observe that attention heads in C utilize their cls self-attention as attention sinks (Xiao et al., 2023) depending on the presence of typography in the image. Further details are provided in Appendix C. Building on those findings we showcase the casual nature of these attention heads in this section.

---

Published as a conference paper at ICLR 2026
Manipulate attention of heads in
Defending against typographic atlacks improves object recognition
token spatial tokens
0.6
0.00 0.39
0.4
a 0.4
o== 0.6
0.6
0.4
Probability
Control lypographic robustness
0.3
0.0 low
1.0 high
Original model
0.2
typographic robustness
Object label
0.1
Typo label
Firearm: 3.2%
Firearm: 75.4%
0.0
0.2
0.4
0.6
0.8
1.0
Banana: 96.8%
Banana: 24.5%
Qr
Figure 5: Controlling typographic vulnerability by manipulating attention sinks in circuit heads. a) We set the cls token attention to and rescale the spatial token attentions to sum to a. b) Increasing raises attention to spatial tokens, amplifying typographic understanding. c) Decreasing a increases typographic robustness, increasing the probability of predicting the true object class.
To demonstrate the causal role of these heads in typographic vulnerability, we manipulate their attention patterns. Specifically, we construct
Ai, [a, (1--(11a)/1472)
(10)
where e {0.0, O.I...., 0.9, 1.0). The scaling factor (1 1-)aaallAAlll ensures that the attention distribution remains normalized.
We then evaluate the effectiveness of typographic attacks under these manipulations by tracing the predicted label probabilities p(hrex) and p(yyyo) function of on the ImageNet-100-y1 dataset.
Fig. 5 shows that increasing causally reduces the effectiveness of typographic attacks, while decreasing a amplifies This provides direct causal evidence: circuit heads allocate more weight to the el s sink, the attack signal is suppressed; conversely, when is low and spatial attention dominates, p(hyyoo increases, indicating that typographic information is transferred from spatial tokens into the cls representation.
Results:
5.3 EVALUATING DYSLEXIC MODELS
To evaluate Dyslexify we construct dyslexic OpenClip model variants: ViT-B, L. H. G, and BigG and conduct zero-shot classification experiments. Concretely we first evaluate the effectiveness of our defense against typographic attacks and secondly measure the zero-shot object classification capabilities on non-typographic datasets. For each model, we record the accuracy difference between the original model and dyslexic model. A detailed description of the datasets is given in Appendix F.
Results: Table shows that Dyslexify yields consistent robustness improvements across both real- world typographic attack datasets and synthetic benchmarks. The observed gains are substantial up +31% accuracy and occur across all evaluated datasets, suggesting that the identified typographic circuits capture generalizable failure modes rather than dataset-specific artifacts.
Table 2 further demonstrates that Dyslexify preserves performance on standard vision datasets. In nearly all cases, deviations remain within + 11% of the base model, the only exception being ViT-L showing the largest decline (-1.74%) on Aircraft and (-1.17%) on Food-101 101, which is close to the tolerance bound 1%. This indicates that Dyslexify achieves favorable cobustness-accuracy trade-off: substantial robustness gains are obtained while standard zero-shot performance essentially maintained.
7

---

Published as a conference paper at ICLR 2026
Table I: Comparison of dyslexic model performance datasets of typographic attacks across model sizes, showing accuracy changes relative to the base model, with improvements (t) or declines (1). IN denotes ImageNet, and the suflix -T indicates the corresponding typographic version of the dataset.
Real Typographic
Synthetic Typographic
Model
RTA-100
Disentangling
Paint
Food-101- IN-100-T
Aircraft-T
B
68.30112.00
85.00
12.53
66.84
78.27
16.23
L
71.00
60.56,10.00
76.36114555
72.22
82.15
23.34
18.30
72.22
72.91
75.3412126
128.68
29.40 29.40
G
62.00
69.2219.44
71.82116.36
68. 08.76622.06
73.05
27.69
Big-G
72.90
68.33
r9.09
18.64
84.69
f16.2
Table 2: Comparison of dyslexic model performance on non-typographic datasets across model sizes, showing accuracy changes relative to the base model, with improvements (1) or declines (1).
Not Typographic
Model
Aircraft
Food-101
ImageNet-100
B
27.72.0.12
84.97.0.99
75.0010.64
L
34.62.1.74
89.3
79.52,0024
H
43.98r0.12
92.2910.24
83.4010.34
G
44.07
91.47
82.58 10.66
Big-G
50.47.0.39
92.55
84.7210.34
5.4 COMPARING TO BASELINES
We compare Dyslexify to Pefense-Prefix (DP) (Azuma Matsui, 2023). which introduces learnable prefix token for CLIP's language transformer OpenCLIP ViT-L without fine-tuming the full ViT. Following their setup, we train the DP on the ImageNet- 100-typo training split with learning rate of 0.002, batch size of 64, and hyperparameters = 3.0 and = 1.0 for 6 epochs.
Table shows that Dyslexify outperforms DP on two out of three typographic benchmarks, while DP retains slightly higher performance two out of three non-typographic benchmarks. Notably, DP yields a modest accuracy improvement on the corresponding non-typographic ImageNet-100 likely caused by the choice of ImageNet-100-1e the black-box optimization in DP captures features relevant not only to typographic defense but 100
the training set for DP. We hypothesize that
also to ImageNet- 100 classification, thereby limiting its generalization. In contrast, method focuses on key mechanisms relevant to typographic attacks, leading to more robust transfer across non-typographic benchmarks.
Table 3: Performance comparison of Dyslexify and Defense- Prefix (DP) on ViT-L across typographic and non-typogn nonttypographic datasets. For each method we show the accuracy followed by the deviation from the baseline, (1) for improvement, (1) for decline.
Real Typographic
Training
Non- Non-Tyyographic
Method
RTA-100
Disentangling
PAINT
ImageNet- 100
Food-101
Aircraft
Baseline
54.40
50.56
61.81
79.76
90.48
36.36
DP
62.20; 7.80
82.78732.20
71.82:10.01
81.7011.94
19.83
32.94.3.42
Dyslexify
71.00
60.56.10.00
76.36,14.55
79.52.0.24
89.31
34.62,1.74
5.5 DEFENDING AGAINST TYPOGRAPHIC ATTACKS IN MELANOMA DETECTION
Safety domains such as medicine are particularly vulnerable, as Al decisions may impact human lives. Therefore, we investigate whether typographic attacks transfer this setting and
8

---

Published as a conference paper at ICLR 2026
Typographic attacks result in misdiagnosis
Increasing typographic robustness with Dyslexity
1yc20019typo
-15 755
dllanooyic Nevi
Me anoma typo
Typogra
119 119.3%
BCN20K-typo
+6.8%
HAM10K-typo
9%
original
lypographic attack
ISI22019
Standard
-0.3%
malignant (BCC)
benign (NV)
Melanoma
40.3%
CLIP
BCN20K
+0.5%
Base Model Dystexhfied
Dystexic-
malignant (BCC)
malignant (BCC)
+4.5%
CLIP
20
60
100
Accuracy (%) Y
Figure 6: Typographic attacks in melanoma detection. (a) Adding adversarial text may cause CLIP to misdiagnose a malignant lesion as benign. (b) Applying Dyslexify mitigates these failures, increasing robustness to typographic attacks and even improving accuracy in several non-attacked cases.
whether our defense remains effective. Specifically, we analyze WhyL.esionCL.I a foundation model for skin lesion classification (Yang al., 2024), i.c. melanoma detection, based on OpenClip-
We utilize the same setting as in Section 5.1 retrieving the typographic attention scores, but deviate from Section 5.2 in that construct C by setting ISIC2019 Dimg and ISIC2019-Typo Drypo-
The results in Fig. and in Table reveal two key insights: (i) Typographic attacks reduce the zero- shot accuracy of melanoma detection by up 22% and (ii) Dyslexify proves effective in defending a medical foundation model from a relevant attack vector. Not only does Dyslexify increase the accuracy under typographic attack by up 19.3%, but it also increases the base models accuracy in three out of the four datasets. An additional medical use-case is analyzed in Appendix E.
CONCLUSION
We present mechanistic defense against typographic attack in CLIP using interpretability-fiss approach. We reveal that small number of attention heads located in the later layers of the vision encoder are responsible for the effectiveness of typographic attacks. By selectively ablating typographic circuit, Dyslexify is able defend CLIP against typographic attacks without requiring fine-tuning steps, offering practical and interpretable method for controlling model behavior. To our knowledge, this is the first work to address typographic attacks in CLIP through causal interventions. Dyslexify demonstrates that fineggrained control over model capabilities is achievable through targeted architectural manipulations withou! retraining, and thus paves the way for more robust and modular deployment of multimodal models. Beyond standard benchmarks, we further show that typographic attacks constitute realistic threat vector in the medical domain, where they can mislead safety-critical models, and that Dyslexify substantially mitigates this vulnerability. We believe this work motivates a broader shift toward mechanistic interpretability as tool not only for understanding, but for controlling safety-relevant behaviors in deep transformer models. Finally, we release a family of dyslexic CLIP models that are significantly more robust against typographic attacks. These models serve as drop-in replacements for safety-critical applications where the risks posed by adversarial text manipulation outweigh the benefits of typographic understanding.
Limitations and future work: Dyslexify enhances the typographic robustness of the cl token by preventing specialized typographic attention heads from writing to it. However, many multimodal applications, such as LLaVA and IP adapters (Ye et al., 2023; Liu et al., 2023; 2024b), leverage not only the cls token but also spatial tokens, allowing typographic information propagate into downstream tasks. This might limits the impact of Dyslexify on improving robustness in applications, and calls for further investigation into its generalizability to VLM setups.
Furthermore is it standard practice to evaluate adversarial defenses against adaptive attacks (Tramer al., 2020), i.e., attacks that explicitly optimized circumvent the defense mechanism. In our

---

Published as a conference paper at ICLR 2026
case, however, such evaluation is not feasible: typographic attacks are inherently non-differentiable, which prevents constructing adaptive variants that directly optimize against Dyslexify.
Misuse Potential: While we aim to enhance the safety of multimodal systems, acknowledge that our insights into CLIP's behaivor under typographic attacks could exploited by attackers. Specifically, adversarial inputs might be crafted to increase the spatial attention of heads in C, making typographic attacks even more effective.
ACKNOWLEDGEMENTS
This work was supported by the German Research Foundation (DFG) as research unit DeSBi [KI-FOR 5363] (459422098).
ETHICS STATEMENT
We affirm that this work adheres the ICLR Code of Ethics. All authors have read and agree with the Code. We note that our study does not involve human subjects or sensitive personal data. While insights into typographic attacks may inform adversarial strategies, our primary aim is to strengthen robustness and safety of multimodal models.
REPRODUCIBILITY STATEMENT
We have taken efforts to make our results reproducible. Our code is open-sourced and provided to reviewers in anonymized form. All results and the majority of plots in the paper can be reproduced with the released code.
REFERENCES
Hiroki Azuma and Yusuke Matsui. lefense-preity for preventing typographic attacks on clip. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3644-3653 2023.
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food- 101---nmmiinggg discriminative compo- nents with random forests. In Computer vision-ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part VI 13, pp. 446-461. Springer, 2014.
G. Bradski. The OpenCV Library. Dr. Dobb's 's Journal of Software Tools, 2000.
Yue Cao, Yun Xing, Jie Zhang, Di Lin, Tianwei Zhang, Ivor Tsang, Yang Liu, and Qing Guo. Scenetap: Scene-coherent typographic adversarial planner against languala models in real-world environments. arXiv preprint arXiv:2412.00114, 2024.
Hao Cheng, Erjia Xiao, Jiayan Yang, Jiahang Cao, Qiang Zhang, Jize Zhang, Kaidi Xu, Jindong Gu, and Renjing Xu. Uncovering vision modality threats image-to-image tasks. arXiv preprint arXiv:2412.05538 2024.
Noel CF Codella, David Gutman, M Emre Celebi, Brian Helba, Michael A Marchetti, Stephen W Dusza, Aadi Kalloo, Konstantinos Liopyris, Nabin Mishra, Harald Kittler, et al. Skin lesion analysis toward melanoma detection: A challenge at the 2017 international symposium imaging (isbi), hosted by the international skin imaging collaboration (isic). In 2018 IEEE 15th international symposium on biomedical imaging (ISBI 2018), pp. 168-172. IEEE, 2018.
biomedical
Mare Combalia, Noel CF Codella, Veronica Rotemberg, Brian Helba, Veronica Vilaplana, Ofer Reiter, Cristina Carrera, Alicia Barreiro, Allan C Halpern, Susana Puig, et al. Ben20000: Dermoscopic lesions the wild. arXiv preprint arXiv: 00088002888...... 2019.
Cheng Cui, Ting Sun, Manhui Lin, Tingquan Gao, Yubo Zhang, Jiaxuan Liu, Xueqing Wang, Zelun Zhang, Changda Zhou, Hongen Liu, Yue Zhang, Wenyu Lv, Kui Huang, Yichao Zhang, Jing Zhang, Jun Zhang, Yi Liu, Dianhai Yu, and Yanjun Ma. Paddlcoer 3.0 technical report, 2025. URL 05595.
https: /ttt:///rxiii org/ abs / /507 a250
10

---

Published as a conference paper at ICLR 2026
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A aarge-scale hierarchical image database. In 2009 JEEE conference on computer vision and pattern recognition, pp. 248-255. Icee, 2009.
Maximilian Dreyer, Lorenz Hufe, Jim Berend, Thomas Wiegand, Sebastian Lapuschkin, and Wojciech Samek. From what to how: Attributing clip's latent components reveals unexpected semantic reliance. arXiv preprint arXiv:2505. 20229, 2025.
Sedigheh Eslami, Christoph Meinel, and Gerard De Melo. Pubmedclip: How much does clip benefit visual question answering in the medical domain? In Findings the the Association for Computational Linguistics: EACL 2023, pp. 1151-1163, 2023.
Gabriel Goh, Nick Cammarata, Chelsea Voss, Shan Carter, Michael Petrov, Ludwig Schubert, Alec Radford, and Chris Olah. Multimodal neurons in artificial neural networks. Distill, 2021. doi: 10,2391 1 00030. https://distill.pubb200 hmtp::/lllllpp/002//mmm/mmoaa /ttmmmmmmaaal-eeeeeeeeeeeo neurons..
Yichen Gong, Delong Ran, Jinyuan Liu, Conglei Wang, Tianshuo Cong, Anyu Wang, Sisi Duan, and Xiaoyun Wang. Figstep: Jailbreaking large vision- lasion-anguuae models via typographic visual prompts. In Proceedings of the AAAI Conference Artificial Intelligence, volume 39, pp. 23951 233559, 2025.
Kuo-Han Hung, Ching Yun Ko, Ambrish Rawat, I Chung, Winston Hsu, Pin-Yo Yu Chen, al. Attention tracker: Detecting prompt injection attacks in llms. arXiv preprint arXiv:2411.00348, 2024.
Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannanch Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip, July 2021. URL https: /twp///....///////// ort.ol..rrr1..000 5281/ zenodo. 5143773.
Gabriel Ilharco, Mitchell Wortsman, Samir Yitzhak Gadre, Shuran Song, Hannaneh Hajishirzi, Simon Kornblith, Ali Farhadi, and Ludwig Schmidt. Patching vocabblcyy models by interpolating weights. Advances in Neural Information Processing Systems, 35:29262-22 212-222222227... 2022.
Sonia Joseph, Praneet Suresh, Ethan Goldfarb, Lorenz Hufe, Yossi Gandelsman, Robert Graham, Danilo Bzdok, Wojciech Samek, and Blake Aaron Richards. Steering clip's s vision transformer with sparse autoencoders. arXiv preprint S00iv:2500.082299 2025.
Subaru Kimura, Ryota Tanaka, Shumpei Miyawaki, Jun Suzuki, and Keisuke Sakaguchi. Empirical analysis of large languae models against goal hijacking via visual prompt injection. arXiv preprint arXiv:2408.0335 2024.
Xiang Li, Congcong Wen, Yuan Hu, and Nan Zhou. Rs-clip: Zero shot remote sensing scene classification via contrastive vision-language supervision. International Journal of Applied Earth Observation and Geoinformation, 124: 103490 2023.
Fan Liu, Delong Chen, Zhangqingyun Guan, Xiaocong Zhou, Jiale Zhu, Qiaolin Ye, Liyong Fu, and Jun Zhou. Remoteclip: A vision language foundation model for remote sensing. IEEE Transactions on Geoscience and Remote Sensing, 62:1-16. 2024a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023.
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024b. URL https:
//llava-vl.giee git.gbitbbb....... hto/blog//244 01-30- 30- llava-nextt
Mingrui Liu, Sixiao Zhang, and Cheng Long. Wukong framework for not safe for work detection in text-to-image systems. arXiv preprint arXiv:2508..0055) 2025.
S. Maji, J. Kannala, E. Rahtu, M. Blaschko, and A. Vedaldi. Fine iineggrained visual classification of aircraft. Technical report, 2013.
11

---

Published as a conference paper at ICLR 2026
Joanna Materzyúiska, Antonio Torralba, and David Bau. Disentangling visual and written concepts in clip. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 16410-16419, 2022.
A. Mishra, K. Alahari, and C. V. Jawahar. Scene text recognition using higher order language priors. In BMVC, 2012.
Maan Qraitem, Nazia Tasnim, Piotr Teterwak, Kate Saenko, and Bryan A Plummer. Vision-llms can fool themselves with self generaened typographic attacks. arXiv preprint arXiv:2402.00620 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. PMLR, 2021.
8748-8760
Camilo Carvajal Reyes, Joaquín Fontbona, and Felipe Tobar. Towards sfw sampling for diffusion models via external conditioning, arXiv preprint arXiv:2505. 088177 2025.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in neural information processing systems, 01225278-22244 2022.
Florian Tramer, Nicholas Carlini, Wieland Brendel, and Aleksander Madry. On adaptive attacks to adversarial example defenses. Advances in neural information processing systems, 33:1633-1645, 2020.
Philipp Tschandl, Cliff Rosendahl, and Harald Kittler. The ham 1000 dataset, large collection of multi-source dermatoscopic images common pigmented skin lesions. Scientific data, 5(1):1-9. 2018.
Vicente Vivanco, Gaurav Kumar Nayak, and Mubarak Shah. Geoclip: hlip-insprred alignment between locations and images for effective worldwide geo-localization. In Advances in Neural Information Processing Systems, 2023.
Zifeng Wang, Zhenbang Wu, Dinesh Agarwal, and Jimeng Sun. Medclip: Contrastive learning from unpaired medical images and text. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. Conference on Empirical Methods in Natural Language Processing. volume 2022, pp. 3876, 2022.
Justus Westerhoff, Erblina Purellku, Jakob Hackstein, Leo Pinetzki, and Lorenz Hufe. Scam: A real-world typographic robustness evaluation for multimodal foundation models. arXiv preprint 05430:2500404893, 2025.
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309. /0009./7743... 2023.
Yue Yang, Mona Gandhi, Yufei Wang, Yifan Wu, Michael Yao, Chris Callison-Burde James Gee, and Mark Yatskar. A textbook remedy for domain shifts: Knowledge priors for medical image analysis. Advances in neural information processing systems, 37:90683-90713, 2024.
Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. arXiv preprint arXiv:2308.06721, 2023.
12

---

Published as a conference paper at ICLR 2026
CONTENTS OF APPENDIX
A Use of Large Language Models
13
B MLP layers compress information
13
Attention sinks for typographic attention heads
14
Ablation Curves
15
Further details on Medical Application
16
F Datasets
17
G Circuit size
17
H OCR capabilities
17
I Runtime and memory requirements
18
Blurring baseline
18
K Relationship between probes and Tie scores
19
A USE OF LARGE LANGUAGE MODELS
Large Language Models (LLMs) were used in the preparation of this work. Specifically, they assisted polishing the text, generating code snippets, and conducting literature searches. All conceptual contributions, experimental designs, analyses, and conclusions are the sole work of the authors.
B MLP LAYERS COMPRESS INFORMATION
In Section we observe that Ace( (e((Pmgg)) and Acc Acc(Piypee)) increases after attention layers, but consistently drops following MIP blocks. Thus we estimate the intrinsic dimensionality (ID) of cls-token representations across layers using PCA. From a 5% split the ImageNet-100- 100-TTTTo training set, extract residual stream activations for each layer. For each embedding h2 E g we fit PCA and define ID as the smallest number of principal components such that the cumulative explained variance exceeds 95%:
Aj 2 0.95
ID min k
A
レ==
where X, the PCA eigenvalues. A larger ID indicates higher representational complexity. We report ID across layers and token types to analyze how typographic inputs affect the geometry of CLIP representations.
Results Fig. compares linear probe accuracy (left) and ID (right) across layers of a VIT-B model. We observe a consistent pattern: attention layers tend increase both probe accuracy and intrinsic dimensionality, while MLP layers reduce them. This trend is most prominent the middle layers (3-7), suggesting that attention blocks introduce linearly accessible information, whereas MLPs compress or remove
This pattern is not consistent throughout the model. The MLPs at layers 9 and 11 exhibit increases in ID, deviating from the overall compression trend. Conversely, the attention block at layer causes a sharp drop in ID. These exceptions indicate that deeper layers may more specialized roles.
13

---

Published as a conference paper at ICLR 2026
Attention Layers add information to the CLS tokens, while MLPs remove information from it
a) This can observed the probe accuracy
b) And in intrinsic dimensionality analysis
Layor
Layer
11
Typographic Probes Object Probes
ID
350
B
Attention adds
300
Probs Accuracy
component
Attention adds
Trend breaks
200
0.4
last layers
150
100
MLP removes
information
0.0
Attention Block
MLP removes Information
MLP Block
Figure 7: Attention layers increase, MLP layers reduce cls token information. (a) Linear probe accuracy rises after attention blocks and drops after MLPs, indicating improved linear accessibility followed by compression. (b) Intrinsic dimensionality shows a matching trend, especially in layers 3-7. Exceptions include MLPs at layers and (ID increase) and the attention block at layer 11 (ID drop), suggesting deeper-layer specialization.
C ATTENTION SINKS FOR TYPOGRAPHIE ATTENTION HEADS
We analyze the attention patterns of head in layer 6 (Ha,6) in VIT-B, which has the highest Ti score in the model. More specifically we evaluate spatial attention norm ||A), all on ImageNet-100 and ImageNet- -angeNett-000. TypoT
As shown in Fig. 8, ||AA,,lll is systematically higher on ImageNet-1000 100-Typo than on ImageNet-106 While the distribution on typopgrahic images is unimodal, the distribution the original dataset is bimodal. Manual inspection reveals that the text in, such watermarks copyright tags.
mode in ImageNet-100 contains incidental
One possible interpretation of these results is that Hs.6 uses the cls-to-clo s attention as an attention sink (Xiao al., 2023), selectively adjust the impact of this specialized typographic attention head.
Building on these finding we evaluate Hss 's capabilities to predict if sample x originates from ImageNet-100 ImageNet- pogeeNett100-tppo 100-typo. The score ||A4 ROC-AUC of 0.887, indicating that this attention signal can be used as a robust classifier. In comparison, a linear classifier trained on the same task reaches an ROC-AUC of 1.0, but overfits superficial typographic features specific to the ImageNet- Sereeeeet-10--Typp 100-Typo construction. Many images in the original dataset that contain real-world typography are still correctly classified as non-typographic by this probe supporting the conclusion that it does not generalize beyond the synthetic intervention.
14

---

Published as a conference paper at ICLR 2026
Spatial attention norm is selective
Spatial Attention Norm
0.2
0.6
0.8
1.0
5
Higher spatial on typographic images
norm exhibits bimodal distribution on normal images
Density
Typographic Normal
Mode a)
Mode b)
Images without typography
Images with typography
Figure 8: Distribution of the spatial attention norm of VIT-B head H3,6 across ImageNet- 100 and mageNNet-100-Typo The norm is consistently higher for typographic images, while the non- typographic distribution is bimodal. Manual inspection links the higher mode to incidental text, suggesting that the head selectively activates in response to typography, regardless of its origin.
D ABLATION CURVES
VIT-
VITL
VISH
50-
60-
10-
50-
45-
15--
Number ofAclaee feads
Nurbbrr of Abbudd Heads
Number olAMibe Heads
VIT-G
WitdigG
Whrflesiinc...
52
20.
50-
25.
85
80
60-
75
70
Imagahesi-00 Imagaheei-0000 100-732
Image
Image/ 1ma.0/a
50-
20
25
25
Number A'Allde reads
Marber
otAMitt
Figure 9: Tradeoff between general accuracy and typographic robustness as a function of the number of ablated heads. Ablations are applied in decreasing order of Ti.
15

---

Published as a conference paper at ICLR 2026
E FURTHER DETAILS ON MEDICAL APPLICATION
We evaluated Dyslexify in the safety-critical context of melanoma detection, using WhyLesionCLIIP based OpenCLIP ViT-L the underlying foundation model. To construct typographic attack datasets, we introduced textual labels into ISIC2019, HAMI0K, and BCN20k, resulting in paired typographic variants. The typographic circuit was selected using the same procedure in Section 5.2, with tolerance c = 0.01 and maximum skips k = 10.
Table reports detailed results. We find that typographic attacks reduce zero-shot accuracy by up to 22. %, while Dyslexify recovers up to 19.3% accuracy, and even improves performance on non-attacked datasets some cases. These results confirm that typographic attacks pose realistic failure mode for medical Al systems.
Fig. 10 further illustrates that typographic probes perform more strongly in WhyXrayCLIP than in WhyLesionCLIP, which may be explained by X-ray training data. This suggests that the degree of vulnerability depends on the presence of typographic features in the training distribution.
frequent typographic artifacts (e.g., "R" markers)
Table 4: Comparison of dyslexic model performance on WhylesionCLIIP dataset, showing accuracy changes relative to the base model, with improvements (1) or declines (1). The suffix -T indicates the corresponding typographic version of the dataset.
Not Typographic
Synthetic Typographic
Model ISIC2019
Melanoma
BCN20K
HAM1OK
ISIC2019-T
Melanoma-T
BCN20k-T HAMIOK-T
Base
77.80
84.10
31.99
35.66
55.19
62. 62.10
19.65
19.48
Ours
77.47 10.33
84.40 10.30
32.49 70.50
40.20 14.54
70.92 115.73
81.40 119.30
26.46 16.81
24.42 14.94
a) Typographic probes perform more strongly in the lesion model than the xray model
WhyLesionC ip Layer
WhyXrayClic Layer
5 001112231151111770000002222
012345
Object Probe
Daject Probe
Typographic Probe
0.8
Typographic Probe
0.8
Proba Accuracy
Probe Accuracc
0.6
0.6
0.4
0.0
0.0
b) Typographic artifacts in training data (e.g., R-markers) may explain robustness
R
R
Figure 10: (a) Linear probes on CLS embeddings show that typographic probes perform less well in WhyLesionCLIIP compared to WhyXrayCLIIP, indicating weaker encoding of typographic features in the lesion model. (b) Examples of typographic artifacts (e.g., "R" markers) commonly found in X-ray training data, which may influence probe performance.
16

---

Published as a conference paper at ICLR 2026
F DATASETS
For the zero-shot evaluation we tested a variety of datasets, grouped below by purpose.
RTA-100 (Azuma & Matsui, 2023) consists of 1,000 handcrafted typographic attacks, each written on Post-it note and overlaid onto natural images.
Disentangling (Materzyúska et al., 2022) includes 180 typographic attacks, also written on Post-its, designed to probe the separation of visual and textual features in multimodal models.
PAINT-DS (Ilharco et al., 2022) comprises 110 Post-it-based typographic attacks and serves to evaluate patch-level vulnerabilities in vision-language models.
Food-101 (Bossard et al., 2014) is a standard image classification dataset containing 101 food categories, each with 1,000 images.
FGVC-Aircraft (Maji et al., 2013) (referred as Aircraft our paper) is fine-grained classification benchmark consisting of 10,000 images across 100 aircraft variants.
ImageNet- 100 a subset of the ImageNet-Ik dataset (Deng et al., 2009), containing 100 object and animal classes with 1,000 images per class.
ISIC2019 contains 25,331 images available for the classification dermoscopic images among nine different diagnostic Tschandl et al. (2018); Combalia et al. (2019); Codella et al. (2018). In this paper we evaluate the binary classification task into the classes "Benign" and "Malignant".
HAMIOK (Tschandl et al., 2018 includes 10,015 dermatoscopic images labeled into seven diagnostic categories: Actinic Keratoses, Basal Cell Carcinoma, Benign Keratosis-like Lesions, Dermatofiroma, Melanoma, Melanocytic Nevi, and Vascular Lesions.
BCN20K (Combalia et al., 2019) comprises 19,424 dermatoscopic images collected at the Hospital Clínic de Barcelona, annotated into eight categories: Actinic Keratoses, Basal Cell Carcinoma, Benign Keratosis-like Lesions, Dermatofibroma, Melanocytic Nevi, Melanoma, Squamous Cell Carcinoma, and Vascular Lesions.
For the generation of the -typo dataset we used the following fonts: Times New Roman, Georgia, Arial.
G CIRCUIT SIZE
Table 5: Number of selected and total heads Hi,s in circuit C model
Model
Selected
Total
Percentage (%)
B
8
144
5,6
29
288
10.1
H
24
384
6.2
29
480
6.0
Big-G
26
576
4.5
H OCR CAPABILITIES
Dyslexify is designed to suppress typographic understanding in favor of robustness to typographic attacks. As a consequence, we expect a degradation on tasks that require optical character recognition (OCR).
To quantify this effect, we evaluate the base and dyslexic OpenCLIP models on the IITSK word recognition benchmark (Mishra et al., 2012), using the standard small and medium lexicon settings.
Pattps:/ http///w////g/t//e/// keggle. Le. con datasets///essa.. s ambi haw.l/yyaaaamggggeeeeeeeeem imagenet100
ppa p //www. kaggl e oom aset sa bv a 2-12
mages- for-al assi fication
17

---

Published as a conference paper at ICLR 2026
For each image, treat the lexicon entries candidate labels and perform zero-shot classification with CLIP, reporting the top-1 word accuracy.
Table 6 reports the results. Across all model sizes, Dyslexify substantially reduces OCR performance, with drops between 8 and 30 percentage points depending on the model and lexicon size. The effect strongest for the smaller VYT-B and ViT-L models and remains pronounced even for the larger ViT-bigG.
This confirms that dyslexic models not appropriate for applications that benefit from text recogni- tion. Instead, they are intended as drop-in replacements in safety-criticn settings where the risks of adversarial text manipulation outweigh the utility of typographic understanding.
Table Comparison of dyslexic model performance on IITTKK OCR evaluation, showing accuracy changes relative to the base model, with improvements (1) declines (1).
Model
Small Lexicon
Medium Lexicon
B
25.03
11 11.37
L.
p3.27
15.30
H
51.27
26.40.8.57
51.40
27.27
Big-G
54.47
29.40
I RUNTIME AND MEMORY REQUIREMENTS
We benchmark the computational cost of Dyslexify against Defense-Prefix (DP) in terms of wall-clock runtime and memory requirements. All experiments were run on single NVIDIA H200 GPU using identical data pipelines and batch sizes. Importantly, both methods were evaluated using the entire ImageNet-1100 100 training split for circuit construction and prefix optimization, respectively.
Table 7: Runtime comparison (in seconds) of Dyslexify and Defense-Prefix (DP) in seconds on a single NVIDIA H200 GPU. Lower is better.
Model
Dyslexify
DP
VIT-B
3818
145998
ViT-L
23518
23 145
VIT-H
24
39 483
Dyslexify is 3.8x faster than DP on ViT-B, 1.6x faster VIT-H, and comparable VIT-L. The cost gap reflects the fact that Dyslexify requires only forward passes with circuit ablations, whereas DP performs gradient-base optimization
learnable prefix.
Runtime reduction through subset evaluation. The reported Dyslexify runtimes the full ImageNet- 100 training set. In practice, Dyslexify does not require the entire dataset: we observe that using significantly smaller random subset yields nearly identical circuits and comparable robustness gains.
VRAM requirements. Dyslexify further has a low memory footprint. It runs on all evaluated models, including those exceeding 1B parameters, Prefix cannot be trained beyond ViT-L. due to CUDA out-of-memory errors. This makes Dyslexify suitable for low-resource environments where finetuning-basee defenses are infeasible.
single NVIDIA Titan RTX (24 GB). Defense-
J BLURRING BASELINE
Following the reviewer's 's suggestion, we implemented a simple OCR- based defense baseline that removes typographic content directly from the image. We use the PaddleOCR text detector (Cui et al.,
18

---

Published as a conference paper at ICLR 2026
2025) to localize text regions and then apply a Gaussian blur to each detected bounding box using OpenCV (Bradski, 2000)..
In brief, for each input image we (1) run OCR pass to obtain polygonal text boxes; (2) compute the pixel bounds of each box; and (3) blur only the corresponding subregion with a large-kernel Gaussian filter (kernll size 51,
25). No other pixels are modified. This produces text-ablated variant of ImageNet-100-Tyne that preserves the underlying image content while suppressing overlaid words.
This baseline allows direct comparison between Dyslexify (a mechanistic intervention) and purely image-level text-removal defense.
The OCR+blur baseline is strong: it consistently outperforms Dyslexify on ImageNet- 100cc---00yyyppp across all model sizes. This confirms that removing text at the pixel level is an effective strategy for mitigating typographic attacks. On non-typographic datasets, the blur baseline has minimal impact- the accuracy remains within +0.3% of the original models indicating that the image modifications do not substantially harm object-centris features. Dyslexify shows similarly small degradation (<1%), but delivers lower absolute robustmess than OCR+blur on this benchmark.
While OCR-based preprocessing competitive in absolute robustness, it comes with clear limita- tions. First, OCR+blur must be applied at every inference step, adding recurring computational overhead, whereas Dyslexify is executed once per model and introduces no inference-time cost. Second, OCR+blur removes visual information rather than modifying internal causal pathways; it therefore does not provide mechanistic insight into model behavior or contribute to understanding the underlying failure modes.
Overall, the results highlight that OCR-based preprocessing is strong baseline, but Dyslexify serves different purpose: it demonstrates that mechanistie analysis can yield actionable interventions that improve robustness without retraining and without modifying the input image.
We hope that this comparison highlights broader point: while preprocessing -ased defenses can be strong, they do not replace the need for mechanistic approaches. Our results show that even simple circuit-level interventions can yield meaningful robustness gains without retraining, without modifying inputs, and without inference-time overhead. We hope that this work encourages further research actionable interpretability using mechanistic understanding not only analyze models, but to directly improve their safety and reliability.
Table 8: Comparison of dyslexic model performance on ImageNet- 100 with and without blur.
ImageNet-
L.
H
G
Big-G
Normal
74.36
79.76
83.74
83.24
85.06
Blurred
74.32
79.48
83.88
83.20
84.94
Dyslexify
75.00
79.52
83.40
82.58
84.72
Both
74.64
79.50
83.50
82.52
84.60
Table 9: Comparison of dyslexic model performance
1000eeett-100-Tpoo with and without blur.
1mageNet-100-Typo
B
L
H
G
Big-G
Normal
46.94
51.90
54.08
46.70
61.90
Blurred
68.52
75.12
79.98
79.56
80.84
Dyslexify
66.84
72.22
75.34
68.76
78.64
Both
67.86
74.30
80.00
78.48
80.52
K RELATIONSHIP BETWEEN PROBES AND Tie SCORES
As discussed in Section 3, we observe strong correspondence between layers with elevated Tie scores and those where the typographic probe Pryma exhibits sharp increases in accuracy. In this section, extend this analysis across all evaluated model sizes to support the trends previously
19

---

Published as a conference paper at ICLR 2026
shown for VIT-B. Fig. 11 to Fig. 15 visualize this relationship for each model. To improve readability. we transpose the probe accuracy plots: the y-axis denotes the layer index, while the z-axis indicates probe accuracy.
Typographic Scores
18.5 29.8 13.3 20.3 14.7 30.3 12.6 75 18.7 62
9.7
35
21.7 16.2 11.6 12.2 16.6 13.5 37.3 15.2 8.2
33.8
9.0
69.
11
H2
23.5
11.2 13.7
19.4 15.9 23.0 15.6
7.9
5.4
10.
14.7
45.2
38.6
18.5 17.3 14.2 15.4 10.0
12.8
19.7
15.9
4.8
36.7
40.0
Head
16.6
15.7
12.6
7.8
7.1
21.6
HB
21.4
25.1
23.0
39.2 12.9 15.9 21.1 12.2 20.4
33.3
6.1
103
59.
9.1 54.0
54.0-18
11
32.4 21.6 21.2 13.4 14.4 22.2
18.1
8.0
7.4 91 18.0
58.3-110
H11
Thresholded Scores
1.0
Typooraphic Probe
Heads with Tu. 2or
H1
H2
Probe Accuracy
0.6
Heed
HB
-11
H12
10
12
L6
Layer
Figure 11: VIT-B typographic attention scores T.e (top), thresholded and compared with linear probe accuracies (bottom). Layers with higher typographic attention scores align with increased probe accuracy, highlighting correspondence between attention patterns and typographic feature decodability.
20

---

Published as a conference paper at ICLR 2026
Typographic
14X
11.5
12.2
102
17.0
TA
12.7
14.3
18-4
54.5
13.0
17.4
62.2
10.5
16.6
2.2
43.4
261
23.3
24.7
17.4
15.0
22.4
24.
LAT
306
20L1
2533
6-2
13.1
135
13.4
34.5
16.5
14.1
143
21.
37.9
41.3
16.1
187
7.2
6.4
12.3
16.
1.0
12V
15.4
96.5
22A
16.3
X1.
00.3
12.4
3.1
13.0
20.9
15.5
352
4.3
32
264
15.2
2.0
22.7
10.
157
22
124
13.6
78
191
15.0
16.1
556
16.
123
22.6
182
12.
16.7
11.
16.7
13.4
4311
11.7
15.4
855
20.5
278
23.3
180
15.1
1077 17.
12.1
10.8
172
22.9
10.2
38
287
135
82
9.2 12.9 21.1
9/9 15.0 16.7 18.5 35.7 12.5
133
11.8
192 12.9
2400
16.
26.4
143
02.0
16.0
11.0 167
14.7 96
17.6 233
8.
26.5 2044 11.4
244
16.0 115
6.6
944
10.0 23.4 16.4
15.0 143
92.4
16.4
10.4
H92
263
8.
20.7 22.3 23.3 15.4
135
57.6 13.5
3611
16.3
20.0
5.8
32.0
11.8 2011 35.7
30.8 10.2
30
323
44.1
26.7
48.0
105
11.4
12.0 11.3 16.1
16.0
12.5
15.3
15.2
157
10.0
15.6
16.7
15.4
55.9
1144
220
5.8
223
27.0
350
213
Thresholded Scores
Sppograpt:: hrbe
Hoads with 2ox
Figure 12: ViT-L typographic attention scores T., (top), thresholded compared with linear probe accuracies (bottom). Layers with higher typographic attention scores align with increased probe accuracy, highlighting a correspondence between attention patterns and typographic feature decodability.
21

---

Published as a conference paper at ICLR 2026
Typographic
20.2
22.6
222
120
174
203
325
162
233
93
10.3
382
D6L
27.
153
22.2
14L
283
360
6.2
21.1
462
170
383
16.9
25.6
12.7
22.6
323
33.6
405
62
533
432--1
13.5
37.5
40.
344
15.0
353
28.4
21.9
23.5
223
22.0
15.7
164
257
36.
169
55
25
533
432
224
92.
228
2010
20.1
267
26.4
323
262
145
23.3
6.5
29.4
130
7A
15.
93
16.0
359
26-7
13.3
21.
392
150
836
37A
11.0
174
160
233
20.0
262
404
21.3
350
40.8
432
402
50.5
133
359
33.
92
32.8
15.0
12.3
10.2
360
15.5
33
46.7
17.4 141
313
11.5
323
16.
15.7
385
264
244
25.5
342
31.7
335
27.5
31.8
323
45.5-192
11.9
2033
1500
17.3
15.5
136
3544
21.
227
153
22.7
41.2
357
36.
112
100
10.
16.7
302
23.6
262
300
367
204
26.3
13.4 22. 36.4 10.1
7.5
16.0
13.5
192
352
15.7
22.6
21. 143
10.3
326 140 22.6 12.0 323 21.
383
11.7
297
22.6
155
352
303
203
23.7
35.8
266
Thresholded Scores
-1
Figure 13: ViT-H typographic attention scores T.Ą (top). thresholded and compared with linear probe accuracies (bottom). Layers with higher typographic attention scores align with increased probe accuracy, highlighting a correspondence between attention patterns and typographic feature decodability.
22

---

Published as a conference paper at ICLR 2026
Tyoegeapii: ScorEs
210
114
110
18.0
Therchatted Screes
Figure 14: VIT-G typographic attention scores Ti (top), thresholded and compared with linear probe accuracies (bottom). Layers with higher typographic attention scores align with increased probe accuracy, highlighting correspondence between attention patterns and typographic feature decodability.
23

---

Published as a conference paper at ICLR 2026
Figure 15: ViT-BigG typographic attention scores Ti (top), thresholded and compared with linear probe accuracies (bottom). Layers with higher typographic attention scores align with increased probe accuracy, highlighting correspondence between attention patterns and typographic feature decodability.
24


Reviews:
[{"review_id": "C4XmE8FQhR", "reviewer": "Reviewer_scR4", "summary": "This paper proposes an image recognition method that is robust against typographic attacks. The authors first identify the key modules responsible for typographic attacks. Then, they compute scores for the attention heads within each module and suppress typographic attacks by removing modules based on these scores.", "questions": "I would like to hear the authors' opinion regarding the training aspect. Specifically, compared to other methods in the literature, what kind of data is actually required for the training in the proposed method, and how much difference is there in computational cost or time?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "The strengths of this paper are as follows.\n\n## Analysis in Sec. 3\n\nThe analysis in Section 3 and Figure 2, where the authors identify the modules necessary for typographic attacks, is fascinating. By examining the model's actual responses, they determine that the later stages are more critical and further discover that the attention layers contribute additional information. This type of analysis is significant and is expected to inspire further research and analysis.\n\n## Design Principles of the Proposed Method\n\nThe proposed method described in Sec. 4.2 and Algorithm 1 determine module utilization by considering performance on both the standard dataset and the typographic dataset, which is a direct and practical approach. This simple and straightforward design principle is clear and compelling.", "weaknesses": "The weaknesses of this paper are as follows. While I like the proposed method itself, the inconsistencies with the claims in the introduction and the notational issues described below lower the overall completeness of the paper. Therefore, my current evaluation leans slightly toward rejection, though it is very close to the borderline.\n\n## Claim of \"Training-Free,\" but Dataset Actually Required\n\nIn Section 1, the authors claim that their proposed method requires no training (\"... we introduce Dyslexify, a training-free defense that ...\"). However, as far as I understand, to execute the proposed method in Algorithm 1, it is necessary to compute `Acc_img` and `Acc_typo`, which in turn require access to the datasets. Therefore, the proposed method involves a form of training. Hence, the claim made in the introduction appears to be inaccurate.\n\n## Notational Issues\n\nThere is room for improvement in the paper's notation.\n\n- In Eq. 3, $h_\\mathrm{cls}^l$ is not properly defined. There may be other variables used without clear definitions as well.\n- In Eq. 4, the expression $H \\mapsto 0$ is used, but $\\mapsto$ typically denotes a mapping between input and output values, so this notation implies the existence of a function $f(H) = 0$, which is likely a different meaning from what the authors intended. Perhaps what they meant was to set $H$ to zero, i.e., $H \\gets 0$?\n- In Eq. 6, the term $A_{i, l}^*(x)_t$ appears identically in both the numerator and denominator, so it should cancel out. If this is not the case, then there is likely a notational or typographical error in the equation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CahnozUWye", "reviewer": "Reviewer_xwqg", "summary": "The paper proposes a training-free mechanistic method to defend CLIP models from typographic attacks. The attacks considered are adversarial insertion of misleading text into images, which causes CLIP to misclassify the image. The authors first analyze CLIP’s vision encoder and identify a small set of attention heads in later layers that causally transmit text information (OCR) to the CLS token. They then locate the most OCR-sensitive heads. Then they ablate this “OCR circuit” by zeroing out the identified heads’ contributions to the CLS token during inference. This ablation improves robustness to typographic attacks, shown via experiments on ImageNet-100-Typo, while maintaining standard classification accuracy (<1% drop). The strength of the method is that it does not require fin-tuning. The authors also validate their approach on medical foundation models (melanoma detection) and release a family of “dyslexic” CLIP models.", "questions": "I'd like the authors to answer questions 1-4 in the weakness section, specifically\n\n1) does the method perform consistently better than the baselines on a variety of datasets?\n\n2) how does this method compare to a simple OCR + remove baseline, in terms of ease of implementation, computation costs and accuracy?\n\n3) what is the computational cost of the method, and how does it compare to other methods?\n\n4) how would the method perform on tasks that need OCR?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper makes several contributions\n\n1) A thorough investigation of where OCR capabilities emerge in the CLI Pmodel\n- showing that OCR capabilites emerge in the second half of the CLIP layers is an interesting observation\n- showing that MLP layers reduce OCR information while attention heads improve it\n\n2) Disabling a small typographic circuit can make CLIP robust without retraining. \n- finding the circuit is interesting but alone does not guarantee that it can be disabled to remove OCR function\n- the paper shows that it indeed can, which means that OCR is highly isolated in the model's attention heads, \n\n3) Mechanistic interpretability can help us come up with practical algorithms\n- personally, this is the first work I've seen that uses circuits to do something useful, ie create effective safety interventions\n- the approach was evaluated on a variety of image datasets, including medical images", "weaknesses": "The overal idea in the paper is interesting,but a few questions remain unanswered for me.\n\n1) Experimental results. \nThe main table comparing the proposed method to baselines is Table 1, which shows a performance comparison of the proposed Dyslexify and the baseline Defense-Prefix (DP) measured in terms of the accuracy of the method on classifying data affected by typographic insertion attacs. If I understand correctly, higher is better. While Dislexify beats the DP baseline on RTA-100 and Paint dataset, it is considerably worse on the Disentangling dataset. On average, the proposed method gets 69% while the DP baselines gets 72%. This raises the question of whether the method is consistently better than existing methods on a variety of datasets.\n\n2) Baselines. \nSome simpler defenses are not considered as baselines. For example, using an off the shelf OCR model to identify text and blurring it out or removing it in some way directly in the image. This would destroy the text information at the source (image) instead of relying on the ablation process to \"catch\" the OCR information inside the network. It is possible that this approach does not work as well or reduces performance on the primary tasks, but it should be considered. \n\n3) Unclear computation cost. \nAlthough the proposed defense is training free, it is not computation free. The ablation process requires some computation to identify the circuits, which presumably needs to be done for each new model. Furtthermore, a new typographic dataset must be constructed for each new problem. In the paper the authors create an attack for the ImageNet dataset to identify the attention heads. It is not clear how well this will transfer to new tasks that are not ImageNet.\n\n4) What about tasks that need OCR?\nThere are some image classification tasks where the ability to understand text is helpful for the model. By completely ablating the OCR capability, how does this affect performance on such tasks? For example, fine-grained classification of airplanes by airline could benefit from reading the text of the airline; classifying food or drink products could benefit from OCR as well to read the packaging text.\n\n5) (minor) CLIP models have also been shown to be susceptible to *graphic* attacks that include text OR logos. It would be good to comment on the method's applicatbility to defending against such attacks, but not required.\n\n6) (minor) I'm not sure how I feel about the name 'Dislexic' for the method. But that's a personal preference, does not affect my score.\n\n7) The medical case study (melanoma detection) is compelling rhetorically but I'm not sure how practical it really is. It uses synthetic attacks (overlayed words like “benign” or “malignant”) rather than realistic clinical artifacts. There’s no discussion of whether such text would appear in practice, or who may want to create such attacks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "24nAvJNcPJ", "reviewer": "Reviewer_SCJM", "summary": "The paper proposes Dyslexify, a training-free defense that select attention head circuits using attention head score to defend the model from typographic attacks. Through layer-wise probing and attention-pattern analysis, the authors identify a small set of late-layer attention heads that specifically transmit typographic information to the CLS token. The method yields substantial robustness gains on several typographic attack benchmarks while preserve the model performances.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper provides a clear causal explanation of where typographic vulnerability arises inside CLIP.\n\n2. Dyslexify does not require fine-tuning, gradients, or retraining, making it scalable to large CLIP backbones.\n\n3. The method improves the robustness significantly across multiple datasets. \n\n4. The presentation and explanation are rather clear.", "weaknesses": "1. The defense intentionally reduces CLIP’s ability to read text within images, which may negatively affect tasks where typography is a key visual feature (e.g., signs, product labels, book covers). The authors should include ablation studies measuring performance drops on text-dependent classes to quantify this trade-off.\n\n2. The typographic attention score is computed using English printed text placed in fixed locations. It remains unclear whether the identified circuit generalizes to other forms of typography, such as stylized or handwritten fonts, varied color/contrast conditions, or non-English scripts.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hhXIjGGwFd", "reviewer": "Reviewer_hKir", "summary": "This paper addresses typographic attacks on CLIP models by identifying and ablating specific attention heads responsible for processing typographic information. The authors introduce the Typographic Attention Score to locate specialized heads in the latter layers of CLIP's vision encoder that extract and transmit text information to the CLS token. Their training-free defense method, Dyslexify, selectively ablates these heads to improve robustness against typographic attacks on ImageNet-100-typo while maintaining standard accuracy.", "questions": "1. Transferability across CLIP variants. The experiments focus on OpenCLIP models. Do the identified typographic circuits transfer to other CLIP variants? Are the same head positions specialized for typography across architectures?\n2. Interaction with other capabilities. Does ablating typographic circuits affect related capabilities like OCR, scene text understanding, or processing images with incidental text (e.g., street signs, book covers)? \n3. Mechanism of typographic understanding. Why do typographic heads emerge specifically in the latter half of the network? Is this related to the hierarchical feature learning in vision transformers? Does pre-training data composition (text-heavy vs. text-sparse images) influence circuit formation?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper provides a novel mechanistic interpretation approach for the localization of typographic processing, such as linear probes showing abrupt emergence of typographic understanding in later layers and the Typographic Attention Score identifying specific high-attention heads. The causal intervention experiments demonstrate that manipulating attention weights in identified heads directly controls typographic vulnerability.\n2. The paper introduces practical training-free defense with scalability, unlike existing defenses that require gradient-based optimization. Dyslexify operates purely through inference-time circuit ablation. This approach scales seamlessly to billion-parameter models and achieves superior performance compared to training-based baselines.", "weaknesses": "1. Limited analysis of failure modes and edge cases. The paper does not thoroughly investigate when and why Dyslexify fails. For instance, what types of typographic attacks remain effective after ablation? The paper would benefit from failure case analysis showing examples where the defense is insufficient and discussing the underlying reasons.\n2. Incomplete evaluation of spatial token applications. The authors acknowledge that downstream applications use spatial tokens rather than only the CLS token, which limits Dyslexify's applicability. However, no experiments quantify this limitation. The paper may include experiments like (a) whether typographic information persists in spatial tokens after CLS ablation, (b) how this affects downstream task performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)