Pay Less Attention to Function Words for Free Robustness of Vision-Language Models
Authors: Qiwei Tian
Chenhao Lin
Zhengyu Zhao
Chao Shen
Venue: ICLR 2026 Poster (2026)
Keywords: Vision-Language Model
Concentrated Attention
Adversarial Robustness
Primary area: alignment, fairness, safety, privacy, and societal considerations

Abstract: To address the trade-off between robustness and performance for robust VLM, we observe that function words could incur vulnerability of VLMs against cross-modal adversarial attacks, and propose Function-word De-Attention (FDA) accordingly to mitigate the vulnerability brought by function words. Inspired by differential transformers, our FDA calculates the original and the function-word cross-attention within attention heads, and differentially subtracts the latter from the former for more robust alignment. Comprehensive experiments include 2 SOTA baselines under 6 different attacks on 2 downstream tasks, 3 datasets, and 3 models. Overall, our FDA yields an average 18/13/53\% ASR drop with only 0.2/0.3/0.6\% performance drops on the 3 tested models on retrieval, and a 90\% ASR drop with a 0.3\% performance gain on visual grounding. We demonstrate the scalability, generalization, and zero-shot performance of FDA experimentally, as well as in-depth ablation studies and analysis. Code is available at https://github.com/michaeltian108/FDA.

Published as a conference paper at ICLR 2026
PAY LESS ATTENTION TO FUNCTION WORDS FOR FREE ROBUSTNESS OF VISION-LANGUAGE MODELS
Qiwei Tian, Chenhao Lin, Zhengyu Zhao, Chao Shen School of Cyher Science and Engineering, Xi'an Jiaotong University, Xi'an 710049, China mi chael tqwê stu. xjtu. edu. cn, {1 (linchenhao, zhengyu. (aangyu.zhaap @xjtu.edu.cm, chaoshen@main x tu. en
ABSTRACT
To address the trade-off between robustness and performance for robust VLM, we observe that function words could incur vulnerability of VI.Ms against cross- modal adversarial attacks, and propose Function-word De-Attention (FDA) ac- cordingly to mitigate the vulnerability brought by function words. Inspired by differential transformers, our FDA calculates the original and the function-word cross-attention within attention heads, and differentially subtracts the latter from the former for more robust alignment. Comprehensive experiments include 2 SOTA baselines under 6 different attacks on 2 downstream tasks, 3 datasets, and models. Overall, our FDA yields an average 18/13/53% ASR drop with only 0.2/0.3/0.6% performance drops on the 3 tested models retrieval, and a 90% ASR drop with a 0.3% performance gain on visual grounding. We demon- strate the scalability, generalization, and zero-shot performance of FDA experi- mentally, as well as in-depth ablation studies and analysis. Code is available at http://gghhbb.coom//mchaiaaall
INTRODUCTION
Building robust vision-language models (VLMs) has garnered profound academic focus because of the necessity of defending VLMs against various adversarial attacks. To this end, many works have been proposed to defend models against adversarial images by enhancing model robustness (Schlar-
mann et al.,, 2024 Mao al. 2022), purifying perturbations (Yoon et al. [021] Lei et al. (025), or detecting potential adversaries (Metzen et al.) (017)] Klingner et al.) (022). Among them, adver-
sarial training (AT) shows superior enhancement in the robustness of VLMs, at cost of significant performance drops and computational costs.
In this paper, we propose enhance adversarial robustness from overlooked yet intuitive per- spective. We hypothesize that, due to the innate semantic gaps between the clean image and targeted texts, cross-modal perturbation are likely guided towards closer tokens (i.e., smaller semantic gaps). In this regard, function words, which carry little semantic information but have high word frequency, become more approachable than other tokens, incurring unwanted vulnerability. To demonstrate the above hypothesis, we record and visualize the relative clean R @ and Attack Success Rate (ASR) when removing verbs, adjectives, and function words, compared to not removing. As shown left of [ only function words reduce ASR among all word types and cause the least clean perfor- mance drop. The results qualitatively confirms the impact of function words regarding in bringing adversarial vulnerability. We further provide the attention maps (Sclvaraju al. 2017) to visualize the impact of removing different words, presented in the right of Fig All above results validate our above hypothesis, implying that function words carry certain adversarial semantic, and remov- ing them properly could tighten the cross-modal alignment, defending VLMs against attacks.
Consequently, inspired by the setting of differential transformers (Ye et al. 2024) and differential amplifiers, we propose Function-word De-Attention (FDA) as the first method to build robust VLMs by refining languae alignment. Specificallyy, our FDA works by deploying a parallel pipeline on multi-attention heads within fusion-encoders, calculating the cross-attcntion between function
"Corresponding author.

---

Published as a conference paper at ICLR 2026
le Remove
Clean Image
Adversarial Image
Aorieec All
Original
Original
Bemoving Adjjctiress
Removing
Removing
Original Text:
young fonale stadout performing downmarrd kick to brialt baard by her karate justructor
Figure Left: Histograms on of relative clean R@1 and ASR of remove, Removing Adj, Verbs and Function words. Results are collected from Flickr30k retrieval testset on ALBEF. Re- sults qualitatively show that removing function words helps reduce ASR and maintain clean performance. Right: Attention maps of VLM under white-box attacks through perturbed images. The texts are given at the bottom, with words marked by corresponding colors. VLM correctly recognizes the corresponding character the clean image given the token her, but gets distracted towards the incorrect character on the adversarial image. Only removing function words helps the VLM 'looks back at" the correct character.
words and the input images, i.e., distractions. We further softmax along the dimensions of visual and textual tokens to highlight the most misleading textual/visual tokens. Finally, subtract the above distractions from the original attention for the output. To validate the effectiveness of FDA, SOTA baselines, 3 models, tasks, 3 datasets, and
conduct comprehensive experiments attacks. Overall, our FDA yields an average 18/13/53% ASR drop with only 0.2/0.3/006%% perfor- mance drops the tested models for retrieval, and 90% ASR drop with better clean performance on grounding. We also find that FDA improves the zero-shot performance.
Overall, our contributions summarized as follows:
1) We identify that function words are vulnerability for VLMs and subsequently propose Function- word De-Ate Ee-Aeettioon (FDA) pay less attention to function words for more aligned vision-language models with free robustness.
2) We conduct comprehensive experiments on two SOTA baselines, models, tasks, and 3 datasets, under 6 attacks, and validate the effectiveness of FDA in enhancing robustness while preserving performance.
3) We provide ablation studies to show the hyperparameters insensitivity, backbone generalization, and zero-shot performance enhancement of our method with in-depth analysis.
RELATED WORK
Adversarial attacks vision-language models. In light of the advancement in VLMSs (Li al.
2021 2022 Radford et al. 2021 Bai et al. 2023 Team et al.] 2024 Bai et al.) 2025 Yang et al. Liu [023))severa several aspects (Li al. 2025b a Zha et al. 2025), such as token compres- sion and broader capabilities, have garnered enormous research interests. Among them, adversarial attacks VLMs become the most threatening direction they fool VLMs into incorrect mislead- ing outputs. Recent studies on white-bor box attacks (Croce & Hein 2020 have exhibited impressive results. Besides. black- box attack:Zhang al.) (2022); Lu al. (2023): Yin et al. (2023); He et al.
2025
2023 Tian et al. (2026) have also demonstrated significant effectiveness against, pre-trained VLMMs through transferable cross-modal attacks.
Adversarial Defense on vision-language models For defenses, adversarial training (AT) RRice (ct al. 2020 Zhang et al., 2019) Tian et al. 2024) has exhibited significant effectiveness in de- fending models against various adversarial attacks against classification, retrieval, etc. Several AT- based methods (Schlarmann et al. 2024; Mao et al. 2022) have demonstrated impressive robustness boost CLIP models. However, AT notoriously well-known for downgrading performance sig- nificantly due to the inclusion adversarial examples into training. Although Schlarmann et al.) 2024) proposed FARE the visual embeddings of vanilla models supervision to balance the
2

---

Published as a conference paper at ICLR 2026
Concat
Function Word De- Attertion (Bq.4)
Attack Succeed
Altack Defended
Cross-Attention Calculation (Eq. 2/4/5)
Imsgee
Alignnent
V
Vlayer
Layer
tayer
Antocl (succeed)
*
Attack (lefendel)
FV
Fy
FT
FTy
Figure 2: Left: An illustration of our Function-word De-Attention (FDA) method, On the exist- ing process of attention calculation, which uses Fv and Fr, we add parallel pipeline to calculate the attentions between function words Fr, and the images Fv. Afterwards, the function-attention passes control gate G before entering the FDA module (triangle) differentially subtract distrac- tions as presented in oq Right: An illustration on how FDA defends attacks. We speculate that attacks can cross the boundary easily to cause misalignments vanilla models (top), and by remov- ing fundtion-word distractions, models can learn a tightened embedding (bottom), preventing such misalignments.
trade-off between clean performance and robustness, the performance drops remain considerably noticeable. Besides, the high computational costs also hinder broader applications in practice.
METHODOLOGY
In this section, first provide brief preliminary for the original calculation pipeline of cross- attention and introduce our Function-words De-Attention (FDA).
3.1 PRELIMINARY
For given textual/visual encoder T /VV input images / and texts T are fed into respective encoders with their attention masks Mr/ Mr, for the embeddings Fr, Fv E Rdd:
Fr = = T(T Fv = V(I
(1)
Then, cross-attention scores caleulated by inputting these hidden states into the fusion encoder:
(FFr))((v) dim -1) v(Fv)
AuL.B = 50f lmmaa
Vdz
where Q/K/V is the query/key/value layers, and Hl is the index of layers and attention heads.
3.2 FUNCTION-WORD DE- DE-ATTENTION (FDA)
Built upon our previous hypothesis that function words are potential distractions in vision-language alignment, we propose Function-word De-Attention (FDA) to remove such distractions: we add parallel pipeline upon the existing cross-attention calculation specifically acquire the cross- attention between function words and the input images, namely the distraction, and then subtract them from the original attention. An illustration of our FDA is given in Fig We first parallelly extract the features of all function words (denoted as Tς) within the input texts by masking all other tokens, excluding [CLS]]
3

---

Published as a conference paper at ICLR 2026
Fr, = Tr, E
(3)
where Mr, is the function word mask, and D is the function word dictionary, which is dictionary shortlisted from the stopwords list in (Li et al. 2020 We then use a parallel pipeline to calculate function words' attention scores:
SL. Q ((FV))
ddz
With the function words attention scores, we then conduct softmax along the dimensions of visual tokens and textual tokens, respectively. In this way, we highlight the visual tokens with false activa- tion under token words or the most misleading tokens with the largest visual activation:
Att = softmar (sa, ,dmm -1)V,
Att softmar (S1,"" ,dim - (5)
Afterwards, subtract both attention from Att individually and take the minimum value as the final attention scores, with & being a control gate for subtraction intensity.
Att min (Att G(Âtt, Att ((Att..))
(6)
Finally, after complete calculation of FDA, denoted as D(-), we concatenate the attention scores from all attention heads:
Att L.H Concal (FDA(
FDA can be flexibly implemented on any number fusion layers/attention heads, as each may specialize differently (Kang et al., 2025). A general intuition is to remove these distractions in the early layers instead of the later ones to avoid possible 'absorption', but not so exquisitely upfront that it may undermine the contextual integrity of the textual inputs. We provide ablation and analysis in Sec 4.3
4 EXPERIMENTS
Vision-Language DatsesDs To thoroughly evaluate the performance and robustness of FDA, incorporate several downstream tasks, including Text-to-image//magttoo--- (T2IR/12TR) and Visual Grounding (VG). For datasets, we use the Flickr30k et al. 2015) and MSCOCO (Lin et al. 2014) dataset for retrieval, and RefCOCO+ Yu al. 2016) for VG.
Retrieval
Models. For T2IR/12TR, we test our method on the ALBEF (Li et al. 2021). TCL Yang et al.
2022), and BLIP(Li et al. 2022), using 4AM/14M/124M pretrained images, respectively.For VG. we use ALBEF All models use the ViT-B/32 (Dosovitskiy et al.] [202]) as visual encoders and BERT DDvlinn al. 2019) as textual encoders. Specifically,, TCL shares the same backbones as ALBEF but uses different training strategy (triplet contrastive learning), while BLIP uses a larger pre-trained encoder with an extra decoder. CLIP (Radford et al. 2021) is not included due to the absence of fusion encoders.
Baselines. As for baselines, we adopt the two SOTA methods for robust CLIP, i.e., TeCoA (Mao) et al. 2022) and FARE (Schlarmann al.) 2024), on all the models adversarial fine-tuning base- lines. To account for the robustness and accuracy trade-off, we lower the perturbation strength of each method to ensure similar clean performance as FDA, such that TeCoA and FARE serve as a reference to compare robustness. Specificallyy, we use Image/Image-to-Ters Text Retrieval, adversarially fine-tune using TeCoA and FARE for 4/1/1 epochs for ALBEF/TCL/BLIC For Visual Grounding, we adversarially train models with both TeCoA and FARE for epoch, as both methods ineur significant performance drops.
for TeCoA and FARE. For Text-to-
Attacks. To thoroughly evaluate the robustness of our models, we test all models with three ad- versarial attacks and use the average of all attacks for robustness evaluations. Specifically,
4

---

Published as a conference paper at ICLR 2026
Table 1: Attack success rate (ASR) of PGI/APGID/MAPGII (masked APGD) against for Text- to- po-Image/Image-to-Pee Text Retrieval (T2IR/12TR) on Flickr30k and COCO. Results are presented in percentage (%). 1 1L indicates increased/decressed ASR (higher values preferred). f indicates higher performance than clean models. (Full results are given in Sec the Appendix.) Our FDA consistently shows the best performance and overall robustness on ALL models.
Text-to-lmage Retrieval
Image- text Text Retricval
ASR drop
Clean (R@1)
ASR
ASR (
PGD
APGD
MAPGD
Ause f
No Defense
95.90
3.38
11.68
65.88
85.60
0.71
14,98
58.85
TeCoA
91.20
2.56
19.39
73.12
81.44
0.55
17.45
61.30
3.02
FARE
91.10
2,46
17.29
70.15
81,48
0.55
16.55
65,90
5.36
ALBCI
3.37
12.44
No Defense
8.72
16.09
80.92
7.20
15.89
77.14
TeCoA
91.20
9:13
19.34
K5-48
81.44
4.60
18:45
79.60
2.15
PARE
91.10
9.27
18.60
86.25
81.48
5.20
18.35
80.60
2.48
95.60
7,90
13.70
75.80
85.50
13.70
14.69
Defense
84.02
60.79
Flickr
TeCoA
92.10
11.08
66.31
59.11
80.40
4.10
70.80
46.85
2.78
FARE
91.20
11.72
67.47
60.98
78.22
4,60
61.25
47.85
1.09
TCL
FDA
94,40
8.52
48.38
68.48
83.82
3.30
44.50
57.50
14.29
Defense
37.66
84.02
TeCoA
92.10
44.29
80.62
80.08
80.40
35,40
76.60
67.95
4.16
FARE
91.70
46.21
81.03
79.64
78.22
38.05
76.95
67.35
6.12
FDA
94,40
30.36
58.64
86.24
83.82
24.25
56.50
77,80
13.55
Defense
25.10
87.30
19.28
48.67
FARE
89.7
20.24
66.53
54.92
77.72
10.00
58.00
46.65
3.09
DLIP
FDA
96.50
7.66
18.96
40.99
86.84
5.50
13.75
35,00
51.60
V7.30
86.39
62.39
PARE
89. 89.70
66.29
92.35
80.49
77.72
67.30
90.50
82.50
7.04
FDA
96.50
15,86
28.37
60.64
86.84
14.45
16.30
55.50
56.36
No Defense
77.60
0.95
11.01
30.47
60.70
0.35
8.86
19.10
18.56
34.23
2,87
FARE
69.2 68.28
0.26
22.66 68
32.71
53.58
0.02
14.59
16.76
0.53
COOD
ALBEF
FDA
77.70-
0.84
9.65
27.60
60.63
0.28
8.03
18.02
9.28
No Defense
77.60
4.71
14.48
51.20
60.70
2.41
12,18
36.17
68 04
1,40
16.08
FDA
77.70-
3.82
11.87
44.92
60.63
2.05
10.57
32,83
14.43
Projected Gradient Descent (Madry al., 2017) and AutoAttack (Croce & Hein 2020). denoted as PGD and APGD. As for the adaptive attack, we apply function word masks to the input texts for APGD to evade our FDA, denoted Masked APGD (MAPGD). All attacks are fully white-box,
i.e., the attackers can access FDA. For each attack, we follow the settings in Mao et al. (2022) and Schlarmann et al. 2024 to attack images with perturbation bounded by 4.0 255 255 Specifi- cally, for targeted attacks in T2IR/I2TR, we apply circular shift targeted to ensure non-overlapping unmatched targets. For targeted attacks VG, we follow the settings of Gao et al. (2024) to not
apply patched attacks.
We use the common metric, Attack Success Rate (ASR), to indicate the efficacy of all adversarial attacks. For ASR defended model and the baseline model, denoted as ASRM and ASRB, respectively, we calculate the relative ASR change in percentage using AASR = ASR ASRM x 100%. Consequently, a positive/larger ASR indicates improved/stronger robustness, while a negative/lower A ASR implies decreased/weaker robustness, with 0% (100%) meaning no robustness gain (completely defended). Details are given in the Sec B the Appendix.
Metrics.
ASna
Implementation Details. Since our FDA parallelly computes distracted attention for subtraction, fine-tuning models with FDA is identical to downstream fine-tuning without extra modifications or parameters. Following the settings in Li et al. (202I), we fine-tune the model by 10 epochs and the last-epoch model for all tasks/models. For the layer index, use L and H for all models/tasks/datasets with ablation studies on the selection of the layer and attention head indices for all models and tasks in Sec 43
5

---

Published as a conference paper at ICLR 2026
Table Attack success rate (ASR) of PGD/APGD/MAPGD against for Visual Grounding (VG) on ReiCOCO+. Results are presented in percentage (%). 7 /4 indicates increased/decreased ASR (higher values preferred). i indicates higher performance than clean models. (Full results are given
in Sec C of the Appendix.) Our FDA consistently leads in performance and overall robustness
on ALL models.
Clean (Acc)
ASR on Test A Split ())
ASR on Test B Split ())
Avg ASR drop
100
Defense
Val.d
Test
Test
PGD
APGD
MAPGD
PGD
APGD
MAPGD
AAsa 1
Defense
58.50
65.90
46.30
6.70
11.16
11.16
6.07
7.08
7.42
TeCoA
57.20
64.70
45,00
6.81
7.72
8.01
3,39
6.28
6.10
21221
FARE
56.40
64.20
44.70
6.13
10.42
9,96
4,54
6.72
6.21
12.08
FDA
58.10
66.801
46.10
1.36
2.41
1.80
0.00
0.00
0.00
93.16
No Defense
58.50
65,90
46.30
7.89
11.16
11.75
4.39
8.06
8,06
TeCoA
57.20
64.70
45,00
6,57
8.17
8.46
3.56
6,10
6.44
1
FARE
56.40
64.20
44.70
6.74
9.66
10.27
4.03
7.06
6.55
13.28
FDA
58.10
66,80
46.10
1.50
2.10
2.10
0.34
0.00
0.00
99.50
4.1
MAIN RESULTS
In this section, we compare the robustness of FDA and other baselines on T2IR/1TRR and VG tasks.
T2IR/ITTR. Results of T2IR/I2TR ALBEF, TCL, and BLIP for all methods given in Ta- [ Overall on Flickr, our FDA consistently exhibits the best robustness with the best clean performance all models over all other baselines for 14.29/13.55% and 51. $1.60/56.366 6.%% average ASR drop over all 3 attacks on ALBEF/CC//BLI with negligible 0.30/0.1094 0.50/0.225 22%, and 0.70/0.46% performance drops in R I for T2IR/I2TR on each model, respectively. On MSCOCO, similar patterns exist as our FDA boosts the ASR drop by 9/14% for / 4 with 0.1 1% clean performance boost on T2IR and a 0.07% drop on I2TR.
/// 224 yielding 22.26/14.699
224
i. Attack-wise, on Flickr, our FDA exhibits the best defense against PGD and APGD in 22 out of 24 results, leading TeCoA/FARE by 60/65% on the BLIP model, demonstrating the effectiveness of FDA in enhancing robustness against various attacks. For the strongest adaptive attack, MAPGD, our FDA maintains its lead over TeCoA and FARE on ABLEF and BLIP, with an average lead by over 10%. Although our FDA shows more vulnerability against APGD on the TCL model, it retains the best comprehensive robustness the other two baselines, yielding a 10-20% lead for 2/4. TeCoA/FARE become ineffective for attacks with = 4, while our FDA retains its effectiveness facing stronger attacks. Similar trends also exist on MSCOCO.
ii. Performance wise,, all baseline methods (i.c., TeCoA and FARE) suffered from performance drop by an average 4/3/9% on ALBEF/TCL/BLIP Nevertheless, our FDA only causes minor or little drops of less than 1% for all models, yielding lead of TeCoA and FARE by approximately 4%, 3%, and 7% on average, demonstrating the feasibility of paying less attention to function words for free robustness.
iii. Scalability-wise, we find that the effectiveness of FDA benefits significantly as the model scales: on ALBEF/TCL, which uses 14M pre-trained images, FDA enhances robustness of each model by roughly 15%; while on BLIP, which uses 124M pre-trained images, FDA achieves an impressive 54% overall increase in AASR We attribute the enhancement to the capability of the backbone model, which enables the encoders to better capture visual clues.
Grounding. Similar patterns persist for VG shown in Table 2 Our FDA achieves almost com- plete defense for all attacks, yielding an over 90% ASR drop while performing better on clean examples than the vanilla model. Specifically, FDA shows 13.11/91..544 ASR drop for 2/4. While TeCoA and FARE show comparative clean performance, they achieve 21.21/21 .63% and 12.08/13.2854 ASR drop, respectively, with an over 1% drop on clean examples. These results confirm the efficacy of FDA in enhancing robustness for similar/better clean performance.
4.2 UNTARGETED ATTACKS
We further evaluate the robustness against untargeted attacks. Thus, retrained all models using TeCoA/FARE and their combination with our FDA to validate the effectiveness of FDA in defending against untargeted attacks.

---

Published as a conference paper at ICLR 2026
Table 3:
Robustness evaluations ALBEF using FDA a plug-and-play tool with TeCoA and Retrieval. Results are averaged
FARE against untargeted attacks for marttoommma//mmmgee--eee
over T2IR and 12TR. Full results are provided in See of the Appendix. FDA consistently boosts
clean performance and/or robustness against all attacks.
Defense
Clean (R@1)
Average ASR 9/265 (1)
Average ASR 1/26 (L)
Arg. ASR drop
VLM
T21R
12TR
PGD
APGD
MAPGD
PGD
APGD
MAPGI
AaSa 1
No Defense
95.90
85.60
72.67
68.13
63.19
94.71
83.81
81.75
TeCoA
92.30
75.84
97.49
81.41
0.47
ALBEF
TeCoA FDA
92.50
81.86
75.52
63.22
60.44
97.57
80.84
82.01
1.31
FARE
91.20
80,76
69.87
48.18
44,00
96.43
75.79
75.79
13.09
FARE FDA
91.40
80,80
70.70
47.95
44.54
96.42
74.84
73.57
13.46
No Defense
97.20
87.30
78.17
77.08
67.65
99.80
94.01
89.82
TeCoA
81.50
68.00
48.01
41.23
95.37
75.41
72.39
28.23
BLIP
TeCoA FDA
80,40
67.78
43.80
38.20
35.63
94.26
72.20
69.67
32.18
FARE
89.70
77.72
47.51
53.62
51,45
90.37
78.71
77.01
22.36
FARE FDA
89.80
77.72
45.07
49.54
46.97
89,96
76.11
74.15
25.91
Table Robustness evaluations of FDA as a plug-and-pe and-play tool with TeCoA and FARE against un- targeted attacks for Visual Grounding, Results are averaged over Test-A and Test-B. Full results are provided in See of the Appendix. FDA consistently boosts clean performance and robustness against all attacks.
Clean (Ace)
Average ASR
8/20s (1)
Average ASR 4/255 (L)
Avg ASR drop
Defense
Val_d
Test A
Test B
PGD
APGD
MAPGD
PGD
APGD
MAPGD
Aase" 1
No Defense
58.50
65.90
46.30
27.46
20.06
19.82
32.33
23.48
23.31
TeCoA
57.20
64.70
45.00
9.67
12.68
12.80
9.64
16.22
16.43
1 39.68
TeCoA + FDA
57.00
64.90
45.30
10.37
12.85
12.01
9.84
15.22
15.67
1 40.30
FARE
56.40
64.20
44.70
10.56
14.45
14.49
10.79
16.70
16.97
34.69
FARB FDA
56.10
63.70
44.70
11.25
13.02
13.05
10.46
15.20
15.50
39.35
Table 5:
Ablation studies the encoders and dictionary of FDA. We use T2IR/I2TR for evaluation.
Clean (R@1)
Average ASR 4/250 ())
Average ASR f/25n
Avg ASR drop
Defense
T2IR
12TR
Avg
PGD
APGD
MAPGD
PGD
APGD
MAPGD
AARR
w/o FDA
95.90
85.60
90.75
2.04
14.83
62.37
7.96
15.99
79.03
T T & H
95.10
85.28
90.19
2.10
21.86
8.15
10.66
24.70
17.30
2.54
93.80
85.00
89.40
2.01
17.61
15.82
9.06
21.91
20.99
15.61
95.60
85,50
90.55
1.86
12.50
55.00
6.40
13.70
73.12
18.48
Full Dict Shorllittd Dict
95.10
84.46
89.78
2.03
13.54
56.60
6.46
14.47
74.65
4.22
95.40
85.40
90.40
1.71
13.60
36.78
6.92
14.15
75.07
6.45
For T2IR/I2TR, as presented in Table 3 Overall, FDA consistently boosts the robustness of TeCoA and FARE for all untargeted attacks on all models. Specifically, the scalability FDA also persists with TeCo//FARE: both methods benefit more from FDA on the larger backbone of BLIP, yielding a 4/3% robustness boost. Furthermore, we notice that FDA also boosts the clean performance of both methods ALBEF considerably. For VG, we observe identical patterns: implementing FDA yields solid robustness gain. For example, FARE experiences a significant robustness enhancement regarding untargeted attacks by 5%. In sum, FDA compatibility works with both TeCoA and FARE to further boost their robustness against untargeted attacks.
4.3 ABLATION STUDY
All results conducted targeted attacks, hyperparameters of FDA: encoder, dictionary, layer/head, variant FDA, and zero-shot performance. (See full results in Sec E of the Appendix.)
Hyperparameters. The implementation of FDA, especially the macro- hyperparameterss influencing where to implement, would largely impact the subsequent performance models. We first provide relative ablation studies to help understand the mechanics and design of our FDA.
7

---

Published as a conference paper at ICLR 2026
Table 6: Ablation studies on the layer/head index H.HH of FDA on fext-oo-mmage/mmag-too----- Text Retrieval on ALBEF, TCL and BLIP. Results are averaged over T2IR/2TR. Shallower layers/heads (smaller 1/1 consistently outperform over others on retrieval tasks.
Clean (R@1)
Average ASR 3/ms (
Average ASR 1/as (L)
||
Avg ASR drop
VLM
Defense
T2IR
12TR
Avg
PGD
APGD
MAPGD
PGD
APGD
MAPGD
AASR 1
w/o FDA
95.90
85.60
90.75
2.04
14.83
62.37
7.96
15.99
79.03
D Hool
95.50
85,54
90.52
2.06
14.82
60.98
7.82
16.15
80.70
t 1,56
ALBEF
Lok, H6-11
95.00
84.96
89.98
2.31
17.76
65.95
7.91
19.46
83.70
8.70
Dell HO
95.40
85,40
1.71
13.60
56.78
6.92
14.15
75.07
6.45
L -H-- 1,
95.60
85,50
90,55
1.86
12.50
55,00
6.40
13.70
73.12
18.48
95.40
85.32
90.36
1.81
12.30
54.87
6.17
13.45
72.71
16.91
FDA
94.90
7.20
68.07
79.74
79.03
||
TCL
D H
94.10
83,98
89.04
6.17
54.34
65.59
29.24
66.24
84,47
8.52
1. H0-5
94.40
83.82
89.11
6.06
46.44
62.99
27.31
57.57
82.02
13.92
94.20
83,96
89.08
6.42
48.64
64.82
28.22
61.30
83.44
11.14
FDA
97.20
87.30
18.46
61.67
47.27
64.09
86.23
71.44
||
BLIP
T.lli H
96.50
86.94
91.72
16.60
22.74
43.06
61.09
31,38
65.19
1 26.46
D HP
96.80
86.86
91.83
6.43
17:41
39.79
15.85
23.51
59.32
52.51
10-1 -0-3
96.70
86.84
91.77
6.58
16.36
37.99
15.15
22.34
58.07
53.98
Table 7: Ablation studies on the layer/head index L/H of FDA on VG on ALBEF, TCL, and BLIP. Results are given in average.
Clean (Acc)
|
Average ASR 2/2ss (1)
Average ASR 1/2ss (J)
Avg ASR drop
Defense
Val.d
Test A
Test B
Avg
PGD
APGD
MAPGD
PGD
APGD
MAPGD
AASa 1
w/o FDA
58.50
65.90
46.30
56.90
6.38
9.12
9.29
6.14
9.61
9.90
H
$7,90
65.80
46.40
56.70
1.02
2.06
2.27
0.77
2.38
2.22
88883
E
H
58.00
65,90
46.40
56.77
1.72
3.09
2.93
1.85
2.60
2.76
71.43
1--1
58.10
66.80
46.10
57.00
0.59
0.87
0.73
0.92
0.72
0.88
92.33
(1) Sicoders&&Dictionns
We start by comparing three implementations: FDA text encoders, fusion encoders, and both, denoted as T, H. and T&H. As presented in the top rows of T5 find that T performs the worst among all, indicating that an early subtraction is insufficient for removing such subtraction. Although T&H provides a significant robustness boost, it costs an evident 2% performance drop on performance, implying that subtraction on both encoders is too strong and potentially causes contextual distortion. H performs the best as it helps models concentrate while preserving the contextual meaning.
For the dictionary, we use the off-the-shele stopwords dictionary in (Li et al. 2020), containing 208 words/symbols, denoted as Full Dict. Furthermore, we use shortlisted dictionary, by only using the most commonly used function words, containing 91 crucial function words, denoted as Shortlisted Dict. Both dictionary settings are trained with FDA Iall to maximize their impacts on training. As presented in the lower row of Table there are significant performance gaps between the two settings, with Full Dict performing slightly worse regarding both clean and adversarial examples. We attribute the minor degradation to the length of the stopwords dictionary. which could unnecessarily skim words and distort the context. We provide the shortlisted dictionary in Sec of the Appendix.
(2) Astention Head &Layer: We then investigate the index of the layers L and attention heads H for retrieval and grounding, as presented in Table and Table Specifically, we train a series models using FDA but using different L and H: for layers, we use all, 0-1, and 0 layers, denoted as L.lli r I,; for attention heads, we use all heads, 1st half (0-5) and the second half (6-11), denoted as Halll HU-5 16-11 For T2IR/12TR, as shown in T we find that the shallow implementations of FDA, i.e., L9/L0-1 H consistently yield the best performance on robust- ness on all models. Specifically, I H constantly achieves the best clean performance, lead- ing other counterparts by 0.1-0.2% We further test the leading 3 settings on retrieval tasks, i.e., T /T// /T.0/TT0 -,, 0 5 on VG. As shown in 7 we find the shallow T,0-1 -0-5 settings still top w.r.t. both adversarial and clean examples, leading other settings by 10-20% respectively. Overall, FDA remains solid and insensitive to the head/layer parameters.
FDA v.s. Masking & Other DA. We start by providing comparisons of our FDA and fine-tuning models by directly masking function words, including content words and nouns for sanity checks.
8

---

Published as a conference paper at ICLR 2026
For variants of FDA, further compare our FDA with Adjective DA (ADA) and Deter- miner DA (DDA). Specifically, we choose de- terminers (DET) and adjectives because DET indicates using small subset (i.e., a/an/the) of function words, while ADJ adopts completely different set of words. Results are presented in 8 Note: We only test on PGD and APGD since MAPGD not applicable for nouns and content words.
Table 8: Comparison for directly removing
content words (CONT), nouns (NOUN), function words (FUNC). Determiner DA (DDA), and Ad- jective DA (ADA). AaSA is presented using the average results for PGD and APGD. shows significant advantages over directly masking and other de-attention variants.
Maksd
Clean (R @ 1) (t)
Avg ASR Drop
Words
T2IR
12TR
ASR (
N/A
95.90
85.60
Overall, masking content words and nouns yields the largest performance drop, making it inviable for robustness evaluation. This aligns with the intuition that these words carry ex- tensive semantic information crucial for VLM tasks. Furthermore, naively masking fune- tion words leads to evident performance drop (~1%) and brings negligible robustness. As a subset of FDA, DDA has identical clean performance but much less robustness. ADA also shows subpar performance compared with FDA. In sum, FDA leads the clean and adver- sarial performance among other variants.
CONT
21.50
11.10
NOUN
68.60
44.62
FUNC
94.00
80.86
t 1.56
DDA
95.60
85.42
1 9.28
ADA
95.50
85.38
15.10
FDA
95.60
85.50
23.07
Table 9:
Zero-shot performance by applying FDA as a plug-and-play tool on T2IR/12TR on ALBEF/BLIP and VG on ALBEF. T2IR/12TR uses R@ 1/5/10, while VG uses accuracies.
Zero-shot Performance. We also adopt three different head/layer settings of FDA for zero- shot T2IR/ITR/VG on ALBEF and BIIIP with H0-5 Results are presented in 9 We find that aall performs the best for all tasks and all models. This not only suggests that La serves as the most generalizable setting for multiple VL tasks and models, but also implies the feasibility of FDA for performance boost on zero-shot tasks.
Tasks
Models
Method
Avg Performance
ALBEF
w/o FDA
92.01
L
92.02
0.01
Retrieval
1
92.41
0.40
L
92.17
0.16
w/o FDA
92.24
BLIP
I
92.19
0.05
92.22
0.02
Iall
92.71
0.47
Takeaway: Implementing FDA on the fusion encoder would be ideal practice for all mod- els. While we do not rule out potentially bet- ter performance by using different L/H, one can adopt the setting in the paper on similar downstream tasks for stability, or simply use Lall, 00-5 as a safe choice.
ALBEF
w/o FDA
53.12
VG
T,0
52.72
0.40
1O-1
$2.68
0.44
Lall
53,34
0.22
Image-Taat Simllanty
1 0.113
4.4 ANALYSIS AND VISUALIZATION
We notice that the results of APGD and MAPGD somewhat worsen after adversarial fine-tuning, e.g., TeCoA and FARE on ALBEF, FARE on BLIP in Table etc. As previously illustrated in 2 defending against targeted attacks requires a more aligned vasion-language embedding. Consequently, hypothesize that such abnormality potentially originates from the disruption in vision-language alignment brought by adversarial noise for enhanced ro- bustness.
TeGrA
VIM FDA
Figure 3: Left: T-SNE of the lasiunnlanguage embedding of vanilla VLM, FDA, FARE, and TeCoA. Our FDA is the most aligned model. Right: Comparison of text-image similarity for vanilla VLM versus VLM FDA. Our FDA yields tightened alignment with larger similar- ities and smaller variances.
To validate our speculation, we visualize the vision-language distribution of ALBEF to-

---

Published as a conference paper at ICLR 2026
gether with TeCoA, FARE, and FDA, shown in the left graph Fig From the left graph, find that both FARE and TeCoA (left column) yield a severely disrupted embedding, where images and texts sparsely scatter away from each other. On the other hand, FDA (lower right) has the most aligned cross-modal embedding, as all images and texts remain tightly aligned with each other. To numerically compare the alignment of FDA and the vanilla model, record the top 200 aver- age white-box text-image similarity scores. As shown in the right figure of Fig applying FDA generates higher average text-image similarity scores and lower variations.
4.5 LIMITATION
Although direct subtraction introduces superior efficiency, FDA could be potentially improved through modular or algorithmic approach for more refined removal. Furthermore, we did not implement FDA to fine-tune a larger VLM or verify the effectiveness of FDA using PEFT methods such Hu al. (2022) due to the hardware limitation. Additionally, our FDA is designed and implemented on backbones featuring a fusion encoder, but not been applied to projector-base els such as LLaVA
mod- al., 2023) series and Qwen(Bai al. 2023 Team et al.! 2024 Bai al., 2025 Yang et al., 2025) series, etc. Nevertheless, FDA could be seamlessly adopted on these back- bones, which is a valuable exploration for future work. Finally, although have built shortlisted dictionary based on existing ones, it is possible that the optimal dictionary might have less/more words, which requires enormous effort for validation.
CONCLUSION
In this paper, propose Function-word De-Attention (FDA) calculates the original and the function-word cross-attention within attention heads, and differentially subtracts the latter from the former for more aligned and robust VLMs. Specifically, 3 datasets, and models, and evaluated all methods under 6 attacks. By comparing with existing SOTA defenses, our FDA shows superiority of FDA in boosting robustness and clean performance. We also provide an in-depth analysis of FDA and validate its boost on zero-shot performance. Our work validate the redundancy of function words in vision- languaee models and the subsequent vul- nerability brought by these words, calling for more research interest in further exploration.
tested the FDA 2 downstream tasks,
ACKNOWLEDGEMENT
This work was supported by National Key Research and Development Program China (2033YB3100400), the National Natural Science Foundation of China (T2341003, 62521002, U2441240, U24B20185, 62376210, 62132011, 62406240, U244120060).
ETHICS STATEMENT
We acknowledge that all authors of our papers are required to read the Code of Ethics, adhere to it, and explicitly acknowledge this during the submission process. Contribute society and human well-being. All authors: i) uphold high standards of scientific excellence; ii) avoid harm; iii) be honest, trustworthy, and transparent; iv) be fair and take action avoid discrimination; v) respect the work required to produce new ideas and artefacts; vi) respect privacy; vii) honour confidentiality.
REPRODUCIBILITY STATEMENT
The detailed information about the implementation of FDA is provided in Section The data used in this paper is open-source, and the details/full results are stated in the Appendix. The code and the checkpoint will be publicly available along with sufficient instructions to faithfully reproduce the main experimental results/visualization, and detailed instructions transfer to other backbones.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, al. Qwen technical report. arXiv preprint arXiv:2309. 16609, 2023.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint Skiv:2502.1392..3 2025.
Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In International conference machine learning, pp. 2206- 2216. PMLR, 2020.
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, 2019. URL https://azxiv.orr
abs/1810. 04805
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale, 2021. URL https:://// /awpiv.vvvv..////////a////aaavva wwt/aaa.aaa...eeeeo...e....e.aee 20100.001...99999
Kuofeng Gao, Yang Bai, Jiawang Bai, Yong Yang, and Shu-Tao Xia. Adversarial robustness for visual grounding of multimodal large language models. arXiv preprint urXiv:2405.0998/ 2024.
Bangyan He, Xiaojun Jia, Siyuan Liang. Tianrui Lou, Yang Liu, and Xiaochun Cao. Sa-attack: Im- proving adversarial transfcrability of Ausion-language pre-training models via self-augmentation, 2023. URL htRhttts: http//aaivvv... wrg/ 2albo//23122 01.312229999
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2)3, 2022.
Seil Kang, Jinyeong Kim, Junhyeok Kim, and Seong Jae Hwang. Your large vision- model only needs few attention heads for visual grounding. In Proceedings of the Computer Vision and Puttern Recognition Conference, pp. 9339-9350, 2025.
Marvin Klingner, Varun Ravi Kumar, Senthil Yogamani, Andreas Bär, and Tim Fingscheidt. De- tecting adversarial perturbations in multi-task perception. In 2022 IEEE/RSJ International Con- ference on Intelligent Robots and Systems (IROS), pp. 13050-13057, 2022. doi: 10.1
IROS47612..0222 202.099211159
Chun Tong Lei, Hon Ming Yam, Zhongliang Guo, Yifei Qian, and Chun Pong Lau. Instant ad- versarial purification with adversarial consistency distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 24331-24340, June 2025.
Jiaao Li, Kaiyuan Li, Chen Gao, Yong Li, and Xinlei Chen. Egoprune: Efficient token pruning for egomotion video reasoning in embodied agent. arXiv preprint arXiv: 15428, 2025a.
Junnan Li, Ramprasaath R. Selvaraju, Akhilesh Deepak Gotmare, Shafiq Joty, Caiming Xiong, and Steven Hoi. Align before fuse: Vision and language representation learning with momentum distillation. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), A.vaanccs in Neural Information Processing Systems, 2021. URL pettts::/oppereviiii wttp://.eeeeffforeoeeoee foormm? 1d=OJIaKwiXSSx
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre- training for unified vision- vision-language understanding and generation. In International conference on machine learning, pp. 12888-12900. PMLR, 2022.
Kaiyuan Li, Xiaoyue Chen, Chen Gao, Yong Li, and Xinlei Chen. Balanced token pruning: Ac- celerating vision language models beyond local optimization. arXiv preprint arXiv:2505.2200 2025b.
Linyang Li, Ruotian Ma, Qipeng Guo, Xiangyang Xue, and Xipeng Qiu. Bert-attack: Adversarial attack against bert using bert. arXiv preprini arXiv:2004.0998. 099844, 2020.
11

---

Published as a conference paper at ICLR 2026
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer viston, pp. 740-755. Springer, 2014.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892-349111 2023.
Dong Lu, Zhiqiang Wang. Teng Wang, Weili Guan, Hongchang Gao, and Feng Zheng. Set-level guidance attack: Boosting adversarial transferability of vision-language pre-training models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 102- 111, October 2023.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv: arXiv:1706.0608. 2017.
Chengzhi Mao, Scott Geng, Junfeng Yang, Xin Wang, and Carl Vondrick. Understanding zero-shot adversarial robustness for large-scale models. arXiv preprint arXiv:22/2.070111 2022.
Jan Hendrik Metzen, Tim Genewein, Volker Fischer, and Bastian Bischoff. On detecting adversarial perturbations, 2017. URL https::/ /aapss///axiv... aariv.iiivv org/ abs / 1002. 04267
Bryan A. Plummer, Liwei Wang, Chris M. Cervantes, Juan C. Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. In Proceedings of the IEEE International Conference Computer Vision (ICCV), December 2015.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp- 8748-8763. PmLR, 2021.
Leslie Rice, Eric Wong, and Zico Kolter. Overfitting in adversarially robust deep learning. In International conference on machine learning, pp. (093-8104, PMLR, 2020.
Christian Schlarmann, Naman Deep Singh, Francesco Croce, and Matthias Hein. Robust clip: Un- supervised adversarial fine-tuning of vision embeddings for robust large vision-language models. arXiv preprint ar arXiv:2402.12336 2024.
Ramprasaath Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based local- ization. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), Oct 2017.
Qwen Team et al. Qwen2 technical report. arXiv preprint arXiv:2407. 2(3), 2024. 2024.
Qiwei Tian, Chenhao Lin, Zhengyu Zhao, Qian Li, and Chao Shen. Collapse-aware triplet decou- pling for adversarially robust image retrieval. Proceedings of the 41st International Conference Machine Learning, ICML'24, JMLR.org, 2024.
Qiwei Tian, Chenhao Lin, Zhengyu Zhao, Shuai Liu, Qian Li, and Chao Shen. Adversarial video promotion against text-to- video retrieval. IEEE Transactions Information Forensics and Secu- rity, 21:1484414444 1496, 2026. doi: 10.1109/TIFS.2022 2026.365709..
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint 25X.v:2505.09388. 2025.
Jinyu Yang. Jiali Duan, Son Tran, Yi Xu, Sampath Chanda, Liqun Chen, Belinda Zeng, Trishul Chilimbi, and Junzhou Huang. Dision-languag pre-training with triple contrastive learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 15671- 15680, 2022.
12

---

Published as a conference paper at ICLR 2026
Tianzhu Ye, Li Dong, Yuqing Xia, Yutao Sun, Yi Zhu, Gao Huang, and Furu Wei. Differential transformer. arXiv preprint arXiv:2410.05258, 2024.
Ziyi Yin, Muchao Ye, Tianrong Zhang, Tianyu Du, Jinguo Zhu, Han Liu, Jinghui Chen, Ting Wang, and Fenglong Ma. Vlattack: Multimodal adversarial attacks
vision-tanguage tasks via pre-trained models. In A. Oh, T. Naumann, A. Glober- son, K. Saenko, M. Hardt, and S. Levine (eds.), Advances Neural Information Pro- cessing Systems, volume 36. pp. 52936-52955 Curran Associates, Inc., 2023. URL
htttps::/prroceedinge ngs. neur cc. https/cccppppeeeeeeeeeee / file/ a
Jongmin Yoon, Sung Ju Hwang, and Juho Lee. Adversarial purification with score-based gener- ative models. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 12062-12072. PMLR, 18-24 Jul 2021. URL httpp:://proceeddng.......
1139/yoon21ar html
Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara Berg. Modeling context in referring expressions. In European conference on computer vision, pp. 69-85. Springer, 2016.
Jirong Zha, Yuxuan Fan, Xiao Yang, Chen Gao, and Xinlei Chen. How enable Ilm with 3d capacity? survey of spatial reasoning in Ilm. arXiv preprint arXiv:2504.05786 2025.
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustness and accuracy. In International conference on machine learning, pp. 7472-7482. PMLR, 2019.
Jiaming Zhang, Qi Yi, and Jitao Sang. Towards adversarial attack on vision-language pre-training models. Proceedings of the 30th ACM International Conference on Multimedia, MM '22, pp. 5005-5013, New York, NY, USA, 2022. Association for Computing Machinery. ISBN 9781450392037. doi: 10.1145/3003611.55430000
URL https://dci. ort.:ii..rr111....1....
35031 61 .5547801
13

---

Published as a conference paper at ICLR 2026
APPENDIX FOR PAY LESS ATTENTION TO FUNCTION WORDS FOR FREE ROBUSTNESS OF VISION-LANGUAGE MODELS
A FDA V.S. ADAPTIVE WORDS SELECTION
We further qualitatively verify FDA with 3 adaptive selection methods (i.e., using per-token similar- ity of text and image features to choose tokens with low similarity): i) setting threshold of 6; ii) setting threshold of μ 26, with μ., being the mean and std of the text-image similarity. We further choose the lowest N tokens, with N being the number of function words in the texts. We denote them as SIM-5, SIM-26, and SIM-N, respectively. Furthermore, we record the % of selected words that are in our shortlisted funccion words dictionary. Results are shown in Table 10
In summary, we find that the gained robustness of VLMs increased as the proportion of function words increased. While cannot design an adaptive mechanism that perfectly aligns with using the function words dictionary, we find that while the vulnerability of VLMs does not necessarily come from low-similarity (or low semantic) words, there is an evident correlation between the percentage of function words and the gained robustness.
B DETAILS FOR ATTACKS AND EVALUATION METRICS
We first introduce the attacks and evaluation metries for each VL task, including the scenarios where a targeted attack considered successful and the corresponding metries.
Text-to- tot--mma///maae-ttte Text Retrieval. For T2IR, successful targeted attack is only when the manipulated images emerge in the Top 1/5 position given the targeted text queries; for 12TR, suc- cessful attack is only when the targeted texts emerge in the Top 1/5 position given the manipulated images as the query. Consequently, the ASR of T2IR/12TR would be the hit rate at the top 1/5, i.e., the probability of appearance in the top 1/5 position, denoted as ASR @ 1/5. In the main paper, use the average of ASR@ 1/5 as the overall ASR. Untargeted attacks follow the identical setting of existing works, i.e., lowering the R@ 1/5 of the victim models.
Visual Grounding. For visual grounding, we choose to obfuscate the model by fooling it into recognizing other objects the target, or, if there is only object in the image, locating the position the object incorrectly (top-left corner). A successful attack is when the IOU of the targeted bounding box and the model bounding box is larger than 0.5, i.e., the model locates the object within the targeted bounding box. As for untargeted attacks, we follow existing settings to lower the accuracy of the victim model and calculate the drops as ASR.
C FULL RESULTS FOR TARGETED ATTACKS
Table 10: Comparison between FDA and adap- tive selection, i.e., using image-text similarity to choose less informative tokens. SIM-5/-26 indi- cates using μ - 8 and μ 28 as the de-attention threshold, with μ4, ô being the mean and std of the text-image similarity. SIM-N refers to choos- ing the lowest N tokens, with N being the num- ber of function words in the text. % of words means the percentage of function words in the se- lected ones. Results confirm the correlation be- tween the proportion of function words and the gained robustness.
In this section, we provide full results for all tar- geted attacks on all models and tasks. Specifi- cally, for T2IR and 12TR, results on ALBEF is given in Table I1 and Table 12 results on TCL given in Table 13 and Table 14 and results on BLIP is given in Table and Table T6. respec-
tively. Targeted attacks for visual grounding on ALBEF are given in Table 17
D FULL RESULTS
FOR UNTARGETED ATTACKS
Defense & of Words
Clean R (1)
||
ASR Drup AASR (f)
Dictionary
T2IR
2/ss
4/2ss
Avg
In this section, we provide full results for all untargeted attacks on all models and tasks. Specifically, for T2IR and 12TR, results on AL- BEF is given in Table 18 and Table and
SIM-N
25.95
95.90
85.50
2.44
12.81
7.62
SIM-26
74.53
95.60
85.32
9.38
13.86
8.39
SIM-5
79.49
95.30
85.38
12.85
11.54
12.41
100.00
27.61
18.53
23,07
14

---

Published as a conference paper at ICLR 2026
Table 11: ASR of white-box targeted attacks against Text-to-Image Retrieval on Flickr30k and COCO. The model is ALBEF, Changes over unattacked values are presented in parentheses. All results are in percentage (%). ASR @ 1/5 indicates the attack success rate of the adversarial image showing up in the top-1/5 position of the targeted text queries.
Daaaase
1x
Defense
Clean
PGD
APGD
MAPGD
ASR
ASR05
ASR
ASRe5 J
ASRe1 1
A
No Defense
95,90
(10.20)
7.50 (16.50)
14.60 (114.50)
(:1.70))
50,10
(180,99)) (10000))
TeCoA
91.20
0:20 (10.20)
(14.90)
18:70 (818.70)
(+20000)
59.90 (139.90)
86.40 86,40 (186.00)
FARE
91.10
0.10 (40.0 10)
5.10 (+4.80)
17.20 (+17.20)
(-17.30)
58. 10 (+57.900
80,50 (+80.00)
FDA-L"
95.60
0.10 (+0.10)
7.30 (+6.60)
12.10 (0)
(-12070)
43.60 (+443.60)
73.90 (+73.20)
Fliekr
FDA-L
95.40
0.20 (+0.20)
6,90 (+6.10)
12.00 (+12.00)
12.80 (+12.00)
43.30 (+133.30)
73.60 (+72.8 80)
HDA-L.at
95.40
0.40 (40.40)
5.90 (45.20)
12,80 (+12,80)
14,20 (-13,500
43,50 (+43.550
77.30 (476. 60)
No Defense
(+4.20) 20)
(+13.100
16. (4-1640)
16.80 (-15.60)
75:00 (474,900)
(+86.00) (0)
TeCoA
91.20
3.90 (+3.90)
14.70 (+14.30)
19.40 (+19.40)
19.60 (-19.220)
81.00 (+8) .000
90.00 (+59. .0)
FARE
91.10
4.00 (14.00)
14,50 (1 14.50)
18, 18,80 (8.8800)
18.80 (1 18.30)
79.70 (1 79.70)
85.90 (185,40)
FDA-I"
2.90 (+2.90)
(+12.80)
13:90 (+13.90)
14:10 (-13-40)
69.00
00)
(+81.70)
HDA-1.9-1
95. 40
3.00 (+3.00)
12.40 (+11 (.0)
13,90 (+13.900)
14,00 (-13.221)
68. 10 (+68. 10)
81, 30 (+80.50)
FDA-I.
95.4
3.00 (13.00)
13.50 (x12.80)
14.60 (4.66))
14,80 (411.00)
68.90 (1 68.900
10 (183.40)
Defense
0:22 (10.18)
(.1.72
10:18 (1 (0:14)
(1:94(1111886
20: (120144)
(10.80)
3/000
TeCoA FARE
68.04
0.10 (+0.08)
0.66 (+0.56)
16.12 (+16, 40)
17.40 (-17,34)
24.52 (+24.50)
44.02 (+13 .2)
69.28
0.08 (+0.06)
0.55 (+0.45)
19.64 (+19.62)
25,82 (+25,72)
21.76 (+21.74)
43.74 (+43.66
COCO
FDA-L p-1
77.70
0.26 (40.22)
1.58 (+1.46)
9:00 (+ 8.988)
10.40 (-10.30)
18:28 (+1..266)
37.00 (436.90)
No Defense
77.60
2.10 (42.06)
7.44 (+7.36)
14:26 (+14,22)
14,80 (+14.72)
43.74 (443,700
58.72 (458.64)
5/853
TeCoA FARE
68.04
0.52 (+0.50)
2.74 (+2.64)
25,00 (+24.96)
26,46 (+26.36) 36)
$3.56 (+53.54)
66. (+66. 18)
69.28
0.40 (+0.38)
(+2.42)
30.94 (+30.92)
33.82 (+33.72)
54.46 (+54.44)
72.48 (+72.38)
FDA-L
77.70
1.74 (+1.70)
6,06 (+5.9H)
11.76 (+11.44)
12.08 (-11.98)
37.92 (+37.90)
51.98 (+$1.88)
Table 12: ASR of white-box targeted attacks against Image-to- Text Retrieval on Flickr30k and
COCO. The model is ALBEF, Changes over unattacked values are presented in parentheses. All results are in percentage (%). @ 1/5 indicates the attack success rate of the targeted text queries showing up in the top-1/5 position of the adversarial image.
PGD
APGD
MAPGD
Dataset
Defense
Clean
ASR J
ASKØ5 |
ASK
ASR 4
ASR 1
ASK 5|
No Defense
85.60
0.30 (40.30)
(+1,
(+14444)
(-15:40)
(453.555
63630 50)
TeCoA FARE
(+0.08)
1.00 (+1.00)
(+16.40)
(714..31
56.90 (+56.90)
65.70 70)
81.48
0.10 (+0.10)
1.00 (+1.00)
19.64 (a 19.62)
25.82 (+25.72)
56.30 (+56.33)
65.90 (+65.90) 90)
FDA-F"
85,50
(+0.10)
(-0.6))
12,30 (x1..90)
12.80 (+12. 80)
46.50 (146 (6)
56, 20 (156.20)
Flickr
FD
85.32
0.10 (+0.10)
0.80 (+0.80)
12.10 (+1..10)
12.50 (-12.50)
46.90 (+46 66)90)
55,90 (+55 90)
IDA-Lo!!
85:40
0.20 (+0.20)
(+1.00)
13,50 (+13.50)
13,70 (+13.70)
48.50(+18.500
58.00 (+58.00)
Defense
85.60
(14.50)
(49.0))
(+1.70)
(11199)))
(44,40((144
79:00 (s 79.00)
TeCoA
81.14
2,80 (+2.80) 80)
6.10 6-1.04+
(..8..011.300 30)
18.60 (-18. 60)
78.10 (+78. 10)
81810 10)
V/nn
FARE
81.48
3.40 (+3.40)
7.00
(4.300(e185 (8.33)
18.40 (-18.40)
77.00 (+77.00) (77000
80.60 (-80.66)
FDA-19
85,50
3.30 (+3,30)
6.50 (46.50)
(+13.77)))70)
13:70 (-13 70)
68, (+68 80)
72,40 (+72.440)) .40)
FDA-T.9-1
85,32
3.10 (+3.100)
6,90 (46.90)
13.40
(31.40
13.50 (-13.50)
69.30 (469. 30)
72. 30 (+72..90) (0)
FDA-L-9
85,40
(+4.00)
7,80 (+7.80)
14.10 14.10 (+1.010)
14.20 (-14.20)
72.20 (+72.20)
75.20 (+75.20)
No Defense
60.70
0.22 (40.22)
0.50 (-0.48)
7.68 (+7 68)
(-10.02))
14.64 (+14 (4)
24.16 (+21 14)
TeCoA
53.07
0.02 (+0.02)
0.12 (+0.12)
10.82 (+10.82)
13.58 (-13.54)
14.32 (+14.32)
(+33.74)
FARE
53.58
(+0.00)
0.06 (+0.04)
11.80 (1.88)
17.40 (-17.88)
21.62 (+1 (2.62)
20,92 (+20.90)
COCO
FDA-
60.63
0.16 (+0.10)
0.42 (+0.40)
(. 7.14)
12.34 (-12.52)
16.72 (+16.72)
26.66 (+26.64)
Defense
60.70
1.46 (+1.46)
3.38 (43.36)
11.11 (+11.10)
13.26 (+13.22)
50,10 (450,00)
81,90 (480.90)
TeCoA
53.07
0.16 (+0.16)
0.56 (+0.56)
18.58 (+18 58)
25.56 (-25, 56)
38.42 (+38.42)
$2.00 (+51, 96)
V225
FARE
53.58
0.22 (+0.22)
0.50 (+0.48)
21.84 (+21.84)
26.88 (-26.86)
31.94 (+31.94)
47,30 (+47.28)
FDA-L 1
60.63
(.2299))90) (+1.22)
(+ 9.88)
11.28 (+11.26)
27.74(+27.74)
37.94 (+37.99)
results on BLIP is given in Table 20 and Ta- respectively, Untargeted attacks for vi-
ble 21 21
sual grounding ALBEF are given in Ta- ble
E FULL RESULTS FOR ABLATION STUDIES
In this section, we provide full results for all ablation studies. T2IR and 12TR results are given in Table and Table 24 Zero-shot performance is given in Table 25
15

---

Published as a conference paper at ICLR 2026
Table 13: ASR of white-box targeted attacks against Text-to-Image Retrieval on Flickr30k. The model is TCL. Changes over unattacked values are presented in parentheses. All results are in percentage (%). ASR @ 1/5 indicates the attack success rate of the adversarial image showing up in the top-1/5 position of the targeted text queries.
PGD
APGD
MAPGD
Defense
Clean
ASR@1
ASRØ5 4
ASR@1 J
A
ASR@1 4
ASR
No Defense
94.90
(+2.10))
18.80 (+18.40)
66.00 (+66.00)
75:20 (+74.80)
50.90 (+500900
82.50 10)
TcCoA
92.10
3.10 (+3.10)
19.30 (+19.00)
61,00 (+61,00)
71.70 (+71.40)
44.20 (+44,200
74.10 (+73.80)
3/005
FARE
91.70
3.20 (43.20)
20,40 (+20.20)
63.10 (+63.10)
71.90 (+71.70)
46.80 (+46.80)
75.20 (475.00)
FDA-10
94.40
1.90 (+1 .0)
15.40 (+15.. 10)
44.90 (+-44, 90)
52.00 (+51. 770)
52.40 (+52.40)
84. 60 (+84.30 30)
FDA-L9-1
94.20
2.40 (+2.40)
17.00 (+16.60)
46.30 (+46.30)
55 80.80 (+55. 00)
54,40 (+54.40)
86.70 (+85,90)
FDA-f
94.10
2.30 (+2.30)
16 80 (+ 16.40)
50.80 (+500.80 80)
60.90 (-60.50)
54,90 (+55.90)
87. 10 (+86. 70)
No Defense
94,90
21.50 (+21.50)
54.00 (+53.66)
80,30 (+80.30)
82.00 (+81.60)
75.20 (+75,20)
88.10 (+87.70)
TeCoA
92.10
27.80 (+27.80)
60.90 (+60.60)
79.70 (+ (+79.70)
81 60 (+81. (3)
74. 10 (+74.10)
86. 10 (+85.80)
FARE
91.70
30.20 (+ 30.20)
62,30 (+62. 10)
80.30 (-80,30)
81 80 (+81 60)
73.20 (+-73.20)
86. 10 (+85. 90)
FDA-L9
94.40
17.90 (+17.990)
43.00 (+42.70)
56.80 (+56.80)
60.60 (+60.30)
80.20 (+80.20)
92.30 (+92.00)
FDA-L9-1
94.20
18.80 (+18.80)
44.90 (+44.50)
60:20 (-60.20)
64.50 (-63. 70)
80,40 (+80,40)
00.90 (+92.10)
FDA-I.aR 94.10
19.00 (+ 19.00)
44.90 (+44.50)
66. 10 (+66. 10)
69. 69.10 (+68.70)
79.80 (+79.800
94. $4.50 (+94. 10)
Table 14: ASR of white-box targeted attacks against Image-to-Text Retrieval on Flickr30k. The model is TCL. Changes over unattacked values are presented in parentheses. All results are in percentage (%). ASR @ 1/5 indicates the attack success rate of the targeted text queries showing up
in the top- 1/5 position of the adversarial image.
PGD
APGD
MAPGD
100
Defense
Clean
ASR@1
ASR@5 L
ASR@1
ASR@5
ASR@1 1
ASR
No Defense
84.02
2.50 (+2.50)
(45.70)
64.10 (+64.11
66.80 (+66.80) 8U)
50,70 (+50,70)
(+38.90)
TeCoA
80.40
(+2.
(+6.10))
69.60 (+69.60)
72.00 (+72.00)
(+65.50)
70.40 (+70.40)
3/505
FARE
78.22
(43.10))
(46.10)
59.70 (+59,70) 70)
62.80 (+62.80) 80)
65,20 (+65,20)
69. (469.50)
FDA-19
83.82
1.90 (+1.90)
5.30 (+5.30)
43.40 (+43.44)
45.60 (+45.60)
52,90 (+52.90)
62.10 (+62.10)
FDA-19-1
83.96
1.80 (+1.80)
4.80 (+4.80)
45,30 (-)
47.50 (4))
53,90 (+53.90)
64.40 (+64,40)
FDA-
83.98
1.30 (+1,30)
4.60 (+4.60)
51.50 (+51,50)
54,30 (+54. 30)
56.00 (+56.00)
64.40 (+64.40)
No Defense
84.02
24.60 (+24.60)
34.70 (434.70)
78.80 (+77.80)
78.60 (+78.60)
70.40 (+70.40)
75.50 (+75.50)
TeCoA
80.40
29.60 (+29.60) 6D)
34.70 (+34.70)
76.00 (+76.00) 00)
77.20 (+77. 20)
65.50 (+65..50
70.4 (+70.40)
4/566
FARE
78.22
33.30 (+33.30)
39.50 (+39.50)
76.20 (+76.20)
77.70 7777770)
65.20 (+65.20)
69.50 (+69.50)
FDA-L
83.82
20.00 (+20:00)
28.50 (+28.50)
56. (+56.10)
56.90 (+56.900 90)
75.50 (+75.50)
80. 10.10 (+80.10)
FDA-19
83.96
20. 10 (+20. 10)
29. (+29.30)
60.00 (+60.00)
60.80 (+60..8)
77.80 (+77.80)
82. 70 (+82.70)
FDA-Lalt
83.98
22.20 (+22.20)
31.10 (+31.10)
64.40 (+64,40)
65 50 (+65.50)
79.55 (+79.50)
84.10 (+84.10)
Table 15: ASR of white-box targeted attacks against Text-to-Image Retrieval on Flickr30k. The model is BLIP. Changes over unattacked values are presented in parentheses. All results are in percentage (%). ASR @ 1/5 indicates the attack success rate of the adversarial image showing up in the 1/5 position of the targeted text queries.
PGD
APGD
MAPGD
bm
Defense
Clean
ASR@1
ASR@5 1
ASR@1
ASR@5
ASR@1 1
ASR
No Defense
97.20
2.50 (+2.550
46.10 (446.10) 10)
80.50 (+81.10)
(-74.
(+577.99
(+84.30)
TeClA
81.50
4.30 (+4.30)
16.20 (+16.20)
19.30 (+19.30)
45.70 (+45 60)
14,80 (+14.80)
39.20 (+39.10)
3/55a
FARE
79.40
1.00 (+1.00)
10.30 (+10.10)
12,90 (+12.90)
46.20 (+46.20)
9.70 (+ 9.70)
38.00 (+37.90)
FDA-19
96.80
3.10 (+3.10)
12.00 (+11.90)
13.60 (+13.60)
26.30 (+26.20)
24.20 (+24.20)
61.70 (+61.60)
FDA-L2
96.50
3.00 (+3.00)
12.40 (+12.30) 30)
12.30 (+12.30)
25 25.60 (+25.70)
22,10 (+22.10)
59.90 (+59.80)
96.50
3.20 (+3,20)
42.00 (+41.80)
16.00 (+16.00)
39.60 (+39.40)
24.50 (+24.50)
66.30 (+66.30)
Defense
97.20
31.80 (+31.80)
90.60 (490.20)
79.80 (+79.80)
93.00 (492.60)
$7.90 (+57.990)
(+84.30)
TeCoA
81.50
46. 20 (+46.20)
73.00 (+72.90)
60.40 (+60.40 40)
83, (-83. 00)
49.50 (+-49.50)
74, 80 (+74 80) 74.70 (+74.70)
4/256
FARE
79.40
23.90 (+23.90)
61.50 (+61.30)
42.30 (+42.330
82.70 (+82.60)
33,90 (+33.900
FDA-L"
96.80
13. 13.60)
80 (+19.70)
16.40 (+16.40)
43.60 (=43,50)
44,70 (+444.70)
78.50 (+78.40)
FDA-1
96.50
13.20 (+13.20)
18.60 (+18.50 50)
15.10 (+15.10)
41.60 (+41.70)
43.40 (+-13.40)
77.90 (+77.80)
FDA-Lelt
96.50
31.60 (+31.60)
86.80 (+86.60)
21.40 (+21.40)
62.00 (+61.80) 80)
50.00 (+50.00)
81.90 (+81.70)
16

---

Published as a conference paper at ICLR 2026
Table 16: ASR of white-box targeted attacks against Image-to-Text Retrieval on Flickr30k. The model is BLIP. Changes over unattacked values are presented in parentheses. All results are in percentage (%). ASR @ 1/5 indicates the attack success rate of the targeted text queries showing up in the top-1/5 1/5 position of the adversarial image.
PGD
APGD
MAPGD
Defense
Clean
ASR@1 4
ASR
ASR@1
ASRG5 1
ASR@1
ASR |
Defense
(+7.000
(-1666))
54,90 (+54.90)
65650
37.60 (+37.60)
(+500.9))
TeCoA
68.00
2.40 (+2,40)
5.90 5.90)
13.90 (+13.90)
25. (+ 10)
9.30 (+ 9.30)
19:80 (+19.80)
TeCoA FDA-E
67.78
2.20 (42.20)
6:20 6.20)
12,90 (4-12.90)
25.40 (425.40)
8.90
19.60 (+)19.00
3/206
FARE
65.64
0.40 (+0.40)
(+ 1.90)
11.30 (+11.30)
19.60 (+19.60)
8,00 (+ 8.00)
15.10 (+15. 10)
FARE FDA-L
66.22
0.40 (+0.40)
2+ 2.10)
9.40 9.40)
17. (+17:50)
6.00 6.00)
13,60 (+13.60)
IDA-L0
(+4.40)
5.30 5.30)
14.10 (+14. 10)
15. (515
31.70 (+31.70)
60 (+41.60)
FDA-L5-1
$6.86
4.60 (+1.60)
4,50 (+ 4.80)
13.10 (+13.10)
14.40 (+ (+14.40)
30.60 (+30.60)
39.40 (+39.40)
FDA-LSR
86.94
(+3.20)
(+14.60)
16.90 (+16.90)
(+ 18.60)
(+35.00)
(+-46.50)
No Defense
$4.02
58.80 (458.80) 80)
(4.90-774.900 (+74.90)
83,70 (+83.70) 700
88. 10 (088.10) 10)
67.00 (-07
90 (+75.900
TeCoA
68.00
46. 60 (+46.60)
58. (-58.10)
58.90 (+588.90 90)
69.00 (+69.00)
47.10 (+47.10) 10)
60.10 (+-0.. 10)
TeCoA + FDA-1
67.78
44.50 (+44.50)
(9.10 (+59.10)
59.10 (+59.10)
70.30 (+70.30)
47.40 (+11.44)
61.10 (+61.10)
FARE
65.64
23.50 (+233.90)
37. (+ 30)
45, (+455.77 70)
60:90 (+00990)
34.00 (+.4..00)
50:90 (4-50.90)
FARE FDA-
70 (+24.70)
37.10 (+37.10)
43,90 (+43.900)
58.60 (+58.60)
(+31.90)
(+-48.20)
IDA-L FDA-
$6.86
14,50 (+14.80)
15.30 (-15..3)) 30)
16.80 (+ (6)))80)
(+17.30)
53.40 (+:53.40)
60.70 (+60. 70)
14.20 (+14.20)
(-14.70)
16.20 (+16.20)
16.40 (+16,40)
(+52.00)
(+59.00)
FDA-L.R
86.94
$5,20 (+52.220
70.80 (+70.00)
21,80 (+21.80)
22. 10 (+ (+22.10 10)
60,20 (+60.20)
68. (+68. 70)
Table 17: Attack success rate (ASR) of targeted PGD/APGD/MAPGD (masked APGD) against for Visual Grounding (VG) on RefCOCO+ All results are presented in percentage (%). Changes over unattacked values are presented in parentheses.
Clean Performance
Test A Split (L)
Test B Split (.)
In
Defense
Val d
Test A
Test
PGD
APGD
MAPGD
PGD
APGD
MAPGD
No Defense
46.30
20.40 (-6.00)
(+10.00)
(+1.50)
26.53
(+6.37)
3/855
TeCoA
57.20
64.70
45.00
17.87 (-6.00)
18.67 (+ 6.80)
18.93 (+ 7.06)
24.00 (+1.67)
26.27 (+4.94)
26.13 (+4.80)
FARE
56.40
64.20
44.70
18.40 (:5.33)
(1 9.06)
22.00 (1. 8.93)
24.27 (13.60)
(15.3)
25.87 (+5.20)
FDA-L"
65.90
12.53 (-1.73)
14.40
3.60)
20)
(+1.73)
21.07 (+1.47)
FDA- FDA-10-1
58.10
66.80
46.10
12.67 (-1.20)
13.60 (+ 2.13)
13.06 (+ 1.59)
20.26 (.0.14)
19.87 (-0.53)
20.13 (40.27)
Fa
57,90
65.50
46.40
12.27 (-2,11)
(2 2.80)
13.30 3.17)
20.55 (-0.27)
21.60 (+0.80)
21.60 (+0.80)
No Defemse
65:90
16.30
17.17 (-7.07)
20.40 (+10.00
20.40 (+10.00)
24.10 (43.47)
26.53 (+5.60)
(+6.37)
TeCoA
57.20
64.70
45.00
18.00 18.00
19.07 (+ 7.20)
19.33 (+ 7.46)
24.13 (+2.80)
26.13 (++.80)
26. (+1.80)
FARE
56.40
64.20
44.70
18.93 (-5.86)
21.47 (1 8.40)
22.00 (+ 8.93)
24.27 (13.60)
26.27 (+5.60)
25.87 (+5.20)
FDA-L"
13.07 (-2.27)
14.40
(40.93)
20.53 (+0.93)
20,80
FDA FDA-D0-1
58.10
66.30
46.10
12.80 (-1.33)
13.33 (+ 1.86)
13.33 (+ 1.86)
20.67 (+0.27)
19.87 (-0.53)
20.13 (40.27)
FDA
57,90
65. 55.80
46.40
12.27 (-2.14)
13,20 3.07)
13:47 2,67)
20.13 (-0.67)
21,87 (+1.07)
21.73 (+0.93)
F DETAILS FOR FUNCTION WORD DICTIONARY
We provide the function word dictionary we used as follows: "am, is, are, was, were, be, been, being, have, has, had, do, does, did, will, would, shall, should, may, might, must, can, could, ought, dare, need, used, to, a, an, the, and, but, if. or, because, as, until, while, of. at, by, for, with, about, against, between, into, through, during, before, after, above, below, from, in, out, on, off over, under, again, further, then, once, here, there, when, where, why, how, all, any, both, each, few, more, most, other, some, such, no, nor, not, only, own, same, so, than, too, very".
G VISUALIZATION OF ATTENTION SCORES
Finally, we provide an illustration of original attention, FDA with one subtraction, and FDA, as shown in Fig 4
17

---

Published as a conference paper at ICLR 2026
Table 18: ASR of white-box untargeted attacks against Text-to-Image Retrieval Flickr30k.
The model is ALBEF. After-attack R @@k values are presented in parentheses. All results are in percentage (%). ASR @ 1/5 indicates the drop of R@ 1/5 after attacks.
100
Defense
Clean
PGD
APGD
MAPGD
ASR@1 L
AS L
ASR@1 &
ASR@5 L
ASR@1 4
ASR@5 1
No Defense
95.90
78.54 (21.46)
57.39 (42.61)
55.89 (25.30))
(29.077)
(51.8))
TeCoA
91.20
(18.78) 78)
60.20 (39.80)
76.02 (23.98)
(41.87)))
67.95 (32.05)
46.01 (53.99)
2/200
TeCoA FDA-I.-1
91.60
80.73 (19.27)
(41.28)
68.87 (31.13)
(49.80)
(32.25)
(55.88)
FARE
91.10
74.39 (25.61)
51.52 (48.48)
54.35 (45.65)
70.95 (29.05)
49.29 (50.71)
23.79 (76.21)
FARE + FDA-L
90.60
76.73 (23.27)
53.46 (46.54)
52.95 (47.05)
69.31 (30.69)
49.19 (50.81)
26.32 (73.68)
No Defense
95.90
96.15 (3,85)
92.00 (8.00)
88,11 (11.89)
76.73 23)
87.80 (12.20)
72.97 (27.03)
TeCoA
91.20
98.98 (1.02)
95.33 (4.67)
85.44 (14.56)
73.91 09)
87.36 (12.64)
71.49 (28.51)
4/200
TcCoA FDA-19-1
91.60
98.68 (1.32)
94.73 (5.27)
85.09 (14.91)
72.41 (27.59)
87.02 (12.98)
69:47 (30.53)
FARE
91.10
98.08 (1.92)
93.93 (6.07)
80.57 (19:43)
63.56 (36.44)
80,57 (19:43)
63.56 (36.44)
FARE FDA-L
90.60
98.17 (1.83)
93.90 (6.10)
79.67 (20.33)
62.60 (37.40)
(19.82)
(41.67)
Table 19: ASR of white-box untargeted attacks against Image-to-Text Retrieval on Flickr30k.
The model is ALBEF. After-attack R @@k values are presented in parentheses. All results are in
percentage (%). ASR @ 1/5 indicates the drop of R@ 1/5 after attacks.
Defense
Clean
PGD
APGD
MAPGD
ASR@1 L
ASR L
ASR@1 &
ASR
ASR@1 1
ASR@5 1
Defense
85.60
(15.
69.91 (30.09)
77.34 (22.66)
64.58
(23.98)
57.65 (42.35)
TeCoA
81.44
87.68 (12.32)
74.26 (25.74)
74.70 (25.30)
61.62 (38.38)
74.49 (25.51)
$8.16 58.16 (41.84)
2/200
TeCoA FDA-J0-1
81.80
88.59 (11.4))
74.02 (25.98)
73.91 (26.09)
59.89 (40.11)
73.26 (26.74)
56.63 (43.37)
FARE
81.48
84.09 (15.91)
69.48 (30.52)
64.18 (35.82)
74.06 (25.94)
61.80 (38.20)
41.13 (58.87)
FARE + FDA-L
80.14
83.61 (16.39)
68.98 (31.02)
63.70 (36.30)
73.27 (26.73)
61.39 (38.61)
61.25 (58.75)
No Defense
85.60
97.08 (2.92)
93.61 (6. (6.39)
89.11 (10.89)
81 (18.
88.34 (1
77.89 (22.11)
TeCoA
81.44
99.13 (0.87)
96.51 (3.49)
86.92 (13.08)
79.35 (20.65)
90.81 ( 9.19)
81.08 92)
4/25
TeCoA FDA-
81.80
99.72 (0.76)
97.61 (2.39)
(13.04)
78.91 (21.09)
(14.18)
(19.67)
FARE
81.48
98.27 (1.73)
95.45 (4.55)
84,96 (15.04)
74.06 (4.
84,96 (15.04)
74.06 (25.94)
FARE + FDA-LO-1
80.14
98.35 (1.65)
95,27 (4.73)
83.83 (16.17)
73.27 (26.73)
84.93 (15.07)
70.85 (29.15)
Table 20: ASR of white-box untargeted attacks against Text-to- Retrieval on Flickr30k. The model is BLIP. After-attack R @@ k values are presented in parentheses. All results are in percentage (%). ASR @ 1/5 indicates the drop of @ 1/5 after attacks.
PGD
APGD
MAPGD
Defense
Clean
ASR@I
ASR@5 J
ASR@ 4
ASR@5
ASR L
ASR@5 J
Delense
97.20
82.45(17.55)
61.79 (38.21)
80:94 (19.66
(32.66
72.22 (27.78)
(47.04)
TeCoA
81.50
55.33 (44.70)
32.79 (67.21)
(52.77)
29.22 (70.78)
44.16 (55.84)
26.40
TeCoA FDN-L
80.40
51.30 (48.70)
(71.26)))
(55.75)
28.09 (71.91)
(59.44)
(75.92)
FARE
79.40
54.19 (45.81)
28.29 (71.71)
60.72 (+39.22 23)
36. (63.87)
58.65 (41.35)
34.93 (65.07)
FARE + FDA-L0-1
79.30
51.41 (48.59)
29.18 (70.82)
56.62 (+43.38)
33.08 (66.92)
54.01 (45.99)
30.59 (69.41)
No Defense
97.20
99.90 (0.10)
99.60 0.40)
95.69 (4.31)
90.57 (9.43)
93,18 (6.82)
83.75 (16.25)
ToCoA
81.50
96.97 (3.03)
93.94 6.06)
80.95 (19.05)
68.40 (31. 66))
80.98 (19:02)
66.38 (33.62)
Vana
TeCoA FDA-
80.40
96.64 (3.36)
92.73 7.27)
79.07 (20.93)
65.08 (31 (1.92)
77.99 (22.01)
63.81 (36.19)
FARE
79.40
94.23 (5.77)
84.87 (15.13)
85.09 (14.91)
66, (33.73)
83.46 (16.54)
63.76(36.45) (36.42)
FARE+ FDA-L0-1
79.30
(5.86)
82.86 (17.14)
82.43 (17.57)
63.12 (36.88)
80.04 (19.96)
60.95 (39.05)
18

---

Published as a conference paper at ICLR 2026
Table 21: ASR of white-box untargeted attacks against Image-to-Text Retrieval on Flickr30k. The model is BLIP. After-attack R @@ values are presented in parentheses. All results are in percentage (%). ASR @ 1/5 indicates the drop of R @ 1/5 after attacks.
Defense
Clean
PGD
APGD
MAPGD
@ 4
ASR L
ASR@1 &
ASR L
ASR@1 4
AS 1
No Defense
87.30
89.68 (10.32)
(21.26)
85:45 (14.55)
74.51 (25.49)
(19.81)
63.22 (34.78)
ToCoA
68.00
62.58 (37.42)
41.35 (58.65)
53.87 (46.13)
34.11 89)
50.67 (49.33)
31.41 (68 (9)
9/000
TeCoA + FDA-L
67.78
58.08 (41.92)
37.06 (62.94)
50.37 (49.63)
30.10 (69.90)
49,50 (50.50)
28.36 (71.64)
FARE
65.64
63.35 (36.65)
44.21 (55.79)
68.64 36)
48. (51.01)
66,25 (33.75)
45.97 (54.03)
FARE FDA-L
66.22
59.73 (40:27)
39.97 (60.03)
64,06 (35,94)
44.39 (55.61)
61.54 (38.46)
41.74 (58.26)
No Defense
84.02
99.90 (0)
99.79 (0.21)
96.70 ( 3.30)
93.09 (6.91)
94,01 (5.99)
88.34 (11.66)
TeCoA
68.00
97. 42 (2.58)
93.13 (6.87)
82.94 (17.06)
69.33 (30.67)
80.98 (19.02)
66 38 (33.62)
4/00
ToCoA + FDA-L0-1
67.78
96.64 (3,36)
91.04 (8.96)
80.22 (19.78)
64,43 (35.57)
77.99 (22.01)
63.81 19).
FARE
65.64
94,21 (5.79)
88.16 (11.8))
87.41 (12.59)
76.07 (23.93)
86.52 (13.48)
74.31 (25 69)
FARE + FDA.I.--
66.22
94.45 (5.55)
88.40 (11.6)
86.00 (14.00)
72.89 127.11)
84,24 (15.76)
71.37 (28.63)
Table 22: Attack success rate (ASR) of untargeted PGD/APGD/MAPGD (masked APGD) against for Visual Grounding (VG) on RefCOCO+. After-attack accuracies are presented in parentheses. All results are in percentage (%). ASR indicates an accuracy drop after attacks.
Defense
Clean Performance
Test Split ())
Test Split ())
Val.dl
Test A
Test
||
PGD
APGD
MAPGD
PGD
APGD
MAPGD
No Defense
$8.50
65.90
46.30
(7.40 50)
15:90 (50.00)
(50.00))
13:20 (33.10)
(38.90)
(38.70)
TeCoA
$7.20
64.70
45.00
8.20 (56 50)
9.50 (54.90)
9.80 (54.90)
3.00 (42.00)
4.50 (40.40)
4.70 (40.30)
TeCoA+ + FDA-I
$7.10
64.90
45.30
8.30 (56.60)
9.80 (55.10)
9.70 (55.20)
3.60 (41.70)
4.80 (40.50)
5.00 (40.30)
FARE
56.40
64,20
44.70
9.40 (54.80)
11.80 (52.40)
(52,20)
2.90 (41.80)
4.70 (40.00)
4,50 (40.10)
FARE+ FDA-
56.10
63.70
(54.50)
10.90 (53.10)
10.80 (53.20)
3:40 (11.00)
(40.40)
4.10 (10.30)
No Defense
58.50
65.90
16.30
(14.0
18.70 (47,20)
18.20 (17.070
14.90 (31/40)
8.50 (37.70)
8.80 (37.50)
TeCoA
$7.20
61.70
15:00
8,30 (56.40)
12.50 (52.20)
12.20 (52.50)
2.90 (42.10)
5.90 (39.10)
6.30 (38. 70)
TeCoA FDA-
$7.10
64.90
45.30
7.90 (57 .00
11. (53.60)
11.60 (53.30)
3.40 (41.90)
5:90 (39.40)
6.10 (39.20)
56.40
64.20
(54.80)
11 (52.40)
13:60 (50.60)
3.40 (41.60)
(39.10)
5.70
FARE+ + F
56.10
63.70
44.70
9.50 (54.50)
10.90 (53.10)
(2.200
2.70 (41.70)
5.10 (39.30)
5.30 (39.10)
Table 23: ASR of ablation studies against Text-to-Image Retrieval on Flickr30k. The model is ALBEF. Changes
unattacked values are presented in parentheses. All results are in percentage
(%). ASR @ 1/5 indicates the attack success rate of the adversarial image showing up in the top- position of the targeted text queries.
Defense
Clean t
PGD
APGD
MAPGD
ASR@1 4
ASR 1
ASR J
ASR@5 1
ASR 1
ASR
No Defense
95.90
0.30 (+0.20)
7.50 (46.50)
(+14..5))
50.10 (+14077)
(480909)
FDA-T
95.10
0.40 (+0.40)
6.40 (+6.20)
20.70 (-20.70)
23.70 50)
4.80 (+ 4,80)
28.30 10)
FDA T & H
93,80
0.50 (+0.50)
6.90 (+6.60)
16.80 (+20.99)
19.30 (+19.30)
13.60 (+13.60)
21.80 (+21.80)
FDA- N
95.60
0.10 (+0.10)
7.30 (+6.60)
12.10 (-12.10)
13.40
43.60 (+43.60)
73.90 (+73.20)
Lall Hoil
95.50
0.10 (+0. (+0.10) 10)
7.30 (46.80)
14,40 (+14.40)
15.90 (+15.40)
46.50 (+46.50)
84,90 (+84.40)
3/660
poil 10-11
95.00
0.20 (+0.20)
8.00 (+7.70)
16.70 (- 16. 70)
19.50 (+ 19.20
48. (+48.90)
85.60 (+85. 30)
Lou Hu-s
95.40
0.40 (+0.40)
5.90 (+5.20)
12.80 (+12.80)
14,20 (+13 50)
43,50 (+43,50) 505)
77.30 (+76.60)
1 HO-5
95.60
0.10 (+0.10)
7.30 (+6.60)
12.10 (+12.10)
13.40 (+12.70)
43.60 (+43.60)
73,90 (+73.20)
1. 5
95.40
0.20 (+0.20)
6.90 (+6.10)
12.00 (+12.00)
12.80 (+12.000
43,30 (+43.30)
73.60 (+72, 8D)
Pull Diet
95.40
0.40 (+0.40)
5.90 (+5. 20)
12,80 (+12.80)
14,20 (+13.55)
43. 60 (+43.60)
77.50 (+77.00)
Shortlisted Dict
90.10
0.30 (+0.30)
6.60 (+).10)
12,80 (+ (=12.80)
14,20 (+13.55)
43,50 (+43.50)
77.30 (+76.60)
No Defense
95.90
4,30 (4+4220)
14,10 (+13,10)
16.50 (+16.40)
16.60 (405.0+15660 60)
75.00 (474.90)
87, (+86.00)
FDA T
95.10
5. 10 (+0.200
20.00 (+19. 80)
24. 80 (+24.80)
25. 50 (+25 30)
12. 10 (+1 10)
40.00 (+39. 80)
FDA T & H
93.80
4,70 (+4.70)
16.70 70 (+16.40)
20.90 (+20.90)
22.20 (+21.99)
19.90 (+19.90)
23.50 (+23.20)
FDA- N
95.60
2,90 (+2.90)
13.50 (+12.80)
13.90 (+13.990
14,10 (+13.440
69.00 (+69.00)
82.40 (+81,70)
Lill Hull
95.50
3.90 (+3.90)
14.70 (+14.70)
16.30 (+16.30)
16.70 (+16.20)
76.70 (+76.70)
92.10 (2.100(+91.60)
1ed He-11
95.00
3.30 (+3.30)
15,70 (+15.40)
19.60 (+19.60)
20.10 (+19.80)
77.70 (+77.70)
91,70 (+91.40) 144
Loil Hu-s
95.40
3.00 (+3.00)
13,50 (+12.80) 80)
14.60 (+14.60)
14,80 (+14.10)
68. (+68.90) 90)
84.10 (+83.40)
1P, HO-S
95.60
2.90 (+2.90)
[3.50 (+12.
13,90 (+13.90)
14,10 (+.3.44+ 13.40)
69.00 (469.00)
82.40 (+81.70)
10-1 -0-5
95.40
3.00 (+3.00)
12,40 (+11.60)
13.90 (+13.90)
14,00 (+13 13.
68. 10 (+68. 10)
81.30 (+80.50)
Pall Diet
95.40
3.00 (+3.00)
13.50 (+12 80)
14,90 (+14.90)
15.20 (+14.70)
69. (469.
83, 33 (+83.40)
Shortlisted Diet
95.10
3.70 (+3.70)
11.60 (+11. 10)
12.80 (+ 12.80)
14,20 13.50)
43,50 (+43.50)
77.30 (+76.60)
19

---

Published as a conference paper at ICLR 2026
Table 24: ASR of ablation studies against Image-to- Text Retrieval on Flickr30k. The model is ALBEF. Changes over unattacked values are presented parentheses. All results are percentage (%). ASR@ @ 1/5 indicates the attack success rate of the adversarial image showing in the top- 1/5 position of the targeted text queries.
Defense
Clean
PGD
APGD
MAPGD
ASR
ASR
ASR85
ASRØ5
Defens
85.60
(+0.300)
(-1.(1)
14:40
(5140 40)
(-.30
63.50 (-63.50)
FDA
85.28
(+0.300
(+1.50)
20.90 (+20.90)
22.30 30)
5.90 5,90)
(-140440
FDA T& H
93.80
0.10 (+0.10)
1.10 (+1.10)
16.80 (+20.90)
17.80 (+17 80)
13.00 (+13.00)
1-,10
(+0.10))
(+12.30)
(+56.20)
Fell Hou
85.51
(+0.300
1.00 (+1.00)
14:40 [4.40 (+14.40)
15.00 (+15.00)
(+.1.7055)70)
60. 60,90 (-60.90
7/335
Lan, 170-11
84.96
0.30 (+0.30)
1.10 (=1.10)
17.30 (x.330)
17.80 (+17.80)
55.60 (+48.90)
72.30 (+72.30)
Lan H0-5
85.40
0.20 (+0.20)
1.00 (+1.00)
13.50 (3.555)
13.70 (+13.70)
48.50 (-18.50)
58.00 (-58.00)
1. H0-5
50
0.10 (+0.100)
0.60 (+0.60)
12.30 500
12.80 (+122 80)
46,50 (+46.50)
56, (+56,
10-1 10
85.32
(+12.10)
(+12.50)
(+1609))
55.90 (+55.90)
1,99 H - Pull Diat
84
0.40 (+0-400
5,90 (+5.20)
13.50 (+13.700
1::70 (+13.70)
48.00 (+48.00)
57. (-57,
1. 179-3- Shortlisted
40
0.20
(=100
13.50 (x
13:70 (+13.770
(148.50)
58.00 (+58.00)
4.50
FDA T
95.10
6.40 (+6.40)
11.30 (=11.30)
24.50 (4)
24.90 (+24.90)
15.50 15.50)
19.10 (=19.10)
FDA T &
93.80
(45.04511)))
10.00 (=10.00)
20 (4.20 80)
21 (+21 10)
19,30 (+ 19,80)
21 (-21.00)
H
85.
3,30
6.50)
13.70 (3.70)
13:70 (+
68.80 (168.80)
72.40 (:72.40)
gell Maa
85.54
5.20 (+5.20)
7.90 7.90 (+7.90)
15.90 (+ 90)
16.10 (+.6.11 10)
78.90 (+78.00
82.50 (-82.50)
84.96
4,10 (+4.10)
8.80 (+8.80)
19.10 (+19.10)
19.30 (+)))3)
82.00 (+82.00)
85,40 (-85.40)
Eali IP1-5
(:2.20))
(175.20)
20 110-
$5.50
(+3.30))
6.50 (+6.50)
13:70 (+13.70)
13:70 (+ 70)
68.50 (+68.80)
72.40 (+72.40)
10-1 190m
85.32
(..10)))
(+13.50)
(169.30))
72.30 (172.90)
Tell H0-2 Full Diet
84.46
3.60 (+3.60)
7.10 (+7.10)
14.10 (+14.10)
14.10 (+14.10)
70.80 (-70.80)
74:10 (+74.10)
1, 8d Shortlisted Diet
85.40
(.0.004)00
7,80 (+7.80)
(4110
14.20 (+14.20)
72.20.477.200
75.20 (475.20)
Table 25: Zero-shot performance by applying FDA plug-and-pla tool T2IR, I2TR
ALBEF/BLIP and VG on ALBEF
Tasks
Models
Method
Zero-shot Performance())
Average
w/o FDA
88.50
98.50
99.20
75.88
93.34
88.50
92.01
ALBEF
Loll
89.10
98.60
99.40
75.56
93.70
96.66
92.17
t 0.16
I
89,00
98.50
99.30
75.38
93.20
96.72
92.02
0.01
T2IR/1TRR
L0-1
89.60
98.80
99.40
76.16
93.70
96.80
92.41
0.40
w/o FDA
87.20
98.00
99.10
78.20
94.08
96.88
92.24
BLIP
T
88.70
98.40
99.30
78.80
94.20
96.88
92.71
0.47
L
87.00
98.00
99,10
78.14
94.10
96.82
92.19
0.05
L LO-1
87.10
98.00
99.10
78.12
94.16
96.82
92.22
0.02
w/o FDA
54.50
61.77
43.10
$3.12
VG
ALBEF
I Lall
$4.73
62.17
43.11
53.34
0.22
L
54.10
61.71
42.34
52.72
4
0.40
L
54.14
61.47
42.42
52.68
0.44
20

---

Published as a conference paper at ICLR 2026
Original Attention
Two Subtractions
1003
0.12
434
0064
Figure 4: A heatmap of attention probabilities given the same image and text inputs. Left: Original attention probabilities are relatively 'noisy' and have several visible stripes with very low probabil- ities, implying the existence of some less relevant visual tokens that activated, with negligible contributions. Mid: Attention probabilities with one FDA subtraction show much less aforemen- tioned 'stripes', with much cleaner and more focused attentions. However, exist and remain visible. Right: Attention probabilities with two subtractions show the cleanest attention maps and have the most negligible distractions, with only strong activations the most relevant visual tokens, i.c., with higher probabilities.
distractions still
21


Reviews:
[{"review_id": "iAHCX1XdcL", "reviewer": "Reviewer_h995", "summary": "This paper proposes Function-word De-Attention (FDA), a training-free defense that subtracts attention from function words in VLMs (e.g., ALBEF, BLIP).\nIt shows robustness gains under image-space attacks without hurting clean performance. However, the idea is limited to cross-attention models, and lacks ablation evidence to support its claims.", "questions": "Please refer to the Weaknesses section.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The function-word “de-attention mechanism” is straightforward and easy to integrate into VLMs.\n2. Zero training cost and minimal inference overhead.\n3. Works stably across three representative fusion-based VLMs (ALBEF, BLIP, TCL).", "weaknesses": "1. Limited scope\n    * FDA fundamentally requires cross-attention between text and image; thus, it cannot be applied to CLIP-style encoders or LLM-based multimodal models (LLaVA, InternVL).\n    * Therefore, I think the paper’s title (“robustness of VLMs”) overstates generality.\n2. Kind of trivial core observation\n    * Figure 1’s finding — that attacks disrupt function-word attention but not content-word grounding — is unsurprising and follows directly from semantic anchoring.\n3. Missing ablation\n    * The simplest verification — masking function words entirely and measuring retrieval performance — is absent.\n    * Without it, it’s unclear which is more important; “attention subtraction” or just “ignoring function words.”\n4. Over-claiming “training-free”\n    * Some sections mention optional fine-tuning for λ-scaling and per-layer adaptation, which contradicts the strict “zero-training” claim.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8IFucUMs1c", "reviewer": "Reviewer_nHus", "summary": "This paper hypothesizes that function words (e.g., am, is, are)—which are semantically vague and non-specific—make Vision-Language Models (VLMs) vulnerable to cross-modal adversarial attacks.\nTo address this, the authors propose Function-word De-Attention (FDA), which introduces a parallel attention path that computes the cross-attention between function words and images, and then subtracts this “distraction” from the original attention through a differential subtraction mechanism.\n\nOn retrieval tasks, FDA reduces the Attack Success Rate (ASR) by 18/13/53% on ALBEF/TCL/BLIP, with only 0.2–0.6% clean performance degradation.\nOn visual grounding, it achieves a 90% ASR reduction with a 0.3% accuracy improvement.\nThe robustness gain scales with model size (e.g., +54% ΔASR for BLIP) and even improves zero-shot performance by +0.4% without fine-tuning.", "questions": "see weaknesses", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Conceptually novel to focus on function words as a source of VLM vulnerability.\n- Simple and efficient method. No additional parameters or adversarial training required.\n- Consistent and strong improvement across multiple models, datasets, and tasks.", "weaknesses": "- The paper does not perform a true white-box attack against FDA.\nWhile baselines are evaluated under full white-box settings (with gradient access), FDA’s differential subtraction module is hidden from the attacker.\nThe proposed Masked APGD (MAPGD) does not actually backpropagate through the FDA operation, so the comparison is not fair.\n\n- The “observation” section is weak.\nThe reported 80.3% statistic lacks detail—what dataset, model, or attack setting?\nShowing only one qualitative example in Fig. 1 is not convincing as empirical evidence.\n\n- The paper does not include any ablation or control experiment on other word classes (e.g., content words, nouns, verbs, adjectives).\n\n- Table 1 presentation is confusing: R@1 (↑, higher is better) and ASR (↓, lower is better) are shown in the same rows without clear separation.\n\n- The comparison is too limited: only TeCoA/FARE trained with ε = 1/255 are considered. Robustness should be compared against baselines trained under the same threat levels (ε = 2/255 or 4/255), as standard adversarial training does.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nNtGkI5Nrs", "reviewer": "Reviewer_N7H5", "summary": "The paper introduces Function-word De-Attention (FDA) to improve the robustness and performance of VLMs by reducing attention to function words. FDA works by subtracting the cross-attention between function words and images from the original attention. Experimental results demonstrate that FDA reduces ASR with minimal or even improved performance.", "questions": "See Weaknesses.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The motivation and introduction of this paper is clear while use visualization to help readers quickly understand the task.\n\n2. The experiments are thorough, and the performance is verified on three models, two tasks, and three datasets.", "weaknesses": "1. The authors' proposed operation, \"proper removal of function words could potentially defend VLMs against such attacks,\" appears to have some limitations on VLM's use cases: the paper does not demonstrate its ability to defend against the fundamental task of vision-language models, such as classification. Furthermore, defense methods like FARE [1], which were originally designed to enhance the robustness of the LVLM's vision encoder, seems also inapplicable to this scenario.\n\n2. In contribution summarization, the proposed “pioneer the theory” may be replaced with observation, motivation or other words. The paper does not use theory to prove that the difference in similarity after paying less attention can increase or decrease, or that the difference is bounded.\n\n3. In Figure 2, how the conclusion “models can learn a more aligned cross-modal embedding” get? The observational evidence provided in the introduction doesn't suggest that this phenomenon is achieved by learning a more aligned cross-modal embedding. On the contrary, if the function word is removed, intuitively, the effect of alignment should be weakened.\n\n4. Considering the trade-off between robustness and performance, it is recommended to compare with the TRADES [2] method.\n\n[1] Schlarmann C, Singh N D, Croce F, et al. Robust clip: Unsupervised adversarial fine-tuning of vision embeddings for robust large vision-language models[J]. arXiv preprint arXiv:2402.12336, 2024.\n\n[2] Zhang H, Yu Y, Jiao J, et al. Theoretically principled trade-off between robustness and accuracy[C]//International conference on machine learning. 2019: 7472-7482.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Hc5C7l24OX", "reviewer": "Reviewer_QdtA", "summary": "This paper introduces Function-word De-Attention (FDA), a lightweight and novel method to improve the robustness of vision-language models (VLMs) without compromising their clean performance. The authors observe that function words (e.g., “the,” “is,” “and”) can make VLMs more vulnerable to cross-modal adversarial attacks because these words are frequent but semantically uninformative. FDA operates by computing and subtracting the cross-attention between function words and image tokens from the original attention maps. While the underlying mechanism behind FDA’s effectiveness is not yet fully understood, the authors provide extensive experiments demonstrating consistent improvements across multiple models, datasets, and attack settings.", "questions": "I have made my suggestions along with weaknesses. Here are several open questions for the authors:\n\n1. Have the authors examined whether the vulnerability arises specifically from function words, or more generally from irrelevant or low-information words in the inputs?\n\n2. Could there be an adaptive mechanism to identify which words to down-weight or exclude, rather than relying on a fixed dictionary?\n\n3. Are there cases where function words contribute meaningfully to visual grounding or alignment, and if so, how does FDA handle those instances?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. Identifying function words as a source of vulnerability in VLMs is an original and well-motivated idea. Good observation.\n\n2. FDA is conceptually straightforward and computationally efficient. It requires only a differential subtraction step within existing attention computations. It does not require additional parameters or retraining complexity.\n\n3. The authors evaluate FDA across multiple models, datasets, and both targeted and untargeted attacks. The improvement over adversarial training baselines (TeCoA, FARE) is consistent and substantial.", "weaknesses": "1. It's an interesting observation that function words can lead as a source of vulnerability. However, the evidence provided are far from satisfying. Namely, a single sentence (“80.3% of images show higher similarity scores toward function words than content words after attacks”) and one visualization of distracted attention. It is insufficient to establish that this is a systematic rather than a coincidental effect. Additional qualitative and quantitative analyses would strengthen the claim. I also recommend including a baseline where function words are simply removed from the inputs, to better isolate and validate the proposed phenomenon.\n\n2. Lack of clarity regarding experimental setup. FDA introduces a hyperparameter $\\lambda$ to control the strength of “de-attention,” where $\\lambda = 0$ reduces the method to standard attention. However, I find it confusing that the authors did not specify the $\\lambda$ values used in the reported results. In particular, for the clean (non-attacked) setting, it is unclear whether the authors used a very small or even zero $\\lambda$, which would make the claim of minimal performance drop less convincing. Moreover, the paper does not analyze how varying $\\lambda$ affects performance and robustness, which is essential for understanding the method’s sensitivity and general applicability.\n\n3. Several presentation issues. 1) Figure 1 seems to have noticeably lower resolution than other figures, even for texts in the figure. 2) Typo in Equation (4), $S_{T_f}$ instead of $S_{t_f}$. 3) Mistake in line 197-198, \" Specifically, **TCL** shares the same backbones as **TCL** ...\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)