Sparse Black-Box Multimodal Attack for Vision-Language Adversary Generation
Authors: Zhen Yu
Zhou Qin
Zhenhua Chen
Meihui Lian
fu Jun Hao
Weigao Wen
Hui Xue'
Kun He
Venue: EMNLP 2023 Findings (2023)
Keywords: Vision-and-Language
adversarial learning
multimodal attack
black-box attack
TL;DR: This paper proposes a novel black-box multimodal attack termed Sparse Multimodal Attack that leverages sparse perturbations in the black-box scenario to simulate the adversarial behavior exhibited by illegal merchants.

Abstract: Deep neural networks have been widely applied in  real-world scenarios, such as product restrictions on e-commerce and hate speech monitoring on social media, to ensure secure governance of various platforms. However, illegal merchants often deceive the detection models by adding large-scale perturbations to prohibited products, so as to earn illegal profits. Current adversarial attacks using imperceptible perturbations encounter challenges in simulating such adversarial behavior and evaluating the vulnerabilities of detection models to such perturbations. To address this issue, we propose a novel black-box multimodal attack, termed Sparse Multimodal Attack (SparseMA), which leverages sparse perturbations to simulate the adversarial behavior exhibited by illegal merchants in the black-box scenario. Moreover, SparseMA bridges the gap between images and texts by treating the separated image patches and text words uniformly in the discrete space. Extensive experiments demonstrate that SparseMA can identify the vulnerability of the model to different modalities, outperforming existing multimodal attacks and unimodal attacks. SparseMA,  which is the first proposed method for black-box multimodal attacks to our knowledge, would be used as an effective tool for evaluating the robustness of multimodal models to different modalities.

Sparse Black-Box Multimodal Attack for Vision-Language Adversary Generation
Zhen Yu!; Zhou Qin?; Zhenhua Chen'; Meihui Lian?, Haojun Fu?, Weigao Wen?, Hui Xue?,
Kun Hel
School of Computer Science and Technology, Huazhong University of Science and Technology, Wuhan, China Alibaba Group baiding15@hun 5@hust. cnu brooklet6@chust.. edu. cn
Abstract
Pharmacy
Pharmacy
Deep neural networks have been widely ap- plied in real-world scenarios, such as product restrictions on e-commerce and hate speech monitoring on social media, to ensure se- cure governance of various platforms. How- ever, illegal merchants often deceive the de- tection models by adding large-scale pertur- bations to prohibited products, so as to earn illegal profits. Current adversarial attacks us- ing imperceptible perturbations encounter chal- lenges simulating such adversarial behav- ior and evaluating the vulnerabilities of de- tection models to such perturbations. To ad- dress this issue, we propose novel black-box multimodal attack, termed Sparse Multimodal Attack (SparseMA), which leverages sparse perturbations to simulate the adversarial be- havior exhibited by illegal merchants in the black-box scenario. Moreover, SparseMA bridges the gap between images and texts by treating the separated image patches and text words uniformly in the discrete space. Exten- sive experiments demonstrate that SparseMA can identify the vulnerability of the model different modalities, outperforming exist- ing multimodal attacks and unimodal attacks. SparseMA, which is the first proposed method knowl-
TABLETS Ae
"/BLETS
Glless marchmns)
Smear key information and perturt the text
Titl: Drigs sale
Tilee Medicnas on sale
Recogrized illegal drig the detection model
nocmal the derection
Damage!!!
An example of a prohibited drug, where some key information is smeared by the illegal merchant, which is identified as compliant by the detection model.
Figure I:
However, there are always some illegal merchants who attempt to sell prohibited goods to make ille- gal profits, such as illegal drugs or smuggled goods, which violate the rules and terms of platforms. Due to the lack of professional knowledge in adversar- ial attacks (Szegedy et al., 2014; Papernot et al.. 201 6b), they often adopt large-scale and sparse per- turbations, including smearing and mosaicing, deceive detection models deployed by various com- panics, thereby posing a significant threat to the sccurity of platforms, as shown in Figure 1.
for black-box multimodal attacks edge, would be used as an effective tool for evaluating the robustness of multimodal mod- els different modalities. Code is available at
Current adversarial attacks (Goodfellow et al., 2015; Papernot et al., 2016b) typically mislead the victim model by adding imperceptible perturba- tions to benign samples, which can not accurately evaluate the vulnerability of detection models to the adversarial behavior exhibited by illegal merchants. In addition, prior research has mainly focused on unimodal adversarial attacks in Computer Vision (CV) (Goodfellow et al., 2015; Madry et al., 2018) and Natural Language Processing (NLP) (Papernot et al., 2016b; Liang et al., 2018), with little atten- tion paid to the vulnerability of multimodal mod- els (Yu al., 2020; Yang et al., 2021) that are more challenging but widely used in the real world. This motivates us to develop a multimodal adversarial attack that simulates physical adversarial behaviors
1 Introduction
With the rapid development of Deep Neural Net- works (DNNs), vasion-language multimodal clas- sification has been applied in various real-world applications, such as product restrictions on e- commerce (Zahavy et al., 2018) and hate speech monitoring on social media (Kiela et al., 2020), to ensure secure governance of various platforms.
"The first three authors contributed equally. "Corresponding author.

---

in the black-box scenario, which can help ensure secure governance of various platforms.
ity in the model's decision-making process. This information is valuable for researchers to better understand the multimodal model's behavior and enhance its robustness against adversarial attacks.
Existing multimodal attacks (Evtimov et al., 2020; Zhang et al.. 2022) typically process images and texts independently due to the vastly different data attributes of the two modalities, such as con- tinuous images vs. discrete texts. When employing different attack strategies to perturb images and texts, these methods unable to effectively com- bine features from different input modalities, lead- ing to relatively low attack performance. And these attacks run in white-box attack settings, which are almost impossible to apply to the real-world.
Related Work
This section provides a brief overview of unimodal adversarial attacks on images or texts, as well as multimodal adversarial attacks.
2.1 Unimodal Adversarial Attack
Image adversarial attacks Szegedy et al. (2014) first show the existence of adversarial examples. Subsequently, numerous works based on the loo or l2 norm, including FGSM (Goodfellow et al., 2015), PGD (Madry al., 2018), MIM (Dong al., 2018). NIM (Lin et al., 2020), etc., are proposed to enhance the attack performance. JSMA (Pa- pernot et al., 2016a) is the first to generate sparse perturbations by minimizing the lo norm. Croce and Hein (2019) introduce the to norm variant of PGD, known as PGDo, and a black-box sparse at- tack of CornerSearch that evaluates the saliency of each pixel based on the changes on logits. LocSearchAdv (Narodytska and Kasiviswanathan, 2017), ADMM (Zhao et al., 2019), and Sparse- RS (Croce et al., 2022) adopt local search, Bayes optimization, and random search algorithms,, re- spectively, to search for the optimal sparse adver- sarial example in the black-box setting.
To address the above issue and simulate the sparse perturbations commonly used by illegal mer- chants, we propose to map them into the same discrete space to bridge the gap between continu- ous images and discrete texts. And then we perturb both images and texts simultaneously using the same sparse attack strategy. Although these sparse perturbations are indeed visible, they generally do not alter the semantics.
In this work, we propose a novel black-box mul- timodal attack, named Sparse Multimodal Attack (SparseMA). SparseMA splits the input image into multiple patches, then evaluates the impact of cach patch in the image and each word in the input text on the victim model based on the output logits. After sorting all patches and words according to their impacts, we sequentially replace the original data with suitable candidates until an adversarial example is found. Note that SparseMA only needs access to the model output, making it more feasi- ble in the real-world than white-box attacks, which require full access to the model, including architec- ture, parameter, gradient, output, etc.
Despite the prosperity of sparse box at- tacks, they generally search for the optimal solution or evaluate the importance for each pixel, which re- quire more than 10 queries on images of size (224, 224). Even if ADMM and Sparse-RS optimize the query efficiency, tens of thousands of queries are still required, making it challenging to scale to large-scale datasets, such ImageNet (Deng et al., 2009). In this work, SparseMA splits the image into multiple patches, and does not need to pro- cess each pixel. It allows to achieve good attack performance with only a few hundred queries. Text adversarial attacks Existing text adver- sarial attacks typically modify the character, word, or sentence of the benign text to maximize the loss on the ground-truie class, among which word- level attacks (Li et al., 2020; Garg and Ramakr- ishnan, 2020; Li al., 2021a: Maheshwary et al., 2021; Yu et al., 2022) show excellent performance. PWWS (Ren et al., 2019) and TextFooler (Jin et al., 2020) greedily substitute important words with syn-
To validate the effectiveness of the proposed SparseMA, we do comparison with one image attack, two text attacks, and two multimodal at- tacks on the task of vision- laiiune multimodal classification. Extensive experiments demonstrate that SparseMA could identify the vulnerability of the model to different modalities, achieving a higher attack success rate than almost all the base- lines. Moreover, SparseMA could generate better sparse adversarial examples with fewer perturba- tions, making them more similar benign sam- ples and more applicable to real-world scenarios. Through analyzing the impact of perturbations on the model's output in each modality, SparseMA can reveal the relative importance of each modal-

---

Sample is nen-aleesorial
the optimal contidate soquentally unnil we find ativerary
Set ssch panch alllllll
and by Viiv Fop.
Recovery
Inirial advoneey
Candidale set
cich [UNK]
Split input
Replacement Order Determination
Candidate Substitution
Ontpet adversary
The overall framework of the proposed SparseMA algorithm. SparseMA first splits the image into multiple sparse patches and the text into multiple words. Then it evaluates their importance based on the model output by setting each patch as all Os or each word as [UNK], and determines the replacement order. According to the replacement order, SparseMA substitutes each element with suitable candidate until we find an adversary. Finally, it randomly recovers the redundant substitution back to the original data, thereby reducing the perturbation size.
Figure 2:
onyms based the output logits. GA (Alzantot et al., 2018) and PSO (Zang al., 2020) adopt evo- lutionary algorithms to search for near-optimal text adversarial example.
3 Methodology
In this section, we first formalize the problem of multimodal adversarial attack and then provide a detailed description of our method.
2.2 Multimodal Adversarial Attack
3.1 Multimodal Adversarial Attack
To the best of our knowledge, there are few works focusing on the vulnerability of multimodal mod- els. Yu et al. (2020), Tian and Xu (2021) and Yang et al. (2021) find that samples could be misclassi- fied by perturbing only single modality using PGD, revealing the vulnerability of multimodal models. Evtimov et al. (2020) perturb images with PGD and texts with HotFlip (Ebrahimi et al., 2018) gener- ate hateful posts that could fool multimodal models. MUROAN (Vishwamitra et al., 2021) utilizes the fusion features to decouple the input modalities and generate adversarial examples by removing salient data points. Co-Attack (Zhang et al., 2022) first per- turbs the text and then perturbs the image according the perturbed fusion features conduct collab- orative attack on the pre-trained multimodal model fool all downstream tasks.
Given an input space X' containing all input images and texts, and an output space y = {VI...... yk. we have a pre-trained multimodal classifier f X + V, which maps the input image æz and text Py w1 W2 Wn, to its ground-truth label y E V. The adversary adds an imperceptible perturbation on the classified sample x (æ1; ; Ø7 to craft an adversarial example (x+ xr--) that misleads the clas- sifier f to output a wrong prediction:
argmaax f(ulzj, x7-) + v.
MEY
Meanwhile, we should also guarantee the imper- ceptibility of adversarial perturbations. Therefore, we propose evaluating the quality generated ad- versarial examples using multimodal similarity. To achieve this, we input both images and texts into pre-trained BLIP-2 (Li et al., 2023) and extract the vector of the last hidden layer as the multimodal feature vector of the sample. Then, we calculate the cosine similarity between these vectors the multimodal similarity. This approach provides more comprehensive evaluation of the generated adversarial examples, taking into account both vi- sual and textual aspects of the data.
In this work, by splitting the input image and text into multiple sparse components, SparseMA estab- lishes more fine-grained connection between each element rather than between the entire image and text, allowing us to generate more well-designed multimodal perturbations efficiently. To the best our knowledge, SparseMA is the first proposed method for black-box multimodal attacks.

---

In this work, we propose novel sparse black-box multimodal attack approach, named SparseMA, in which only the output score of the target model is needed.
Algorithm 1: The SparseMA Algorithm Input: Input sample that contains the image
Ii and the text øT with lable y; Target classifier f: Number iterations N for random recovery
3.2 The Proposed SparseMA Method
Output: Adversarial example
Given a benign sample that contains image Ty and text 2y, SparseMA first splits image xι into mul- tiple patches and text 27 into multiple words. As illustrated in Figure 2, SparseMA consists of two main stages: i.e., replacement order determination and candidate substitution. Replacement order determination determines the importance and re- placement order of each patch and word. Can- didate substitution sequentially substitutes each patch or word with suitable candidate and then optimizes the redundant substitutions by random recovery to generate optimal ad adversary.
/* Replacement Order Determination
*/
Split the image into patches for each patch in x and each word in Fy do
Compute the importance score via Eq.
Sort all patches and words based on the importance score in descending order
/* Candidate Substitution
Construct the candidate set for each patch and word for each element in replacement order do Replace on the sample (xI, with its optimal candidate via Eq. 2 to craft new sample (T)
3.2.1 Replacement Order Determination
order to map the image and text into dis- crete space, we split the image 2j into m patches (p1.P2..... Pm. of size (s, 8) and the text ay into n words [wi, ....... wr Intuitively, more im- portant components with greater impact on the vic- tim model should be replaced. To evaluate the importance of each patch pi word wy. set the patch p; be all Os, or the word wy to be unknown, i.e., [UNK], to obtain modified sam- ple f (,,,,r)) or (x),,r)) We feed it into the target model, and calculate the importance score of each patch p; by examining the change of the model output:
8
if argmax,ycc> [ (T) # then break
9
10
if argmax f(ul|=I, = y then
11
L return None
// Attack fails
12 for to N do
13
Replace a randomly selected perturbed element on (#1, with the original data to craft a new sample (ă1, (T)) if Targmax,, f( # then 14
15
Update (I1,2T) +- (f1, 2T)
16 return (21,TT);
// Attack succeeds
I(ps) f(yla1,0)) fyylt,,,))) 21)
For each image patch, select all-black, all-white, and crossed black-and- aenkk-nnd-whitee patches as its candi- date set to simulate the perturbation behavior that illegal merchants are most likely to adopt in reality. For each word, we select its top nearest synonyms in the counter- fitted embedding space (Mrkšić et al.. 2016) 6) its candidate set.
Similarly, we could obtain the importance score Iww;) of each word wy. I()) measures the impact of the unit perturbation on the victim model by taking the perturbation size d(-,-) into account. allows us balance the choice perturbation be- tween image patches and text words to achieve maximum impact on the model with minimal dis- ruption. In the end, we sort all patches and words in descending order based on the importance score the replacement order.
For the patch pi word wj be replaced on sample x' (a*, x4)) at the t-th iteration along the obtained replacement order, we greedily substitute with the optimal candidate that has the greatest impact AP on the model from its candidate sets:
3.2.2 Candidate Substitution
To find the suitable candidate for substitution, pre-define candidate sets for each patch and word to ensure that the replacement operation has little impact on the semantics and similarity of samples.
AP f(yl x(r) flyla7t1 1+1 .x7 x(t1), (2)
where the sample r++ = (a/ 2T
(1
(+1) is obtained
by replacing the patch pæ or word wy with a candi-

---

Dataset
#Class
Train
Test
Avg. words
CLIPvr
CLIPRes Res
ALBEF
MVSA-Single
3
3,511
1,000
14
64.6
63.6
64.6
MVSA-Multi
3
17,886
1,000
14
76.6
73.4
72.0
CrisisMMD
7
17,126
1,000
118
95.8
95.3
89.5
Table 1: The statistics on the datasets and the classification accuracy (%) of victim models on the test set.
date on sample z'. Then, we sequentially substitute each patch or word with its optimal candidate ac- cording to the replacement order until we find an adversarial example successfully.
Baselines Since there are only a few multi- modal attacks proposed recently, we select repre- sentative works from unimodal and multimodal black-box attacks as our baselines. For image ad- versarial attacks, we choose the state-of-the-art sparse black-bo bbx attack Sparse-RS (Croce et al., 2022). For text adversarial attacks, we choose the word-level black-box attacks PWWS (Ren et al., 2019) and PSO (Zang al., 2020). For multimodal attacks, combine Sparse-RS and PWWS to at- tack images and texts respectively to perform the multimodal attack, named Sp-RS&PWWS.
Since some perturbed elements may have little influence on the initial adversarial examples, i.e., there are redundant perturbations, greedy substi- tution would result in suboptimal adversarial ex- amples. To further reduce the perturbation while keeping adversarial, we randomly change the per- turbed patches or words back to the original data. We continue this recovery operation on the result- ing sample if it is still adversarial to find an opti- mal adversarial example. The overall algorithm of SparseMA is summarized in Algorithm 1.
Experimental Settings The side length of image patches set 20, and the number of iter- ations N for random recovery is set to four times the number of perturbed elements the initial ad- versarial example. To ensure high semantic sim- ilarity between adversarial texts and benign texts for SparseMA and all text baselines, we also set the number of synonyms to 4. All evaluations are conducted on 1,000 randomly sampled texts from the corresponding test set. For fair comparison with our SparseMA, we increase the maximum number of pixels perturbed to 3,600 and decrease the number of iterations accordingly to 2,000 for Sparse-RS.
4 Experiments
In this section, we conduct extensive experiments to validate the effectiveness of SparseMA on three datasets and three models that are widely used.
4.1 Experimental Setup
Datasets We adopt three widely investigated datasets for vision-language multimodal classi- fication, including WVVSA-Single (Niu al., 2016), MVSA-Multi (Niu et al., 2016) and Cri- sisMMD (Alam et al., 2018). MVSA-Single and MVSA-Multi are two sentiment classification datasets that contain three sentiments (positive, neg- ative and neutral). CrisisMMD is a multimodal dataset consisting of tweets and associated images collected during seven disaster events.
4.2 Evaluation on Attack Effectiveness
To validate the effectiveness SparseMA, we con- duct experiments for vision- languae classification on three victim models using three datasets. The results, including attack success rate, multimodal similarity and the number of queries, are summa- rized in Table 2.
Victim Models We consider ALBEF (Li et al., 1b) and CLIP (Radford et al., 2021) as the vic- tim models in this work. As CLIP is an aligned model, we construct 2-layer multilayer percep- tron for predicting the target label. Addition- ally, consider two different image encoders for CLIP: VIT-B/16 (Dosovitskiy et al., 2021) and ResNet-50 (He et al., 2016), denoted as CLIPvTT and CLIPRes: respectively. More details of these datasets and the classification accuracy of each model are shown in Table
We could observe that SparseM achieves the highest attack success rate on of 9 cases com- pared to all unimodal and multimodal baselines and is only slightly weaker than Sp-RS&PWWS in the other two cases. And we always generate ad- versarial examples that are more similar the orig- inal samples, compared to Sp-RS&PWSS indicat- ing that we generate better adversarial examples. Meanwhile, the number of queries of SparseMA

---

MVSA-Multi
MVSA-Single
CrisisMMD
Model
Attack
Succ.
Sim.
Query
Succ.
Sim.
Query
Succ.
Sim.
Query
Sparse-RS
70.7
95.3
914
80.8
95.2
779
12.9
95.0
1,840
PWWS
20.9
96.7
56
24.7
95.6
60
10.0
99.1
64
CLIPvIT
PSO
27.6
97.1
3,427
29.4
95.4
3,568
13.3
98.9
4,443
Sp-RS&PWWS
71.8
94.5
969
81.2
94.0
838
17.7
94.4
1,903
SparseMA
82.5
96.6
388
80.0
96.9
366
27.1
97.1
557
Sparse-RS
54.2
95.3
1,184
67.2
95.2
937
8.0
95.4
1,894
PWWS
17.0
96.6
57
26.2
96.9
60
9.8
99.1
64
CLIPRes
PSO
20.8
96.5
3,702
30.7
96.0
3,527
13.3
99.1
4,446
Sp-RS&PWWS
56.4
96.4
1,240
69.6
94.2
996
13.4
94.3
1,957
SparseMA
64.5
96.7
410
81.1
96.7
336
24.2
97.6
563
Sparse-RS
22.1
96.7
1,666
50.4
97.2
1,258
30.4
97.2
1,583
PWWS
22.3
97.7
56
39.2
95.9
66
34.4
99.1
61
ALBEF
PSO
31.4
97.1
1,001
44.6
96.5
732
39.8
99.2
999
Sp-RS&PWWS
29.1
95.0
1,721
61.8
94.9
1,316
46.5
96.5
1,643
SparseMA 40.7
96.6
428
61.7
96.1
341
56.3
98.4
375
Table 2: Attack success rate (Suce., %)
multimodal similarity (Sim., %) and the number of queries (Query) of
various attacks (image attack Sparse-RS, text attacks PWWS and PSO, multimodal attacks Sp-RS&PWWS and SparseMA) three models using three datasets for multimodal classification. The highest attack success rate is highlighted in bold. The second highest attack success rate underlined.
is lower than that of all baselines except PWWS, being about 1/3 that of the multimodal attack Sp- RS&PWWS. Despite the lower query number and higher similarity of PWWS, our attack success rate is significantly higher than PWWS. Therefore, cannot be concluded that PWWS implies a better effectiveness.
Then, present two instances of adversarial examples in Figure 3. It can be seen that SparseMA adds patches to the image generate adversarial examples, which have fewer perturbations and are easier to add for attackers the real-world. Al- though the perturbations are indeed visible, they do not alter the semantics of adversarial images, which are still recognizable by humans. Addi- tionally, SparseMA perturbs fewer words than Sp-RS&PWWS. These evaluations demonstrate the high quality and practicality in the physical world of the adversarial examples generated by SparseMA.
In conclusion, SparseMA achieves a higher at- tack success rate and similarity with a lower query number compared to the combined multimodal at- tack Sp-RS&PW WS, demonstrating the necessity of bridging the gap between different modalities for effective multimodal attack. These results validate the superiority of our proposed method. Moreover, we find that by perturbing only 2% to 3% of the top-level features, we can make the victim model output completely different predictions, thus con- firming the vulnerability of the multimodal model. These discoveries suggest that the model relies heavily these small subsets of high-level fea- tures determine its output, but these features highly susceptible adversarial attacks. Therefore, improving the robustness of the model's high-level features against perturbations would be crucial method to enhance the model's robustness.
4.3 Vulnerability on Different Modalities
Multimodal classifier utilizes information from var- ious modalities to predict the classification results, which is expected to be robust to all modalities and achieves better performance. However, we find that multimodal models tend to be robust one modality while being vulnerable to another. When attacking the robust modality, the attack per- formance usually poor. In contrast, attacking the vulnerable modality would result in good at- tack performance. For instance, PWWS for text

---

fewer perturbations
fewer perturbations
cusier to add
Bak nantis) sntienfscaosl baree Heres modelt oo pambuu cope Tipe rffor Scherhookkssssingg apocilaajoppe
Bakk sentlcjacheed sbow Hare's has cope Tipe from parretin expor Gannsorzla..
How siffoca) eotlii lagpen?? cas, 3 disaaaatiin dertnred drachiee lissee Scrrp, 166a feeg sioegtaafic ig comooddo g 3e25reeiiodoo
How does cptiii) happes? atthren dineniee topkee Banp; His ryep Sisetaatie 100 Radio
fewer perturbations
fewer perturbations
Sp-RS&PWWS
SparseMA
Sp-RS&PWWS
SparseMA
(a) The sample with positive label perturbed by various attacks is misclassified us negative.
(b) The sample with negative label perturbed by various atlacks is misclussified as positive.
Figure Adversarial examples generated by SparseMA and Sp-RS&PWWS on MVSA-Mulh dataset. The adversarial images generated by SparseMA have fewer perturbations and are easier to add for attackers in the real-world than that of Sp-RS&PWWS. And SparseMA perturbs fewer words than Sp-RS&PWWS. We highlight the words replaced by the attacks in Red. The corresponding original words are highlighted in Green.
MVSA-Multi
MVSA-Single
CrisisMMD
Model
Image
Text
Both
Image
Text
Both
Image
Text
Both
CLIPvT
51.0
3.4
45.2
60.0
3.6
36.0
15.0
15.8
68.9
CLIPRe Res
58.0
1.0
41.0
56.0
2.5
41.0
17.0
17.8
64.9
ALBEF
20.0
24.0
56.3
27.0
28.6
44.4
12.0
33.9
54.2
Table 3: Percentage (%) of adversarial examples generated by SparseMA that perturb only on image or text, or both.
attack shows poor attack performance when attack- ing CLIPvI model on MVSA-Multi dataset, as shown in Table 2. Conversely, Sparse-RS for im- age attack performs better. We believe this due to the model over-relying on one modality while disregarding the other. To better evaluate the vul- nerability robustness of multimodal models to different modalities, we present the percentage of adversarial examples generated by SparseMA that perturb only on image or text, or both, in Table
Attack
Clean
Brig.
Cont.
Satu.
Shar.
Sparse-RS
70.7
28.0
30.2
41.5
32.5
Sp-RS&PWWS
71.8
36.5
39.8
46.7
39.8
SparseMA
82.5
38.9
40.9
53.4
45.0
Table Attack success rate (Suce., %) of diffcrent attacks CLIPvir model using MVSA-Multi dataset when Brightness (Brig.). Contrast (Cont.). Saturation (Satu.) or Sharpness (Shar.) varies in the real-world.
4.4 Evaluation in the Real-World
We can see that SparseMA prefers to perturb the more vulnerable modality. For example, SparseMA perturbs more images on the CLIP viT model using the MVSA-Multi dataset, where image attacks per- form well. In addition. SparseMA tends to perturb both modalities simultaneously when the multi- modal model is robust to both modalities, such on the ALBEF model using the MVSA- dataset, where both image attacks and text attacks perform poorly. This demonstrates that SparseMA has discovered the vulnerability of the model and generated corresponding perturbations for the at- tack. It also indicates that SparseMA may be good metric to evaluate the vulnerability or robust- ness of the model different modalities.
Adversarial images captured by digital devices are usually affected by physical factors, such bright- ness, saturation, contrast or sharpness, etc. Thus, good physical attack should generate perturbations that are easy to be added to images, and also re- sistant to various physical factors. To evaluate the effectiveness of SparseMA in the real-world, we randomly adjust the brightness, contrast, saturation, and sharpness of the adversarial images generated by various attacks CLIP model using MVSA- Multi dataset to 0.5 1.5 times the original value. Then we evaluate their attack success rate shown in Table 4. The results are averaged on five runs to eliminate randomness. We could observe that brightness typically has the largest effect, and sat-

---

Succ.
Pert
Pertr
Sim
Simr
Sim
Image
Text
Both
Query
8
83.7
2.8
5.6
94.0
85.6
97.0
70.0
0.5
29.4
1,642
12
83.8
5.1
5.9
91.6
84.3
96.8
66.0
1,4
32.9
811
16
76.3
7.4
8.1
89.9
78.5
96.4
57.0
3.9
39.6
510
20
80.0
8.7
6.7
88.7
81.2
96.9
60.0
3.6
36.0
366
24
77.3
10.7
7.3
86.9
79.8
96.5
57.0
5.4
37.9
273
28
78.8
12.8
7.2
85.1
79.7
96.7
56.0
5.0
39.1
188
32
75.4
15.5
8.0
82.7
77.1
95.5
51.0
6.4
42.6
159
Table Attack performance and query number (Query) SparseMA varying different side length of the patch CLIPwT model using MVSA-Single dataset.
uration has the least effect on all attacks. Notably, SparseMA consistently achieves the highest attack success rate among all baselines, demonstrating the practicality of SparseMA in the real-world.
adversarial example with different numbers of per- turbed elements, we set the number of iterations N to be an integer time of the number of perturbed elements. We evaluate the final perturbation rate and multimodal similarity in final adversarial ex- amples using various iterations from 2 to 8 times the number of perturbed elements with interval of 2. We observe that they achieve similar per- turbation rates (8.71% 8.78% in images and 6.73% 6.92% texts) and similar multimodal similarity (96.91% ~ 96.94%). More iterations will result in a lower perturbation rate but more queries, so we choose an intermediate value, i.e., N is four times the number of perturbed elements.
4.5 Parameter Study
To investigate the impact of the hyper-paranee paper-parrmeters in SparseMA, including the side length of patches and the number of iterations N for random recov- ery, conduct a series of experiments on CL model using MVSA-Sine -Snglee dataset. We addition- ally evaluate the similarity to the original samples using the Structural Similarity (SSIM) (Wang et al., 2004) for images and the Universal Sequence En- coder (USE) (Cer et al., 2018) 8) for texts, denoted Sim; and Simy. And we adopt the perturbation rate to measure the percentage of pixels perturbed in an image or words perturbed in text, denoted Perty and Perty.
Conclusion
We propose vision-language black-box adversar- ial attack method called Sparse Multimodal Attack (SparseMA). SparseMA maps the input image and text into discrete space by splitting the image into patches and the text into words. Then it evaluates the importance of each patch or word based on its affects to the model's output. By greedily sub- stituting important patches words with suitable candidates, SparseMA generates adversarial exam- ples with fewer perturbations and higher quality, which achieves a higher attack success rate than the baselines. Additionally, SparseMA can reveal the vulnerability multimodal models to differ- ent modalities and concentrate on perturbing the more vulnerable ones. It could be served as a good metric to measure the vulnerability of multimodal models to different modalities. And it can help better understand the multimodal model's behav- ior, and enhance its robustness against adversarial attacks. Experiments show that the sparse pertur- bations generated by SparseMA are more practical in the physical world that are easier to be added
On the side length of patches The side length restricts the minimum perturbation unit images. Table 5, we evaluate the attack performance of SparseMA using various side lengths from 8 to 32 with an interval of 4. SparseMA performs well when 8, achieving a high attack success rate. It also has the lowest perturbation rate and the highest similarity, which prefers perturb images individually. However, it has a high query cost for the victim model. As we increase the side length of the image patch, the attack success rate will change unpredictably. Also, increasing the patch size results in a decrease in the number of queries but an increase in the perturbation rate. Addition- ally, SparseMA gradually boosts its preference for text perturbations. To balance attack performance and query cost, we select an intermediate value of s = 20 for our experiments.
On the number of iterations for random re- covery To perform random recovery in the initial

---

for attackers, and perform well against real-world influences. SparseMA would be a strong baseline for future works and may inspire more researches on multimodal attacks.
Alexey Dosovitskiy, Lucas Beyer. Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An image is worth 16x 16 words: Transformers for image recognition scale. In International Conference on Learning Representations.
Limitations
SparseMA focuses on the two most typical modali- ties in the multimodal classification task, i.e., con- tinuous images and discrete text. It does not take other types of modalities, such as audio and sig- nal, into consideration. Actually, data from these modalities can also be processed by similar sparse strategy, and then apply our method to generate ad- versarial examples. We will continue to investigate the potential of SparscMA in our future work.
Javid Ebrahimi. Anyi Rao. Daniel Lowd, and Dejing Dou. 2018. HotFlip: White-box adversarial exam- ples for text classification. In Association for Com- putational Linguisties.
Ivan Evtimov, Russel Howes, Brian Dolhansky, Hamed Firooz, and Cristian Canton Ferrer. 2020, Adversar- ial evaluation of multimodal models under realistic gray box assumption. In ArXiv preprint.
Siddhant Garg and Goutham Ramakrishnan. 2020. BAE: BERT-based adversarial examples for text clas- sification. In Conference Empirical Methods in Natural Language Processing.
A Acknowledgement
This work is supported by National Natural Science Foundation (62076105, U22B2017).
Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy, 2015. Explaining and harnessing adversar- ial examples. In International Conference on Learn- ing Representations.
References
Firoj Alam, Ferda Ofli, and Muhammad Imran. 2018. Crisismmd: Multimodal twitter datasets from natural disasters. In AAAI conference on web and social media.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recogni- tion. In Conference on Computer Vision and Pattern Recognition.
Moustafa Alzantot, Yash Sharma, Ahmed Elgohary, Bo-Jhang Ho, Mani Srivastava, and Kai- Wei Chang. 2018. Generating natural language adversarial exam- ples. In Conference on Empirical Methods in Natural Language Processing.
Di Jin, Zhijing Jin, Joey Tianyi Zhou, and Peter Szolovits. 2020. Is BERT really robust? A strong baseline for natural language attack on text classifica- tion and entailment. AnAI Conference on Artificial Intelligence.
Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St John, Noah Constant, Mario Guajardo-Céspeedes, Steve Yuan, Chris Tar, et al. 2018. Universal sentence encoder. In arXiv preprimt arXiv:1803.11
Douwe Kiela, Hamed Firooz, Aravind Mohan, Vedanuj Goswami, Amanpreet Singh, Pratík Ringshia, and Davide Testuggine. 2020. The hateful memes chal- lenge: Detecting hate speech in multimodal memes. In Advances in Neural Information Processing Sys- tems.
Francesco Croce, Maksym Andriushchenko, Naman D Singh, Nicolas Flammarion, and Matthias Hein. 2022. Sparse-rs: versatile framework for query-efficient sparse black-bor. box adversarial attacks. In AAAI Con- ference on Artificial Intelligence.
Dianqi Li. Yizhe Zhang, Hao Peng, Liqun Chen, Chris Brockett, Ming-Ting Sun, and Bill Dolan. 2021a. Contextualized perturbation for textual adversarial attack. In North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
Francesco Croce and Matthias Hein. 2019. Sparse and imperceivable adversarial attacks. In /EEE/CVF In- ternational Conference Computer Vision.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large lan- guage models. In ArXiv preprint.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei Li. 2009. Imagenet: A large-scale hierar- chical image database. In Conference on Computer Vision and Pattern Recognition.
Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafi Joty, Caiming Xiong, and Steven Chu Hong Hoi. 202 Align hefore fuse: Vision and language representation learning with momentum distillation. In Advances in Neural Information Processing Sys- lems.
Yinpeng Dong, Fangzhou Liao, Tianyu Pang, Hang Su, Jun Zhu, Xiaolin Hu, and Jianguo Li. 2018 Boosting adversarial attacks with momentum. In Conference on Computer Vision and Pattern Recognition.

---

Linyang Li, Ruotian Ma. Qipeng Guo, Xiangyang Xue, and Xipeng Qiu. 2020. -ATTKK Adversarial attack against BERT using BERT. In Conference on Empirical Methods in Natural Language Processing.
Shuhuai Ren, Yihe Deng, Kun He, and Wanxiang Che. 2019. Generating natural language adversarial exam- ples through probability weighted word saliency. In Association for Computational Linguistics.
Christian Szegedy, Wojciech Zaremba. llya Sutskever, Joan Bruna, Dumitru Erhan, Ian J. Goodfellow, and Rob Fergus. 2014, Intriguing properties of neural networks. In International Conference on Learning Representations.
Bin Liang, Hongcheng Li, Miaoqiang Su, Pan Bian, Xirong Li, and Wenchang Shi. 2018. Deep text clas- sification can be fooled. In International Joint Con- ference on Artificial Intelligence.
Jiadong Lin, Chuanbiao Song, Kun He, Liwei Wang. and John E. Hopcroft. 2020. Nesterov accelerated gradient and scale invariance for adversarial attacks. In Intermational Conference on Learning Representa- tions.
Yapeng Tian and Chenliang Xu. 2021. Can audio-visus integration strengthen robustness under multimodal attacks? In IEEE/CVF Conference on Computer Vision and Pattern Recognition.
Nishant Vishwamitra, Hongxin Hu, Ziming Zhao, Long Cheng, and Feng Luo. 2021. Understanding and measuring robustness of multimodal learning. In ArXiv preprint.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. 2018. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. 2004. Image quality assessment: from error visibility to structural similarity. In IEEE trans- actions on image processing.
Rishabh Maheshwary, Saket Maheshwary, and Vikram Pudi. 2021. Generating natural language attacks in hard label black box setting. In AAAI Conference on Artificial Intelligene.
Karren Yang, Wan- Yi Lin, Manash Barman, Filipe Con- dessa, and Zico Kolter. 2021. Defending multimodal fusion models against single-source adversaries. In IEEE/CVF Conference on Computer Vision and Pat- tern Recognition.
Nikola Mrkšić, Diarmuid Ó Séaghdha, Blaise Thom- son, Milica Gašić, Lina M. Rojas-Barahona, Pei-Hao Su, David Vandyke. Tsung-Hsiie Hsien Wen, and Steve Young. 2016. Counter foutter-itting word vectors to linguis- tic constraints. In North American Chapter the Association for Computational Linguistics: Human Language Technologies.
Youngjoon Yu, Hong Joo Lee, Byeong Cheon Kim, Jung Uk Kim, and Yong Man Ro. 2020. Investigating vulnerability to adversarial examples on multimodal data fusion in deep learning. In ArXiv preprint.
Nina Narodytska and Shiva Prasad Kasiviswanathan. 2017. Simple black-box adversarial attacks deep neural networks. In IEEE Conference on Computer Vision and Pattern Recognition Workshops.
Zhen Yu, Xiaosen Wang. Wanxiang Che, and Kun He. 2022. Texthacker: Learning based hybrid lo- cal search algorithm for text hard-label adversarial attack. In Findings of the Association for Computa- tional Linguistics: EMNLP 2022, pages 622-637.
Teng Niu, Shiai Zhu, Lei Pang. and Abdulmotaleb EI Saddik. 2016. Sentiment analysis on multi-vies view so- cial data. In International Conference on Multimedia Modeling.
Tom Zahavy, Abhinandan Krishnan, Alessandro Mag- nani, and Shie Mannor. 2018 Is picture worth thousand words? deep multi-modal architecture for product classification in e-commerce. In AAAI Conference on Artificial Intelligence.
Nicolas Papernot, Patrick McDaniel, Somesh Jha, Matt Fredrikson, Z Berkay Celik, and Ananthram Swami. 2016a. The limitations of deep learning in adversarial settings. In IEEE European Symposium on Security and Privacy (EuroS&P).
Yuan Zang, Fanchao Qi, Chenghao Yang, Zhiyuan Liu, Meng Zhang. Qun Liu, and Maosong Sun. 2020. Word-level textual adversarial attacking as combi- natorial optimization. In Association for Computa- tional Linguistics.
Nicolas Papernot, Patrick McDaniel, Ananthram Swami, and Richard Harang. 2016b. Crafting adversarial in- put sequences for recurrent neural networks. In MIL- COM IEEE Military Communications Conference.
Jiaming Zhang. Qi Yi, and Jitao Sang. 2022. Towards adversarial attack on vision-language pre-training models. In ACM International Conference on Multi- media.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- ur, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learn- ing transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning, ICML.
Pu Zhao, Sijia Liu, Pin- Yu Chen, Nghia Hoang, Kaidi Xu. Bhavya Kailkhura, and Xue Lin. 2019. On the design of black-box adversarial examples by leverag- ing gradient- free optimization and operator splitiing method. In IEEE/CVF International Conference on Computer Vision.


Reviews:
[{"review_id": "zscZizPDuW", "reviewer": "Reviewer_GDxW", "summary": "", "questions": "1. What would be the major reason that the SparseMA underperforms existing baselines on the MVSA-Single dataset?\n\n2. I stated concerns in the weakness section. Please address them to validate the work.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful comments which we will address below.\n\nQuestion 1 The use of importance score is questionable.\n\nAnswer 1: In theory, a completely robust model would exhibit minimal changes in its prediction scores when subjected to perturbations in specific patches or words. Under this scenario, the importance score (replacement order) will be less informative, as you commented. However, in practice, when we conduct adversarial attacks on existing multimodal models, we observe a high attack success rate of 80%, showing the value of importance score. This finding clearly indicates that current models are far from achieving complete robustness. The existence of such vulnerabilities underscores the significance of researching adversarial attacks. \n\nThese attacks serve as a catalyst for advancing the development of models that are more robust and resistant to such adversarial attempts. By studying and understanding these attacks, we can identify weaknesses in models and work towards enhancing their robustness, ensuring the reliability and security of their predictions.\n\nQuestion 2 The effectiveness of the proposed attack on these adversarially-trained models.\n\nAnswer 2: We appreciate your acknowledgement that attacking adversarially-trained models can highlight the effectiveness of our attack. In the real world, there are abundant samples with sparse perturbations, whereas the academic community primarily focuses on imperceptible perturbations. As a result, models adversarially trained on imperceptible perturbations may struggle to defend against adversarial samples with sparse perturbations. However, there is a lack of prior research concerning sparse attacks and adversarial training in the domain of multimodal adversarial attack now. We develop SparseMA with the intention of fostering the advancement of adversarial training against sparse perturbations. We will explore further research regarding adversarial training with SparseMA in our future work and provide a good baseline for defenses.\n\nQuestion 3 Is the perturbation of patch a brute-force approach?\n\nAnswer 3: Perturbing objects obtained by semantic segmentation is an excellent approach to enhance the interpretability of adversarial examples. However, our investigation reveal that in black-box scenario, it can be challenging to fully understand which objects the model is focusing on. These objects may range from real-world entities like flowers, cats, and dogs to more abstract semantic objects like a smile or a logo. However current semantic segmentation models primarily concentrate on real objects, making it difficult to comprehensively cover all semantic targets. This limitation can significantly decrease the number of viable targets for perturbation and make it challenging to control the number of perturbed pixels, which can lead to a decline in the attack effectiveness.\n\nTo address these challenges, we have adopted a strategy of replacing patches instead. By doing so, we can locate the regions of interest for the model without the need to specifically identify the exact objects it is focused on. Although this approach sacrifices the interpretability of adversarial samples, it greatly enhances the effectiveness of our attack.\n\nQuestion 4 More effective search algorithms rather than grid search\n\nAnswer 4: Let me clarify that in our paper, we primarily employed a greedy algorithm rather than grid search. Greedy search we employed and other search algorithms, including genetic algorithms, possess different strengths and weaknesses. In the context of text attacks, utilizing Particle Swarm Optimization (PSO), a variant of genetic algorithms, would result in higher attack success rates but incur significant costs for accessing the model. In contrast, the greedy-based attack PWWS requires fewer queries but achieves a slightly lower attack success rate. When dealing with large multimodal models, the query cost becomes significantly high, thus motivating our selection of the greedy strategy.\n\nQuestion 5 The connection between the problem illustrated in this figure and the investigated task.\n\nAnswer 5: Figure 1 illustrates a scenario where illegal online merchants deceive detection models by perturbing a prohibited drug. This poses a severe security threat to numerous e-commerce companies. Therefore, our objective is to develop a sparse multimodal adversarial attack method capable of simulating these perturbation behaviors. In this paper, we employ techniques such as blacking out, whitening, and pixelation to emulate these behaviors. In practice, we achieve a 94% attack success rate on Alibaba's online detection model. We hope that SparseMA can identify more robust models or enhance the security of these applications through adversarial training. We will emphasize this connection in the revised version of our paper and explore the further research regarding adversarial training with SparseMA in the future work.\n\nQuestion 6 The reason that SparseMA underperforms Sp-RS&PWWS on the MVSA-Single dataset?\n\nAnswer 6: The parameter study in Table 5 shows that SparseMA achieves an attack success rate of 83.8% when the side length of the patch is set to 12, significantly surpassing Sp-RS&PWWS, which achieves an attack success rate of 81.2%. This may mean that the important objects in the MVSA-Single dataset are smaller, thereby requiring a smaller patch size to achieve a higher attack success rate.\n\nWe sincerely hope the reviewer find our response helpful, and update the scores if the concerns have been resolved. We are more than happy to further discuss any concerns that you find not fully addressed.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is well-written with a clear description of how the framework is constructed.\n2. The authors develop the framework in the black-box setting, which can potentially inspire future works against models of large parameter scales.\n3. The authors explore adversarial attacks on multimodal models, which is a relatively underexplored research problem in past literature.", "reasons_to_reject": "1. The use of importance score is questionable. The representativeness of the candidate patches depends on the variation of the model prediction score. If the model is sufficiently invariant/robust such that there are no significant prediction changes for all patches, the importance score (replacement order) will be less informative for deciding the perturbation location. In other words, Line 10 - Line 11 in Algorithm 1 can happen frequently.\n\n2. Past adversarial works [1,2] have explored semantic attacks that manipulate attributes/appearances/textures of the object to fool the target model. A natural defense against these attacks is to perform adversarial training over the generated semantic adversarial examples. To show the validity of the work, authors should add additional discussions on the effectiveness of the proposed attack on these adversarially-trained models.\n\n3. Dividing the input through patches to evaluate sensitivity is a brute-force approach. Authors should argue more about why the object saliency map or object segmentation is not adopted to pre-process the semantics in the image for better attack success rates.\n\n4. There are many kinds of black-box optimization rather than the grid search adopted by the paper. Authors shall consider using more effective search algorithms (e.g., natural evolution strategy) for generating black-box attacks.\n\n5. The paper claims the motivation concerning illegal online merchants. However, few discussions have been raised on how to defend these malicious edits using the proposed framework. It will be contributive for the paper to show the real-world applicability (e.g., perform adversarial training on this attack to get more robust models).\n\n[1] SemanticAdv: Generating Adversarial Examples via Attribute-conditional Image Editing (ECCV 2020)\n\n[2] Adversarial Camouflage: Hiding Physical-World Attacks with Natural Styles (CVPR 2020)", "questions_to_authors": "1. What would be the major reason that the SparseMA underperforms existing baselines on the MVSA-Single dataset?\n\n2. I stated concerns in the weakness section. Please address them to validate the work.", "paper_topic_and_main_contributions": "Authors propose a black-box attack on the semantic level against multimodal models, named Sparse Multimodal Attack (SparseMA). Motivated by malicious editing from certain online merchants, SparseMA evaluates the image patch sensitivity w.r.t. the text prompt (through the variation of model prediction) and locates the most sensitive semantic component for perturbation. The experiments show that SparseMA has higher attack success rates (ASR) compared to existing multimodal attacks.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "CkfBb7XMGN", "reviewer": "Reviewer_BaP4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful comments which we will address below.\n\nQuestion 1: The computational cost of Replacement Order Determination and Candidate Substitution\n\nAnswer 1: The computational cost of adversarial attacks can be measured by the number of queries to victim the model. In our case, we can calculate that the Replacement Order Determination operation only requires 144 queries ((224/20) \\* (224/20)) to determine the importance of patches, and an average of 14 queries for word importance on the MVSA-Multi dataset, totaling 158 queries. On the other hand, the Candidate Substitution operation requires a maximum of 488 queries (144 \\* 3 + 14 \\* 4). From Table 2, it can be observed that our query count ranges from 300 to 400. In contrast, Sp-RS&PWWS and PSO require thousands of queries. Therefore, SparseMA achieves more efficient attacks compared to Sp-RS&PWWS and PSO. We will clarify it in the revision. Thanks. \n\nQuestion 2: The explanation of term \"Sparse\" in SparseMA\n\nAnswer 2: The experiments in Section 4.2 validate that Sp-RS&PWWS, which individually attacks images and text, performs worse than SparseMA, and this establishes connections between images and text. Due to their significant differences in data attributes, such as continuous images versus discrete text, it is challenging to simultaneously handle both modalities. As a result, we adopt a strategy of dividing the image into patches to convert it into discrete data, facilitating improved communication between image and text information. Meanwhile, this discretization approach allows us to handle high-resolution images more efficiently. In contrast, other sparse attacks in the image domain typically require manipulation of each pixel, resulting in a computational cost of 10^6 queries. This substantial computational cost makes them less practical and efficient.\n\nQuesiton 3: The evaluation on images where text is embedded in an image.\n\nAnswer 3: In practical applications, when dealing with images like the one illustrated in Figure 1, it is common practice to employ OCR tool to extract text from the image. The extracted text is subsequently combined with the image and fed into a multimodal model for classification or attack purposes. This approach ensures that our evaluation aligns with real-world applications by leveraging both image and text information. Therefore, we can maintain consistency between our assessment and practical scenarios.\n\nQuestion 4: Line 236 \"Replacement order Determination\", determine should be \"Replacement order determination.\"\n\nAnswer 4: Thanks for your careful inspection, we'll fix it in the revised version.\n\nWe sincerely hope the responses provided above address your questions. We are more than happy to further discuss any concerns that you find not fully addressed.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The research motivation is effectively presented, particularly exemplified through the insightful Figure 1.\n2. The paper gives a clear algorithmic description and an illustrative overview figure of SparseMA, contributing to its comprehensibility.\n3. SparseMA attains the highest attack success rate in 7 out of 9 cases, surpassing all unimodal and multimodal baseline methods.", "reasons_to_reject": "1. The operations of Replacement Order Determination and Candidate Substitution raise concerns about significant computational overhead. Surprisingly, the paper neglects to address this issue.\n2. The term \"Sparse\" in SparseMA remains inadequately explained within the methodology section. The rationale for expecting sparse perturbation due to such design choices is not clearly elucidated.\n3. The real-world multimodal scenario depicted in Figure 1, where text is embedded in an image, lacks evaluation. The evaluation is restricted to separate images and text.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper introduces SparseMA, a black-box sparse multimodal adversarial attack approach designed to deceive multimodal deep learning models. The effectiveness of SparseMA has been evaluated in comparison with three baseline methods across three distinct datasets.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Line 236  \"Replacement order Determination\" determine should be \"Replacement order determination.\"\n"}, "extra_text": {}}, {"review_id": "Vw7QWCpyeG", "reviewer": "Reviewer_RAMp", "summary": "", "questions": "Question A:\nCan the multi-modal approach described in this paper be extended to other modalities?\n\nQuestion B:\nHow do the authors make the adversarial perturbations imperceptible to the human eyes?\n\nQuestion C:\nHow do the authors make sure that the semantics of the adversarial images are not changed?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful comments which we will address below.\n\nQuestion 1: How do the authors make sure that the semantics of the adversarial images are not changed? How do the authors make the adversarial perturbations imperceptible to the human eyes?\n\nAnswer 1: We suspect that the reviewer may have misunderstood the application scenario of our attacks. Previous attacks have primarily focused on the imperceptibility of adversarial perturbations. In contrast, our proposed adversarial scenario is more prevalent in real-world situations, particularly in e-commerce or compliance domains. In these scenarios, attackers aim to deceive classifiers through perturbations while still allowing users to understand the non-compliant aspects of the sample, as depicted in Figure 1. We will clarify the difference in the revision.\n\nIn fact, images with localized pixelation are often used on various online platforms, yet people can still effectively comprehend the content and meaning of the image. To automatically evaluate whether users can understand the information conveyed by our generated adversarial examples, we define a multimodal similarity metric similar to the commonly used text similarity tool called Universal Sentence Encoder (USE).[1]. It involves using a robust model to extract representations of samples and calculating the similarity of these representations as a measure of similarity. To extract feature vectors from multimodal inputs, we chose the BLIP2[2] model, which has demonstrated excellent performance across various downstream tasks. The experiment in Table 2 validates that our generated adversarial samples possess high quality and maintain high semantic similarity to the original samples. Therefore, although these sparse perturbations are indeed visible, they generally do not alter the semantics.\n\nQuestion 2: Can the multi-modal approach described in this paper be extended to other modalities?\n\nAnswer 2: Yes. As stated in limitation section, data from other modalities can also be processed by a similar sparse strategy, such as audio. Then SparseMA could efficiently perturb these samples to generate adversarial examples.\n\nQuestion 3: How do the authors make the adversarial perturbations imperceptible to the human eyes?\n\nAnswer 3: Please see Answer 1.\n\nQuestion 4: How do the authors make sure that the semantics of the adversarial images are not changed?\n\nAnswer 4: Please see Answer 1.\n\nWe sincerely hope the reviewer find our response helpful, and update the scores if the concerns have been resolved. We are more than happy to further discuss any concerns that you find not fully addressed.\n\n\n[1] Universal sentence encoder\n\n[2] BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.\tThe performance of the proposed method is promising.\n\n2.\tThis paper focuses on black-box multimodal attacks, which is an interesting and timely research topic.", "reasons_to_reject": "1.\tThese sparse perturbations are visible, maybe they will change the semantics.\nIn Figure 2, the authors split the image into 7 × 7 blocks and then, based on the model output, evaluate their importance by setting each patch to all zeros or each word to [UNK] and determine the order of replacement. The resulting disturbance may lead to changes in the semantics of the original images. How do the authors make sure that the semantics of the adversarial images are not changed?\n\n2.\tTraditional adversarial attack methods limit the magnitude of disturbance in order to make the disturbance invisible to the human eyes. How do the authors make the adversarial perturbations imperceptible to the human eyes?", "questions_to_authors": "Question A:\nCan the multi-modal approach described in this paper be extended to other modalities?\n\nQuestion B:\nHow do the authors make the adversarial perturbations imperceptible to the human eyes?\n\nQuestion C:\nHow do the authors make sure that the semantics of the adversarial images are not changed?", "paper_topic_and_main_contributions": "This paper proposes a vision-language black-box adversarial attack method called Sparse Multimodal Attack (SparseMA). Experiments show that the sparse perturbations generated by SparseMA are more practical in the physical world and are easier to be added for attackers.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "VqApDqNQbH", "reviewer": "Reviewer_jyg5", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful comments which we will address below.\n\nQuestion 1 The adversarial tactics employed by illicit merchants do indeed effect semantic changes\n\nAnswer 1: We suspect that the reviewer may have misunderstood the application scenario of our attacks. Previous attacks have primarily focused on the imperceptibility of adversarial perturbations. In contrast, our proposed adversarial scenario is more prevalent in real-world situations, particularly in e-commerce or compliance domains. In these scenarios, attackers aim to deceive classifiers through perturbations while still allowing users to understand the non-compliant aspects of the sample, as depicted in Figure 1. We will clarify the difference in the revision.\n\nIn fact, images with localized pixelation are often used on various online platforms, yet people can still effectively comprehend the content and meaning of the image. To automatically evaluate whether users can understand the information conveyed by our generated adversarial examples, we define a multimodal similarity metric similar to the commonly used text similarity tool called Universal Sentence Encoder (USE).[1]. It involves using a robust model to extract representations of samples and calculating the similarity of these representations as a measure of similarity. To extract feature vectors from multimodal inputs, we chose the BLIP2[2] model, which has demonstrated excellent performance across various downstream tasks. The experiment in Table 2 validates that our generated adversarial samples possess high quality and maintain high semantic similarity to the original samples. Therefore, although these sparse perturbations are indeed visible, they generally do not alter the semantics.\n\nAs shown in Table 2, SparseMA achieves a 96% similarity between generated adversarial examples and original samples. Moreover, based on statistical analysis, we discovered that our image perturbation algorithm only affects an average of 1% to 3% of the image area. These slight perturbations generally do not alter the semantics of the samples.\n\nQuestion 2 The research on black-box multimodal attack\n\nAnswer 2: To the best of our knowledge, there have been no reported works of transfer attacks or zeroth-order optimization black-box attacks in multimodal settings. However, direct transfer attacks of image domain often yield poor attack performance when applied to black-box models. Therefore, our work focuses on bridging the research gap in black-box attacks within the multimodal domain. By doing so, we aim to provide subsequent researchers with a valuable reference for conducting black-box adversarial attacks.\n\nQuestion 3: The potential applications of SparseMA on other multimodal task.\n\nAnswer 3: This study serves as a mere validation of the efficacy of a multimodal attack approach, using multimodal classification as a case in point. SparseMA exhibits model-agnostic characteristics and maintains loose associations with the downstream task. Its core idea lies in establishing bridges between different modalities by means of discretization, enabling a more efficient generation of adversarial samples. This conceptual framework can equally be applied to other multimodal tasks.\n\nQuestion 4 Vision-language multimodal classification is a niche field\n\nAnswer 4: We respectively do not agree with you. In the modern era, information is commonly presented in a multimodal manner, integrating different forms such as text and images. This can be observed in news articles, social media posts, and various other sources. Therefore, vision-language multimodal classification represents a prevalent scenario in the real world. The research on the vulnerabilities and robustness of multimodal classification models is highly valuable across various domains, including e-commerce categorization and compliance. \n\n\nQuestion 5: The connection between the problem illustrated in this figure and the investigated task.\n\nAnswer 5: Thank you for your question. We provide a detailed explanation of the relationship between these two entities as follows. Figure 1 illustrates a scenario where illegal online merchants deceive detection models by perturbing a prohibited drug. This poses a severe security threat to numerous e-commerce companies. Therefore, our objective is to develop a sparse multimodal adversarial attack method capable of simulating these perturbation behaviors. In this paper, we employ techniques such as blacking out, whitening, and pixelation to emulate these behaviors. In practice, we achieve a 94% attack success rate on Alibaba's online detection model. We hope that SparseMA can identify more robust models or enhance the security of these applications through adversarial training. We will emphasize this connection in the revised version of our paper.\n\n\nQuestion 6: The implement details of Sp-RS&PWWS\n\nAnswer 6: Spare-RS and PWWS are two different attacks that cannot be simultaneously applied in the same space. As a result, we apply Spare-RS in the image space and PWWS in the text space, combining the generated samples from both approaches to obtain the final adversarial examples. We will provide further clarification in the revised version.\n\n\nQuestion 7: About the multimodal similarity metric\n\nAnswer 7: As stated in Answer 2, we utilize BLIP2 model to define a multimodal similarity metric. Although the absolute values of this multimodal similarity metric may be subject to some bias due to limitations of the model, the relative values obtained through comparison still possess significant reference value. Moreover, in most cases, SparseMA performs on par with, or even surpasses, the baselines. This observation suggests that our SparseMA approach minimally disrupts the semantic of samples and generates high-quality adversarial examples.\n\nWe sincerely hope the reviewer find our response helpful, and update the scores if the concerns have been resolved. We are more than happy to further discuss any concerns that you find not fully addressed.\n\n\n[1] Universal sentence encoder", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper proposes a novel approach for generating adversarial attacks against vision-language multimodal classifiers. Their approach performs attacks in the joint space of image patches and text tokens, and achieves superior performance against a baseline that performs attacks separately in image space and text space.", "reasons_to_reject": "In Line 87, the authors assert that their proposed attack preserves the semantic integrity of the data. However, upon examining Figure 1, it appears that the adversarial tactics employed by illicit merchants do indeed effect semantic changes. This disparity merits further clarification from the authors.\n\nThe authors assert that their methodology represents the first-ever approach for black-box multimodal attacks. However, it's noteworthy that existing white-box attacks can potentially be transformed into black-box attacks using strategies such as Substitute Training (where a surrogate model is trained to mimic the black-box model and then attacked as a white-box model) or Zeroth-order Optimization. Thus, the distinctiveness and significance of the authors' contributions might warrant further detailed explanation.\n\nVision-language multimodal learning spans diverse tasks such as image captioning, visual question answering, text-to-image synthesis, image-to-text retrieval, and visual grounding. However, the authors only elucidate their focus on vision-language multimodal classification in their experiments section. The manuscript's organization could benefit from earlier clarification of this focus, as its current structure may lead to confusion for readers.\n\nGiven that vision-language multimodal classification is a niche field, the paper's contributions might be of limited impact and applicability.\n\nThe approach presented appears to draw its motivation from Figure 1. However, the connection between the problem illustrated in this figure and the investigated task of vision-language multimodal classification is not sufficiently clear.\n\nThe paper lacks sufficient implementation details for the Sp-RS&PWWS baseline. It remains unclear whether it employs an alternative optimization approach—alternating between attacks in the image space and the text space—or conducts simultaneous optimization in both spaces.\n\nThe 'multimodal similarity' metric, employed for perturbation measurement, lacks a clear definition, leaving room for ambiguity. This raises questions about the fairness and validity of the comparisons made with baseline models.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper investigates the robustness of vision-language multimodal classifiers, presenting a novel strategy for implementing adversarial attacks on such models. The authors' method discretizes the input image and performs attacks in the combined space of image patches and input tokens. Compared to a baseline that conducts attacks separately in image and text spaces, their approach demonstrates superior performance.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a vision-language black-box adversarial attack method called Sparse Multimodal Attack (SparseMA). The SparseMA discretizes the input image and performs attacks in the combined space of image patches and input tokens. The effectiveness of SparseMA has been evaluated in comparison with three baseline methods across three distinct datasets. Compared to a baseline that conducts attacks separately in image and text spaces, their approach demonstrates superior performance.
In summary, all reviewers agreed that this paper focuses on black-box multimodal attacks, which is an interesting and timely research topic. The paper is well-written with a clear description of how the framework  and algorithmic are constructed. Their approach performs attacks in the joint space of image patches and text tokens, and achieves superior performance against a baseline that performs attacks separately in image space and text space. However, as three reviewers have pointed out, the effectiveness of the proposed method has not been sufficiently demonstrated, so ｍore experimental evaluations are needed to validate the contribution of this work. Specifically, the strong and significant differences with past adversarial works and the visibility of the perturbations, etc. There is a possibility that the paper will be accepted to findings.
Decision: Accept-Findings