Stop Reasoning! When Multimodal LLM with Chain-of-Thought Reasoning Meets Adversarial Image
Authors: Zefeng Wang
Zhen Han
Shuo Chen
Fan Xue
Zifeng Ding
Xun Xiao
Volker Tresp
Philip Torr
Jindong Gu
Venue: COLM (2024)
Keywords: Multimodal LLMs, Adversarial Robustness, Vision Language Models, Chain-of-thought
TL;DR: We explored the influence of Chain-of-Thought on Multimodal-LLM's robustness, and proposed a novel effective attack method "stop-reasoning attack". And, the generated rationales open a window for revealing the reason the model has wrong predictions.

Abstract: Multimodal LLMs (MLLMs) with a great ability of text and image under- standing have received great attention. To achieve better reasoning with MLLMs, Chain-of-Thought (CoT) reasoning has been widely explored, which further promotes MLLMs’ explainability by giving intermediate reasoning steps. Despite the strong power demonstrated by MLLMs in multimodal reasoning, recent studies show that MLLMs still suffer from ad- versarial images. This raises the following open questions: Does CoT also enhance the adversarial robustness of MLLMs? What do the intermediate reasoning steps of CoT entail under adversarial attacks? To answer these questions, we first generalize existing attacks to CoT-based inferences by attacking the two main components, i.e., rationale and answer. We find that CoT indeed improves MLLMs’ adversarial robustness against the existing attack methods by leveraging the multi-step reasoning process, but not substantially. Based on our findings, we further propose a novel attack method, termed as stop-reasoning attack, that attacks the model while by- passing the CoT reasoning process. Experiments on three MLLMs and two visual reasoning datasets verify the effectiveness of our proposed method. We show that stop-reasoning attack can result in misled predictions and outperform baseline attacks by a significant margin.

Fan Xue Published as a conference paper at COLM 2024
Stop Reasoning! When Multimodal LLM with Chain-of- Thought Reasoning Meets Adversarial Image
Zefeng Wang"
Zhen Han" LMU Munich
Shuo Chen LMU Munich
Technical University of Munich
Zifeng Ding LMU Munich
Xun Xiao Huawei Technologies
Technical University of Munich
Volker Tresp
Philip Torr University of Oxford
Jindong Gu University of Oxford
LMU Munich
Abstract
Multimodal LLMs (MLLMs) with a great ability of text and image under- standing have received great attention. To achieve better reasoning with MLLMs, Shain-of-Thought (CoT)) reasoning has been widely explored, which further promotes MLLMs' explainability by giving intermediate reasoning steps. Despite the strong power demonstrated by MLLMs in multimodal reasoning, recent studies show that MLLMs still suffer from ad- versarial images. This raises the following open questions: Does CoT also enhance the adversarial robustness of MLLMs? What do the intermediate reasoning steps of CoT entail under adversarial attacks? To answer these questions, first generalize existing attacks to CoT-basee inferences by attacking the two main components, i.e., rationale and answer. We find that CoT indeed improves MLLMs' adversarial robustness against the existing attack methods by leveraging the multi-step reasoning process, but not substantially. Based on our findings, we further propose a novel attack method, termed as stop-reasoning attack, that attacks the model while by- passing the CoT reasoning process. Experiments on three MLLMs and two visual reasoning datasets verify the effectiveness of our proposed method. We show that stop-reasoning attack can result in misled predictions and outperform baseline attacks by significant margin. The code is available here
Introduction
Previous research has shown that traditional vision models (e.g. image classifiers) are vulnerable to images with imperceptible perturbations, p exposing significant challenge in AI security (Szegedy et al. 2013; Goodfellow et al.) 2014). Recently, multimodal large language models (MLLMs) have demonstrated impressive competence in image understanding with the knowledge learned by LLMs, which arises the interest in studying whether MLLMs also show vulnerability to adversarial images. Some recent works confirm that MLLMs are also vulnerable to adversarial images with significant performance drops (Zhao et al., 2023 Bailey et al. 2023 Luo et al. 2024), showing the importance in studying the adversarial robustness of MLLMs.
To improve MLLM's performance in understanding images with complex content, Chain- of-Thought (CoT) reasoning has been explored in MLLMS (Lu et al. 2022 Zhang et al. 2023 He et al. 2023). CoT reasoning generates intermediate reasoning steps, known rationale, betore predicting the answer. This approach not only improves models' inference
"Equal contribution.
I 'Corresponding author: mooonnggu@@ottoook.com

---

Published as a conference paper at COLM 2024
Q: What is in the dlass of the Image?
Without Attack
The image shows a close-up giant panda eating bamboo... The class of the image is (B) giant panda.
howler mankey, (B) giant panda.
(C) bison, (D) chow...
image depicts cartoon monkey, looking up the camera with smile on its face. The monkey wearing sunglasses
Anewer Attack
cigar its mouth. The image class is (A) howler monkey
The image is cartoon Illustration a baby panda bear. panda big. round eyes appearance. it wearing little hat on its heed and has e small, toy fish its mouth. The class of the image howler monkey.
cute, fluffy
Rationale Attack
Stop
Reasoning
The class of the image is (A) howler monkey.
Attack
User
Agent
Figure 1: Given adversarial images, answer attack and rationale attack make an MLLM output an explanation for the incorrect predictions with CoT The phrases highlighted with red are found to inaccurately depict the actual facts. Apart from these two attacks, stop-reasoning attack shows the ability to restrain the reasoning process and make an MLLM output an incorrect answer even if the model prompted to leverage the CoT explicitly.
power but also introduces explainability, which is essential in critical domains such as clinical decision-making (Singhal et al. 2022). Nevertheless, the performance of CoT-based inference in MLLMs when facing adversarial images is still not fully investigated. In this work, we primarily explore the following questions:
Does CoT enhance the adversarial robustness of MILM?
What do the intermediate reasoning steps of CoT entail under adversarial attacks?
Since CoT--ased inference consists of two parts, i.e., rationale and final answer, we investi- gate the adversarial robustness of MLLMs by attacking both of them. First, two existing attacks are generalized to MLLMs with CoT, i.e., answer attack and rationale attack. Answer attack attacks only the extracted choice letter in the answer, e-g,, the "B" character in Figure ① which is suitable for both MLLMs with or without COT. The other attack, rationale allack, attacks not only the choice letter the answer but also the preceding rationale (Figure rationale attack). We find that models employing CoT tend to demonstrate considerably higher robustness under both answer and rationale atfacks compared with models without COT.
Based on this observation, we further devise new attacking method called stop-reasoning attack. Stop-reasoning attack aims to interrupt the reasoning process and force the model to directly answer the question even with an explicit requirement of CoT in the prompt. Meanwhile, the choice letter is also attacked, leading the model to predict an incorrect answer (as shown in Figure [sto-rresoonnng attack). In this way, the enhancement brought by CoT is limited, making MLLMs more vulnerable to adversarial attacks.
Furthermore, with the existing two attacks, i.e., answer atfack and rationale attack, the CoT mechanism elucidates the model's intermediate reasoning steps, which opens window for us to understand the reason for an incorrect answer when encountering adversarial images. As shown in Figure with answer attack, even though only the choice is attacked ("B"= "A"), the rationale changes correspondingly and reveals the reason: the panda with black eyes misidentified as a monkey with sunglasses. For rationale allack, although the panda is correctly recognized in the rationale, the other wrong information in the rationale influences the answer and leads to a wrong answer.
We conduct experiments with MiniGPT4 (huu et al. 2023
OpenFlamingo Awadalla
et al. 2023), and LLaVA (Liu et al. 2023a) as the representatives of victim MLLMs on two visual question answering datasets that require understanding
complex images, i.e, A-OKVQA (Schwenk et al., 2002)) and ScienceQA (Lu et al.) 2022). Experimental results demonstrate that MLLMs with CoT exhibit enhanced robustness compared to MLLMMs
2

---

Published as a conference paper at COLM 2024
without CoT across diverse datasets. Our stop-reasoning attack can restrain the CoT reasoning process even with the explicit prompt requiring CoT. It leads to a higher success rate, results in misled predictions, and outperforms baselines by sigmificant margin.
To summarize, we have the following contributions:
We study the influence of CoT the adversarial robustness of MLLMs by perform- ing attacks on the two core components of CoT, i.e, rational and answer.
We propose a novel attack method, i.e., stop-reasoning attack, for MLLMs with CoT, which effective at the most.
We show that the rationale opens a window for understanding the reason for an incorrect answer with an adversarial image.
Extensive experiments conducted on representative MILMSs and two datasets under the proposed attacking methods to justify our proposal.
2 Related work
2.1 Adversarial attacks
Deep learning models are known to be vulnerable to adversarial attacks (Szegedy et al. 2013
Goodfellow et al. 2014). Extensive previous studies have a primary focus on image recogni-
tion (Szegedy
2013 Goodfellow et al. 2014 Athalye et 2018 Carlini & Wagner 2017
Gu et al. 2021) and many well-known adversarial methods are proposed such as Projected
Gradient Descent (PGD) (Madry et al.) 2017), Fast Gradient Sign (ethod(FGSM) (Goodfel
low et al. 2014). These studies aim to mislead the models to generate wrong predictions
while only adding minimal and imperceptible perturbations to the images (Goodfellow et al.,
2014). Unicorn (Tu et al. (023) explores robustness against adversarial attacks and OOD generalization but does not address the influence of CoT reasoning on MILMs' robustness.
AVIBench (Zhang et al. 2024) and MM-Safety Bench (Liu et al. 2023b) provide frameworks for evaluating MLLMs robustness against adversarial attacks, but they do not focus on the
CoT reasoning process. Despite the effectiveness of these attacks, it is still hard to interpret the model behavior during the attacks and understand why the attacks could succeed (Gul & Tresp 2019: Li et al. 2022). Recent studies have also investigated the vulnerability of
large language models iZou et al., 2023 Kumar et al. 2023) and multimodal LLMs (Zhao)
et al. 2023 Gan et al. 2020 Gao et al.) 2024 Han et al. (023) under adversarial attacks. However, the adversarial robustness of multimodal LLMs with CoT reasoning ability is still under-explored. Since CoT reasoning reveals the model's decision process (Wei et al.) 2023), this reported intermediate process can serve as a good proxy for to understand the model behavior before and after the adversarial attacks, which additionally brings explainability. Different from previous studies, this work focuses on evaluating the adversarial robustness of MLLMs with CoT by designing effective attack methods and understanding why the model would behave under such adversarial attacks.
2.2 Chain-of-thought reasoning multimodal LLMs
CoT generates a series of intermediate logical reasoning steps and assists LLMs in thinking step by step before generating the final answer (Wei et al., 2023). CoT has been widely applied to LLMs (Wei et al. 2023 Kojima et al. 2023)
et al. 2022) and has significantly
improved the performance in various tasks, such as arithmetic problems (Wei et al. 2023 and symbolic reasoning (Wei et al.) 2023). Some studies have noticed that Col can bring extra robustness to the L.Ms (Wu et al. 2023) and have designed better CoT method
for better robustness (Wang et al. 2022) Recently, on MLLMs, various studies have also
Ms can bring superior performances as well (Lu et al. 2022 Zhang et al., 2023 He et al., 2023). However, the robustness of CoT on MLLMs against adversarial attacks has not been investigated. is still an open question whether CoTrrasoniig is beneficial, indifferent, or even harmful to the robustness of MLLMs under adversarial attacks. This study aims to first evaluate the adversarial robustness of CoT on MI.LMs and then understand how the attacks affect the model behavior.
shown that adopting COT on MLL
3

---

Published as a conference paper at COLM 2024
Clean Prediction
Quesiton: What the imagee class?
Rationale
Answer
MLLM
Answer
Choice: monkey, (8)ginat
Adversartal Pradiction
Rationale
(C) bison....
Pixel Addition
Attack
Answeer
Stop
Reasoning Attack
Figure 2: Pipeline. The dotted line indicates tes clean prediction with the original image. The solid line visualizes the attack pipeline in one iteration. The adversarial image Unto is built with the corresponding attack method.
3 Methodology
3.1 Threat models
This work examines the influence of the CoT reasoning process on MLLMs' adversarial robustness. We follow the principles introduced by Carlini et al. 2019) to define our adversary goals, adversarial capabilities, and adversary knowledge. The adversary goal is to cause the model to output wrong answer. Given the scenario that MLLMs with a prompt are applied to extract information from user images, the images are assumed to be manipulated by attacker to mislead MLLMs. Hence, restrict the adversarial capability to perturb the image in an imperceptible range and assume text prompts are unmodifiable. The restrictions on images
(1) where D(.) is the distance between images, Vorg is the original input image, Vado is the perturbed image, and e is a predefined boundary. As for the adversary knowledge, we assume the full knowledge of the model. Thus, solid attacks can be performed with the PGD (Madry et al. 2017) method for convincing results.
D(Vorg Vmdo) max Vorg - Undo S
3.2 Attack pipeline
We denote visual question answering (VQA) inference as (v,,9) + f, where f(-) represents an MLLM, is the input image, is input text formulated as question with its multiple answer choices, and is the output of the MILM. To make models use the CoT reasoning process, we add prompt generate a rationale with at least three sentences that can be used to infer the answer to the question. At last, infer the answer according to the question, the image, and the rationale.".
explicit instruction after the question and choices, e.g., "First,
As depicted Figure dotted line, both the textual question and corresponding image are fed to an MLLM to produce an initial clean prediction. This clean prediction, denoted as telemn serves as the basis for calculating losses according to three attack methods.
In one attack iteration (Figure 2 solid line), the MLLM takes both the perturbed image from the last attack iteration D Valojast and text as input and generates an adversarial output. Then, the new adversarial image Vadv,, built by leveraging different attack methods. In the first attack iteration, initial perturbation is performed before the image is fed into the model. The corresponding optimization problem can be defined as:
arg max D Paoor
C(f(Vador 4), (Vorg, 9))
(2)
the optimization problem can be solved with the PGD method.

---

Published as a conference paper at COLM 2024
Rationale
Bafore Attack
The woman is wearing goggles to protect her eyes from the sur's glare while sking down the mountain. The goggies provide UV protection, which
The woman is wearing goggles protect eyes sun's glare while skiing down mountain. goggles provide UV protection, which
The dog trying to catch frisbee. Answer: The answer is (B) frisbee.
After Attack
The answer is (B) protection.
The answer is (B) protection.
answer (D) ball.
Prediction
Answer
(a) Prediction with COT.
(b) Stop-Reasoning Attack.
Figure 3: Models output rationale and answer prediction without attack. After stop- reasoning allack, models output only the answer. (a) Prediction with COT. The complete prediction with CoT can be divided into two components: the rationale and the answer. (b) Stop-Reasoning Attack. After stop-rensoning attack, MLLMs skip the reasoning part and output the answer directly without rationale.
3.3 Generalized attacks
As shown in Figure 3a model inference with CoT reasoning provides an answer and rationale as its prediction output. We first generalize two existing attack methods to MLLMs with CoT, i.e. answer atlack and rationale atlack
Answer Attack. The answer attack focuses exclusively on attacking the answer part of the output, aiming to manipulate the model to infer wrong answer.
To alter the answer the prediction, a cross-entropy loss is computed between the generated answer and the ground truth. We extract the explicit answer choice to ensure that the loss computation focuses solely on the chosen response (please refer to Appendix B.2 Given the loss depends only on one character, the influence of the prediction's length is mitigated. The loss function defined as follows:
Lans (tatur felean) = (E ((fclean)
where g(-) is the answer extraction function, CE is the cross-entropy function. With escalat- ing the loss, models infer alternative answers, deviating from the correct responses.
Rationale Attack. Upon revealing the inferences of models with CoT under answer attack, an interesting observation surfaced: despite the disregard for the rationale in the attack's s design, the rationale part also changes in most cases. Building on this insight, the rationale attack is performed, which, in addition to targeting the answer part, also aims at modifying the rationale. We utilize the Kullback-Leibler (KL.) divergence to induce changes in the rationale because high KL divergence indicates a high information loss that fits the target, making the rationale more useless, while cross-entropy does not fit to measure the information entropy. Specifically, the loss function of rationale attack is as follows
Lish (fndor (llmm) = (L raa (clem) + Lams (tattr Felean)
where the aadp is the rationale in the adversarial output and the "claa is the rationale in the clean prediction. As the KI. divergence increases by perturbing image, the adversarial rationale diverges from the clean rationale. Hence, an alternative answer is predicted based on the altered rationale.
The results indicate that CoT slightly boosts the adversarial robustness of MLLMs against the aforementioned two existing methods and introduces the explainability of the incorrectness.
3.4 Stop-reasoning attack
Having explored the influence of CoT on the adversarial robustness of MLLMs, we found that the rationale is important for the inference process. A pertinent question arises: how will the model behave when the reasoning process is halted? Inspired by this question, we introduce stap-reagnning altack, method that targets the rationale to interrupt the model's
5

---

Published as a conference paper at COLM 2024
w/oCoT CoT
with CoT
Model
Dataset
w/o Attack
Answer Attack
Answer Attack
Rationale Attack
Stop Reasoning Attack
A-OKVQA
61,38
0,76
16.06
29.06
2.87
MiniGPT+-7B
ScienceQA
66.28
1.17
31.51
44.40
A-OKVQA
42.65
1.45
17.97
36.84
10.53
MiniGPT4-138
ScienceQA
63.64
13.75
33.78
45.69
A-OKVQA
34.80
3.52
11.14
10.79
4.95
Open-Flamingo
ScienceQA
34.55
34.73
28.87
A-OKVQA
92.21
0.74
36,22
21 .8
12.02
LLaVA
ScienceQA
83.17
1.13
56.96
49:27
22.39
Table 1: Inference accuracy (%) results of victim models. The samples achieve 100% accuracy with models employing the CoT reasoning process. Across diverse attacks, when models are prompted with CoT, stop-reasoning aftack emerges as the most effective method. For further studies and the experiment with ImageNet dataset, please refer to Appendix D
reasoning process. The objective of this attack is to compel the model to predict wrong answer directly without engaging in the intermediate reasoning process.
In the text input, predefined specific answer template, denoted faar, to prompt the model to output the answer in a uniform format. The upper part of Figure 3b/swwss that well-finetuned MILMs are able to produce answers following the prompt. Therefore, when the initial tokens align with the answer format tar, the model is forced to directly output the answers in the predefined format and bypass the reasoning process even if the model is prompted explicitly to leverage the CoT (refer to the lower part of Figure 3b).
Stop-reasontng attack formulates a cross-entropy loss to drive the model towards inferring the answer directly without reasoning process:
(stop(tader telean taar) - CE ttar) + Lans (fadur felean)
where tar is predefined answer template, e.g., "The answer is (){EOS)". By increasing the loss, MLLMs directly output the answer by aligning the initial tokens with the specified answer format and alter the answer into wrong one. This approach bypasses the reasoning process and thus, it eliminates the robustness boost introduced by COT. The results on all models and datasets reveal its effectiveness. Stop-rensoning attack outperforms the two existing methods by large margin and can be close to the results of MLLMs without CoT.
Experiments
4.1 Experimental settings
ScienceQA (Lu et al. 2022) and A-OKVQA (Schwenk et al., 2022) are used to evaluate the impact of the CoT reasoning process on the adversarial robustness of MLLMs, where both datasets comprise multiple-choice questions and rationales and require understanding on complex images. ScienceQA is sourced from elementary and high school science curricula and includes reasoning tasks. A-OKVQA requires commonsense reasoning about the depicted scene in the image and is known as a prevalent choice for VQA reasoning tasks. We perform attacks on data samples that are correctly answered by MLLMs with CoT (need be attacked (correctly answered) and can be attacked with rationale and stop-reasoning attacks (with Col)).
Datasets.
Victim Models. Three representative MLLMs are used our experiments, ie., MiniGIT4 (Zhu et al. 2023), OpenFlamingo Awadalla et al. 2023) and LLaVA (Liu et al. 2023a). Commercial MLLMs like GPT-4 (OpenAl et al. 2023), which operate as black-box products, excluded from the experiments because the first-order gradients for per- turbation are inaccessible. Note that MiniGPT4 and OpenFlamingo can infer with CoT without fine-tuning, while LLaVA initially lacks the CoT capability. LLaVA acquires the

---

Published as a conference paper at COLM 2024
CoT capability through fine-tuning with CEQA (ai et al. 2023). In our work, experi- ment on MiniGPT4-7B, OpenFlamingo-99. and LL For detailed parameters and experiment settings, please refer to Appendix
How does CoT influence the robustness MLLMs?
In this section, we present the evaluation results of the three victim models under the three proposed attacks to answer the following two questions:
Does the CoT reasoning bring extra robustness to MLLMs against adversarial images?
From the results of answer attack and rationale attack in Table the CoT brings marginal robustness boost against the two existing attacks.
Is there any specific attack targeting MLLMs with CoT that is effective? Comparisons in Table show that slop-reasoning allack is the most effective for MLLMs with CoT.
We provide more diverse examples in Appendix where the consequences caused by the three attacks are illustrated.
4.2.1 CoT marginally enhances robustness only on existing attacks
As shown in Table without CoTT, the considered models exhibit high vulnerability. Under answer aftack, on the A-OKVQA dataset, the accuracy of MiniGPT4 without CoT drops to 0.76%, while its accuracy dataset, the accuracy answer atlack, while if with CoT, it can remain at 31.51% %. We observe the same trends for the ScienceQA and A-OKVQA datasets on Open and LLaVA and find that models' robustness is relatively boosted by using COT.
still remain at 16.06% with CoT. Similarly, on the ScienceQA MiniGPT4 drops to 1.17% when answering without CoT under
MiniGPT4
OpenFlamingo
LLaVA
Changed
100
84.25
97.89
Not Changed
0
15.75
2.11
Table 2:
Distribution (%) of rationale changes. When answer attack succeeds on MLLMs with CoT, although answer attack specifically targets on the final answer, a majority of samples exhibit altered rationales.
After reviewing the examples, an important observation noted that the majority of samples suffering successful answer attacks exhibit altered rationales, though answer attack does not aim at the rationale part (Table 2). This implies that attacking a model with CoT requires changing both the answer and rationale parts.
The person is wearing a white jacket because he a chee..nswer: The answer (D) chef.
The person is wearing white jacket because he is a nuswer:nswer: The answer (A) nurse.
The books located the shaives because the books are the eheives. Answer: The answer (A) books.
The shelves located the library.
and didss are located the lbbrar. Answer: The answer (D) food.
Before Altack
After Atlack
Before Allack
Aller Allask
(a) Key change visualization.
(b) Trivial change visualization.
Figure 4: Rationale with key changes and trivial changes after attacks. (a) Key change visualization. The replication of the answer serves the key information to infer the answer from the rationale. After answer attack, the keyword in the rationale is also altered, even though the attack exclusively targets on the answer ("D" + "A"). (b) Trivial change visualization. The replication of the answer is the key information to infer the answer from the rationale. After ansueer attack, the keyword is not changed (the word "books" not changed), while the other part of the rationale is changed.
Based on the observation above, rationale attack is performed. Rationale attack exhibits superior performance OpenFlamingo and LLaVA compared to answer attack (Table
7

---

Published as a conference paper at COLM 2024
with marginal improvement (56.96% to 49.27 49.27% on ScinceQA LLaVA, 11.14% to 10.79% on A-OKVQA on OpenFlamingo)) Conversely, on MiniGPT4, the rationale attack proves less effective than answer attack on both datasets (16.06% under answer attack against 29.06% under rationale attack).
To understand why rationale attack does not always work, we pick 100 samples of each victim model on A-OKVQA under rationale attack. We classify these samples into two categories according to their changes in rationale: key changes and trivial changes. key change refers to the modifications on words crucial for deducing a correct answer, as shown in Figure 4a trivial change (as illustrated with the example in Figure on the other hand, refers to those modifications on words that are non-crucial for deducing correct answer while leaving key information untouched.
= Kay Change
Trival Change No Change
40
32
300
25
Number
20
17
10 10
10
Falure MrnGPPT 14
Success MrnGPT4
Failure OpenFlamingo
Success Openi
Fallure LLaVA
Success
LLaVA
Figure 5: Classifications of different types of changes made to rationale in three victim mod- els under rational attack (based on 100 Samples Sapples/Moon Modell. The groups "Failure" and "Success" indicate whether the attack failed or succeeded. "Failure" indicates an unsuccessful attack where the model's prediction remains correct, while "Success" denotes a successful attack resulting in change from correct to an incorrect prediction.
Figure glives statistical comparisons of the respective numbers of different types of changes made via rationale altack to the three victim models. The classifications indicate that success- fully attacked samples under rationale attack are often associated with significant modifica- tions key information within the rationale. Conversely, samples lacking altered rationales or featuring only minor adjustments tend to preserve their correct answers. This suggests the critical role of key information in influencing the inference of the final answer. However, precisely identifying the crucial information is hard, and modifying it efficiently is even more difficult. To this end, rationale aflack can only slightly enhance the attack performance in comparison with answer attack, and our results further support the finding that CoT improves adversarial robustness against generalized attack methods.
4.2.2 Stop-reasoning attack's effectiveness
Given the ineffectiveness of the answer and rationale atfacks, we introduce stop-reasoning attack halt the model's reasoning. The results demonstrate that stop-reasoning attack out- performs both other attacks (11.20% against 31.51% % and 44.40% on SincecQA on MiniGPT4, 4.95% against 11.1 11.14% and 10.79% on A-OKVQA on Open-Flamingo, and 12.02% against 36.22% and 21.88% on A-OKVQA on LLaVA). It even approaches the performance observed when attacking models without CoT (2.87% against 0.76% on A-OKVQA on MiniGPT4), indicating its remarkable potency in mitigating the additional robustness introduced by the CoT reasoning process. Biguueelllusttteeee an example where both rationale and answer attacks fail, and only stop-reasoning attack succeeds. In this work, simple outcheck defense does not work, resulting high false positive rate, given the fact that a simple image does not require COT.
To understand the effectiveness stop-reasoning, we examine the results of stop-reasoning attack and observe that after the attack, the model outputs the answer directly without
8

---

Published as a conference paper at COLM 2024
Q: What the light the room?
light source laptop screen.
Atlack
Answer: The answer is (D) laptop screen.
C: (A) lamp, (B) window, (C) flashight, (D) laplop screen
The light source is a fleshight, which is a small
Choice
light
Answer: The answer (D) laptop screen.
Rationale
The light source is flashight.
Attack
Answer: The answer (D) laptop screen
Stop
Reasoning
The answer is (B) window.
User
Agent
Figure 6: Example of all attacks. Slop-reasoning atlack is potent. At the top, it shows the callouts of user with an input image and their associated textual questions. The four callouts below are the answers from the MLLM under each type of attack. Only stop-reasoning attack achieves the goal of failing the model by providing a wrong answer (highlighted in red color).
leveraging CoT, aligning with the fundamental concept of stop-reasoning attack - aiming to halt the CoT reasoning process (Figure and more examples in Appendix (F). When slop-reasoning atlack succeeds, the model disregards the prompt's CoT reasoning process requirement and directly infers the answer.
As revealed in Section 4.2.1 the extra robustness boost is intricately tied to the generated rationale. If this CoT reasoning process is halted by stop-reasoning attack, the additional robustness generated during the CoT reasoning process will be diminished as well. Thus, achieving the adversary's 's goal becomes comparatively easier.
Concerning the performance gap between the stop-reasoning attack on MLLMs with CoT and the ansuer altack on MLLMs without COT. The difference might be caused by two factors: Firstly, the prompts for CoT and non-CoT scenarios are naturally different. Explicit instructions are added to prompt the MLLMs to leverage the COT. In comparison, the MLLMs are prompted to infer the answer directly under "w/o CoT" scenarios. Secondly, the stop-reasoning attack cannot always stop MLLMs from performing reasoning. We observed that not all samples successfully skipped the reasoning phase after the stop-reasoning attack (e.g,, when evaluating the stop-reasoming attack with the MiniGPT4 model on the A-OKVQA dataset, the model did not skip the reasoning phase for 5.35% samples).
The image depicts a giant panda sitting on tree branch, looking downwards. The panda has black fur with white patches on its ears, face, and chest. It has long, curved claw on each paw. The class the image giant panda.
The image depicts a giant panda sitting on tree branch lush green forest. The panda black and white with a distinctive black patch on its forehead.
The class of the image giant panda.
The image is black and white photograph bison. The bison large and has shaggy coal with long mane. It is standing looking around with its head tilled. The class of the image is bison.
The image shows black dog lying on its back a grassy field. The dog is wearing collar and has its paws in the air. There are trees and mountains in the background.
center the enclosure,
The class of the image is chow.
(a) CoT brings explainability for answer attack.
(b) CoT brings explainability for rationale attack.
Figure 7: Comparison of CoT under different attacks. (a) CoT brings explainability under answer attack. On the top, monkey is falsely recognized a panda. On the bottom, panda falsely recognized as a bison. (b) CoT brings explainability under rationale attack. On the top, monkey is falsely recognized as panda. On the bottom, bison is falsely recognized as a chow.

---

Published as a conference paper at COLM 2024
4.3
What does the rationale entail under adversarial attacks?
Although CoT brings marginal robustness to MLLMs against existing attacks, MLLMs are still vulnerable adversarial images, similar traditional vision models. When traditional vision models make inferences, e.g., on classification tasks, our understanding is confined to the correctness of the answer. Delving deeper into the model's reasoning process and answering the question of why the model infers wrong answer with an adversarial image is difficult. In comparison, when MLLMs perform inference with CoT reasoning, it opens window into the intermediate reasoning steps that models employ to derive the final answer. The intermediate reasoning steps (rationale part) generated by the CoT reasoning process provide insights and potentially reveal the reasoning process of the MLLMs.
To look deeper into the explainability introduced by CoT, we conducted image classification tasks ImageNet (Russakovsky al. 2015). These tasks involved constructing multi- choice questions by extracting subsets from ImageNet (please refer to Appendix C for selected classes). We provide two example pairs to illustrate the rationale's 's changes under answer attack and rationale attack.
Figure 7a illustrates CoT inference under answer atlack. In the upper example, CoT erro- neously interprets the partial color of the monkey as white, resulting in the misclassification of the monkey as a panda. In the bottom example, the rationale falsely asserts that the black-and-whiie patterns on the panda's 's body resemble the black-white picture of a bison. This misconception leads to the incorrect inference of a bison. Figure 7bb displays examples under rationale attack. In the upper example, the rationale incorrectly states that the black forehead is distinctive black patch, leading the model to inaccurately classify the image as panda instead of monkey. In the bottom example, the horn of a bison is misinterpreted as a collar, resulting in the false classification of the bison as a chow.
Conclusion
In this paper, we fully investigated the impact of CoT on the robustness of MLLMs. Specifi- cally, we introduced stop-reasoning attack, a novel attack method tailored for MLLMs with COT. Our findings reveal that CoT can slightly enhance the robustness MLLMs against answer atfack and rationale attack. This extra robustness is attributed to the complexity of changing precisely the key information in the rationale part. For stop-reasoning attack, our test results show that MLLMs with CoT still suffer and the expected extra robustness is eliminated. At last, examples are provided to reveal the changes in CoT when MLLMs infer wrong answers with adversarial images.
Acknowledgements
This paper is supported by the DAAD programme Konrad Zuse Schools of Excellence in Artificial Intelligence, sponsored by the Federal Ministry of Education and Research.
References
Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples. In International conference on machine learning, pp. 274-283. PMLR, 2018
Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, Jenia Jitsev, Simon Kornblith, Pang Wei Koh, Gabriel Ilharco, Mitchell Wortsman, and Ludwig Schmidt. OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Models, 01 01390 arXiv:2308.01390 [cs].
August 2023. URL
Jiaxin Bai, Xin Liu, Weiqi Wang, Chen Luo, and Yangqiu Song. Complex query answer- ing on eventuality knowledge graph with implicit logical constraints. arXio preprint ar MrXi::2055.19066 2023.
10

---

Published as a conference paper at COLM 2024
Luke Bailey, Euan Ong, Stuart Russell, and Scott Emmons. Image hijacking: Adversarial images can control generative models at runtime. arv preprint ar Stt::23090002336 2023,
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 ieee symposium on security and privacy (sp), pp. 39-79 leee, 2017.
Nicholas Carlini, Anish Athalye, Nicolas Papernot, Wieland Brendel, Jonas Rauber, Dim- itris Tsipras, Ian Goodfellow, Aleksander Madry, and Alexey Kurakin. On Evaluat- ing Adversarial Robustness, February 2019. URL http::/arxww.orgg/aaaa.. wtwp://gg//gg...o... 06705 arXiv:1902.06705 [cs, stat].
Zhe Gan, Yen-Chun Chen, Linjie Li, Chen Zhu, Yu Cheng, and Jingjing Liu. Large-scale adversarial training for vision-and-language representation learning. Advances in Neural Information Processing Systems, 33:6616 6-6628, 2020.
Kuofeng Gao, Yang Bai, Jindong Gu, Shu-Tao Xia, Philip Torr, Zhifeng Li, and Wei Liu. Inducing high energy-latency of large vision-language models with verbose images. arviv preprint Xxx::2401.11170, 2024.
lan Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. Xrv preprint arXio:1412.6572, 2014.
Jindong Gu and Volker Tresp. Saliency methods for explaining adversarial attacks. Xiv preprint ar Xiix:1908.0841.1 2019.
Jindong Gu, Baoyuan Wu, and Volker Tresp. Effective and efficient vote attack on capsule networks. Xrx preprint ar xi1o:21022.1005, 2021.
Dongchen Han, Xiaojun Jia, Yang Bai, Jindong Gu, Yang Liu, and Xiaochun Cao. Ot-attack: Enhancing adversarial transferability of vision-language models via optimal transport optimization. ar Xiv preprint arXit:2312.04403 2023.
Liqi He, Zuchao Li, Xiantao Cai, and Ping Wang. Multi-modal latent space learning for chain-of-thought reasoning in language models. arXiv preprint xxXio:2312.08762, 2023.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large Language Models are Zero-Shot Reasoners, January 2023. /Rtp:/ttp//r abs/2205. 11916 arXiv:2205.11911 [cs].
Aounon Kumar, Chirag Agarwal, Suraj Srinivas, Soheil Feizi, and Hima Lakkaraju. Certify- ing I1m safety against adversarial prompting. ar preprint 7Xiv:2309.02705 2023.
Xuhong Li, Haoyi Xiong, Xingjian Li, Xuanyu Wu, Xiao Zhang, Liu, Jiang Bian, and Dejing Dou. Interpretable deep learning: Interpretation, interpretability, trustworthiness, and beyond. Knowledge and Information Systems, 64(12):3197-3233 3234, 2022.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved Baselines with Visual Instruction Tuning, October 2023a. URL arXiv:2310.03744 [cs].
Xin Liu, Yichen Zhu, Jindong Gu, Yunshi Lan, Chao Yang, and Yu Qiao. Mm--afetybench: A benchmark for safety evaluation of multimodal large language models, 2023b.
Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering, October 2022. URL http::///a/r org/ abs/2209. 09513 arXiv:2209.05513 [cs].
Haochen Luo, Jindong Gu, Fengyuan Liu, and Philip Torr. An image is worth 1000 lies: Transferability of adversarial images across prompts on vision-language models. In To appear in ICLR, 2024. URL
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. Xiv preprint arXit:1706.06083 2017.
11

---

Published as a conference paper at COLM 2024
OpenAI, :, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haim- ing Bao, Mo Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapire, Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brak- man, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Lukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Lukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, I'atricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob Mc- Grew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong MU, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reiichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cullen O'Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascaanooo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Šherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojeiech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. Gpt-4 technical report, 2023.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. Imagenet large scale visual recognition challenge, 2015.
Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A-OKVQA: A Benchmark for Visual Question Answering using World Knowl- edge, June 2022. URL http:///xxxiv.oorr//////////////
01718 2rXiv:2206.01718 [cs].
12

---

Published as a conference paper at COLM 2024
Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al. Large language models encode clinical knowledge. arXiv preprint arXio:2212.13138 2022.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian So
Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arv preprint arXiv:1312.6199, 2013.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshom, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, lliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open Foundation and Sane-Tuned Chat Models, July 2023. URL http:/ttttii...ee/// wtw.orggabb/b/oo/// 09288 AxXiv:2307.09255 [cs].
Haoqin Tu, Chenhang Cui, Zijun Wang, Yiyang Zhou, Bingchen Zhao, Junlin Han, Wangchunshu Zhou, Huaxiu Yao, and Cihang Xie. How many unicorns are in this image? safety evaluation benchmark for vision Ilms, 2023.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Repre- sentations, 2022.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-Thought Prompting Elicits Reasoning in Large Lan- guage Models, January 2023. Uttp://///arxi htttt::/aaxxiv.rgg/bbb////// [cs].
atgg/bbs/2211 11903 arXiv:2201.11903
Skyler Wu, Eric Meng Shen, Charumathi Badrinath, Jiaqi Ma, and Himabindu Lakkaraju. Analyzing chain-of-thouge prompting in large language models via gradient-based feature attributions. arviv preprint 1rX1v:2307.13339, 2023.
Hao Zhang, Wenqi Shao, Hong Liu, Yongqiang Ma, Ping Luo, Yu Qiao, and Kaipeng Zhang. Avibench: Towards evaluating the robustness of large vision-language model on adversarial visual-instructions, 2024.
Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. Automatic Chain of Thought Prompting in Large Language Models, October 2022. URL httphtt///aaxxi.org//aass 03493 arXiv:2210.034993 [cs].
Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. Multimodal Chain-of-Thought Reasoning in Language Models, February 2023. URL 00923 arXiv:2302.009) [cs].
http:://arxiv.org/bbbsttse
Yunqing Zhao, Tianyu Pang, Chao Du, Xiao Yang, Chongxuan Li, Ngai-Man Cheung, and Min Lin. On evaluating adversarial robustness of large vision-language models. arXiv preprint ArXiv:3005.1693 2023.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models, 10592 arXiv:2304.1059) [cs].
April 2023. http: http:tpppp//rxi...ogg/////////// wrg./bb/////////.//
Andy Zou, Zifan Wang, J Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15044 2023.
13

---

Published as a conference paper at COLM 2024
Appendix
A Attack settings
Across all attack scenarios, the perturbation constraint e is set to 16. The maximum number of attack iterations is capped at 200. To generate the adversarial output fudv we opt for the forward(-) function over the generate(-) function in MLLMs. This choice driven by the fact that the generate(-) function demands significantly much more time, rendering the attack impractical due prolonged running times across extensive iterations. The prediction is updated every 10 iterations to mitigate the gap between the forward(() method and the generatel.) method. In every attack test, all victim models use 0-shot prompt to output their final answer. Every attack method starts with a random perturbation on the image in the very first iteration, then follows its individual loss function and uses PGD method to generate
perturbed image for the next iteration. The attacks are performed on a single NVIDIA 40G A100 GPU or single NVIDIA 80G A100 GPU for 13B model. To measure the robustness of the MLLMs, employ accuracy the performance metric. Low accuracy indicates a low robustness.
B
Implementation details
B.1 Pseudo code
The algorithm for the entire pipeline is outlined in Algorithm 1 In this algorithm:
fgen(> represents the model's inference using the generate(-) method.
f-x(-) signifies the model's inference using the forward method.
the perturbation constraint.
The initial adversarial image is created by introducing Gaussian noise to the original image.
Regular updates the prediction are essential alleviate the performance gap between the forward(.) and generate(.) methods.
Algorithm Pipeline
Require:
original image Vorg, question boundary é, step a, maximum iteration prediction feleun = fgen (Vorg. 9)
initial adversarial image Vade
truncate adversarial image to fit D D(org. vadv) S
for = 1 to 1 do adversarial output lado = fno q, 4.
loss calculation with L(taduv ppred)
grad of Vado from loss
new adversarial image Vado = Vado + a sign(grad)
check and truncate D(Vorg vadv) S
if update prediction is true then
prediction felemn = fgen (Vorge 9) end if
if stop criteria satisfied then break end if
end for fado = fgen (Uadv, 9)
save Undo return Indv
14

---

Published as a conference paper at COLM 2024
B.2 Extract answer
To perform an exact attack, the model is prompted to answer the multiple-choice questions in specific form and explicitly show the answer choice. As shown in F only the choice letter in the answer sentence will be considered as the answer. The choice content and choices appearing in other sentences will not be accepted.
The man in The image does not show any signs sun or snow, which eliminates options (C) and (D). Therefore, the answer is (A) hail.
The image shows street with a building on the left side bicycle parked in front of it. There are trucks or airplanes visible the Image. Therefore, the answer (C) bicycle...
(a) Extract answer.
(b) Split rationale.
Figure 8: (a) Extract answer. Only the choice letter (green) in the answer sentence will be considered as the answer. Other choice letters or choice content (red) will be ignored. (b) Split rationale. Only the sentences (bold) before the answer extracted (green) will be contoured as the rationale.
B.3 Split rationale and answer
To perform the rationale attack, the rationale and answer parts in the output logit matrix should be split if the model answers the question with the CoT process. As the used LLMs are all generative models, it is not deterministic where the rationale is, where the answer is, and how long each is. So, the output logit matrix is decoded and split into sub-sentences first. As the model is imperfect and the instruction prompt is not strong enough, the model may not follow the instructions exactly. The inference may mix the answer and the rationale part. The part before the sentence the answer is extracted from is the rationale. As shown in Figure 8 (b), inferences can be roughly divided into two parts from the answer. The sentences from the answer are regarded the answer part of the model, even though there are some other sentences. The sentences before the answer belong to the rationale part. The corresponding logit matrix will be extracted.
B.4 Stop criteria
The general stop criteria shared in all attack scenarios is whether the inferred answer is wrong in the perturbation loop. The perturbation process will be stopped the answer wrong. Then, the perturbed image will be fed into the model again to infer the final answer with the generate(.) method. The sample won't be attacked again, regardless of the correctness of the final answer generated with the generate( ()) method. When the MLLM is under stop-reasoning attack, the stop criteria is combined with an additional stop check, which checks if the answer extracted from the first sentence.
B.5 Forward vs. Generate
In the context of language models, the forward(-) function often refers to the process of passing input data through the model to obtain predictions or activations. For LLMs used in MLLMs like Llama2 (Touvron et al. 2023), the forward(.)) function has the same length in output as the input. "The output token is the predicted next token to the input token at the same position. The generate(-) function generates output by iteratively using the forward(-) function. Specifically, in each iteration, only the last token, the next token to the whole input, is extracted and concatenated after the input sequence. The new sequence will be used as input in the next iteration until there is an end-of-sequence token [EOS]. To generate an output, the generate(.) function costs much more time than the forward(.)) ), if ground truth can be provided to the froward function, because the generate(.) function goes through the forward(-) function several times while the forward with ground truth needs only one iteration. However, it's important to note that to generate all tokens at once, the forward method requires a ground truth, and there a performance gap between the forward(() and the generate(.) functions.
15

---

Published as a conference paper at COLM 2024
As outlined in appendix [.1] adopt the clean prediction as the pseudo ground truth for the forward(-) method. As the perturbation progresses, the input image differs from the one used for the clean prediction. Consequently, the pseudo ground truth deviates from the actual ground truth, leading to a divergence in the adversarial output. To address the disparity between the real adversarial output and the actual adversarial output, the clean prediction should be updated with the latest adversarial image every several iterations.
C ImageNet subclasses
We create classification tasks by extracting 4, 8, and 16 classes. All the classes randomly picked from the dataset. The specific classes selected for each scenario are as follows:
4 classes: English setter, Persian cat, school bus, pineapple.
bison, howler monkey, hippopotamus, chow, giant panda, American Staffordshire terrier, Shetland sheepdog, Great Pyrenees.
8 classes:
16 classes:
piggy bank, street sign, bell cote, fountain pen, Windsor tie, volleyball, overskirt, sarong, purse, bolo tie, bib, parachute, sleeping bag, television, swimming trunks, measuring cup.
All tasks had uniform question: "What is the class of the image?"
D Further studies
D.1 What If CoT is not necessary for tasks?
We randomly picked several classes from the ImageNet dataset (Appendix C) Surprisingly, as indicated in Table 3 the tests with 8 classes show worse performances than the tests with 16 classes. The reason maybe some of the chosen classes are similar, and the similarity makes the classification task more complex, e.g., American Staffordshire terrier and Shetland sheepdog look similar. However, the conclusion improvement in performance brought about by CoT suggests that the rationale may not be essential for simple tasks. Although the two main results outlined in Section 4.2.1shhar commonalities, notable gap exists between the accuracy values of the ImageNet series and those of the A-OKVQA and ScienceQA datasets when models with CoT are subjected to the answer attack. This discrepancy can be attributed to the inherent complexity of VQA tasks compared to the straightforward classification tasks on the ImageNet dataset.
all tests consistent: the marginal
Further examination of the A-OKVQA and ScienceQA datasets reveals that the A-OKVQA dataset is relatively easier, as illustrated in Table This performance difference consistent across all three models. By comparing the accuracy of the classification task ImageNet with the VQA tasks on A-OKVQA and ScienceQA, a significant observation emerges: CoT has almost no impact the robustness of simple tasks.
w/o CoT
with CoT
# Classes
w/o Attack
Answer Attack
Answer Attack
Rationale Attack
Stop Reasoning Attack
85.34
0.00
4.19
4.71
0.52
82.04
0.00
1.06
0.00
0.00
16
74.32
0.00
3.32
2.42
0.30
Table 3: Inference accuracy (%) on ImageNet classification task. All samples are correctly inferred when inferring with CoT. # classes signifies the number of classes extracted from the ImageNet dataset for multi-choice classification tasks. Figure 9b reveals larger values lead to earlier convergence and better performance, but performance plateaus beyond certain threshold.
16

---

Published as a conference paper at COLM 2024
D.2 Ablation study adversarial capability
During the ablation study, the adversarial capability is changed by narrowing or expanding the limited boundary (e) to 2, 4, 8, 32 (as described in Section 3.1cioon3.1.. 3.1 Table presents results of with e set to 8. The results are consistent with Table 1
the M and Open-Fallmnn indicating that the CoT reasoning process enhances the robustness of MLLMs. Furthermore, Figure 9 shows the results of ScienceQA on MiniGPT4 with different c. Figure shows slop-reasoning atlack consistently outperforms rationale and answer attacks as increases.
w/o
Model
Dataset
w/o Attack
Answer Attack
Answer Attack
Rationale Attack
Stop Reasoning, Attack
A-OKVQA
61.38
17.59
30.98
MiniGPT4
ScienceQA
66.28
3.12
25.55
47 47.66
16.93
A-OKVQA
34.80
4.19
13.94
11.73
6.47
Open-Flamingo
ScienceQA
34.55
7.13
39.18
40.51
31.71
Table 4: Accuracy table with reduced adversarial capability.
70
Ralionale
apsilon
apailon
60
Stop Ressoning Atlack
Answer Atiack (wl CoT)
Answer Attack CoT)
70
60
epsikon 16
opsilion 52
epsikon
50
40
30
20
10
32
Epsiion c
max attack iterations
(a) Ablation epsilon on all attacks.
(b) Ablation stop-reasoning attack.
Figure 9: Ablation studies on adversarial capability e and maximum attack iterations on ScienceQA on MiniGPT4. (a) Accuracy under various attacks and e. (b) Accuracy under stop-reasoning attack with different
D.3 Targeted attack
Besides the aforementioned attacks, we also conducted targeted attack on MiniGPT4. When the MILM answers the question correctly, we randomly pick a wrong answer and use as the target. The input images are perturbed so that the MLLM answers the question with the picked target choice. shbls the success rate of the attacks. The results on both datasets confirm that the stop-reasoning attack achieves the highest mapping success rate on MLLMs with CoT, ie., 64.91% against 49.87% and 38.22% on ScienceQA, 62.84% against 61.19% and 52.23% on A-OKVQA.
CoT
Dataset
Answer Attack
Answer Attack
Rationale Attack
Stop Reasoning Attack
A-OKVQA
86.92
61.19
52.23
62.84
ScienceQA
85.27
49.87
Table 5; Targeted attack on MiniGPT4. Results indicate the success rate of the MLLM infers a predefined wrong choice with an adversarial image.
17

---

Published as a conference paper at COLM 2024
E
Image comparison
Figures 10 11 and provide visual representations of adversarial images generated for MiniGP 4, wppeeeaainnggee and LLaVA.
Examples
In the provided set of samples:
First 4 Samples (IDs: 10, 12, 40, 43 ): All attacks (answer attack, rationale attack, stop-reasoning attack) succeed.
Next 2 Samples (IDs: 23, 1024): Only the stop-reasoning attack succeeds.
Next Samples (IDs: 21, 51): Both answer attack and stop-reasoning attack succeed.
Last 2 Samples (IDs: 112, 207): Both rationale attack and stop-reasoning attack succeed.
Original imae
(wbb Call)
Figure 10: Image comparison of attacks on MiniGPT4. The figure saowsases the original and adversarial images generated during attacks on MiniGPT4. All the attacks succeed.
Original Image
Antovur Allack, (wil CoT)
Arsowe Allack (wit CuT)
Reiznall Alladi
Step Ressoning Allace
Figure 11: Image comparison of attacks on OpenFlamingo. The figure shows the original and adversarial images generated during attacks on OpenFlamingo. All the attacks succeed.
Original Image
Figure 12: Image comparison of attacks on LLaVA. The figure showcases the original and adversarial images generated during attacks on LLaVA. All the attacks succeed.
18

---

Published as a conference paper at COLM 2024
Qunstion:
What country do these planes belong to?
Choices: (A) united states, (B) garmany, (C) canada, (D) maxico
The planes are the Smithsonian
The planes are painted in the colors of
The planes are from the Franch Air Forne. (D) mexico.
The first plane is B-2 Spini, bomber, plane Eurofighter Typhoon, which is fighter jet.
Space Museum, which located in Washington, D.C. The pari Smilhsonisn
Canadian flag. question is asked in the context of millary base. Answer:
Institution, which is Tederal agency of the
Answer:: anower (A) united states.
Attack
Rationale Attakk
Reasoning Atlack
Figure 13: Sample id 10. All the three attacks succeed. The answer attack and the rationale attack alter the answer, while the model stopped before providing an answer under the stop-reasoning attack.
other?
Choices: (A) grapes, (B) apples, (C) lettuce, (D) radish
The question
The question asking about the fruit the salad. The green, and apples and grapes are red and only respectively. The
The question asking about the frui the salad.
The question is asking about the color of the fruit,
asking about the fruit the salad.
only
the grape. The grape cut in half and darker than the other only darker than the other fruit ls the grape.
which cut The banana is darker than the other fruil, which answer is (D) radiah.
Anewer: The answer is (C) lettuce.
black olive, Answer: The answer is (D) radiahi
Answer: The answer (D) radiss
(A)
(a) Attak
(5) Ancweor Attack
(C) Rationalc Altakk
(6) Stop Reasoning Attock
Figure 14: Sample id 12. All the three attacks succeed. The answers are changed under all three attacks.
19

---

Published as a conference paper at COLM 2024
type
Choioes: (A) dog. (B) figer, (C) cat, (D) lion
question
animal cow, which commonly used ferming, Answer The lion.
animal which commanly used farming, Answer The
a
asking about the type animal leash, only animal commonly seen on leash is dog Answerr The
which is type of equine.
(b)
10)
Stop Reasoning
Figure 15: Sample id 40. All the three attacks succeed. The answer attack and the rationale attack alter the answer, while the model stopped before providing an answer under the stop-reasoning attack.
Question: Which food Item the knife for? Choices: broad, (B) vagetables,
The kniee for nutting meat, which the only food itom that is not vegetable Answer; meat.
The knile for cutting vagetables. Answer: answer is (C) vegotablcs.
The knfie woffle,
fruit.
type of bread. Answer: The answer is (A) bread
Without Aftack
Antwer Atteck
(6) Rationale Attack
Reasoning Atteck
Figure 16: Sample id 43. All the three attacks succeed. The answers are changed under all three attacks.
20

---

Published as a conference paper at COLM 2024
What people
Choices: resort, (B) safari, (C) (D) tropical resort
The people are skiing, and the presence snow the further supports
The people are di tropical resort, and the Pyramid the backiground tropical resorts. Answer: The answer is (C) sii resort
The people are al tropical resort, and the palm and support answer is (C) ski resort:
answer is (D) Tropical resort)
that the resort a ski resort, Answer: The
What the occupation of the driving? ? Choices: waiter, (B) leacher
(0) Ansseor Attak
(0)
(c) Stop Reasoning Attack
Figure 17: Sample id 23. Only the stop-reasoning attack succeed. The answer attack and the rationale attack fail.
Question:
What room
Choices: (A) badroom, (B) living room, (C) hospital ward, (D) hotel
The question
The room is hospital ward. Answer: The hospital.
The room is hospital ward, and the presence hospital bed and room further support idea that the room is hospital ward. Answer: (D) hospilal.
The answer is (C) hospital ward:
asking about the the room. Answer: The answer is (D) hotel,
[n] Without
(bg Arteane Atback
(e) Retionale
(6) Stop Resenning Atteck
Figure 18: Sample id 1024. Only the stop-reasoning attack succeed. The answer attack and the rationale attack fail.
trying
Choices: (A) person, (B) frisboo, (C) kito, (D) ball
The dog trying to catch fribbe. (B) frisbes.
The dog trying to calich frisbeo, Answer: The is
The dog trying to catch frisbee. Answer: (B)
The anewer is ball
WithoutAtackk
Answee Attack
Rationale Attock
Atteck
Figure 19: Sample id 21. The answer attack and the stop-reasoning attack succeed, The rationale attack fails.
21

---

Published as a conference paper at COLM 2024
Question: In what envirnment
skateboards?
Choloes: (A) beach, (B) city (C) rural, suburban
The question asking about the environment which skaleboarding moet popular: The anawer is (B) city.
The question asking about the environment which skateboarding The anewer is (C) rural.
The question asking about the environment which skatehoardiig Iikey moet likey environment for skatteboarding ls question states the people wearing suils lies. Answer: The answer &s (B) city.
The answer is (C) rural.
330 Aitck
(b) Answerr Attack
(c) Rationale Attakk
Stop Reasoning Attack
Figure 20: Sample id 51. The answer attack and the stop-reasoning attack succeed. The rationale attack fails.
Question: Which century the item the holding invented in?
Choices: (B) twenty eighteenth
phone, which was invented in the twentieth century. Answer: answer (A)
camena, and the woman is photogrspher, The
phone,
phono, which was
which was invented the twenlieth century. Answer: (D)
invented in the bwentieth
was invented nineteenth century.
eighteenthh.
is (A) twertieth.
Answer: The answer is (C) tenth. Question:
Question: What is the accupation of
What occupation the picture? Choices: (A) waiter, (B) teacher, (C
driving?
Choices: (A) waiter,
(B) teacher. (C)
cashier, (D)
(a) Aitaku
(bs Answer Attack
(6) Rationalo Attack
(c) Stap Reasoning Attack
Figure 21:
Sample id 112. The rationale attack and the stop-reasoning attack succeed. The rationale attack fails.
Question:
What animal is next to the baby animal?
Choices: (A) roceter, chicken, (C) cow, (D)
The baby snimal next is oow. Answer: The answer (C) bow
The dog and baby pupoy: The dog not puppy, so the answer (C)
The baby snimal next is sheep. Answer: The answer is (B) chicken.
is (D) hen.
(e) Withred Atteck
2b) Ansieer Atbeck
(6) Retionale Abackk
Stop Resecning Alteck
Figure 22: Sample id 207. The rationale attack and the stop-reasoning attack succeed. The rationale attack fails.
22


Reviews:
[{"review_id": "cv9fRwMEx0", "reviewer": "Reviewer_hxRq", "summary": "This paper first found that the CoT technique can improve the adversarial robustness of MLLMs. Then they proposed a novel attack method that train visual noise using designed losses that are accosiated with CoT components. Experiments are conducted on two VQA datasets using three different MLLMs, and the resutls presented confim the effectiveness of the method.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed feedback and constructive suggestions.\n1. Generalization: \n- Regarding the dataset, our primary focus is to explore the influence of CoT on the adversarial robustness of MLLMs, where reasoning is necessary. The two chosen VQA datasets are representative of tasks that demand reasoning. In comparison, captioning tasks typically do not involve reasoning steps, which is why they were excluded. Moreover, since many tasks can be formulated as VQA task, our approach should remain generalizable even though we have only performed our method to two VQA tasks.\n- Concerning targeted attacks, we have conducted additional experiments on MiniGPT4 with pre-defined wrong answer. The results on both datasets confirm that the stop-reasoning attack achieves the highest mapping success rate on MLLMs with CoT, i.e., 64.91% (stop-reasoning attack) against 49.87% (answer attack) and 38.22% (rationale attack) on ScienceQA, 62.84% (stop-reasoning) against 61.19% (answer) and 52.23% (rationale) on A-OKVQA.\n2. Model Size: We ran experiments on MiniGPT4-Vicuna13B. The results are consistent with our findings presented in the paper, demonstrating that the stop-reasoning attack is the most effective. The detailed results are presented below:\n|| w/o CoT (acc%)|w/o CoT| w/ CoT (acc%)|w/ CoT|w/ CoT|\n|-|-|-|-|-|-|\n|Datasets|w/o Attack|Answer Attack|Answer Attack|Rational Attack|Stop Reasoning Attack|\n|A-OKVQA|42.65|1.45|17.97|36.84|10.53|\n|ScienceQA|63.64|13.75|33.78|45.69|30.89|\n3. Ablation Studies: More ablations on $\\epsilon$ and iteration were conducted, please see figure at [here](https://anonymous.4open.science/r/stop_reasoning_colm-BF9B). Ablation indicates that with different $\\epsilon$, our stop-reasoning attack is consistently better than rationale attack and answer attack on MLLMs with CoT. Ablation on iterations shows the convergence of the different attacks. \n4. Missing References: Thank you for pointing out the references. We will incorporate the suggested references into the related work. Specifically:\n- AVIBench (Zhang H, et al.) and MM-SafetyBench (Liu X, et al.) provide frameworks for evaluating MLLMs' robustness against adversarial attacks, but they do not focus on the CoT reasoning process.\n- Unicorn (Tu H, et al.) explores robustness against adversarial attacks and OOD generalization but does not address the influence of CoT reasoning on MLLMs' robustness.\n\nThank you again for your valuable feedback and helpful review.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The writing is straightforward and easy to follow.\n2. The method seems to be effective given the results presented.\n3. Discussion sections in the paper are insightful and the picked examples are vivid.", "reasons_to_reject": "1. Generalization: the paper only perform their method on two VQA task. On the one hand, can the method perform well on multimodal generation tasks like captioning? On the other hand, while your current attack goal is to only \"avoid the right answer\", can your method be applied to more strict tasks like \"generate the pre-defined wrong answer \" (e.g., the jailbreaking task)?\n2. Model Size: only MLLMs with 7B size LLM are applied , I'm wondering how's effectiveness of your stop-reasoning on larger MLLMs, such as models with 13B LLMs.\n3. Ablation of $\\epsilon$ and the training iteration: the pertubation budget $\\epsilon$ and training iteration are main parameters in adversarial learning domain. Maybe you can draw a picture about the attack success rate of your method under different $\\epsilon$ and/or training steps in the main part.\n4. Missing reference: several references that are highly related to your paper are missing [1-3].\n\n[1] Zhang H, Shao W, Liu H, et al. AVIBench: Towards Evaluating the Robustness of Large Vision-Language Model on Adversarial Visual-Instructions[J]. arXiv preprint arXiv:2403.09346, 2024.\n\n[2] Tu H, Cui C, Wang Z, et al. How many unicorns are in this image? a safety evaluation benchmark for vision llms[J]. arXiv preprint arXiv:2311.16101, 2023.\n\n[3] Liu X, Zhu Y, Lan Y, et al. Query-relevant images jailbreak large multi-modal models[J]. arXiv preprint arXiv:2311.17600, 2023.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "46TrUVOGyT", "reviewer": "Reviewer_VqF6", "summary": "This research focuses on multimodal language models (MLLMs) and introduces a novel adversarial attack method called \"stop-reasoning attack\" to investigate and assess the impact of CoT reasoning on the adversarial robustness of MLLMs. Experimental results demonstrate that CoT reasoning indeed enhances the robustness of MLLMs against existing adversarial attack methods.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback.\n\nResponse for \"Reasons To Reject\":\n\nRegarding the concern about the novelty of our method, we think there might be a misunderstanding. The primary aim of our paper is not to introduce a completely new attack method, but rather to investigate the impact of CoT reasoning on the adversarial robustness of MLLMs. We highlighted two main research questions in the introduction: (1) Does CoT enhance the adversarial robustness of MLLMs? (2) What do the intermediate reasoning steps of CoT entail under adversarial attacks?\n\nThe CoT reasoning process is known to improve MLLM performance on complex reasoning tasks, but its effect on robustness has not been thoroughly explored. In our study, we conducted three different attacks on MLLMs with CoT and one attack on MLLMs without CoT. Our results indicate that while CoT provides marginal robustness enhancements, these can be negated when the model is led to skip the reasoning phase through the stop-reasoning attack. Furthermore, our analysis of the rationale changes under different attacks demonstrates how CoT serves as a window to understand why MLLMs make incorrect answers when confronted with adversarial images.\n\nResponse for \"Questions To Authors\":\n\nConcerning your question about the performance gap between the stop-reasoning attack on MLLMs with CoT and the answer attack on MLLMs without CoT, we think this is not due to the additional optimization cost of the reasoning component. We ensured that the attack iterations were sufficiently long and the attacks converged in all experiments, ruling out optimization cost as the cause of performance differences.\n\nThe difference might be caused by two factors:\n1. The prompts for CoT and non-CoT scenarios are naturally different. We add explicit instructions such as \"first, generate a rationale that can be used to infer the answer to the question\" to prompt the MLLMs to leverage the CoT. In comparison, we ask MLLMs to infer the answer directly under \"w/o CoT\" scenarios.\n2. The stop-reasoning attack cannot always stop MLLMs from performing reasoning. We observed that not all samples successfully skipped the reasoning phase after the stop-reasoning attack (e.g., when evaluating the stop-reasoning attack with the MiniGPT4 model on the A-OKVQA dataset, the model did not skip the reasoning phase for 5.35% samples). \n\nThese two factors likely account for the performance gap between the two attacks.\n\nThank you once again for your valuable insights.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The purpose and approach of the attack are interesting.\n2. The writing of the paper is also quite clear.", "reasons_to_reject": "1. I am concerned about the novelty of the method. The overall pipeline of the attack is similar to other methods, where noise is used to perturb images so that they are mapped to incorrect target locations in the feature space after passing through the MLLM image encoder.", "questions_to_authors": "1. I have a question about the difference between \"stop reasoning attack\" and simply not using CoT and then attacking the answer. I believe they both ultimately attack the answer. What might be the performance difference between the \"Answer Attack\" column in Table 1 for \"w/o CoT\" and the \"Stop Reasoning Attack\" column for \"with CoT\"? Could this difference in performance be due to the need for additional optimization of a reasoning component in the stop reasoning attack", "extra_scores": {}}, {"review_id": "MWMUUqC1SU", "reviewer": "Reviewer_3eKb", "summary": "This paper considers the intersection of adversarial images + chain-of-thought prompting.\n\nThe setup involves a model answering multiple choice questions given an image. The model tries to give reasoning first before spitting out a letter from A to E. There are several attacks:\n* answer attack, in making the model spit out the wrong answer\n* rationale attack, in making the model point out incorrect observations\n* stop reasoning attack, in making the model not do chain-of-thought at all\n\nthe authors consider various image+LLMs like MiniGPT4, OpenFlamingo, and LLaVa; then A-OKVQA and ScienceQA. the results show that these various attacks can be successful.\n\n---\nedit post rebuttal: still agree to accept this paper, I am less concerned about novelty and don't think one negative review saying that should be reason to reject. so overall should be reasonable to accept this paper given all the reviews.", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive feedback and acceptance recommendation.\n\nWe are very happy that you highlighted the contributions in the paper. The results in the paper reveal that CoT marginally enhances the robustness of MLLMs, and the stop-reasoning attack, which can bypass the reasoning steps, is the most effective attack method compared with answer attack and rationale attack on MLLMs with CoT.\n\nWe are thrilled to hear that our work has sparked interest beyond the security domain. We also hope that our work can inspire further research across various fields, leveraging the insights we have uncovered. We will continue our investigations in this area to contribute to the broader understanding of CoT and MLLMs' robustness and vulnerabilities.\n\nRegarding the point about the assumed knowledge of the model, the reason is that our primary focus is not solely on the robustness against adversarial images but also on how the CoT reasoning process influences MLLMs' robustness. To better understand this influence, we assumed that the adversary possesses knowledge about the MLLMs.\n\nThank you once again for your thoughtful review and encouragement.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This seems like an interesting paper to this reviewer. It extends the domain of adversarial inputs to the multiple-choice and CoT regime, both of which are very popular in today's LLM era. It also suggests\n* CoT provides a marginal robustness boost\n* Stop-reasoning is the most effective attack (so it is possible to perturb the images such that models stop reasoning)\n\nthese seem pretty useful for the community to know!", "reasons_to_reject": "Possibly, this is a bit of a narrow attack that assumes a lot of knowledge about the model - the model's parameters, the domain, etc. To this reviewer though this is interesting even outside of the security domain, so I still recommend acceptance.", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper considers the intersection of adversarial images and chain-of-thought prompting in multimodal LLMs. After author rebuttal, it received scores of 678. The authors have done a good job of rebuttal, and all the reviewers are happy about the paper after rebuttal, commenting that (1) the writing is clear and easy to follow, (2) the method seems to be effective given the results presented, and (3) it extends the domain of adversarial inputs to the multiple-choice and CoT regime in the setting of multimodal LLMs. The additional results added during rebuttal is also helpful to address reviewers' concerns. Therefore, the AC would like to recommend acceptance of the paper.
Decision: Accept