Stop Reasoning! When Multimodal LLM with Chain-of-Thought Reasoning Meets Adversarial Image

Authors: zefeng  wang
zhen  han
shuo  chen
lmu  munich
fan  xue
xun  xiao
volker  tresp
philip  torr
jindong  gu
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-21 17:09:39

Abstract: Multimodal LLMs (MLLMs) with a great ability of text and image under- standing have received great attention. To achieve better reasoning with MLLMs, Chain-of-Thought (CoT) reasoning has been widely explored, which further promotes MLLMs’ explainability by giving intermediate reasoning steps. Despite the strong power demonstrated by MLLMs in multimodal reasoning, recent studies show that MLLMs still suffer from ad- versarial images. This raises the following open questions: Does CoT also enhance the adversarial robustness of MLLMs? What do the intermediate reasoning steps of CoT entail under adversarial attacks? To answer these questions, we first generalize existing attacks to CoT-based inferences by attacking the two main components, i.e., rationale and answer. We find that CoT indeed improves MLLMs’ adversarial robustness against the existing attack methods by leveraging the multi-step reasoning process, but not substantially. Based on our findings, we further propose a novel attack method, termed as stop-reasoning attack, that attacks the model while by- passing the CoT reasoning process. Experiments on three MLLMs and two visual reasoning datasets verify the effectiveness of our proposed method. We show that stop-reasoning attack can result in misled predictions and outperform baseline attacks by a significant margin.

Main paper: 
 introduction: 
 Previous research has shown that traditional vision models (e.g. image classifiers) are vulnerable to images with imperceptible perturbations, exposing a significant challenge in AI security (Szegedy et al., 2013; Goodfellow et al., 2014) . Recently, multimodal large language models (MLLMs) have demonstrated impressive competence in image understanding with the knowledge learned by LLMs, which arises the interest in studying whether MLLMs also show vulnerability to adversarial images. Some recent works confirm that MLLMs are also vulnerable to adversarial images with significant performance drops (Zhao et al., 2023; Bailey et al., 2023; Luo et al., 2024) , showing the importance in studying the adversarial robustness of MLLMs.To improve MLLM's performance in understanding images with complex content, Chainof-Thought (CoT) reasoning has been explored in MLLMs (Lu et al., 2022; Zhang et al., 2023; He et al., 2023) . CoT reasoning generates intermediate reasoning steps, known as rationale, before predicting the answer. This approach not only improves models' inference The image is a cartoon illustration of a baby panda bear. The panda has big, round eyes and a cute, fluffy appearance. It is wearing a little hat on its head and has a small, toy fish in its mouth.The class of the image is (A) howler monkey.The image shows a close-up of a giant panda bear eating bamboo...The class of the image is (B) giant panda.Answer Attack Rationale AttackFigure 1 : Given adversarial images, answer attack and rationale attack make an MLLM output an explanation for the incorrect predictions with CoT . The phrases highlighted with red are found to inaccurately depict the actual facts. Apart from these two attacks, stop-reasoning attack shows the ability to restrain the reasoning process and make an MLLM output an incorrect answer even if the model is prompted to leverage the CoT explicitly.power but also introduces explainability, which is essential in critical domains such as clinical decision-making (Singhal et al., 2022) . Nevertheless, the performance of CoT-based inference in MLLMs when facing adversarial images is still not fully investigated. In this work, we primarily explore the following questions:• Does CoT enhance the adversarial robustness of MLLM?• What do the intermediate reasoning steps of CoT entail under adversarial attacks?Since CoT-based inference consists of two parts, i.e., rationale and final answer, we investigate the adversarial robustness of MLLMs by attacking both of them. First, two existing attacks are generalized to MLLMs with CoT, i.e., answer attack and rationale attack. Answer attack attacks only the extracted choice letter in the answer, e.g., the "B" character in Figure 1 , which is suitable for both MLLMs with or without CoT. The other attack, rationale attack, attacks not only the choice letter in the answer but also the preceding rationale (Figure 1 rationale attack). We find that models employing CoT tend to demonstrate considerably higher robustness under both answer and rationale attacks compared with models without CoT.Based on this observation, we further devise a new attacking method called stop-reasoning attack. Stop-reasoning attack aims to interrupt the reasoning process and force the model to directly answer the question even with an explicit requirement of CoT in the prompt. Meanwhile, the choice letter is also attacked, leading the model to predict an incorrect answer (as shown in Figure 1 stop-reasoning attack). In this way, the enhancement brought by CoT is limited, making MLLMs more vulnerable to adversarial attacks.Furthermore, with the existing two attacks, i.e., answer attack and rationale attack, the CoT mechanism elucidates the model's intermediate reasoning steps, which opens a window for us to understand the reason for an incorrect answer when encountering adversarial images. As shown in Figure 1 , with answer attack, even though only the choice is attacked ("B" → "A"), the rationale changes correspondingly and reveals the reason: the panda with black eyes is misidentified as a monkey with sunglasses. For rationale attack, although the panda is correctly recognized in the rationale, the other wrong information in the rationale influences the answer and leads to a wrong answer.without CoT across diverse datasets. Our stop-reasoning attack can restrain the CoT reasoning process even with the explicit prompt requiring CoT. It leads to a higher success rate, results in misled predictions, and outperforms baselines by a significant margin.To summarize, we have the following contributions:• We study the influence of CoT on the adversarial robustness of MLLMs by performing attacks on the two core components of CoT, i.e., rational and answer.• We propose a novel attack method, i.e., stop-reasoning attack, for MLLMs with CoT, which is effective at the most.• We show that the rationale opens a window for understanding the reason for an incorrect answer with an adversarial image.• Extensive experiments are conducted on representative MLLMs and two datasets under the proposed attacking methods to justify our proposal.2 Related work 
adversarial attacks: 
 Deep learning models are known to be vulnerable to adversarial attacks (Szegedy et al., 2013; Goodfellow et al., 2014) . Extensive previous studies have a primary focus on image recognition (Szegedy et al., 2013; Goodfellow et al., 2014; Athalye et al., 2018; Carlini & Wagner, 2017; Gu et al., 2021) and many well-known adversarial methods are proposed such as Projected Gradient Descent (PGD) (Madry et al., 2017) , Fast Gradient Sign Method(FGSM) (Goodfellow et al., 2014) . These studies aim to mislead the models to generate wrong predictions while only adding minimal and imperceptible perturbations to the images (Goodfellow et al., 2014) . Unicorn (Tu et al., 2023) explores robustness against adversarial attacks and OOD generalization but does not address the influence of CoT reasoning on MLLMs' robustness. AVIBench (Zhang et al., 2024) and MM-SafetyBench (Liu et al., 2023b) provide frameworks for evaluating MLLMs' robustness against adversarial attacks, but they do not focus on the CoT reasoning process. Despite the effectiveness of these attacks, it is still hard to interpret the model behavior during the attacks and understand why the attacks could succeed (Gu & Tresp, 2019; Li et al., 2022) . Recent studies have also investigated the vulnerability of large language models (Zou et al., 2023; Kumar et al., 2023) and multimodal LLMs (Zhao et al., 2023; Gan et al., 2020; Gao et al., 2024; Han et al., 2023) under adversarial attacks. However, the adversarial robustness of multimodal LLMs with CoT reasoning ability is still under-explored. Since CoT reasoning reveals the model's decision process (Wei et al., 2023) , this reported intermediate process can serve as a good proxy for to understand the model behavior before and after the adversarial attacks, which additionally brings explainability. Different from previous studies, this work focuses on evaluating the adversarial robustness of MLLMs with CoT by designing effective attack methods and understanding why the model would behave under such adversarial attacks.CoT generates a series of intermediate logical reasoning steps and assists LLMs in thinking step by step before generating the final answer (Wei et al., 2023) . CoT has been widely applied to LLMs (Wei et al., 2023; Kojima et al., 2023; Zhang et al., 2022) and has significantly improved the performance in various tasks, such as arithmetic problems (Wei et al., 2023) and symbolic reasoning (Wei et al., 2023) . Some studies have noticed that CoT can bring extra robustness to the LLMs (Wu et al., 2023) and have designed a better CoT method for better robustness (Wang et al., 2022) Recently, on MLLMs, various studies have also shown that adopting CoT on MLLMs can bring superior performances as well (Lu et al., 2022; Zhang et al., 2023; He et al., 2023) . However, the robustness of CoT on MLLMs against adversarial attacks has not been investigated.  
threat models: 
 This work examines the influence of the CoT reasoning process on MLLMs' adversarial robustness. We follow the principles introduced by Carlini et al. (2019) to define our adversary goals, adversarial capabilities, and adversary knowledge. The adversary goal is to cause the model to output a wrong answer. Given the scenario that MLLMs with a prompt are applied to extract information from user images, the images are assumed to be manipulated by an attacker to mislead MLLMs. Hence, we restrict the adversarial capability to perturb the image in an imperceptible range and assume text prompts are unmodifiable. The restrictions on images areD(v org , v adv ) = max v org -v adv ≤ ϵ (1)where D(•) is the distance between images, v org is the original input image, v adv is the perturbed image, and ϵ is a predefined boundary. As for the adversary knowledge, we assume the full knowledge of the model. Thus, solid attacks can be performed with the PGD (Madry et al., 2017) method for convincing results.We denote a visual question answering (VQA) inference as f (v, q) → t, where f (•) represents an MLLM, v is the input image, q is input text formulated as a question with its multiple answer choices, and t is the output of the MLLM. To make models use the CoT reasoning process, we add a prompt as explicit instruction after the question and choices, e.g., "First, generate a rationale with at least three sentences that can be used to infer the answer to the question. At last, infer the answer according to the question, the image, and the rationale.".As depicted in Figure 2 dotted line, both the textual question and corresponding image are fed to an MLLM to produce an initial clean prediction. This clean prediction, denoted as t clean , serves as the basis for calculating losses according to three attack methods.In one attack iteration (Figure 2 solid line), the MLLM takes both the perturbed image from the last attack iteration v adv l ast and text q as input and generates an adversarial output. Then, the new adversarial image v adv n ew is built by leveraging different attack methods. In the first attack iteration, an initial perturbation is performed before the image is fed into the model. The corresponding optimization problem can be defined as:EQUATIONthe optimization problem can be solved with the PGD method.The woman is wearing goggles to protect her eyes from the sun's glare while skiing down the mountain. The goggles provide UV protection, which is ......... The answer is (B) protection.The woman is wearing goggles to protect her eyes from the sun's glare while skiing down the mountain. The goggles provide UV protection, which is .........The answer is (B) protection.(a) Prediction with CoT.The answer is (D) ball.The dog is trying to catch a frisbee.Answer: The answer is (B) frisbee. As shown in Figure 3a , model inference with CoT reasoning provides an answer and a rationale as its prediction output. We first generalize two existing attack methods to MLLMs with CoT, i.e. answer attack and rationale attack Answer Attack. The answer attack focuses exclusively on attacking the answer part of the output, aiming to manipulate the model to infer a wrong answer.To alter the answer in the prediction, a cross-entropy loss is computed between the generated answer and the ground truth. We extract the explicit answer choice to ensure that the loss computation focuses solely on the chosen response (please refer to Appendix B.2) Given the loss depends only on one character, the influence of the prediction's length is mitigated. The loss function is defined as follows:L ans (t adv , t clean ) = CE(g(t adv ), g(t clean ))(3)where g(•) is the answer extraction function, CE is the cross-entropy function. With escalating the loss, models infer alternative answers, deviating from the correct responses.Rationale Attack. Upon revealing the inferences of models with CoT under answer attack, an interesting observation surfaced: despite the disregard for the rationale in the attack's design, the rationale part also changes in most cases. Building on this insight, the rationale attack is performed, which, in addition to targeting the answer part, also aims at modifying the rationale. We utilize the Kullback-Leibler (KL) divergence to induce changes in the rationale because a high KL divergence indicates a high information loss that fits the target, making the rationale more useless, while cross-entropy does not fit to measure the information entropy. Specifically, the loss function of rationale attack is as followsL rsn (t adv , t clean ) = KL(t rat adv , t rat clean ) + L ans (t adv , t clean ) (4)where the t rat adv is the rationale in the adversarial output and the t rat clean is the rationale in the clean prediction. As the KL divergence increases by perturbing the image, the adversarial rationale diverges from the clean rationale. Hence, an alternative answer is predicted based on the altered rationale.The results indicate that CoT slightly boosts the adversarial robustness of MLLMs against the aforementioned two existing methods and introduces the explainability of the incorrectness.Having explored the influence of CoT on the adversarial robustness of MLLMs, we found that the rationale is important for the inference process. A pertinent question arises: how will the model behave when the reasoning process is halted? Inspired by this question, we introduce stop-reasoning attack, a method that targets the rationale to interrupt the model's In the text input, we predefined a specific answer template, denoted as t tar , to prompt the model to output the answer in a uniform format. The upper part of Figure 3b shows that well-finetuned MLLMs are able to produce answers following the prompt. Therefore, when the initial tokens align with the answer format t tar , the model is forced to directly output the answers in the predefined format and bypass the reasoning process even if the model is prompted explicitly to leverage the CoT (refer to the lower part of Figure 3b ).Stop-reasoning attack formulates a cross-entropy loss to drive the model towards inferring the answer directly without a reasoning process:L stop (t adv , t clean , t tar ) = -CE(t adv , t tar ) + L ans (t adv , t clean )where t tar is a predefined answer template, e.g., "The answer is ().[EOS] ′′ . By increasing the loss, MLLMs directly output the answer by aligning the initial tokens with the specified answer format and alter the answer into a wrong one. This approach bypasses the reasoning process and thus, it eliminates the robustness boost introduced by CoT. The results on all models and datasets reveal its effectiveness. Stop-reasoning attack outperforms the two existing methods by a large margin and can be close to the results of MLLMs without CoT. 
experimental settings: 
 Datasets. ScienceQA (Lu et al., 2022) and A-OKVQA (Schwenk et al., 2022) are used to evaluate the impact of the CoT reasoning process on the adversarial robustness of MLLMs, where both datasets comprise multiple-choice questions and rationales and require understanding on complex images. ScienceQA is sourced from elementary and high school science curricula and includes reasoning tasks. A-OKVQA requires commonsense reasoning about the depicted scene in the image and is known as a prevalent choice for VQA reasoning tasks. We perform attacks on data samples that are correctly answered by MLLMs with CoT (need to be attacked (correctly answered) and can be attacked with rationale and stop-reasoning attacks (with CoT)).Victim Models. Three representative MLLMs are used in our experiments, i.e., MiniGPT4 (Zhu et al., 2023) , OpenFlamingo (Awadalla et al., 2023) and LLaVA (Liu et al., 2023a) . Commercial MLLMs like GPT-4 (OpenAI et al., 2023) , which operate as black-box products, are excluded from the experiments because the first-order gradients for perturbation are inaccessible. Note that MiniGPT4 and OpenFlamingo can infer with CoT without fine-tuning, while LLaVA initially lacks the CoT capability. LLaVA acquires the CoT capability through fine-tuning with CEQA (Bai et al., 2023) . In our work, we experiment on MiniGPT4-7B, OpenFlamingo-9B, and LLaVA-1.5-7B. For detailed parameters and experiment settings, please refer to Appendix A.In this section, we present the evaluation results of the three victim models under the three proposed attacks to answer the following two questions:• Does the CoT reasoning bring extra robustness to MLLMs against adversarial images? From the results of answer attack and rationale attack in Table 1 , the CoT brings a marginal robustness boost against the two existing attacks.• Is there any specific attack targeting MLLMs with CoT that is effective? Comparisons in Table 1 show that stop-reasoning attack is the most effective for MLLMs with CoT.We provide more diverse examples in Appendix F, where the consequences caused by the three attacks are illustrated.As shown in CoT, although answer attack specifically targets on the final answer, a majority of samples exhibit altered rationales.After reviewing the examples, an important observation is noted that the majority of samples suffering successful answer attacks exhibit altered rationales, even though answer attack does not aim at the rationale part (Table 2 ). This implies that attacking a model with CoT requires changing both the answer and rationale parts.The person is wearing a white jacket because he is a chef.Answer: The answer is (D) chef.The person is wearing a white jacket because he is a nurse.Answer: The answer is (A) nurse.(a) Key change visualization.The books are located on the shelves, and the books are located on the shelves because the books are on the shelves.Answer: The answer is (A) books.The shelves are located in the library, and the books and dvds are located in the library. Answer:The answer is (D) food.After Attack The replication of the answer serves as the key information to infer the answer from the rationale. After answer attack, the keyword in the rationale is also altered, even though the attack exclusively targets on the answer ("D" → "A"). (b) Trivial change visualization. The replication of the answer is the key information to infer the answer from the rationale. After answer attack, the keyword is not changed (the word "books" is not changed), while the other part of the rationale is changed.Based on the observation above, rationale attack is performed. Rationale attack exhibits superior performance on OpenFlamingo and LLaVA compared to answer attack (Table 1 ), with marginal improvement (56.96% to 49.27% on ScinceQA on LLaVA, 11.14% to 10.79% on A-OKVQA on OpenFlamingo). Conversely, on MiniGPT4, the rationale attack proves less effective than answer attack on both datasets (16.06% under answer attack against 29.06% under rationale attack).To understand why rationale attack does not always work, we pick 100 samples of each victim model on A-OKVQA under rationale attack. We classify these samples into two categories according to their changes in rationale: key changes and trivial changes. A key change refers to the modifications on words crucial for deducing a correct answer, as shown in Figure 4a . A trivial change (as illustrated with the example in Figure 4b ), on the other hand, refers to those modifications on words that are non-crucial for deducing a correct answer while leaving key information untouched. Figure 5 : Classifications of different types of changes made to rationale in three victim models under rational attack (based on 100 Samples/Model). The groups "Failure" and "Success" indicate whether the attack failed or succeeded. "Failure" indicates an unsuccessful attack where the model's prediction remains correct, while "Success" denotes a successful attack resulting in a change from a correct to an incorrect prediction.Figure 5 gives statistical comparisons of the respective numbers of different types of changes made via rationale attack to the three victim models. The classifications indicate that successfully attacked samples under rationale attack are often associated with significant modifications to key information within the rationale. Conversely, samples lacking altered rationales or featuring only minor adjustments tend to preserve their correct answers. This suggests the critical role of key information in influencing the inference of the final answer. However, precisely identifying the crucial information is hard, and modifying it efficiently is even more difficult. To this end, rationale attack can only slightly enhance the attack performance in comparison with answer attack, and our results further support the finding that CoT improves adversarial robustness against generalized attack methods.Given the ineffectiveness of the answer and rationale attacks, we introduce stop-reasoning attack to halt the model's reasoning. The results demonstrate that stop-reasoning attack outperforms both other attacks (11.20% against 31.51% and 44.40% on SincecQA on MiniGPT4, 4.95% against 11.14% and 10.79% on A-OKVQA on Open-Flamingo, and 12.02% against 36.22% and 21.88% on A-OKVQA on LLaVA). It even approaches the performance observed when attacking models without CoT (2.87% against 0.76% on A-OKVQA on MiniGPT4), indicating its remarkable potency in mitigating the additional robustness introduced by the CoT reasoning process. Figure 6 illustrates an example where both rationale and answer attacks fail, and only stop-reasoning attack succeeds. In this work, simple outcheck defense does not work, resulting high false positive rate, given the fact that a simple image does not require CoT.To understand the effectiveness of stop-reasoning, we examine the results of stop-reasoning attack and observe that after the attack, the model outputs the answer directly without The light source is a flashlight, which is a small light source.Answer: The answer is (D) laptop screen.The answer is (B) window.The light source is a flashlight.Answer: The answer is (D) laptop screenThe light source is the laptop screen.Answer: The answer is (D) laptop screen. leveraging CoT, aligning with the fundamental concept of stop-reasoning attack -aiming to halt the CoT reasoning process (Figure 6 and more examples in Appendix F). When stop-reasoning attack succeeds, the model disregards the prompt's CoT reasoning process requirement and directly infers the answer.As revealed in Section 4.2.1, the extra robustness boost is intricately tied to the generated rationale. If this CoT reasoning process is halted by stop-reasoning attack, the additional robustness generated during the CoT reasoning process will be diminished as well. Thus, achieving the adversary's goal becomes comparatively easier. (a) CoT brings explainability for answer attack.The image depicts a giant panda sitting on a tree branch in a lush green forest.The panda has black and white fur, with a distinctive black patch on its forehead.The class of the image is giant panda.The image shows a black dog lying on its back in a grassy field. The dog is wearing a collar and has its paws in the air. There are trees and mountains in the background.The class of the image is chow.(b) CoT brings explainability for rationale attack. On the top, a monkey is falsely recognized as a panda. On the bottom, a bison is falsely recognized as a chow.Although CoT brings marginal robustness to MLLMs against existing attacks, MLLMs are still vulnerable to adversarial images, similar to traditional vision models. When traditional vision models make inferences, e.g., on classification tasks, our understanding is confined to the correctness of the answer. Delving deeper into the model's reasoning process and answering the question of why the model infers a wrong answer with an adversarial image is difficult. In comparison, when MLLMs perform inference with CoT reasoning, it opens a window into the intermediate reasoning steps that models employ to derive the final answer. The intermediate reasoning steps (rationale part) generated by the CoT reasoning process provide insights and potentially reveal the reasoning process of the MLLMs.To look deeper into the explainability introduced by CoT, we conducted image classification tasks on ImageNet (Russakovsky et al., 2015) . These tasks involved constructing multichoice questions by extracting subsets from ImageNet (please refer to Appendix C for selected classes). We provide two example pairs to illustrate the rationale's changes under answer attack and rationale attack.CoT inference under answer attack. In the upper example, CoT erroneously interprets the partial color of the monkey as white, resulting in the misclassification of the monkey as a panda. In the bottom example, the rationale falsely asserts that the black-and-white patterns on the panda's body resemble the black-white picture of a bison. This misconception leads to the incorrect inference of a bison. Figure 7b displays examples under rationale attack. In the upper example, the rationale incorrectly states that the black forehead is a distinctive black patch, leading the model to inaccurately classify the image as a panda instead of a monkey. In the bottom example, the horn of a bison is misinterpreted as a collar, resulting in the false classification of the bison as a chow. 
conclusion: 
 In this paper, we fully investigated the impact of CoT on the robustness of MLLMs. Specifically, we introduced stop-reasoning attack, a novel attack method tailored for MLLMs with CoT. Our findings reveal that CoT can slightly enhance the robustness of MLLMs against answer attack and rationale attack. This extra robustness is attributed to the complexity of changing precisely the key information in the rationale part. For stop-reasoning attack, our test results show that MLLMs with CoT still suffer and the expected extra robustness is eliminated. At last, examples are provided to reveal the changes in CoT when MLLMs infer wrong answers with adversarial images. 
 
 Appendix: To perform an exact attack, the model is prompted to answer the multiple-choice questions in a specific form and explicitly show the answer choice. As shown in Figure 8 (a), only the choice letter in the answer sentence will be considered as the answer. The choice content and choices appearing in other sentences will not be accepted.The man in ... The image does not show any signs of sun or snow, which eliminates options (C) and (D).Therefore, the answer is (A) hail.(a) Extract answer.The image shows a street with a building on the left side and a bicycle parked in front of it. There are no trucks or airplanes visible in the image. Therefore, the answer is (C) bicycle....(b) Split rationale.Figure 8 : (a) Extract answer. Only the choice letter (green) in the answer sentence will be considered as the answer. Other choice letters or choice content (red) will be ignored. (b) Split rationale. Only the sentences (bold) before the answer extracted (green) will be contoured as the rationale.To perform the rationale attack, the rationale and answer parts in the output logit matrix should be split if the model answers the question with the CoT process. As the used LLMs are all generative models, it is not deterministic where the rationale is, where the answer is, and how long each is. So, the output logit matrix is decoded and split into sub-sentences first. As the model is imperfect and the instruction prompt is not strong enough, the model may not follow the instructions exactly. The inference may mix the answer and the rationale part. The part before the sentence the answer is extracted from is the rationale. As shown in Figure 8 (b), inferences can be roughly divided into two parts from the answer. The sentences from the answer are regarded as the answer part of the model, even though there are some other sentences. The sentences before the answer belong to the rationale part. The corresponding logit matrix will be extracted.The general stop criteria shared in all attack scenarios is whether the inferred answer is wrong in the perturbation loop. The perturbation process will be stopped if the answer is wrong. Then, the perturbed image will be fed into the model again to infer the final answer with the generate(•) method. The sample won't be attacked again, regardless of the correctness of the final answer generated with the generate(•) method. When the MLLM is under stop-reasoning attack, the stop criteria is combined with an additional stop check, which checks if the answer is extracted from the first sentence.In the context of language models, the f orward(•) function often refers to the process of passing input data through the model to obtain predictions or activations. For LLMs used in MLLMs like Llama2 (Touvron et al., 2023) , the f orward(•) function has the same length in output as the input. The output token is the predicted next token to the input token at the same position. The generate(•) function generates output by iteratively using the f orward(•) function. Specifically, in each iteration, only the last token, the next token to the whole input, is extracted and concatenated after the input sequence. The new sequence will be used as input in the next iteration until there is an end-of-sequence token [EOS] . To generate an output, the generate(•) function costs much more time than the f orward(•), if a ground truth can be provided to the f roward(•) function, because the generate(•) function goes through the f orward(•) function several times while the f orward(•) with ground truth needs only one iteration. However, it's important to note that to generate all tokens at once, the f orward(•) method requires a ground truth, and there is a performance gap between the f orward(•) and the generate(•) functions.As outlined in appendix B.1, we adopt the clean prediction as the pseudo ground truth for the f orward(•) method. As the perturbation progresses, the input image differs from the one used for the clean prediction. Consequently, the pseudo ground truth deviates from the actual ground truth, leading to a divergence in the adversarial output. To address the disparity between the real adversarial output and the actual adversarial output, the clean prediction should be updated with the latest adversarial image every several iterations.We create classification tasks by extracting 4, 8, and 16 classes. All the classes are randomly picked from the dataset. The specific classes selected for each scenario are as follows:• 4 classes: English setter, Persian cat, school bus, pineapple.• 8 classes: bison, howler monkey, hippopotamus, chow, giant panda, American Staffordshire terrier, Shetland sheepdog, Great Pyrenees.• 16 classes: piggy bank, street sign, bell cote, fountain pen, Windsor tie, volleyball, overskirt, sarong, purse, bolo tie, bib, parachute, sleeping bag, television, swimming trunks, measuring cup.All tasks had a uniform question: "What is the class of the image?"We randomly picked several classes from the ImageNet dataset (Appendix C). Surprisingly, as indicated in Table 3 , the tests with 8 classes show worse performances than the tests with 16 classes. The reason maybe some of the chosen classes are similar, and the similarity makes the classification task more complex, e.g., American Staffordshire terrier and Shetland sheepdog look similar. However, the conclusion in all tests is consistent: the marginal improvement in performance brought about by CoT suggests that the rationale may not be essential for simple tasks. Although the two main results outlined in Section 4.2.1 share commonalities, a notable gap exists between the accuracy values of the ImageNet series and those of the A-OKVQA and ScienceQA datasets when models with CoT are subjected to the answer attack. This discrepancy can be attributed to the inherent complexity of VQA tasks compared to the straightforward classification tasks on the ImageNet dataset.Further examination of the A-OKVQA and ScienceQA datasets reveals that the A-OKVQA dataset is relatively easier, as illustrated in Table 1 . This performance difference is consistent across all three models. By comparing the accuracy of the classification task on ImageNet with the VQA tasks on A-OKVQA and ScienceQA, a significant observation emerges: CoT has almost no impact on the robustness of simple tasks. During the ablation study, the adversarial capability is changed by narrowing or expanding the limited boundary (ϵ) to 2, 4, 8, 32 (as described in Section 3.1). Table 4 presents results of the MiniGPT4 and Open-Falmingo with ϵ set to 8. The results are consistent with Table 1 , indicating that the CoT reasoning process enhances the robustness of MLLMs. Furthermore, Figure 9 shows the results of ScienceQA on MiniGPT4 with different ϵ. Figure 9a shows stop-reasoning attack consistently outperforms rationale and answer attacks as ϵ increases. Besides the aforementioned attacks, we also conducted targeted attack on MiniGPT4. When the MLLM answers the question correctly, we randomly pick a wrong answer and use it as the target. The input images are perturbed so that the MLLM answers the question with the picked target choice. Table5 shows the success rate of the attacks. In the provided set of 11 samples:• First 4 Samples (IDs: 10, 12, 40, 43 ): All attacks (answer attack, rationale attack, stop-reasoning attack) succeed.• Next 2 Samples (IDs: 23, 1024): Only the stop-reasoning attack succeeds.• Next 2 Samples (IDs: 21, 51): Both answer attack and stop-reasoning attack succeed. The first plane is a B-2 Spirit, which is a stealth bomber, and the second plane is a Eurofighter Typhoon, which is a fighter jet. The question is asking about the fruit in the salad. The lettuce is green, the radish is red, and the apples and grapes are red and green, respectively. The only fruit that is darker than the others is the black olive. Answer: The answer is (D) radish.The question is asking about the fruit in the salad. The only fruit in the salad is the banana, which is cut in half. The banana is darker than the other fruit, which is the reason why the answer is (D) radish.Answer: The answer is (D) radish The question is asking about the fruit in the salad. The only fruit in the salad is the grape. The grape is cut in half and darker than the other fruit. The only fruit that is cut in half and darker than the other fruit is the grape.Answer: The answer is (A)The question is asking about the color of the fruit, and the answer is the only one that is cut in half.Answer: The answer is (C) lettuce. The animal is a cow, which is commonly used in farming. Answer: The answer is (D) lion.The animal is a cow, which is commonly used in farming. Answer: The answer is (D) lion.The question is asking about the type of animal on a leash, and the only animal that is commonly seen on a leash is a dog. Answer: The answer is (A) dog.The animal is a horse, which is a type of equine. The knife is for cutting vegetables. Answer: The answer is (C) vegetables.The knife is for the waffle, which is a type of bread. Answer: The answer is (A) bread.The knife is for cutting meat, which is the only food item that is not a vegetable. Answer:The answer is (D) meat.The answer is (B) fruit.  


Decision: This paper considers the intersection of adversarial images and chain-of-thought prompting in multimodal LLMs. After author rebuttal, it received scores of 678. The authors have done a good job of rebuttal, and all the reviewers are happy about the paper after rebuttal, commenting that (1) the writing is clear and easy to follow, (2) the method seems to be effective given the results presented, and (3) it extends the domain of adversarial inputs to the multiple-choice and CoT regime in the setting of multimodal LLMs. The additional results added during rebuttal is also helpful to address reviewers' concerns. Therefore, the AC would like to recommend acceptance of the paper.

Reviews: review: paper_summary: paper_summary: This paper first found that the CoT technique can improve the adversarial robustness of MLLMs. Then they proposed a novel attack method that train visual noise using designed losses that are accosiated with CoT components. Experiments are conducted on two VQA datasets using three different MLLMs, and the resutls presented confim the effectiveness of the method.
strength_weakness: strength_weakness: 1. Generalization: the paper only perform their method on two VQA task. On the one hand, can the method perform well on multimodal generation tasks like captioning? On the other hand, while your current attack goal is to only "avoid the right answer", can your method be applied to more strict tasks like "generate the pre-defined wrong answer " (e.g., the jailbreaking task)?
2. Model Size: only MLLMs with 7B size LLM are applied , I'm wondering how's effectiveness of your stop-reasoning on larger MLLMs, such as models with 13B LLMs.
3. Ablation of $\epsilon$ and the training iteration: the pertubation budget $\epsilon$ and training iteration are main parameters in adversarial learning domain. Maybe you can draw a picture about the attack success rate of your method under different $\epsilon$ and/or training steps in the main part.
4. Missing reference: several references that are highly related to your paper are missing [1-3].

[1] Zhang H, Shao W, Liu H, et al. AVIBench: Towards Evaluating the Robustness of Large Vision-Language Model on Adversarial Visual-Instructions[J]. arXiv preprint arXiv:2403.09346, 2024.

[2] Tu H, Cui C, Wang Z, et al. How many unicorns are in this image? a safety evaluation benchmark for vision llms[J]. arXiv preprint arXiv:2311.16101, 2023.

[3] Liu X, Zhu Y, Lan Y, et al. Query-relevant images jailbreak large multi-modal models[J]. arXiv preprint arXiv:2311.17600, 2023.
paper_summary: paper_summary: This research focuses on multimodal language models (MLLMs) and introduces a novel adversarial attack method called "stop-reasoning attack" to investigate and assess the impact of CoT reasoning on the adversarial robustness of MLLMs. Experimental results demonstrate that CoT reasoning indeed enhances the robustness of MLLMs against existing adversarial attack methods.
strength_weakness: strength_weakness: 1. I am concerned about the novelty of the method. The overall pipeline of the attack is similar to other methods, where noise is used to perturb images so that they are mapped to incorrect target locations in the feature space after passing through the MLLM image encoder.
questions: questions: 1. I have a question about the difference between "stop reasoning attack" and simply not using CoT and then attacking the answer. I believe they both ultimately attack the answer. What might be the performance difference between the "Answer Attack" column in Table 1 for "w/o CoT" and the "Stop Reasoning Attack" column for "with CoT"? Could this difference in performance be due to the need for additional optimization of a reasoning component in the stop reasoning attack
paper_summary: paper_summary: This paper considers the intersection of adversarial images + chain-of-thought prompting.

The setup involves a model answering multiple choice questions given an image. The model tries to give reasoning first before spitting out a letter from A to E. There are several attacks:
* answer attack, in making the model spit out the wrong answer
* rationale attack, in making the model point out incorrect observations
* stop reasoning attack, in making the model not do chain-of-thought at all

the authors consider various image+LLMs like MiniGPT4, OpenFlamingo, and LLaVa; then A-OKVQA and ScienceQA. the results show that these various attacks can be successful.

---
edit post rebuttal: still agree to accept this paper, I am less concerned about novelty and don't think one negative review saying that should be reason to reject. so overall should be reasonable to accept this paper given all the reviews.
strength_weakness: strength_weakness: Possibly, this is a bit of a narrow attack that assumes a lot of knowledge about the model - the model's parameters, the domain, etc. To this reviewer though this is interesting even outside of the security domain, so I still recommend acceptance.

Comments: title: Official Comment by Reviewer VqF6
comment: Thanks for your additional results, I’ll increase my score by 1.
Thank you very much for your response. It has addressed my concerns and clarified a previous misunderstanding I had about this paper. I have adjusted my rating：）