言語処理学会 第30回年次大会 発表論文集（2024年3月）

Evaluation of the Adversarial Robustness
in LLM-based Visual Dialog System
Yahan Yu Fei Cheng Chenhui Chu
Kyoto University
yahan@nlp.ist.i.kyoto-u.ac.jp {feicheng, chu}@i.kyoto-u.ac.jp

Abstract
Large Language Models (LLMs) are widely employed.
However, their susceptibility to adversarial attacks posesa signiﬁcant security concern. In this paper, we focus
on LLM-based visual dialog systems and delve into their
sensitivity in the aspect of both visual attack and textual
attack. Our work aims to investigate the robustness ofthese systems, and give researchers the understanding of
the security challenges that LLMs may face in practicalapplications.

1

Introduction

In recent years, the ﬁeld of natural language processing has witnessed a surge in the utilization of Large Language Models (LLMs) [1] and their multi-modal extensions[2, 3, 4], underscoring their pivotal role in various applications. In contrast to non-LLM-based multi-modal systems,
such as ViLBERT-based and CLIP-based, the LLM-based
approaches often have more powerful abilities for contextual understanding and transfer learning because of theirlarge-scale training. Despite the success of LLM-based
multi-modal systems, an escalating concern surrounds the
security robustness [5] of these models, particularly in theface of adversarial attacks [6].
Adversarial attacks [7], characterized by purposeful manipulations of input data, exploit the inherent vulnerabilities in model architectures, posing formidable challenges
to the reliability and security of LLMs. In multi-modal
scenarios, the system may exhibit varying sensitivities to
inputs from diﬀerent modalities. Attackers can exploit thisby selectively targeting the most sensitive modality during
adversarial attacks to achieve more eﬀective manipulation.
It is necessary to consider the adversarial robustness of thesystem on multiple modalities simultaneously to obtain acomprehensive evaluation.

In response to this security risk and insuﬃcient research
in LLM-based multi-modal systems, our work directs its
focus toward the speciﬁc domain of LLM-based visual dialog systems [8]. Given that Chatbot [9] represents a foundational function of the application of LLMs, our work
aims to evaluate their robustness boundaries. To emulate real-world scenarios, we introduce a zero-shot setting
system tailored for the visual dialog task. This system
serves to evaluate the robustness of LLM-based multimodal systems against adversarial attacks, incorporatingassessments in both text and visual modalities. Speciﬁcally, we scrutinize robustness through Fast Gradient Sign
Method (FGSM) attacks [7], introducing adversarial noise
to images, and coreference attacks [10], manipulating the
textual input.
Our contributions are summarised as below:
1. To reﬂect the real robustness of LLMs, we construct
the LLMs-based visual dialog system in zero-shot settings.
2. Our successful execution of FGSM and coreference
attacks sheds light on the nuanced vulnerabilities inherent in LLM-based visual dialog systems.

2

Related Work

2.1

Visual Dialog Task

Visual dialog [8] has been introduced as an extended taskof Visual Question Answering (VQA) [11]. In Visual dialog, the system is tasked with responding to a sequence ofinterconnected questions, leveraging both an image and a
dialog history. Prior works have explored attention mechanisms [12] that account for the intricate interactions among
modalities of the image, dialog history, and question. Some
investigations [13] have delved into uncovering the semantic structures within the dialog through graph neural net-

― 2069 ―This work is licensed by the author(s) under CC BY 4.0(https://creativecommons.org/licenses/by/4.0/).

Coreference Attack
Is there any text on the screen?

Image
Encoder

Add noise to image

Concat:History Emb.
Q-former Emb.
Answer Emb.

Q-formerGet gradient of Visual Emb.

FGSM Attack

OPT

OutputLog-likelihood
score

Rank
AnswerCandidates

Figure 1

The proposed structure of LLM-based zero-shot visual dialog system. We evaluate its adversarial robustness against FGSM
attack and coreference attack. Example of coreference attack is shown in Table 1.

works. Our work takes into account the excellent zero-shotand the textual attack as TA(·). After attacking, we input

generalization capability of LLMs, and leverages this capa-

the perturbed image 𝑉 ′ = VA(𝑉) or the perturbed dialog

bility to facilitate interactions and inference across modal-

history 𝐻 ′ = TA(𝐻) into the model, and expected theities within the visual dialog system.

answer 𝐴𝑛𝑠′ should be diﬀerent from the ground truth

2.2

answer.

Adversarial Attacks on Multi-modal
ModelsIn the domain of multi-modal models, a few studies haverecently delved into examining their susceptibility to adversarial attacks. In [14], this work attacked textual inputs

4

Method

4.1 LLM-based Zero-shot Visual Dialog
Sysytemusing the methods of BERT-Attack and TextFooler, provid-

The proposed structure is shown in Figure 1.

ing the ﬁrst investigation into the robustness of visual dialog

Selection for LLM Considering the advanced perfor-

models against textual attacks. In [15], this work demon-

mance and open-source implementation, we choose BLIP-

strated that imperceptible attacks on images, altering theoutput of an image caption model, can be exploited by ma-

2 [16] as the base model in the zero-shot visual dialog
system, which is a recent multi-modal model that gained

licious content providers to harm users. In [10], both image

signiﬁcant attention. BLIP-2 seamlessly integrates visionand text-level attacks were imposed on a ViLBERT-based

and language understanding by combining a pre-trained

visual dialog system, evaluating the adversarial robustness

visual model with a LLM. Consequently, it possesses theacross multiple modalities. Our evaluation builds on thesecapability to handle both visual and textual inputs, leading

insights, concentrating on LLM-based multi-modal mod-to the generation of coherent natural language outputs.

els and demonstrating the eﬀectiveness of the attacks on
the visual dialog task.

3

Zero-shot Setting We construct a zero-shot system forthe evaluation of adversarial robustness. The zero-shotsetting aims to closely mimic real-world application sce-

Task Deﬁnition

narios, where models may encounter previously unseen

For the visual dialog task, we deﬁne the input image

categories or tasks. This conﬁguration better simulates

as 𝑉, the dialog history as 𝐻 = {𝐻1 , 𝐻2 , ..., 𝐻𝑡 } where 𝑡

the robustness observed in practical use, providing a moremeans dialog turns and 𝐻𝑡 = {𝑄 𝑡 , 𝐴𝑡 } means the question

realistic assessment of the model’s performance.
Scoring of Answers To select the appropriate answer

𝑄 𝑡 and answer 𝐴𝑡 in one turn, and the ﬁnal question as
𝑄 0 . The objective of this task is to enable the model tofrom the candidate set, we calculate the log-likelihood

select the appropriate answer 𝐴𝑛𝑠 from a set containing

score [8] of all the candidates because our visual dialog

𝑁 candidate answers { 𝐴𝑛𝑠1 , 𝐴𝑛𝑠2 , ..., 𝐴𝑛𝑠 𝑁 } based on thesystem contains a generative decoder. In the evaluation

input information.

phase, the rank of the candidate set is given by their log-

As for the attacks, we deﬁne the visual attack as VA(·)

likelihood scores.

― 2070 ―This work is licensed by the author(s) under CC BY 4.0(https://creativecommons.org/licenses/by/4.0/).

4.2

Adversarial Visual Attack

We make the assumption that an attacker possesses thecapability to introduce minor perturbations to the visual
inputs of the model. Additionally, we assume that theattacker has unrestricted access to all model weights.

Table 1 Example of adversarial textual attack. The red words
indicate that the target word for the attack is adults which is
replaced with grownup after the attack.
Original Text
Attacked Text
Q: what is in the background? Q: what is in the background?
A: trees and buildings
A: trees and buildings
Q: how many are adults?
Q: how many are grownup?
A: 1 adults
A: 1 grownup

In this setting, the Fast Gradient Sign Method [7] is anattack method that introduces perturbations to visual inputs
guided by the gradients of the loss concerning the visual
inputs:
FGSM(𝑉) = 𝑉 + 𝜖 · sign(∇𝑉 𝐿𝑜𝑠𝑠(𝑉, 𝑙𝑎𝑏𝑒𝑙)),

(1)

4.3

Adversarial Textual Attack

where 𝑉 and 𝑙𝑎𝑏𝑒𝑙 represent the visual inputs and their

We also investigate adversarial robustness against textual

corresponding ground-truth labels, respectively. ∇𝐿𝑜𝑠𝑠(·)

attacks. As shown in Table 1, we employ the coreferencemeans the gradient of the model. The hyperparameter 𝜖attack [10], wherein noun phrases in the dialog historyis utilized to modulate the intensity of perturbations. And

are replaced with their synonyms to deceive the models.

the sign(·) is a mathematical function that returns the sign

Coreference means that the target words and the replace-of a real number. Speciﬁcally, it maps a positive number
to 1, a negative number to -1, and zero to 0. The sign(·)

ment words not only have semantic similarity but also referto the same object. In this method, we utilize an oﬀ-theshelf neural coreference resolution tool [17] to identifyfunction is denoted as follows:

words in the dialog history referring to objects mentioned



1




sign(𝑥) = −1




0


in a given question. We perform a greedy substitution of

if 𝑥 > 0
if 𝑥 < 0(2)

if 𝑥 = 0words with their synonyms, selecting those with the minimum cosine distance in the counter-ﬁtting word embedding
space [18]. After going through these steps, we obtained
the attacked text samples.

However, in contrast to the aforementioned conﬁguration, each question in the visual dialog task may have one

5

Experiments

5.1

Datasetor more pertinent answers within the list of candidate answers. This is because some candidate answers may besemantically identical (e.g. ’yeah’ and ’yes’). So the validation dataset of visual dialog task is annotated with therelevance score [8] between each of the candidate answers
and ground truth, which ranges from 0 to 1. Therefore, we
modify the FGSM attack as follows:𝜖 · sign(

which is widely used in the visual dialog task. It consistsof a collection of dialogues between humans discussing
images. The dataset is designed to facilitate research on
the intersection of computer vision and natural language
processing, particularly in tasks involving dialog under-

FGSM(𝑉) = 𝑉+𝑁∑

VisDial v1.0 [8] is a version of the Visual Dialog dataset,

𝑅𝑒𝑙 𝑛 · ∇𝑉 𝐿𝑜𝑠𝑠((𝑉, 𝐻, 𝑄), 𝐴𝑛𝑠 𝑛 )),𝑛=1

standing and visual reasoning. The VisDial v1.0 dataset
contains 123k, 2k, and 8k dialogs as train, validation, andtest split. In our zero-shot setting, we only use the valida-(3)
where 𝐴𝑛𝑠 𝑛 means the 𝑛−𝑡ℎ candidate answer, 𝑅𝑒𝑙 𝑛 means
the relevent score between ground truth answer and the 𝑛 −tion set.

5.2

Implementation𝑡ℎ candidate answer. Equation signiﬁes that the gradients

As for the BLIP-2 model, in our work, we opted for CLIPof the loss with respect to all relevant answers are taken(ViT-L/14) [19] as the Image Encoder and the decoder-

into account in the FGSM attack.

based OPT model (2.7B and 6.7B) [20] as the LLM.

― 2071 ―This work is licensed by the author(s) under CC BY 4.0(https://creativecommons.org/licenses/by/4.0/).

Table 2

Results on LLM-based zero-shot visual dialog system
without attacks. The ViLBERT-based, CLIP-based and FROMAGe-based approaches are the existing methods, while two
BLIP-2-based approaches are our baselines. We emphasize thecurrent state-of-the-art results.
w/o attacks
Base Model
NDCG MRR R@1 R@5 R@10
ViLBERT [21]
11.6
6.9
2.6
7.2
11.3
CLIP (Vit-L/14) [19]
10.9
8.5
3.1
8.7
15.9
FROMAGe [22]16.5 22.0 17.6 20.1
25.1
BLIP-2 (OPT2.7B)
13.9 20.8 15.9 18.4
23.2
BLIP-2 (OPT6.7B)
14.2 20.9 16.2 18.6
23.8

Table 3Results on LLM-based zero-shot visual dialog system with adversarial visual attack. We adjust the perturbation
strength by varying the hyperparameter 𝜖, observing changes in
the model’s performance.
NDCG
Base Model𝜖 = 0 𝜖 = 0.01 𝜖 = 0.03 𝜖 = 0.05
BLIP-2 (OPT2.7b)
13.9
13.8
13.8
13.8
Model
𝜖 = 0.1
BLIP-2 (OPT2.7b)13.8

𝜖 = 0.2
13.8

𝜖 = 0.3
13.7

Model
𝜖 = 0.5
BLIP-2 (OPT2.7b)13.7

𝜖 = 0.6
13.6

𝜖 = 0.7
13.5

𝜖 = 0.4
13.7

Table 4

5.3

Metrics

We adhere to the standardized evaluation protocol introduced in [8]. Generative tasks for visual dialog models areappraised using retrieval-based evaluation metrics, including normalized discounted cumulative gain (NDCG), mean
reciprocal rank (MRR) and recall@k (R@k, k = 1, 5, 10).
Each dialogue comprises a list of 100 answer candidates
for every question, with one ground-truth answer includedin the candidates. The model arranges the answer candidates based on log-likelihood scores and is subsequently
evaluated using the aforementioned three metrics. NDCG
considers all relevant answers from the candidate set, while
MRR and R@k take into account the rank of the single
ground-truth answer. So, NDCG is considered the primary evaluation metric in current works.

5.4

Results on LLM-based Zero-shot Visual Dialog System

Table 2 shows the results on our proposed system without
adversarial attacks in contrast with other zero-shot visual
dialog systems. In details, BLIP-2 has 188M trainable
parameters in the pretraining stage, while ViLBERT has
114M, CLIP has 300M and FROMAGe has 5.5M. The
performance and trainable parameters of BLIP-2-basedmodel are close to the existing zero-shot visual dialog models, which can illustrate that the structure of BLIP-2-basedmodels is reasonable, and can be used in the attack process.

5.5

Results on Adversarial Visual Attack

Table 3 shows the results on LLM-based zero-shot visual dialog system with the visual attack. Hyperparameter𝜖 adjusts the intensity of perturbations. As the 𝜖 rises,
the perturbation applied to the image becomes stronger
and NDCG drops, which means that the FGSM attack is
successful on the BLIP-2-based model. In the ViLBERT-

Results on LLM-based zero-shot visual dialog systemwith adversarial textual attack.
NDCG
Base Modelw/o attacks w/ textual attack
BLIP-2 (OPT2.7b)
13.9
16.8

based visual dialog model [10], FGSM usually causes adrop by 30%. In our work, the slight drop in NDCG can be
attributed to the robustness of BLIP-2 because of the pretraining on multiple vision datasets such as LAION [23].

5.6

Results on Adversarial Textual Attack

Table 4 shows the results on LLM-based zero-shot visual dialog system with the coreference attack. In the
ViLBERT-based visual dialog model [10], NDCG usually
drops by 4%. But in our work, NDCG improves. We
hypothesize that, based on the language understanding andreasoning capabilities of LLMs, the substitution of synonymous words in the text contributes to the semantic completion of the dialog history. This indicates that LLM-basedmodels are robust against our coreference attack.

6

Conclusion
In conclusion, our study underscores the widespreadutilization of LLMs while highlighting a critical security
concern due to their vulnerability to adversarial attacks.
The identiﬁed weaknesses in security extend not only to
LLMs but also to LLM-based multi-modal models. Specifically focusing on LLM-based visual dialog systems, ourresearch ﬁnds their sensitivity to visual attacks, and the opposite impact to textual attacks. Future work could involvethe design of more advanced adversarial attacks to attainstronger visual attacks and more eﬀective textual attacks.
By developing novel adversarial strategies, researchers cangain a deeper understanding of the security boundary in
LLM-based visual dialog systems, contributing to robust
defense mechanisms.

― 2072 ―This work is licensed by the author(s) under CC BY 4.0(https://creativecommons.org/licenses/by/4.0/).

der G Schwing. Factor graph attention. In Proceedings

Acknowledgements
This work was supported by 京都大学大学院情報学研究科 博士後期課程の学生に対する支援事業.

References[1] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann
Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, andTatsunori B Hashimoto. Stanford alpaca: An instructionfollowing llama model, 2023.
[2] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae
Lee.
Visual instruction tuning.
arXiv preprint
arXiv:2304.08485, 2023.
[3] Xi Chen, Josip Djolonga, Piotr Padlewski, Basil Mustafa,
Soravit Changpinyo, Jialin Wu, Carlos Riquelme Ruiz,
Sebastian Goodman, Xiao Wang, Yi Tay, et al. Pali-x:
On scaling up a multilingual vision and language model.
arXiv preprint arXiv:2305.18565, 2023.
[4] Yanzhe Zhang, Ruiyi Zhang, Jiuxiang Gu, Yufan Zhou,
Nedim Lipka, Diyi Yang, and Tong Sun. Llavar: Enhanced
visual instruction tuning for text-rich image understanding.
arXiv preprint arXiv:2306.17107, 2023.
[5] Jun Yan, Vikas Yadav, Shiyang Li, Lichang Chen,
Zheng Tang, Hai Wang, Vijay Srinivasan, Xiang
Ren, and Hongxia Jin. Virtual prompt injection for
instruction-tuned large language models. arXiv preprint
arXiv:2307.16888, 2023.
[6] Andy Zou, Zifan Wang, J Zico Kolter, and Matt Fredrikson.
Universal and transferable adversarial attacks on aligned
language models. arXiv preprint arXiv:2307.15043,
2023.
[7] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy.
Explaining and harnessing adversarial examples. arXiv
preprint arXiv:1412.6572, 2014.
[8] Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh,
Deshraj Yadav, José MF Moura, Devi Parikh, and Dhruv
Batra. Visual dialog. In Proceedings of the IEEE conference on computer vision and pattern recognition,
pp. 326–335, 2017.
[9] Sara Montagna, Stefano Ferretti, Lorenz Cuno Klopfenstein, Antonio Florio, and Martino Francesco Pengo. Data
decentralisation of llm-based chatbot systems in chronic
disease self-management. In Proceedings of the 2023
ACM Conference on Information Technology for Social Good, pp. 205–212, 2023.

[10] Gi-Cheon Kang, Sungdong Kim, Jin-Hwa Kim, Donghyun
Kwak, and Byoung-Tak Zhang. The dialog must go on:
Improving visual dialog via generative self-training. In
Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6746–6756,

2023.
[11] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret
Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi
Parikh. Vqa: Visual question answering. In Proceedingsof the IEEE/CVF Conference on Computer Vision and
Pattern Recognition, pp. 2039–2048, 2019.

[13] Zilong Zheng, Wenguan Wang, Siyuan Qi, and Song-Chun
Zhu. Reasoning visual dialogs with structural and partial
observations. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.
6669–6678, 2019.
[14] Lu Yu and Verena Rieser. Adversarial textual robustnessof visual dialog. In 61st Annual Meeting of the Association for Computational Linguistics 2023, pp. 3422–
3438. Association for Computational Linguistics, 2023.
[15] Christian Schlarmann and Matthias Hein. On the adversarial robustness of multi-modal foundation models. In Proceedings of the IEEE/CVF International Conference
on Computer Vision, pp. 3677–3685, 2023.

[16] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi.
Blip-2: Bootstrapping language-image pre-training withfrozen image encoders and large language models. arXiv
preprint arXiv:2301.12597, 2023.
[17] Kevin Clark and Christopher D Manning. Deep reinforcement learning for mention-ranking coreference models.
arXiv preprint arXiv:1609.08667, 2016.
[18] Di Jin, Zhijing Jin, Joey Tianyi Zhou, and Peter Szolovits.
Is bert really robust? a strong baseline for natural language
attack on text classiﬁcation and entailment. In Proceedings of the AAAI conference on artiﬁcial intelligence,Vol. 34, pp. 8018–8025, 2020.
[19] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya
Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry,
Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748–8763. PMLR, 2021.
[20] Susan Zhang, Stephen Roller, Naman Goyal, Mikel
Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan,
Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open
pre-trained transformer language models. arXiv preprint
arXiv:2205.01068, 2022.
[21] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. Advances in neural
information processing systems, Vol. 32, , 2019.
[22] Jing Yu Koh, Ruslan Salakhutdinov, and Daniel Fried.
Grounding language models to images for multimodal inputs and outputs. In International Conference on Machine Learning, pp. 17283–17300. PMLR, 2023.
[23] Christoph Schuhmann, Romain Beaumont, Richard
Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo
Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset fortraining next generation image-text models. Advances in
Neural Information Processing Systems, Vol. 35, pp.
25278–25294, 2022.

of the IEEE international conference on computer vision, pp. 2425–2433, 2015.

[12] Idan Schwartz, Seunghak Yu, Tamir Hazan, and Alexan-― 2073 ―This work is licensed by the author(s) under CC BY 4.0(https://creativecommons.org/licenses/by/4.0/).