Adversarial Textual Robustness of Visual Dialog
Year: 2023

Abstract:
Adversarial robustness evaluates the worst-case performance scenario of a machine learning model to ensure its safety and reliability. For example, cases where the user input contains a minimal change, e.g. a synonym, which causes the previously correct model to return a wrong answer. Using this scenario, this study is the first to investigate the robustness of visually grounded dialog models towards textual attacks. We first aim to understand how multimodal input components contribute to model robustness. Our results show that models which encode dialog history are more robust by providing redundant information. This is in contrast to prior work which finds that dialog history is negligible for model performance on this task. We also evaluate how to generate adversarial test examples which successfully fool the model but remain undetected by the user/software designer. Our analysis shows that the textual, as well as the visual context are important to generate plausible attacks.

heading: 1 Introduction
text: Adversarial robustness has recently gained increased attention within the NLP community (e.g. Moradi and Samwald, 2021; Chang et al., 2021; Goel et al., 2021; Wang et al., 2021). In contrast to this previous wok, which focuses on text-only models, we evaluate the adversarial robustness of Visual Dialog (VisDial) models with the aim to understand how different input components contribute to robustness. For example, it has previously been established that multiple input modalities increase robustness of pre-neural conversational interfaces, e.g. (Oviatt, 2002; Bangalore and Johnston, 2009). Here, we want to know which modalities can mitigate input attacks on neural visual dialog systems, and to what extent. This is important, since worst-
∗ Work done at Heriot-Watt University. † Now at Google DeepMind.
case input permutations can also happen spontaneously during user interaction.
A successful attack in our setting is a perturbation which changes the model prediction, but, at the same time, remains unnoticed by the user/ software developer. While this is relatively simple for pixel-level visual attacks, textual perturbations are substantially harder to conceal. In the following we thus investigate the trade-off between effectiveness and linguistic quality for text-based attacks. The focus on text-only attacks is further motivated by our preliminary experiments, which show that encoding visual information has very little gain for performance and robustness (around 5%, cf. Table 1), confirming previous results (Massiceti et al., 2018; Agarwal et al., 2020). In addition, we argue that black box visual attacks are very unlikely (if not impossible) to occur in the real-world: They are highly inefficient and expensive to run at real-time (mainly due to the way features are pre-computed using FasterRCNN).
3422
To the best of our knowledge, we are the first to explore adversarial attacks on VisDial, which was introduced as a shared task by (Das et al., 2017a). A visual dialog system consists of three components: an image (with a caption), a question and the dialog history, i.e. previous user and system turns. In order to answer the question accurately, the AI agent has to ground the question in the image and infer the context from history, see Fig. 1. Most existing research has focused on improving the modelling performance on this task, (e.g. Das et al., 2017b; Kottur et al., 2018; Jain et al., 2018; Zheng et al., 2019; Niu et al., 2019; Yang et al., 2019; Qi et al., 2020; Murahari et al., 2020; Agarwal et al., 2020; Nguyen et al., 2020; Wang et al., 2020; Chen et al., 2022a), whereas our aim is to evaluate adversarial robustness. In addition, we use these attacks to improve our understanding of how the model works (i.e. interpretability). Our main contributions are: • We show that dialog history contributes to model robustness: We attack ten VisDial models which represent a snapshot of current methods, including different encoding and attention mechanisms, as well as graphical networks and knowledge transfer using pretraining. We measure the performance change before and after attack and show that encoding history helps to increase the robustness against adversarial questions by providing redundant information. We also show that models become more uncertain when the history is attacked. • We evaluate adversarial text-generation within VisDial: We leverage well-established Synonym Substitution methods for adversarial black-box attack (Jin et al., 2020; Li et al., 2020) and show that BERT-based models are able to generate more contextually coherent perturbations. We also conduct a detailed analysis to study the trade-off between the effectiveness of the attack versus the text quality. • We show that human evaluator are able to identify an attack from the textual and multimodal context: We conduct a detailed human evaluation to investigate the trade-off between successful attacks and their ability to remain unnoticed by humans. In particular, we evaluate semantic similarity, fluency/grammaticality and label consistency. All code will be made available.

heading: 2 Related Work
text: Adversarial attacks have been widely investigated within uni-modal applications, foremost for com-
puter vision (Narodytska and Kasiviswanathan, 2016; Dong et al., 2018; Xie et al., 2019; Mahmood et al., 2021). Adversarial attacks on text are more challenging due to its discrete nature, which makes it harder to stay undetected. Adversarial textual attack methods can be divided into three levels of granularity (Zhang et al., 2020; Wang et al., 2019): character-level, word-level and sentence-level attacks. While the former two are relatively easy to detect, recent word-level attack methods (e.g. Lee et al., 2022; Zang et al., 2020; Jin et al., 2020; Li et al., 2020; Ren et al., 2019), on the other hand, are more subtle: they are targeted towards ‘vulnerable’ words, which are substituted via their synonyms in order to preserve semantic meaning. In our paper, we explore word-level attack methods on VisDial.
Research on adversarial attacks for multi-modal tasks is relatively scarce, including studies for Optical Character Recognition (Song and Shmatikov, 2018), Scene Text Recognition (Yuan et al., 2020), Image Captioning (Chen et al., 2017, 2022b) and VQA (Li et al., 2021; Shi et al., 2018). Most of these works utilise white box attack, where the parameters, gradient and architecture of the model are available, e.g. by attacking attention (Xu et al., 2018; Sharma et al., 2018). Whereas we follow a more realistic black-box setting which assumes that the attacker only has access to the model’s prediction on test data.
Shi et al. (2018) is closest related to our work: they generate adversarial textual attacks for the VQA task using contrastive examples and thus don’t pay attention to semantic similarity. In contrast, we are interested in generating adversarial attacks which follow three desiderata, as outlined by (Morris et al., 2020): An adversarial text should (1) keep the same semantic meaning (semantic similarity); (2) guarantee fluency and grammar (grammaticality); (3) stay unnoticed by humans, i.e. the human still assigns the correct label, while the model prediction changes (label consistency).

heading: 3 Method

heading: 3.1 Problem Formulation
text: VisDial is formulated as a discriminative learning task, where the model is given an image I , the dialog history (including the image caption C) H = ( C︸︷︷︸
H0 , (Q1, A1)︸ ︷︷ ︸ H1 , ..., (Qt−1, At−1)︸ ︷︷ ︸ Ht−1 ), the
question Qt, and N = 100 candidate answers At = (A 1 t , A 2 t , ..., A 100 t ) to rank, including the
ground truth (GT), which is labelled Yt, where t indicates the round ID.
In the following, we focus on generating textual adversarial examples for the question and history (including the caption). That is, for a sentence X ∈ {Q,H}, and F (X) = Y , a successful adversarial attack sentence Xadv should result in F (Xadv) ̸= Y , while meeting the following requirements:
• Semantic Similarity: Sim(X,Xadv) ≥ ε, where Sim(·) is a semantic and syntactic similarity function. The semantic similarity between the original sentence X and the adversarial attack sentence Xadv should above a similarity threshold ε; Following (Jin et al., 2020), we use Universal Sentence Encoder (Cer et al., 2018) to encode the two sentences into high dimensional vectors and use their cosine similarity score as an approximation of semantic similarity. • Grammaticality: The adversarial attack sentence Xadv should be fluent and grammatical. • Label Consistency: Human annotators still assigns the correct GT label Y after the original sentence X changes to Xadv.

heading: 3.2 Visual Dialog Models
text: We explore a wide range of ten recent VisDial models to attack – representing a snapshot of current techniques and architectures popular for VisDial.1
Agarwal et al. (2020) experiment with several multi-modal encodings based on Modular CoAttention (MCA) networks (Yu et al., 2019b): MCA-I encodes the image and question representation using late fusion; MCA-H only encodes the textual history with late fusion; MCA-I-H encodes image and history with late fusion; MCA-I-HGQ encodes all three input modalities using early fusion between question and history; MCA-I-VGH is another early fusion variant which first grounds the image and history.
We also consider Recursive Visual Attention (RvA) (Niu et al., 2019) as an alternative to MCA, encoding history and image information.
In addition, we test two variants of causal graphs from (Qi et al., 2020) by adding to causal principles P1/P2: P1 removes the history input to the model to avoid a harmful shortcut bias; P2 adds one new (unobserved) node U and three new links to history, question and answer respectively.
1Details on model architecture can be found in the original papers.
Finally, we test a Knowledge Transfer (KT) method based on Sparse Graph Learning (SGL) (Kang et al., 2021), which uses P1/P2 as pre-trained models.

heading: 3.3 Synonym-based Methods
text: For generating attacks, we explore two popular synonym-based methods, which first find the vulnerable words of the sentence, and then replace them with a semantically similar word.2
• TextFooler (Jin et al., 2020) performs embedding-similarity-based perturbations as a constraint to generate semantically consistent adversarial examples. • BERT-Attack (Li et al., 2020) generates context-aware perturbations using BERT.
Following these previous works, we first detect vulnerable words by calculating prediction change before and after deleting a word. We then impose additional constraints to improve the quality (in particular the grammaticality) of our attacks, which we further analyse in Section 6. We apply a stop word list before synonym substitution, extending the list by (Jin et al., 2020; Li et al., 2020) for our domain. We also filter antonyms following the original BERT-Attack implementation (Li et al., 2020). We then apply additional quality checks for selecting synonym candidates: We filter by partof-speech (POS)3 to maintain the grammar of the sentence. We then experiment with a semantic similarity threshold ε to choose the top k synonyms. Finally, we iteratively select the word with the highest similarity until the attack is successful. See Appendix B for further details.

heading: 3.4 Adversarial Attack on Visual Dialog Models
text: We perform attacks on 2 textual inputs to the model: The current question and the previous history.

heading: 3.4.1 Question Attack
text: We first attack the current input question in VisDial. In order to do this, we have to adapt textual attacks methods to the unique challenges of this setting, which differs from other common textual attacks (such as sentiment classification, image captioning or news classification) both in terms of textual
2Note that previous work refers to these methods as “synonym-based”, e.g. (Morris et al., 2020), but not all of the substitutions are synonyms. They can also include different lemmatas of the same lexeme, such as singular and plural, as well as different spellings, etc. Also see Table 8.
3Using SpaCy https://spacy.io/api/tagger.
input, as well as in predicted output: First, the question in VisDial is generally much shorter than a typical declarative sentence in the above tasks. The average length of the question in the VisDial dataset is 6.2 words, which makes it harder to find a word to attack. For instance, “Is it sunny?”, “What color?”, “How many?”, there is only one word left to attack after filtering out the stop words, i.e. {is, it, what, how}.
For the VisDial task, the model ranks N possible candidate answers according to its log-likelihood scores. The attack is considered successful once the top ranked answer differs from the GT. However, there can be several candidate answers which are semantically similar or equivalent, such as “yes/yep/yeah”. This is different from other labelling tasks, such as “positive/neutral/negative” sentiment. We account for this fact by considering several common retrieval metrics before and after the attack, including R@k (k=1,5,10), Mean Reciprocal Rank (MRR), and Normalized Discounted Cumulative Gain (NDCG) – a measure of ranking quality according to manually annotated semantic relevance scores in a 2k subset of VisDial.

heading: 3.4.2 History Attack
text: We also attack the textual history using the same procedure. The use of history is the main distinguishing feature between the VisDial and the VQA task, and thus of central interest to our work. History is mainly used for contextual question understanding, including co-reference resolution, e.g. “What color are they?”, and ellipsis, e.g. “Any others?” (Yu et al., 2019a; Li and Moens, 2021).
Our preliminary results indicate that attacking history is hardly ever successful, i.e. does not result in label change. This is in line with previous work, which suggests that history only plays a negligible role for improving model performance on the VisDial task, e.g. (Massiceti et al., 2018; Agarwal et al., 2020). However, there is also some evidence that history helps, but to a smaller extent. For example, Yang et al. (2019) show that accuracy can be improved when forcing the model to pay attention to history. Similarly, Agarwal et al. (2020) show that history matters for a sub-set of the data.
In a similar vein, we investigate how history contributes to the model’s robustness and, in particular, can increase the model’s certainty in making a prediction. We adopt the perplexity metric, following (Sankar et al., 2019), to measure the change of prediction distribution after (unsuccessfully) attack-
ing the history, i.e. after adding the perturbation to the history while the top-1 prediction is unchanged. The difference between the perplexity before and after the attack reflects the uncertainty change of the model. The perplexity with the original history input is calculated with the following equation:
PPL(F (X), Y ) = − ∑
X
F (X)log2Y (1)
And the perplexity after attack is:
PPL(F (Xadv), Y ) = − ∑
Xadv
F (Xadv)log2Y (2)

heading: 4 Experimental Setup

heading: 4.1 Dataset
text: We use the VisDial v1.0 dataset, which contains 123,287 dialogs for training and 2,064 dialogs for validation. The ten target models are trained on the training set and the adversarial attacks are generated for validation set (as the test set is only available to challenge participants).

heading: 4.2 Automatic Evaluation Metrics
text: In order to assess the impact of an attack, we use the automatic evaluation metrics from (Jin et al., 2020): The accuracy of the model tested on the original validation data is indicated as original accuracy and after accuracy on the adversarial samples – the larger gap between these two accuracy means the more successful of our attack (cf. relative performance drop [∆]). The perturbed word percentage is the ratio of the perturbed words and the length of the text. The semantic similarity measures the similarity between the original text and the adversarial text by cosine similarity score. The number of queries shows the efficiency of the attack (lower better). In addition, we use retrieval based metrics to account for the fact that VisDial is a ranking task: original/after R@{5, 10} measures the performance of top 5/10 results before and after attack (where R@1 corresponds to accuracy); we also report original/after mean reciprocal rank (MRR) and original/after Normalized Discounted Cumulative Gain (NDCG) which measure the quality of the ranking. Detailed results with R@k (k=10) are shown in Appendix B and C. Further implementation details are given in Appendix A.

heading: 5 Results

heading: 5.1 Question Attack
text: Table 1 summarises the results for attacking the question. Across the ten target models, we first
compare different input encodings and fusion mechanisms, answering the question whether multiple inputs can help robustness. We find that MCAI (with image input only) is the least robust model with a relative performance drop of over 22% on R@1 using TextFooler. MCA-H (with no image input) is vulnerable with respect to R@1, but does well on NDCG, suggesting that history helps to produce a semantically similar response despite the lack of encoding the input image. One possible explanation of these results is given by previous research claiming that VisDial models mainly pay attention to text while ignoring the image, e.g. (Massiceti et al., 2018). However, in contrast to claims by (Massiceti et al., 2018), we find that history is important for robustness: In general, models encoding history are more robust with the MCA-I-H model being the least vulnerable model. Note that this is also the best performing model in (Agarwal et al., 2020).
Next, we compare attention mechanisms on the input encodings. Recursive visual Attention (RvA) in general shows lower robustness than MCA-based methods. Causal encodings using graphs lead to comparable robustness results for P1. Adding P2 results in a slight drop in robustness. This is interesting, because P2 adds an unobserved node to represent history while avoiding spurious
correlations from training data. This drop thus might suggest that previous robustness is due to the very same bias. Additionally, we observe that knowledge transfer (KT) via pre-training for the SLG method helps to boost the performance of NDCG, however not the robustness.
We further perform an example based analysis of the top-1 predicted answer changes after a successful question attack, see Fig. 2. We observe answer changes to the opposite meaning (e.g. from “no” to “yes”), which can be considered as a maximum successful attack. Some answers, however, change to a very similar meaning in context (e.g. from “No pets or people” to “No”), which is reflected in fewer NDCG changes. In some cases, the answer changes from certain / definite to uncertain / noncommittal and the other way round (e.g. from “white” to “Not sure”).
Next, we compare the two attack methods. We find that TextFooler is more effective: It achieves up to 4.5% higher drop than BERT-Attack. However, BERT-Attack is more efficient: It reduces the number of queries (Quer.) about four times compared to TextFooler. Efficiency is important in attack settings, as attackers always run into danger of being discovered. Furthermore, the perturbed word percentage (Pert.) for both methods is around 17%, which means the average perturbation is about one word for each question (since the average length of the question is 6.2). Similarly, the semantic similarity (S.S.) is over 70% which is about the same across all models.
We further compare TextFooler and BERTAttack using an example-based analysis, see Fig. 3. We find that TextFooler is not able to distinguish words with multiple meanings (homonyms), whereas BERT-Attack is able to use BERT contextembeddings to disambiguate. Consider the examples where TextFooler replaces “flat” (adverb) with “loft” (noun) and “faces” (noun) with “confront” (verb), which POS tagger failed to catch.
Based on the above results, we use BERTAttack to attack the MCA-I-H model in the following experiments.

heading: 5.2 History Attack
text: Next, we analyse the results for attacking dialog history. As explained in Section 3.4.2, we consider an attack ‘successful’ once the probability of the corresponding GT decreases and we use perplexity to measure the uncertainty of the prediction. The results in Table 2 show that attacking history increases the uncertainty of almost all the models, especially when the history is the unique input component (MCA-H model).4 This confirms our preliminary results that encoding history increases robustness.
We then analyse which part of history gets attacked the most, see Table 3. We find that 44.9% of the time the image caption was attacked, followed by system answer 30.8% and user question 24.3%. We thus conclude that the image caption, i.e. a description of what can be seen in the picture, is the most vulnerable part (and ergo the most informative) compared to the rest of history. We hypothesise that the image caption can “replace” information corrupted by the attack. Thus, history contributes to robustness by providing redundant information.

heading: 6 Detailed Analysis of Linguistic Quality Constraints
text: Next, we analyze the impact of the linguistic quality constraints. We are interested in the trade-off between using these constraints to produce high quality text (which increases the chance of the attack to remain unnoticed by humans) versus an effective attack (which increases the chance of the model changing its prediction).
4Attacking the history of MCA-I-VGH model doesn’t change the prediction distribution because its encoder only uses a single round of history following (Agarwal et al., 2020).
Effect of Selecting Vulnerable Words First, we compare the results of choosing a random word in text to attack and our vulnerable word attack. The results in Table 4 confirm that attacking the vulnerable word achieves a 2.0% higher relative drop for R@1, NDCG and MRR.
Effect of Stop Words Next, we compare the results with/without stop words. The results in Table 5 show that attacking all words leads to more successful attack in terms of R@1 and NDCG, while attacking with stopwords leads more successful attacks for MRR. We use stop words list for all the experiments since attacking question words, preposition or pronouns result in highly ungrammatical sentences.
Effect of Semantic Similarity The semantic similarity threshold between the original text and adversarial text is used to guarantee the similar meaning of the attack. In the previous experiments, we set 0.5 as default threshold. Table 6 shows results with different semantic similarity thresholds (0.1, 0.3, 0.5 and 0.7) respectively. The results show that when increasing the threshold ε from 0.1 to 0.7, the number of successful attack decreases 4.1%, while R@1 and NDCG drop around 3% after attack, which means there are more successful attacks if we loosen the semantic similarity constraint. In addition, the examples in Fig. 4 illustrate that a lower semantic similarity threshold comes at the cost of lower fluency and grammaticality, i.e. at the price of being more easily detectable by humans. We will explore this trade-off in more detail in the human evaluation study.
Next, we analyze the combined effect of adding POS, semantic similarity constraint and grammar

heading: 7 Human Evaluation Study
text: We evaluate the quality of our generated adversarial question attack by asking human judges on Amazon Mechanical Turk (AMT) to rate three aspects: if the generated question preserve the semantic similarity (semantic similarity with/without given image); if the generated question is natural
and grammatical (grammaticality); if the human’s prediction is unchanged for the generated question (label consistency). We evaluate a total of 198 generated attacks, randomly sampled from the development set, where three users are asked to rate each instance, following best practices for human evaluation in NLP (van der Lee et al., 2019). 5 See Appendix for further details.
Evaluation of Semantics We first ask crowd workers to evaluate whether the original and the adversarial question still have the same meaning on a scale from 1 to 4, where 1 is “One text means something completely different” and 4 is “They have exactly the same meaning”. Appendix E shows the interface and instructions. We elicit ratings with and without showing the image in order to measure the effects of multimodal grounding. Our results show that the semantic similarity is rated slightly lower when shown together with the original image (average score 3.518 / 4) than without image (average score 3.564 / 4). This indicates that the visual context can change the semantic similarity ratings, as illustrated in Fig. 5. Therefore, a promising future avenue is to use visually grounded word embeddings for generating synonyms for V+L tasks.
Evaluation of Grammaticality Next, we evaluate whether the utterance is fluent and grammatical (as defined in Appendix E) on a scale from 1-5, where 1 is “Not understandable” and 5 is “Everything is perfect; could have been produced by a native speaker”. Overall, our attacks are rated as highly grammatical (average score 4.429 / 5). We furthermore investigate the effect of different attacks, where we manually label five common types of successful attacks, see Table 8. We find that
5For example, van der Lee et al. (2019) report that standard evaluations only include an average of 100 samples rated by a median of 4 annotators. We chose to increase the sample size in order to increase the effect and chose 3 annotators in order to allow for ties.
synonyms/near synonyms is the main type of attack, closely followed by British vs. American English (e.g. “color” vs. “colour”). In addition, we identify Singular vs. Plural, and Comparatives and Superlatives (e.g. “great/greater/greatest”) , as well as Others which mainly include grammar operations like uncaught POS change (e.g. “sunny” vs. “sun”) and tense change (e.g. “eat” vs. “ate”). Looking at the grammar ratings, we find that substituting British vs. American English has the least impact on grammaticality, whereas grammatical operations, such as replacing singular with plural, as well as changes classified under Others have the highest impact on the perceived linguistic quality.
Evaluation of Label Consistency Finally, we evaluate label consistency by asking users to judge whether the answer remains unchanged for the adversarial question by selecting among “1 - Yes, answer is correct”, “2 - No, answer is incorrect” and “3 - Unsure” as shown in Appendix E. We ask three judges to rate each instance and describe results by averaging and by (a more conservative) majority vote to assign a gold label. The results show that most (82.0% by averaging and 86.4% by majority vote) crowdworkers think the answer is unchanged, few (9.6% and 8.1%) think the answer changes, and the rest (8.4% and 5.5%) are not sure about the change. We thus conclude that synonym-based attacks mostly remain undetected by humans.

heading: 8 Conclusions and Limitations
text: We present a detailed study investigating adversarial robustness of visual dialog models. We find that multiple inputs increase robustness, and in particular dialog history contributes to robustness, despite previous results which suggest that history has negligible effect on model performance, e.g. (Massiceti et al., 2018; Agarwal et al., 2020). We investigate the trade-off between effectiveness and linguistic quality, where we show limitations of current synonym-based textual attack models, and stress the importance of context (both textual as well as multi-modal) to generate semantically coherent and grammatically fluent adversarial attacks, which are likely remain undetected. While the observed effects of visually-grounded interpretations in our human evaluation were relatively small, we do believe that it is an important future direction. For example, we expect improved results by using synonym substitution methods based on visuallygrounded word embeddings.
Ethics Statement
We use adversarial attack as a tool to evaluate the robustness of visual dialog models. However, the same techniques can also be used to maliciously attack the system. Our experiments demonstrate that most synonym-based attacks are successful in remaining undetected by humans. However, our results also show that the most effective attacks are also the ones which are easiest for humans to detect. Further work is thus needed to automatically detect malicious attacks, e.g. using our proposed grammaticality and contextual multimodal methods.

heading: Acknowledgments
text: Lu Yu and Verena Rieser were supported by the EPSRC project ‘AISEC: AI Secure and Explainable by Construction’ (EP/T026952/1), Lu Yu was also supported by NSFC (NO.62202331), Verena Rieser was also supported by ‘Gender Bias in Conversational AI’ (EP/T023767/1) and ‘Equally Safe Online’ (EP/W025493/1), as well as by a Leverhulme Trust Senior Research Fellowship (SRF/R1/201100).

heading: A Licence
text: Visual Dialog annotations and this website are licensed under a Creative Commons Attribution 4.0 International License.
B Implementation Details
All models are implemented with Pytorch. We embedded BERT-Attack and TextFooler to our VisDial system6. We initially set the semantic similarity threshold 0.5 for attacking both question and history (but see detailed study of different threshold in Table 6).
To ensure the high quality of generated attack, We did the following modifications for original BERT-attack method:
• We updated the stop-word list by adding some new words in and removing some words out due to the visual dialogue domain, compared to the original stop-word list7. • We filter by part-of-speech (POS) to maintain the grammar of the sentence for selecting synonym candidates. • We used the grammar tool from (Morris et al., 2020) to check the sentence after the attack.
6BERT-Attack code from https://github.com/ LinyangLee/BERT-Attack and TextFooler code from https://github.com/jind11/TextFooler.
7The stop-word list is updated to {does, do, see, look, seem, be, some, sort, have, has, had, so, a, about, above, across, after, afterwards, again, against, ain, all, almost, alone, along, already, also, although, am, among, amongst, an, and, another, any, anyhow, anyone, anything, anyway, anywhere, are, aren, aren’t, around, as, at, back, been, before, beforehand, behind, being, below, beside, besides, between, beyond, both, but, by, can, cannot, could, couldn, couldn’t, d, didn, didn’t, doesn, doesn’t, don, don’t, down, due, during, either, else, elsewhere, even, ever, everyone, everything, everywhere, except, for, former, formerly, from, hadn, hadn’t, hasn, hasn’t, haven, haven’t, he,hence, her, here, hereafter, hereby, herein, hereupon, hers, herself, him, himself, his, how, however, i, if, in, indeed, into, is, isn, isn’t, it, it’s, its, itself, just, latter, latterly, least, ll, may, me, meanwhile, mightn, mightn’t, mine, moreover, must, mustn, mustn’t, my, myself, namely, needn, needn’t, neither, never, nevertheless, next, no, nobody, none, noone, nor, not, nothing, now, nowhere, o, of, off, on, once, only, onto, or, other, others, otherwise, our, ours, ourselves, out, over, per, please,s, shan, shan’t, she, she’s, should’ve, shouldn, shouldn’t, somehow, something, sometime, somewhere, such, t, than, that, that’ll, the, their, theirs, them, themselves, then, thence, there, thereafter, thereby, therefore, therein, thereupon, these, they,this, those, through, throughout, thru, thus, to, too,toward, towards, under, unless, until, up, upon, used, ve, was, wasn, wasn’t, we, were, weren, weren’t, what, whatever, when,whence, whenever, where, whereafter, whereas, whereby, wherein, whereupon, wherever, whether, which,while, whither, who, whoever, whole, whom, whose, why,with, within, without, won, won’t, would, wouldn, wouldn’t, y, yet, you, you’d, you’ll, you’re, you’ve, your, yours, yourself, yourselves.}

heading: C Full Table of Question Attack
text: We show the full table of question attack results including R@10 in Table 9 as supplement of Table 1.

heading: D Detailed Results for Lingustic Analysis
text: The full tables of the in-depths studies are shown in Table 10, Table 11, Table 12 and Table 13, as supplement Table 4, Table 5, Table 6, Table 7 respectively.

heading: E Details of Human Evaluation Study
text: Here, we provide more details on the human study. We show the interface of semantic similarity experiment for AMT task in Figure 7, including the instruction (top). Two versions of this interface are conducted, where one is provided with image, one is without image. The interface of fluency/grammaticality experiment for AMT task is shown in Figure 8. Two versions of this interface are done as well, where one is with grammar checker and one is without. Finally, the interface of label consistency experiment is shown in Figure 9.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: section 8

heading: 3 A2. Did you discuss any potential risks of your work?
text: Section Ethics Statement

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Abstract and section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 7 Did you use or create scientific artifacts? Left blank.

heading: 3 B1. Did you cite the creators of artifacts you used?
text: all

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: appendix A

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Section 4.1
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Section 4
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. section 4.1
C 3 Did you run computational experiments? Section 5

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Appendix B
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Appendix B

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 5 and 7

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 5 and appendix B
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: While the observed effects of visually-grounded interpretations in our human evaluation were relatively small, we do believe that it is an important future direction. For example, we expect improved results by using synonym substitution methods based on visuallygrounded word embeddings