Human-Adversarial Visual Question Answering
Authors: Sasha Sheng
Amanpreet Singh
Vedanuj Goswami
Jose Alberto Lopez Magana
Tristan Thrush
Wojciech Galuba
Devi Parikh
Douwe Kiela
Venue: NeurIPS 2021 Poster (2021)
Keywords: vqa
visual question answering
adversarial evaluation
model in the loop

Abstract: Performance on the most commonly used Visual Question Answering dataset (VQA v2) is starting to approach human accuracy. However, in interacting with state-of-the-art VQA models, it is clear that the problem is far from being solved. In order to stress test VQA models, we benchmark them against human-adversarial examples. Human subjects interact with a state-of-the-art VQA model, and for each image in the dataset, attempt to find a question where the model’s predicted answer is incorrect. We find that a wide range of state-of-the-art models perform poorly when evaluated on these examples. We conduct an extensive analysis of the collected adversarial examples and provide guidance on future research directions. We hope that this Adversarial VQA (AdVQA) benchmark can help drive progress in the field and advance the state of the art.

Human-Adversarial Visual Question Answering
Sasha Sheng!
Amanpreet Singh
Vedanuj Goswami
Jose Alberto Lopez Magana
Tristan Thrush
Wojeiech Galuba
Devi Parikh
Douwe Kiela
Facebook AI Research
Tecnológico de Monterrey
8 Georgia Tech
https://adverssaiillq...org
Abstract
Performance on the most commonly used Visual Question Answering dataset (VQA v2) is starting approach human accuracy. However, in interacting with state-of-the-art VQA models, it clear that the problem is far from being solved. In order to stress test VQA models, we benchmark them against auvan-adversarial examples. Human subjects interact with a state-of-the-art VQA model, and for each image in the dataset, attempt to find question where the model's 's predicted answer is incorrect. We find that a wide range of state-of-the-ar models perform poorly when evaluated on these examples. We conduct an extensive analysis of the collected adversarial examples and provide guidance on future research directions. We hope that this Adversarial VQA (AdVQA) benchmark can help drive progress in the field and advance the state of the art.
Introduction
Visual question answering (VQA) is widely recognized as an important evaluation task for vision and language research, Besides direct applications such as helping the visually impaired or multimodal content understanding the web, it offers a mechanism for probing machine understanding of images via natural language queries. Making progress on VQA requires bringing together different subfields in AI combining advances from natural language processing (NLP) and computer vision together with those in multimodal fusion making an exciting task in AI research.
Over the years, the performance of VQA models has started to plateau on the popular VQA v2 dataset [20] - approaching inter-human agreement as evidenced by Fig. This raises important questions for the field: To what extent have we solved the problem? If we haven't, t, what are we still missing? How good are we really?
An intriguing method for investigating these questions is dynamic data collection [29], where human annotators and state-of-the-ar models are put "in the loop" together to collect data adversarially. Annotators are tasked with and rewarded for finding model-fooling examples, which are then verified by other humans. The easier it is to find such examples, the worse the model's performance can be said to be. The collected data can be used to "stress test" current VQA models and serve as the next iteration of the VQA benchmark helping drive further progress.
The commonly used VQA dataset [20] was collected by instructing annotators to "ask a question about this scene that [a] smart robot probably can not answer" [4]. One way of thinking about our proposed human- human-adversee adversarial data collection is that it explicitly ensures that the questions can not be answered by today's "smartest" models.
"Equal contribution. Correspondence advga@fb.com. t Work done intern Facebook Research.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Table 1: Contrastive examples from VQA and AdVQA. Predictions given for the VisualBERT, ViLBERT and UniT models, respectively. Models can answer VQA questions accurately, but consistently fail on AdVQA questions.
Image
VQA
AdvQA
Q: How many cats are in the image?
Q: What brand is the tv?
A: Ig
Model: sony, samsung, samsung
Model: 2, 2,
Q: Does the cat look happy?
Q: How many cartoon drawings
A:
present the cat's tic?
Model: no, no,
A: 4
Model: 1.2
Q: What kind of floor is the sitting on?
Q: Did someone else take this picture?
wood
A: Model: yes, yes, yes
Model: wood, wood, wood
This work is, the best of knowledge, the first apply this human-adversarial approach to an image and language multimodal problem. We introduce Adversarial VQA (AdVQA), a large evaluation dataset of 46,807 examples in total, all of which fooled the VQA 2020 challenge winner, the Mo MCAA [45] model.
We evaluate wide range of existing VQA models on AdVQA and find that their performance is significantly lower than on the commonly used VQA v2 dataset [20] (see Table 1). Furthermore, we conduct an extensive analysis of AdVQÅ characteristics, and contrast with the VQA v2 dataset.
We hope that this new benchmark can help advance the state of the art by shedding important light on the current model shortcomings. Our findings suggest that there is still considerable room for continued improvement, with much more work remaining to done.
2 Related Work
Stress testing VQA. Several attempts exist for stress testing VQA models. Some examine to what extent VQA models' predictions are grounded in the image content, as opposed them relying primarily on language biases learned from the training dataset. The widely used VQA v2 dataset [20] was one attempt at this. The dataset contains pairs of similar images that have different answers to the same question, rendering language- based prior inadequate. VQA under changing priors (VQA-CP) [2] is more stringent test where the linguistic prior not only is weak in the test set, but is adversarial relative to the training dataset (i.e., answers that are popular for question type during training are rarer in the test set and vice versa). Other complementary datasets test model robustness against certain conditions, such as different logicial compositions [18], question

---

rephrasings [52], suprious correlations [1], and consistency in answering sub-questin sub-questions [51]. Datasets which benchmark specific capabiliities in VQA models also exist, such as reading and reasoning
about text in images [57], leveraging external knowledge [44], and spatial [26] or compositional
reasoning [28]. Other vision and language datasets, such as Hateful Memes [31], have also tried to
make sure the task involves true multimodal reasoning, as opposed to any of the individual modalities
sufficing for arriving at the correct label. Compared to the previous work, with AdVQA, we aim to have a more holistic and general dataset that pushes the boundaries of the late-of-the-art on VQA.
Saturating prior work. The VQA v2 [20] challenge has been running yearly since 2016 and has seen tremendous progress, as can be seen in Figure From simple LSTM [23] and VGGNet [54] fused models to more advanced fusion techniques (MCB [16]. Pythia [27]) to better object detec- tors (MCAN [61], Mo MONA [45], BUTD [3] to transformers [59, 56] and self-supervised pretraining (MCAN [61], UNIMO [38]), the community has brought the performance of models on the VQA v2 dataset close to human accuracy, thus starting saturate progress on dataset. As we show through AdVQA, however, saturation is far from achieved on the overall task, and hence there a need for new datasets continue benchmarking progress vision and language reasoning.
Adversarial datasets. As AI models starting to work well enough in some narrow contexts for real world deployment, there increasing concerns about the robustness of these models. This has led to fertile research both in designing adversarial examples to "attack" models (e.g., minor imperceptible noise added to image pixels that significantly changes the model's prediction [19]), as well as in approaches to make models more robust to "defend" against these attacks [9, 12]. The human-in-tee -i-tn-ii-tth--ooy thop adversarial setting that we consider in this paper qualitatively differ- ent from the statistical perturbations typically explored when creating adversarial examples. This type of human-in- somaa--n--the-loy adversarial data collection has been explored in NLP c.g..
80
VOA Accuracy
70
Human
65
UNIMO
MeVIE-MCAN
MCAN
60
Pythia
BUTD
55
MCB
2015
2016
2017
2018
2019
2020
2021
natural language inference [46], question an- swering [5, 50], sentiment analysis [49], hate speech detection [60], next video-and-lane
Figure 1: Progress on the VQA v2 dataset over time.
event prediction [35] and dialogue safety [15]. To
the best of our knowledge, ours is the first work to explore an image and language multimodal suman-adversane benchmark.'
3 AdVQA
The aim of this work is to investigate state of the VQA model performance via human-adversarial data collection. In this samaan-ndd-ooeel----hh----e finding examples that fool the model. In this case, annotators shown image and are tasked with asking difficult but valid questions that the model answers incorrectly. We collected our dataset using Dynabench [29], a platform for dynamic adversarial data collection and benchmarking. For details on our labeling user interface, please refer to the supplementary material.
paradigm, human annotators are tasked with
The VQA v2 dataset is based COCO [39] images. We collected adversarial questions on both val2017 COCO images and testdev2015 COCO images. We then random sampled the collected set down to 2 questions per val2017 COCO images (10,000) and question per testdev2015 COCO images (36,807) The random sampling was done to balance the annotator's contributions. The data collection involves three phrases (Figure 2): First, in the question collection phase, Amazon Mechanical Turk workers interact with state-of-the-art VQA model and self-report image-question pairs that the model failed to answer correctly. Second, in the question validation phase, set
CConcureett work [36] jointly available with AdVQA at https://adversarialvg..or also focuses on the same problem but uses images from a different source.
3

---

imege
mozell
validarors
answerers
yes
yes
writer
по
question
check model prediction
Figure 2: Diagram of the AdVQA human-adversarial data collection flow.
46
(b) Magnifier, available both in question collection. validation answer collection helps an- notators ask and answer more spe- cific questions.
(a) Main interface. The annotators follow the instructions provided to write the question in the text field provided which is then answered by the model-in-the-lop.
Figure Question collection interface showing the first stage of AdVQA collection setup.
of workers validates whether the answers provided by the VQA model for the image-question pairs collected in the first phrase are indeed incorrect. Finally, in the answer collection phase, we collect 10 ground truth answers for the image-question pairs validated the second phase. In what follows, we provide further details for each of these steps.
3.1 Question Collection
In this phase, annotators on Amazon MTTurk are shown an image and are tasked with asking questions about this image. The interface has state-of-the-art VQA model in the loop. For each question the annotator asks, the model produces an answer. Annotators are asked to come with questions that fool the model. Annotators are done (i.e., can submit the HIT) once they successfully fooled the model or after they tried a minimum of 10 questions (whichever occurs first). To account for the fact that it may be hard to think of many questions for some images, annotators are given the option to skip an image after providing non-fooling three questions. We use the VQA Challenge 2020 winner- MoViE+MCAN [45] trained on the COCO 2017 train set as the model the loop. This is to ensure that the adversarial benchmark we collect is challenging for the current state-of oftt--of-the-art in VQA.
The interface provides digital magnifier zoom into area of the image when the mouse hovers over it, allowing workers to examine the image closely if necessary. See Figure 3b. Keyboard shortcuts were provided for convenience.
3.2 Question Validation
Note that in the question collection phase, annotators self-report when they have identified question that incorrectly answered by the model. Whether or not this was actually the case is verified in the question validation phase. Two different annotators shown the image, question, and answer

---

predicted by the model, and asked whether the model's answer is correct. As an additional quality control, the annotator also asked whether the question is "valid". A valid question is one where the image is necessary and sufficient to answer the question. Examples of invalid questions are: "What is the capital of USA?" (does not need an image) or "What the person doing?" when the image does not contain any people or where there are multiple people doing multiple things. If the two validators disagree, a third annotator is used to break the tie, Examples are added to the dataset only if least two annotators agree that the question is valid.
3.3 Answer Collection
In the final stage of data collection, following [20, 57], we collect 10 answers per question providing instructions similar to those used in [20] making sure that no annotator sees the same question twice. In addition, as an extra step of caution, to filter bad questions that might have passed through last two stages and to account for ambiguity that can be present in questions, we allow annotators to select "unanswerable" as an answer. Further, to ensure superior data quality, we occasionally provide annotators with hand-crafted questions for which we know the non-ambiguous single true answer, as a means to identify and filter out annotators providing poor quality responses.
3.4 Human-AAverssarial Annotation Statistics
The statistics for the first two stages (question collection and validation) are shown in Table 2. We find that annotators took about tries and on average around 4 minutes to find model-fooling example. For computing the model error rate, we can look at the instances where annotators claimed they had fooled the model, and where annotators were verified by other annotators to have fooled the model. The latter has been argued to be particularly good metric for measuring model performance [29]. We also further confirm that the model was indeed fooled by running the podel-in-the-looe on subset of examples in which human agreement was 100% and found the accuracy to be 0.15% on val, and O. 18% on test splits of AdVQA respectively.
Table 2: AdVQA human-adversarial question collection and dataset statistics. The model error rate is the percentage of examples where the submitted questions fooled the model (either as claimed during question collection, or after validation). We also report the number of attempts (tries) needed before validated model-fooling example was found, and how long this took, in seconds
Total
Model error rate validated
Tries
Time in see
claimed
mean/median per ex.
208,932 40.94% (85,537) 36.17% (75,571) 5.33/4.0 203.22/107.26
Interestingly, the "claimed" model error rate, based on self-reported model-fooling questions, is similar that of text-based adversarial question answering, which was at 44.0% for RoBERTa and 47.1% for BERT [5]. The validated error rate for our task is much higher than e.g. for ANLI [46], which was 9.52% overall, suggesting that fooling models is lot easier for VQA than for NLI. appears to be not too difficult for annotators to find examples that the model fails to predict correctly.
4
Model Evaluation
4.1 Baselines and Methods
We analyze the performance of several basclines and wide variety of statc-of-the-art VQA models on the AdVQA dataset. We evaluate the same set of models on VQA v2 dataset as direct comparison.
Prior baselines. We start by evaluating two prior baselines: answering based on the overall majority answer, the per answer type majority answer in the validation dataset. We use the same answer types as in [4]. The overall majority answer in AdVQA is no. The majority answer is no for "yes/no", 2 for "numbers" and unanswerable for "others". See Section 5 for more details answer types.
Unimodal baselines. Next, we evaluate two unimodal pretrained models: i) ResNet-152 [22] pretrained on Imagenet [13]; and ii) BERT [14], both finetuned on the task using the visual (image)
5

---

or textual (question) modality respectively, while ignoring the other modality. We observe that the unimodal text model performs better than unimodal image for both VQA and AdVQA.
Multimodal methods. We evaluate two varieties of multimodal models: i) unimodal pretrained and ii) multimodal pretrained. In the unimodal pretrained category explore MMBT [30], MoViE+MCAN [45] and UniT [24]. These models are initialized from unimodal pretrained weights: BERT pretraining for MMBT; Imagenet + Visual Genome [34] detection pretraining for MoViE+MCAN; and Imagenet+ + COCO [40] detection pretraining for the image encoder part and BERT pretraining for the text encoder part in UniT. In the multimodal pretrained category, we explore VisualBERT [37], VilBERT [42, 43], ViIT [32]. UNITER[1 1] and VILLA [17]. These models are first initialized from pretrained unimodal models and then pretrained on different multimodal datasets on proxy self=supervise/semmi- sttee///eemmspprviiseeee supervised tasks before finetuning on VQA. VisualBERT is pretrained
on COCO Captions [10]: VilBERT is pretrained on Conceptual Captions [53]; ViLT, UNITER and VILLA models are pretrained on COCO Captions [10] + Visual Genome [34] Conceptual Captions [53] SBU Captions [47] datasets.
We find that multimodal models in general perform better than unimodal models, as we would expect given that both modalities are important for the VQA task.
Multimodal OCR methods. As we will see in Section 5, a significant amount of questions in AdVQA can be answered using scene text.
We test a state-of-the-ard TextVQA [55] model, M4C [25] on AdVQA. We evaluate two versions: (i) trained on VQA 2.0 dataset, and (ii) trained on TextVQA [55] and STVQA [6]. In both cases, we use OCR tokens extracted using the Rosetta OCR system [7]. We also use the same answer vocabulary used by other models for fair comparison.
Table 3: Model performance on VQA v2 and AdVQA val and test sets. indicates that this model architecture (but this model instance) used in the data collection loop.
Model
VQA
AdVQA
VQA
AdVQA
test-dev
test
val
Human performance
80.78
89.01
84.73
88.46
Majority answer (overall)
16.79
24.67
16.98
Mujority answer (per answer type)
31.86
31.01
33.38
Model in loop
MoViE+MCAN [45]
73.58
13.89
73.51
14.08
ResNet-152 [22]
26.66
20.59
24.85
19.02
Unimodal
BERT [14]
43.59
30.24
43.71
31.89
MoViE+MCAN" [45]
69.81
30.02
69.77
31.31
Multimodal
(unimodal pretrain)
MMBT [30]
49.27
30.80
49.36
32.57
UniT [24]
64.34
32.12
64.32
33.94
VisualBERT [37]
70.40
31.96
69.98
28.09
ViLBERT [42]
59.45
32.01
59.78
33.67
Multimodal (multimodal pretrain)
ViLT [32]
62.30
31.00
62.33
32.48
UNITER Base [11]
70.67
27.56
69.30
29.44
UNITER Large [11]
73.58
29.66
72.82
32.08
VILLA Base [17]
71.17
27.55
69.87
29.36
VILLA [17]
72.02
28.59
71.1
30.58
Multimodal
M4C (Tex(VQA+STVQA) [25]
32.89
33.84
31.44
34.05
(unimodal pretrain OCR)
M4C (VQA v2 train set) [25]
67.66
36.57
66,21
36,93
4.2 Discussion
Our model evaluation results show some surprising findings. We discuss observations and hypotheses around these findings in this section.
Baseline comparison. Surprisingly, most multimodal models are unable to outperform unimodal baseline (BERT [14]) and simple majority answer (per answer type) prior baseline which only predicts the most-frequently occurring word for the question's 's category (no for "yes/no", 2 for "numbers" and

---

umanswerable for "others"). A detailed breakdown of category- -eiee performance provided in Table 4 suggests that even though "yes/no" questions are often considered be easy, the baseline multimodal models we evaluate are unable to beat the majority answer prediction. We also observe varied but close to majority answer performance in the "numbers" category, which is question type known to be difficult for VQA models [45]. In contrast, the models outperform the majority answer in the "others" category even though "unanswerable" (the majority answer) is not in the answer vocabulary used. Interestingly, M4C outperforms all on "numbers" and "others" categories possibly thanks to its text reading capabilities. These trends showcase the difficulty of AdVQA and suggest that we have a long way go still, given that are as yet apparently unable to beat such simple baselines.
Model rankings. First, M4C [25] performs the best among the evaluated models. Interestingly, it is smaller than many of the more sophisticated model architectures that score higher on VQA. This is probably due to the importance of the ability to read and reason about text in the image for answering some AdVQA questions. Second, among models that can't 't read text, UniT [24] is the best model, despite (or perhaps because of?) being only unimodal pretrained. As UniT was trained jointly unimodal as well as multimodal tasks, perhaps, it is less prone to biases in multimodal datasets. Third, the adversarialll--rraine VILLA [17] model performs surprisingly poorly. While adversarial examples, it appears to be less so against
may be more robust to pattitticall-generaate suman-adversaan examples. Fourth, we find that all of these models perform poorly compared to humans, while model performance on VQA is much closer that of humans. A more detailed analysis on model capabilities required for AdVQA and BERT's s superior performance can be found in Section B.2 and B.3.
Human performance. Another surprising find- ing is that inter-human agreement is higher on the AdVQA dataset than on VQA. This could be
Table 4: The category-wise performance of VQA models. The state-of-the-art VQA models perform very close to the majority class prior. il- lustrating the challenge and difficulty of AdVQA.
due to different data annotation procedures, re- quirements on annotators or just statistical noise. culman-adversartii questions may also be more specific due to annotators having to make crisp decisions about the model failing not.
Question Type
Model
yes/no numbers
others
Model-in-the -otploo'--ss-ssp's performance. Interest- ingly, the MoViE+MCAN model that was not used in the loop and trained with a different seed, performs very similarly to other models. This suggests that to some extent, annotators overfit to the model instance. An alternative explanation is that model selection for all evalu- ated models was done the AdVQA validation set, which was (obviously) not possible for the model in the loop used to construct the dataset. In Adversarial NIJI [46], the entire model class of the in-the-loop model was affected. Note however that all VQA models perform poorly on AdVQA, suggesting that the examples are by and large representative of shortcomings of VQA techniques overall, and not of an individual model instance or class.
Majority Class
65.87
35.55
8.83
ResNet-152
65.72
0.35
0.19
BERT
67.25
30.27
14.39
VisualBERT
56.77
37.06
21.23
ViLBERT
57.09
35.19
20.00
MoViE+MCAN"
50.42
34.53
19.26
M4C (VQA2)
64.66
38.01
21.62
Train vs Test Distribution. We experiment with finetuning VisualBERT and ViIBERT further on the AdVQA val set, finding that this improved test accuracy from 31.96 to 39.49 and 32.01 to 38.11, respectively, suggesting a difference in the VQA and AdVQA distributions, as we would expect.
4.3 Training Details
We train all the models in Table the VQA train val split excluding the COCO 2017 validation images. We also add questions from Visual Genome [34] corresponding to the images that overlap with the VQA training set our training split. The VQA val set in our results contains all questions associated with the images in the COCO 2017 validation split. This is also consistent with the training split used for the model the loop. We collect our AdVQA validation set on the images from the COCO 2017 val split, ensuring there is no overlap with the training set images. We choose the best checkpoint for each model by validating on the AdVQA validation set.
7

---

quaetior sotuvalle
Cumulttae percet-spe of questies
VIA 2.0 A:VOA ToXEVQA
80
AdVeA
60
6
VQA
Adv04
20
TestV0A
2 11
183 za vo voo anz 1883
7
10
Number question
occuming
(b)
Figure 5: Quantitative statistics for AdVQA val set questions and answers showing longer question length, more diversity and better human- agreemeet. (a) Percentage of questions solvable with particular question length in AdVQA val set. We see that the average question length (7.82) in
AdVQA is higher than the prior work. (b) Percentage of questions solvable when particular number of top k most occurring answers are selected from each dataset. The plot suggests that AdVQA has much more diverse answer vocabulary compared [20] but also not quite as challenging as [57]. (c)
Cumulative human agreement scores. We see that human agreement is better and higher on AdVQA compared to [20].
For all our experiments, we use the standard ar- chitecture for the models as provided by their authors. For the models that are initialized from pretrained models (whether unimodal or mul- timodal pretrained), we use off-the-shelf pre- trained model weights and then finetune our training set. We do not do any hyperparamter search for these models and the best hyper- params as provided by respective authors. We finetune each model with three different seeds and report average accuracy.
there titls
the
many
there tho
how
We most of experiments NVIDIA V100 GPUs. The maximum number of GPUs used for training is 8 for larger models. Maxi- mum training time is days. More details about hyperparameters, number of devices and time for training each model are provided in the sup- plementary material.
which
what
color
the
Dataset Analysis
Questions. We first analyze the question diver- sity in AdVQA and compare them with popular VQA datasets. AdVQA contains 46,807 ques- tions (10,000 in val and 36,807 in test) each with 10 answers.
Figure Sunburst distribution for the first four words in the AdVQA val set questions. Most questions start with "what" "how".
Fig. 5a shows the distribution of question length compared with [20. 57). The average question length in AdVQA is 8.1, which higher than the VQA v2 (6.3), and TextVQA (7.2). The workers often need to get creative and more specific to fool the model-in-the-loop leading to somewhat longer questions (e.g. specifying a particular person to ask about). Fig. 6a shows the top 15 most occurring questions from the val set, showcasing that questions involving text (e.g. time, sign) and counting (e.g. how many) are major failures for current state-of-the-as VQA models, corroborating the findings of prior work in [57, 21, 58, 28). Fig. 4 shows sunburst plot for the first 4 words in the AdVQA val set questions. We can observe that questions in AdVQA often start with "what" or "how" frequently inquiring about aspects like "many" (count) and "brand" (text).
Answers. In AdVQA val set, 75.4% (1,930) answers only occur once compared to 56.2% VQA v2 [20], suggesting that the diversity of possible answers is much larger in AdVQA compared to

---

Number occurrences
14000
many
r0
many
what man ty
10000
yallow
how many vistbre eyes the photo
how many peop
shate
8000
2013
52
what does the how many pilnss are thene
6000
berch
horizo rtal
pespers
fow many fingers vtible
& there toeth vbible now
4000
keys
waible
viable
people
100
200
300
400
500
60
25
100
325
150
175
Rank the answer
(8)
(b)
Figure 6: Qualitative statistics for AdVQA val questions and answers showing top questions, answers and word distribution. (a) Top 15 most occurring questions in AdVQA val. Most of the top questions start with "what". (b) Total occurrences for 500 most common answers with markers for particular ranks.
VQA v2. Fig. 5b shows the percentage of questions that are solvable with a vocabulary of top most occurring answers. We observe that more questions in VQA v2 while fewer question in TextVQA are solvable with smaller vocabulary compare to AdVQA. This suggests that AdVQA is more diverse and difficult than VQA v2 but not narrowly focused TextVQA, making it great testbed for future VQA models. We also showcase more qualitative examples in our supplementary to demonstrate that AdVQA's diversity doesn't ' lead unnatural questions. Fig. Sc shows the cumulative percentage of questions where more than a particular number of annotators agree with each other. Fig, 6b shows the top 500 most occurring answers in the AdVQA val set; starting from very common answers such as "no" and counts ("1", "5"), to gradually more specific and targeted answers like "32", "peppers" and "horizontal".
Answer Vocabulary. To showcase the chal- lenge of AdVQA, take the original VQA v2 vocabulary used in the Pythia v0.3 model [55]. We find that 87.2% of the AdVQA val set's questions are answerable using this vocab- ulary, suggesting that model with powerful reasoning capability won't be heavily limited by vocabulary on AdvQA. But, also note that for high performance on AdVQA, model will need to understand and reason about rare concepts, as 52.6% of the answers in AdVQA val and test sets don't 't occur in VQA v2 train set.
Table 5: The category-wise distribution of an- swers. Compared to VQA, AdVQA contains more mnumber"based and lesser "yes/no" questions sup- porting the prior work's observations around fail- ure of VQA models to count and read text.
Question Type
VQA test-dev
AdVQA
VQA
AdvQA val
test
yes/no
38.36
23.22
37.70
24.58
number
12.31
35.73
11.48
32.44
Question Types. Table shows the category- wise distribution for AdVQA questions com- pared with VQA v2 [20]. We can observe a shift
others
49.33
41.05
50.82
42.98
from more easy questions of the "yes/no" category in VQA v2 dataset to difficult questions in "numbers" category (as suggested in prior work [45]) in AdVQA. Please refer to Section B. for further detailed breakdown question types.
Human Agreement. In the AdVQA val set, 27.8% human annotators agree on all 10 answers while 3 more annotators agree an answer for 97.8%. which is higher compared to 93.4% on VQA v2 [20], even though as discussed AdVQA contains large number of rare concepts.
Relationship to TextVQA [55]. To understand if the ability to read text is crucial for AdVQA, we extract Rosetta [7] tokens on the AdVQA val set and determine how many questions can be answered using OCR tokens at edit distance of 1 to account for OCR errors. We find that 1 11.1% % of the AdVQA val questions are solvable using OCR tokens suggesting that ability to read scene text would play crucial role for AdVQA. Refer to Section B.4 for comparison with other important datasets.

---

6 Conclusion, Limitations & Outlook
We introduced the AdVQA dataset, a novel huwan-adversarial multimodal dataset designed for accelerating progress on Visual Question Answering (VQA). Current VQA datasets have started plateauing and are approaching saturation with respect to human performance. In this work, we demonstrate that the problem is far from solved.
In particular, our analysis and model evaluation results suggest that current state-of-the-art models underperform on AdVQA due to reasoning gup incurred from a combination of (i) inability to read text: (ii) inability to count; (iii) heavy bias towards the VQA v2 question and answer distribution: (iv) external knowledge; (v) rare unseen concepts; and (vi) weak multimodal understanding. We 've shown the gap unlikely to be due to (i) limited answer vocabulary; (ii) language representation (BERT per- formance compared to other); (iii) no pretraining (UniT)); or (iv) lack of adversarial training (VILLA performance). The evaluation benchmark for AdVQA is available the community and we hope that AdVQA will help bridge the gap by serving as a dynamic new benchmark for visual reasoning with a large amount of headroom for further progress the field.
attps://adversariallg..or for
In future work, it would be interesting to continue Ad as dynamic benchmark. If a new state of the art emerges, those models can be put the loop to examine how we can improve even further.
Broader Impact
This work analyzed state-of-the-art Visual Question Answering (VQA) models via dataset con- structed using a dynamic human-adversarial approach. We hope that this work can help make VQA models more robust.
VQA datasets contain biases, both in the distribution of images in these datasets, as well as the corresponding questions and answers which are likely amplified by the VQA models trained these datasets. Biases studied the context image captioning [8, 62] are also relevant for VQA. English is the only language represented in this work and most annotators are based in the United States.
VQA models can be useful for aiding visually impaired users. The commonly used VQA datasets are not representative of the needs of visually impaired users - both in terms of the distribution of images (typically consumer photographs from the web), and in terms of the questions contained in the datasets (typically asked by sighted individuals while looking at the image). In contrast, the VizWiz dataset [21] contains questions asked by visually impaired users on images taken by them to accomplish day-to-day tasks. It is also worth mentioning to the researchers working on the VQA problem that VQA systems can also possibly be used surveillance systems. would be beneficial for the community to collect larger datasets in this context to enable progress towards useful and relevant technology while making efforts to reduce the harmful side-effects and applications. would be important to do this with input from all relevant stakeholders, and in a responsible and manner. VQA models are currently far from being accurate enough to be uesful or safe in these contexts.
Acknowledgments and Disclosure of Funding
We thank our collaborators in the Dynabench team for their support.
References
[1] Vedika Agarwal, Rakshith Shetty, and Mario Fritz. Towards causal vqa: Revealing and reducing spurious correlations by invariant and covariant semantic editing. In Proceedings of IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 9690-9698, 2020.
[2] Aishwarya Agrawal, Dhruv Batra, Devi Parikh, and Aniruddha Kembhavi. Don't just assume; look and answer: Overcoming priors for visual question answering. In Proceedings of IEEF/CVF Conference on Computer Vision and Pattern Recognition, 2018.
We suggest that the community continues to report results on VQA v2 and other stress tests too, since we don't want models to improve performance AdVQA the expense of VQAv2 performance.
10

---

[3] Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answering. In Proceedings of the IEFE conference on computer vision and pattern recognition, pages 6077-6086, 2018.
[4] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitncck, and Devi Parikh. VQA: Visual Question Answering. In Proceedings of IEEE/CVF International Conference on Computer Vision, 2015.
[5] Max Bartolo, Alastair Roberts, Johannes Welbl, Sebastian Riedel, and Pontus Stenetorp, Beat the ai: Investigating adversarial human annotation for reading comprehension. Transactions of the Association for Computational Linguistics, 8:662-678, 2020.
[6] Ali Furkan Biten, Ruben Tito, Andres Maila, Lluis Gomez, Marçal Rusinol, Ernest Valveny, CV Jawahar, and Dimosthenis Karatzas. Scene text visual question answering. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4291-4301, 2019.
[7] Fedor Borisyuk, Albert Gordo, and Viswanath Sivakumar. Rosetta: Large scale system for text detection and recognition images. In Proceedings of the 24th ACM SIGKDD International Conference Knowledge Discovery & Data Mining, pages 71-79, 2018.
[8] Kaylee Burns, Lisa Anne Hendricks, Trevor Darrell, and Anna Rohrbach. Women also snow- board: Overcoming bias in captioning models. In Proceedings of European Conference on Computer Vision, 2018.
[9] Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 ieee symposium on security and privacy (sp). pages 39-57. IEEE, 2017.
[10] Xinlei Chen, Hao Fang, Tsung- Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C LLwrence Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint 150, 2015.
[11] Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In Proceedings of European Conference on Computer Vision, pages 104-120. Springer, 2020.
[12] Moustapha Cisse, Piotr Bojanowski, Edouard Grave, Yann Dauphin, and Nicolas Usunier. Par- seval networks: Improving robustness adversarial examples. In Proceedings on International Conference on Machine Learning, pages 854-863, PMLR, 2017.
[13] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proceedings of JEEE/CVF Conference on Computer Vision and Pattern Recognition, 2009.
[14] Jacob Devlin, Ming- Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 1XX0:1810004805, 2018.
[15] Emily Dinan, Samuel Humeau, Bharath Chintagunta, and Jason Weston. Build it break it fix it for dialogue safety: Robustness from adversarial human attack. In Proceedings of Empirical Methods in Natural Language Processing, 2019.
[16] Akira Fukui, Dong Huk Park, Daylen Yang, Anna Rohrbach, Trevor Darrell, and Marcus Rohrbach. Multimodal compact bilinear pooling for visual question answering and visual grounding. In Proceedings of Empirical Methods in Natural Language Processing, 2016.
[17] Zhe Gan, Yen-Chun Chen, Linjie Li, Chen Zhu, Yu Cheng, and Jingjing Liu. Large-scale adver- sarial training for lnd-lagga representation learning. In Proceedings of Conference on Advances in Neural Information Processing Systems, 2020.
[18] Tejas Gokhale, Pratyay Banerjee, Chitta Baral, and Yezhou Yang. Vqa-lol: Visual question answering under the lens of logic. In Proceedings of European Conference on Computer Vision, pages 379-396 Springer, 2020.
11

---

[19] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adver- sarial examples. In Proceedings of International Conference on Learning Representations. 2015.
[20] Yash Goyal, Tejas Khot, Douglas Summers-Stur Dhruy Batra, and Devi Parikh. Making the V in VQA matter: Elevating the role of image understanding in Visual Question Answering. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2017.
[21] Danna Gurari, Qing Li, Abigale J. Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P. Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of IEEE/CVF Conference on Computer Vision and Puttern Recognition, 2018.
[22] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for im- age recognition. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 770-778, 2016.
[23] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, (00x1735-1780, 1997.
[24] Ronghang Hu and Amanpreet Singh. Unit: Multimodal multitask learning with unified transformer. arXiv preprint arXiv:2102. 10772, 2021.
[25] Ronghang Hu, Amanpreet Singh, Trevor Darrell, and Marcus Rohrbach. Iterative answer prediction with pointer-augmenter multimodal transformers for textvqa. In Proceedings of JEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 9992-10002, 2020.
[26] Drew A. Hudson and Christopher D. Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019.
[27] Yu Jiang, Vivek Natarajan, Xinlei Chen, Marcus Rohrbach, Dhruv Batra, and Devi Parikh. Pythia vO. 1: the winning entry to the vqa challenge 2018. arXiv preprint arXiv: 10Xivv/807.0990 8807009956 2018.
[28] Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross B. Girshick. CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of IEEE/CVF Conference Computer Vision and Pattern Recognition, 2017.
[29] Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, Zhiyi Ma, Tristan Thrush, Sebastian Riedel, Zeerak Waseem, Pontus Stenetorp. Robin Jia, Mohit Bansal, Christopher Potts, and Adina Williams. Dynabench: Rethinking benchmarking in nlp. In Proceedings of Annual Conference of the North Americun Chupter of the Association for Computational Linguistics, 2021.
[30] Douwe Kiela, Suvrat Bhooshan, Hamed Firooz, Ethan Perez, and Davide Testuggine. Supervised multimodal bitransformers for classifying images and text. arXiv preprint 2019.
[31] Douwe Kiela, Hamed Firooz, Aravind Mohan, Vedanuj Goswami, Amanpreet Singh, Pratik Ringshia, and Davide Testuggine. The hateful memes challenge: Detecting hate speech in multimodal memes. In Proceedings of Conference on Advances in Neural Information Processing Systems, 2021.
[32] Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Vision-and-language transformer without convolution or region supervision. arXiv preprint arXiv:2102.03334, 2021.
[33] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proceedings of International Conference on Learning Representations, 2015.
[34] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. Proceedings of International Journal of Computer Vision, 123(1):32-73, 2017.
12

---

[35] Jie Lei, Licheng Yu, Tamara L Berg, and Mohit Bansal, What is more likely to happen next? sodeo-and-aanuagee future event prediction. In Proceedings of Empirical Methods in Natural Language Processing, 2020.
[36] Linjie Li, Jie Lei, Zhe Gan, and Jingjing Liu. Adversarial vqa: A new benchmark for evaluating the robustness of vqa models. In Proceedings of IEEE/CVE International Conference on Computer Vision, 2021.
[37] Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. Visualbert: A simple and performant baseline for vision and language. In Proceedings of Annual Meeting of the Association for Computational Linguistics, 2020.
[38] Wei Li, Can Gao, Guocheng Niu, Xinyan Xiao, Hao Liu, Jiachen Liu, Hua Wu, and Haifeng Wang. Unimo: Towards unified-modal understanding and generation via cross-modal contrastive 2020.
learning, arXiv preprint
[39] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects context. In Proceedings of European Conference on Computer Vision, pages 740-755. Springer, 2014.
[40] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740-755. Springer. 2014.
[41] Uya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In Proceedings of International Conference on Learning Representations, 2019.
[42] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for lana-uugee tasks. arXiv preprint 2019.
[43] Jiasen Lu, Vedanuj Goswami, Marcus Rohrbach, Devi Parikh, and Stefan Lee. 12-in-1: Multi- task vision and language representation learning. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10437-10446, 2020.
[44] Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Proceedings of IEEF/CVF Conference on Computer Vision and Pattern Recognition, 2019.
[45] Duy-Kien Nguyen, Vedanuj Goswami, and Xinlei Chen. Revisiting modulated convolutions for visual counting and beyond. In Proceedings of International Conference on Learning Representations, 2021.
[46] Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. Adversarial nli: A new benchmark for natural language understanding. In Proceedings of Annual Meeting of the Association for Computational Linguistics, 2020.
[47] Vicente Ordonez, Girish Kulkarni, and Tamara Berg. Im2text: Describing images using I million captioned photographs. Proceedings of Conference on Advances Neural Information Processing Systems, 24:1143 1e:1143-111,, 2011.
[48] Adam Paszke. Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang. Zach Vetito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Proceedings of Conference on Advances in Neural Information Processing Systems, 2019.
[49] Christopher Potts, Zhengxuan Wu, Atticus Geiger, and Douwe Kiela. Dynasent: A dynamic benchmark for sentiment analysis. In Proceedings of Annual Meeting of the Association for Computational Linguistics, 2021.
[50] Pedro Rodriguez, Shi Feng, Mohit lyyer, He He, and Jordan Boyd-Graber. Quizbowl: The case for incremental question answering, arXiv preprint arXiv:1904.0479). 2019.
13

---

[51] Ramprasaath R Selvaraju. Purva Tendulkar, Devi Parikh, Eric Horvitz, Marco Tulio Ribeiro, Besmira Nushi, and Ece Kamar. Squinting at vqa models: Introspecting vqa models with sub-questions. In Proceedings of IEEF/CVF Conference on Computer Vision and Puttern Recognition, pages 10003-10011, 2020.
[52] Meet Shah, Xinlei Chen, Marcus Rohrbach, and Devi Parikh. Sycle-consistency for robust visual question answering. In Proceedings of IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 6649-6658, 2019.
[53] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypemnmed, image alt-text dataset for automatic image captioning. In Proceedings of Annual Meeting of the Association for Computational Linguistics, pages 2556-2565, 2018.
[54] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for scale image recognition. arXiv preprint arXiv: 1556, 2014.
[55] Amanpreet Singh, Vedanuj Goswami, Vivek Natarajan, Yu Jiang, Xinlei Chen, Meet Shah, Marcus Rohrbach, Dhruv Batra, and Devi Parikh. Mmf: A multimodal framework for vision and language research. https: htt///gttttbbbtt com/ ftt.//oboborreeeaccc/mmttt 2020.
[56]
Amanpreet Singh, Vedanuj Goswami, and Devi Parikh. Are we pretraining it right? digging deeper into visio-linguistic pretraining. arXiv preprint 2rXiv:2004.08744, 2020.
[57] Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of IEFF/CVF Conference on Computer Vision and Pattern Recognition, 2019.
[58] Alexander Trott, Caiming Xiong, and Richard Socher. Interpretable counting for visual question answering. In Proceedings of International Conference on Learning Representations, 2018.
[59] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proceedings of Conference on Advances in Neural Information Processing Systems, 2017.
[60] Bertie Vidgen, Tristan Thrush, Zeerak Waseem, and Douwe Kiela. Learning from the worst: Dynamically generated datasets to improve online hate detection. In Proceedings of Annual Meeting of the Association for Computational Linguistics, 2021.
[61] Zhou Yu, Jun Yu, Yuhao Cui, Dacheng Tao, and Qi Tian. Deep modular co-attention networks for visual question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6281-6290, 2019.
[62] Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kei Chang. Men also like shopping: Reducing gender bias amplification using corpus-level constraints. In Proceedings of Empirical Methods in Natural Language Processing, 2017.
14


Reviews:
[{"review_id": "y3WCVkYg4R-", "reviewer": "Reviewer_MEbN", "summary": "This paper proposes AdVQA, a new (adversarial) test set for the Visual Question Answering.  Annotators and a SOTA model are put in the loop to make sure the model can be fooled. Experiments show that other VQA models tend to perform poorly on AdVQA, and a comprehensive analysis discusses the questions and answers that are in this dataset (compared with VQA and TextVQA).", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nTo this reviewer, this paper seems strong. It is perhaps not surprising that performance is lower on AdVQA, however, it is impressive that it has higher human agreement than the non-adversarial VQA. The magnitude of how much models drop in performance is significant (at least this reviewer): from the 70% range to 33% at most. It seems like this could be a great addition in the community (assuming it is released, etc.) for modeling work on VQA-V2 to report on AdVQA in addition.\n\nThe qualitative analysis is insightful and much appreciated (to this reviewer). \n\nThe additional experiments are helpful. Interestingly, the discussion at L222 (suggesting that performance goes up just a little bit if finetuned on val) seems qualitatively different to trends with other datasets. For instance, in a lot of past work (e.g. Adversarial Filtering (from Le Bras et al ICML 2020 and earlier works)) one round of 'fooling the model' is shown to be not enough. However, it's great that it works here (possibly though because val is small).\n\nThe limitations and broader impact sections could be somewhat improved (see the next section).", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitations of this work do not seem to be addressed in the \"limitations\" section, but rather the Broader Impact section. The main limitation stated is that this dataset isn't as good as others (like VizWiz) for actually aiding the visually impaired, even though this is often mentioned as a goal to this work.\n\nThe broader impact statement touches on social biases, but it could flesh this out a bit more (e.g. from what countries are annotators from? what kinds of social biases emerge and from where in the pipeline during your annotation setup?) It could also be improved by discussing end impacts of this work, e.g. downstream use of VQA like models for surveillance applications.  ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1"}, "extra_text": {}}, {"review_id": "ixTtSeT3AX", "reviewer": "Reviewer_bpjd", "summary": "This paper collects a new validation dataset via crowdsourcing to benchmark the progress of state-of-the-art VQA models. \nThe dataset (containing 22K questions and 10x answers) is adversarially collected such that annotators have to come up with questions that can successfully fool a state-of-the-art VQA model, and such questions are undoubtedly more challenging to various existing models.\nThe authors perform thorough data analysis as well as extensive experiments with a variety existing models, to validate the usefulness of the new dataset.\nThis new dataset can potentially be a valuable benchmark to track the progress of VQA research, if there is no obvious bias in the dataset (potential dataset bias is not thoroughly discussed in this paper).\n\n========================================================================================\n\nThanks for the response from the authors. I've updated my rating after reading the rebuttal as well as the comments from other reviewers.\n\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. Originality\n* In my view, this is mainly a resource paper as the major contribution would be the collected new dataset. The data collection procedure is standard, but collecting high quality dataset is definitely not trivial especially for human-in-the-loop dataset collection. \n* The authors claim this is the first work to explore multimodal human-adversarial benchmark.\n\n2. Quality\n* Extensive analysis about the new dataset is provided in the paper, and experiments with existing VQA methods show the usefulness of the dataset. \n* As a resource paper, technical contribution of this work is limited as it evaluated existing methods off-the-shelf on this new dataset.\n\n3. Clarity\n* The paper is well-written and quite easy to follow. \n\n4. Significance\n* This benchmark has the potential to be an important \"stress test\" testbed for new VQA methods, if there is no obvious bias in the collected dataset (which is not thoroughly tested thus unknown from this paper).", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "* My major concern is about the quality of the collected dataset (i.e., whether it's biased), and the authors also mention this issue in L212-213. Specifically, since only one SOTA model is used as \"adversary\" for human to fight against, it is very possible that the new dataset can be \"biased\" towards this specific model checkpoint (i.e. human has the capability to find a \"shortcut\" to fool a model). For example, annotators may quickly learn that asking questions about embedded text can easily fool the model, thus leading to the new dataset containing many questions regarding text. Also From Table 3, it shows a same architectured but differently seeded model shows similar performance. The authors discussed a bit about this in L219-L221 (by the way it is not surprising that VQA models will perform poorly on AdVQA as simple questions are all ruled out), but more evidence would be needed to better address the concern (quality of the dataset). For example, if a M4C model (capable of reading text) was selected as \"adversary model\" in the dataset collection phase, would that cause the collected dataset to differ significantly from current version? If an ensemble (containing multiple instances of a variety of different type of models) are used as adversary, probably the collected dataset will be of higher quality. As a benchmark, if the dataset is flawed with bias it may lead the community to an undesired direction. For example, VQA-CP v2 is a benchmark measuring model robustness against priors, but since it's programmatically reversed the answer distribution (\"more frequent\" in training = \"less frequent\" in test), some debiasing methods exploit this characteristic and achieves strong performance on VQA-CP v2 benchmark but are not really generalizable [1][2]. Therefore I am leaning towards being conservative and cautious on such benchmarks.\n\n[1] Teney, Damien, et al. \"On the value of out-of-distribution testing: An example of goodhart's law.\" arXiv preprint arXiv:2005.09241 (2020).\n[2] Kervadec, Corentin, et al. \"Roses Are Red, Violets Are Blue... but Should Vqa Expect Them To?.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "_6HMhTMn7ce", "reviewer": "Reviewer_BJYR", "summary": "This paper works on the topic of visual question answering and aims to test how well state-of-the-art models perform on adversarial examples. Specifically, the authors collect a new adversarial benchmark on VQA, named \"Adversarial VQA\" (AdVQA), including a large evaluation dataset of 28,522 examples in total. The data samples in AdVQA are collected by putting both human annotators and state-of-the-art models in the collection loop: Annotators are requested to create questions that can fool the state-of-the-art models, and these questions are further verified by other annotators. The authors benchmark state-of-the-art models on these collected adversarial questions and their performance drops significantly.\n\nThe main contribution of this paper is the collected AdVQA dataset which includes adversarial examples created by human. The experimental results suggest future research directions on VQA.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. It is interesting and novel to collect adversarial examples by making human annotators interact with and fool state-of-the-art models. The questions generated in this way are semantically meaningful to humans, compared with adversarial examples created by adding imperceptible noise.\n\n2. The collected benchmark AdVQA is valuable to the community. It provides a quantitive measure of how well current state-of-the-art models perform on adversarial examples and can further spur the progress along this direction. The authors promise to make a public evaluation server to evaluate on AdVQA.\n\n3. The authors conduct extensive experiments on AdVQA and benchmark tens of baseline methods. The analysis points out several reasons for the performance gap between humans and models, such as inability to read text and count, heavy bias, and external knowledge.\n\n4. The paper is overall well-written and easy to follow.\n\nWeaknesses:\n1. My major concern is that the collected examples are more of \"out-of-distribution\" than \"adversarial\". As explained in Line 302~303, 15.4% of the questions in AdVQA require detecting texts in images, as shown by the first example in Table 1. The state-of-the-art models that are trained on VQA v2 definitely fail on these questions, since they are never trained on such examples. It is similar to the questions requiring external knowledge or involving unseen concepts. \n\n2. It is unclear what is the most significant difference between AdVQA and other VQA datasets, such as OKVQA (requiring external knowledge), TextVQA (involving texts in images), and VQA-CP (changing prior). Although AdVQA is collected in a human-adversarial manner, the collected questions seems a combination of previous datasets. The authors might clarify if there are some unique points in AdVQA, or if not, why such a combination is meaningful and what benefits it brings compared with previous datasets.\n\n\n======post rebuttal======\n\nThank the authors for their responses. After reading these responses and other reviews, I keep my original rating and vote for accept.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "It is unclear if the collected questions are \"out-of-distribution\" or \"adversarial\". The authors might perform some analysis to demonstrate the gap between AdVQA and VQA v2, such as the overlap between word vocabularies of AdVQA and VQA v2, the overlap between answer vocabularies of AdVQA and VQA v2, the answer distribution for each question type (like counting questions), etc.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "TY2zCfx2Lt", "reviewer": "Reviewer_JzmN", "summary": "Update after rebuttal:\nThanks to the authors for their rebuttal in response to my questions/concerns. After reading the rebuttal and the other reviews, I am updating my score to 6, to better reflect the contributions made by the work.\n_______________________________________________________________________________________________________________\nThe paper introduces an adversarial VQA dataset collected with human-and-model in the loop by directly asking humans to write questions to attack the winning model of VQA challenge. Extensive evaluation of existing models on this adversarial VQA dataset suggest that current VQA systems are far from really solving the VQA problem. Many well-performed models on VQA v2 seem to fail on AdVQA. AdVQA potentially can be used to benchmark the advances of VQA models in the future, considering VQA v2 performance is close to human parity.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "`Missing discussions on related VQA stress test datasets`\n\nThere are many other VQA datasets built upon VQA v2 to stress test VQA models. For example, VQA-LOL [1], VQA-Introspect [2], IV-VQA/CV-VQA [3]and VQA-Rephrasings [4]. I suggest the authors to add discussions about these works, to highlight the key differences and contributions in AdVQA.\n\n[1] Tejas Gokhale, Pratyay Banerjee, Chitta Baral, and Yezhou Yang. Vqa-lol: Visual question answering under the lens of logic. ECCV, 2020.\n\n[2] Ramprasaath R Selvaraju, Purva Tendulkar, Devi Parikh, Eric Horvitz, Marco Ribeiro, Besmira Nushi, and Ece Ka- mar. Squinting at vqa models: Interrogating vqa models with sub-questions. CVPR, 2020.\n\n[3] Vedika Agarwal, Rakshith Shetty, and Mario Fritz. Towards causal vqa: Revealing and reducing spurious correlations by invariant and covariant semantic editing.\n\n[4] M Shah, X Chen, M Rohrbach, and D Parikh. Cycle- consistency for robust visual question answering. In CVPR, 2019\n\n`L34-35, AdVQA probably is not the first work to have human adversarial approach in Multimodal space`\n\nVLEP [5] is a related work in video+language domain, which leverages human-and-model-in-the-loop to collect harder event candidates for future event prediction task. I suggest the author to add discussion about this work.\n\n[5] JieLei, Licheng Yu,Tamara L Berg, and Mohit Bansal. What is more likely to happen next? video-and-language future event prediction. In EMNLP, 2020\n\n`Detailed question type analysis`\n\nIn ANLI, the authors have provided detailed analysis on the collected adversarial statements (Table ?? and Section ?? of ANLI) to provide insights on (1) What kind of statements would successfully attack the model and (2) Distribution of different failure cases to understand what human annotators focuses on when attacking the model. \n\nI see that some statistics are provided in Figure 5/6 and Table 5. However, I would be more interested in analysis on detailed question types that is similar to Section ?? in ANLI. \n\nFor example, since \"Other\" questions contribute to more than 40% of AdVQA questions, are these other questions more about testing model's reasoning skills or about recognition of small/novel objects in the image. What kind of reasoning questions are there? Are there more positional reasoning than relational reasoning?\n\n`Answers in AdVQA`\n\n(1) L222-224 suggest that the low performance can be from the answer distribution differences between training and testing data. When further finetune on AdVQA val set, what's the performance on VQA v2?\n\n(2) I also want to understand how much does the answer distribution shift contribute to the low performance.\n\nFirst, there are questions that are not answerable by VQA v2 answer vocabularies. What's the upper bound of model performance on val and test set if using the VQA v2 answer vocabularies in Pythia?\n\nSecond, can we keep adding more training data from AdVQA and see when does the model performance saturate? This may not be easy as no training set is collected for AdVQA. Maybe we can keep a smaller set of test set aside and use all others as training data. I will be happy to discuss more about this point.\n\n`BERT performance on AdVQA`\n\nBERT performance on AdVQA test set seems to be on par with multimodal models. Does this suggest that the multimodal models are not paying attention to the visual signals at all? Or at least it is not paying attention to the image regions that really matters to answer the AdVQA questions. Maybe some visualizations of attention maps could be helpful to understand this result.\n\n`Differences between AdVQA and ANLI`\n\nAdVQA largely follows ANLI, but differs in several data collection settings. I hope to learn from the authors about the reason behind these differences.\n\n(1) Human generated adversarial training data is provided in ANLI, but not in AdVQA. As L222-224 suggested, the training data may be helpful in improving performance on AdVQA.\n\n(2) ANLI was collected for multiple rounds, but AdVQA only collect one round. This point is somewhat related to (1). Without training data, we could not have a stronger model for the second round of attack. \n\n(3) ANLI include diverse context from multiple domains, while AdVQA collects on images from COCO only\n\n`Best Practice with AdVQA`\n\nAs the VQA v2 performance is saturating. Would it still make sense to report VQA v2 performance, or should the future work just report results on AdVQA? What about the other stress test dataset? ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes. It is included in the last section and Broader Impact section.", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a new Adversarial VQA dataset collected with human-and-model in the loop by directly asking humans to write questions to attack SOTA VQA models. After author rebuttal, it has received 4 accept recommendations. All the reviewers are happy about the paper, and agree that this is a solid new benchmark worth sharing with the community, and has the potential to be the next generation of a generic VQA dataset for testing future VQA methods. Therefore, the AC is happy to recommend acceptance of the paper. 
Decision: Accept (Poster)