Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: Update after rebuttal:
Thanks to the authors for their rebuttal in response to my questions/concerns. After reading the rebuttal and the other reviews, I am updating my score to 6, to better reflect the contributions made by the work.
_______________________________________________________________________________________________________________
The paper introduces an adversarial VQA dataset collected with human-and-model in the loop by directly asking humans to write questions to attack the winning model of VQA challenge. Extensive evaluation of existing models on this adversarial VQA dataset suggest that current VQA systems are far from really solving the VQA problem. Many well-performed models on VQA v2 seem to fail on AdVQA. AdVQA potentially can be used to benchmark the advances of VQA models in the future, considering VQA v2 performance is close to human parity.
main_review: `Missing discussions on related VQA stress test datasets`

There are many other VQA datasets built upon VQA v2 to stress test VQA models. For example, VQA-LOL [1], VQA-Introspect [2], IV-VQA/CV-VQA [3]and VQA-Rephrasings [4]. I suggest the authors to add discussions about these works, to highlight the key differences and contributions in AdVQA.

[1] Tejas Gokhale, Pratyay Banerjee, Chitta Baral, and Yezhou Yang. Vqa-lol: Visual question answering under the lens of logic. ECCV, 2020.

[2] Ramprasaath R Selvaraju, Purva Tendulkar, Devi Parikh, Eric Horvitz, Marco Ribeiro, Besmira Nushi, and Ece Ka- mar. Squinting at vqa models: Interrogating vqa models with sub-questions. CVPR, 2020.

[3] Vedika Agarwal, Rakshith Shetty, and Mario Fritz. Towards causal vqa: Revealing and reducing spurious correlations by invariant and covariant semantic editing.

[4] M Shah, X Chen, M Rohrbach, and D Parikh. Cycle- consistency for robust visual question answering. In CVPR, 2019

`L34-35, AdVQA probably is not the first work to have human adversarial approach in Multimodal space`

VLEP [5] is a related work in video+language domain, which leverages human-and-model-in-the-loop to collect harder event candidates for future event prediction task. I suggest the author to add discussion about this work.

[5] JieLei, Licheng Yu,Tamara L Berg, and Mohit Bansal. What is more likely to happen next? video-and-language future event prediction. In EMNLP, 2020

`Detailed question type analysis`

In ANLI, the authors have provided detailed analysis on the collected adversarial statements (Table ?? and Section ?? of ANLI) to provide insights on (1) What kind of statements would successfully attack the model and (2) Distribution of different failure cases to understand what human annotators focuses on when attacking the model. 

I see that some statistics are provided in Figure 5/6 and Table 5. However, I would be more interested in analysis on detailed question types that is similar to Section ?? in ANLI. 

For example, since "Other" questions contribute to more than 40% of AdVQA questions, are these other questions more about testing model's reasoning skills or about recognition of small/novel objects in the image. What kind of reasoning questions are there? Are there more positional reasoning than relational reasoning?

`Answers in AdVQA`

(1) L222-224 suggest that the low performance can be from the answer distribution differences between training and testing data. When further finetune on AdVQA val set, what's the performance on VQA v2?

(2) I also want to understand how much does the answer distribution shift contribute to the low performance.

First, there are questions that are not answerable by VQA v2 answer vocabularies. What's the upper bound of model performance on val and test set if using the VQA v2 answer vocabularies in Pythia?

Second, can we keep adding more training data from AdVQA and see when does the model performance saturate? This may not be easy as no training set is collected for AdVQA. Maybe we can keep a smaller set of test set aside and use all others as training data. I will be happy to discuss more about this point.

`BERT performance on AdVQA`

BERT performance on AdVQA test set seems to be on par with multimodal models. Does this suggest that the multimodal models are not paying attention to the visual signals at all? Or at least it is not paying attention to the image regions that really matters to answer the AdVQA questions. Maybe some visualizations of attention maps could be helpful to understand this result.

`Differences between AdVQA and ANLI`

AdVQA largely follows ANLI, but differs in several data collection settings. I hope to learn from the authors about the reason behind these differences.

(1) Human generated adversarial training data is provided in ANLI, but not in AdVQA. As L222-224 suggested, the training data may be helpful in improving performance on AdVQA.

(2) ANLI was collected for multiple rounds, but AdVQA only collect one round. This point is somewhat related to (1). Without training data, we could not have a stronger model for the second round of attack. 

(3) ANLI include diverse context from multiple domains, while AdVQA collects on images from COCO only

`Best Practice with AdVQA`

As the VQA v2 performance is saturating. Would it still make sense to report VQA v2 performance, or should the future work just report results on AdVQA? What about the other stress test dataset? 
limitations_and_societal_impact: Yes. It is included in the last section and Broader Impact section.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper collects a new validation dataset via crowdsourcing to benchmark the progress of state-of-the-art VQA models. 
The dataset (containing 22K questions and 10x answers) is adversarially collected such that annotators have to come up with questions that can successfully fool a state-of-the-art VQA model, and such questions are undoubtedly more challenging to various existing models.
The authors perform thorough data analysis as well as extensive experiments with a variety existing models, to validate the usefulness of the new dataset.
This new dataset can potentially be a valuable benchmark to track the progress of VQA research, if there is no obvious bias in the dataset (potential dataset bias is not thoroughly discussed in this paper).

========================================================================================

Thanks for the response from the authors. I've updated my rating after reading the rebuttal as well as the comments from other reviewers.


main_review: 1. Originality
* In my view, this is mainly a resource paper as the major contribution would be the collected new dataset. The data collection procedure is standard, but collecting high quality dataset is definitely not trivial especially for human-in-the-loop dataset collection. 
* The authors claim this is the first work to explore multimodal human-adversarial benchmark.

2. Quality
* Extensive analysis about the new dataset is provided in the paper, and experiments with existing VQA methods show the usefulness of the dataset. 
* As a resource paper, technical contribution of this work is limited as it evaluated existing methods off-the-shelf on this new dataset.

3. Clarity
* The paper is well-written and quite easy to follow. 

4. Significance
* This benchmark has the potential to be an important "stress test" testbed for new VQA methods, if there is no obvious bias in the collected dataset (which is not thoroughly tested thus unknown from this paper).
limitations_and_societal_impact: * My major concern is about the quality of the collected dataset (i.e., whether it's biased), and the authors also mention this issue in L212-213. Specifically, since only one SOTA model is used as "adversary" for human to fight against, it is very possible that the new dataset can be "biased" towards this specific model checkpoint (i.e. human has the capability to find a "shortcut" to fool a model). For example, annotators may quickly learn that asking questions about embedded text can easily fool the model, thus leading to the new dataset containing many questions regarding text. Also From Table 3, it shows a same architectured but differently seeded model shows similar performance. The authors discussed a bit about this in L219-L221 (by the way it is not surprising that VQA models will perform poorly on AdVQA as simple questions are all ruled out), but more evidence would be needed to better address the concern (quality of the dataset). For example, if a M4C model (capable of reading text) was selected as "adversary model" in the dataset collection phase, would that cause the collected dataset to differ significantly from current version? If an ensemble (containing multiple instances of a variety of different type of models) are used as adversary, probably the collected dataset will be of higher quality. As a benchmark, if the dataset is flawed with bias it may lead the community to an undesired direction. For example, VQA-CP v2 is a benchmark measuring model robustness against priors, but since it's programmatically reversed the answer distribution ("more frequent" in training = "less frequent" in test), some debiasing methods exploit this characteristic and achieves strong performance on VQA-CP v2 benchmark but are not really generalizable [1][2]. Therefore I am leaning towards being conservative and cautious on such benchmarks.

[1] Teney, Damien, et al. "On the value of out-of-distribution testing: An example of goodhart's law." arXiv preprint arXiv:2005.09241 (2020).
[2] Kervadec, Corentin, et al. "Roses Are Red, Violets Are Blue... but Should Vqa Expect Them To?." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper proposes AdVQA, a new (adversarial) test set for the Visual Question Answering.  Annotators and a SOTA model are put in the loop to make sure the model can be fooled. Experiments show that other VQA models tend to perform poorly on AdVQA, and a comprehensive analysis discusses the questions and answers that are in this dataset (compared with VQA and TextVQA).
main_review: 
To this reviewer, this paper seems strong. It is perhaps not surprising that performance is lower on AdVQA, however, it is impressive that it has higher human agreement than the non-adversarial VQA. The magnitude of how much models drop in performance is significant (at least this reviewer): from the 70% range to 33% at most. It seems like this could be a great addition in the community (assuming it is released, etc.) for modeling work on VQA-V2 to report on AdVQA in addition.

The qualitative analysis is insightful and much appreciated (to this reviewer). 

The additional experiments are helpful. Interestingly, the discussion at L222 (suggesting that performance goes up just a little bit if finetuned on val) seems qualitatively different to trends with other datasets. For instance, in a lot of past work (e.g. Adversarial Filtering (from Le Bras et al ICML 2020 and earlier works)) one round of 'fooling the model' is shown to be not enough. However, it's great that it works here (possibly though because val is small).

The limitations and broader impact sections could be somewhat improved (see the next section).
limitations_and_societal_impact: The limitations of this work do not seem to be addressed in the "limitations" section, but rather the Broader Impact section. The main limitation stated is that this dataset isn't as good as others (like VizWiz) for actually aiding the visually impaired, even though this is often mentioned as a goal to this work.

The broader impact statement touches on social biases, but it could flesh this out a bit more (e.g. from what countries are annotators from? what kinds of social biases emerge and from where in the pipeline during your annotation setup?) It could also be improved by discussing end impacts of this work, e.g. downstream use of VQA like models for surveillance applications.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper works on the topic of visual question answering and aims to test how well state-of-the-art models perform on adversarial examples. Specifically, the authors collect a new adversarial benchmark on VQA, named "Adversarial VQA" (AdVQA), including a large evaluation dataset of 28,522 examples in total. The data samples in AdVQA are collected by putting both human annotators and state-of-the-art models in the collection loop: Annotators are requested to create questions that can fool the state-of-the-art models, and these questions are further verified by other annotators. The authors benchmark state-of-the-art models on these collected adversarial questions and their performance drops significantly.

The main contribution of this paper is the collected AdVQA dataset which includes adversarial examples created by human. The experimental results suggest future research directions on VQA.
main_review: Strengths:
1. It is interesting and novel to collect adversarial examples by making human annotators interact with and fool state-of-the-art models. The questions generated in this way are semantically meaningful to humans, compared with adversarial examples created by adding imperceptible noise.

2. The collected benchmark AdVQA is valuable to the community. It provides a quantitive measure of how well current state-of-the-art models perform on adversarial examples and can further spur the progress along this direction. The authors promise to make a public evaluation server to evaluate on AdVQA.

3. The authors conduct extensive experiments on AdVQA and benchmark tens of baseline methods. The analysis points out several reasons for the performance gap between humans and models, such as inability to read text and count, heavy bias, and external knowledge.

4. The paper is overall well-written and easy to follow.

Weaknesses:
1. My major concern is that the collected examples are more of "out-of-distribution" than "adversarial". As explained in Line 302~303, 15.4% of the questions in AdVQA require detecting texts in images, as shown by the first example in Table 1. The state-of-the-art models that are trained on VQA v2 definitely fail on these questions, since they are never trained on such examples. It is similar to the questions requiring external knowledge or involving unseen concepts. 

2. It is unclear what is the most significant difference between AdVQA and other VQA datasets, such as OKVQA (requiring external knowledge), TextVQA (involving texts in images), and VQA-CP (changing prior). Although AdVQA is collected in a human-adversarial manner, the collected questions seems a combination of previous datasets. The authors might clarify if there are some unique points in AdVQA, or if not, why such a combination is meaningful and what benefits it brings compared with previous datasets.


======post rebuttal======

Thank the authors for their responses. After reading these responses and other reviews, I keep my original rating and vote for accept.
limitations_and_societal_impact: It is unclear if the collected questions are "out-of-distribution" or "adversarial". The authors might perform some analysis to demonstrate the gap between AdVQA and VQA v2, such as the overlap between word vocabularies of AdVQA and VQA v2, the overlap between answer vocabularies of AdVQA and VQA v2, the answer distribution for each question type (like counting questions), etc.
review_title: null

Meta-review: This paper introduces a new Adversarial VQA dataset collected with human-and-model in the loop by directly asking humans to write questions to attack SOTA VQA models. After author rebuttal, it has received 4 accept recommendations. All the reviewers are happy about the paper, and agree that this is a solid new benchmark worth sharing with the community, and has the potential to be the next generation of a generic VQA dataset for testing future VQA methods. Therefore, the AC is happy to recommend acceptance of the paper. 
Decision: Accept (Poster)