Venue: NAACL 2022

# All You May Need for VQA are Image Captions

Soravit Changpinyo\({}^{*}\) Doron Kukliansky\({}^{*}\)

Idan Szpektor Xi Chen Nan Ding Radu Soricut

Google Research {schangpi,doronk,szpektor,chillxichen,dingnan,rsoricut}@google.com

 Equal contribution

###### Abstract

Visual Question Answering (VQA) has benefited from increasingly sophisticated models, but has not enjoyed the same level of engagement in terms of data creation. In this paper, we propose a method that automatically derives VQA examples at volume, by leveraging the abundance of existing image-caption annotations combined with neural models for textual question generation. We show that the resulting data is of high-quality. VQA models trained on our data improve state-of-the-art zero-shot accuracy by double digits and achieve a level of robustness that lacks in the same model trained on human-annotated VQA data.

## 1 Introduction

Visual Question Answering (VQA) is a complex multimodal task that, to be successfully modeled and evaluated, requires large amounts of annotations that are not naturally produced by existing business processes, the way translation-pair annotations Guo et al. (2018) or image alt-text annotations Sharma et al. (2018) are produced.

At present, a main bottleneck for developing robust VQA systems that are useful for downstream applications, such as for visually-impaired people and in the medical and education domains, appears to be a lack of large image-question-answer training triplets (on the order of millions). Manual annotation of such triplets is costly, time-consuming, and prone to a variety of human biases that are difficult to account for Yuan (2021). In addition, the brittleness of VQA systems trained on such manual annotations is well-understood and documented Agrawal et al. (2018); Kafle and Kanan (2017).

To address the data limitation, we turn to a potential source for creating VQA examples: image-English caption pairs Chen et al. (2015); Sharma et al. (2018). Large-scale image caption datasets exist with millions Changpinyo et al. (2021), several hundreds millions Radford et al. (2021), or even billions Jia et al. (2021) of examples. Captions come mostly in the form of declarative sentences, e.g., "_two bears are laying down on the ice_". Yet, the task of converting declarative captions into VQA question/answer pairs is still largely unexplored. It requires automatically inducing candidate answers fitting the VQA task, along with their respective questions based on the caption text (Fig. 1). We note that transforming declarative form to interrogative form plus answer(s) seems crucial, as there exists evidence that a vision-and-language model trained on declarative-language data cannot be successfully adapted or transferred "out-of-the-box" for VQA Wang et al. (2021).

In this paper, we explore the automatic creation of millions of VQA training data using neural models for textual question generation and question answering. We refer to this method as \(\mathrm{VQ^{2}A}\),

Figure 1: Given an English caption (along with its corresponding image), our \(\mathrm{VQ^{2}A}\) method generates high-quality question-answer pairs. These image-question-answer triplet data can be automatically produced at volume (millions of examples) and used to effectively train VQA systems.



for Visual **Q**uestion Generation with **Q**uestion Answering validation. We demonstrate that VQA models trained on such data, with no exposure to human-annotated VQA data at all, exhibit high zero-shot performance. Our best models obtain 61.1% accuracy on VQA2.0, 52.1% on GQA, around 15-17 points higher than previous zero-shot state-of-the-art results, and getting close to fully-supervised performance. In addition, taking our generated examples as a test set, we provide further evidence for the brittleness of VQA systems built with human-annotated examples, as well as evidence for the robustness of VQA systems built with the automatically-induced \({\rm V}{\rm Q}^{2}\!{\rm A}\) data.

## 2 Related Work

### Question generation in NLP

Question Generation (QG) is an active research topic in NLP. It is explored as a standalone task Heilman and Smith (2009); Nema et al. (2019), as a pre-training task for language models Narayan et al. (2020) and as a component in solutions for other textual tasks, such as question answering Alberti et al. (2019); Puri et al. (2020), information retrieval Mass et al. (2020); Gaur et al. (2021) and generation evaluation Durmus et al. (2020); Wang et al. (2020); Honovich et al. (2021). There are two main directions to QG: template-based Heilman and Smith (2009); Lyu et al. (2021); Dhole and Manning (2020) and neural-based, with the latter achieving state-of-the-art results Alberti et al. (2019); Narayan et al. (2020).

### Question generation in computer vision

Question generation in computer vision aims at generating _visual_ questions about a given image (or video), either for generating questions without knowing the answer Mostafazadeh et al. (2016); Zhang et al. (2017); Yang et al. (2018); Uehara et al. (2018); Krishna et al. (2019), e.g., for them to to be answered by humans, or to help improving the VQA task Kafle et al. (2017); Li et al. (2018); Shah et al. (2019); Xu et al. (2021); Kil et al. (2021); Akula et al. (2021), e.g., for additional evaluation and as means of data augmentation. Such QG models are typically based on VQA triplets as training data, whose language complexity is often limited, or require the collection of visual QG data Mostafazadeh et al. (2016). We take a different approach by leveraging models trained on _textual_ QA datasets instead.

Multiple works leverage image captions or video transcripts as training sources Ren et al. (2015); Banerjee et al. (2021); Yang et al. (2021); Lee et al. (2021). In this approach, question-answer pairs are automatically generated from the text, ignoring the visual source, and are then combined with the related image/video to produce image-question-answer triplets. Banerjee et al. (2021) propose WeaQA, in which they generate questions from MSCOCO image captions Chen et al. (2015) using an improved template-based approach in COOQA Ren et al. (2015) as well as QA-SRL methods, enhanced by paraphrasing and backtranslation for linguistic variations. Lee et al. (2021) similarly train a VQA model from question-answer pairs derived from MSCOCO Captions but only use noun phrases as candidate answers, focusing on using it to verify generated captions but not on the VQA task itself. Yang et al. (2021) generate question-answer pairs from instructional video ASR transcripts, which are then coupled with the related video.

In this work, we follow this direction, investigating what requires to generate data with good coverage for the VQA task in the image domain. We show that our neural-based textual question generation approach with captions is much more effective than previous approaches. Further, unlike previous work, we also explore automatically-curated out-of-domain image-text data sources.

### Transfer learning for and in VQA

Existing work also explores the relationship between the image captioning task and the VQA task without question generation (Section 2.2). Fisch et al. (2020) perform image captioning by anticipating visual questions (i.e., using VQA data as additional supervision and post-inference evaluation). Wu et al. (2019) generate question-relevant image captions to aid VQA. Yang et al. (2021) prompt the GPT-3 Brown et al. (2020) to answer knowledge-based visual questions based on generated captions and tags and a few VQA examples.

Evidence suggests that image-text pre-training, especially when performed at scale, benefits vision-and-language tasks, including VQA Lu et al. (2019); Li et al. (2019); Chen et al. (2020); Tan and Bansal (2019); Su et al. (2020); Lu et al. (2020); Zhou et al. (2020); Li et al. (2020); Zhang et al. (2021); Cho et al. (2021); Wang et al. (2021); Yuan et al. (2021). However, these approaches do not work 

well without fine-tuning on the downstream VQA data (Wang et al., 2021). Further, prompt-based learning and inference Liu et al. (2021) from a pre-trained image-text model that works for VQA is still an open research problem. In contrast, our approach directly works with the training data, explicitly transforms them into the interrogative form of question-answer pairs.

Our focus is the _zero-shot_ transfer setting in WeaQA Banerjee et al. (2021) in which no manually-created VQA triplets are available during training. Note that the term zero-shot here is different from the one used in Teney and Hengel (2016), in which the model still has access to manually-created VQA triplets but is evaluated with unseen questions at test time. Similar to this, Chao et al. (2018) explore cross-dataset VQA but they solely focus on human-annotated data along with approaches to transfer.

## 3 Textual Question Generation for Vqa

We study whether automatically producing VQA annotations from existing image-text resources can alleviate or completely replace the need for manual data annotation. We only focus on English in this paper. To this end, we follow and improve upon some of the recent directions in Section 2.2 on automatic question-answer generation from text.

We start with a given dataset of image-caption pairs \(D{=}\{\mathrm{img}_{i},\mathrm{cap}_{i}\}_{i=1}^{N}\). An important assumption we take is that the information conveyed by the caption is, in the vast majority of cases, present in the image, i.e., captions do not contain an excessive amount of external-world or personal knowledge (e.g., "_my friend at my birthday party_").

For each pair \(\{\mathrm{img}_{i},\mathrm{cap}_{i}\}\), an initial set of candidate answers \(\{\mathrm{a}_{i,j}\}_{j=1}^{M_{i}}\) is first automatically derived from \(\mathrm{cap}_{i}\). For each such candidate answer, a question is generated by a neural model \(\mathrm{q}_{i,j}=QG(\mathrm{a}_{i,j},\mathrm{cap}_{i})\). Each generated question-answer pair undergoes a validation step, and, if validated, is coupled with the corresponding image \(\mathrm{img}_{i}\) to induce a VQA example triplet \(\{\mathrm{img}_{i},\mathrm{q}_{i,j},\mathrm{a}_{i,j}\}\).

We refer to this method as \(\mathrm{VQ}^{2}_{\mbox{A}}\) (**U**isual **Q**uestion Generation with **Q**uestion **A**nswering validation). Figure 2 provides an overview of our approach. We next detail the steps in \(\mathrm{VQ}^{2}_{\mbox{A}}\).

### Candidate Answer Extraction

The only prior work on neural question generation from captions we are aware of Lee et al. (2021), focuses on noun phrases as candidate answers. Yet, these are not enough to cover the answer types included in typical VQA benchmarks such as VQA2.0 (as we will show in Section 5.1), such as boolean, attribute, and verb answers, to name a few, which are required for questions like as "_Is there..._", "_What color..._", "_What is the dog doing_". We present a method that covers all of these answer types.

To extract candidate answers from a given caption, we parse it using spaSCy1 and then extract candidates based on the Part-of-Speech (POS) and dependency parse tree annotations, as follows:

Footnote 1: [https://spacy.io/](https://spacy.io/)

**Noun Phrases**. We extract all noun phrases annotated by spaSCy, including named entities.

**POS Spans**. We extract sequences that begin with an open-class POS (nouns, verbs, adjectives and adverbts), that end with an open-class POS or an adverbial particle, and that do not contain any other POS in between except closed-class POS for determiners, adpositions and conjunctions.

**Parse Tree Spans**. We consider all sub-trees that include at least one open-class POS and no more than 3 words altogether. We only extract maximal spans, i.e., not extracting sub-trees that are fully included in other extracted sub-trees.

**Boolean**. Boolean questions are frequent in VQA benchmarks Goyal et al. (2017). Yet, '_yes_' and '_no_' are not found in captions, and so cannot be extracted as candidates by extracting text spans from captions. To this end, we also add '_yes_' and '_no_' as candidate answers and generate one question per candidate (see Section 3.2).

* about 4.5%.

Our extraction method covers various answer candidates such as compound nouns, noun phrases, 

named entities, boolean answers, cardinal and ordinal numbers, verbs and their compounds, (multi-word) adjectives and prepositional phrases. Table 1 provides an example of candidate answers of various types and the mechanism used to extract them.

### Question Generation

Our question generation model, \(q=QG(a,cap)\), takes as input a caption, \(cap\), and a candidate answer span within it, \(a\), and generates a question \(q\), whose answer given the input caption is the input answer span. Importantly, the answer a does not need to appear verbatim in the caption, enabling the generation of questions for answer types like boolean and zero counts (see Section 3.1).

Given the advances in neural text generation, including models like T5 Raffel et al. (2020), we choose to use a neural generation model as \(QG\). Concretely, we use a T5-XXL model and further fine-tune it on SQuAD1.1 Rajpurkar et al. (2016) for question generation. We take the top-scoring generated question for each caption-answer input. We note that our QG model is trained on a question answering dataset that is not caption-specific, and therefore is not optimized for caption inputs. From manual inspection of hundreds of generated questions, our QG model copes well with captions as input; see examples in Table 2 and Section 3.5.

### Question-Answer Filtering

Generative models may hallucinate, that is, generate content that is inconsistent with its input source Alberti et al. (2019); Honovich et al. (2021). To mitigate this, we follow Alberti et al. (2019) and apply round-trip consistency by answering the generated question on the caption text with a question answering model. If the answer does not match the answer candidate offered as input to the question generation model, the generated question is discarded.

We use the token-level F1 score Wang et al. (2020) to determine if the candidate answer and the QA model's answer is a match; If the score is above a threshold (manually set to 0.54, exemplified in Table 2), the pair is a match. For question answering, we use a T5-XXL model and further fine-tune it on SQuAD2.0 Rajpurkar et al. (2018) and Natural Questions Kwiatkowski et al. (2019).

### Sources of Image/Caption Data

To gain insights on \(VQ^{2}\!A\) potential performance, we generate VQA triplets with \(VQ^{2}\!A\) from two sources of image captions: MSCOCO Captions COCO-CAP Chen et al. (2015) and Conceptual Captions (CC3M) Sharma et al. (2018). COCO-CAP captions contains 123,287 images from the COCO dataset Lin et al. (2014), each with 5 _gold_ captions manually created by raters with careful guidelines. CC3M contains 3.32M images

Figure 2: Visual **Question Generation with **Q**uestion **A**nswering validation (\(VQ^{2}\!A\)) has three main stages: Candidate Answer Extraction (Section 3.1), Question Generation (Section 3.2), and Question-Answering Filtering (Question Answering + Answer Validation, Section 3.3).

 

automatically-collected from the web, each with one associated alt-text which we treat as a _silver_ caption.

These datasets are quite different. Both the amount and the domain of CC3M images are larger and its captions look more plausible for capturing a larger set of object/attribute/action annotations. On the other hand, COCO-CAP's captions are cleaner and represent image content more adequately (see also Section 3.5). Thus, using COCO-CAP would show the potential of training a VQA model using \({\rm VO^{2}A}\) in a "cleaner" zero-shot setup, where captions are human-curated. Using CC3M would indicate the potential of training on noisy web image-alt-text pairs, where scaling up to billions of examples is possible.

To quantify the impact of our method, we focus on VQA classification for the VQA2.0 Goyal et al. (2017), GQA Hudson and Manning (2019), and OKVQA Marino et al. (2019) benchmarks (see Section 4.2). We thus restrict our classifier to top 5,971 answers that are part of a unified answer vocabulary from these benchmarks Appendix B.1). To this end, we remove triplets whose answers are not in the target answer vocabulary, and leave the study of using all generated triplets to future work. We then split our datasets into train/dev sets. In particular, since the images in VQA2.0 are taken from COCO, we split the COCO dataset based on the standard VQA2.0 train/dev splits of *train2014 and minival2014 Jiang et al. (2018)2. For the CC3M dataset, we use the default CC3M train/dev splits Sharma et al. (2018). For each unique image-question pair in the dev split, we construct an answer target of size 10, following VQA2.0, by reducing or expanding the set of seed answers that occur for this image-question pair. Additional details are in Appendix B.1.

Footnote 2: With the exception of OKVQA in which we split into train2014/val2014 to avoid using test images during training.

Table 3 depicts the size of the induced datasets, named \({\rm VG^{2}A}\)-COCO and \({\rm VG^{2}A}\)-CC3M, as well as the VQA datasets used in our experiments.

### Quality Analysis

To measure the quality of the generated datasets, we sampled 800 examples from each of the \({\rm VG^{2}A}\)-COCO and \({\rm VG^{2}A}\)-CC3M datasets. The sample was split between four authors, who assessed whether the answer to the question in an example is justified based on the example's image. For each dataset, 50 examples were rated by all raters, resulting in a free-margin Kappa Randolph (2005) of 0.71 for \({\rm VG^{2}A}\)-COCO and 0.59 for \({\rm VG^{2}A}\)-CC3M, corresponding to high inter-rater agreement. The measured percentage of valid triplets is 87.3% for \({\rm VG^{2}A}\)-COCO and 66.0% for \({\rm VG^{2}A}\)-CC3M. This shows the difference between the high-quality captions of COCO-CAP and the noisier web-based ones of CC3M.

Fig. 3 demonstrates the diversity of questions generates in the \({\rm VG^{2}A}\) datasets. One can see that a significant amount of questions generated by \({\rm VG^{2}A}\) for the shared VQA2.0/COCO image do not appear in VQA2.0. Additional analysis and examples are in Appendix A.

## 4 Visual Question Answering (VQA)

To assess the effectiveness of our automatic generation of VQA annotations, we perform extrinsic

\begin{table}
\begin{tabular}{|l|c|c|c|} \hline Candidate Answer & Generated Question & Validated Answer & Match Score \& Result \\ \hline _’noo’_ & ‘_How many bears are laying on the ice?_’ & _‘two’_ & 1.0 (Pass) \\ _`bears’_ & ’_What are the two animals laying on the ice?_’ & _‘bears’_ & 1.0 (Pass) \\ _’no bears’_ & ‘_How many bears are laying on the ice?_’ & _‘two’_ & 1.0 (Pass) \\ _‘laying’_ & ‘_What are the beans doing?_’ & ‘_laying down on the ice_’ & 0.4 (Fail) \\ _‘laying down’_ & ‘_What are the bears doing?_’ & ‘_leaving down on the ice_’ & 0.7 (Pass) \\ _‘ice’_ & ‘_Two bears are laying down on what?_’ & _‘the ice_’ & 1.0 (Pass) \\ _‘the ice’_ & ‘_Where are the bears laying?_’ & _‘on the ice_’ & 0.7 (Pass) \\ _‘on the ice_’ & ‘_Where are the beats sleeping?_’ & _‘on the ice_’ & 1.0 (Pass) \\ _‘no’_ & ‘_Are the bears sleeping?_’ & _‘yes’_ & 0.0 (Fail) \\ _‘yes’_ & ‘_Are the beans on the ice?_’ & _‘yes’_ & 1.0 (Pass) \\ _‘zero’_ & ‘_How many people are sitting down?_’ & - & Pass by definition \\ \hline \end{tabular}
\end{table}
Table 2: Question/answer pairs generated from the sentence “_two bears are laying down on the ice_” and the filtering decision. For answer candidate ‘_zero_’, no validation is performed .

\begin{table}
\begin{tabular}{l|r|r|r|r} \multirow{2}{*}{Dataset} & \multicolumn{2}{c|}{Image} & \multicolumn{2}{c}{VQA examples} \\ \cline{2-5}  & train & dev & train & dev \\ \hline \({\rm VG^{2}A}\) COCO & 114.9K & 8.4K & 3.50M & 257.5K \\ \({\rm VG^{2}A}\) CC3M & 3.32M & 15.8K & 13.29M & 61.2K \\ \hline COCOQAA & 64.5K & 4.7K & 108.7K & 38.6K \\ \hline VQA2.0 & 114.9K & 8.4K & 582K & 65.1K \\ GOA & 82.4K & 0.4K & 1.08M & 12.6K \\ OKVQA & 9K & 5K & 8.3K & 4.7K \\ \hline \end{tabular}
\end{table}
Table 3: **Sizes** of our generated \({\rm VG^{2}A}\) data (top two rows) and VQA datasets used in our experiments.

 

evaluations of the generated data by measuring its impact on a variety of established VQA benchmarks. We first describe the model, followed by the experimental setup and the results.

### VQA Formulation and Model

Following the literature, we treat VQA as a classification task, i.e., vocab-based VQA. In particular, we treat our target answers as labels, where a label could be multi-token (e.g., "Christmas tree", "black and white", "play tennis"). We define our set of labels based on top answers in the training set of downstream VQA datasets, which allows for a fair comparison with most work in the VQA literature since Antol et al. (2015).

Since our work explores the impact of automatically-generated training data, we fix the VQA model architecture across all experimental conditions. Our model fuses the input image and question (Fig. 4). On the image side, we take global image features from ResNet-152 He et al. (2016) pre-trained on ImageNet Russakovsky et al. (2015) plus 16 region-of-interest image features from Faster R-CNN Ren et al. (2015) pre-trained on Visual Genome Krishna et al. (2017), following the bottom-up-features paradigm Anderson et al. (2018). On the question side, we use the encoder of a pre-trained T5-base checkpoint Radfel et al. (2020). Given the image features and the output token embeddings of the question encoder, a Transformer Vaswani et al. (2017) fuses the multi-modal intermediate representation and classifies it into the predefined answer space. We train the (randomly-initialized) fusing encoder and the text encoder end-to-end using standard cross-entropy loss. The parameters of both ResNet and Faster R-CNN are frozen during training. Additional details are given in Appendix B.2.

### Experimental Setup

We consider three VQA benchmarks: VQA2.0 Goyal et al. (2017), GQA Hudson and Manning (2019), and OKVQA Marino et al. (2019). These datasets have their own characteristics and thus test different capability of VQA models. For instance, GQA puts emphasis on reasoning and OKVQA on external knowledge, whereas VQA2.0 is more general; VQA2.0 and GQA are order-of-magnitude larger than OKVQA; GQA is generated using a question engine while VQA2.0 and OKVQA are human-annotated.

For training and evaluating on VQA2.0, we use the standard train/dev splits *train2014 and minival2014 Jiang et al. (2018). For GQA, we use the balanced v1.2 and combine the train and val splits for training and use the testdev split for evaluation, following the official guideline3 and Tan and Bansal (2019). For OKVQA, we use the train/val splits for training/evaluation. Table 3 summarizes the sizes of the different datasets.

Footnote 3: [https://cs.stanford.edu/people/dorard/gqa/evaluate.html](https://cs.stanford.edu/people/dorard/gqa/evaluate.html)

**Evaluation Settings and Baselines**. The main

Figure 4: **VQA model used in our experiments. The text encoder is initialized from a T5-base checkpoint, while the image-text encoder is initialized from scratch. The parameters of ResNet and Faster R-CNN are frozen during VQA training.**



goal of our experiments is to explore the utility of our \({\rm VQ^{2}A}\) data for transfer learning, as training or evaluation data.

Our main focus in this paper is on zero-shot evaluation. Still, fine-tuning would provide additional insight on using our induced data for pre-training. Therefore, following Banerjee et al. (2021), we train VQA models on the generated \({\rm VQ^{2}A}\) data and then evaluate them in two settings: (i) **zero-shot** evaluation, in which we evaluate our models as-is on the dev split of VQA2.0, GQA, or OKVQA; and (ii) **fully-supervised** fine-tuning evaluation, in which we further fine-tune our models on the training split of \({\rm VQ^{2}A}\), GQA, or OKVQA before evaluating them. When training on \({\rm VQ^{2}A}\) data, we explore training on \({\rm VQ^{2}A}\)-COCO only, \({\rm VQ^{2}A}\)-CC3M only, and a two-stage training \({\rm VQ^{2}A}\)-CC3M followed by \({\rm VQ^{2}A}\)-COCO (\({\rm VQ^{2}A}\) CC3M \(\rightarrow\) COCO).

Our baselines, which do not use \({\rm VQ^{2}A}\) data, include (i) our VQA model trained on template-based question generation data COCOQA4Ren et al. (2015a), (ii) state-of-the-art zero-shot WeqaD Banerjee et al. (2021) and its fully-supervised variants, and (iii) our VQA model trained supervised on each of the target benchmarks' training data.

Footnote 4: Train/dev based on the standard VQA2.0 train/dev splits.

**Metrics**. To be compatible with prior work, on VQA2.0 and OKVQA we measure the standard _VQA Accuracy_. It is the average score over 9 subsets of the ground-truth 10 answers5, where each score is: \(min(\frac{\#answer\ occurrences}{3},1)\). On GQA, we measure _Top-1 Accuracy_ against the single ground-truth answer.

Footnote 5: $targets in OKVQA, replicated twice Marino et al. (2019).

## 5 Results

We report several sets of experimental results that shed light both on the accuracy and on the robustness of VQA models trained on \({\rm VQ^{2}A}\) data in this section, with additional results, analysis and ablation studies in Appendix C.

### Zero-Shot Setting

Table 4 summarizes the outcomes of our VQA experiments on various benchmarks. Our main result is that the \({\rm VQ^{2}A}\) models achieve new state-of-the-art results in the zero-shot transfer learning setting. The improvement in performance is large: to the best of our knowledge, previous state-of-the-art zero-shot accuracy was 46.8% on VQA2.0 and 33.7% on GQA by WeaqA Banerjee et al. (2021), which also induces their training VQA data from COCO Captions. Our \({\rm VQ^{2}A}\)-COCO model reaches 60.0% on VQA2.0 and 51.3% on GQA, an absolute improvement of +13.2% and +17.6%, respectively. Even higher accuracy for the zero-shot setting - 61.1% (VQA2.0) and 52.1% (GQA) - is reached with the \({\rm VQ^{2}A}\) CC3M \(\rightarrow\) COCO model (trained first on the CC3M-derived data and then fine-tuned on the COCO-derived data), establishing new state-of-the-art results.

Training the same model architecture on the manually-constructed VQA2.0 and GQA training sets in a fully-supervised manner achieves 68.8% and 61.8% accuracy, respectively. Hence, our results significantly close the performance gap between automatically-generated and manually-constructed training sources, indicating that the \({\rm VQ^{2}A}\) method may reduce the need for human curated VQA training examples.

The captions for COCO images are carefully annotated to be of high-quality Chen et al. (2015). Additionally, the VQA2.0 images are taken from COCO. To test the robustness of \({\rm VQ^{2}A}\), we also evaluate a \({\rm VQ^{2}A}\)-CC3M model. While CC3M contains more image-alt-text pairs than COCO (see Table 3), the images are from a different distribution and the text annotations are noisier and may represent a larger spectrum of discourse intents 

(Alikhani et al., 2020). In spite of these differences, the gap between COCO-based and CC3M-based VQ\({}^{2}\)A models is not large, 60.0% vs 56.5% on VQA2.0 and 51.3% vs. 49.9% on GQA. This result strengthens our previous observation, in that it does not seem to be crucial that the starting captions are manually-annotated; it appears that "silver" annotations such as the ones provided by CC3M are competitive in zero-shot VQA performance.

To cover the types of answers present in VQA benchmarks, there is a need for thorough extraction of various answer/question types (Section 3). The QACE model (Lee et al., 2021), for example, focuses only on noun-phrases as answer types. By analyzing the VQA2 devset, we find that only 32% of its answers are nouns. As such, it makes sense that, when limiting to only this answer type, the _VQA Accuracy_ of VQE\({}^{2}\)A-COCO is 10.5%, compared to the 60% achieved with a full coverage. As another example, our model trained COCOQA (Ren et al., 2015a), which focuses on a few answer types and one-word answers, barely surpasses the accuracy of our _COCO, nouns only_ baseline. For similar reasons, we want to be able to generate '_how many_' questions from the CC3M data, even though the published annotations have been stripped of digits and numerals. To solve this problem, we recover the original captions from the CC3M URLs, generate questions of the type '_how many_', and train an additional VQ\({}^{2}\)A-CC3M +D model. The results in Table 4 show a small but consistent improvement over vanilla VQ\({}^{2}\)A-CC3M, further closing the gap between VQ\({}^{2}\)A models using curated "gold" captions and noisier "silver" captions.

To gain further insights, we provide a breakdown of _VQA Accuracy_ per VQA2.0 question types in Table 5. Boolean questions are the easiest and all models perform well on them. More challenging question types are '_How many?_' and '_What is_'. One reason could be the validity of various answers, like "_several_" for counts. '_What time?_' is the most difficult, probably due to lack of such information in captions.

Finally, we provide zero-shot results on the more difficult OKVQA benchmark. In this setting, a supervised model reaches 22.1% accuracy, while VQ\({}^{2}\)A models in zero-shot setting achieve close to that - 18.0% with COCO and 19.1% with CC3M, while their combination reaches 19.7%, -2.4% shy of the supervised level. This result also supports the conclusion that creating training data with the VQ\({}^{2}\)A method is a good replacement for small-scale supervised training data.

### Fully-Supervised Setting

Another aspect of the VQ\({}^{2}\)A method that we want to evaluate is whether it produces training data that is similar with the human-annotated data, or it complements it. To this end, we perform experiments in which we first train a model using the VQ\({}^{2}\)A data, and then fine-tune it in a supervised manner using the human-annotated training data.

The results, in the _Fully-supervised_ part of Table 4, tell two stories. For VQA2.0 and GQA, there is a small yet consistent improvement of the fineluned models on top of a model trained directly on the supervised data in each benchmark (labeled _w/o_ VQ\({}^{2}\)A). This indicates that, at least for these two benchmarks, there is a high overlap in the nature of the signal between the human-annotated data and the VQ\({}^{2}\)A data.

The results on OKVQA show a different trend. Here, training first with VQ\({}^{2}\)A boosts performance by +17.2% compared to supervised training without VQ\({}^{2}\)A (22.1% \(\rightarrow\) 39.3%). The small scale of the OKVQA training set (Table 3) certainly contributes to this effect, but it also points to another aspect: question-answer pairs that subsume world knowledge can only be made available at-scale to models by means that are not bottlenecked by human-annotation processes.

### Robustness of Existing VQA Training Sets

So far we have assessed the capability of models trained on VQ\({}^{2}\)A data. As a complementary study, we use 500 manually-validated random samples (see Section 3.5) from the dev part of each VQ\({}^{2}\)A dataset to assess VQA robustness for various training setups. We use the _VQA Accuracy_ metric for the VQ\({}^{2}\)A datasets (10 target answers, see Section 3.4), and _Top-1 Accuracy_ on COCOQA (one target answer).

Table 6 shows the results. The fully-supervised

\begin{table}
\begin{tabular}{l|c|c|c} \hline Question & VQA2.0 & VQ\({}^{2}\)A-COCO & VQ\({}^{2}\)A-CC3M \\ Prefix & Supervised & Zero-shot & Zero-shot \\ \hline Boolean & 96.3 & 93.2 & 94.2 \\ ‘_What color_’ & 69.2 & 64.8 & 56.8 \\ ’_What kind/is_’ & 52.6 & 36.9 & 32.1 \\ ‘_How many_’ & 49.3 & 29.4 & 19.5 \\ ‘_Where aren/is_’ & 38.0 & 30.0 & 25.3 \\ ‘_What does_’ & 33.0 & 24.1 & 20.3 \\ ‘_What time_’ & 23.6 & 11.9 & 12.7 \\ \hline \end{tabular}
\end{table}
Table 5: Aggregated average Accuracy on VQA2.0 for the most common question types.

 

models (diagonal, similar training and test distributions) achieve in-domain Accuracy around 70%, with \({\rm VQ^{2}A}\) CC3M achieving slightly higher 76.4% accuracy. When tested on out-of-domain (non-diagonal), however, each model poses performance degradation at different degrees. First, the model based on template-generated COCOQA does not generalize at all. Second, the VQA2.0 model sees significant accuracy drops, even on the COCO (44.4%) and COCOQA (35.9%), which share a similar image domain with VQA2.0. This result provides another evidence that progress made on the VQA2.0 benchmark may not reflect progress on the VQA task in full Chao et al. (2018); Bras et al. (2020).

In contrast, both \({\rm VQ^{2}A}\) COCO and \({\rm VQ^{2}A}\) CC3M perform robustly with more modest performance drops. For instance, on COCOQA, \({\rm VQ^{2}A}\) CC3M achieves even better performance than VQA2.0 (42.1% vs. 35.9%) despite being tested on out-of-domain images. This suggests that the \({\rm VQ^{2}A}\) training data possesses a higher degree of question variations, provides better answer coverage, and exhibits less biases than the manually-curated VQA2.0 training data, at least enough to address these different benchmarks.

## 6 Considerations and Limitations

Automatic data generation is prone to erroneous outputs. In \({\rm VQ^{2}A}\) these may include hallucinations of the generative model, incorrect negative sampling, and bad answer span extraction. In addition, the image captions may contain details not in the image, e.g. additional details only aware to the photo taker or personal opinions, or information that is inconsistent with the image due to human mistakes and biases. We address some of these issues automatically, filtering bad generations via question answering round-trip validation. In addition, the classification task itself curbs the effects of such errors through the use of a fixed answer vocabulary. Yet, for automatic generation to be more robust, additional methods to narrow down mistakes or mismatches need to be developed.

The resulting VQA model incorporates and may reinforce some of the biases and stereotypes present in the data. For instance, it may learn that answering questions such as "What is the gender of this person?" is a binary choice dictated by shallow cues, or that the answer to "For whom is this room decorated?" depends on stereotypical features present (or not) in the room depicted in the image. Mitigation strategies for such issues go beyond the scope of this paper, but we encourage the research community to consider addressing these issues as central for the successful deployment of this technology.

## 7 Conclusions

In this paper, we show that high-quality VQA training data can be automatically induced at scale from existing image-caption datasets. Our method, \({\rm VQ^{2}A}\), annotates candidate answers using syntactic parsing of the captions and then derives questions for them using neural models for question generation and question answering verification. We demonstrate that VQA models trained only on such data exhibit high zero-shot performance with new state-of-the-art results on VQA2.0 and GQA. Additionally, we provide evidence for the brittleness of VQA systems built with human-annotated examples compared to the ones built with automatically-induced image-question-answer triplets using \({\rm VQ^{2}A}\).

For future work, we plan to explore even larger automatically-curated image-text datasets, consisting of billions of examples. In addition, we want to test the applicability of \({\rm VQ^{2}A}\) to languages other than English, for which human-annotated VQA data is scarce.

**Acknowledgments**. We would like to thank Or Honovich, Hagai Taitelbaum and Roee Aharoni for their help with question generation, Sebastian Goodman for his help with the VQA infrastructure, Piyush Sharma for his help with the Conceptual Captions, Nassim Oufatole for his early exploration of question generation, Gal Elidan, Sasha Goldshtein, and Avinatan Hassidim for their useful feedback.

\begin{table}
\begin{tabular}{c|c|c|c|c}  & \multicolumn{3}{c}{Evaluation Benchmark (Acc \%)} \\ \cline{2-5} Training & COCO: & VQA2.0 & \({\rm VQ^{2}A}\) & \({\rm VQ^{2}A}\) \\ data & QA & & COCO & CCM \\ \hline COCOQA & 70.3 & 11.7 & 13.2 & 5.8 \\ \hline \({\rm VQA2.0}\) & **35.9** & 68.8 & 44.4 & 41.6 \\ \hline \({\rm VQ^{2}A}\) COCO & **55.9** & **60.0** & 72.6 & **56.8** \\ \({\rm VQ^{2}A}\) CC3M & 42.1 & 56.5 & **65.6** & 76.4 \\ \end{tabular}
\end{table}
Table 6: **Manually-validated \({\rm VQ^{2}A}\) data for robustness evaluation**: Accuracy of training on “row” and tested on “column”; diagonal (gray) numbers denote supervised setting, non-diagonal numbers denote zero-shot cross-dataset setting. Best zero-shot is in **bold**.

 



## References

* A. Agrawal, D. Batra, D. Parikh, and A. Kembhavi (2018)Don't just assume; look and answer: overcoming priors for visual question answering. In CVPR, Cited by: SS3.4.
* A. Akula, S. Changpinyo, B. Gong, P. Sharma, S. Zhu, and R. Soricut (2021)CrossVQA: scalably generating benchmarks for systematically testing vqa generalization. In EMNLP, Cited by: SS5.
* C. Alberti, D. Andor, E. Pitler, J. Devlin, and M. Collins (2019)Synthetic QA corpora generation with roundtrip consistency. In ACL, Cited by: SS6.
* M. Alikhani, P. Sharma, S. Li, R. Soricut, and M. Stone (2020)Cross-modal coherence modeling for caption generation. In ACL, Cited by: SS1, SS2.
* P. Anderson, X. He, C. Buehler, D. Teney, M. Johnson, S. Gould, and L. Zhang (2018)Bottom-up and top-down attention for image captioning and visual question answering. In CVPR, Cited by: SS1, SS2.
* S. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra, C. Lawrence Zitnick, and D. Parikh (2015)VOA: Visual question answering. In ICCV, Cited by: SS1, SS2.
* P. Banerjee, T. Gokhale, Y. Yang, and C. Baral (2021)WeaQA: weak supervision via captions for visual question answering. In Findings of ACL-IJCNLP, Cited by: SS1, SS2.
* R. Le Bras, S. Swayamdipta, C. Bhagavatula, R. Zellers, M. E. Peters, A. Sabharwal, and Y. Choi (2020)Adversarial filters of dataset biases. In ICML, Cited by: SS1, SS2.
* T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Woss, G. Krueger, T. J. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei (2020)Language models are few-shot learners. In NeurIPS, Cited by: SS1, SS2.
* S. Changpinyo, P. Sharma, N. Ding, and R. Soricut (2021)Conceptual 12M: pushing web-scale image-text pre-training to recognize long-tail visual concepts. In CVPR, Cited by: SS1, SS2.
* W. Chao, H. Hu, and F. Sha (2018)Being negative but constructively: lessons learnt from creating better visual question answering datasets. In NAACL, Cited by: SS1, SS2.
* W. Chao, H. Hu, and F. Sha (2018)Cross-dataset adaptation for visual question answering. In CVPR, Cited by: SS1, SS2.
* X. Chen, H. Fang, T. Lin, R. Vedantam, S. Gupta, P. Dollar, and C. L. Zitnick (2015)Microsoft COCO Captions: data collection and evaluation server. arXiv preprint arXiv:1504.00325. Cited by: SS1, SS2.
* Y. Chen, L. Li, L. Yu, A. El Kholy, F. Ahmed, Z. Gan, Y. Cheng, and J. Liu (2020)UNITER: learning universal image-text representations. In ECCV, Cited by: SS1, SS2.
* J. Cho, J. Lei, H. Tan, and M. Bansal (2021)Unifying vision-and-language tasks via text generation. In ICML, Cited by: SS1, SS2.
* K. D. Dhole and C. D. Manning (2020)Synd-QG: syntactic and shallow semantic rules for question generation. In ACL, Cited by: SS1, SS2.
* E. Durmus, H. He, and M. Diab (2020)FEQDA: a question answering evaluation framework for faithfulness assessment in abstractive summarization. In ACL, Cited by: SS1, SS2.
* A. Fisch, K. Lee, M. Chang, J. H. Clark, and R. Barzilay (2020)CapWay: Captioning with a purpose. In EMNLP, Cited by: SS1, SS2.
* M. Gaur, K. Gunarantina, V. Srinivasan, and H. Jin (2021)Iseq: information seeking question generation using dynamic meta-information retrieval and knowledge graphs. arXiv preprint arXiv:2112.07622. Cited by: SS1, SS2.
* Y. Goyal, T. Khot, D. Summers-Stay, D. Batra, and D. Parikh (2017)Making the V in VQA matter: elevating the role of image understanding in visual question answering. In CVPR, Cited by: SS1, SS2.
* M. Guo, Q. Shen, Y. Yang, H. Ge, D. Cer, G. Abrego, K. Stevens, N. Constant, Y. Sung, B. Strope, and R. Kurzweil (2018)Effective parallel corpus mining using bilingual sentence embeddings. In WMT, Cited by: SS1, SS2.
* K. He, X. Zhang, S. Ren, and J. Sun (2016)Deep residual learning for image recognition. In CVPR, Cited by: SS1, SS2.
* M. Heileman and N. A. Smith (2009)Question generation via overgenerating transformations and ranking. Technical report Carnegie Mellon University. Cited by: SS1, SS2.
* O. Honovich, L. Choshen, R. Aharoni, E. Neeman, I. Szpekltor, and O. Abend (2021)\({Q}^{2}\): evaluating factual consistency in knowledge-grounded dialogues via question generation and question answering. In EMNLP, Cited by: SS1, SS2.
* D. A. Hudson and C. D. Manning (2019)GQA: a new dataset for real-world visual reasoning and compositional question answering. In CVPR, Cited by: SS1, SS2.
* C. Jia, Y. Yang, Y. Xia, Y. Chen, Z. Parekh, H. Pham, Q. V. Le, Y. Sung, Z. Li, and T. Duerig (2021)Scaling up visual and vision-language representation learning with noisy text supervision. In ICML, Cited by: SS1, SS2.
* Y. Jiang, V. Natarajan, X. Chen, M. Rohrbach, D. Batra, and D. Parikh (2018)Pythia v0.1: the winning entry to the vqa challenge 2018. arXiv preprint arXiv:1807.09956. Cited by: SS1, SS2.
* K. Kafle and C. Kanan (2017)An analysis of visual question answering algorithms. In ICCV, Cited by: SS1, SS2.
* K. Kafle, M. Yousefhussien, and C. Kanan (2017)Data augmentation for visual question answering. In INLG, Cited by: SS1, SS2.
* J. Kil, C. Zhang, D. Xuan, and W. Chao (2021)Discovering the unknown knowns: turning implicit knowledge in the dataset into explicit training examples for visual question answering. In EMNLP,

Ranjay Krishna, Michael Bernstein, and Li Fei-Fei. 2019. Information maximizing visual question generation. In _CVPR_.
* Krishna et al. (2017) Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael Bernstein, and L Li Fei-Fei. 2017. Visual Genome: Connecting language and vision using crowdsourced dense image annotations. _IJCV_, 123(1):32-73.
* Kwiatkowski et al. (2019) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee., Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural Questions: a benchmark for question answering research. _TACL_, 7:453-466.
* Lee et al. (2021)Hwanhee Lee, Thomas Scialom, Seunghyun Yoon, Franck Dernoncourt, and Kyomin Jung. 2021. QACE: Asking questions to evaluate an image caption. In _Findings of EMNLP_.
* Li et al. (2019) Liuinian Harold Li, Mark Yatskar, Da Yin, Cho-Juh Hsieh, and Kai-Wei Chang. 2019. VisualBERT: A simple and performant baseline for vision and language. _arXiv preprint arXiv:1908.03557_.
* Li et al. (2020) Xiujun Li, Xi Yin, Chunyuan Li, Xiaowei Hu, Pengchuan Zhang, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, Yejin Choi, and Jianfeng Gao. 2020. Oscar: Object-semantics aligned pre-training for vision-language tasks. In _ECCV_.
* Li et al. (2018)Yikang Li, Nan Duan, Bolei Zhou, Xiao Chu, Wanli Ouyang, Xiaogang Wang, and Ming Zhou. 2018. Visual question generation as dual task of visual question answering. In _CVPR_.
* Lin et al. (2014) Tsung-Yi Lin, Michael Maire, Serge Belongie, Lubonnir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dollar. 2014. Microsoft COCO: Common objects in context. In _ECCV_.
* Liu et al. (2021)Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2021. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. _arXiv preprint arXiv:2107.13586_.
* Lu et al. (2019) Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019. ViLBERT: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. In _NeurlPS_.
* Lu et al. (2020) Jiasen Lu, Vedanuj Goswami, Marcus Rohrbach, Devi Parikh, and Stefan Lee. 2020. 12-in-1: Multi-task vision and language representation learning. In _CVPR_.
* Lyu et al. (2021)Chenyang Lyu, Lifeng Shang, Yvette Graham, Jennifer Foster, Xin Jiang, and Qun Liu. 2021. Improving unsupervised question answering via summarization-informed question generation. In _EMNLP_.
* Marino et al. (2019)Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. 2019. OK-VQA: A visual question answering benchmark requiring external knowledge. In _CVPR_.
* Mass et al. (2020)Yosi Mass, Boaz Carmeli, Haggai Roitman, and David Konopmicki. 2020. Unsupervised FAQ retrieval with question generation and BERT. In _ACL_.
* Mostafazadeh et al. (2016) Nasrin Mostafazadeh, Ishan Misra, Jacob Devlin, Margaret Mitchell, Xiaodong He, and Lucy Vanderwende. 2016. Generating natural questions about an image. In _ACL_.
* Narayan et al. (2020)Shashi Narayan, Goncalo Simoes, Ji Ma, Hannah Craighead, and Ryan Mcdonald. 2020. QUIROUS: Question generation pretraining for text generation. _arXiv preprint arXiv:2004.11026_.
* Nema et al. (2019)Preksha Nema, Akash Kumar Mohankumar, Mitesh M. Khapra, Balaji Vasan Srinivasan, and Balaraman Ravindran. 2019. Let's ask again: Refine network for automatic question generation. In _EMNLP-IJCNLP_.
* Puri et al. (2020)Raul Puri, Ryan Spring, Mohammad Shoeybi, Mostofa Patwary, and Bryan Catanzaro. 2020. Training question answering models from synthetic data. In _EMNLP_.
* Radford et al. (2021)Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning transferable visual models from natural language supervision. In _ICML_.
* Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. _JMLR_.
* Rajpurkar et al. (2018)Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don't know: Unanswerable questions for SQuAD. In _ACL_.
* Rajpurkar et al. (2016)Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In _EMNLP_.
* Randolph (2005)Justus J. Randolph. 2005. Free-marginal multirater kappa (mulitar\({}^{\prime}\)x\({}^{\prime}\)ise): An alternative to Fleiss" fixed-marginal multirater kappa. _Joensuu Learning and Instruction Symposium_.
* Ren et al. (2015a) Mengye Ren, Ryan Kiros, and Richard Zemel. 2015a. Exploring models and data for image question answering. In _NIPS_.
* Ren et al. (2015b)Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015b. Faster R-CNN: Towards real-time object detection with region proposal networks. In _NIPS_.
* Russakovsky et al. (2015)Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khola, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. 2015. ImageNet large scale visual recognition challenge. _IJCV_, 115(3):211-252.
* Shah et al. (2019)Meet Shah, Xilien Chen, Marcus Rohrbach, and Devi Parikh. 2019. Cycle-consistency for robust visual question answering. In _CVPR_.
* Sharma et al. (2018)Pyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. 2018. Conceptual Captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In _ACL_.
* Shazeer and Stern (2018)Noam Shazeer and Mitchell Stern. 2018. Adafactor: Adaptive learning rates with sublinear memory cost. In _ICLR_.
* Singh et al. (2020)Amanuel Singh, Vedanuj Goswami, Vivek Natarajan, Yu Jiang, Xinlei Chen, Meet Shah, Marcus Rohrbach, Dhruv Batra, and Devi Parikh. 2020. DMFF: A multimodal framework for vision and language research. [https://github.com/facebookresearch/mmf](https://github.com/facebookresearch/mmf) 

Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. 2020. VL-BERT: Pre-training of generic visual-linguistic representations. In _ICLR_.
* Tan and Bansal (2019) Hao Tan and Mohit Bansal. 2019. LXMERT: Learning cross-modality encoder representations from transformers. In _EMNLP-IJCNLP_.
* Teney and van den Hengel (2016) Damien Teney and Anton van den Hengel. 2016. Zero-shot visual question answering. _arXiv preprint arXiv:1611.05546_.
* Uehara et al. (2018) Kohei Uehara, Antonio Tejero-de-Pablos, Yoshitaka Ushiku, and Tatsuya Harada. 2018. Visual question generation for class acquisition of unknown objects. In _ECCV_.
* Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In _NeurIPS_.
* Wang et al. (2020) Alex Wang, Kyunghyun Cho, and Mike Lewis. 2020. Asking and answering questions to evaluate the factual consistency of summaries. In _ACL_.
* Wang et al. (2021) Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. 2021. SimVLM: Simple visual language model pretraining with weak supervision. _arXiv preprint arXiv:2108.10904_.
* Wu et al. (2019)Jialin Wu, Zeyuan Hu, and Raymond J. Mooney. 2019. Generating question relevant captions to aid visual question answering. In _NeurIPS_.
* Xu et al. (2021) Xing Xu, Tan Wang, Yang Yang, Alan Hanjalic, and Heng Tao Shen. 2021. Radial graph convolutional network for visual question generation. _IEEE Transactions on Neural Networks and Learning Systems_, 32(4):1654-1667.
* Yang et al. (2021a) Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid. 2021a. Just ask: Learning to answer questions from millions of narrated videos. In _ICCV_.
* Yang et al. (2018) Jianwei Yang, Jiasen Lu, Stefan Lee, Dhruv Batra, and Devi Parikh. 2018. Visual curiosity: Learning to ask questions to learn visual recognition. In _CoRL_.
* Yang et al. (2021b) Zhengyuan Yang, Zhe Gan, Jianfeng Wang, Xiaowei Hu, Yunao Lu, Zicheng Liu, and Lijuan Wang. 2021b. An empirical study of GPT-3 for few-shot knowledge-based VQA. _arXiv preprint arXiv:2109.05014_.
* Yuan (2021) Desen Yuan. 2021. Language bias in visual question answering: A survey and taxonomy. _arXiv preprint arXiv:2111.08531_.
* Yuan et al. (2021) Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Kiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, Ce Liu, Mengchen Liu, Zicheng Liu, Yumao Lu, Yu Shi, Lijuan Wang, Jianfeng Wang, Bin Xiao, Zhenxiao, Jianwei Yang, Michael Zeng, Ludwei Zhou, and Pengchuan Zhang. 2021. Florence: A new foundation model for computer vision. _arXiv preprint arXiv:2111.11432_.
* Zhang et al. (2021) Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao. 2021. VinVL: Revisiting visual representations in vision-language models. In _CVPR_.
* Zhang et al. (2017) Shijiie Zhang, Lizhen Qu, Shaodi You, Zhenglu Yang, and Jiawan Zhang. 2017. Automatic generation of grounded visual questions. In _IJCAI_.
* Zhou et al. (2020) Louwei Zhou, Hamid Palangi, Lei Zhang, Houdong Hu, Jason J. Corso, and Jianfeng Gao. 2020. Unified vision-language pre-training for image captioning and VQA. In _AAAI_.

 

Additional Examples and Analysis of Generated Data

Fig. 5 provides additional examples of \({\rm VQ^{2}A}\) COCO and CC3M generated VQA triplets, showing the diversity compared to what can be found in VQA2.0.

Table 7 presents the top question prefixes and their distribution in the VQA2.0 and \({\rm VQ^{2}A}\)-based dev sets, showing significant differences between datasets. Many questions in VQA2.0 are of boolean answer type, e.g. _'is the"_, _'is there'_ and _'does the'_, summing to 29.2%. In addition, (_'how many'_) questions are frequent, 11%. Finally, questions for the color attribute are standing out with 9%. On the other hand, COCO and CC3M questions are more explanatory in nature, with the majority of questions (45.5% in COCO, 43.9% in CC3M) of the form _'what is/are/do/does/type'_. Another type that is more prominent in COCO and CC3M are _'where is/are'_ questions, which are more than twice frequent compared to VQA2.0.

Another difference between the manually curated VQA2.0 dataset and the \({\rm VQ^{2}A}\) automatically generated datasets is question and answer word length distribution (Fig. 6 and 7). The questions in \({\rm VQ^{2}A}\)-CC3M and \({\rm VQ^{2}A}\)-COCO have an average word length of 8.3 and 7.8 respectively, while the average VQA2.0 is 6.3. Inspecting the generated questions, we noticed that QG model tends to quote parts of the caption, extending the question length. The average answer word length in \({\rm VQ^{2}A}\)-CC3M and \({\rm VQ^{2}A}\)-COCO is 1.76 and 1.85 words respectively, while in VQA2.0 it is 1.1. While all answers tend to be short, the \({\rm VQ^{2}A}\)-induced datasets have more "detailed" answers of length 2-3 words.

Fig. 8 offers a more visual view of the differences between question type distribution presented in Table 7.

Table 8 depicts the percentage of questions of each type (prefix) that were retained (not filtered out) when applying the question answer validation phase of \({\rm VQ^{2}A}\) (Section 3.3).

## Appendix B Implementation Details

### Details on Data Processing

Our default question and answer preprocessor is based on Jiang et al. (2018); Singh et al. (2020)6, with the exception of GQA which we use 7. The unified answer vocabulary used in our experiments is the union of top answers from existing COCO-based VQA benchmarks: VQA2.0 (3,128, minimum answer frequency=9), GQA (1,843, all), OKVQA (2,000, top), and Visual7W (3,140, minimum answer frequency=3) of total size 5,971

Footnote 6: [https://github.com/facebookresearch/mmf/blob/main/mmf/datasets/processors/processors.py](https://github.com/facebookresearch/mmf/blob/main/mmf/datasets/processors/processors.py)

For each image-unique question pair generated by our \({\rm VQ^{2}A}\) approach, we reduce or expand a list of possibly different candidate answers based on the list length, such that we eventually have a target list of answers of size 10. In particular, we first sort the answers based on their lengths ("dog" before "black dog"), and select up to top-10 answers. If the list length is less than 10, we replicate each of the top answers one-by-one until we have the list of size 10, similar to the process in OKVQAMarino et al. (2019). This is to ensure that we can adopt _VQA Accuracy_ to make the performance comparison.

### Details on Training and Evaluating Visual Question Answering

Our code for the VQA model is based on the Flaxformer framework8. Both the text encoder and the multi-modal encoder have 6 blocks of Transformers, each of which consists of self-attention and a feed-forward network. We use 12 heads of inner dimension of 64, the embedding dimension of 768, and the MLP dimension of 2048. During training, we use Adafactor Shazeer and Stern (2018), with an initial learning rate of 0.0025, a linear warm-up step of 5K for (pre-)training and 1K for fine-tuning, and an "inverse square root" learning rate schedule \(\frac{1}{\sqrt{\max(n,k)}}\), where \(n\) is the current training iteration and \(k\) is the number of warm-up steps. We use a dropout rate of 0.0. We train each of the models with data parallelism using 16 Cloud TPU Pods9, each with a batch size of 256, unless otherwise stated.

Footnote 8: [https://github.com/google/flaxformer](https://github.com/google/flaxformer)

Footnote 9: [https://cloud.google.com/tpu](https://cloud.google.com/tpu)

The default numbers of training steps during training and fine-tuning are 100K and 30K, respectively. The exceptions are OKVQA (30K/15K) and \({\rm VQ^{2}A}\) CC3M (150K/NA). In addition, in the two-stage training where we fine-tune a \({\rm VQ^{2}A}\)-CC3M model with \({\rm VQ^{2}A}\) COCO, we also use 100K steps. Each single training run on average took fewer than 10 hours, including the time used to evaluate 

a checkpoint -- every 1K iterations. For instance, training on VQA2.0 took approximately 7 hours, \({\rm VG}^{2}\!{\rm A}\) COCO 13 hours, \({\rm VG}^{2}\!{\rm A}\) CC3M 10 hours. Note that \({\rm VG}^{2}\!{\rm A}\) COCO has larger evaluation set than other datasets, hence taking longer time to to train then \({\rm VG}^{2}\!{\rm A}\) CC3M.

The hyperparameters for Transformers are selected to be consistent with a T5-base checkpoint, which has 220 million parameters [86] (except that now we have 2 encoders rather than the ones in Table 4). The best performing model is the best performing model and the best performing model is the best performing model. We note that our models are not able to predict the correct performance of our model.

Figure 5: Additional examples from \({\rm VG}^{2}\!{\rm A}\) COCO (top) and \({\rm VG}^{2}\!{\rm A}\) CC3M (bottom). Questions with the green background are present in VQA2.0.

\begin{table}
\begin{tabular}{l|c|c|c|l} \hline Question Prefix & VQA2.0 \% & \({\rm VG}^{2}\!{\rm A}\)-COCO \% & \({\rm VG}^{2}\!{\rm A}\)-CC3M \% & Question Example from \({\rm VG}^{2}\!{\rm A}\)-COCO \\ \hline ‘_What is’_ & 0.140 & 0.288 & 0.217 & ’_What is the man swinging?_” \\ ‘_How many_’ & 0.110 & 0.022 & 0.005 & “_How many people are standing in front of a tv?_” \\ ‘_Is thee_’ & 0.098 & 0.084 & 0.053 & _‘Is the baby wearing a Santa hat?_” \\ ‘_What color_’ & 0.090 & 0.022 & 0.018 & _‘What color is the man’s hair?_’ \\ ‘_Is this’_ & 0.082 & 0.008 & 0.015 & _‘Is this a safe way to fly?_” \\ ‘_Is there_’ & 0.037 & 0.011 & 0.022 & _‘Is there a pool in the backyard?_’ \\ ‘_What kind_’ & 0.025 & 0.049 & 0.078 & _‘What kind of track is the yellow one?_’ \\ ‘_What are_’ & 0.024 & 0.049 & 0.022 & _‘What are the sheep and other animals roaming?_’ \\ ‘_Are thee_’ & 0.024 & 0.022 & 0.007 & ‘_Are the apples on the cutting board green?_’ \\ ‘_Are there_’ & 0.020 & 0.002 & 0.004 & _‘Are there any exceptions to this rule?_’ \\ ‘_Where is’_ & 0.019 & 0.071 & 0.034 & _‘Where is the tennis player pictured?_’ \\ ‘_What type_’ & 0.018 & 0.006 & 0.022 & _‘What type of picture is this?_’ \\ ‘_Is it_’ & 0.017 & 0.001 & 0.005 & _‘Is it possible to eat a whole pizza?_’ \\ ‘_Does the_’ & 0.014 & 0.007 & 0.007 & _‘Does the adult giraffe have any young?_’ \\ ‘_What does_’ & 0.011 & 0.015 & 0.038 & _‘What does a giraffe do with its long neck?_’ \\ ‘_Where are_’ & 0.006 & 0.032 & 0.014 & _‘Where are the skateboarders in the photo?_’ \\ ‘_'Who is’_ & 0.005 & 0.054 & 0.020 & _‘Who is in the photo?_’ \\ ‘_What do_’ & 0.002 & 0.003 & 0.018 & _‘What do the father and son ride?_’ \\ ‘_What was_’ & 0.000 & 0.009 & 0.023 & _‘What was the woman looking at?_’ \\ ‘_What did_’ & 0.000 & 0.001 & 0.021 & _‘What did the cat lay inside of?_’ \\ \end{tabular}
\end{table}
Table 7: Most popular question prefix distribution on valid questions whose answers are in the 6k target vocabulary.

Figure 6: Question length distributions per dataset.

 

than an encoder and a decoder). We initially tuned the initial learning rate (0.0125, 0.075, 0025, 0.00125, 0.00075) and the dropout rate (0.0, 0.1, 0.2) on a fully-supervised model on VQA2.0 baseline using _VQA Accuracy_ and observed that 0.0025 and 0.0 work robustly across our experiments but we did not extensively tuned them in all of our experiments.

We implement _VQA Accuracy_ ourselves based on the official challenge page for VQA2.010.

Footnote 10: [https://visualqa.org/evaluation.html](https://visualqa.org/evaluation.html)

## Appendix C Additional Results

Table 9 offers the Accuracy of the supervised VQA2.0 model, as well as of the zero-shot \({\rm VQ^{2}A}\) models, on the VQA2.0 devset, split by most common question prefixes. The Table is sorted by the supervised model's Accuracy. It shows a several performance differences, first between all types of boolean questions, which all have high precision on all models, vs. other types, which show not only lower performance for all models, but also more significant performance drop between the supervised and zero-shot models.

Table 10 shows the zero-shot performance of models when using all of the \({\rm VQ^{2}A}\) dev sets, not only the manually validated sample, for which Table 6 reports results. What we see is that the difference in performance on the whole \({\rm VQ^{2}A}\) dev sets (Table 10) is similar in magnitude to that of the manually validated dev samples (Table 6), and most

Figure 8: VQA2.0 (top), \({\rm VQ^{2}A}\)-COCO (middle), \({\rm VQ^{2}A}\)-CC3M (bottom) sunburst plots of question prefixes.

\begin{table}
\begin{tabular}{|c|c|c|} \hline Question & \multicolumn{2}{l|}{\({\rm VQ^{2}A}\)-COCO} & \multicolumn{2}{l|}{\({\rm VQ^{2}A}\)-CC3M} \\ Prefix & Filter Pass Ratio & Filter Pass Ratio \\ \hline ‘What is’ & 0.73 & 0.65 \\ ‘Is hee’ & 0.64 & 0.39 \\ ‘What kind’ & 0.84 & 0.80 \\ ‘How many’ & 0.83 & 0.51 \\ ‘What color’ & 0.92 & 0.90 \\ ‘Where is’ & 0.79 & 0.79 \\ ‘Is this’ & 0.83 & 0.62 \\ ‘What are’ & 0.75 & 0.71 \\ ‘Who is’ & 0.85 & 0.79 \\ ‘Is there’ & 0.73 & 0.47 \\ ‘What does’ & 0.75 & 0.67 \\ ‘Are the’ & 0.58 & 0.32 \\ ‘Where are’ & 0.80 & 0.81 \\ ‘What type’ & 0.84 & 0.81 \\ ‘What was’ & 0.72 & 0.67 \\ ‘Does the’ & 0.60 & 0.43 \\ ‘Are there’ & 0.80 & 0.62 \\ ‘What do’ & 0.76 & 0.72 \\ ‘What did’ & 0.69 & 0.64 \\ ‘Is it’ & 0.62 & 0.59 \\ \hline \end{tabular}
\end{table}
Table 8: Question filtering stats.

 

importantly, it keeps the order of models in terms of capabilities/performance. We therefore suggests that the utility of the \({\rm VQ}^{2}_{\mbox{A}}\) approach could go beyond training; it can be used as an automatic test-bed for VQA robustness, if not for absolute figures, for ranking models for robustness zero-shot capabilities.

Table 11 shows the effect of candidate answer types on the VQA2.0 performance. We train our model on \({\rm VQ}^{2}\!\!{\rm A}\) COCO or \({\rm VQ}^{2}\!\!{\rm A}\) CC3M subsets with questions with (i) noun answers, (ii) yes/no answers, (iii) answers containing color-related tokens based on a list of common colors from Wikipedia, and (iv) answers containing digits from 0 to 100. We then evaluate models trained on these subsets on VQA2.0 using _VQA Accuracy_ and the normalized version (by the percentage of evaluation questions with corresponding answer types. This highlights the importance of incorporating diverse answer candidates in our datasets. We also observe that \({\rm VQ}^{2}\!\!{\rm A}\) CC3M is on par with \({\rm VQ}^{2}\!\!{\rm A}\) COCO on yes/no-answer questions but are behind on nouns, color, and number, which we attribute to their lower degree image-text relevance, less mentioning of colors (due to the style of alt-texts vs. captions), and digit substitution.

Table 12 shows the effect of scale on the VQA2.0 performance. We randomly sampled 10%, 20%, and 50% of \({\rm VQ}^{2}\!\!{\rm A}\) COCO or \({\rm VQ}^{2}\!\!{\rm A}\) CC3M training data. We observe that the bigger the data, the higher the accuracy. However, the gain is diminishing. We identify improving the data generation process to achieve higher degree of diversity in the output as interesting future work.

Table 13 provides question-only baselines (no image features as input). Interestingly, the models trained on our generated \({\rm VQ}^{2}\!\!{\rm A}\) data has similar answer distributions to those of existing VQA benchmarks. At the same time, this reveals the exploitation of the language bias, suggesting that additional research on bias mitigation is needed, both in terms of model and data (existing benchmarks and our datasets).

## Appendix D Further Considerations

**Information that names or uniquely identifies individual people or offensive content**. COCO

\begin{table}
\begin{tabular}{l|c|c c}  & \multicolumn{4}{c}{Evaluation Benchmark} \\ \cline{2-4} Training & COCO- & VQA2.0 & \({\rm VQ}^{2}\!\!{\rm A}\) & \({\rm VQ}^{2}\!\!{\rm A}\) \\ data & QA & & COCO & CC3M \\ \hline COCO & 70.3 & 11.7 & 11.5 & 3.7 \\ \hline VQA2.0 & 35.9 & 68.8 & 41.1 & 33.3 \\ \hline \({\rm VQ}^{2}\!\!{\rm A}\) COCO & **55.9** & **60.0** & 71.2 & **49.3** \\ \({\rm VQ}^{2}\!\!{\rm A}\) CC3M & 42.1 & 56.5 & **60.3** & 69.5 \\ \end{tabular}
\end{table}
Table 10: \({\rm VQ}^{2}\!\!{\rm A}\) **as evaluation data** for measuring robustness: VQA Accuracy when training on “row” and tested on “column”; diagonal (gray) numbers denote the supervised setting, non-diagonal numbers denote zero-shot cross-dataset setting. Best zero-shot is **bold**.

\begin{table}
\begin{tabular}{l|c|c} Training data & _VQA Accuracy_ on VQA2.0 \\ \cline{2-3}  & Standard & Normalized \\ \hline \({\rm VQ}^{2}\!\!{\rm A}\) COCO & 60.0 & 60.0 \\ \hline \({\rm VQ}^{2}\!\!{\rm A}\) COCO nouns & 10.5 & 32.5 \\ \({\rm VQ}^{2}\!\!{\rm A}\) COCO yes/no & 38.4 & 94.3 \\ \({\rm VQ}^{2}\!\!{\rm A}\) COCO color & 6.7 & 55.6 \\ \({\rm VQ}^{2}\!\!{\rm A}\) COCO number & 3.9 & 25.4 \\ \hline \hline \({\rm VQ}^{2}\!\!{\rm A}\) CC3M & 56.5 & 56.5 \\ \hline \({\rm VQ}^{2}\!\!{\rm A}\) CC3M nouns & 8.8 & 27.2 \\ \({\rm VQ}^{2}\!\!{\rm A}\) CC3M yes/no & 38.4 & 94.4 \\ \({\rm VQ}^{2}\!\!{\rm A}\) CC3M color & 6.0 & 49.5 \\ \({\rm VQ}^{2}\!\!{\rm A}\) CC3M number & 3.4 & 22.1 \\ \hline \hline \end{tabular}
\end{table}
Table 11: **Effect of candidate answer types on the VQA2.0 performance.**

\begin{table}
\begin{tabular}{c|c|c|c}  & \multicolumn{2}{c}{Evaluation Benchmark} \\ \cline{2-3}  & COCO- & VQA2.0 & \({\rm VQ}^{2}\!\!{\rm A}\) CC3M \\ data & QA & & COCO & CC3M \\ \hline COCO & 70.3 & 11.7 & 11.5 & 3.7 \\ \hline \({\rm VQ}^{2}\!\!{\rm A}\) OCCO & **55.9** & 60.0 & 71.2 & **49.3** \\ \({\rm VQ}^{2}\!\!{\rm A}\) CC3M & 42.1 & 56.5 & **60.3** & 69.5 \\ \end{tabular}
\end{table}
Table 10: \({\rm VQ}^{2}\!\!{\rm A}\) **as evaluation data** for measuring robustness: VQA Accuracy when training on “row” and tested on “column”; diagonal (gray) numbers denote the supervised setting, non-diagonal numbers denote zero-shot cross-dataset setting. Best zero-shot is **bold**.

\begin{table}
\begin{tabular}{c|c|c|c} Training data & _VQA Accuracy_ on VQA2.0 \\ \cline{2-3}  & Standard & Normalized \\ \hline \({\rm VQ}^{2}\!\!{\rm A}\) COCO & 60.0 & 60.0 \\ \hline \({\rm VQ}^{2}\!\!{\rm A}\) COCO nouns & 10.5 & 32.5 \\ \({\rm VQ}^{2}\!\!{\rm A}\) COCO yes/no & 38.4 & 94.3 \\ \({\rm VQ}^{2}\!\!{\rm A}\) COCO color & 6.7 & 55.6 \\ \({\rm VQ}^{2}\!\!{\rm A}\) COCO number & 3.9 & 25.4 \\ \hline \hline \({\rm VQ}^{2}\!\!{\rm A}\) CC3M & 56.5 & 56.5 \\ \hline \({\rm VQ}^{2}\!\!{\rm A}\) CC3M nouns & 8.8 & 27.2 \\ \({\rm VQ}^{2}\!\!{\rm A}\) CCM3M yes/no & 38.4 & 94.4 \\ \({\rm VQ}^{2}\!\!{\rm A}\) CC3M color & 6.0 & 49.5 \\ \({\rm VQ}^{2}\!\!{\rm A}\) CC3M color & 3.4 & 22.1 \\ \hline \hline \end{tabular}
\end{table}
Table 11: **Effect of candidate answer types on the VQA2.0 performance.** 

Captions are human-curated and cleaned while the approach to collection of CC3M upholds rigorous privacy and ethics standards such as the removal of offensive content and hypernymization. This significantly mitigates the risks that our datasets would contain such information.

**Intended uses**. Due to considerations and limitations as we mention in Section 6, COCO Captions, CC3M, and our induced are intended to be used for research-only purposes.