SlowGuess/NAACL_2022_OCR huggingface dataset:
Soravit Changpinyo* Doron Kukliansky*  
Idan Szpektor Xi Chen Nan Ding Radu Soricut  
Google Research
{schangpi,doronk,szpektor,chillxichen,dingnan,rsoricut}@google.com
Visual Question Answering (VQA) has benefited from increasingly sophisticated models, but has not enjoyed the same level of engagement in terms of data creation. In this paper, we propose a method that automatically derives VQA examples at volume, by leveraging the abundance of existing image-caption annotations combined with neural models for textual question generation. We show that the resulting data is of high-quality. VQA models trained on our data improve state-of-the-art zero-shot accuracy by double digits and achieve a level of robustness that lacks in the same model trained on human-annotated VQA data.
Visual Question Answering (VQA) is a complex multimodal task that, to be successfully modeled and evaluated, requires large amounts of annotations that are not naturally produced by existing business processes, the way translation-pair annotations (Guo et al., 2018) or image alt-text annotations (Sharma et al., 2018) are produced.
At present, a main bottleneck for developing robust VQA systems that are useful for downstream applications, such as for visually-impaired people and in the medical and education domains, appears to be a lack of large image-question-answer training triplets (on the order of millions). Manual annotation of such triplets is costly, time-consuming, and prone to a variety of human biases that are difficult to account for (Yuan, 2021). In addition, the brittleness of VQA systems trained on such manual annotations is well-understood and documented (Agrawal et al., 2018; Kafle and Kanan, 2017).
To address the data limitation, we turn to a potential source for creating VQA examples: image-English caption pairs (Chen et al., 2015; Sharma et al., 2018). Large-scale image caption datasets
Figure 1: Given an English caption (along with its corresponding image), our  $\mathrm{VQ^2A}$  method generates high-quality question-answer pairs. These image-question-answer triplet data can be automatically produced at volume (millions of examples) and used to effectively train VQA systems.
exist with millions (Changpinyo et al., 2021), several hundreds millions (Radford et al., 2021), or even billions (Jia et al., 2021) of examples. Captions come mostly in the form of declarative sentences, e.g., "two bears are laying down on the ice". Yet, the task of converting declarative captions into VQA question/answer pairs is still largely unexplored. It requires automatically inducing candidate answers fitting the VQA task, along with their respective questions based on the caption text (Fig. 1). We note that transforming declarative form to interrogative form plus answer(s) seems crucial, as there exists evidence that a vision-and-language model trained on declarative-language data cannot be successfully adapted or transferred "out-of-the-box" for VQA (Wang et al., 2021).
In this paper, we explore the automatic creation of millions of VQA training data using neural models for textual question generation and question answering. We refer to this method as  $\mathrm{VQ^2A}$
Human Language Technologies, pages 1947 - 1963
for Visual Question Generation with Question Answering validation. We demonstrate that VQA models trained on such data, with no exposure to human-annotated VQA data at all, exhibit high zero-shot performance. Our best models obtain  $61.1\%$  accuracy on VQA2.0,  $52.1\%$  on GQA, around 15-17 points higher than previous zero-shot state-of-the-art results, and getting close to fully-supervised performance. In addition, taking our generated examples as a test set, we provide further evidence for the brittleness of VQA systems built with human-annotated examples, as well as evidence for the robustness of VQA systems built with the automatically-induced  $\mathrm{VQ}^2\mathrm{A}$  data.
2.1 Question generation in NLP
Question Generation (QG) is an active research topic in NLP. It is explored as a standalone task (Heilman and Smith, 2009; Nema et al., 2019), as a pre-training task for language models (Narayan et al., 2020) and as a component in solutions for other textual tasks, such as question answering (Alberti et al., 2019; Puri et al., 2020), information retrieval (Mass et al., 2020; Gaur et al., 2021) and generation evaluation (Durmus et al., 2020; Wang et al., 2020; Honovich et al., 2021). There are two main directions to QG: template-based (Heilman and Smith, 2009; Lyu et al., 2021; Dhole and Manning, 2020) and neural-based, with the latter achieving state-of-the-art results (Alberti et al., 2019; Narayan et al., 2020).
2.2 Question generation in computer vision
Question generation in computer vision aims at generating visual questions about a given image (or video), either for generating questions without knowing the answer (Mostafazadeh et al., 2016; Zhang et al., 2017; Yang et al., 2018; Uehara et al., 2018; Krishna et al., 2019), e.g., for them to be answered by humans, or to help improving the VQA task (Kafle et al., 2017; Li et al., 2018; Shah et al., 2019; Xu et al., 2021; Kil et al., 2021; Akula et al., 2021), e.g., for additional evaluation and as means of data augmentation. Such QG models are typically based on VQA triplets as training data, whose language complexity is often limited, or require the collection of visual QG data (Mostafazadeh et al., 2016). We take a different approach by leveraging models trained on textual QA datasets instead.
Multiple works leverage image captions or video transcripts as training sources (Ren et al., 2015a; Banerjee et al., 2021; Yang et al., 2021a; Lee et al., 2021). In this approach, question-answer pairs are automatically generated from the text, ignoring the visual source, and are then combined with the related image/video to produce image-question-answer triplets. Banerjee et al. (2021) propose WeaQA, in which they generate questions from MSCOCO image captions (Chen et al., 2015) using an improved template-based approach in CO-COQA (Ren et al., 2015a) as well as QA-SRL methods, enhanced by paraphrasing and backtranslation for linguistic variations. Lee et al. (2021) similarly train a VQA model from question-answer pairs derived from MSCOCO Captions but only use noun phrases as candidate answers, focusing on using it to verify generated captions but not on the VQA task itself. Yang et al. (2021a) generate question-answer pairs from instructional video ASR transcripts, which are then coupled with the related video.
In this work, we follow this direction, investigating what requires to generate data with good coverage for the VQA task in the image domain. We show that our neural-based textual question generation approach with captions is much more effective than previous approaches. Further, unlike previous work, we also explore automatically-curated out-of-domain image-text data sources.
2.3 Transfer learning for and in VQA
Existing work also explores the relationship between the image captioning task and the VQA task without question generation (Section 2.2). Fisch et al. (2020) perform image captioning by anticipating visual questions (i.e., using VQA data as additional supervision and post-inference evaluation). Wu et al. (2019) generate question-relevant image captions to aid VQA. Yang et al. (2021b) prompt the GPT-3 (Brown et al., 2020) to answer knowledge-based visual questions based on generated captions and tags and a few VQA examples.
Evidence suggests that image-text pre-training, especially when performed at scale, benefits vision-and-language tasks, including VQA (Lu et al., 2019; Li et al., 2019; Chen et al., 2020; Tan and Bansal, 2019; Su et al., 2020; Lu et al., 2020; Zhou et al., 2020; Li et al., 2020; Zhang et al., 2021; Cho et al., 2021; Wang et al., 2021; Yuan et al., 2021). However, these approaches do not work
well without fine-tuning on the downstream VQA data (Wang et al., 2021). Further, prompt-based learning and inference (Liu et al., 2021) from a pre-trained image-text model that works for VQA is still an open research problem. In contrast, our approach directly works with the training data, explicitly transforms them into the interrogative form of question-answer pairs.
Our focus is the zero-shot transfer setting in WeaQA (Banerjee et al., 2021) in which no manually-created VQA triplets are available during training. Note that the term zero-shot here is different from the one used in (Teney and Hengel, 2016), in which the model still has access to manually-created VQA triplets but is evaluated with unseen questions at test time. Similar to this, Chao et al. (2018b) explore cross-dataset VQA but they solely focus on human-annotated data along with approaches to transfer.
3 Textual Question Generation for VQA
We study whether automatically producing VQA annotations from existing image-text resources can alleviate or completely replace the need for manual data annotation. We only focus on English in this paper. To this end, we follow and improve upon some of the recent directions in Section 2.2 on automatic question-answer generation from text.
We start with a given dataset of image-caption pairs  $D = \{\mathrm{img}_i, \mathrm{cap}_i\}_{i=1}^N$ . An important assumption we take is that the information conveyed by the caption is, in the vast majority of cases, present in the image, i.e., captions do not contain an excessive amount of external-world or personal knowledge (e.g., "my friend at my birthday party").
For each pair  $\{\mathrm{img}_i,\mathrm{cap}_i\}$  , an initial set of candidate answers  $\{\mathrm{a}_{i,j}\}_{j = 1}^{M_i}$  is first automatically derived from  $\mathrm{cap}_i$  . For each such candidate answer, a question is generated by a neural model  $\mathrm{q}_{i,j} = QG(\mathrm{a}_{i,j},\mathrm{cap}_i)$  . Each generated question-answer pair undergoes a validation step, and, if validated, is coupled with the corresponding image  $\mathrm{img}_i$  to induce a VQA example triplet  $\{\mathrm{img}_i,\mathrm{q}_{i,j},\mathrm{a}_{i,j}\}$
We refer to this method as  $\mathrm{VQ^2A}$  (Visual Question Generation with Question Answering validation). Figure 2 provides an overview of our approach. We next detail the steps in  $\mathrm{VQ^2A}$ .
3.1 Candidate Answer Extraction
The only prior work on neural question generation from captions we are aware of, Lee et al.
(2021), focuses on noun phrases as candidate answers. Yet, these are not enough to cover the answer types included in typical VQA benchmarks such as VQA2.0 (as we will show in Section 5.1), such as boolean, attribute, and verb answers, to name a few, which are required for questions like "Is there...", "What color...", "What is the dog doing". We present a method that covers all of these answer types.
To extract candidate answers from a given caption, we parse it using  $\mathsf{spaCy}^1$  and then extract candidates based on the Part-of-Speech (POS) and dependency parse tree annotations, as follows:
Noun Phrases. We extract all noun phrases annotated by spaCy, including named entities.
POS Spans. We extract sequences that begin with an open-class POS (nouns, verbs, adjectives and adverbs), that end with an open-class POS or an adverbial particle, and that do not contain any other POS in between except closed-class POS for determiners, adpositions and conjunctions.
Parse Tree Spans. We consider all sub-trees that include at least one open-class POS and no more than 3 words altogether. We only extract maximal spans, i.e., not extracting sub-trees that are fully included in other extracted sub-trees.
Boolean. Boolean questions are frequent in VQA benchmarks (Goyal et al., 2017). Yet, 'yes' and 'no' are not found in captions, and so cannot be extracted as candidates by extracting text spans from captions. To this end, we also add 'yes' and 'no' as candidate answers and generate one question per candidate (see Section 3.2).
How many? 0. Captions do not normally contain mentions of ‘zero’ object counts. Hence, marking spans in a caption does not generate questions with the answer ‘0’. Therefore, we randomly sample a generated “How many?” question (with a non-zero answer) from a different caption and add it with the answer changed to ‘zero’ to the candidate set of the target caption. This procedure is potentially noisy because the answer for the sampled question could be non-zero also for the target image. From a manual inspection of 200 such questions, we found this to happen infrequently – about  $4.5\%$ .
Our extraction method covers various answer candidates such as compound nouns, noun phrases,
'https://spacy.io/
Figure 2: Visual Question Generation with Question Answering validation  $(\mathrm{VQ}^2\mathrm{A})$  has three main stages: Candidate Answer Extraction (Section 3.1), Question Generation (Section 3.2), and Question-Answering Filtering (Question Answering + Answer Validation, Section 3.3).
Candidate Answer	Noun Phrase	POS	Parse Tree	Boolean
‘two’	V	V
‘bears’	V
‘two bears’	V	V
‘laying’	V
‘laying down’	V
‘ice’	V
‘the ice’	V
‘on the ice’	V
‘no’	V
‘yes’	V
Table 1: Answer candidates extracted from the sentence "two bears are laying down on the ice" and the mechanism used to extract them.
named entities, boolean answers, cardinal and ordinal numbers, verbs and their compounds, (multiword) adjectives and prepositional phrases. Table 1 provides an example of candidate answers of various types and the mechanism used to extract them.
Our question generation model,  $\mathrm{q} = QG(\mathrm{a},\mathrm{cap})$ , takes as input a caption, cap, and a candidate answer span within it, a, and generates a question q, whose answer given the input caption is the input answer span. Importantly, the answer a does not need to appear verbatim in the caption, enabling the generation of questions for answer types like boolean and zero counts (see Section 3.1).
Given the advances in neural text generation, including models like T5 (Raffel et al., 2020), we choose to use a neural generation model as  $QG$ . Concretely, we use a T5-XXL model and further fine-tune it on SQuAD1.1 (Rajpurkar et al., 2016) for question generation. We take the top-scoring generated question for each caption-answer input. We note that our QG model is trained on a question
answering dataset that is not caption-specific, and therefore is not optimized for caption inputs. From manual inspection of hundreds of generated questions, our QG model copes well with captions as input; see examples in Table 2 and Section 3.5.
3.3 Question-Answer Filtering
Generative models may hallucinate, that is, generate content that is inconsistent with its input source (Alberti et al., 2019; Honovich et al., 2021). To mitigate this, we follow (Alberti et al., 2019) and apply round-trip consistency by answering the generated question on the caption text with a question answering model. If the answer does not match the answer candidate offered as input to the question generation model, the generated question is discarded.
We use the token-level F1 score (Wang et al., 2020) to determine if the candidate answer and the QA model's answer is a match; If the score is above a threshold (manually set to 0.54, exemplified in Table 2), the pair is a match. For question answering, we use a T5-XXL model and further fine-tune it on SQuAD2.0 (Rajpurkar et al., 2018) and Natural Questions (Kwiatkowski et al., 2019).
3.4 Sources of Image/Caption Data
To gain insights on  $\mathrm{VQ^2A}$  potential performance, we generate VQA triplets with  $\mathrm{VQ^2A}$  from two sources of image captions: MSCOCO Captions (COCO-CAP) (Chen et al., 2015) and Conceptual Captions (CC3M) (Sharma et al., 2018). COCOCAP captions contains 123,287 images from the COCO dataset (Lin et al., 2014), each with 5 gold captions manually created by raters with careful guidelines. CC3M contains 3.32M images
Candidate Answer	Generated Question	Validated Answer	Match Score & Result
‘two’	‘How many bears are laying on the ice?’	‘two’	1.0 (Pass)
‘bears’	‘What are the two animals laying on the ice?’	‘bears’	1.0 (Pass)
‘two bears’	‘How many bears are laying on the ice?’	‘two’	1.0 (Pass)
‘laying’	‘What are the bears doing?’	‘laying down on the ice’	0.4 (Fail)
‘laying down’	‘What are the bears doing?’	‘laying down on the ice’	0.7 (Pass)
‘ice’	‘Two bears are laying down on what?’	‘the ice’	1.0 (Pass)
‘the ice’	‘Where are the bears laying?’	‘on the ice’	0.7 (Pass)
‘on the ice’	‘Where are the bears laying?’	‘on the ice’	1.0 (Pass)
‘no’	‘Are the bears sleeping?’	‘yes’	0.0 (Fail)
‘yes’	‘Are the bears on the ice?’	‘yes’	1.0 (Pass)
‘zero’	‘How many people are sitting down?’	-	Pass by definition
Table 2: Question/answer pairs generated from the sentence "two bears are laying down on the ice" and the filtering decision. For answer candidate 'zero', no validation is performed.
Dataset	Image	VQA examples
train	dev	train	dev
\( VQ^2A \) COCO	114.9K	8.4K	3.50M	257.5K
\( VQ^2A \) CC3M	3.32M	15.8K	13.29M	61.2K
COCOQA	64.5K	4.7K	108.7K	38.6K
VQA2.0	114.9K	8.4K	582K	65.1K
GQA	82.4K	0.4K	1.08M	12.6K
OKVQA	9K	5K	8.3K	4.7K
Table 3: Sizes of our generated VQ $^2$ A data (top two rows) and VQA datasets used in our experiments.
automatically-collected from the web, each with one associated alt-text which we treat as a silver caption.
These datasets are quite different. Both the amount and the domain of CC3M images are larger and its captions look more plausible for capturing a larger set of object/attribute/action annotations. On the other hand, COCO-CAP's captions are cleaner and represent image content more adequately (see also Section 3.5). Thus, using COCO-CAP would show the potential of training a VQA model using  $\mathrm{VQ^2A}$  in a "cleaner" zero-shot setup, where captions are human-curated. Using CC3M would indicate the potential of training on noisy web image-alt-text pairs, where scaling up to billions of examples is possible.
To quantify the impact of our method, we focus on VQA classification for the VQA2.0 (Goyal et al., 2017), GQA (Hudson and Manning, 2019), and OKVQA (Marino et al., 2019) benchmarks (see Section 4.2). We thus restrict our classifier to top 5,971 answers that are part of a unified answer vocabulary from these benchmarks (Appendix B.1). To this end, we remove triplets whose answers are not in the target answer vocabulary, and leave the study of using all generated triplets to future work. We then split our datasets into train/dev sets. In particular, since the images in VQA2.0 are taken from COCO, we split the COCO dataset based on the standard VQA2.0 train/dev splits of *train2014 and minival2014 (Jiang et al.,
2018) $^{2}$ . For the CC3M dataset, we use the default CC3M train/dev splits (Sharma et al., 2018). For each unique image-question pair in the dev split, we construct an answer target of size 10, following VQA2.0, by reducing or expanding the set of seed answers that occur for this image-question pair. Additional details are in Appendix B.1.
Table 3 depicts the size of the induced datasets, named VQ $^2$ A-COCO and VQ $^2$ A-CC3M, as well as the VQA datasets used in our experiments.
3.5 Quality Analysis
To measure the quality of the generated datasets, we sampled 800 examples from each of the  $\mathrm{VQ^2A}$  COCO and  $\mathrm{VQ^2A}$  -CC3M datasets. The sample was split between four authors, who assessed whether the answer to the question in an example is justified based on the example's image. For each dataset, 50 examples were rated by all raters, resulting in a free-margin Kappa (Randolph, 2005) of 0.71 for  $\mathrm{VQ^2A}$  -COCO and 0.59 for  $\mathrm{VQ^2A}$  -CC3M, corresponding to high inter-rater agreement. The measured percentage of valid triplets is  $87.3\%$  for  $\mathrm{VQ^2A}$  -COCO and  $66.0\%$  for  $\mathrm{VQ^2A}$  -CC3M. This shows the difference between the high-quality captions of COCO-CAP and the noisier web-based ones of CC3M.
Fig. 3 demonstrates the diversity of questions generates in the  $\mathrm{VQ^2A}$  datasets. One can see that a significant amount of questions generated by  $\mathrm{VQ^2A}$  for the shared VQA2.0/COCO image do not appear in VQA2.0. Additional analysis and examples are in Appendix A.
4 Visual Question Answering (VQA)
To assess the effectiveness of our automatic generation of VQA annotations, we perform extrinsic
2With the exception of OKVQA in which we split into train2014/val2014 to avoid using test images during training.
Question	Answers
How many pieces of fruit are in the bowl	“0”
Is there a refrigerator in the kitchen	“yes”
What color are the cabinets in the kitchen	“white”
Is the kitchen lit or dark	“lit”
Is there a stove in the kitchen	“no”
What color is the formula in the kitchen	“white”
What is on the door of the refrigerator	“papers”, “several papers”
Where are the papers on the refrigerator	“door”
What kind of kitchen does the house have	“small white formula kitchen”
Question	Answers
Is the tumbler dishwasher safe	“no”
Is the tumbler insulated	“yes”
What color is the tumbler	“blue”, “shiny blue”
What kind of blue is on the tumbler	“shiny”
What is the name of the blue drinkware item	“tumbler”
What is ceramic used for in the tumbler	“lining”
Which part of the tumbler is made of stainless steel	“exterior”
Which part of the tumbler is ceramic	“inner”
What is the purpose of the stainless steel exterior of the tumbler	“insulating”
What kind of steel is on the outside of the tumbler	“stainless”
What material is the inside of the tumbler made of	“ceramic”
What material is the outside of the tumbler made of	“steel”, “stainless steel”
Question	Answers
What is flying over the ocean	"eagle", "brown eagle"
What color is the bird's head	"white"
What color is the bird	"brown"
Is the brown eagle flying over land	"No"
Is the brown eagle flying over the ocean	"Yes"
What does the bird do over the water	"glides"
Which part of the bird is white	"head"
What color are the bird's wings	"brown", "brown wings"
What color is the eagle in the picture	"brown"
Why is the bird flying down to the water	"To catch food"
Question	Answers
What type of art is this	“vector”
Aside from watches what else is included in the illustration	“clocks”
What is the illustration of	“clocks and watches”
Is this a hand drawn clock and watch set	“yes”
How were the clocks and watches drawn	“hand”
In addition to gold what color is used for the clocks and watches in this illustration	“gray”
What colors are the clocks and watches in the illustration	“gray and gold”
What kind of illustration is this	“vector art”
Figure 3: Examples from VQ $^2$ A COCO (top) and VQ $^2$ A CC3M (bottom). Questions with the green background are also present in VQA2.0.
Figure 4: VQA model used in our experiments. The text encoder is initialized from a T5-base checkpoint, while the image-text encoder is initialized from scratch. The parameters of ResNet and Faster R-CNN are frozen during VQA training.
evaluations of the generated data by measuring its impact on a variety of established VQA benchmarks. We first describe the model, followed by the experimental setup and the results.
4.1 VQA Formulation and Model
Following the literature, we treat VQA as a classification task, i.e., vocab-based VQA. In particular, we treat our target answers as labels, where a label could be multi-token (e.g., "Christmas tree", "black and white", "play tennis"). We define our set of labels based on top answers in the training set of downstream VQA datasets, which allows for a fair comparison with most work in the VQA literature since Antol et al. (2015).
Since our work explores the impact of automatically-generated training data, we fix the VQA model architecture across all experimental conditions. Our model fuses the input image and question (Fig. 4). On the image side, we take global image features from ResNet-152 (He et al., 2016) pre-trained on ImageNet (Russakovsky et al., 2015) plus 16 region-of-interest image features from Faster R-CNN (Ren et al., 2015b) pre-trained on Visual Genome (Krishna et al., 2017), following the bottom-up-features paradigm (Anderson et al., 2018). On the question side, we use the encoder