SlowGuess/NAACL_2022_OCR huggingface dataset:
Nasrin Mostafazadeh, Ishan Misra, Jacob Devlin, Margaret Mitchell, Xiaodong He, and Lucy Vanderwende. 2016. Generating natural questions about an image. In ACL.
Shashi Narayan, Goncalo Simoes, Ji Ma, Hannah Craighead, and Ryan Mcdonald. 2020. QURIOUS: Question generation pretraining for text generation. arXiv preprint arXiv:2004.11026.
Preksha Nema, Akash Kumar Mohankumar, Mitesh M. Khapra, Balaji Vasan Srinivasan, and Balaraman Ravindran. 2019. Let's ask again: Refine network for automatic question generation. In EMNLP-IJCNLP.
Raul Puri, Ryan Spring, Mohammad Shoeybi, Mostofa Patwary, and Bryan Catanzaro. 2020. Training question answering models from synthetic data. In EMNLP.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning transferable visual models from natural language supervision. In ICML.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. JMLR.
Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don't know: Unanswerable questions for SQuAD. In ACL.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In EMNLP.
Justus J. Randolph. 2005. Free-marginal multirater kappa (multirater  $\kappa_{\mathrm{free}}$ ): An alternative to Fleiss' fixed-marginal multirater kappa. Joensuu Learning and Instruction Symposium.
Mengye Ren, Ryan Kiros, and Richard Zemel. 2015a. Exploring models and data for image question answering. In NIPS.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015b. Faster R-CNN: Towards real-time object detection with region proposal networks. In NIPS.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. 2015. ImageNet large scale visual recognition challenge. IJCV, 115(3):211-252.
Meet Shah, Xinlei Chen, Marcus Rohrbach, and Devi Parikh. 2019. Cycle-consistency for robust visual question answering. In CVPR.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. 2018. Conceptual Captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In ACL.
Noam Shazeer and Mitchell Stern. 2018. Adafactor: Adaptive learning rates with sublinear memory cost. In ICLR.
Amanpreet Singh, Vedanuj Goswami, Vivek Natarajan, Yu Jiang, Xinlei Chen, Meet Shah, Marcus Rohrbach, Dhruv Batra, and Devi Parikh. 2020. MMF: A multimodal framework for vision and language research. https://github.com/facebookresearch/mmf.
Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. 2020. VL-BERT: Pre-training of generic visual-linguistic representations. In ICLR.
Hao Tan and Mohit Bansal. 2019. LXMERT: Learning cross-modality encoder representations from transformers. In EMNLP-IJCNLP.
Damien Teney and Anton van den Hengel. 2016. Zero-shot visual question answering. arXiv preprint arXiv:1611.05546.
Kohei Uehara, Antonio Tejero-de-Pablos, Yoshitaka Ushiku, and Tatsuya Harada. 2018. Visual question generation for class acquisition of unknown objects. In ECCV.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NeurIPS.
Alex Wang, Kyunghyun Cho, and Mike Lewis. 2020. Asking and answering questions to evaluate the factual consistency of summaries. In ACL.
Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. 2021. SimVLM: Simple visual language model pretraining with weak supervision. arXiv preprint arXiv:2108.10904.
Jialin Wu, Zeyuan Hu, and Raymond J. Mooney. 2019. Generating question relevant captions to aid visual question answering. In NeurIPS.
Xing Xu, Tan Wang, Yang Yang, Alan Hanjalic, and Heng Tao Shen. 2021. Radial graph convolutional network for visual question generation. IEEE Transactions on Neural Networks and Learning Systems, 32(4):1654-1667.
Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid. 2021a. Just ask: Learning to answer questions from millions of narrated videos. In ICCV.
Jianwei Yang, Jiasen Lu, Stefan Lee, Dhruv Batra, and Devi Parikh. 2018. Visual curiosity: Learning to ask questions to learn visual recognition. In CoRL.
Zhengyuan Yang, Zhe Gan, Jianfeng Wang, Xiaowei Hu, Yu-mao Lu, Zicheng Liu, and Lijuan Wang. 2021b. An empirical study of GPT-3 for few-shot knowledge-based VQA. arXiv preprint arXiv:2109.05014.
Desen Yuan. 2021. Language bias in visual question answering: A survey and taxonomy. arXiv preprint arXiv:2111.08531.
Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, Ce Liu, Mengchen Liu, Zicheng Liu, Yumao Lu, Yu Shi, Lijuan Wang, Jianfeng Wang, Bin Xiao, Zhen Xiao, Jianwei Yang, Michael Zeng, Luowei Zhou, and Pengchuan Zhang. 2021. Florence: A new foundation model for computer vision. arXiv preprint arXiv:2111.11432.
Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao. 2021. VinVL: Revisiting visual representations in vision-language models. In CVPR.
Shijie Zhang, Lizhen Qu, Shaodi You, Zhenglu Yang, and Jiawan Zhang. 2017. Automatic generation of grounded visual questions. In *IJCAI*.
Luowei Zhou, Hamid Palangi, Lei Zhang, Houdong Hu, Jason J. Corso, and Jianfeng Gao. 2020. Unified vision-language pre-training for image captioning and VQA. In AAAI.
A Additional Examples and Analysis of Generated Data
Fig. 5 provides additional examples of  $\mathrm{VQ^2A}$  COCO and CC3M generated VQA triplets, showing the diversity compared to what can be found in VQA2.0.
Table 7 presents the top question prefixes and their distribution in the VQA2.0 and  $\mathrm{VQ^2A}$  -based dev sets, showing significant differences between datasets. Many questions in VQA2.0 are of boolean answer type, e.g. 'is the', 'is there' and 'does the', summing to  $29.2\%$  . In addition, (how many') questions are frequent,  $11\%$  . Finally, questions for the color attribute are standing out with  $9 \%$  .On the other hand, COCO and CC3M questions are more explanatory in nature, with the majority of questions  $(45.5\%)$  in COCO,  $43.9\%$  in CC3M) of the form what is/are/do/does/type'. Another type that is more prominent in COCO and CC3M are where is/are' questions, which are more than twice frequent compared to VQA2.0.
Another difference between the manually curated VQA2.0 dataset and the  $\mathrm{VQ^2A}$  automatically generated datasets is question and answer word length distribution (Fig. 6 and 7). The questions in  $\mathrm{VQ^2A - CC3M}$  and  $\mathrm{VQ^2A - COCO}$  have an average word length of 8.3 and 7.8 respectively, while the average VQA2.0 is 6.3. Inspecting the generated questions, we noticed that QG model tends to quote parts of the caption, extending the question length. The average answer word length in  $\mathrm{VQ^2A - CC3M}$  and  $\mathrm{VQ^2A - COCO}$  is 1.76 and 1.85 words respectively, while in VQA2.0 it is 1.1. While all answers tend to be short, the  $\mathrm{VQ^2A}$ -induced datasets have more "detailed" answers of length 2-3 words.
Fig. 8 offers a more visual view of the differences between question type distribution presented in Table 7.
Table 8 depicts the percentage of questions of each type (prefix) that were retained (not filtered out) when applying the question answer validation phase of  $\mathrm{VQ^2A}$  (Section 3.3).
B.1 Details on Data Processing
Our default question and answer preprocessor is based on (Jiang et al., 2018; Singh et al., 2020)<sup>6</sup>,
with the exception of GQA which we use<sup>7</sup>. The unified answer vocabulary used in our experiments is the union of top answers from existing COCO-based VQA benchmarks: VQA2.0 (3,128, minimum answer frequency=9), GQA (1,843, all), OKVQA (2,000, top), and Visual7W (3,140, minimum answer frequency=3) of total size 5,971
For each image-unique question pair generated by our  $\mathrm{VQ^2A}$  approach, we reduce or expand a list of possibly different candidate answers based on the list length, such that we eventually have a target list of answers of size 10. In particular, we first sort the answers based on their lengths ("dog" before "black dog"), and select up to top-10 answers. If the list length is less than 10, we replicate each of the top answers one-by-one until we have the list of size 10, similar to the process in OKVQA(Marino et al., 2019). This is to ensure that we can adopt VQA Accuracy to make the performance comparison.
B.2 Details on Training and Evaluating Visual Question Answering
Our code for the VQA model is based on the Flaxformer framework. Both the text encoder and the multi-modal encoder have 6 blocks of Transformers, each of which consists of self-attention and a feed-forward network. We use 12 heads of inner dimension of 64, the embedding dimension of 768, and the MLP dimension of 2048. During training, we use Adafactor (Shazeer and Stern, 2018), with an initial learning rate of 0.0025, a linear warm-up step of 5K for (pre-)training and 1K for fine-tuning, and an "inverse square root" learning rate schedule  $\frac{1}{\sqrt{\max(n,k)}}$ , where  $n$  is the current training iteration and  $k$  is the number of warm-up steps. We use a dropout rate of 0.0. We train each of the models with data parallelism using 16 Cloud TPU Pods<sup>9</sup>, each with a batch size of 256, unless otherwise stated.
The default numbers of training steps during training and fine-tuning are 100K and 30K, respectively. The exceptions are OKVQA (30K/15K) and  $\mathrm{VQ^2A}$  CC3M (150K/NA). In addition, in the two-stage training where we fine-tune a  $\mathrm{VQ^2A}$ -CC3M model with  $\mathrm{VQ^2A}$  COCO, we also use 100K steps. Each single training run on average took fewer than 10 hours, including the time used to evaluate
$^{6}$ https://github.com/facebookresearch/ mmf/blob/main/mmf/datasets/processors/ processors.py
$^{7}$ https://github.com/stanfordnlp/mac-network/blob/gqa/preprocess.py $^{8}$ https://github.com/google/flaxformer $^{9}$ https://cloud.google.com(tpu
Question	Answers
How many people are in the picture	“0”
Is there a stoplight at the intersection of S Lane St. and 12th Ave S.	“no”
Is there a street sign in the picture	“yes”
On what is a street sign pictured	“hill”
What color is the street sign next to the dirt field by the playground	“green”
What is in front of the street sign	“gravel”
Which avenue meets S Lane St	“12th”
Which street does 12th Ave S intersect with	“lane st”
What does a street sign do at the intersection of S Lane St. and 12th Ave S.	“marks”
Figure 5: Additional examples from VQ $^2$ A COCO (top) and VQ $^2$ A CC3M (bottom). Questions with the green background are present in VQA2.0.
Question	Answers
Where is the woman in the photo	“car”
Is the woman in the picture doing her makeup	“yes”
What adjective would you use to describe the woman in the car	“pretty”
Who is in the car doing makeup	“pretty young woman”
What is the woman in the car doing	“makeup” “doing makeup”
Question	Answers
Pink shoes are for whom	"women"
What are the pink shoes for	"for women"
What item for women is all pink	"shoes"
What are all pink	"shoes for women"
Question Prefix	VQA2.0 %	\( \mathrm{VQ}^2\mathrm{A}\text{-}\mathrm{COCO}\% \)	\( \mathrm{VQ}^2\mathrm{A}\text{-}\mathrm{CC3M}\% \)	Question Example from \( \mathrm{VQ}^2\mathrm{A}\text{-}\mathrm{COCO} \)
‘What is’	0.140	0.288	0.217	‘What is the man swinging?’
‘How many’	0.110	0.022	0.005	‘How many people are standing in front of a tv?’
‘Is the’	0.098	0.084	0.053	‘Is the baby wearing a Santa hat?’
‘What color’	0.090	0.022	0.018	‘What color is the man’s hair?’
‘Is this’	0.082	0.008	0.015	‘Is this a safe way to fly?’
‘Is there’	0.037	0.011	0.022	‘Is there a pool in the backyard?’
‘What kind’	0.025	0.049	0.078	‘What kind of truck is the yellow one?’
‘What are’	0.024	0.049	0.022	‘What are the sheep and other animals roaming?’
‘Are the’	0.024	0.022	0.007	‘Are the apples on the cutting board green?’
‘Are there’	0.020	0.002	0.004	‘Are there any exceptions to this rule?’
‘Where is’	0.019	0.071	0.034	‘Where is the tennis player pictured?’
‘What type’	0.018	0.006	0.022	‘What type of picture is this?’
‘Is it’	0.017	0.001	0.005	‘Is it possible to eat a whole pizza?’
‘Does the’	0.014	0.007	0.007	‘Does the adult giraffe have any young?’
‘What does’	0.011	0.015	0.038	‘What does a giraffe do with its long neck?’
‘Where are’	0.006	0.032	0.014	‘Where are the skateboarders in the photo?’
‘Who is’	0.005	0.054	0.020	‘Who is in the photo?’
‘What do’	0.002	0.003	0.018	‘What do the father and son ride?’
‘What was’	0.000	0.009	0.023	‘What was the woman looking at?’
‘What did’	0.000	0.001	0.021	‘What did the cat lay inside of?’
Table 7: Most popular question prefix distribution on valid questions whose answers are in the  $6\mathrm{k}$  target vocabulary.
Figure 6: Question length distributions per dataset.
Figure 7: Answer length distributions per dataset.
a checkpoint — every 1K iterations. For instance, training on VQA2.0 took approximately 7 hours,  $\mathrm{VQ^2A}$  COCO 13 hours,  $\mathrm{VQ^2A}$  CC3M 10 hours. Note that  $\mathrm{VQ^2A}$  COCO has larger evaluation set than other datasets, hence taking longer time to to
train then  $\mathrm{VQ^2A}$  CC3M.
The hyperparameters for Transformers are selected to be consistent with a T5-base checkpoint, which has 220 million parameters (Raffel et al., 2020) (except that now we have 2 encoders rather
Figure 8: VQA2.0 (top),  $\mathrm{VQ^2A}$  -COCO (middle),  $\mathrm{VQ^2A}$  -CC3M (bottom) sunburst plots of question prefixes.
Question Prefix	VQ2A-COCO Filter Pass Ratio	VQ2A-CC3M Filter Pass Ratio
‘What is’	0.73	0.65
‘Is the’	0.64	0.39
‘What kind’	0.84	0.80
‘How many’	0.83	0.51
‘What color’	0.92	0.90
‘Where is’	0.79	0.79
‘Is this’	0.83	0.62
‘What are’	0.75	0.71
‘Who is’	0.85	0.79
‘Is there’	0.73	0.47
‘What does’	0.75	0.67
‘Are the’	0.58	0.32
‘Where are’	0.80	0.81
‘What type’	0.84	0.81
‘What was’	0.72	0.67
‘Does the’	0.60	0.43
‘Are there’	0.80	0.62
‘What do’	0.76	0.72
‘What did’	0.69	0.64
‘Is it’	0.62	0.59
Table 8: Question filtering stats.
than an encoder and a decoder). We initially tuned the initial learning rate (0.0125, 0.075, 0025, 0.00125, 0.00075) and the dropout rate (0.0, 0.1, 0.2) on a fully-supervised model on VQA2.0 baseline using VQA Accuracy and observed that 0.0025 and 0.0 work robustly across our experiments but we did not extensively tuned them in all of our experiments.
We implement VQA Accuracy ourselves based on the official challenge page for VQA2.0<sup>10</sup>.
Table 9 offers the Accuracy of the supervised VQA2.0 model, as well as of the zero-shot  $\mathrm{VQ^2A}$  models, on the VQA2.0 devset, split by most common question prefixes. The Table is sorted by the supervised model's Accuracy. It shows a several performance differences, first between all types of boolean questions, which all have high precision on all models, vs. other types, which show not only lower performance for all models, but also more significant performance drop between the supervised and zero-shot models.
Table 10 shows the zero-shot performance of models when using all of the  $\mathrm{VQ^2A}$  dev sets, not only the manually validated sample, for which Table 6 reports results. What we see is that the difference in performance on the whole  $\mathrm{VQ^2A}$  dev sets (Table 10) is similar in magnitude to that of the manually validated dev samples (Table 6), and most
10https://visualqa.org/evaluation.html
Question Prefix	VQA2.0 Supervised	VQ2A-COCO Zero-shot	VQ2A-CC3M Zero-shot
‘is there’	98.6	98.1	98.2
‘are there’	98.0	97.1	97.2
‘does this’	98.0	95.1	95.8
‘are they’	96.9	95.0	95.3
‘does the’	96.4	95.2	95.9
‘is it’	96.3	91.4	92.7
‘is this’	96.1	91.2	92.8
‘are the’	95.6	92.1	93.1
‘is the’	95.3	91.7	92.9
‘are these’	95.1	90.7	92.2
‘what color’	69.2	64.8	56.8
‘what kind’	56.3	35.8	31.4
‘what type’	54.4	32.3	30.8
‘what are’	51.3	40.2	33.9
‘how many’	49.3	29.4	19.5
‘what is’	48.5	39.4	32.2
‘where are’	40.9	33.9	27.6
‘where is’	35.1	26.0	23.0
‘what does’	33.0	24.1	20.3
‘what time’	23.6	11.9	12.7
Table 9: Average accuracy  $(\%)$  on VQA2.0 for the most common question prefixes.
Training data	Evaluation Benchmark
COCO-QA	VQA2.0	VQ2ACOCO	VQ2AC3M
COCOQA	70.3	11.7	11.5	3.7
VQA2.0	35.9	68.8	41.1	33.3
VQ2ACOCO	55.9	60.0	71.2	49.3
VQ2AC3M	42.1	56.5	60.3	69.5
Table 10:  $\mathrm{VQ^2A}$  as evaluation data for measuring robustness: VQA Accuracy when training on "row" and tested on "column"; diagonal (gray) numbers denote the supervised setting, non-diagonal numbers denote zero-shot cross-dataset setting. Best zero-shot is bold.
importantly, it keeps the order of models in terms of capabilities/performance. We therefore suggests that the utility of the  $\mathrm{VQ^2A}$  approach could go beyond training; it can be used as an automatic test-bed for VQA robustness, if not for absolute figures, for ranking models for robustness zero-shot capabilities.
Table 11 shows the effect of candidate answer types on the VQA2.0 performance. We train our model on  $\mathrm{VQ}^2\mathrm{A}$  COCO or  $\mathrm{VQ}^2\mathrm{A}$  CC3M subsets with questions with (i) noun answers, (ii) yes/no answers, (iii) answers containing color-related tokens based on a list of common colors from Wikipedia, and (iv) answers containing digits from 0 to 100. We then evaluate models trained on these subsets on VQA2.0 using VQA Accuracy and the normalized version (by the percentage of evaluation questions with corresponding answer types. This highlights the importance of incorporating diverse answer candidates in our datasets. We also observe that  $\mathrm{VQ}^2\mathrm{A}$  CC3M is on par with  $\mathrm{VQ}^2\mathrm{A}$  COCO on
Training data	VQA Accuracy on VQA2.0
Standard	Normalized
VQ2A COCO	60.0	60.0
VQ2A COCO nouns	10.5	32.5
VQ2A COCO yes/no	38.4	94.3
VQ2A COCO color	6.7	55.6
VQ2A COCO number	3.9	25.4
VQ2A CC3M	56.5	56.5
VQ2A CC3M nouns	8.8	27.2
VQ2A CC3M yes/no	38.4	94.4
VQ2A CC3M color	6.0	49.5
VQ2A CC3M number	3.4	22.1
Table 11: Effect of candidate answer types on the VQA2.0 performance.
Training data	VQA Accuracy on VQA2.0
VQ2A COCO (100%)	60.0
VQ2A COCO (50%)	58.5
VQ2A COCO (20%)	56.7
VQ2A COCO (10%)	55.4
VQ2A CC3M (100%)	56.5
VQ2A CC3M (50%)	55.8
VQ2A CC3M (20%)	54.8
VQ2A CC3M (10%)	53.8
Table 12: Effect of dataset sizes on the VQA2.0 performance.
yes/no-answer questions but are behind on nouns, color, and number, which we attribute to their lower degree image-text relevance, less mentioning of colors (due to the style of alt-texts vs. captions), and digit substitution.
Table 12 shows the effect of scale on the VQA2.0 performance. We randomly sampled  $10\%$ ,  $20\%$ , and  $50\%$  of  $\mathrm{VQ^2A}$  COCO or  $\mathrm{VQ^2A}$  CC3M training data. We observe that the bigger the data, the higher the accuracy. However, the gain is diminishing. We identify improving the data generation process to achieve higher degree of diversity in the output as interesting future work.
Table 13 provides question-only baselines (no image features as input). Interestingly, the models trained on our generated  $\mathrm{VQ^2A}$  data has similar answer distributions to those of existing VQA benchmarks. At the same time, this reveals the exploitation of the language bias, suggesting that additional research on bias mitigation is needed, both in terms of model and data (existing benchmarks and our datasets).
D Further Considerations
Information that names or uniquely identifies individual people or offensive content. COCO
Approach	Evaluation Benchmark
VQA2.0	GQA	OKVQA
Zero-shot
questions VQ2A COCO	48.9	44.4	11.4
questions VQ2A CC3M	47.8	44.6	11.9
VQ2A COCO	60.0	51.3	18.0
VQ2A CC3M	56.5	49.9	19.1
Table 13: Zero-shot question-only baselines using  $\mathrm{VQ^2A}$  as training data.
Captions are human-curated and cleaned while the approach to collection of CC3M upholds rigorous privacy and ethics standards such as the removal of offensive content and hypernymization. This significantly mitigates the risks that our  $\mathrm{VQ^2A}$  datasets would contain such information.
Intended uses. Due to considerations and limitations as we mention in Section 6, COCO Captions, CC3M, and our induced  $\mathrm{VQ^2A}$  are intended to be used for research-only purposes.