SlowGuess/NAACL_2022_OCR huggingface dataset:
of a pre-trained T5-base checkpoint (Raffel et al., 2020). Given the image features and the output token embeddings of the question encoder, a Transformer (Vaswani et al., 2017) fuses the multi-modal intermediate representation and classifies it into the predefined answer space. We train the (randomly-initialized) fusing encoder and the text encoder end-to-end using standard cross-entropy loss. The parameters of both ResNet and Faster R-CNN are frozen during training. Additional details are given in Appendix B.2.
We consider three VQA benchmarks: VQA2.0 (Goyal et al., 2017), GQA (Hudson and Manning, 2019), and OKVQA (Marino et al., 2019). These datasets have their own characteristics and thus test different capability of VQA models. For instance, GQA puts emphasis on reasoning and OKVQA on external knowledge, whereas VQA2.0 is more general; VQA2.0 and GQA are order-of-magnitude larger than OKVQA; GQA is generated using a question engine while VQA2.0 and OKVQA are human-annotated.
For training and evaluating on VQA2.0, we use the standard train/dev splits *train2014 and minival2014 (Jiang et al., 2018). For GQA, we use the balanced v1.2 and combine the train and val splits for training and use the testdev split for evaluation, following the official guideline<sup>3</sup> and (Tan and Bansal, 2019). For OKVQA, we use the train/val splits for training/evaluation. Table 3 summarizes the sizes of the different datasets.
Evaluation Settings and Baselines. The main
3https://cs.stanford.edu/people/dorarad/gqa/evaluate.html
goal of our experiments is to explore the utility of our  $\mathrm{VQ^2A}$  data for transfer learning, as training or evaluation data.
Our main focus in this paper is on zero-shot evaluation. Still, fine-tuning would provide additional insight on using our induced data for pre-training. Therefore, following (Banerjee et al., 2021), we train VQA models on the generated  $\mathrm{VQ^2A}$  data and then evaluate them in two settings: (i) zero-shot evaluation, in which we evaluate our models as-is on the dev split of VQA2.0, GQA, or OKVQA; and (ii) fully-supervised fine-tuning evaluation, in which we further fine-tune our models on the training split of  $\mathrm{VQ^2A}$ , GQA, or OKVQA before evaluating them. When training on  $\mathrm{VQ^2A}$  data, we explore training on  $\mathrm{VQ^2A}$ -COCO only,  $\mathrm{VQ^2A}$ -CC3M only, and a two-stage training  $\mathrm{VQ^2A}$ -CC3M followed by  $\mathrm{VQ^2A}$ -COCO ( $\mathrm{VQ^2A}$  CC3M  $\rightarrow$  COCO).
Our baselines, which do not use  $\mathrm{VQ}^2\mathrm{A}$  data, include (i) our VQA model trained on template-based question generation data COCOQA4 (Ren et al., 2015a), (ii) state-of-the-art zero-shot WeaQA (Banerjee et al., 2021) and its fully-supervised variants, and (iii) our VQA model trained supervisely on each of the target benchmarks' training data.
Metrics. To be compatible with prior work, on VQA2.0 and OKVQA we measure the standard VQA Accuracy. It is the average score over 9 subsets of the ground-truth 10 answers<sup>5</sup>, where each score is: min(#answer occurrences/3, 1). On GQA, we measure Top-1 Accuracy against the single ground-truth answer.
We report several sets of experimental results that shed light both on the accuracy and on the robustness of VQA models trained on  $\mathrm{VQ^2A}$  data in this section, with additional results, analysis and ablation studies in Appendix C.
5.1 Zero-Shot Setting
Table 4 summarizes the outcomes of our VQA experiments on various benchmarks. Our main result is that the  $\mathrm{VQ^2A}$  models achieve new state-of-the-art results in the zero-shot transfer learning setting. The improvement in performance is large: to the best of our knowledge, previous state-ofthe-art zero-shot accuracy was  $46.8\%$  on VQA2.0
Approach	Evaluation Benchmark
VQA2.0	GQA	OKVQA
Zero-shot
VQ2A COCO, nouns only	10.5	-	-
COCOQA	11.7	4.4	6.3
WeaQA ZSL	46.8	33.7	-
VQ2A COCO	60.0	51.3	18.0
VQ2A CC3M	56.5	49.9	19.1
VQ2A CC3M → COCO	61.1	52.1	19.7
VQ2A CC3M +D	57.9	50.0	19.8
Fully-supervised
WeaQA FSL	65.3	55.2	-
w/o VQ2A data	68.8	61.8	22.1
w. VQ2A COCO	71.6	63.3	36.0
w. VQ2A CC3M	71.3	63.4	39.0
w. VQ2A CC3M → COCO	71.4	64.0	39.3
Human performance	82.4†	89.3‡	82.8†
† from the inter-annotator agreement of ground-truth answers.  
‡ from (Hudson and Manning, 2019).
Table 4:  $\mathrm{VQ^2A}$  as training data. Accuracy in zero-shot and fully-supervised settings. All results use our architecture, except WeaQA ZSL and WeaQA FSL, which are the zero-shot (ZSL + Patches + Encoder) and fully-supervised (FSL + Patches + Encoder) models in (Banerjee et al., 2021), respectively. +D stands for recovered raw CC3M alt-texts with digits.
and  $33.7\%$  on GQA by WeaQA (Banerjee et al., 2021), which also induces their training VQA data from COCO Captions. Our  $\mathrm{VQ^2A}$  -COCO model reaches  $60.0\%$  on VQA2.0 and  $51.3\%$  on GQA, an absolute improvement of  $+13.2\%$  and  $+17.6\%$ , respectively. Even higher accuracy for the zero-shot setting  $-61.1\%$  (VQA2.0) and  $52.1\%$  (GQA) is reached with the  $\mathrm{VQ^2A}$  CC3M  $\rightarrow$  COCO model (trained first on the CC3M-derived data and then fine-tuned on the COCO-derived data), establishing new state-of-the-art results.
Training the same model architecture on the manually-constructed VQA2.0 and GQA training sets in a fully-supervised manner achieves  $68.8\%$  and  $61.8\%$  accuracy, respectively. Hence, our results significantly close the performance gap between automatically-generated and manually-constructed training sources, indicating that the  $\mathrm{VQ^2A}$  method may reduce the need for human curated VQA training examples.
The captions for COCO images are carefully annotated to be of high-quality (Chen et al., 2015). Additionally, the VQA2.0 images are taken from COCO. To test the robustness of  $\mathrm{VQ^2A}$ , we also evaluate a  $\mathrm{VQ^2A - CC3M}$  model. While CC3M contains more image-alt-text pairs than COCO (see Table 3), the images are from a different distribution and the text annotations are noisier and may represent a larger spectrum of discourse intents
4Train/dev based on the standard VQA2.0 train/dev splits. 5 targets in OKVQA, replicated twice (Marino et al., 2019).
(Alikhani et al., 2020). In spite of these differences, the gap between COCO-based and CC3M-based  $\mathrm{VQ^2A}$  models is not large,  $60.0\%$  vs  $56.5\%$  on VQA2.0 and  $51.3\%$  vs.  $49.9\%$  on GQA. This result strengthens our previous observation, in that it does not seem to be crucial that the starting captions are manually-annotated; it appears that "silver" annotations such as the ones provided by CC3M are competitive in zero-shot VQA performance.
To cover the types of answers present in VQA benchmarks, there is a need for thorough extraction of various answer/question types (Section 3). The QACE model (Lee et al., 2021), for example, focuses only on noun- phrases as answer types. By analyzing the VQA2 devset, we find that only  $32\%$  of its answers are nouns. As such, it makes sense that, when limiting to only this answer type, the VQA Accuracy of  $\mathrm{VQ^2A}$  -COCO is  $10.5\%$  , compared to the  $60\%$  achieved with a full coverage. As another example, our model trained COCOQA (Ren et al., 2015a), which focuses on a few answer types and one-word answers, barely surpasses the accuracy of our COCO, nouns only baseline. For similar reasons, we want to be able to generate 'how many' questions from the CC3M data, even though the published annotations have been stripped of digits and numerals. To solve this problem, we recover the original captions from the CC3M URLs, generate questions of the type 'how many', and train an additional  $\mathrm{VQ^2A - CC3M + D}$  model. The results in Table 4 show a small but consistent improvement over vanilla  $\mathrm{VQ^2A - CC3M}$  , further closing the gap between  $\mathrm{VQ^2A}$  models using curated "gold" captions and noisier "silver" captions.
To gain further insights, we provide a breakdown of VQA Accuracy per VQA2.0 question types in Table 5. Boolean questions are the easiest and all models perform well on them. More challenging question types are 'How many?' and 'What is'. One reason could be the validity of various answers, like "several" for counts. 'What time?' is the most difficult, probably due to lack of such information in captions.
Finally, we provide zero-shot results on the more difficult OKVQA benchmark. In this setting, a supervised model reaches  $22.1\%$  accuracy, while  $\mathrm{VQ^2A}$  models in zero-shot setting achieve close to that -  $18.0\%$  with COCO and  $19.1\%$  with CC3M, while their combination reaches  $19.7\%$ ,  $-2.4\%$  shy of the supervised level. This result also supports the conclusion that creating training data with the
Question Prefix	VQA2.0 Supervised	VQ2A-COCO Zero-shot	VQ2A-CC3M Zero-shot
Boolean	96.3	93.2	94.2
‘What color’	69.2	64.8	56.8
‘What kind/is’	52.6	36.9	32.1
‘How many’	49.3	29.4	19.5
‘Where are/is’	38.0	30.0	25.3
‘What does’	33.0	24.1	20.3
‘What time’	23.6	11.9	12.7
Table 5: Aggregated average Accuracy on VQA2.0 for the most common question types.
$\mathrm{VQ^2A}$  method is a good replacement for small-scale supervised training data.
5.2 Fully-Supervised Setting
Another aspect of the  $\mathrm{VQ^2A}$  method that we want to evaluate is whether it produces training data that is similar with the human-annotated data, or it complements it. To this end, we perform experiments in which we first train a model using the  $\mathrm{VQ^2A}$  data, and then fine-tune it in a supervised manner using the human-annotated training data.
The results, in the Fully-supervised part of Table 4, tell two stories. For VQA2.0 and GQA, there is a small yet consistent improvement of the finetuned models on top of a model trained directly on the supervised data in each benchmark (labeled w/o  $\mathrm{VQ^2A}$ ). This indicates that, at least for these two benchmarks, there is a high overlap in the nature of the signal between the human-annotated data and the  $\mathrm{VQ^2A}$  data.
The results on OKVQA show a different trend. Here, training first with  $\mathrm{VQ^2A}$  boosts performance by  $+17.2\%$  compared to supervised training without  $\mathrm{VQ^2A}$ $(22.1\% \rightarrow 39.3\%)$ . The small scale of the OKVQA training set (Table 3) certainly contributes to this effect, but it also points to another aspect: question-answer pairs that subsume world knowledge can only be made available at-scale to models by means that are not bottlenecked by human-annotation processes.
5.3 Robustness of Existing VQA Training Sets
So far we have assessed the capability of models trained on  $\mathrm{VQ^2A}$  data. As a complementary study, we use 500 manually-validated random samples (see Section 3.5) from the dev part of each  $\mathrm{VQ^2A}$  dataset to assess VQA robustness for various training setups. We use the VQA Accuracy metric for the  $\mathrm{VQ^2A}$  datasets (10 target answers, see Section 3.4), and Top-1 Accuracy on COCOQA (one target answer).
Table 6 shows the results. The fully-supervised
Training data	Evaluation Benchmark (Acc %)
COCO-QA	VQA2.0	VQ2A COCO	VQ2A CC3M
COCOQA	70.3	11.7	13.2	5.8
VQA2.0	35.9	68.8	44.4	41.6
VQ2A COCO	55.9	60.0	72.6	56.8
VQ2A CC3M	42.1	56.5	65.6	76.4
Table 6: Manually-validated VQ²A data for robustness evaluation: Accuracy of training on "row" and tested on "column"; diagonal (gray) numbers denote supervised setting, non-diagonal numbers denote zero-shot cross-dataset setting. Best zero-shot is in bold.
models (diagonal, similar training and test distributions) achieve in-domain Accuracy around  $70\%$ , with  $\mathrm{VQ^2A}$  CC3M achieving slightly higher  $76.4\%$  Accuracy. When tested on out-of-domain (nondiagonal), however, each model poses performance degradation at different degrees. First, the model based on template-generated COCOQA does not generalize at all. Second, the VQA2.0 model sees significant accuracy drops, even on the COCO  $(44.4\%)$  and COCOQA  $(35.9\%)$ , which share a similar image domain with VQA2.0. This result provides another evidence that progress made on the VQA2.0 benchmark may not reflect progress on the VQA task in full (Chao et al., 2018a; Bras et al., 2020).
In contrast, both VQ $^2$ A COCO and VQ $^2$ A CC3M perform robustly with more modest performance drops. For instance, on COCOQA, VQ $^2$ A CC3M achieves even better performance than VQA2.0 (42.1% vs. 35.9%) despite being tested on out-of-domain images. This suggests that the VQ $^2$ A training data possesses a higher degree of question variations, provides better answer coverage, and exhibits less biases than the manually-curated VQA2.0 training data, at least enough to address these different benchmarks.
6 Considerations and Limitations
Automatic data generation is prone to erroneous outputs. In  $\mathrm{VQ^2A}$  these may include hallucinations of the generative model, incorrect negative sampling, and bad answer span extraction. In addition, the image captions may contain details not in the image, e.g. additional details only aware to the photo taker or personal opinions, or information that is inconsistent with the image due to human mistakes and biases. We address some of these issues automatically, filtering bad generations via question answering round-trip validation. In addition, the classification task itself curbs the effects
of such errors through the use of a fixed answer vocabulary. Yet, for automatic generation to be more robust, additional methods to narrow down mistakes or mismatches need to be developed.
The resulting VQA model incorporates and may reinforce some of the biases and stereotypes present in the data. For instance, it may learn that answering questions such as "What is the gender of this person?" is a binary choice dictated by shallow cues, or that the answer to "For whom is this room decorated?" depends on stereotypical features present (or not) in the room depicted in the image. Mitigation strategies for such issues go beyond the scope of this paper, but we encourage the research community to consider addressing these issues as central for the successful deployment of this technology.
In this paper, we show that high-quality VQA training data can be automatically induced at scale from existing image-caption datasets. Our method,  $\mathrm{VQ^2A}$ , annotates candidate answers using syntactic parsing of the captions and then derives questions for them using neural models for question generation and question answering verification. We demonstrate that VQA models trained only on such data exhibit high zero-shot performance with new state-of-the-art results on VQA2.0 and GQA. Additionally, we provide evidence for the brittleness of VQA systems built with human-annotated examples compared to the ones built with automatically-induced image-question-answer triplets using  $\mathrm{VQ^2A}$ .
For future work, we plan to explore even larger automatically-curated image-text datasets, consisting of billions of examples. In addition, we want to test the applicability of  $\mathrm{VQ}^2\mathrm{A}$  to languages other than English, for which human-annotated VQA data is scarce.
Acknowledgments. We would like to thank Or Honovich, Hagai Taitelbaum and Roee Aharoni for their help with question generation, Sebastian Goodman for his help with the VQA infrastructure, Piyush Sharma for his help with the Conceptual Captions, Nassim Oufattole for his early exploration of question generation, Gal Elidan, Sasha Goldshtein, and Avinatan Hassidim for their useful feedback.
Aishwarya Agrawal, Dhruv Batra, Devi Parikh, and Aniruddha Kembhavi. 2018. Don't just assume; look and answer: Overcoming priors for visual question answering. In CVPR.
Arjun Akula, Soravit Changpinyo, Boqing Gong, Piyush Sharma, Song-Chun Zhu, and Radu Soricut. 2021. CrossVQA: Scalably generating benchmarks for systematically testing vqa generalization. In EMNLP.
Chris Alberti, Daniel Andor, Emily Pittler, Jacob Devlin, and Michael Collins. 2019. Synthetic QA corpora generation with roundtrip consistency. In ACL.
Malihe Alikhani, Piyush Sharma, Shengjie Li, Radu Soricut, and Matthew Stone. 2020. Cross-modal coherence modeling for caption generation. In ACL.
Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. 2018. Bottom-up and top-down attention for image captioning and visual question answering. In CVPR.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. 2015. VQA: Visual question answering. In ICCV.
Pratyay Banerjee, Tejas Gokhale, Yezhou Yang, and Chitta Baral. 2021. WeaQA: Weak supervision via captions for visual question answering. In Findings of ACL-IJCNLP.
Ronan Le Bras, Swabha Swayamdipta, Chandra Bhagavatula, Rowan Zellers, Matthew E. Peters, Ashish Sabharwal, and Yejin Choi. 2020. Adversarial filters of dataset biases. In ICML.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, T. J. Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeff Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In NeurIPS.
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. 2021. Conceptual 12M: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In CVPR.
Wei-Lun Chao, Hexiang Hu, and Fei Sha. 2018a. Being negative but constructively: Lessons learnt from creating better visual question answering datasets. In NAACL.
Wei-Lun Chao, Hexiang Hu, and Fei Sha. 2018b. Cross-dataset adaptation for visual question answering. In CVPR.
Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dolkar, and C. Lawrence Zitnick. 2015. Microsoft COCO Captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325.
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. 2020. UNITER: Learning UNiversal Image-TExt Representations. In ECCV.
Jaemin Cho, Jie Lei, Hao Tan, and Mohit Bansal. 2021. Unifying vision-and-language tasks via text generation. In ICML.
Kaustubh D. Dhole and Christopher D. Manning. 2020. SynQG: Syntactic and shallow semantic rules for question generation. In ACL.
Esin Durmus, He He, and Mona Diab. 2020. FEQA: A question answering evaluation framework for faithfulness assessment in abstractive summarization. In ACL.
Adam Fisch, Kenton Lee, Ming-Wei Chang, Jonathan H. Clark, and Regina Barzilay. 2020. CapWap: Captioning with a purpose. In EMNLP.
Manas Gaur, Kalpa Gunaratna, Vijay Srinivasan, and Hongxia Jin. 2021. ISEEq: Information seeking question generation using dynamic meta-information retrieval and knowledge graphs. arXiv preprint arXiv:2112.07622.
Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017. Making the V in VQA matter: Elevating the role of image understanding in visual question answering. In CVPR.
Mandy Guo, Qinlan Shen, Yinfei Yang, Heming Ge, Daniel Cer, Gustavo Abrego, Keith Stevens, Noah Constant, Yun-Hsuan Sung, Brian Strope, and Ray Kurzweil. 2018. Effective parallel corpus mining using bilingual sentence embeddings. In WMT.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In CVPR.
Michael Heilman and Noah A. Smith. 2009. Question generation via overgenerating transformations and ranking. Technical report, Carnegie Mellon University.
Or Honovich, Leshem Choshen, Roee Aharoni, Ella Neeman, Idan Szpektor, and Omri Abend. 2021.  $Q^2$ : Evaluating factual consistency in knowledge-grounded dialogues via question generation and question answering. In EMNLP.
Drew A. Hudson and Christopher D. Manning. 2019. GQA: A new dataset for real-world visual reasoning and compositional question answering. In CVPR.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V. Le, Yunhsuan Sung, Zhen Li, and Tom Duerig. 2021. Scaling up visual and vision-language representation learning with noisy text supervision. In ICML.
Yu Jiang, Vivek Natarajan, Xinlei Chen, Marcus Rohrbach, Dhruv Batra, and Devi Parikh. 2018. Pythia v0.1: the winning entry to the vqa challenge 2018. arXiv preprint arXiv:1807.09956.
Kushal Kafle and Christopher Kanan. 2017. An analysis of visual question answering algorithms. In ICCV.
Kushal Kafle, Mohammed Yousefhussien, and Christopher Kanan. 2017. Data augmentation for visual question answering. In INLG.
Jihyung Kil, Cheng Zhang, Dong Xuan, and Wei-Lun Chao. 2021. Discovering the unknown knowns: Turning implicit knowledge in the dataset into explicit training examples for visual question answering. In EMNLP.
Ranjay Krishna, Michael Bernstein, and Li Fei-Fei. 2019. Information maximizing visual question generation. In CVPR.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael Bernstein, and Li Fei-Fei. 2017. Visual Genome: Connecting language and vision using crowdsourced dense image annotations. IJCV, 123(1):32-73.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural Questions: a benchmark for question answering research. TACL, 7:453-466.
Hwanhee Lee, Thomas Scialom, Seunghyun Yoon, Franck Dernoncourt, and Kyomin Jung. 2021. QACE: Asking questions to evaluate an image caption. In Findings of EMNLP.
Lianian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. 2019. VisualBERT: A simple and performant baseline for vision and language. arXiv preprint arXiv:1908.03557.
Xiujun Li, Xi Yin, Chunyuan Li, Xiaowei Hu, Pengchuan Zhang, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, Yejin Choi, and Jianfeng Gao. 2020. Oscar: Object-semantics aligned pre-training for vision-language tasks. In ECCV.
Yikang Li, Nan Duan, Bolei Zhou, Xiao Chu, Wanli Ouyang, Xiaogang Wang, and Ming Zhou. 2018. Visual question generation as dual task of visual question answering. In CVPR.
Tsung-Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dólar. 2014. Microsoft COCO: Common objects in context. In ECCV.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2021. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. arXiv preprint arXiv:2107.13586.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019. ViLBERT: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. In NeurIPS.
Jiasen Lu, Vedanuj Goswami, Marcus Rohrbach, Devi Parikh, and Stefan Lee. 2020. 12-in-1: Multi-task vision and language representation learning. In CVPR.
Chenyang Lyu, Lifeng Shang, Yvette Graham, Jennifer Foster, Xin Jiang, and Qun Liu. 2021. Improving unsupervised question answering via summarization-informed question generation. In EMNLP.
Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. 2019. OK-VQA: A visual question answering benchmark requiring external knowledge. In CVPR.
Yosi Mass, Boaz Carmeli, Haggai Roitman, and David Konopnicki. 2020. Unsupervised FAQ retrieval with question generation and BERT. In ACL.