SlowGuess/NAACL_2022_OCR huggingface dataset:
Comparing the fully fine-tuned setting across models and datasets it becomes clear that, although oftentimes performing comparably, our CCQA seq2seq pre-trained model underperforms the vanilla models in most cases. Seq2seq indomain pre-training on CCQA only reaches superior performance on the ELI5 dataset for all models, as well as on the GooAQ dataset for T5 Small. Showing that seq2seq pre-training on CCQA is effective in zero-shot scenarios, however only partially improves over baselines in the fine-tuned set
<sup>8</sup>Experiments are executed on Nvidia V100 32GB GPUs.
Zero-Shot	Fine-Tuned
Model	TQA AR	NQ-Short AR	NQ-Long R-L	ELI5 R-L	GooAQ R-L	TQA EM	NQ-Short EM	NQ-Long R-L	ELI5 R-L	GooAQ R-L
BART Large	BART Large
Rand. Init.	0.04	0.11	0.10	0.26	0.16	0.71	0.75	16.04	14.37	16.21
Vanilla	†4.91	†1.93	10.39	11.88	14.67	28.67	23.79	23.47	16.96	35.67
Vanillaa	26.50
CCQA	†5.14	†2.16	12.18	†15.21	†17.5	25.82	22.91	21.25	17.23	32.53
CCQA-d	4.80	2.13	10.33	11.91	14.88	27.84	23.96	24.56	17.27	35.92
T5 Small	T5 Small
Rand. Init.	0.05	0.11	1.13	1.49	0.80	0.44	0.54	10.86	13.06	8.71
Vanilla	†5.06	†1.74	9.16	7.55	†8.92	21.02	21.16	22.09	16.28	24.70
Vanilla b	19.00	23.00
CCQA	†5.13	†1.86	†13.63	†15.28	†15.46	17.55	19.50	22.05	16.33	25.35
T5 Base	T5 Base
Rand. Init.	0.04	0.11	0.00	0.00	0.00	0.32	0.38	13.58	12.72	7.93
Vanilla	†5.49	†2.02	†14.39	12.27	†14.99	26.25	23.04	25.36	16.58	29.36
Vanilla c	23.63	25.94
CCQA	†7.15	†3.19	†15.08	†15.69	†15.85	22.69	22.32	24.73	16.64	29.09
Results from Lewis et al. (2021a) b Khashabi et al. (2021) Roberts et al. (2020)
Table 3: Closed-book zero-shot and fine-tuned results. Best performance of fairly computed results per sub-table bold.  ${}^{ \dagger  }$  Zero-shot model outperforms fully fine-tuned randomly initialized transformer of same architecture. -d extension indicates denoising CCQA pre-training task. AR=Answer-level recall, EM=Exact Match, RL=Rouge-L.
ting, we investigate: (1) Additional experiments using the CCQA dataset for denoising-style pretraining  $(-d$  in Table 3) and (2) Evaluate additional low-resource scenarios, shown in Figure 4.
For our denoising-style in-domain pre-training experiment, we keep the available markup information, in line with HTLM (Aghajanyan et al., 2022b). As shown in Table 3, the in-domain CCQA denoising objective outperforms the vanilla BART Large model on 4 out of 5 benchmarks in the finetuned setting. We believe that this result, alongside the zero-shot performance of the seq2seq CCQA model, clearly shows the usefulness and generality of our CCQA corpus for closed-book open-domain question-answering.
Taking a closer look at low-resource scenarios, we evaluate the vanilla T5 Small model against our in-domain pre-trained approach using 5 proper subsets of the NQ-Long, GooAQ and ELI5 benchmark datasets, drawn at random. As presented in Figure 4, our CCQA model mostly outperforms the vanilla T5 Small model in low-resource scenarios with up to 10,000 data points. While the performance of our CCQA model is consistently better
on the ELI5 test-set, the vanilla baselines outperform our models fastest on the NQ-Long corpus. Additional low-resource experiments on T5 Base are shown in Table 6, in Appendix F.
4.2 Passage Retrieval
4.2.1 Task
For the passage retrieval task, an important component of most open-book QA systems (e.g., Lewis et al. (2020b); Izacard and Grave (2021)), models aim to extract a set of evidence passages from a large collection of documents through conditional ranking. To align our corpus with the passage retrieval task, we aggregate every question into a single data point, consisting of the question itself, alongside all available answers as either positive or negative contexts. If available, answer votes are used as a proxy to determine positive and negative (sometimes called "hard-negative") contexts. Following the practice in Fan et al. (2019), we assign every answer with at least 2 more upvotes than downvotes as a positive context and all other answers as negative. If answer votes are not available, we use the accepted/suggested label (shown
Figure 4: Low resource experiments comparing the Rouge-L score of vanilla T5 Small with our CCQA pre-trained models on NQ-long (left), GooAQ (center) and ELI5 (right).
Figure 5: High-level overview of the CCQA passage retrieval in-domain pre-training step (yellow) as part of the training pipeline for DPR. Language model pretraining shown in green. Task-dependent fine-tuning presented in red. Evaluation in blue. (1) Baseline pre-training/fine-tuning pipeline, (2) In-domain pretraining/fine-tuning pipeline.
in Figure 2) as an indicator for positive and negative contexts. In the absence of either criterion, we use all available answers as positive contexts.
4.2.2 Models & Training
For passage retrieval, we choose the Dense Passage Retriever (DPR) (Karpukhin et al., 2020), used in a variety of popular end-to-end open-book QA models, such as RAG (Lewis et al., 2020b) and FiD (Izacard and Grave, 2021). As shown in Figure 5, we start with the vanilla DPR model based on BERT (Devlin et al., 2019) and in-domain pre-train using questions and positive/negative passages from the CCQA dataset (yellow box in Figure 5), similar to Oğuz et al. (2022). In line with the training-flows of the closed-book models, we train DPR using either the vanilla setup (pre-training  $\rightarrow$  fine-tuning) or the in-domain pre-training ap
Model	TQA	NQ-Short
Acc@20	Acc@100	Acc@20	Acc@100
DPR	79.4	85.0	78.4	85.4
DPR v2	79.5	85.3	78.3	85.6
CCQA DPR	80.0	85.6	79.1	86.3
Table 4: Fine-tuned Dense Passage Retriever (DPR) accuracy measure on the TQA and NQ-Short datasets. DPR represents the original DPR model (Karpukhin et al., 2020), DPR v2 (Oğuz et al., 2022) indicates the updated codebase. CCQA DPR uses our CCQA pretrained DPR model for retrieval fine-tuning.
Bench. (test)	TQA	NQ-S	NQ-L	ELI5	GooAQ
Bench. (train)	11.9	4.9	5.2	3.0	26.9
CCQA (train)	0.4	1.9	2.3	0.5	26.9
Table 5: 8-gram question overlap (in %) between training sets and benchmark test-sets (inspired by Radford et al. (2019)). Bench (train) refers to the overlap between the respective training- and test-portion of the benchmark datasets, CCQA (train) identified overlaps between our dataset and the test-splits. False positive rate upper-bound by  $\frac{1}{10^8}$ . All inputs are normalized and lower-cased. NQ-S=NQ-Short, NQ-L=NQ-Long.
proach (pre-training  $\rightarrow$  in-domain pre-training  $\rightarrow$  fine-tuning), shown as circles (1) and (2) in Figure 5, respectively.
4.2.3 Datasets & Metrics
Following the original DPR paper (Karpukhin et al., 2020), we evaluate the passage retrieval task on the NQ-Short and TQA datasets presented in section 4.1.3, using the top-20 and top-100 retrieval accuracy (Acc@20/Acc@100) measures.
4.2.4 Hyper-Parameters
We use the default DPR hyper-parameters whenever possible (Karpukhin et al., 2020). For in
domain pre-training, we limit training to  $800k$  steps using a batch-size of 1,536 samples. During finetuning, we restrict the number of updates to  $20k$  steps with a batch-size of 128. The best checkpoint is selected based on the Mean Reciprocal Rank (MRR) measure, following Oğuz et al. (2022). We do not perform any hyper-parameter search.
4.2.5 Results
For the passage retrieval experiments, we compare our CCQA in-domain pre-trained DPR model against the vanilla DPR model published in Karpukhin et al. (2020), as well as the recently enhanced version (Oguz et al., 2022). Table 4 contains our empirical results, showing consistent improvements of our CCQA DPR model over the vanilla baselines. More specifically, the in-domain CCQA pre-training step increases the top-20 and top-100 accuracy score on the TQA benchmark dataset by over half a point, while the performance gap on NQ-Short shows consistent improvement of over  $0.7\%$ .
4.3 Evaluation Fairness: Dataset Overlap
With modern pre-training approaches using increasingly large datasets, accidental overlaps between pre-training corpora and benchmark datasets become more and more common (Lewis et al., 2021a). To analyze this threat to the integrity of our dataset and empirical analysis, we follow Radford et al. (2019) and evaluate the 8-gram question overlap of our CCQA training portion with the test-split of benchmark datasets using bloom filters. Table 5 shows a consistently smaller question overlap between CCQA and the benchmark test set, compared to the benchmark training split itself.
In this work, we presented our new web-scale CCQA dataset for in-domain model pre-training. Orthogonal to recent efforts on improving task-specific pre-training objectives, we show our dataset generation process, followed by detailed insights into key corpus dimensions of this new, largescale, natural, and diverse question-answering dataset. In a set of empirical evaluations, we confirm the initial intuition that the corpus presents a valuable resource for open-domain question-answering research. In our zero-shot, low-resource and fine-tuned experiments for open- and closed-book QA tasks, we show promising results across
multiple model architectures. With around 130 million question-answer pairs (55M unique) as well as additional meta-data, our CCQA dataset presents a versatile source of information, which has a large variety of applications in future work (e.g., question summarization, answer rating, answer ranking and many more).
We now discuss the three major ethical considerations impacting this paper:
Hate-speech, Harmful Gender and Racial Biases: With general web-data potentially containing hate-speech and harmful gender and racial biases, we believe that our extracted dataset based on the schema.org annotations is less impacted by these issues, with the schema.org annotation representing a good proxy for high-quality, professionally curated websites. As a result, we believe that the severity of this issue is significantly reduced. Furthermore, in our human evaluation, we find no signs of the above mentioned biases. We leave computational approaches to determine dataset biases for future work (e.g., the Word Embedding Association Test (Caliskan et al., 2017) and Sentence Encoder Association Test (May et al., 2019)).
Data Availability: We do not directly provide the CCQA dataset, but enable third parties to generate the corpus through our published dataset generation scripts available at https://github.com/facebookresearch/CCQA.
Hallucinations and Factual Errors: As shown in the evaluation section, our model is able to generate reasonable answers for factoid and long-form questions. The inferred answers are fluent and human-like, but may also contain hallucinations and factual errors, especially for the challenging closed-book question-answering task. Without a guarantee of the predicted answers being factually correct, they can potentially spread misinformation if not properly corrected.
We want to thank the anonymous reviewers for their valuable feedback and acknowledge our colleagues and peers for their help throughout this project, specifically Anchit Gupta, Gagan Aneja and Patrick Lewis for their valuable input on important decisions.
Armen Aghajanyan, Anchit Gupta, Akshat Shrivastava, Xilun Chen, Luke Zettlemoyer, and Sonal Gupta. 2021. Muppet: Massive multi-task representations with pre-finetuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5799-5811, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Armen Aghajanyan, Bernie Huang, Candace Ross, Vladimir Karpukhin, Hu Xu, Naman Goyal, Dmytro Okhonko, Mandar Joshi, Gargi Ghosh, Mike Lewis, et al. 2022a. Cm3: A causal masked multimodal model of the internet. arXiv preprint arXiv:2201.07520.
Armen Aghajanyan, Dmytro Okhonko, Mike Lewis, Mandar Joshi, Hu Xu, Gargi Ghosh, and Luke Zettle-moyer. 2022b. HTLM: Hyper-text pre-training and prompting of language models. In International Conference on Learning Representations.
Aylin Caliskan, Joanna J. Bryson, and Arvind Narayanan. 2017. Semantics derived automatically from language corpora contain human-like biases. Science, 356(6334):183-186.
Wei-Cheng Chang, Felix X. Yu, Yin-Wen Chang, Yiming Yang, and Sanjiv Kumar. 2020. Pre-training tasks for embedding-based large-scale retrieval. In International Conference on Learning Representations.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Édouard Grave, Myle Ott, Luke Zettle-moyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440-8451.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. 2019. ELI5: Long form question answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3558-3567, Florence, Italy. Association for Computational Linguistics.
Suchin Gururangan, Ana Marasovic, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don't stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the
Association for Computational Linguistics, pages 8342-8360, Online. Association for Computational Linguistics.
Gautier Izacard and Édouard Grave. 2021. Leveraging passage retrieval with generative models for open domain question answering. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 874-880.
Robin Jia, Mike Lewis, and Luke Zettlemoyer. 2021. Question answering infused pre-training of general-purpose contextualized representations. arXiv preprint arXiv:2106.08190.
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1601-1611, Vancouver, Canada. Association for Computational Linguistics.
Armand Joulin, Edouard Grave, Piotr Bojanowski, Matthijs Douze, Hérve Jégou, and Tomas Mikolov. 2016. Fasttext.zip: Compressing text classification models. arXiv preprint arXiv:1612.03651.
Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2017. Bag of tricks for efficient text classification. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 427-431, Valencia, Spain. Association for Computational Linguistics.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769-6781, Online. Association for Computational Linguistics.
Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannaneh Hajishirzi. 2020. UNIFIEDQA: Crossing format boundaries with a single QA system. In *Findings of the Association for Computational Linguistics: EMNLP* 2020, pages 1896-1907, Online. Association for Computational Linguistics.
Daniel Khashabi, Amos Ng, Tushar Khot, Ashish Sabharwal, Hannaneh Hajishirzi, and Chris Callison-Burch. 2021. Gooaq: Open question answering with diverse answer types. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 421-433.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones,
Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452-466.
Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. 2019. Latent retrieval for weakly supervised open domain question answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6086–6096, Florence, Italy. Association for Computational Linguistics.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020a. BART: Denoising sequence-to-sequence pretraining for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871-7880, Online. Association for Computational Linguistics.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Kuttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020b. Retrieval-augmented generation for knowledge-intensive nlp tasks. In Advances in Neural Information Processing Systems, volume 33, pages 9459-9474. Curran Associates, Inc.
Patrick Lewis, Pontus Stenetorp, and Sebastian Riedel. 2021a. Question and answer test-train overlap in open-domain question answering datasets. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 1000-1008.
Patrick Lewis, Yuxiang Wu, Linqing Liu, Pasquale Minervini, Heinrich Kuttler, Aleksandra Piktus, Pontus Stenetorp, and Sebastian Riedel. 2021b. PAQ: 65 million probably-asked questions and what you can do with them. Transactions of the Association for Computational Linguistics, 9:1098-1115.
Chandler May, Alex Wang, Shikha Bordia, Samuel R. Bowman, and Rachel Rudinger. 2019. On measuring social biases in sentence encoders. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 622-628, Minneapolis, Minnesota. Association for Computational Linguistics.
Barlas Oğuz, Kushal Lakhotia, Anchit Gupta, Patrick Lewis, Vladimir Karpukhin, Aleksandra Piktus, Xilun Chen, Sebastian Riedel, Wen-tau Yih, Sonal Gupta, et al. 2022. Domain-matched pre-training tasks for dense retrieval. In Findings of the Association for Computational Linguistics: NAACL-HLT 2022.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67.
Adam Roberts, Colin Raffel, and Noam Shazeer. 2020. How much knowledge can you pack into the parameters of a language model? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5418-5426, Online. Association for Computational Linguistics.
Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Edouard Grave. 2020. CCNet: Extracting high quality monolingual datasets from web crawl data. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 4003-4012, Marseille, France. European Language Resources Association.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtopicz, et al. 2019. Huggingface's transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771.
Wenhan Xiong, Xiang Li, Srini Iyer, Jingfei Du, Patrick Lewis, William Yang Wang, Yashar Mehdad, Scott Yih, Sebastian Riedel, Douwe Kiela, and Barlas Oguz. 2021. Answering complex open-domain questions with multi-hop dense retrieval. In International Conference on Learning Representations.
A CCQA Dataset Generation Algorithm
B Detailed Topic Distribution
Topic	Top 5 Appearances in CCQA
Top-Level	Regional (38.90%)	Society (21.14%)	Business (8.36%)	Sports (7.04%)	Rec. (6.20%)
Regional	North America (61.48%)	Europe (34.69%)	Asia (1.28%)
Society	Issues (76.89%)	Religion (18.39%)	Philosophy (2.36%)	Law (1.41%)
Business	Industrial Goods (13.41%)	Energy (9.75%)	Textiles (9.75%)	Construction (7.31%)	Business Services (6.09%)
Sports	Golf (81.08)	Aquatiques (10.81%)	Events (2.70%)	Water Sports (2.70%)	Lacrosse (1.35%)
Recreational	Food (56.92)	Outdoors (23.07%)	Travel (12.30%)	Motorcycles (3.07%)	Pets (1.53%)
C Detailed Question Word Distribution
Table 6: Fine-grained CCQA dataset topic distribution of 1000 randomly chosen domains retrieved through the DMOZ/Curlie annotation at https://curlie.org/. Only showing sub-topics with  $\geq 1\%$ .
Question-Word	What	How	When	Which	Where	Why	Who	Whose
Frequency	5.3M	4.3M	1.4M	1.4M	881k	717k	514k	25k
(36.20%)	(29.80%)	(9.68%)	(9.64%)	(6.04%)	(4.92%)	(3.53%)	(0.17%)
Table 7: Question word distribution for all 8 English question words with their number of appearance in the CCQA corpus and their relative frequency.
D HTML Markup Tag Distribution
Rank	HTML Markup Tag Distribution
1-5	p (28.48%)	a (14.89%)	br (14.87%)	li (10.04%)	span (5.77%)
6-10	strong (4.93%)	code (4.59%)	em (2.79)	div (2.38%)	ul (2.27%)
11-15	pre (1.80%)	b (1.70%)	blockquote (1.14%)	h3 (0.89%)	td (0.88%)
16-20	h2 (0.48%)	ol (0.42%)	tr (0.42%)	h1 (0.35%)	i (0.24%)
21-25	sup (0.17%)	tbody (0.12%)	table (0.12%)	u (0.12%)	sub (0.11%)
E Sensibility and Answerability Examples
Table 8: Distribution of the 25 most common HTML tags in CCQA.
Metric	Type	Example	Explanation
Q-sensibility	Pos	What languages do you speak?	Q-Sensible, since question internally makes sense
Neg	How blue is the number 7?	Not Q-Sensible, since question internally makes no sense
Q-answerability	Pos	How can I purchase affordable Flats in Vancouver?	Q-Answerable, since a single answer exists
Neg	What languages do you speak?	Not Q-Answerable, since no single answer exists, but depends on the (unavailable) context
QA-sensibility	Pos	Which is the busiest month to travel from London to Delhi? → July	QA-Sensible, since question and answer make sense together
Neg	How can I purchase affordable Flats in Vancouver? → There are many affordable Flats available.	Not QA-Sensible, since answer does not answer the question
Table 9: Examples and explanations for Question-sensibility (Q-sensibility), Question-answerability (Q-answerability) and QA-sensibility. Pos = Positive example, Neg = Negative example.
F Full Set of Low Resource Experiments
Figure 6: Low resource experiments comparing the Rouge-L score of vanilla T5 Small (left) and T5 Base (right) with our CCQA pre-trained models on NQ-long (top), GooAQ (center) and ELI5 (bottom).
G Qualitative Dataset Examples