SlowGuess/NAACL_2022_OCR huggingface dataset:
Patrick Huber†*, Armen Aghajanyan‡, Barlas Oğuz‡ Dmytro Okhonko‡, Wen-tau Yih‡, Sonal Gupta‡ and Xilun Chen‡
†University of British Columbia; ‡Meta AI
huberpat@cs.ubc.ca, {armenag,barlaso,oxo,scottyih,sonalgupta,xilun}@fb.com
We propose a novel open-domain question-answering dataset based on the Common Crawl project. With a previously unseen number of around 130 million multilingual question-answer pairs (including about 60 million English data-points), we use our large-scale, natural, diverse and high-quality corpus to in-domain pre-train popular language models for the task of question-answering. In our experiments, we find that our Common Crawl Question Answering dataset (CCQA) achieves promising results in zero-shot, low resource and fine-tuned settings across multiple tasks, models and benchmarks<sup>1</sup>.
Open-domain question-answering (ODQA) has evolved into a core problem in Natural Language Processing (NLP), receiving growing interest from the research community (Raffel et al., 2020; Roberts et al., 2020). Despite the notoriously difficult challenge to correctly answer open-domain questions on arbitrary topics, recent advances of pre-trained language models (such as BERT (Devlin et al., 2019), BART (Lewis et al., 2020a) and T5 (Raffel et al., 2020)) have stimulated new research into additional, task-dependent pre-training steps. Specifically, recent publications show that in-domain pre-training regimes can improve models for several downstream tasks (Gururangan et al., 2020). For open-domain question-answering, newly proposed pre-training tasks such as the Inverse Cloze Task (ICT) (Lee et al., 2019), Body First Selection (BFS), Wiki Link Prediction (WLP) (Chang et al., 2020) and Question Answering Infused Pre-training (QUIP) (Jia et al., 2021) show consistent improvements over baselines. However, most of these approaches still rely
on either unlabeled text, or synthetically generated question-answer (QA) pairs. In this paper, we explore a second, somewhat orthogonal dimension to these lines of work, examining if a web-scale collection of natural QA pairs can support ODQA through in-domain pre-training.
Per definition, an ODQA system should be able to answer any question from an arbitrary domain. We believe that to approach this ability with in-domain pre-training, a suitable dataset should address the following 5 challenges: (1) Size; ODQA requires knowledge of a wide variety of topics. The underlying dataset used for in-domain pretraining hence needs to cover this abundance of domains, requiring a web-scale dataset. (2) Naturalness; While synthetic corpora can potentially capture a wide variety of language phenomena, to understand and generate truly natural language in all facets, synthetic datasets are not sufficient. (3) Quality; Given the requirement for a diverse, large-scale dataset, high data quality in terms of cleanliness and sensibility becomes a major challenge. Given that web-scale data sources require highly automated approaches operating on noisy data, assuring data quality is non-trivial. (4) Diversity; Besides size, another challenge for any ODQA in-domain pre-training dataset is the generality of the corpus. The dataset needs to support answering many diverse questions to allow models to learn general concepts. (5) Evaluation Fairness; A web-scale question-answering dataset potentially overlaps with existing benchmark corpora, leading to inflated performance measures and impeding the evaluation fairness (Lewis et al., 2021a).
To overcome these challenges, we propose a new large-scale dataset for open-domain question-answering called the Common Crawl Question Answering (CCQA) dataset. Similar to popular datasets, such as C4 (Raffel et al., 2020), CCNet (Wenzek et al., 2020), CC-100 (Conneau et al., 2020), HTLM (Aghajanyan et al., 2022b),
*Work done at Meta.
<sup>1</sup>Our dataset generation script and CCQA pre-trained checkpoints can be found at https://github.com/facebookresearch/CCQA
Findings of the Association for Computational Linguistics: NAACL 2022, pages 2402 - 2420 July 10-15, 2022 ©2022 Association for Computational Linguistics
and CM3 (Aghajanyan et al., 2022a) we generate a large-scale, diverse and high-quality question-answering dataset from Common Crawl.
More specifically, Common Crawl allows us to obtain a large number of truly natural question-answer pairs, asked and answered by real humans on the web, rather than inferred through computational methods. Using the abundantly available schema.org question annotation $^2$ , we generate question-answer pairs from explicit annotations, instead of heuristic rules, leading to high-quality data points.
In a large set of evaluations, we show that indomain pre-training on our CCQA dataset achieves promising results across different settings, models and benchmarks. Using the rich information available on the web, we augment our dataset with additional data attributes beyond just question-answer pairs, such as votes, multiple (competing) answers, question summaries and intra-textual HTML markup, which can be used for a variety of tasks beyond question-answering in future work. Furthermore, we evaluate the diversity and evaluation fairness of our dataset by computing topic distributions and train-test overlaps with benchmark datasets, providing additional rationale regarding the quality of our data and experiments. To summarize, our main contributions in this paper are as follows:
- We generate the first truly large-scale, natural question-answering dataset, containing around 130 million unfiltered question-answer pairs (55M unique pairs), including about 60 million English data points (24M unique pairs).
- We present key dataset statistics, confirming the high quality of our question-answer pairs, the wide range of diverse topics and a low overlap with existing benchmarks.
- We show the effectiveness of the dataset for in-domain pre-training by evaluating the performance of the unfiltered English subset on two question-answering tasks, three different settings, four models and five diverse benchmarks.
Figure 1: Dataset generation overview from the initial raw HTML file (top) to general purpose, webpage aggregated question-answer pairs (bottom).  $\mathbf{M} =$  Additional question/answer metadata. Red boxes = Nonquestion-answer related webpage components.
This work is inspired by a range of previous approaches using Commoncrawl web-data, such as the Colossal Clean Crawled Corpus (C4) for language model pre-training (Raffel et al., 2020), the word/sentence representation generation corpus CCNet (Wenzek et al., 2020), the CC-100 dataset for translation (Conneau et al., 2020) and the markup-style language modelling HTLM corpus for zero-shot summarization (Aghajanyan et al., 2022b). Despite all previously mentioned applications directly relying on large-scale web data from Commoncrawl, their scope and application vary significantly. Compared to previously proposed datasets based on Commoncrawl, we are the first to extract well-structured question-answer pairs with additional meta-data, making our corpus a valuable resource for ODQA research, and a multitude of related tasks, such as question summarization, answer rating, and answer ranking.
Further web-based datasets outside the Common Crawl domain are the TriviaQA (Joshi et al., 2017) and ELI5 corpora (Fan et al., 2019), extracting small-scale question-answer datasets from Trivia websites and Reddit threads respectively. The large-scale GooAQ dataset (Khashabi et al., 2021) is similarly based on web data, however exploits the Google auto-complete feature and related answer boxes to generate semi-synthetic question-answer pairs. As a large-scale, completely syn
$^{2}$ https://schema.org/Question
thetic dataset, the PAQ corpus (Lewis et al., 2021b) automatically generates a large set of Probably Asked Questions from Wikipedia articles. In contrast to these previously proposed datasets, our CCQA corpus presents a large-scale, natural and diverse question-answering resource in the same order of magnitude as the largest synthetic datasets.
Besides the generation of the CCQA dataset, we evaluate its potential as an in-domain pre-training corpus for open-domain question-answering. Our work is aligned with previous in-domain pretraining approaches, which have shown to improve a variety of downstream tasks (Gururangan et al., 2020). Similar to in-domain pre-training, multiple domain-dependent pre-training tasks have been proposed for open-domain question-answering. For example, Lee et al. (2019) propose the Inverse Cloze Task (ICT), Chang et al. (2020) introduce Body First Selection (BFS) and Wiki Link Prediction (WLP) and Jia et al. (2021) describe a novel Question Answering Infused Pre-training (QUIP) task. Along similar lines, Aghajanyan et al. (2021) propose pre-finetuning, an alternative to in-domain pre-training, using around 50 domain-dependent datasets, showing that their MUPPET approach generalizes well to many tasks. Khashabi et al. (2020) introduce a similar concept for question-answering in their UnifiedQA framework. While we propose a somewhat orthogonal dimension to most of these works, they nevertheless present us with strong intuition regarding the effectiveness of domain-dependent pre-training.
3 The Common crawl Question Answering (CCQA) Dataset
Our Common crawl Question Answering (CCQA) dataset contains around 130 million question-answer pairs (55M unique), extracted from 13 Common crawl snapshots between May 2020 and May  $2021^{3}$ . A high-level overview of the dataset generation process is depicted in Figure 1. Starting from a set of raw HTML webpages, we make use of the schema.org definition to find relevant tags, such as the question, answer, author and votes (for the full set of tags see Figure 2). Using the explicit schema.org annotation (commonly used for search-engine optimization), instead of simple heuristics (e.g. question marks), we optimize the resulting corpus for high-quality data points. Specifically,
Figure 2: JSON data structure following the schema.org annotation. Fasttext language labels (Joulin et al., 2016, 2017) added for language distinction.
due to the added efforts for website creators to define schema.org conforming meta-data, we believe that annotated question-answer pairs are likely to be relevant to the general public, mostly exclude rhetorical and contextual questions, and as a result constitute high quality QA data, despite the noisy nature of webpages.
During the dataset processing steps, we remove all HTML elements that do not contain valid schema.org markers (red in Figure 1) and subsequently clean every question on the webpage to only conserve markup related to the textual content of schema.org tags<sup>4</sup>. We further remove any unrelated markup attributes (e.g., CSS and JavaScript classes), before converting the content into a well-defined JSON object, shown in Figure 2 and further described in section 3.2.
Using the 13 consecutive Common Crawl snapshots, we generate an initial dataset of 130 million question-answer pairs, naturally containing two types of potential duplicates: (1) Same-URL duplicates; where a webpage is updated between any two Common Crawl snapshots and (2) Content duplicates; where webpages from any Common Crawl snapshot contain same questions with potentially similar answers.
Here, we use the original, uncleaned version of the dataset, presenting a practical performance
4Set of textual tags taken from developer.mozilla. org/en-US/docs/Web/HTML/Element
lower-bound, while leaving the exploration of additional filtering steps for future work<sup>5</sup>.
Our dataset generation procedure is further outlined in Algorithm 1, found in Appendix A. For qualitative examples of our generated dataset format, we refer readers to Appendix G.
3.2 Dataset Format
The structured output of the dataset collection (shown in Figure 2), contains a three-level nested structure: (1) Every top-level data point represents a webpage in Common crawl, encapsulating questions and answers found on the page, together with relevant metadata. (2) On the second level, every question is represented as a tuple containing the question name (a short summary of the question) and question text (the main question). Questions also contain additional metadata as shown in Figure 2. (3) Every question can contain an arbitrary number of associated answers and answer attempts, located on the third and final level of the nested structure. An answer thereby contains a mandatory accepted/suggested label, the answer text as well as optional metadata.
With this nested structure of our CCQA dataset, we allow users to verify question-answer pairs and their metadata on the original webpage, utilize additional parts of the web-document and allow future research to tackle question-answering related tasks, such as answer selection, answer rating or answer ranking.
3.3 Dataset Dimensions
To gain better insights into the massive amount of data, we present a mix of automatically obtained dataset dimensions, a small-scale human pilot study, and a set of key dataset distributions.
Regarding the small-scale human pilot study, we analyze a random subset of 400 individual question-answer pairs and evaluate their sensibility and answerability. We define question sensibility as to whether the annotator understands the questions itself, while question answerability refers to whether the question provides enough context for a perfect question-answering system to correctly answer the question. Furthermore, QA-sensibility denotes if the question-answer pair makes sense<sup>6</sup>. We refer
Q-SensH	Q-AnsH	QA-SensH	Markup	Q-Summ
96.5%	86%	82.25%	47.5%	11.7%
No A	Avg #A*	Mean Q	Mean A	Lang Tags
5.9%	1.41	43	57	77.9%
Table 1: Key CCQA dataset dimensions. Q=Question, A=Answer, QA=Question-answer pair, Sens= Sensibility, Ans=Answerability, Lang=Language, Summ=Summarization, Mean=Average number of words,  ${}^{H}$  Human pilot study,*Excluding questions without answers.
interested readers to Table 9 in Appendix E for further explanations on sensibility/answerability.
As shown in Table 1, our CCQA corpus contains nearly exclusively sensible questions, with the vast majority of them also answerable and sensible as a pair. To complement our small-scale human annotation, we further explore key dataset dimension, including the fraction of samples with advanced markup, questions containing both, question name and question text (as defined by the schema.org annotation), the number of questions without gold-answers, average question and answer length and the number of webpages with a valid language label, all indicating that the schema.org annotation highly correlates with carefully curated webpages.
Besides the key corpus-level statistics, we take a closer look at important dataset distributions in Table 2. Specifically, we present the top 5 domains at the top of Table 2, showing the largest number of webpages originating from the stackexchange domain, accounting for about  $8\%$  of data points. Regarding the topical distribution of our dataset, we use the DMOZ/Curlie taxonomy, automatically extracting hierarchical topic information<sup>7</sup>. We randomly sample 1,000 question webpages and show the top 5 topics in the second row of Table 2. A more detailed topic distribution, also considering second-level assignments, can be found in Table 6 in Appendix B. Regarding the question-word distribution in our CCQA dataset, we observe that the majority of  $36\%$  of question words are what questions, followed by how, when, which and where. A full list of all 8 questions words and their relative appearance in our corpus can be found in Table 7 in Appendix C. Lastly, expanding on the number of non-trivial markup tags presented in Table 1, we explore the frequency of HTML markup tags in our dataset in the last row in Table 2. For a list
<sup>7</sup>https://www.curlie.org
We provide de-duplication scripts for same-URL duplicates due to snapshot overlap at https://github.com/facebookresearch/CCQA.
<sup>6</sup>We do not check the answer for factual correctness but merely evaluate if it could be the answer for the given question.
Figure 3: High-level overview of the closed-book CCQA in-domain pre-training step (yellow) as part of the training pipeline for BART and T5. Language model pre-training shown in green. Task-dependent fine-tuning presented in red. Evaluation in blue. (1) Baseline pre-training/fine-tuning pipeline, (2) In-domain pre-training/fine-tuning pipeline, (3) zero-shot baseline setting and (4) zero-shot in-domain pretraining setting.
of the top-25 tags found in our corpus, we point interested readers to Table 8 in Appendix D.
In this section, we showcase the value of our CCQA dataset with experiments on the closed-book question-answering (section 4.1) and passage retrieval for open-book QA (section 4.2) tasks.
4.1 Closed-Book Question-Answering
4.1.1 Task
The closed-book question-answering task challenges systems to answer questions without the use of additional information sources, such as knowledge bases or evidence documents. As a result, models are solely relying on the question text and the information stored inside the model weights during training. Here, we evaluate our new CCQA dataset as an in-domain pre-training corpus for this highly challenging task by converting the JSON representation into plain question-answer pairs, removing markup tags and additional metadata.
4.1.2 Models & Training
Using the question-answer pairs from the CCQA dataset, we in-domain pre-train large language models for question-answering. We start with vanilla BART and T5 transformer models, shown
on the left side (green) in Figure 3. We then further in-domain pre-train the models using a denoising or sequence-to-sequence (seq2seq) setup (yellow box in Figure 3). For the denoising task, we follow the vanilla BART approach (Lewis et al., 2020a), using a concatenation of  $Q: \| < \text{question} > \| A: \| < \text{answer} >$  as the model input. For the seq2seq task, we train the model to predict the gold answer given a question as input. With the additional in-domain pretraining step, a variety of training-flows emerge, shown as numbered circles in Figure 3:
(1) Using a vanilla pre-trained language model to fine-tune on the benchmark dataset.
(2) Using the CCQA dataset for in-domain pretraining and subsequently fine-tune on the benchmark dataset.
(3) Using a pre-trained language model to directly infer answers on the benchmark dataset (zero-shot).
(4) Using the CCQA in-domain pre-trained model to directly infer answers on the benchmark dataset in zero-shot fashion.
4.1.3 Datasets
We evaluate the performance of our CCQA corpus as an in-domain pre-training dataset on 5 common benchmarks, based on 4 publicly available datasets in the closed-book setting:
TriviaQA (TQA) is a short-form, factoid-style question-answering dataset (Joshi et al., 2017). For the closed-book task, we ignore the available contexts and focus exclusively on question-answer pairs. Since the official test-split of the dataset is not publicly available, we use the official validation set as our test split and randomly sample a validation set from the training data, as commonly done in previous work (Roberts et al., 2020).
Natural Questions (NQ) (Kwiatkowski et al., 2019) represents a popular corpus for question-answering research. Despite most recent work focusing on the short-form answers (NQ-Short), the NQ corpus also provides additional long-form answers (NQ-Long) for a large subset of questions. In this work, we use both, short, factoid answers and long-form responses.
ELI5, introduced by Fan et al. (2019), constitutes the first large-scale long-form dataset for open-ended question-answering. We again do not take available evidence documents into account, but focus on the question-answer pairs only.
GooAQ (Khashabi et al., 2021) contains semiautomatically extracted question-answer pairs from the Google question auto-complete feature.
Metric	Top 5 Appearances in CCQA
Domains	stackexchange (07.78%)	hotels (03.46%)	viamichelin (02.51%)	ccm (01.86%)	vrbo (01.74%)
Topics	Regional (38.90%)	Society (21.10%)	Business (08.30%)	Sports (07.00%)	Rec (06.20%)
Q-words	What (36.20%)	How (29.80%)	When (09.68%)	Which (09.64%)	Where (06.04%)
Markup	p (28.48%)	a (14.89%)	br (14.86%)	li (10.04%)	span (05.77%)
Table 2: CCQA dataset distribution for top 5 domains, topics according to the DMOZ/Curlie annotation, question words (Q-words, only computed on the English subset) and most common markup tags. % for q-words and markup tags presents portion of all q-word/markup appearances. ccm=commentcamarche, Rec=Recreational.
4.1.4 Metrics
For datasets with short-form answers, we use the Exact Match (EM) metric for fine-tuned systems, in line with previous work by Roberts et al. (2020) and Lewis et al. (2021b). While the EM metric works well for systems that are aware of the task-specific format, it punishes potentially correct answers with additional context, which we believe is overly harsh in zero-shot settings, where the specific output format is not known (e.g., training-flows (3) and (4)). Therefore, we propose using the Answer-level Recall (AR) metric for our zero-shot experiments, while limiting the answer length with the max-length and length-penalty inference parameters. As such, the AR metric requires the correct answer to be a continuous sub-sequence of the predicted tokens, while allowing for additional context. Since AR operates on token-level, the prediction of super/sub-words, e.g., fundamental instead of fun, is considered incorrect.
For long-form question-answer datasets, we choose the Rouge-L (RL) score as our evaluation metric, which has shown strong correlation with Rouge-1 and Rouge-2 scores, and is commonly used in previous work (Khashabi et al., 2021).
4.1.5 Hyper-Parameters
We use the default parameters of the BART (Lewis et al., 2020a) and T5 (Raffel et al., 2020) models for in-domain pre-training and fine-tuning whenever possible. Regarding the in-domain pre-training on our CCQA dataset, we limit training to  $800k$  steps using a batch-size of 1,024. During our fine-tuning runs, we limit the number of updates to  $20k$  steps with a batch-size of 256 samples, with exception of the GooAQ dataset, which we fine-tune for  $100k$  steps due to it's large size. We select the best model during our in-domain pre-training runs based on the perplexity measure, and pick the top fine-tuned model according to the final evaluation metric. We do not perform any hyper-parameter search during
in-domain pre-training and fine-tuning.
For the inference step, our hyper-parameter setting is closely related to commonly used summarization parameters. We use a beam-size of 4, max-length of 140, and length-penalty of 2.0. For the fine-tuned short-form task, we choose a max-length of 30, following Xiong et al. (2021) and a length-penalty of 1.0. All model evaluations are based on Huggingface Transformers<sup>8</sup> (Wolf et al., 2019).
4.1.6 Results
Our main results for the closed-book question-answering task are presented in Table 3, showing the zero-shot and fine-tuned performance of the BART Large (top), T5 Small (center) and T5 Base (bottom) models for each of the 5 evaluation datasets. Even though we present a wide variety of benchmark results, from short-form factoid questions to long-form answers, the CCQA seq2seq pretrained model consistently outperforms all other models on the zero-shot question-answering task. Even more importantly, the additional in-domain pre-training step achieves better zero-shot performance than fully fine-tuned, randomly initialized transformer models (as extensively used prior to 2018) in almost all settings. Specifically, our model outperforms the randomly initialized transformers on all benchmarks for T5 Small and T5 Base, as well as on 4 out of 5 datasets using BART Large.