SlowGuess/NAACL_2021_OCR huggingface dataset:
Pratyay Banerjee Tejas Gokhale Chitta Baral
pbanerj6, tgokhale, chitta@asu.edu
Recent work on unsupervised question answering has shown that models can be trained with procedurally generated question-answer pairs and can achieve performance competitive with supervised methods. In this work, we consider the task of unsupervised reading comprehension and present a method that performs "test-time learning" (TTL) on a given context (text passage), without requiring training on large-scale human-authored datasets containing context-question-answer triplets. This method operates directly on a single test context, uses self-supervision to train models on synthetically generated question-answer pairs, and then infers answers to unseen human-authored questions for this context. Our method achieves accuracies competitive with fully supervised methods and significantly outperforms current unsupervised methods. TTL methods with a smaller model are also competitive with the current state-of-the-art in unsupervised reading comprehension.
Reading comprehension is the task in which systems attempt to answer questions about a passage of text. Answers are typically found in the passage as text-spans or can be inferred through various forms of reasoning (Rajpurkar et al., 2016). The answer to the following question:
"Who is the President of the United States?"
depends on the timeframe and context of the passage provided, and will be different for news articles written in 2001 vs. 2021. If the context is the script of the TV series "The West Wing", the answer is "Jed Bartlet", and even in this fictional setting, it will later change to "Matt Santos".
Knowledge sources such as Wikipedia get updated when new events occur (such as the outcome of elections), or new facts about the world are revealed (such as scientific discoveries), with contributors adding new information and removing
information that is no longer valid (Almeida et al., 2007). With such context-dependent answers and continual changes in knowledge, it is hard to justify training models over fixed corpora for tasks such as question answering (QA). We would like models to answer questions based on the given context and not to learn biases from datasets or historical news articles.
Moreover, supervised learning has been shown to perform poorly in QA tasks with adversarial examples (Jia and Liang, 2017), domain shift (Jia and Liang, 2017; Yogatama et al., 2019; Kamath et al., 2020), and biased or imbalanced data (Agrawal et al., 2018; McCoy et al., 2019). For example, QA systems trained on Wikipedia fail to generalize to newer domains such as Natural Questions (Rennie et al., 2020) or biomedical data (Wiese et al., 2017), and suffer a significant drop in accuracy. Even small semantics-preserving changes to input sentences, such as the substitution of words by synonyms, have been shown to degrade performance in NLP tasks (Alzantot et al., 2018; Jia et al., 2019). Continual changes in text corpora are inevitable, thus calling for the development of robust methods that can reliably perform inference without being subject to biases.
Supervised Question Answering faces challenges such as the need for large-scale (usually human-authored) training corpora to train models. Such corpora typically require significant postprocessing and filtering to remove annotation artifacts (Sakaguchi et al., 2020). To address these challenges, some recent methods (Lewis et al., 2019; Li et al., 2020) approach question answering as an unsupervised learning task. A significant advantage of this approach is that it can be extended to domains and languages for which collecting a large-sized human-authored training corpus is challenging. Methods for unsupervised QA procedurally generate a large corpus of (context, question, answer) triples, and train large neural language
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1200-1211 June 6-11, 2021. ©2021 Association for Computational Linguistics
models, such as BERT (Devlin et al., 2019).
In this work, we focus on unsupervised reading comprehension (RC) under evolving contexts and present the "Test-Time Learning" paradigm for this task. RC – the task of answering questions about a passage of text, acts as the perfect setting for robust question-answering systems that do not overfit to training data. While large-scale language models trained on large datasets may contain global information, the answer needs to be extracted from the given context. Thus, our work seeks to learn unsupervised reading comprehension without access to human-authored training data but instead operates independently on each test context. This makes our method 'distribution-blind' where each new context is assumed to be a novel distribution. The test-time learning (TTL) framework enables smaller models to achieve improved performance with small procedurally generated question-answer pairs, and is summarized below:
- a single context (text passage)  $c_{i}$  is given, from which we procedurally generate QA pairs;
- these QA pairs are used to train models to answer questions about  $c_{i}$ ;
- the inference is performed on previously unseen questions for  $c_{i}$ .
This framework has a simple assumption that every context comes from a distinct distribution. Hence, parameters learned for the previous context might not be useful to generalize to other contexts. This assumption holds where the contexts evolve over time, and rote memorization of answers might lead to wrong predictions. As such, the above process is repeated for each new context  $c_{i}$ .
For question-answer generation, we use simple methods such as cloze-translation (Lewis et al., 2019), template-based question-answer generation (Fabbri et al., 2020) and question-answer semantic role labeling (QA-SRL) (He et al., 2015). We use two neural transformer-based language models, BERT-Large (Devlin et al., 2019) and DistilBert (Sanh et al., 2019), to study the efficacy of our framework with large and small transformer models. We evaluate our method on two reading comprehension datasets, SQuAD (Rajpurkar et al., 2016) and NewsQA (Trischler et al., 2017). We investigate test-time training under multiple learning settings: (1) single-context learning – the "standard" setting, (2)  $K$ -neighbor learning – by retrieving top- $K$  multiple related contexts for each test context, (3) curriculum learning – progressively learning on
question-types of increasing order of complexity, (4) online learning - sequentially finetuning models on each incoming test sample.
Our experimental findings are summarized below:
- Test-time learning methods are effective for the task of reading comprehension and surpass current state-of-the-art on two benchmarks: SQuAD and NewsQA.
- Online TTL trained over K-neighboring contexts of the test context is the best version with EM/F1 gains of  $7.3\% / 7.8\%$  on SQuAD 1.1 and  $5.3\% / 6.9\%$  on NewsQA.
- DistilBERT - which has less than  $\frac{1}{5}^{th}$  of the number of model parameters of BERT-Large is competitive with current SOTA methods that use BERT-Large.
2 Test-Time Reading Comprehension
Consider a reading comprehension test dataset  $\mathcal{D}^{test} = \{(c_i,q_i,a_i)\}_{i = 1}^n$  with context text passages  $c_{i}$  , human-authored questions  $q_{i}$  and true answers  $a_{i}$  . The QA model  $g(\cdot)$  is parameterized by  $\theta =$ $(\theta_{f},\theta_{h})$  where  $\theta_{f}$  are parameters for the feature extractor, and  $\theta_{h}$  for the answering head. The answer is predicted as a text-span, given by the start and stop positions  $[y_{start},y_{stop}]$  .Contemporary unsupervised RC models (Lewis, 2019; Li et al., 2020) are trained on a large dataset  $\hat{\mathcal{D}}^{train} = \{(c_i,\hat{q}_i,\hat{a}_i)\}_{i = 1}^n$  , where the QA pairs are synthetically generated from the context.
In our setting, we do not use such large training datasets, but instead directly operate on individual test contexts  $c_{i}\in \mathcal{D}^{test}$ . Given  $c_{i}$ , M synthetic question-answer pairs  $\{(\hat{q}_i^j,\hat{a}_i^j)\}_{j = 1}^M$  are procedurally generated as described in Section 3. The QA model parameters  $\theta$  are trained over the synthetic data to predict the span of the answer  $[\hat{y}_{start},\hat{y}_{stop}]$  by optimizing the loss  $\ell_{ans}$ :
$$
\underset {\theta} {\text {m i n i m i z e}} \sum_ {j = 1} ^ {M} \ell_ {\text {a n s}} \left(c _ {i} ^ {j}, \hat {q} _ {i} ^ {j}, \theta\right) \tag {1}
$$
$$
\ell_ {a n s} = \ell_ {C E} (\hat {y} _ {\text {s t a r t}}, \hat {a} _ {\text {s t a r t}}) + \ell_ {C E} (\hat {y} _ {\text {s t o p}}, \hat {a} _ {\text {s t o p}}) \tag {2}
$$
where  $\ell_{CE}$  is cross-entropy loss. The inference is performed on human-authored questions to predict the answer spans:
$$
[ y _ {\text {s t a r t}}, y _ {\text {s t o p}} ] = g (c, q). \tag {3}
$$
Next, we describe the variants of test-time reading comprehension.
Figure 1: Overview of our self-supervised test-time learning framework for reading comprehension. Our method does not require a human-authored training dataset but operates directly on each single test context and synthetically generates question-answer pairs over which model parameters  $\theta$  are optimized. The inference is performed with trained parameters  $\theta^{*}$  on unseen human authored questions.
Single-Context Test-Time RC. This is the standard formulation of test-time learning in this paper, with Equation 1 optimizing over  $\theta$ , i.e. for each context  $c_{i}$ , the feature extractor  $\theta_{f}$  is re-initialized with pre-trained BERT, and the answering head  $\theta_{h}$  is randomly initialized.
$K$ -neighbor Test-Time RC. In this version,  $K$  contexts similar to the test-context  $c_{i}$  are grouped together, and Equation 1 is optimized over each set of similar contexts as opposed to single contexts in the standard setting. We index contexts in a Lucene-based information retrieval system (Gormley and Tong, 2015) and retrieve top-K similar contexts given  $c_{i}$ , which we call Context Expansion with IR described in Section 3.
Curriculum Test-Time RC. In the curriculum learning version, questions are ordered in increasing order of complexity. We generate different types of questions, such as, semantic role labelling, cloze-completion, template-based and dependency tree-based translation of cloze questions to natural questions. This provides an ordering of complexity and we study the effect of test-time training with such an increasing complexity.
Online Test-Time RC. In the online test-time learning (TTLO), test samples are considered to be encountered in sequence. As such, answering head parameters  $\theta_h$  are updated sequentially without being randomly re-initialized like in the standard single-context setting. For each new test context  $c_i$ ,  $\theta_h$  is initialized with the optimal pa
rameteres from the previous test context  $c_{i-1}$  to optimize Equation 1.
3 Self-Supervised QA Generation
In this section, we detail our framework for procedurally generating QA pairs from a given context. We use named-entity recognition from Spacy (Honnibal and Montani, 2017), dependency parsing from Berkeley Neural Parser (Stern et al., 2017) and semantic role labeling (He et al., 2015) as our core methods to extract plausible answers and generate natural questions. As described in our task formulation, we create a set of  $M$  question-answer pairs  $\{(\hat{q}_i^j,\hat{a}_i^j)\}_{j = 1}^{M}$  for the given context  $c_{i}$ .
Cloze Generation. Statements in which the answer is replaced with a mask or blank token are called cloze questions. We follow the steps provided in Lewis et al. (2019) in which answers are replaced with a special token depending on the answer category. For example, in a sentence,
"They were descended from Norse raiders and pirates from Denmark"
the answer Denmark is replaced by [LOCATION], resulting a cloze question:
"They were descended from Norse raiders and pirates from [LOCATION]".
Cloze Translation is utilized to rephrase cloze questions into more natural questions by using rule-based methods from Lewis et al. (2019).
Template-based Question Generation utilizes simple template-based rules to generate questions. Given a context of format:
[FRAGMENT A][ANSWER][FRAGMENT B]
a template of the format "Wh+B+A+?" replaces the answer with a Wh-word (e.g., who, what, where) as described in Fabbri et al. (2020).
Dependency Parsing-based Question Generation. In this method, we use dependency reconstruction to translate clozes to natural questions as described in Li et al. (2020), according to the following steps:
1. Right child nodes of the answer are retained and left children are pruned.
2. For each node of the parse tree, if the child node's subtree contains the answer, the child node is moved to the first child node.
3. An in-order traversal is performed on the reconstructed tree. A rule-based mapping is applied to replace the special mask token of the cloze with an appropriate "Wh-word".
QA-Semantic Role Labeling (QA-SRL) was proposed by He et al. (2015) as a method to annotate NLP data, by using QA pairs to specify textual arguments and their roles. As seen in Figure 1, for the context sentences:
"They were descended from Norse raiders and pirates from Denmark."
"The distinct cultural and ethnic identity of the Normans emerged initially in the first half of the 10th century and it continued to evolve."
the following QA pairs were generated,
("What was someone descended from?","Norse"),
(What evolved?, distinct cultural and ethnic diversity)
We can observe the questions are short and use generic descriptors and pronouns such as "something" and "someone" instead of specific references calling for the model to have greater semantic understanding of the given context.
Context Expansion using IR is used in the  $K$ -neighbor version of TTL. For Context Expansion, we index all paragraphs present in a Wikipedia dump in ElasticSearch. During test-time learning, we preprocess the context  $c_{i}$  by removing the most frequent stop-words, and use it as a seed query to search and retrieve top-K similar contexts. This provides us with related paragraphs that describe similar topics, and consequently more diverse and slightly larger number of QA pairs to train compared to only  $c_{i}$ . We then generate QA pairs using
the above described methods. We study the effect of varying the number of most similar contexts  $(K)$  on the downstream QA performance.
Datasets. We evaluate our learning framework on two well-known reading comprehension datasets: SQuAD 1.1 (Rajpurkar et al., 2016) and NewsQA (Trischler et al., 2017).
QA Model. We focus on training two transformer-encoder based models, BERT-Large (Devlin et al., 2019) trained with whole-word masking and DistilBERT (Sanh et al., 2019). BERT-Large is used by current state-of-the-art methods on unsupervised extractive QA tasks and has 345 million trainable parameters. On the other hand, DistilBERT is a knowledge-distilled transformer-encoder based model and only has 66 million parameters ( $\sim$ $5 \times$  smaller than BERT-Large), allowing us to study the efficacy of TTL with respect to model-size.
Metrics. We use the standard metrics for extractive QA – macro Exact Match, where the predicted answer span is directly matched with the ground-truth, and macro  $F1$ , which measures the overlap between the predicted and the ground-truth spans. For comparisons with existing unsupervised methods, since TTL operates directly on test instances, we report validation set performance only for SQuAD 1.1, as the test set is hidden.
Training Setup. For all test-time learning variants, we limit the maximum number of questions generated per context to 4000 and the maximum number of training steps to 1500. The number of training steps is linearly dependent on the selected batch size  $\in$  [16, 64]. For our  $K$ -neighbor TTL setup that uses Context Expansion, we limit the number of retrieved contexts to 500. In Curriculum Test-Time RC, we ensure that all variants have an equal number (1000) of generated QA-pairs per-context. We evaluate multiple learning rates within the range 1e-5 to 5e-5. We use the Adam (Kingma and Ba, 2014) optimizer and truncate the paragraphs to a maximum sequence length of 384. The number 384 was chosen by evaluating the  $99^{th}$  percentile of the combined length of question and the contexts, to reduce training overhead and GPU memory size. Long documents are split into multiple windows with a stride of 128. All
Models	SQuAD 1.1	NewsQA
Dev	Test	Dev	Test
DCR (2016)	62.5 / 71.2	62.5 / 71.0	- / -	- / -
mLSTM (2016)	64.1 / 73.9	64.7 / 73.7	34.4 / 49.6*	34.9 / 50.0*
FastQAExt (2017)	70.3 / 78.5	70.8 / 78.9	43.7 / 56.1	42.8 / 56.1
R-NET (2017)	71.1 / 79.5	71.3 / 79.7	- / -	- / -
BERT-Large (2019)	84.2 / 91.1	85.1 / 91.8	- / -	- / -
SpanBERT (2020)	- / -	88.8 / 94.6	- / -	- / 73.6
DistilBERT (2019)	77.7 / 85.8	- / -	57.2 / 64.8	56.1 / 63.5
experiments were conducted on two Nvidia RTX-8000 GPUs. We use ten percent of the training data to perform three hyper-parameter trials for each variant. We train models with three random seeds, and report the mean F1 and EM scores.
Baselines. As we generate our own data using QA-SRL, we use the following strong baselines. First, we train BERT-Large with generated data from previous methods described in Section 3 and our method (which contains additional QA-SRL samples). Second, we replicate the baselines using the low parameter-count model DistilBERT (66 million vs 345 million for BERT-Large). Third, for a fair comparison to Single-Context and  $K$ -neighbor test-time learning where we train models for each context independently, we propose a baseline where we train on all the test contexts together, referred to as "All test contexts". We also evaluate all TTL variants on two initializations of feature-extractor parameters -
1. "default" initialization of BERT-Large, i.e.  $\theta_{f}$  pre-trained on masked language modeling and next-sentence prediction tasks, and  $\theta_{h}$  randomly initialized for each context and trained from scratch, or
2.  $\theta_{f}$  and  $\theta_{h}$  further pre-trained on  $100K$  synthetic QA pairs generated procedurally using our methods described in Section 3 with contexts taken from the Wikipedia corpus.
5.1 Unsupervised Question Answering
We compare our results with current state-of-the-art supervised methods (Table 1) and unsupervised methods (Table 2) on SQuAD 1.1 and NewsQA. The previous best unsupervised method with both BERT-Large and DistilBERT is Li et al. (2020). Our best TTL method is the Online version (TTLO), with a pre-training phase and a randomly-shuffled ordering of QA pairs with an average of 3000 QA pairs per context, trained with only
Table 1: Results (EM / F1) from supervised methods on SQuAD 1.1 and NewsQA.
Models	SQuAD 1.1	NewsQA
Dev	Test	Dev	Test
BERT-Large
+ Dhingra et al.†	28.4 / 35.8	- / -	18.6 / 27.6	18.6 / 27.2
+ Lewis et al.‡	45.4 / 55.6	44.2 / 54.7	19.6 / 28.5	17.9 / 27.0
+ Li et al.	62.5 / 72.6	61.1 / 71.4	33.6 / 46.3	32.1 / 45.1
+ Fabbri et al.	46.1 / 56.8	- / -	21.2 / 29.4	- / -
+ our data	49.4 / 59.1	- / -	28.2 / 37.6	27.3 / 36.4
DistilBERT
+ Lewis et al. data	23.4 / 29.5	- / -	14.1 / 21.6	14.7 / 20.6
+ Li et al. data	42.6 / 48.3	- / -	25.4 / 36.2	27.1 / 35.4
+ Fabbri et al. data	37.5 / 45.6	- / -	16.3 / 22.3	16.1 / 22.9
+ our data	38.9 / 46.8	- / -	23.2 / 31.9	22.4 / 31.1
BERT-Large TTL*	69.8 / 80.4	- / -	38.9 / 53.2	38.2 / 52.6
DistilBERT TTL*	58.1 / 68.9	- / -	32.6 / 46.4	30.5 / 45.2
Table 2: Comparison with previous unsupervised methods on SQuAD 1.1 and NewsQA.  $\star$ We show the best TTL model here, and results from all TTL variants in Table 3. Metrics are EM / F1. Previous SOTA for both models are shaded in gray. *results from Trischler et al. (2017);  $\dagger$ Lewis et al. (2019);  $\ddagger$ Li et al. (2020).
TTL Models	Default init. θf	Pre-trained init. θf
SQuAD 1.1	NewsQA	SQuAD 1.1	NewsQA
BERT-Large
Single-Context	54.9	34.9	59.8	37.5
Single-Context Online	56.1	36.3	61.8	39.1
K-neighbor	66.2	41.6	78.3	50.7
K-neighbor Online	68.7	46.3	80.4	53.2
Curriculum	68.3	46.7	79.7	52.8
All test contexts	64.7	39.8	68.2	43.5
DistilBERT
Single-Context	37.2	23.2	49.4	34.6
Single-Context Online	38.5	25.3	55.6	39.8
K-neighbor	42.4	27.8	64.3	43.5
K-neighbor Online	49.7	29.1	68.9	46.4
Curriculum	49.3	28.7	68.7	45.8
All test contexts	42.4	28.2	47.4	38.7
Table 3: Comparison of Dev-set F1 scores for TTL variants, when  $\theta_{f}$  are trained from default initialization for each test instance, or pre-trained on our generated data. Scores surpassing previous best, are shaded in cyan for SQuAD and red for NewsQA.
100 steps. With this setup, we are able to improve the state-of-the-art for the SQuAD benchmark with BERT-Large by  $7.8\%$  exact-match accuracy and  $7.3\%$  F1 score. With DistilBERT, the best TTL method shows an improvement of  $15.5\%$  EM and  $20.6\%$  F1 over DistilBERT-based baseline, as shown in Table 2. In NewsQA, TTL improves BERT-Large performance by  $5.3\%$  EM and  $6.9\%$  F1 score, and with DistilBERT shows an improvement of  $7.2\%$  EM and  $7.2\%$  F1 score.
Training BERT-Large and DistilBERT with "our data" i.e. with a combined synthetic corpus created via all four QA-pair generation methods, marginally improves the F1 score. This shows that our QA generation methods lead to an improvement over existing unsupervised QA generation methods as shown in Table 2. However, the TTL framework leads to even larger gains ( $\sim 20\%$ )
Figure 2: Comparison of F1 scores of TTL models when trained with an increasing number of labeled training samples on SQuAD. TTLO-Online TTL.
for SQuAD and  $\sim 10\%$  for NewsQA), indicating the benefits of test-time learning. This result also points to the limits of training with a large number of contexts compared to training on individual contexts. This limitation is especially profound in lower parameter models, such as DistilBERT. In Reading Comprehension, since the answer comes from the context, "understanding" the context is much more relevant. It has a higher inductive bias than learning to comprehend a significantly large number of contexts during training.
For instance, there are multiple contexts about Normans in the SQuAD dataset, one of which is shown in Figure 1. But each context may have different historical persons referred to as the leaders or rulers of the Normans. Answers to questions such as "Who was the leader of the Normans" are better learned for each context separately than from all contexts. Pre-training on several contexts is indeed beneficial to obtain better parameter initializations, as observed in Table 2, which can be further independently finetuned for each context during TTL.
5.2 Few-Shot Question Answering
We evaluate our best method under the few-shot setting, i.e. when models are trained with a limited number of human-authored QA pairs from the training datasets. Figure 2 shows a comparison with an increasing number of labeled training samples for SQuAD. TTL-Online is consistently better than existing methods and achieves  $81.6\%$  F1 score with just 100 labeled samples. This indicates that this learning framework can reduce the number of in-domain human-authored samples required for training. TTL-Online is also consistently better than (Li et al., 2020) which the previous best unsupervised method for SQuAD. All methods (which use BERT-Large as backbone) converge to similar
Curriculum Order (Left to Right)	Default init. θf	Pre-trained θf
SQuAD	NewsQA	SQuAD	NewsQA
BERT-Large
Random Shuffled	68.7	46.3	80.4	53.2
QA-SRL > T > DP	68.3	46.7	79.7	52.8
T > QA-SRL > DP	67.6	45.4	77.6	50.0
T > DP > QA-SRL	65.8	44.3	75.3	47.2
DistilBERT
Random Shuffled	49.7	29.1	68.9	46.4
QA-SRL > T > DP	49.3	28.7	68.7	45.8
T > QA-SRL > DP	48.8	28.1	67.2	43.9
T > DP > QA-SRL	47.1	26.5	65.3	39.2
Table 4: Dev-set F1 scores for  $K$ -neighbor Online test-time learning, for different Curriculum Learning orderings of QA-SRL (He et al., 2015), T (template-based methods), DP (dependency parsing).
Figure 3: Comparison of F1 scores of TTL models when trained with an increasing number of contexts, on both SQuAD and NewsQA.