SlowGuess/NAACL_2022_OCR huggingface dataset:
Hongyin Luo $^{1}$  Shang-Wen Li $^{2}$  Mingye Gao $^{3}$  Seunghak Yu $^{2*}$  James Glass $^{1}$ $^{1}$ MIT CSAIL,  $^{2}$ Amazon AI  $^{3}$ MIT MTL
{hyluo,mingye,glass}@mit.edu,{shangwel,yuseungh}@amazon.com
Pretrained language models have significantly improved the performance of downstream language understanding tasks, including extractive question answering, by providing high-quality contextualized word embeddings. However, training question answering models still requires large amounts of annotated data for specific domains. In this work, we propose a cooperative self-training framework, RGX, for automatically generating more non-trivial question-answer pairs to improve model performance. RGX is built upon a masked answer extraction task with an interactive learning environment containing an answer entity Recognizer, a question Generator, and an answer eXtractor. Given a passage with a masked entity, the generator generates a question around the entity, and the extractor is trained to extract the masked entity with the generated question and raw texts. The framework allows the training of question generation and answering models on any text corpora without annotation. We further leverage a self-training technique to improve the performance of both question generation and answer extraction models. Experiment results show that RGX outperforms the state-of-the-art (SOTA) pretrained language models and transfer learning approaches on standard question-answering benchmarks, and yields the new SOTA performance under given model size and transfer learning settings.
Recent studies have shown that language model pretraining provides high-quality text representations and significantly improves neural networks' performance on a variety of natural language processing (NLP) tasks (Peters et al., 2018). Based on the popular Transformer architecture (Vaswani et al., 2017), various language models have been proposed (Devlin et al., 2018; Liu et al., 2019; Clark
Tang Dynasty ... Chengdu became nationally known as a supplier of armies and the home of Du Fu, who is sometimes called China's greatest poet.
AER Agent
a supplier of armies and the home of Du Fu
What was Sichuan known for in the ancient world before 957?
QG Agent
QAE Agent
A supplier of armies
Figure 1: The pipeline of semi-supervised question answering (machine reading comprehension) by RGX. AER (answer entity Recognition) agent recognizes answer entity from a given passage; QG (question Generation) generates a question based on the passage and entity; QAE (question-answering eXtractor) extracts answer from the question and passage.
et al., 2020). These models are pretrained to predict a masked word in a given context from large corpora, and generate a contextual representation that encodes semantic and syntactic information. After finetuning, these representations significantly improve performance on downstream NLP tasks. Although masked language modeling is a powerful self-supervised learning technique, annotation on large-scaled data is still necessary for finetuning on difficult downstream tasks, including extractive question answering (QA) $^{1}$  where a large number of labeled question-answer pairs are required as a training corpora.
Previous studies showed that the QA models can be improved by training on synthetic question-answer pairs, namely self-training (Sachan and Xing, 2018; Puri et al., 2020; Shakeri et al., 2020; Bartolo et al., 2021). The core step of these work is pretraining a question-answer pair synthesis model on a seed corpus, and apply the generator on target
* Work is not related to the employment of the second and fourth authors at Meta and Amazon
<sup>1</sup>Also referred to as machine reading comprehension. The two terms are used interchangeably in this paper.
Human Language Technologies, pages 244 - 257
domains to obtain synthetic training data. The QA model learns domain knowledge after finetuning on the synthetic data, and thus the domain adaptation is improved. However, the gap between the pretraining (i.e., seed) and the target corpus still exists, in terms of domain knowledge, question difficulty, and language style. The gap affects the quality of the synthetic training data.
We thus propose a framework that allows cooperative self-training for both QA pair synthesis and question answering to better adapt the synthesis models to the target domain and improve the learning of the QA models. In the framework, we construct a cooperative environment where a question generator and an answer extractor work together to solve a masked entity prediction problem. We first leverage an entity recognizer to mask out an entity in a provided passage. The question generator then outputs a question based on the masked passage. With the generated question and the original, unmasked passage, we train the answer extractor to select the correct answer spans, which are the masked entity. The extractor is also the final model used for extractive QA. To extract the spans accurately, the generator has to provide a good question, and the extractor should select the most likely tokens. We apply an expectation-maximization algorithm to select high-quality QA pairs and update both question generation and answer extraction models to improve the quality of synthetic data and the accuracy of the self-trained QA model based on synthetic QA pairs. We call our algorithm RGX since it incorporates an answer entity Recognizer, a question Generator, and a question-answering eXtractor. The RGX pipeline is illustrated in Figure 1.
With RGX, we can train a QA model for any unlabeled target domain given the corresponding text corpora and a labeled QA corpus in a seed domain (either the same or different from the target). By training QA models on synthetic QA data generated by RGX and evaluating the trained model on human-labeled evaluation data, we show that RGX outperforms SOTA approaches in QA benchmark datasets when domain specific human labels are not available during finetuning. In this work, we make the following contributions:
1. We propose a cooperative self-training framework, RGX, which contains an answer entity recognition, question generation, and answer span extraction to automatically generate non-
trivial QA pairs on unlabeled corpora.
2. We design a expectation-maximization (EM) synthetic QA selection that identifies difficult but answerable questions without supervision to incrementally train the QA model with challenging examples, and an answer entity recognition (AER) based maximum mutual information (MMI) inference method for question answering.
3. Experiments show that our method significantly outperforms SOTA pretrained QA models and self-training QA baselines.
Reinforcement learning and self-training have emerged recently for learning language generation in addition to maximum likelihood training. To optimize text generation models directly with non-differentiable objective functions, Rennie et al. (2017) proposed self-critical sequence training (SCST) using a policy gradient (Kakade, 2001; Silver et al., 2014). On the other hand, self-training has been shown to be effective in many tasks, such as machine translation (He et al., 2019), image classification (Xie et al., 2020), and structured database-grounded question answering (Xu et al., 2020).
In the domain of question answering, a question generator can be used for joint answer prediction (Tang et al., 2017; Duan et al., 2017), and synthetic QA data are used for in-domain data augmentation (Sachan and Xing, 2018; Puri et al., 2020; Liu et al., 2020; Klein and Nabi, 2019) and out-of-domain adaptation. Lewis et al. (2019b) and Lee et al. (2020) introduced models for question answering under unsupervised/zero-shot settings. Shakeri et al. (2020) proposed generating synthetic question-answer pairs with an end-to-end model simultaneously. Bartolo et al. (2021) improved the question synthesis by training with difficult QA cases from the AdversarialQA corpus (Bartolo et al., 2020) and fine-grained answer synthesis by multi-model voting. We include more related studies in Appendix A.
In this work, we mainly compare our method with latest baselines, Shakeri et al. (2020) and Bartolo et al. (2021) that reported results on out-of-domain adaptation. Besides improved QA performance, our framework, RGX, differs from the previous work in the following aspects: (1) Our method features reinforced finetuning of the QA
Figure 2: The cooperative learning pipeline for question answering. The pipeline starts from a passage and follows the steps: (1) recognizing a potential answer entity, (2) generating a question asking about the answer entity, and (3) answering the question by extracting the answer span in the passage.
Synthesizer, (2) Our framework supports and improves maximize mutual information inference in test time, and (3) Our work did not use complicated data annotation, e.g. AdversarialQA.
3 RGX Framework
In this section, we first introduce (1) the QA synthesis pipeline, (2) cooperative self-training for both QA synthesis and question answering, and (3) an improved maximum mutual information inference strategy. The self-training pipeline of RGX is shown in Figure 2.
3.1 Data Synthesis
Given a passage  $p$ , our goal is generating a set of questions  $q$  and answers  $a$  for the self-training of the QA model. The RGX model first recognize potential answer entities (AE) in  $p$  with an answer entity recognition (AER) model, and then generate question based on the recognized AEs with a question generation (QG) model, and fine-grain the AEs with a pretrained question-answering extraction (QAE) model.
3.1.1 Answer Entity Recognition (AER)
Latest QA synthesis models, QAGen2S (Shakeri et al., 2020) and SynQA (Bartolo et al., 2021), directly generate questions from passages by modeling  $P_{qq}(q|p)$ . In RGX, we first recognize all potential answer entities in a passage before generating questions for (1) increasing question diversity and coverage, and (2) modeling the mutual information between question generation and answering models in test time. The AER model in trained on the seed QA corpus.
We found that using an off-the-shelf named entity recognition (NER) model pretrained on the CONLL 2003 shared task (Bender et al., 2003) performs poorly as a AER model (shown in our experiments). To learn an effective recognizer, given a passage  $p$  and an annotated answer entity  $e$ , we select the sentence  $s$  containing  $e$  from  $p$  and train language models to recognize  $e$  in  $s$ . We tried two models for this task: a BIO sequence tagging model (AER-Tag) and a extractive AER model, which is similar to an extractive question answering model, for easier decoding. The model predicts the start and end positions of the answer entity  $e$ . With this method, we get potential answer entities by probabilities of all candidate spans.
3.1.2 Masked Question Generation
With AER, we replace the answer entity  $e$  in the passage  $p$  with a [MASK] token and obtain the masked passage  $p^*$ . We then build a question generator  $Q$  (denoted as QG interchangeably) that outputs answerable questions  $q$  in natural language with the concatenation of  $p^*$  and  $e$  as input, i.e.,  $q = Q([p^*, e])$ . We adopt the BART sequence-to-sequence model (Lewis et al., 2019a) as the architecture of  $Q$  in our implementation, and we train  $Q$  on the question-answer pairs in the seed corpus by maximizing the likelihood of annotated questions.
3.1.3 Answer Extraction as Fine-grained AER
The answer extraction model  $A$  (denoted as QAE, question-answering extractor) takes generated question  $q$  and the original passage  $p$  as inputs. Following the standard extractive QA method, we predict the answers by
$$
I _ {s t}, I _ {e d} = A ([ q, p ]) \tag {1}
$$
where  $I_{st}$  and  $I_{ed}$  stand for the start and end positions of  $e$  in  $p$ , respectively. We train the QAE model to predict  $I_{st}$  and  $I_{ed}$  separately with cross entropy losses.
Besides being trained with synthetic QA pairs and evaluated for the final QA performance, the QAE model is also a part of the data synthesis pipeline. After generating questions with the QG model, we use a pretrained QAE model to answer the generated questions. The QAE model recognizes better answers spans than the AER model since it takes questions as additional inputs. As a result, the final synthetic dataset is constructed by selecting generated questions and their corresponding QAE outputs. However, we still found the AER model necessary for generating diverse questions.
3.2 Cooperative Self-training
Although the pretrained models can generate synthetic QA pairs from corpora in unseen domains, there is always a domain shift from the seed QA corpus for pretraining to the target. To efficiently adapt the pretrained models to the new domains, we propose a cooperative self-training algorithm that allows finetuning on the target corpora without additional annotations. The finetuning is based on a three-agent (AER, QG, QAE) cooperative framework, RGX. The pipeline is illustrated in Figure 2 and comprises the following steps:
1. Produce a masked passage by replacing an answer entity selected by AER with the '[MASK] token.
2. Generate a question asking about the masked entity.
3. Feed the generated question and the original passage into the QAE to predict an answer span.
4. Optimize the QAE model with selected QA pairs.
5. Optimize the QG model with selected QA pairs.
In the proposed pipeline, all the AER, QG, and QAE models need pretraining to provide a reasonable start point for the cooperative self-training. However, the domain gap between the pretraining and the target corpus causes performance degradation. To mitigate the gap, we propose to measure the quality of generated questions and incorporate the measurement in loss functions. The quality is defined in two folds, correctness and difficulty. Firstly, the question should be fluent and answerable, and secondly, it should not be too trivial. To automatically select high-quality generated QA pairs, we introduce a expectation-maximization (EM) method based on QAE losses that learns the question quality without supervision.
3.2.1 Synthetic QA Selection with EM
To select synthetic QA pairs for finetuning, we first divide the generated questions based on the QAE loss for each question into three groups: low-, medium-, and high- loss questions. We can interpret questions with low loss as simple ones that the QAE model can easily answer. Medium-loss questions are challenging for the QAE, while those with high loss usually contain noise (e.g., containing grammatical errors or asking about incorrect answers). If we train the answering model with all questions, the training signal would be very noisy due to the high-loss questions. If we only reward questions that are correctly answered, the generator will converge to a trivial local optima. Thus, we train the QG and QAE model with the low- and medium- loss questions, namely simple and challenging questions. For the entire pipeline to be fully-automatic, we classify a given QA pair into one of the three types described above. Note that simply setting the thresholds as hyper-parameters is difficult since the loss decreases as the QAE model varies with different passages and domains. In order to find the thresholds adaptively, we apply an expectation-maximization (EM) algorithm to bucket synthetic QA pairs for each passage.
We finetune both QG and QAE models with the selected simple and challenging QA pairs. After the training, re-running the RGX pipeline with the finetuned question generation model leads to improved data synthesis. Training the QAE model on the updated synthetic dataset can significant outperform the previous finetuned QAE model.
3.2.2 Maximum Mutual Information QA
Li and Jurafsky (2016) proposed a maximum mutual information (MMI) decoding method for machine translation, and Tang et al. (2017) proposed a MMI method for jointly learning question generation and answering models. There is no previous study to our knowledge that applies MMI inference in test time of question answering that improves the final performance, because (1) modeling  $P(q|p,a)$  for all possible answers (spans)  $a$  is too inefficient, and (2) Unlike the QAE model that receives loss signals from all words in a given passage, the QG model does not receive loss signal from the passage directly, so  $P_{qq}(q|p,a)$  it is less accurate for ranking answer spans.
However, the AER and self-training strategy en
able efficient MMI inference for QA,
$$
a = \operatorname * {a r g m a x} _ {a} [ \alpha \log P _ {q g} (q | p, a) + \beta \log P _ {q a} (a | p, q) ]
$$
In test time, we run the RGX pipeline for each passage without additional training to get fine-grained AEs and corresponding questions. On the other hand, we take the top span predicted by the QAE model, and the top-k answer entities spans recognized by the RGX pipeline. In practice, we fix  $\beta = 1$ . We used an adaptive  $\alpha$  value by comparing the synthetic question generated by the QG model and the input question. For each answer entity  $a$ , we calculate
$$
\alpha = \max (1 - \mathrm {a b s} (\frac {q _ {i n p u t}}{q _ {g e n}} - 1), 0. 1)
$$
This value normalizes the question probability  $p(q|p, a)$  estimated by the QG model, since generated questions from some answer entities is easier than other spans in the same passage, which makes the QG model assign all natural questions a relative low perplexity.
In this work, we train three modules for building the cooperative self-training environment RGX, i.e., the answer entity recognizer (AER), the question generator (QG), and the question-answering extractor (QAE). We used a BERT (Devlin et al., 2018) model for AER, a BART (Lewis et al., 2019a) model for QG, and an ELECTRA (Clark et al., 2020) model for AER and QAE. To compare with the results reported in Shakeri et al. (2020) and Bartolo et al. (2021), we (1) pretrain question generation and answering models on the seed corpora, (2) generate synthetic QA data on the target domains, (3) finetune QA models with synthetic data, and (4) evaluate the finetuned QA model on human-labeled evaluation sets. The source code and demo are publicly available $^{2}$ .
In our experiment work, we leveraged Natural Questions (Kwiatkowski et al., 2019) and SQuAD v1.1 (Rajpurkar et al., 2016) as the seed corpora for pretraining all modules introduced above. To evaluate the performance of the proposed RGX on question answering tasks with different difficulty levels, we conduct experiments on both SQuAD v1.1 (Rajpurkar et al., 2016) and MRQA (Fisch et al., 2019)
out-of-domains (BioASQ, TextbookQA, RACE, RelationExtraction, DuoRC, and DROP). In the following sections, we use the term SQuAD to represent the SQuAD v1.1 corpus. For self-training, we sample 3000 passages from the training set of each corpus for data synthesis. More details about the data are provided in Appendix B
Pretraining. We pretrain the AER, QG, and QAE models on NaturalQuestions and SQuAD (i.e., the seed) corpora. For NaturalQuestions, we only use the data points containing a short answer. For Cooperative training, we follow the steps described in Section 3.2 for the cooperative training phase.
Self-training. We apply self-training for QG and QAE by finetuning the models on selected synthetic QA pairs using the same method as pretraining. The AER model is fixed after pretraining. The QAE model is finetuned using the official Huggingface (Wolf et al., 2019) training scripts for question answering. We will open-source the RGX framework if the submission is accepted.
Hyperparameters. There are three phases of model training in this work: pretraining on the seed corpora, cooperative adaptation with self-training on the target corpora, and final finetuning on the synthetic data. We adopt most of the hyper-parameters reported in the original BERT (Devlin et al., 2018), BART (Lewis et al., 2019a), and ELECTRA (Clark et al., 2020) papers. We select the final finetuning learning rates from  $\{3e - 5,4e - 5,5e - 5\}$  and report the highest performance. All the other hyper-parameters are the same as reported in the corresponding papers. For all the phases, we fix  $eps = 1e - 6$  and  $s_w = 2000$ , where  $s_w$  is the number of warm-up steps, and we apply no weight decays. We use BART-large (406M parameters) and ELECTRA-large (335M parameters) models for our experiments. We run our experiments on 2 Tesla V100 GPUs. Training the QAE models on augmented data takes about 4 hours.
We assess the performance of RGX with both semi-annotated and zero-annotated evaluation on unseen domains using exact match (EM) and F1 scores. The exact match metric assesses the percentage of predicted spans that are exactly the same as labeled answers, while the F1 score measure the overall token-level overlap between predicted and labeled
$^{2}$ https://github.com/luohongyin/RGX
answers. In our semi-annotated setting, we use the annotated answer entities in the target corpora but utilize QG to generate questions for obtaining the training question-answer pairs. The labeled questions are not used. We employ no annotation from the target corpora for the out-of-domain task but automatically construct the question-answer training pairs with entities and questions inferred by AER and QG on the corpora.
4.3.1 Semi-annotated Evaluation
The model performance with the pretrained QA model, RGX, and SOTA trained with full-supervision is shown in Table 1.
Models	EM	F1
Source domain: NQ, Target domain: SQuAD
ELECTRA-large (NaturalQuestions)	67.8	80.3
RGX	83.1	90.7
-w/o Coop. ST	81.2	89.1
ELECTRA-large (SQuAD)	89.7	94.9
Table 1: The performance of the question answering models in the semi-annotated setting. RGX stands for our cooperative training approach, and Coop. ST stands for cooperative self-training.
Table 1 shows that RGX yields improvement over the pretrained model, approaching the SOTA performance of the fully trained ELECTRA-large-discriminator model. The experiment result suggests that the cooperative learning strategy improves the question generation model with human-annotated answer entities.
4.3.2 Out-of-domain Evaluation
We also evaluate the models in unseen domains, where we do not use any annotated QA for finetuning. We train the QAE models based on the synthetic training data and evaluate the models on the target domains. We compare RGX with latest selftraining QA methods, QAGen2S (Shakeri et al., 2020) and SynQA (Bartolo et al., 2021). Since QAGen2S did not report full MRQA results, we implemented our own version. We first present the RGX performance and the results reported by the authors QAGen2S and SynQA, and then conduct ablation study by training different language models on RGX synthetic QA data.
The full evaluation results on MRQA out-of-domains are shown in Table 2, and the experiment setting comparison is shown in table 3. The results show that the models trained with the RGX framework achieve significantly higher EM and F1