SlowGuess/ACL_2020_OCR huggingface dataset:
Yilin Niu $^{1*}$ , Fangkai Jiao $^{2*}$ , Mantong Zhou $^{1}$ , Ting Yao $^{3}$ , Jingfang Xu $^{3}$ , Minlie Huang $^{1\dagger}$
$^{1}$  Department of Computer Science and Technology, Institute for Artificial Intelligence, State Key Lab of Intelligent Technology and Systems,
Beijing National Research Center for Information Science and Technology,
Tsinghua University, Beijing 100084, China
$^{2}$  School of Computer Science and Technology, Shandong University
Sogou Inc., Beijing, China
niuy1l4@tsinghua.org.cn jiaofangkai@hotmail.com zmt.keke@gmail.com
{yaoting, jingfang}@sogou-inc.com aihuang@tsinghua.edu.cn
Neural models have achieved great success on machine reading comprehension (MRC), many of which typically consist of two components: an evidence extractor and an answer predictor. The former seeks the most relevant information from a reference text, while the latter is to locate or generate answers from the extracted evidence. Despite the importance of evidence labels for training the evidence extractor, they are not cheaply accessible, particularly in many non-extractive MRC tasks such as YES/NO question answering and multi-choice MRC.
To address this problem, we present a Self-Training method (STM), which supervises the evidence extractor with auto-generated evidence labels in an iterative process. At each iteration, a base MRC model is trained with golden answers and noisy evidence labels. The trained model will predict pseudo evidence labels as extra supervision in the next iteration. We evaluate STM on seven datasets over three MRC tasks. Experimental results demonstrate the improvement on existing MRC models, and we also analyze how and why such a self-training method works in MRC.
Machine reading comprehension (MRC) has received increasing attention recently, which can be roughly divided into two categories: extractive and non-extractive MRC. Extractive MRC requires a model to extract an answer span to a question from reference documents, such as the tasks in SQuAD (Rajpurkar et al., 2016) and CoQA (Reddy et al., 2019). In contrast, non-extractive MRC infers answers based on some evidence in reference
documents, including Yes/No question answering (Clark et al., 2019), multiple-choice MRC (Lai et al., 2017; Khashabi et al., 2018; Sun et al., 2019), and open domain question answering (Dhingra et al., 2017b). As shown in Table 1, evidence plays a vital role in MRC (Zhou et al., 2019; Ding et al., 2019; Min et al., 2018), and the coarse-to-fine paradigm has been widely adopted in multiple models (Choi et al., 2017; Li et al., 2018; Wang et al., 2018) where an evidence extractor first seeks the evidence from given documents and then an answer predictor infers the answer based on the evidence. However, it is challenging to learn a good evidence extractor due to the lack of evidence labels for supervision.
Manually annotating the golden evidence is expensive. Therefore, some recent efforts have been dedicated to improving MRC by leveraging noisy evidence labels when training the evidence extractor. Some works (Lin et al., 2018; Min et al., 2018) generate distant labels using hand-crafted rules and external resources. Some studies (Wang et al., 2018; Choi et al., 2017) adopt reinforcement learning (RL) to decide the labels of evidence. However, such RL methods suffer from unstable training. More distant supervision techniques are also used to refine noisy labels, such as deep probability logic (Wang et al., 2019), but they are hard to transfer to other tasks. Nevertheless, improving the evidence extractor remains challenging when golden evidence labels are not available.
In this paper, we present a general and effective method based on Self-Training (Scudder, 1965) to improve MRC with soft evidence extraction when golden evidence labels are not available. Following the Self-Training paradigm, a base MRC model is iteratively trained. At each iteration, the base model is trained with golden answers, as well as noisy evidence labels obtained at the preceding it
† Corresponding author: Minlie Huang.
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3916-3927 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
Q: Did a little boy write the note?
D: ...This note is from a little girl. She wants to be your friend. If you want to be her friend, ...
A: No
Q: Is she carrying something?
D: ...On the step, I find the elderly Chinese lady, small and slight, holding the hand of a little boy. In her other hand, she holds a paper carrier bag. ...
A: Yes
Table 1: Examples of Yes/No question answering. Evidential sentences in bold in reference documents are crucial to answer the questions.
eration. Then, the trained model generates noisy evidence labels, which will be used to supervise evidence extraction at the next iteration. The overview of our method is shown in Figure 1. Through this iterative process, the evidence is labeled automatically to guide the RC model to find answers, and then a better RC model benefits the evidence labeling process in return. Our method works without any manual efforts or external information, and therefore can be applied to any MRC tasks. Besides, the Self-Training algorithm converges more stably than RL. Two main contributions in this paper are summarized as follows:
1. We propose a self-training method to improve machine reading comprehension by soft evidence labeling. Compared with other existing methods, our method is more effective and general.
2. We verify the generalization and effectiveness of STM on several MRC tasks, including Yes/No question answering (YNQA), multiple-choice machine reading comprehension (MMRC), and open-domain question answering (ODQA). Our method is applicable to different base models, including BERT and DSQA (Lin et al., 2018). Experimental results demonstrate that our proposed method improves base models in three MRC tasks remarkably.
Early MRC studies focus on modeling semantic matching between a question and a reference document (Seo et al., 2017; Huang et al., 2018; Zhu
et al., 2018; Mihaylov and Frank, 2018). In order to mimic the reading mode of human, hierarchical coarse-to-fine methods are proposed (Choi et al., 2017; Li et al., 2018). Such models first read the full text to select relevant text spans, and then infer answers from these relevant spans. Extracting such spans in MRC is drawing more and more attention, though still quite challenging (Wang et al., 2019).
Evidence extraction aims at finding evidential and relevant information for downstream processes in a task, which arguably improves the overall performance of the task. Not surprisingly, evidence extraction is useful and becomes an important component in fact verification (Zhou et al., 2019; Yin and Roth, 2018; Hanselowski et al., 2018; Ma et al., 2019), multiple-choice reading comprehension (Wang et al., 2019; Bax, 2013; Yu et al., 2019), open-domain question answering (Lin et al., 2018; Wang et al., 2018), multi-hop reading comprehension (Nishida et al., 2019; Ding et al., 2019), natural language inference (Wang et al., 2017; Chen et al., 2017), and a wide range of other tasks (Nguyen and Nguyen, 2018; Chen and Bansal, 2018).
In general, evidence extraction in MRC can be classified into four types according to the training method. First, unsupervised methods provide no guidance for evidence extraction (Seo et al., 2017; Huang et al., 2019). Second, supervised methods train evidence extraction with golden evidence labels, which sometimes can be generated automatically in extractive MRC settings (Lin et al., 2018; Yin and Roth, 2018; Hanselowski et al., 2018). Third, weakly supervised methods rely on noisy evidence labels, where the labels can be obtained by heuristic rules (Min et al., 2018). Moreover, some data programming techniques, such as deep probability logic, were proposed to refine noisy labels (Wang et al., 2019). Last, if a weak extractor is obtained via unsupervised or weakly supervised pre-training, reinforcement learning can be utilized to learn a better policy of evidence extraction (Wang et al., 2018; Choi et al., 2017).
For non-extractive MRC tasks, such as YNQA and MMRC, it is cumbersome and inefficient to annotate evidence labels (Ma et al., 2019). Although various methods for evidence extraction have been proposed, training an effective extractor is still a challenging problem when golden evidence labels are unavailable. Weakly supervised methods either suffer from the low performance or rely on too many external resources, which makes them diffi
3917
cult to transfer to other tasks. RL methods can indeed train a better extractor without evidence labels. However, they are much more complicated and unstable to train, and highly dependent on model pre-training.
Our method is based on Self-Training, a widely used semi-supervised method. Most related studies follow the framework of traditional Self-Training (Scudder, 1965) and Co-Training (Blum and Mitchell, 1998), and focus on designing better policies for selecting confident samples. CoTrade (Zhang and Zhou, 2011) evaluates the confidence of whether a sample has been correctly labeled via a statistic-based data editing technique (Zighed et al., 2002). Self-paced Co-Training (Ma et al., 2017) adjusts labeled data dynamically according to the consistency between the two models trained on different views. A reinforcement learning method (Wu et al., 2018) designs an additional Q-agent as a sample selector.
3.1 Task Definition and Model Overview
The task of machine reading comprehension can be formalized as follows: given a reference document composed of a number of sentences  $D = \{S_{1}, S_{2}, \dots, S_{m}\}$  and a question  $Q$ , the model should extract or generate an answer  $\hat{A}$  to this question conditioned on the document, formally as
$$
\hat{A} = \operatorname *{argmax}_{A^{\prime}}P(A^{\prime}|Q,D).
$$
The process can be decomposed into two components, i.e., an evidence extractor and an answer predictor. The golden answer  $A$  is given for training the entire model, including the evidence extractor and the answer predictor. Denote  $E_{i}$  as a binary evidence label  $\{0,1\}$  for the  $i$ -th sentence  $S_{i}$ , where  $0/1$  corresponds to the non-evidence/evidence sentence, respectively. An auxiliary loss on the evidence labels can help the training of the evidence extractor.
The overview of our method is shown in Figure 1, which is an iterative process. During training, two data pools are maintained and denoted as  $U$  (unlabeled data) and  $L$  (labeled data). In addition to golden answers, examples in  $L$  are annotated with pseudo evidence labels. In contrast, there are only golden answers provided in  $U$ . At each iteration, the base model is trained on both data pools (two training arrows). After training, the
model makes evidence predictions on unlabeled instances (the labeling arrow), and then Selector chooses the most confident instances from  $U$  to provide noisy evidence labels. In particular, the instances with newly generated evidence labels are moved from  $U$  to  $L$  (the moving arrow), which are used to supervise evidence extraction in the next iteration. This process will iterate several times.
Figure 1: Overview of Self-Training MRC (STM). The base model is trained on both  $L$  and  $U$ . After training, the base model is used to generate evidence labels for the data from  $U$ , and then Selector chooses the most confident samples, which will be used to supervise the evidence extractor at the next iteration. The selected data is moved from  $U$  to  $L$  at each iteration.
3.2 Base Model
As shown in Figure 2, the overall structure of a base model consists of an encoder layer, an evidence extractor, and an answer predictor.
Figure 2: Overall structure of a base model that consists of an encoder layer, an evidence extractor, and an answer predictor. The encoders will obtain  $\pmb{h}^{Q}$  for the question, and  $\pmb{h}_{i}^{D}$  for each sentence in a document. The summary vector  $\pmb{h}^{D}$  will be used to predict the answer.
The encoder layer takes document  $D$  and question  $Q$  as input to obtain contextual representation for each word. Denote  $h_{i,j}^{D}$  as the representation of the  $j$ -th word in  $S_{i}$ , and  $h_{i}^{Q}$  as the representation of the  $i$ -th word in question  $Q$ . Our framework is agnostic to the architecture of the encoder, and we
show improvements on two widely used encoding models, i.e., Transformer (with BERT, Devlin et al., 2019) and LSTM (with DSQA, Lin et al., 2018) in the experiments.
The evidence extractor employs hierarchical attention, including token- and sentence-level attention, to obtain the document representation  $h^{D}$ .
Token-level attention obtains a sentence vector by self-attention (Vaswani et al., 2017) within the words in a sentence, as follows:
$$
\boldsymbol {h} _ {\boldsymbol {i}} ^ {\boldsymbol {D}} = \sum_ {j} ^ {| S _ {i} |} \alpha_ {i, j} \boldsymbol {h} _ {\boldsymbol {i}, \boldsymbol {j}} ^ {\boldsymbol {D}}, \alpha_ {i, j} \propto \exp (F ^ {S} (\boldsymbol {h} ^ {\boldsymbol {Q}}, \boldsymbol {h} _ {\boldsymbol {i}, \boldsymbol {j}} ^ {\boldsymbol {D}})),
$$
$$
\boldsymbol {s} _ {\boldsymbol {i}} ^ {D} = \sum_ {j} ^ {| S _ {i} |} \beta_ {i, j} \boldsymbol {h} _ {i, j} ^ {D}, \beta_ {i, j} \propto \exp (\boldsymbol {w} _ {\boldsymbol {s}} \boldsymbol {h} _ {i, j} ^ {D} + b _ {s}),
$$
where  $h^Q$  is the sentence representation of the question.  $\alpha_{i,j}$  refers to the importance of word  $j$  in sentence  $i$ , and so on for  $\beta_{i,j}$ .  $\mathbf{w}_s$  and  $\mathbf{b}_s$  are learnable parameters. The attention function  $F^S$  follows the bilinear form (Kim et al., 2018).
Sentence-level attention identifies important sentences conditioned on the question in a soft way to get the summary vector  $(h^{D})$ , as follows:
$$
\pmb {h} ^ {D} = \sum_ {i} ^ {m} \gamma_ {i} \pmb {h} _ {i} ^ {D}, \gamma_ {i} \propto \exp (F ^ {D} (\pmb {h} ^ {\pmb {Q}}, \pmb {s} _ {i} ^ {D})),
$$
where  $F^D$  has the same bilinear form as  $F^S$  with different parameters.  $\gamma_{i}$  refers to the importance of the corresponding sentence.
The answer predictor adopts different structures for different MRC tasks. For Yes/No question answering, we use a simple linear classifier to infer answers. For multiple-choice MRC, we use a Multiple Layer Perceptron (MLP) with Softmax to obtain the score of each choice. And for open-domain question answering, one MLP is used to predict the answer start, and another MLP is used to predict the end.
We adopt two loss functions, one for task-specific loss and the other for evidence loss.
The task-specific loss is defined as the negative log-likelihood (NLL) of predicting golden answers, formally as follows:
$$
L _ {A} (D, Q, A) = - \log P (\hat {A} = A | D, Q),
$$
where  $\hat{A}$  denotes the predicted answer and  $A$  is the golden answer.
When the evidence label  $E$  is provided, we can impose supervision on the evidence extractor. For the most general case, we assume that a variable number of evidence sentences exist in each sample  $(Q, A, D)$ . Inspired by the previous work (Nishida et al., 2019) that used multiple pieces of evidence, we calculate the evidence loss step by step. Suppose we will extract  $K$  evidence sentences. In the first step, we compute the loss of selecting the most plausible evidence sentence. In the second step, we compute the loss in the remaining sentences, where the previously selected sentence is masked and not counted in computing the loss at the second step. The overall loss is the average of all the step-by-step loss until we select out  $K$  evidence sentences. In this manner, we devise a BP-able surrogate loss function for choosing the top  $K$  evidence sentences.
Formally, we have
$$
L _ {E} (D, Q, E) = \frac {1}{K} \sum_ {k = 1} ^ {K} H (D, Q, E, M ^ {k}),
$$
where  $K$  is the number of evidence sentences, a pre-specified hyperparameter.  $M^{k} = \{M_{1}^{k}, M_{2}^{k}, \dots, M_{m}^{k}\}$  and each  $M_{i}^{k} \in \{0, -\infty\}$  is a sentence mask, where 0 means sentence  $i$  is not selected before step  $k$ , and  $-\infty$  means selected.
At each step, the model will compute an attention distribution over the unselected sentences, as follows:
$$
\lambda_ {i} ^ {k} = \frac {\exp (F ^ {D} (\pmb {h} ^ {\pmb {Q}} , \pmb {s _ {i}}) + M _ {i} ^ {k})}{\sum_ {j} (\exp (F ^ {D} (\pmb {h} ^ {\pmb {Q}}, \pmb {s _ {j}}) + M _ {j} ^ {k}))}.
$$
As  $M_i^k = -\infty$  for the previously selected sentences, the attention weight on those sentences will be zero, in other words, they are masked out. Then, the step-wise loss can be computed as follows:
$$
H (D, Q, E, M ^ {k}) = - \log \max _ {i} (\lambda_ {i} ^ {k} * E _ {i}),
$$
where  $\lambda_i^k$  indicates the attention weight for sentence  $i$ , and  $E_{i}\in \{0,1\}$  is the evidence label for sentence  $i$ . The sentence with the largest attention weight will be chosen as the  $k$ -th evidence sentence.
For each sentence  $i$ ,  $M_i^1$  is initialized to be 0. At each step  $k(k > 1)$ , the mask  $M_i^k$  will be set to  $-\infty$  if sentence  $i$  is chosen as an evidence sentence at the preceding step  $k - 1$ , and the mask
3919
remains unchanged otherwise. Formally, the mask is updated as follows:
$$
M _ {i} ^ {k} = \left\{ \begin{array}{l l} - \infty & i = \underset {j} {\operatorname {a r g m a x}} (\lambda_ {j} ^ {k - 1} E _ {j}) \\ M _ {i} ^ {k - 1} & o t h e r w i s e \end{array} \right..
$$
During training, the total loss  $L$  is the combination of the task-specific loss and the evidence loss:
$$
\begin{array}{l} L = \sum_ {(D, Q, A) \in U \cup L} L _ {A} (D, Q, A) + \\ \eta \sum_ {(D, Q, E) \in L} L _ {E} (D, Q, E), \tag {1} \\ \end{array}
$$
where  $\eta$  is a factor to balance the two loss terms.  $L$  and  $U$  denote the two sets in which instances with and without evidence labels, respectively. Note that the evidence label in  $L$  is automatically obtained in our self-training method.
3.4 Self-Training MRC (STM)
STM is designed to improve base MRC models via generating pseudo evidence labels for evidence extraction when golden labels are unavailable. STM works in an iterative manner, and each iteration consists of two stages. One is to learn a better base model for answer prediction and evidence labeling. The other is to obtain more precise evidence labels for the next iteration using the updated model.
At each iteration, STM first trains the base model with golden answers and pseudo evidence labels from the preceding iteration using the total loss as defined Equation 1. Then the trained model can predict a distribution of pseudo evidence labels for each unlabelled instance  $(D,Q,A)$ , and decides  $\hat{E}$  as
$$
\hat {E} = \underset {E ^ {\prime}} {\operatorname {a r g m i n}} L _ {E} (D, Q, E ^ {\prime}). \tag {2}
$$
Define the confidence of a labelled instance  $(D,Q,A,\hat{E})$  as
$$
\begin{array}{l} c (D, Q, A, \hat {E}) = \exp (- L _ {A} (D, Q, A)) * \\ \exp (- L _ {E} (D, Q, \hat {E})). \\ \end{array}
$$
Selector selects the instances with the largest confidence scores whose  $L_{A}(D,Q,A)$  and  $L_{E}(D,Q,\hat{E})$  are smaller than the prespecified thresholds. These labelled instances will be moved from  $U$  to  $L$  for the next iteration.
In the first iteration (iteration 0), the initial labeled set  $L$  is set to an empty set. Thus the base
model is supervised only by golden answers. In this case, the evidence extractor is trained in a distant supervised manner.
The procedure of one iteration of STM is illustrated in Algorithm 1.  $\delta$  and  $\epsilon$  are two thresholds (hyper-parameters). sort operation ranks the candidate samples according to their confidence scores  $s$  and returns the top- $n$  samples.  $n$  varies different datasets, and details are presented in the appendix.
Algorithm 1 One iteration of STM
Input: Training sets  $U, L$ ; Thresholds  $\delta$  and  $\epsilon$ ; Number of generated labels  $n$ ; Weight of evidence loss  $\eta$ ;
Output: Trained MRC model  $M$ ; Updated training sets  $U, L$ ;
1: Randomly initialize  $M$ ;
2: Train  $M$  on  $U$  and  $L$ ;
3: Initialize  $L' = \varnothing$
4: for each  $(D, Q, A) \in U$  do
5:  $l_{A} = L_{A}(D, Q, A)$ ;
6: Generate  $\hat{E}$  via Equation 2;
7:  $l_{\hat{E}} = L_E(D, Q, \hat{E})$ ;
8: if  $l_{A} \leq \delta, l_{\hat{E}} \leq \epsilon$  then
9:  $s = c(D, Q, A, \hat{E})$ ;
10: Add  $(D, Q, A, \hat{E}, s)$  to  $L'$ ;
11: end if
12: end for
13:  $L^{\prime} = \text{sort}(L^{\prime}, n)$ ;
14:  $L = L\cup L^{\prime},U = U\backslash L^{\prime}$
15: return  $M, U, L$
To understand why STM can improve evidence extraction and the performance of MRC, we revisit the training process and present a theoretical explanation, as inspired by (Anonymous, 2020).
In Section 3.4, we introduce the simple labeling strategy used in STM. If there is no sample selection, the evidence loss can be formulated as
$$
\mathcal {L} _ {\theta^ {t}} = - \mathbb {E} _ {x \sim p (x)} \mathbb {E} _ {E \sim p _ {\theta^ {t - 1}} (E | x)} \log p _ {\theta^ {t}} (E | x),
$$
where  $x$  represents  $(D,Q,A)$ , and  $\theta^t$  is the parameter of the  $t$ -th iteration. In this case, pseudo evidence labels  $E$  are randomly sampled from  $p_{\theta^{t-1}}(E|x)$  to guide  $p_{\theta^t}(E|x)$ , and therefore minimizing  $\mathcal{L}_{\theta^t}$  will lead to  $\theta^t = \theta^{t-1}$ . As a matter of fact, the sample selection strategy in STM is to filter out the low-quality pseudo labels with two
Model / Dataset	CoQA	MARCO	BoolQ
BERT-MLP	78.0	70.8	71.6
BERT-HA	78.8	71.3	72.9
BERT-HA+RL	79.3	70.3	70.4
BERT-HA+Rule	78.1	70.4	73.8
BERT-HA+STM	80.5†	72.3‡	75.2†
BERT-HA+Gold	82.0	N/A	N/A
Table 2: Classification accuracy on three Yes/No question answering datasets. N/A means there is no golden evidence label. Significance tests were conducted between BERT-HA+STM and the best baseline of each column (t-test).  $\ddagger$  means  $p$ -value  $< 0.01$ , and  $\dagger$  means  $p$ -value  $< 0.05$ .
distribution mappings,  $f$  and  $g$ . The optimizing target becomes
$$
\mathcal {L} _ {\theta^ {t}} ^ {\prime} = - \mathbb {E} _ {x \sim f (p (x))} \mathbb {E} _ {E \sim g (p _ {\theta^ {t - 1}} (E | x))} \log p _ {\theta^ {t}} (E | x).
$$
In STM,  $f$  is a filter function with two pre-specified thresholds,  $\delta$  and  $\epsilon$ .  $g$  is defined as argmax (Equation 2). Compared with random sampling, our strategy tends to prevent  $\theta^t$  from learning wrong knowledge from  $\theta^{t-1}$ . And the subsequent training might benefit from implicitly learning the strategy. In general, the strategy of STM imposes naive prior knowledge on the base models via the two distribution mappings, which may partly explain the performance gains.
4.1.1 Yes/No Question Answering (YNQA)
CoQA (Reddy et al., 2019) is a multi-turn conversational question answering dataset where questions may be incomplete and need historical context to get the answers. We extracted the Yes/No questions from CoQA, along with their histories, to form a YNQA dataset.
BoolQ (Clark et al., 2019) consists of Yes/No questions from the Google search engine. Each question is accompanied by a related paragraph. We expanded each short paragraph by concatenating some randomly sampled sentences.
MS MARCO (Nguyen et al., 2016) is a large MRC dataset. Each question is paired with a set of reference documents, and the answer may not exist in the documents. We extracted all Yes/No questions, and randomly picked some reference documents containing evidence<sup>1</sup>. To balance the ratio of Yes
and No questions, we randomly removed some questions whose answers are Yes.
4.1.2 Multiple-choice MRC
RACE (Lai et al., 2017) consists of about 28,000 passages and 100,000 questions from English exams for middle (RACE-M) and high (RACE-H) schools of China. The average number of sentences per passage in RACE-M and RACE-H is about 16 and 17, respectively.
DREAM (Sun et al., 2019) contains 10,197 multiple-choice questions with 6,444 dialogues, collected from English examinations. In DREAM,  $85\%$  of the questions require reasoning with multiple evidential sentences.