SlowGuess/ACL_2020_OCR huggingface dataset:
MultiRC (Khashabi et al., 2018) is an MMRC dataset where the amount of correct options to each question varies from 1 to 10. Each question in MultiRC is annotated with evidence from its reference document. The average number of annotated evidence sentences for each question is 2.3.
4.1.3 Open-domain QA (ODQA)
Quasar-T (Dhingra et al., 2017b) consists of 43,000 open-domain trivial questions, whose answers were extracted from ClueWeb09. For fair comparison, we retrieved 50 reference sentences from ClueWeb09 for each question the same as DSQA (Lin et al., 2018).
We compared several methods in our experiments, including some powerful base models without evidence supervision and some existing methods (*+Rule/RL/DPL/STM), which improve MRC with noisy evidence labels. Experimental details are shown in the appendix.
YNQA and MMRC: (1) BERT-MLP utilizes a BERT encoder and an MLP answer predictor. The predictor makes classification based on the BERT representation at the position of [CLS]. The parameters of the BERT module were initialized from BERT-base. (2) BERT-HA refers to the base model introduced in Section 3.2, which applies hierarchical attention over words and sentences. (3) Based on BERT-HA, BERT-HA+Rule supervises the evidence extractor with noisy evidence labels, which are derived from hand-crafted rules. We have explored three types of rules based on Jaccard similarity, integer linear programming
<sup>1</sup>The evidence annotation in a document is provided by the
original dataset.
Model / Dataset	RACE-M	RACE-H	MultiRC	DREAM
Dev Acc	Test Acc	Dev Acc	Test Acc	F1m	Dev F1a	EM0	Dev Acc	Test Acc
GPT+DPL	64.2	62.4	58.5	60.2	70.5	67.8	13.3	57.3	57.7
BERT-MLP	66.2	65.5	61.6	59.5	71.8	69.1	21.2	63.9	63.2
BERT-HA	67.8	68.2	62.6	60.4	70.1	68.1	19.9	64.2	62.8
BERT-HA+RL	68.5	66.9	62.5	60.0	72.1	69.5	21.1	63.1	63.4
BERT-HA+Rule	66.6	66.4	61.6	59.0	69.5	66.7	17.9	62.5	63.0
BERT-HA+STM	69.3‡	69.2†	64.7‡	62.6‡	74.0‡	70.9‡	22.0†	65.3‡	65.8†
BERT-HA+Gold	N/A	N/A	N/A	N/A	73.7	70.9	27.2	N/A	N/A
(ILP) (Boudin et al., 2015), and inverse term frequency (ITF) (Wang et al., 2019), among which ITF performed best in most cases. For simplicity, we merely provided experimental results with the rule of ITF. (4) Based on BERT-HA, BERT-HA+RL trains the evidence extractor via reinforcement learning, similar to (Choi et al., 2017). And (5) another deep programming logic (DPL) method, GPT+DPL (Wang et al., 2019), is complicated, and the source code is not provided. Thus we directly used the results from the original paper and did not evaluate it on BERT.
ODQA: (1) For each question, DSQA (Lin et al., 2018) aggregates multiple relevant paragraphs from ClueWeb09, and then infers an answer from these paragraphs. (2) GA (Dhingra et al., 2017a) and BiDAF (Seo et al., 2017) perform semantic matching between questions and paragraphs with attention mechanisms. And (3)  $\mathbf{R}^3$  (Wang et al., 2018) is a reinforcement learning method that explicitly selects the most relevant paragraph to a given question for the subsequent reading comprehension module.
4.3.1 Yes/No Question Answering
Table 2 shows the results on the three YNQA datasets. We merely reported the classification accuracy on the development sets since the test sets are unavailable.
BERT-HA+STM outperformed all the baselines, which demonstrates the effectiveness of our method. Compared with BERT-MLP, BERTHA achieved better performance on all the three
Table 3: Results on three multiple-choice reading comprehension datasets.  $\mathrm{(F1_a}$  : F1 score on all answer-options;  $\mathrm{F1}_m$  : macro-average F1 score of all questions;  $\mathrm{EM_0}$  : exact match.) Note that there is no golden evidence label on RACE and DREAM. The results for DPL (deep programming logic) are copied from (Wang et al., 2019). Significance tests were conducted between BERT-HA+STM and the best baseline of each column (t-test).  $\ddagger$  means  $p$ -value  $< 0.01$ , and  $\dagger$  means  $p$ -value  $< 0.05$ .
Model	EM	F1
GA (Dhingra et al., 2017a)	26.4	26.4
BiDAF (Seo et al., 2017)	25.9	28.5
R3(Wang et al., 2018)	35.3	41.7
DSQA (Lin et al., 2018)	40.7	47.6
+distant supervision	41.7	48.7
+STM	41.8†	49.2†
Table 4: Experimental results on the test set of Quasar-T.  $\mathbf{R}^3$  is a RL-based method. Results of GA, BiDAF and  $\mathbb{R}^3$  are copied from (Lin et al., 2018). DSQA+STM outperforms the best baseline (DSQA+DS) significantly (t-test,  $p$ -value  $< 0.05$ , DS= distant supervision).
datasets, indicating that distant supervision on evidence extraction can benefit Yes-No question answering. However, compared with BERT-HA, BERT-HA+RL made no improvement on MARCO and BoolQ, possibly due to the high variance in training. Similarly, BERT-HA+Rule performed worse than BERT-HA on CoQA and MARCO, implying that it is more difficult for the rule-based methods (inverse term frequency) to find correct evidence in these two datasets. In contrast, our method BERT-HA+STM is more general and performed the best on all datasets. BERT-HA+STM achieved comparable performance with BERT-HA+Gold, which stands for the upper bound by providing golden evidence labels, indicating that the effectiveness of noisy labels in our method.
4.3.2 Multiple-choice MRC
Table 3 shows the experimental results on the three MMRC datasets. We adopt the metrics from the referred papers. STM improved BERT-HA consis
Model/Dataset	CoQA P@1	MultiRC
R@1	R@2	R@3	P@1	P@2	P@3
BERT-HA	20.0	28.2	49.8	62.5	62.3	55.2	46.6
+RL	5.2	10.5	22.3	32.9	24.0	25.3	24.7
+Rule	38.4	32.4	53.6	65.1	71.8	59.6	48.7
+STM (iter 1)	32.7	32.8	57.1	70.1	72.2	63.3	52.5
+STM (iter 2)	37.3	32.9	58.0	71.3	72.7	64.4	53.5
+STM (iter 3)	39.9	31.4	55.3	68.8	69.5	61.6	51.6
BERT-HA+Gold	53.6	33.7	59.5	73.4	74.5	65.9	54.8
Table 5: Evidence extraction evaluation on the development sets of CoQA and MultiRC.  $\mathrm{P@k / R@k}$  represent precision / recall of the generated evidence labels, respectively for top  $k$  predicted evidence sentences.
tently on RACE-H, MultiRC and DREAM in terms of all the metrics. However, the improvement on RACE-M is limited (1.0 gain on the test sets). The reason may be that RACE-M is much simpler than RACE-H, and thus, it is not challenging for the evidence extractor of BERT-HA to find the correct evidence on RACE-M.
4.3.3 Open-domain Question Answering
Table 4 shows the exact match scores and F1 scores on Quasar-T. Distant evidence supervision (DS) indicates whether a passage contains the answer text. Compared with the base models DSQA and DSQA+DS, DSQA+STM achieved better performance in both metrics, which verifies that DSQA can also benefit from Self-Training. Our method is general and can improve both lightweight and heavyweight models, like LSTM-based and BERT-based models, in different tasks.
4.4 Performance of Evidence Extraction
To evaluate the performance of STM on evidence extraction, we validated the evidence labels generated by several methods on the development sets of CoQA and MultiRC. Considering that the evidence of each question in MultiRC is a set of sentences, we adopted precision@k and recall@k as the metrics for MultiRC, which represent the precision and recall of the generated evidence labels, respectively, when  $k$  sentences are predicted as evidence. We adopted only precision@1 as the metric for CoQA as this dataset provides each question with one golden evidence sentence.
Table 5 shows the performance of five methods for evidence labeling on the CoQA and MultiRC development sets. It can be seen that BERT-HA+STM outperformed the base model BERT-HA by a large margin in terms of all the metrics. As a result, the evidence extractor augmented with STM pro
vided more evidential information for the answer predictor, which may explain the improvements of BERT-HA+STM on the two datasets.
4.5 Analysis on Error Propagation
To examine whether error propagation exists and how severe it is in STM, we visualized the evolution of evidence predictions on the development set of CoQA (Figure 3). From the inside to the outside, the four rings show the statistic results of the evidence predicted by BERT-HA (iteration 0) and BERT-HA+STM (iteration 1, 2, 3). Each ring is composed of all the instances from the development set of CoQA, and each radius corresponds to one sample. If the evidence of an instance is predicted correctly, the corresponding radius is marked in green, otherwise in purple. Two examples are shown in the appendix due to space limit.
Self-correction. As the innermost ring shows, about  $80\%$  of the evidence predicted by BERT-HA (iter 0) was incorrect. However, the proportion of wrong instances reduced to  $60\%$  after self-training (iter 3). More concretely,  $27\%$  of the wrong predictions were gradually corrected with high confidence within three self-training iterations, as exemplified by instance A in Figure 3.
Error propagation. We observed that  $4\%$  of the evidence was mistakenly revised by STM, as exemplified by instance  $B$  in Figure 3. In such a case, the incorrect predictions are likely to be retained in the next iteration. But almost  $50\%$  of such mistakes were finally corrected during the subsequent iterations like instance  $C$ . This observation shows that STM can prevent error propagation to avoid catastrophic failure.
Figure 3: Evolution of evidence predictions on the development set of CoQA. From the inside to the outside, the four rings correspond to BERT-HA (iteration 0) and BERT-HA+STM (iteration 1, 2, 3), respectively.
Model/Metric	Ans. Acc	Evi. Acc
RoBERTa-HA	92.6	13.8
RoBERTa-HA+STM	92.7	19.3(+40%)
Table 6: Answer prediction accuracy (Ans. Acc) and evidence extraction accuracy (Evi. Acc) on the development set of CoQA.
4.6 Improvement Over Stronger Pretrained Models
To evaluate the improvement of STM over stronger pre-trained models, we employed RoBERTa-large (Liu et al., 2019) as the encoder in the base model. Table 6 shows the results on CoQA. STM significantly improved the evidence extraction (Evi. Acc) of the base model. However, the improvement on answer prediction (Ans. Acc) is marginal. One reason is that RoBERTa-HA achieved such a high performance that there was limited room to improve. Another possible explanation is that evidence information is not important for such stronger models to generate answers. In other words, they may be more adept at exploiting data bias to make answer prediction. In comparison, weaker pre-trained models, such as BERT-base, can benefit from evidence information due to their weaker ability to exploit data bias.
We present an iterative self-training method (STM) to improve MRC models with soft evidence extraction, when golden evidence labels are unavailable. In this iterative method, we train the base model with golden answers and pseudo evidence labels. The updated model then generates new pseudo evidence labels, which can be used as additional supervision in the next iteration. Experiment results show that our proposed method consistently improves the base models in seven datasets for three MRC tasks, and that better evidence extraction indeed enhances the final performance of MRC.
As future work, we plan to extend our method to other NLP tasks which rely on evidence finding, such as natural language inference.
This work was jointly supported by the NSFC projects (Key project with No. 61936010 and regular project with No. 61876096), and the National Key R&D Program of China (Grant No.
2018YFC0830200). We thank THUNUS NExT Joint-Lab for the support.
Anonymous. 2020. Revisiting self-training for neural sequence generation. ICLR under review.
Stephen Bax. 2013. The cognitive processing of candidates during reading tests: Evidence from eyetracking. Language Testing, 30:441-465.
Avrim Blum and Tom M. Mitchell. 1998. Combining labeled and unlabeled data with co-training. In  $COLT$ , pages 92-100.
Florian Boudin, Hugo Mougard, and Benoit Favre. 2015. Concept-based summarization using integer linear programming: From concept pruning to multiple optimal solutions. In EMNLP, pages 1914-1918.
Qian Chen, Xiaodan Zhu, Zhen-Hua Ling, Si Wei, Hui Jiang, and Diana Inkpen. 2017. Enhanced LSTM for natural language inference. In ACL, pages 1657-1668.
Yen-Chun Chen and Mohit Bansal. 2018. Fast abstractive summarization with reinforce-selected sentence rewriting. In ACL, pages 675-686.
Eunsol Choi, Daniel Hewlett, Jakob Uszkoreit, Illia Polosukhin, Alexandre Lacoste, and Jonathan Berant. 2017. Coarse-to-fine question answering for long documents. In ACL, pages 209-220.
Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. *Boolq: Exploring the surprising difficulty of natural yes/no questions.* In *NAACL*, pages 2924–2936.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In *NAACL*, pages 4171–4186.
Bhuwan Dhingra, Hanxiao Liu, Zhilin Yang, William W. Cohen, and Ruslan Salakhutdinov. 2017a. Gated-attention readers for text comprehension. In ACL, pages 1832–1846.
Bhuwan Dhingra, Kathryn Mazaitis, and William W. Cohen. 2017b. Quasar: Datasets for question answering by search and reading. CoRR, abs/1707.03904.
Ming Ding, Chang Zhou, Qibin Chen, Hongxia Yang, and Jie Tang. 2019. Cognitive graph for multi-hop reading comprehension at scale. In ACL, pages 2694-2703.
Andreas Hanselowski, Hao Zhang, Zile Li, Daniil Sorokin, Benjamin Schiller, Claudia Schulz, and Iryna Gurevych. 2018. Ukp-athene: Multi-sentence
textual entailment for claim verification. CoRR, abs/1809.01479.
Hsin-Yuan Huang, Eunsol Choi, and Wen-tau Yih. 2019. Flowqa: Grasping flow in history for conversational machine comprehension. In ICLR.
Hsin-Yuan Huang, Chenguang Zhu, Yelong Shen, and Weizhu Chen. 2018. Fusionnet: Fusing via fully-aware attention with application to machine comprehension. In ICLR.
Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. 2018. Looking beyond the surface: A challenge set for reading comprehension over multiple sentences. In NAACL, pages 252-262.
Jin-Hwa Kim, Jaehyun Jun, and Byoung-Tak Zhang. 2018. Bilinear attention networks. In NIPS, pages 1571-1581.
Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017. Race: Large-scale reading comprehension dataset from examinations. In EMNLP, pages 785-794.
Weikang Li, Wei Li, and Yunfang Wu. 2018. A unified model for document-based question answering based on human-like reading strategy. In AAAI, pages 604-611.
Yankai Lin, Haozhe Ji, Zhiyuan Liu, and Maosong Sun. 2018. Denoising distantly supervised open-domain question answering. In ACL, pages 1736-1745.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692.
Fan Ma, Deyu Meng, Qi Xie, Zina Li, and Xuanyi Dong. 2017. Self-paced co-training. In ICML, pages 2275-2284.
Jing Ma, Wei Gao, Shafiq R. Joty, and Kam-Fai Wong. 2019. Sentence-level evidence embedding for claim verification with hierarchical attention networks. In ACL, pages 2561-2571.
Todor Mihaylov and Anette Frank. 2018. Knowledgeable reader: Enhancing cloze-style reading comprehension with external commonsense knowledge. In ACL, pages 821-832.
Sewon Min, Victor Zhong, Richard Socher, and Caiming Xiong. 2018. Efficient and robust question answering from minimal context over documents. In ACL, pages 1725-1735.
Minh Nguyen and Thien Nguyen. 2018. Who is killed by police: Introducing supervised attention for hierarchical lstms. In COLING, pages 2277-2287.
Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. MS MARCO: A human generated machine reading comprehension dataset. In NIPS.
Kosuke Nishida, Kyosuke Nishida, Masaaki Nagata, Atsushi Otsuka, Itsumi Saito, Hisako Asano, and Junji Tomita. 2019. Answering while summarizing: Multi-task learning for multi-hop QA with evidence extraction. In ACL, pages 2335-2345.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,  $000+$  questions for machine comprehension of text. In EMNLP, pages 2383-2392.
Siva Reddy, Danqi Chen, and Christopher D. Manning. 2019. Coqa: A conversational question answering challenge. TACL, 7:249-266.
H. J. Scudder. 1965. Probability of error of some adaptive pattern-recognition machines. IEEE Trans. Information Theory, 11.
Min Joon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2017. Bidirectional attention flow for machine comprehension. In ICLR.
Kai Sun, Dian Yu, Jianshu Chen, Dong Yu, Yejin Choi, and Claire Cardie. 2019. DREAM: A challenge dataset and models for dialogue-based reading comprehension. TACL, 7:217-231.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NIPS, pages 6000-6010.
Hai Wang, Dian Yu, Kai Sun, Jianshu Chen, Dong Yu, Dan Roth, and David A. McAllester. 2019. Evidence sentence extraction for machine reading comprehension. CoRR, abs/1902.08852.
Shuohang Wang, Mo Yu, Xiaoxiao Guo, Zhiguo Wang, Tim Klinger, Wei Zhang, Shiyu Chang, Gerry Tesauro, Bowen Zhou, and Jing Jiang. 2018.  $\mathbf{R}^{3}$ : Reinforced ranker-reader for open-domain question answering. In AAAI, pages 5981-5988.
Zhiguo Wang, Wael Hamza, and Radu Florian. 2017. Bilateral multi-perspective matching for natural language sentences. In *IJCAI*, pages 4144-4150.
Jiawei Wu, Lei Li, and William Yang Wang. 2018. Reinforced co-training. In NAACL, pages 1252-1262.
Wenpeng Yin and Dan Roth. 2018. Twowingos: A twowing optimization strategy for evidential claim verification. In EMNLP, pages 105-114.
Jianxing Yu, Zhengjun Zha, and Jian Yin. 2019. Inferential machine comprehension: Answering questions by recursively deducing the evidence chain from text. In ACL, pages 2241-2251.
Min-Ling Zhang and Zhi-Hua Zhou. 2011. Cotrade: Confident co-training with data editing. TSMCB, 41:1612-1626.
Jie Zhou, Xu Han, Cheng Yang, Zhiyuan Liu, Lifeng Wang, Changcheng Li, and Maosong Sun. 2019. GEAR: graph-based evidence aggregating and reasoning for fact verification. In ACL, pages 892-901.
Chenguang Zhu, Michael Zeng, and Xuedong Huang. 2018. Sdnet: Contextualized attention-based deep network for conversational question answering. CoRR, abs/1812.03593.
Djamel A. Zighed, Stéphane Lallich, and Fabrice Muhlenbach. 2002. Separability index in supervised learning. In PKDD, pages 475-487.
A Case Study
Case 1
Case 2
Figure 4: Weight distribution of the two cases from the sentence-level attention.
In Section 4.5 of the main paper, we provide a quantitative analysis of the evolution of evidence predictions, and draw two conclusions: (1) STM can help the base model to correct itself; (2) Error propagation will not result in catastrophic failure, though exists.
To help understand these two conclusions, we provide two corresponding cases from the development set of CoQA (Reddy et al., 2019). The original instances are shown in Table 7, and the weight distribution from the sentence-level attention is shown in Figure 4. In case 1, BERT-HA made wrong evidence prediction, while STM revised it subsequently, which shows the ability of self-correction. In case 2, BERT-HA first selected the correct evidence with high confidence. However, in the iteration 1, BERT-HA with STM was distracted by another plausible sentence. Instead of insisting on the incorrect prediction, STM led BERT-HA back to the right way, which shows that error propagation is not catastrophic.
B Hyper-Parameters for Self-Training
We implemented BERT-HA with BERT-base from a commonly used library $^2$ , and directly used the original source code of  $\mathrm{DSQA}^3$  (Lin et al., 2018). All the codes and datasets will be released after the review period. The hyper-parameters used in BERT-HA and BERT-HA+STM are shown in Table 8.
<sup>3</sup>https://github.com/thunlp/OpenQA
3926
Case 1)
Passage:
...(3) "Why don't you tackle Indian River, Daylight?" (4) Harper advised, at parting. (5) There's whole slathers of creeks and draws draining in up there, and somewhere gold just crying to be found. (6) That's my hunch. (7) There's a big strike coming, and Indian River ain't going to be a million miles away. (8) "And the place is swarming with moose," Joe Ladue added. (9) "Bob Henderson's up there somewhere, been there three years now, swearing something big is going to happen, living off'n straight moose and prospecting around like a crazy man." (10) Daylight decided to go Indian River a flutter, as he expressed it; but Elijah could not be persuaded into accompanying him. Elijah's soul had been seared by famine, and he was obsessed by fear of repeating the experience. (11) "I jest can't bear to separate from grub," he explained. (12) "I know it's downright foolishness, but I jest can't help it..."
Question: Are there many bodies of water there?
Case 2)
(1)If you live in the United States, you can't have a full-time job until you are 16 years old. (2)At 14 or 15, you work part-time after school or on weekends, and during summer vacation you can work 40 hours each week. (3)Does all that mean that if you are younger than 14, you can't make your own money? (4)Of course not! (5)Kids from 10-13 years of age can make money by doing lots of things. (6)Valerie, 11, told us that she made money by cleaning up other people's yards. ...(11)Kids can learn lots of things from making money. (12)By working to make your own money, you are learning the skills you will need in life. (13)These skills can include things like how to get along with others, how to use technology and how to use your time wisely. (14)Some people think that asking for money is a lot easier than making it; however, if you can make your own money, you don't have to depend on anyone else...
Question: Can they learn time management?
Table 7: Examples from the development set of CoQA. Evidential sentences in red in reference passages are crucial to answer the questions. Sentences in blue are distracting as Figure 4 shows.
Dataset	RACE-H	RACE-M	DREAM	MultiRC	CoQA	MARCO	BoolQ
Lmax	380	380	512	512	512	480	512
learning rate	5e-5♣/4e-5♣	5e-5♣/4e-5♣	2e-5	2e-5	2e-5	2e-5	3e-5
epoch	3	3	5	8	3	2♣/3♣	4
η	0.8	0.8	0.8	0.8	0.8	0.8	0.8
batch size	32	32	32	32	6	8	6
ε	0.5	0.5	0.5	0.5	0.6	0.5	0.5
δ	0.9	0.9	0.8	0.8	0.9	0.9	0.7
n	40000	10000	3000	2000	1500	1000	500
Kmax	2	3	4	3	1	1	1
Table 8: Hyper-parameters marked with  $\clubsuit/\spadesuit$  are used in BERT-HA/BERT-HA+STM, respectively. Other unmarked hyper-parameters are shared by these two models.