SlowGuess/ACL_2020_OCR huggingface dataset:
We validate the effectiveness of combining refined and filtered data for our data refinement. We use only refined or filtered data to train our QA model, comparing with the combining approach.
The results are shown in Table 4. We observe
Table 4: Results of using filtered data, refined data, and the combination for data refinement on the SDuAD 1.1 dev set. "Iter." is short for iterative training.
τ	0.0	0.1	0.15	0.2	0.3	0.5	0.7
EM	54.3	61.2	61.8	61.1	59.7	59.2	58.5
F1	69.6	70.4	71.0	70.9	69.4	68.7	67.7
Table 5: Results of using different confidence thresholds during the construction of the refined data and filtered data.
Figure 4: Comparison on filtered data and refined data with different confidence thresholds. "F" is short for using filtered data, "R" is short for using refined data. "R+F" is short for the combination of refined and filtered data.
that both data can help the QA model to achieve better performance. Moreover, the combination of refined and filtered data is more useful than only using one of them. Using iterative training, our combination approach further improves the model performance to 72.6 F1 (1.6 absolute improvement). Besides, using our refined data contributes further improvement compared with filtered data.
5.3.3 Effects of Confidence Threshold
We experiment with several thresholds (0.0, 0.1, 0.15, 0.2, 0.3, 0.5 and 0.7) to filter the predicted answers. Their QA results on SQuAD 1.1 dev set are presented in Table 5. Using threshold of 0.15 achieves better performance.
We also analyze the effects of threshold on refined data and filtered data. As shown in Figure 4, for the filtered data, using a higher confidence threshold achieves better performance, suggesting that using the QA model for filtering makes our examples more credible. For the refined data and the combination, we observe that the threshold 0.15 achieves a better performance than the threshold 0.3, but the EM is greatly reduced when the threshold is set to 0.0. Besides, there are 26,257 answers that do not appear in named entities using the threshold 0.15, compared to 15,004 for the threshold 0.3. Thus, an appropriate threshold can help us improve the answer diversity and get rid of some noisy examples.
6725
Refined	Size	EM / F1
REFQA	-	300k	57.1 / 66.8
OA业PA	X	90k	59.4 / 69.0
OA业PA	✓	90k	50.9 / 64.6
OACPA	X	35k	47.5 / 61.2
OACPA	✓	35k	60.3 / 69.9
Others	X	75k	52.2 / 62.3
Others	✓	75k	58.8 / 69.7
Table 6: Comparison between different types of data refinement on the SQuAD 1.1 dev set.
5.3.4 Effects of Refinement Types
For brevity, we denote the original answer and predicted answer by "OA" and "PA", respectively. In order to analyze the contribution of our refined data, we categorize the data refinements into the following three types:
OA  $\supset$  PA The original answer contains the predicted answer.
OA  $\subset$  PA The predicted answer contains the original answer.
Others The remaining data except for the above two types of refinement.
For each type, we keep the original data or use refined data to train our QA model. We conduct experiments on the non-iterative setting with the data combination.
As shown in Table 6, our refined data improves the QA model in most types of refinement except "OA  $\supset$  PA". The results indicate that the QA model favors longer phrases as answer spans. Moreover, for the "OA  $\subset$  PA" and "Others" types, there are  $47.8\%$  answers that are not extracted by the NER toolkit. The iterative refinement extends the category of answer candidates, which in turn produces novel question-answer pairs.
We show a few examples of our generated data in Table 7. We list one example for each type. For the “OA  $\supset$  PA” refinement, the predicted answer is a sub-span of the extracted named entity, but the complete named entity is more appropriate as an answer. For the “OA  $\subset$  PA” refinement, the QA model can help us extend the original answer to be a longer span, which is more complete and appropriate. Besides, for the “Others” refinement, its prediction can be a new answer, and not appear in named entities extracted by the NER toolkit.
Figure 5: F1 score on the SQuAD 1.1 dev set with various training dataset sizes.
5.4 Few-Shot Learning
Following the evaluation of (Yang et al., 2017; Dhingra et al., 2018), we conduct experiments in a few-shot learning setting. We use the best configuration of our approach to train the unsupervised QA model based on BERT-Large (Whole Word Masking). Then we fine-tune the model with limited SQuAD training examples.
As shown in Figure 5, our method obtains the best performance in the restricted setting, compared with the previous state of the art (Lewis et al., 2019) and directly fine-tuning BERT. Moreover, our approach achieves 79.4 F1 (16.4 absolute gains than other models) with only 100 labeled examples. The results illustrate that our method can greatly reduce the demand of in-domain annotated data. In addition, we observe that the results of different methods become comparable when the labeled data size is greater than 10,000.
In this paper, we present two approaches to improve the quality of synthetic QA data for unsupervised question answering. We first use the Wikipedia paragraphs and its references to construct a synthetic QA data REFQA and then use the QA model to iteratively refine data over REFQA. Our method outperforms the previous unsupervised state-of-the-art models on SQuAD 1.1, and NewsQA, and achieves the best performance in the few-shot learning setting.
The work was partially supported by National Natural Science Foundation of China (NSFC) [Grant No. 61421003].
6726
OA⊃PA	S: In 1938, E. Allen Petersen escaped the advancing Japanese armies by sailing a junk, “Hummel Hummel”, from Shanghai to California with his wife Tani and two White Russians (Tsar loyalists).
Q: Who escaped the advancing Japanese armies by sailing a junk
OA: E. Allen Petersen
PA: Petersen
RQ: Who escaped the advancing Japanese armies by sailing a junk
OA⊂PA	S: Hyundai announced they would be revealing their future rally plans at the 2011 Chicago Auto Show on February 9.
Q: What at they would be revealing their future rally plans on February 9
OA: Chicago Auto Show
PA: the 2011 Chicago Auto Show
RQ: What at their future rally plans they would be revealing on February 9
Others	S: In January 2017, she released the track “That’s What’s Up” that re-imagines the spoken word segment on the Kanye West song “Low Lights”.
Q: What the Kanye West song on the spoken word segment re-imagines
OA: Low Lights
PA: That’s What’s Up
RQ: What the track she released that re-imagines the spoken word segment on the Kanye West song “Low Lights”.
Table 7: The generated and refined question-answer pairs. "S" and "Q" are short for statement and question. "OA", "PA" and "RQ" are short for the original answer, predicted answer and the refined question.
Chris Alberti, Daniel Andor, Emily Pitler, Jacob Devlin, and Michael Collins. 2019. Synthetic QA corpora generation with roundtrip consistency. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6168-6173, Florence, Italy. Association for Computational Linguistics.
Bhuwan Dhingra, Danish Danish, and Dheeraj Rajagopal. 2018. Simple and effective semi-supervised question answering. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 582-587, New Orleans, Louisiana. Association for Computational Linguistics.
Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. 2019. Unified language model pre-training for natural language understanding and generation. In 33rd Conference on Neural Information Processing Systems (NeurIPS 2019).
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2019. SpanBERT: Improving pre-training by representing and predicting spans. arXiv preprint arXiv:1907.10529.
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1601-1611, Vancouver, Canada. Association for Computational Linguistics.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, San Diego, CA.
Nikita Kitaev and Dan Klein. 2018. Constituency parsing with a self-attentive encoder. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2676–2686, Melbourne, Australia. Association for Computational Linguistics.
Guillaume Lample, Ludovic Denoyer, and Marc'Aurelio Ranzato. 2017. Unsupervised machine translation using monolingual corpora only. ArXiv, abs/1711.00043.
Patrick Lewis, Ludovic Denoyer, and Sebastian Riedel. 2019. Unsupervised question answering by cloze translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4896-4910, Florence, Italy. Association for Computational Linguistics.
Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out: Proceedings of the ACL-04 Workshop, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don't know: Unanswerable ques
6727
tions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 784-789, Melbourne, Australia. Association for Computational Linguistics.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383-2392, Austin, Texas. Association for Computational Linguistics.
Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2016. Bidirectional attention flow for machine comprehension. *ArXiv*, abs/1611.01603.
Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. 2017. NewsQA: A machine comprehension dataset. In Proceedings of the 2nd Workshop on Representation Learning for NLP, pages 191-200, Vancouver, Canada. Association for Computational Linguistics.
Shuohang Wang and Jing Jiang. 2016. Machine comprehension using match-lstm and answer pointer. ArXiv, abs/1608.07905.
Wenhui Wang, Nan Yang, Furu Wei, Baobao Chang, and Ming Zhou. 2017. Gated self-matching networks for reading comprehension and question answering. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 189-198, Vancouver, Canada. Association for Computational Linguistics.
Dirk Weissenborn, Georg Wiese, and Laura Seiffe. 2017. Making neural QA as simple as possible but not simpler. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pages 271-280, Vancouver, Canada. Association for Computational Linguistics.
Zhilin Yang, Junjie Hu, Ruslan Salakhutdinov, and William Cohen. 2017. Semi-supervised QA with generative domain-adaptive nets. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1040-1050, Vancouver, Canada. Association for Computational Linguistics.
Xuchen Yao and Benjamin Van Durme. 2014. Information extraction over structured data: Question answering with Freebase. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 956-966, Baltimore, Maryland. Association for Computational Linguistics.
Adams Wei Yu, David Dohan, Minh-Thang Luong, Rui Zhao, Kai Chen, Mohammad Norouzi, and Quoc V. Le. 2018. QANet: Combining local convolution
with global self-attention for reading comprehension. ArXiv, abs/1804.09541.
Yang Yu, Wei Zhang, Kazi Saidul Hasan, Mo Yu, Bing Xiang, and Bowen Zhou. 2016. End-to-end reading comprehension with dynamic answer chunk ranking. ArXiv, abs/1610.09996.
Haichao Zhu, Li Dong, Furu Wei, Bing Qin, and Ting Liu. 2019a. Transforming wikipedia into augmented data for query-focused summarization. ArXiv, abs/1911.03324.
Haichao Zhu, Li Dong, Furu Wei, Wenhui Wang, Bing Qin, and Ting Liu. 2019b. Learning to ask unanswerable questions for machine reading comprehension. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4238-4248, Florence, Italy. Association for Computational Linguistics.
6728