SlowGuess/ACL_2020_OCR huggingface dataset:
Xinya Du and Claire Cardie. 2018. Harvesting paragraph-level question-answer pairs from wikipedia. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018.
Xinya Du, Junru Shao, and Claire Cardie. 2017. Learning to ask: Neural question generation for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017.
Adam Fisch, Alon Talmor, Robin Jia, Minjoon Seo, Eunsol Choi, and Danqi Chen. 2019. Mrqa 2019 shared task: Evaluating generalization in reading comprehension. In EMNLP 2019 MRQA Workshop.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. In Advances in Neural Information Processing Systems, NIPS 2014.
Ian J Goodfellow, David Warde-Farley, Mehdi Mirza, Aaron Courville, and Yoshua Bengio. 2013. Maxout networks. In Proceedings of the 30th International Conference on International Conference on Machine, ICML 2013.
Michael Heilman and Noah A. Smith. 2010. Good question! statistical ranking for question generation. In Proceedings of the 2010 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NACCL-HLT 2010.
Irina Higgins, Loïc Matthew, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. 2017. beta-vae: Learning basic visual concepts with a constrained variational framework. In 5th International Conference on Learning Representations, ICLR 2017.
R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. 2019. Learning deep representations by mutual information estimation and maximization. In International Conference on Learning Representations, ICLR 2019.
Tom Hosking and Sebastian Riedel. 2019. Evaluating rewards for question generation models. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019.
Eric Jang, Shixiang Gu, and Ben Poole. 2017. Categorical reparameterization with gumbel-softmax. In 5th International Conference on Learning Representations, ICLR 2017.
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of
the Association for Computational Linguistics, ACL 2017.
Yanghoon Kim, Hwanhee Lee, Joongbo Shin, and Kyomin Jung. 2019. Improving neural question generation using answer separation. In Proceedings of the AAAI Conference on Artificial Intelligence.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015.
Diederik P. Kingma and Max Welling. 2014. Autoencoding variational bayes. In 2nd International Conference on Learning Representations, ICLR 2014.
Vishwajeet Kumar, Ganesh Ramakrishnan, and Yuan-Fang Li. 2018. A framework for automatic question generation from text using deep reinforcement learning. CoRR.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones, Ming-Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural questions: a benchmark for question answering research. Transactions of the Association of Computational Linguistics, TACL 2019.
Igor Labutov, Sumit Basu, and Lucy Vanderwende. 2015. Deep questions without deep understanding. In Proceedings of the 53rd Annual Meeting of the Association for Computational, ACL 2015.
Chin-Yew Lin and Eduard Hovy. 2002. Manual and automatic evaluation of summaries. In Proceedings of the ACL-02 Workshop on Automatic Summarization-Volume 4.
David Lindberg, Fred Popowich, John C. Nesbit, and Philip H. Winne. 2013. Generating natural language questions to support learning on-line. In Proceedings of the 14th European Workshop on Natural Language Generation, ENLG 2013.
Bang Liu, Mingjun Zhao, Di Niu, Kunfeng Lai, Yancheng He, Haojie Wei, and Yu Xu. 2019. Learning to generate questions by learning what not to generate. In The World Wide Web Conference, WWW 2019.
Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP 2015.
Chris J. Maddison, Andriy Mnih, and Yee Whye Teh. 2017. The concrete distribution: A continuous relaxation of discrete random variables. In 5th International Conference on Learning Representations, ICLR 2017.
Lili Mou, Rui Men, Ge Li, Yan Xu, Lu Zhang, Rui Yan, and Zhi Jin. 2016. Natural language inference by tree-based convolution and heuristic matching. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016.
Preksha Nema and Mitesh M. Khapra. 2018. Towards a better metric for evaluating question generation systems. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP 2018.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, ACL 2002.
Yookoon Park, Jaemin Cho, and Gunhee Kim. 2018. A hierarchical latent structure for variational conversation modeling. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NACCL-HLT 2018.
Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018.
Lisong Qiu, Juntao Li, Wei Bi, Dongyan Zhao, and Rui Yan. 2019. Are training samples correlated? learning to generate dialogue responses with multiple references. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, ACL 2019.
Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don't know: Unanswerable questions for squad. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,  $000+$  questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016.
Mrinmaya Sachan and Eric P. Xing. 2018. Self-training for jointly learning to ask and answer questions. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018.
Iulian Vlad Serban, Alessandro Sordoni, Ryan Lowe, Laurent Charlin, Joelle Pineau, Aaron Courville, and Yoshua Bengio. 2017. A hierarchical latent variable encoder-decoder model for generating dialogues. In Thirty-First AAAI Conference on Artificial Intelligence 2018.
Linfeng Song, Zhiguo Wang, and Wael Hamza. 2017. A unified query-based generative model for question generation and question answering. CoRR.
Linfeng Song, Zhiguo Wang, Wael Hamza, Yue Zhang, and Daniel Gildea. 2018. Leveraging context information for natural question generation. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018.
Jinsong Su, Shan Wu, Deyi Xiong, Yaojie Lu, Xianpei Han, and Biao Zhang. 2018. Variational recurrent neural machine translation. In Thirty-Second AAAI Conference on Artificial Intelligence 2018.
Xingwu Sun, Jing Liu, Yajuan Lyu, Wei He, Yanjun Ma, and Shi Wang. 2018. Answer-focused and position-aware neural question generation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, EMNLP 2018.
Duyu Tang, Nan Duan, Tao Qin, and Ming Zhou. 2017. Question answering and question generation as dual tasks. CoRR.
Duyu Tang, Nan Duan, Zhao Yan, Zhirui Zhang, Yibo Sun, Shujie Liu, Yuanhua Lv, and Ming Zhou. 2018. Learning to collaborate for question answering and asking. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018.
Wenhui Wang, Nan Yang, Furu Wei, Baobao Chang, and Ming Zhou. 2017. Gated self-matching networks for reading comprehension and question answering. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R'emi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface's transformers: State-of-the-art natural language processing. CoRR.
Zhilin Yang, Junjie Hu, Ruslan Salakhutdinov, and William Cohen. 2017. Semi-supervised QA with generative domain-adaptive nets. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017.
Kaichun Yao, Libo Zhang, Tiejian Luo, Lili Tao, and Yanjun Wu. 2018. Teaching machines to ask questions. In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI 2018.
Yi-Ting Yeh and Yun-Nung Chen. 2019. Qainfomax: Learning robust question answering system by mutual information maximization. In Proceedings of
the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019.
Xingdi Yuan, Tong Wang, Caglar Gülcehre, Alessandro Sordoni, Philip Bachman, Saizheng Zhang, Sandeep Subramanian, and Adam Trischler. 2017. Machine comprehension by text-to-text neural question generation. In Proceedings of the 2nd Workshop on Representation Learning for NLP, Rep4NLP@ACL 2017.
Biao Zhang, Deyi Xiong, Jinsong Su, Hong Duan, and Min Zhang. 2016. Variational neural machine translation. CoRR.
Shiyue Zhang and Mohit Bansal. 2019. Addressing semantic drift in question generation for semi-supervised question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019.
Tiancheng Zhao, Ran Zhao, and Maxine Eskenazi. 2017a. Learning discourse-level diversity for neural dialog models using conditional variational autoencoders. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017.
Tiancheng Zhao, Ran Zhao, and Maxine Eskenazi. 2017b. Learning discourse-level diversity for neural dialog models using conditional variational autoencoders. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017.
Yao Zhao, Xiaochuan Ni, Yuanyuan Ding, and Qifa Ke. 2018. Paragraph-level neural question generation with maxout pointer and gated self-attention networks. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP 2018.
Qingyu Zhou, Nan Yang, Furu Wei, Chuanqi Tan, Hangbo Bao, and Ming Zhou. 2017. Neural question generation from text: A preliminary study. In Natural Language Processing and Chinese Computing - 6th CCF International Conference, NLPCC 2017.
Haichao Zhu, Li Dong, Furu Wei, Wenhui Wang, Bing Qin, and Ting Liu. 2019. Learning to ask unanswerable questions for machine reading comprehension. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019.
A Derivation of Variational Lower Bound
Theorem. If we assume conditional independence of  $\mathbf{y}$  and  $\mathbf{z_x}$ , i.e.,  $p_{\theta}(\mathbf{y}|\mathbf{z_x},\mathbf{z_y},\mathbf{c}) = p_{\theta}(\mathbf{y}|\mathbf{z_y},\mathbf{c}),$ $\log p_{\theta}(\mathbf{x},\mathbf{y}|\mathbf{c})\geq \mathcal{L}_{HCVAE}$
The statistics and the data resource are summarized in Table 10.
SQuAD We tokenize questions and contexts with WordPiece tokenizer from BERT. To fairly compare our proposed methods with the existing semi-supervised QA, we follow Zhang and Bansal (2019)'s split, which divides original development set from SQuAD v1.1 (Rajpurkar et al., 2016) into new validation set and test set. We adopt most of the codes from Wolf et al. (2019) for preprocessing data, training, and evaluating the BERT-base QA model.
Natural Questions Other than the original Natural Questions (Kwiatkowski et al., 2019) dataset, we use subset of the dataset provided by MRQA shared task (Fisch et al., 2019) for extractive QA. As semi-supervised setting with SQuAD, we split the validation set provided from MRQA into half for validation set and the others for test set. All the tokens from question and context are tokenized with WordPiece tokenizer from BERT. We generate QA pairs from context not containing html tag, and evaluate QA model with the official MRQA evaluation scripts.
TriviaQA For TriviaQA (Joshi et al., 2017), we also use the training set from MRQA shared task, and divide the development set from MRQA into half for validation set and the other for test set. All the tokens from question and context are tokenized with WordPiece tokenizer from BERT. For evaluation, we follow the MRQA's official evaluation procedure.
HarvestingQA<sup>3</sup> We use paragraphs from HarvestingQA dataset (Du and Cardie, 2018) to generate QA pairs for QA-based Evaluation (QAE) and Reverse QA-based Evaluation (R-QAE). For the baseline QG models such as Maxout-QG and Semantic-QG, we use the same answer spans from the dataset. For the experiments of Maxout-QG baseline, we train the model and generate new questions from the context and answer, while the questions generated by Semantic-QG are provided by the authors (Zhang and Bansal, 2019).
Maxout-QG We use Adam (Kingma and Ba, 2015) optimizer with the batch size of 64 and set the initial learning rate of  $10^{-3}$ . We always set the
3https://github.com/xinyadu/harvestingQA
Datasets	Train (#)	Valid (#)	Source
SQuAD	86,588	10,507	Crowd-sourced questions from Wikipedia paragraph
Natural Questions	104,071	12,836	Questions from actual userfor searching Wikipedia paragraph
TriviaQA	74,160	7,785	Question and answer pairs authored by trivia enthusiasts from the Web
HarvestQA	1,259,691	-	Generated by neural networks from top-ranking 10,000 Wikipedia articles
Table 10: The statistics and the data source of SQuAD, Natural Questions, TriviaQA, and HarvestingQA.
Replace	EM	F1
F1 ≤ 0.0	82.4	89.39
F1 ≤ 20.0	83.11	89.65
F1 ≤ 40.0	83.32	89.79
F1 ≤ 60.0	83.20	89.78
F1 ≤ 80.0	83.09	89.75
Table 11: The effect of F1-based replacement strategy in semi-supervised setting of SQuAD+H×100%. All results are the performance on validation set of Zhang and Bansal (2019).
beam size of 10 for decoding. We also evaluate the Maxout-QG model on our SQuAD validation set with BLEU4 (Papineni et al., 2002), and get 15.68 points.
Selection of Threshold for Replacement As mentioned in our paper, we use the threshold of 40.0 selected via cross-validation of the QA model performance, using both the full SQuAD and HarvestingQA dataset for QAG. The detailed selection processes are as follows: 1) train QA model on only human annotated data, 2) compute F1 score of generated QA pairs, and 3) if the F1 score is lower than the threshold, replace the generated answer with the prediction of QA model. We investigate the optimal value of threshold among [20.0, 40.0, 60.0, 80.0] using our validation set of SQuAD. Table 11 shows the results of cross-validation on the validation set. The optimal value of 40.0 is used for semi-supervised experiments on Natural Questions and TriviaQA. For fully unlabeled semi-supervised experiments on Natural Questions and TriviaQA, the QA model is only trained on SQuAD and used to replace the synthetic QA pairs (denoted in our paper as  $\mathrm{N} \times 1-10$ ,  $\mathrm{T} \times 1-10$ ).
Semi-supervised learning For the semi-supervised learning experiment on SQuAD, we follow Zhang and Bansal (2019)'s split for a fair comparison. Specifically, we receive the unique IDs for QA pairs from the authors and use exactly the same validation and test set as theirs. For the Natural Questions and TriviaQA
experiments, we use our own split as mentioned in the above. We generate QA pairs from the paragraphs of Wikipedia extracted by Du and Cardie (2018) and train BERT-base QA model with the synthetic data for two epochs. Then we further train the model with human-annotated training data for two more epochs. The catastrophic forgetting reported in Zhang and Bansal (2019) does not occur in our cases. We use Adam optimizer (Kingma and Ba, 2015) with batch size 32 and follow the learning rate scheduling as described in (Devlin et al., 2019) with initial learning rate  $2 \cdot 10^{-5}$  and  $3 \cdot 10^{-5}$  for synthetic and human annotated data, respectively.
D Qualitative Examples
The qualitative examples in Table 12, 13, 14 are shown in the next page.
Paragraph-1 Near Tamins-Reichenau the Anterior Rhine and the Posterior Rhine join and form the Rhine. ... This section is nearly 86km long, and descends from a height of 599m to 396m. It flows through a wide glacial alpine valley known as the Rhine Valley (German: Rheintal). Near Sargans a natural dam, only a few metres high, ... The Alpine Rhine begins in the most western part of the Swiss canton of Graubünden, ...
Q-1: how long is the rhine?A-1: 86km long
Q-2: how large is the dam?A-2: a few metres high
Q-3: where does the anterior rhine and the posterior rhine join the rhine?A-3: Tamins-Reichneau
Q-4: what type of valley does the rhine flows through?A-4: glacial alpine
Q-5: what is the rhine valley in german?A-5: Rheintal
Q-6: where deos the alpine rhine begin?A-7: Swiss canton of Graubünden
Paragraph-2 Victoria is the centre of dairy farming in Australia. It is home to 60% of Australia's 3 million dairy cattle and produces nearly two-thirds of the nation's milk, almost 6.4 billion litres. The state also has 2.4 million beef cattle, with more than 2.2 million cattle and calves slaughtered each year. In 2003–04, Victorian commercial fishing crews and aquaculture industry produced 11,634 tonnes of seafood valued at nearly $109 million. ...
Q-1: what industry produced 11,63 million tonnes of seafood in 2003-04 ?A-1: aquaculture
Q-2: what type of cattle is consumed in Victoria?A-2: beef
Q-3: in what year did victorian commercial fishing and aquaculture industry produce a large amount of seafood?A-3: 2003–04
Q-4: how many cattle and calves each year are slaughtered annually?A-4: 2.2 million
Q-5: how much of the nation's milk is produced by the dairy?A-5: two-thrids
Paragraph-3 A teacher's role may vary among cultures. Teachers may provide instruction in literacy and numeracy, craftsmanship or vocational training, the arts, religion, civics, community roles, or life skills.
Q-1: what do a teacher's role vary?A-1: culture
Q-2: what do teachers provide instruction in?A-2: vocational training
Q-3: what is one thing a teacher may provide instruction for?A-3: community roles
Q-4: what is one of the skills that teachers provide in?A-4: life skills
Table 12: Examples of QA pairs generated by our Info-HCVAE. We sample multiple latent variables from  $p_{\psi}(\cdot)$ , and feed them to generation networks. All the paragraphs are from validation set of SQuAD.
Paragraph-1 Super bowl 50 was an american football game to determine the champion of the National Football League (NFL) for the 2015 season. The American Football Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers 24 – 10 to earn their third super bowl title. ...
GT which NFL team represented the AFC at super bowl 50?
Ours-1 what team did the American Football Conference represent?
Ours-2 who won the 2015 American Football Conference?
Ours-3 which team defeated the carolina panthers?
Ours-4 who defeated the panthers in 2015?
Ours-5 what team defeated the carolina panthers in the 2015 season?
Ours-6 who was the champion of the American Football League in the 2015 season?
Ours-7 what team won the 2015 American Football Conference?
Paragraph-2... Some clergy offer healing services, while exorcism is an occasional practice by some clergy in the united methodist church in Africa. ...
GT in what country does some clergy in the umc occasionally practice exorcism?
Ours-1 in what country do some clergy in the united methodist church take place?
Ours-2 in what country is exorcism practice an occasional practice?
Ours-3 use of exorcism is an occasional practice in what country?
Ours-4 is exorcism usually an occasional practice in what country?
Paragraph-3..., the city was the subject of a song, “walking into fresno”, written by hall of fame guitarist Bill Aken ...
GT who wrote “walking in fresno”?
Ours-1 who wrote “walking into fresno”?
Ours-2 “walking into fresno” was written by whom?
Ours-3 the song “walking into fresno” was written by whom?
Table 13: Examples of one-to-many mapping of our Info-HCVAE. Answers are highlighted by pink. We sample multiple question latent variables from  $p_{\psi}(\mathbf{z}_{\mathbf{x}} \mid \mathbf{c})$ , and feed them to question generation networks with a fixed answer. GT denotes ground-truth question, and Seq2Seq denotes question generated by Maxout-QG. All the paragraphs, ground truth questions, and answers are from validation set of SQuAD.
Paragraph-1 Notre Dame is known for its competitive admissions, with the incoming class enrolling in fall 2015 admitting 3,577 from a pool of 18,156 (19.7%). The academic profile of the enrolled class continues to rate among the top 10 to 15 in the nation for national research universities. ... 1,400 of the 3,577 (39.1%) were admitted under the early action plan.
Ori1	Q where does notre dame rank in terms of academic profile among research universities in the us?
A the top 10 to 15 in the nation
Gen	Q where does the academic profile of notre dame rank?
A the top 10 to 15
Q what was the rate of the incoming class enrolling in the fall of 2015?
A 3,577 from a pool of 18,156 (19.7%)
Q how many students attended notre dame?
A 3,577
Ori2	Q what percentage of students at notre dame participated in the early action program?
A 39.1%
Paragraph-2 ... begun as a one-page journal in September 1876, the scholastic magazine is issued twice monthly and ...
In 1987, when some students believed that the observer began to show a ... In spring 2008 an undergraduate journal for political science research, beyond politics, made its debut.
Ori1	Q when did the scholastic magazine of notre dame begin publishing?
A september 1876
Gen	Q when was the scholastic magazine published?
A 1876
Q in what year did notre dame get its liberal newspaper?
A 1987
Q how often is the scholastic magazine published?
A twice
Ori2	Q in what year did notre dame begin its undergraduate journal?
A 2008
Paragraph-3 As at most other universities, notre dame's students run a number of news media outlets. The nine student
-run outlets include ..., and several magazines and journals. .... the dome yearbook is published annually. ...
Ori1	Q what is the daily student paper at notre dame called?
A the observer
Gen	Q how many student media outlets are there at notre dame?
A nine student - run outlets include three
Q what type of media is the student paper at notre dame?
A number of news media
Q how often is the dome published?
A annually
Q how many magazines are published at notre dame?
A several
Ori2	Q how many student news papers are found at notre dame?
A three
Table 14: QA pairs generated by interpolating between two latent codes encoded by our posterior networks. Ori1 and Ori2 are from training set of SQuAD.