SlowGuess/ACL_2020_OCR huggingface dataset:
Especially for baseline-1, the performance gains of our model are more obvious. Our hybrid model-1 outperforms baseline-1 by 1.52 points on Zhou Split and 1.34 points on Du Split, which are large margins for this challenging task. Even based on this weak baseline, our method also achieves the state-of-the-art, 16.55 BLEU-4 score on Du Split for sentence-level QG.
The previous work of CGC-QG (Liu et al., 2019) obtains a 17.55 BLEU-4 score on Zhou Split. But their model relies on many heuristic rules and ad-hoc strategies. In their full model with clue prediction, they do graph convolutional network (GCN) operations on dependency trees, while our model does not use any hand-crafted rules and is lightweight without graphs and trees.
We also conduct experiments on MARCO, and the results are shown in Table 5. Our hybrid models obtain obvious improvements over two baselines, achieving a state-of-the-art BLEU-4 score of 21.61.
Specifically, SQuAD and MARCO are built in different ways. The questions in SQuAD are generated by crowd-workers, while questions in MARCO are sampled from real user queries. The experimental results on two datasets validate the generalization and robustness of our models.
Effect of Multi-task Learning with PG Task As shown in Table 4, the auxiliary PG task brings consistent improvements over both baseline models. On Zhou Split, it increases baseline-1 by 1.38 points and baseline-2 by 0.61 respectively. On Du Split, it increases baseline-1 by 1.16 points and baseline-2 by 0.51 points respectively. The
Previous Works	BLEU-4
s2s(Du et al., 2017)	10.46
s2sa-at-mp-gsa(Zhao et al., 2018b)	16.02
A-P-Hybrid(Sun et al., 2018)	19.45
LM enhanced QG(Zhou et al., 2019a)	20.88
Q-type(Zhou et al., 2019b)	21.59
Our Models
baseline-1	20.13
hybrid model-1	21.15
baseline-2	20.79
hybrid model-2	21.61
Table 5: Main results of our models on MARCO.
reason is that the PG task provides abundant paraphrase knowledge into the model and allows the task-share encoder to learn more paraphrasing representations.
Effect of Diversity Training with Min-loss Function From the results in Table 4, we can see the min-loss strategy improves performances over both baseline models. On Zhou Split, we get a 0.77 improvement over baseline-1 and 0.48 improvement over baseline-2, respectively. On Du Split, we get similar improvements.
Effect of Data Augmentation A straightforward way to leverage paraphrase knowledge is data augmentation. To test whether it works by simply adding paraphrase data as external training data, we also conduct an experiment based on the question paraphrase resource. We add the  $(\mathbf{s},\mathbf{q}^{\prime})$  pairs into the training dataset, where s represents the input sentence and  $\mathbf{q}'$  denotes the paraphrase of the golden reference. Under this setting, we double the training samples. Unfortunately, as shown in Table 4, the baseline-1 model yields much lower BLEU-4 scores on both Zhou Split (13.28) and Du Split (13.36) with such data augmentation. The main reason is that for the same input sentence, there are two different training targets  $(\mathbf{q}$  and  $\mathbf{q}'$ ), making the training process cannot easily converge.
5.2 Diversity Test
To investigate whether the paraphrase knowledge introduces more diverse expressions, we conduct evaluations on the distinct metric (Li et al., 2016), which is calculated as the number of distinct unigrams (distinct-1) and bigrams (distinct-2) divided by the total number of the generated words. The experimental results are shown in Table 6. It shows that our hybrid models obtain obvious gains over baseline models on both distinct-1 and distinct-2
metrics, validating that our models really generate more diverse questions with the help of paraphrase knowledge.
Models	distinct-1	distinct-2
baseline-1	9.49	39.48
hybrid model-1	9.75	41.97
baseline-2	9.81	41.14
hybrid model-2	9.98	42.43
5.3 Ablation Study of Soft Sharing
We also verify the effectiveness of the soft sharing mechanism by removing it from the full hybrid models. The results are displayed in Table 7. After removing the soft sharing mechanism, both of our models have varying degrees of performance degradation. It demonstrates that the soft sharing strategy enhances the influence of paraphrase knowledge on QG decoder.
Table 6: Results of the distinct metric on zhou split.
Models	BLEU-4	METEOR
hybrid model-1	16.23	20.29
w/o soft sharing	15.87	20.04
hybrid model-2	16.93	20.58
w/o soft sharing	16.32	20.34
5.4 Parameters Selection
The soft sharing coefficient hyper-parameter  $\lambda$  is  $1\times 10^{-6}$ , intuitively chosen by balancing the cross-entropy and regularization losses according to Guo et al. (2018b). The other hyper-parameter  $\alpha$  which is to control the balance of QG and PG is tuned by grid search. We set  $\alpha$  to different values to explore the best proportion of two tasks. The experimental results of different  $\alpha$  are shown in Figure 3. Consequently, we set  $\alpha$  to 0.3 for our hybrid model.
Figure 3: The influence of  $\alpha$  on BLEU-4 scores on development set of Zhou Split.
5.5 Human Evaluation
To further assess the quality of generated questions, we perform human evaluation to compare our hybrid model-2 with the strong baseline of language modeling enhanced QG. We randomly select 100 samples from SQuAD (Zhou Split) and ask three annotators to score these generated questions according to three aspects:
Fluency: which measures whether a question is grammatical and fluent;
Relevancy: which measures whether the question is relevant to the input context;
Answerability: which indicates whether the question can be answered by the given answer.
The rating score is set to [0, 2]. The evaluation results are shown in Table 8. The Spearman correlation coefficients between annotators are high, which guarantees the validity of human evaluation. Our hybrid model receives higher scores on all three metrics, indicating that our generated questions have higher quality in different aspects.
Table 7: Ablation studies of soft sharing on Zhou Split.
Models	Fluency	Relevancy	Answerability
baseline-2	1.785	1.535	1.134
hybrid model-2	1.874	1.682	1.333
Spearman	0.722	0.693	0.861
Table 8: Human evaluation results.
We list two examples of generated questions in Table 9. By introducing paraphrase knowledge into generation, the generated questions well capture the paraphrase transitions between contexts and references. Obviously, the questions generated by our hybrid model are more grammatical and coherent.
5.7 Different Paraphrasing Methods
To further test the generalization of our proposed methods, we use other paraphrasing methods to construct the paraphrase dataset.
PPDB: for each non-stop word and phrase, looking it up in PPDB (2013) and replacing it with its synonyms.
NMT: another back-translation method using a pre-trained Transformer (2017) model.
Mixed: expanding input sentences with Google Trans and expanding reference questions with PPDB.
The results are shown in Table 10. Our hybrid model-2 still achieves excellent performances on both BLEU and METEOR. From the results, we
6137
his lab was torn down in 1904, and its contents were sold two years later to satisfy a debt.
torn down
Reference Question:
what happened to his lab?
Baseline Model-2:
what was [UNK] 's lab?
Hybrid Model-2:
what happened to his lab in 1904?
newcastle has a horse racing course at gosforth park.
gosforth park
where is newcastle's horse racing course located?
where does newcastle have a horse racing course?
Table 9: Examples of generated questions.
Paraphrasing Methods	BLEU-4	METEOR
baseline-2	16.13	19.95
PPDB	16.65	20.57
NMT	16.76	20.44
Google Trans	16.93	20.58
Mixed	17.05	20.75
Table 10: Hybrid model-2 performances using different paraphrase expansion methods on SQuAD(Zhou Split).
can observe that the Mixed paraphrase method even obtain better results than the mature Google Translate. It proves that our proposed architecture is effective across different paraphrasing methods and has potential for improvement.
In this paper, we propose a two-hand hybrid model leveraging paraphrase knowledge for QG. The experimental results of independent modules and hybrid models prove that our models are effective and transferable. Besides, human evaluation results demonstrate that the paraphrase knowledge benefits our model to ask more human-like questions of high quality. In the future, we will explore more diverse and advanced paraphrase expanding methods for both sentence and paragraph level QG. Moreover, we will apply our methods to other similar tasks, such as sentence simplification.
We thank Weikang Li and Minghua Zhang for their valuable comments and suggestions. This work is supported by the National Natural Science Foundation of China (61773026) and the Key Project of Natural Science Foundation of China (61936012).
Chris Callison-Burch, Philipp Koehn, and Miles Osborne. 2006. Improved statistical machine translation using paraphrases. In Proceedings of the Human Language Technology Conference of the NAACL, Main Conference, pages 17-24, New York City, USA. Association for Computational Linguistics.
Bo Chen, Le Sun, Xianpei Han, and Bo An. 2016. Sentence rewriting for semantic parsing. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 766-777, Berlin, Germany. Association for Computational Linguistics.
Yu Chen, Lingfei Wu, and Mohammed J. Zaki. 2019. Natural question generation with reinforcement learning based graph-to-sequence model. *ArXiv*, abs/1910.08832.
Michael J. Denkowski and Alon Lavie. 2014. Meteor universal: Language specific translation evaluation for any target language. In WMT@ACL.
Li Dong, Jonathan Mallinson, Siva Reddy, and Mirella Lapata. 2017. Learning to paraphrase for question answering. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 875-886, Copenhagen, Denmark. Association for Computational Linguistics.
Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. 2019. Unified language model pre-training for natural language understanding and generation. In NeurIPS.
Xinya Du and Claire Cardie. 2018. Harvesting paragraph-level question-answer pairs from Wikipedia. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1907-1917, Melbourne, Australia. Association for Computational Linguistics.
Xinya Du, Junru Shao, and Claire Cardie. 2017. Learning to ask: Neural question generation for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1342-1352, Vancouver, Canada. Association for Computational Linguistics.
6138
Nan Duan, Duyu Tang, Peng Chen, and Ming Zhou. 2017. Question generation for question answering. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 866-874, Copenhagen, Denmark. Association for Computational Linguistics.
Anthony Fader, Luke Zettlemoyer, and Oren Etzioni. 2013. Paraphrase-driven learning for open question answering. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1608-1618, Sofia, Bulgaria. Association for Computational Linguistics.
Juri Ganitkevitch, Benjamin Van Durme, and Chris Callison-Burch. 2013. PPDB: The paraphrase database. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 758-764, Atlanta, Georgia. Association for Computational Linguistics.
Yifan Gao, Jianan Wang, Lidong Bing, Irwin King, and Michael R. Lyu. 2018. Difficulty controllable question generation for reading comprehension. *ArXiv*, abs/1807.03586.
Han Guo, Ramakanth Pasunuru, and Mohit Bansal. 2018a. Dynamic multi-level multi-task learning for sentence simplification. In Proceedings of the 27th International Conference on Computational Linguistics, pages 462-476, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Han Guo, Ramakanth Pasunuru, and Mohit Bansal. 2018b. Soft layer-specific multi-task summarization with entailment and question generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 687-697, Melbourne, Australia. Association for Computational Linguistics.
Michael Heilman and Noah A. Smith. 2010. Good question! statistical ranking for question generation. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 609-617, Los Angeles, California. Association for Computational Linguistics.
Yanghoon Kim, Hwanhee Lee, Joongbo Shin, and Kyomin Jung. 2018. Improving neural question generation using answer separation. In AAAI.
Olga Kovaleva, Anna Rumshisky, and Alexey Romanov. 2018. Similarity-based reconstruction loss for meaning representation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4875-4880, Brussels, Belgium. Association for Computational Linguistics.
Reno Kriz, Eleni Miltsakaki, Marianna Apidianaki, and Chris Callison-Burch. 2018. Simplification using paraphrases and context-based lexical substitution. In Proceedings of the 2018 Conference of the
North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 207-217, New Orleans, Louisiana. Association for Computational Linguistics.
Jingjing Li, Yifan Gao, Lidong Bing, Irwin King, and Michael R. Lyu. 2019a. Improving question generation with to the point context. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3216-3226, Hong Kong, China. Association for Computational Linguistics.
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016. A diversity-promoting objective function for neural conversation models. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 110-119, San Diego, California. Association for Computational Linguistics.
Zichao Li, Xin Jiang, Lifeng Shang, and Qun Liu. 2019b. Decomposable neural paraphrase generation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3403-3414, Florence, Italy. Association for Computational Linguistics.
Bang Liu, Mingjun Zhao, Di Niu, Kunfeng Lai, Yancheng He, Haojie Wei, and Yu Xu. 2019. Learning to generate questions by learning what not to generate. ArXiv, abs/1902.10418.
Nasrin Mostafazadeh, Ishan Misra, Jacob Devlin, Margaret Mitchell, Xiaodong He, and Lucy Vanderwende. 2016. Generating natural questions about an image. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1802-1813, Berlin, Germany. Association for Computational Linguistics.
Preksha Nema, Akash Kumar Mohankumar, Mitesh M. Khapra, Balaji Vasan Srinivasan, and Balaraman Ravindran. 2019. Let's ask again: Refine network for automatic question generation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3314-3323, Hong Kong, China. Association for Computational Linguistics.
Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. Ms marco: A human generated machine reading comprehension dataset. ArXiv, abs/1611.09268.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311-318, Philadelphia,
6139
Pennsylvania, USA. Association for Computational Linguistics.
Aaditya Prakash, Sadid A. Hasan, Kathy Lee, Vivek Datla, Ashequl Qadir, Joey Liu, and Oladimeji Farri. 2016. Neural paraphrase generation with stacked residual LSTM networks. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 2923-2934, Osaka, Japan. The COLING 2016 Organizing Committee.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383-2392, Austin, Texas. Association for Computational Linguistics.
Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1073-1083, Vancouver, Canada. Association for Computational Linguistics.
Linfeng Song, Zhiguo Wang, Wael Hamza, Yue Zhang, and Daniel Gildea. 2018. Leveraging context information for natural question generation. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 569-574, New Orleans, Louisiana. Association for Computational Linguistics.
Xingwu Sun, Jing Liu, Yajuan Lyu, Wei He, Yanjun Ma, and Shi Wang. 2018. Answer-focused and position-aware neural question generation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3930-3939, Brussels, Belgium. Association for Computational Linguistics.
Duyu Tang, Nan Duan, Tao Qin, and Ming Zhou. 2017. Question answering and question generation as dual tasks. ArXiv, abs/1706.02027.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NIPS.
Qizhe Xie, Zihang Dai, Eduard H. Hovy, Minh-Thang Luong, and Quoc V. Le. 2019. Unsupervised data augmentation. ArXiv, abs/1904.12848.
Jingjing Xu, Yuechen Wang, Duyu Tang, Nan Duan, Pengcheng Yang, Qi Zeng, Ming Zhou, and Xu Sun. 2019. Asking clarification questions in knowledge-based question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing
(EMNLP-IJCNLP), pages 1618-1629, Hong Kong, China. Association for Computational Linguistics.
Adams Wei Yu, David Dohan, Minh-Thang Luong, Rui Zhao, Kai Chen, Mohammad Norouzi, and Quoc V. Le. 2018. Qanet: Combining local convolution with global self-attention for reading comprehension. ArXiv, abs/1804.09541.
Shiyue Zhang and Mohit Bansal. 2019. Addressing semantic drift in question generation for semi-supervised question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2495-2509, Hong Kong, China. Association for Computational Linguistics.
Sanqiang Zhao, Rui Meng, Daqing He, Andi Saptono, and Bambang Parmanto. 2018a. Integrating transformer and paraphrase rules for sentence simplification. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3164-3173, Brussels, Belgium. Association for Computational Linguistics.
Yao Zhao, Xiaochuan Ni, Yuanyuan Ding, and Qifa Ke. 2018b. Paragraph-level neural question generation with maxout pointer and gated self-attention networks. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3901-3910, Brussels, Belgium. Association for Computational Linguistics.
Qingyu Zhou, Nan Yang, Furu Wei, Chuanqi Tan, Hangbo Bao, and Ming Zhou. 2017. Neural question generation from text: A preliminary study. In NLPCC.
Wenjie Zhou, Minghua Zhang, and Yunfang Wu. 2019a. Multi-task learning with language modeling for question generation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3394-3399, Hong Kong, China. Association for Computational Linguistics.
Wenjie Zhou, Minghua Zhang, and Yunfang Wu. 2019b. Question-type driven question generation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 6032-6037, Hong Kong, China. Association for Computational Linguistics.
6140