SlowGuess/ACL_2020_OCR huggingface dataset:
Figure 4: Here, "killer" is the natural language description of "Brutus" in the MURDER.01 plesse
years and kept only those phrases which ranked in the top  $50\%$ .
3.8 Back-Translation
Back-translation has been used quite often in grammatical error correction (Xie et al., 2018) and is well known to translate noisy and ungrammatical sentences. Their cleaner high probability counterparts. We exploit this observation to clean questions with noisy and inconsistent PropBank descriptions like "wanter" (Figure 5). We use two state-of-the-art (SOTA) pre-trained transformer models transformer.wmt19.en-de and transformer.wmt19.de-en from Ott et al. (2019) trained on the English-German and German-English translation tasks of WMT 2019.
Figure 6 in the Appendix shows the output of all the five sets of templates applied together over one
Figure 5: Back translation and Implicature. Since the entant modality of "murder" is unsure, no questions are generated.
sentence (along-with implicature).
4 Evaluation and Results
Most of the prior QG studies have evaluated the performance of the generated questions using automatic evaluation metrics used in the machine translation literature. We use the traditional BLEU scores (Papineni et al., 2002) and compare the performance of Syn-QG on the SQuAD (Rajpurkar et al., 2016) test split created by Zhou et al. (2017). BLEU measures the average n-gram precision on a set of reference sentences. A question lexically and syntactically similar to a human question would score high on such n-gram metrics. Despite not utilizing any training data, Syn-QG performs better than the previous SOTA on two evaluation metrics BLEU-3 and BLEU-4 and close to SOTA on BLEU-1 and BLEU-2 (Table 4) at the time of submission. The high scores obtained without conducting any training arguably shed a little light on the predictable nature of the SQuAD dataset too.
Besides SRL, Dependency, and NER templates,
Architecture	BLEU-1	BLEU-2	BLEU-3	BLEU-4
PCFG-Trans (Heilman and Smith, 2010)	28.77	17.81	12.64	9.47
SeqCopyNet (Zhou et al., 2018)	13.02
NQG++ (Zhou et al., 2017)	42.36	26.33	18.46	13.51
MPQG (Song et al., 2017)	13.91
Answer-focused Position-aware model (Sun et al., 2018)	43.02	28.14	20.51	15.64
To the Point Context (Li et al., 2019)	44.40	29.48	21.54	16.37
s2sa-at-mp-gsa (Zhao et al., 2018)	44.51	29.07	21.06	15.82
ASs2s (Kim et al., 2019)	16.17
CGC-QG (Liu et al., 2019)	46.58	30.9	22.82	17.55
Capturing Greater Context (Tuan et al., 2019)	46.60	31.94	23.44	17.76
Natural QG with RL based Graph-to-Sequence (Chen et al., 2019)	-	-	-	17.94
RefineNet (Nema et al., 2019)	47.27	31.88	23.65	18.16
QPP&QAP (Zhang and Bansal, 2019)	-	-	-	18.37
ACS-QG* (Liu et al., 2020)	52.30*	36.70*	28.00*	22.05
UNILM* (Wang et al., 2020)	-	-	-	24.32
ERNIE-GEN* (Xiao et al., 2020)	-	-	-	25.57
UNILMv2* (Bao et al., 2020)	-	-	-	26.30
ProphetNet* (Yan et al., 2020)	-	-	-	26.72*
Syn-QG	45.55	30.24	23.84	18.72
Table 4: Automatic Evaluation Results on SQuAD of different QG models. PCFG-TRA-S and Sy-OG are two rule-based models. *Work contemporaneous with or subsequent to the submission of this paper.
System	#Questions Generated	Avg. #Questions Per Sentence	Gammaticity	Relevance
H&S	381	3.81	3.40	4.23
NQG	100	1	8	3.28
QPP&QAP	—	—	3.9	4.03
Syn-QG	654	6.54	3.93	4.34
Table 5: Comparison of human evaluation with H&S (Heilmann and Smith, 2017), NQG (Du et al., 2017) and QPP&QAP (Zhang and Bansal, 2019)
System	Avg. novel unigrams	Avg. novel bigrams	Avg. novel trigrams
H&S	23.6	5.64	52.22
Syn-QG (w/o BT)	26.8	43.93	53.4
Syn-QG	39.34	64.08	76.24
SQUAD	42.86	74.2	86.35
Syn-QG (BT vs w/o-BT)	28.7	55.18	67.81
Table 6: The percentage of n-grams, the generated questions which are not present in the source sentence. The last row indicates the percentage of n-grams present in the non-backtranslated questions.
Syn-QG's questions also arise from Ver. 19's predicates and PropBank descriptions, which indeed by nature describe even not mentioned explicitly within the fact. In Fig. 2, the sentence with the event "and" results in a question with a stateful event of "or". Deducible questions like these have a good chance of having a distribution of n-grams quite different from the source sentences, possibly exposing the weakness of traditional n-gram metrics and rendering them less useful for a task like QG.
In order to have a complete and more reliable evaluation to gauge the system, we also carry out a human evaluation using two of the metrics used in QG-STEC Task B (Rus et al., 2012), namely grammaticality, and relevance which we define below. We compared the questions generated from our sys
tem against the constituency-based H&S (Heilman and Smith, 2009), a neural system NQG (Du et al., 2017) which does not depend on a separate answer extractor and QPP&QAP<sup>5</sup> (Zhang and Bansal, 2019) which has outperformed existing methods. We fed a total of 100 facts randomly picked from Wikipedia and 5 commercial domains (IT, Healthcare, Sports, Banking and Politics) combined, to each of the four systems. We then conducted a crowd-sourced evaluation over Amazon Mechanical Turk for the generated questions.
- Grammatical Correctness: Raters had to rate a question on how grammatically correct
Since the QPP&QAP model does not have a separate answer extractor, we use the answer spans computed from Syn-QG (412 in total after discarding overlaps).
it is or how syntactically fluent it is, disregarding its underlying meaning.
- Relevance Score: Raters had to give a score on how relevant the generated question is to the given fact. The relevance score helps us gauge whether the question should have been generated or not irrespective of its grammaticality.[6]
Each question was evaluated by three people scoring grammaticality and relevance on a 5 point Likert scale. The inter-rater agreement (Krippendorff's co-efficient) among human evaluations was 0.72. The instructions given to the Mturk raters are provided in the Appendix Figure 7. The results of the evaluation are shown in Table 5. Syn-QG generates a larger number of questions than H&S and performs strongly on grammaticality ratings. Syn-QG is also able to generate highly relevant questions without the use of a ranker. Also, rule-based approaches seem to be much better at generating relevant questions than neural ones.
QG-STEC also used variety and question types as their evaluation criteria and rewarded systems to generate questions meeting a range of specific question types. Syn-QG's questions cover each of those question types.
Since many times, despite the ability to paraphrase (Table 6), back-translated output tends to change the meaning of the original sentence, we also measured back-translation's impact on above QG metrics. We considered questions generated from 50 facts of Wikipedia measuring the grammaticality and relevance before and after back-translation. While grammaticality increased from 3.54 to 4.11, questions' relevance fell a bit from 3.96 to 3.88. This observation is along with the performance of QP & CP show in Table 4, accentuates that when new models are learning syntactic structures well, there is still some progress to be made to generate relevant questions.
We introduced Syn-QG, a set of broad coverage rules leveraging event-based and sub-event based sentence views along with verb-specific argument descriptions. Automatic and manual evaluations
show that Syn-QG is able to generate a large number of diverse and highly relevant questions with better fluency. Verb-focused rules help approach long-distance dependencies and reduce the need for explicit sentence simplification by breaking down a sentence into clauses while custom rules like implications serve a purpose similar to a re-ranker to discard irrelevant questions but with increased determinism. While our work focuses on sentence-level QG, it would be interesting to see how questions generated from VerbNet predicates would have an impact on multi-sentence or passage level QG, where the verb-agnostic states of the participants would change as a function of multiple verbs. The larger goal of QG is currently far from being solved. Understanding abstract representations, leveraging world knowledge, are reasoning about them is crucial. However, we believe that with an extensible and transparent architecture, it is very much possible to keep improving the system continuously in order to achieve this larger goal.
We thank the three anonymous reviewers for their helpful comments and invaluable suggestions. We also thank the members of Amelia Science, RnD IPson, - Manjunath Hegde, Anant Khandel, Ashish Shrivastava for their work in QG and especially Viswa Teja Ravi, for helping in replicating Mazidi and Tarau (2016)'s work. We also thank Uday Chinta and IPsoft, India, for supporting and providing access to Amazon Mechanical Turk.
Hangbo Bao, Li Dong, Furu Wei, Wenhui Wang, Nan Yang, Xiaodong Liu, Yu Wang, Songhao Piao, Jianfeng Gao, Ming Zhou, et al. 2020. Unilmv2: Pseudo-masked language models for unified language model pre-training. arXiv preprint arXiv:2002.12804.
Susan Windisch Brown, Julia Bonn, James Gung, Annie Zaenen, James Pustejovsky, and Martha Palmer. 2019. VerbNet representations: Subevent semantics for transfer verbs. In Proceedings of the First International Workshop on Designing Meaning Representations, pages 154-163.
Yu Chen, Lingfei Wu, and Mohammed J Zaki. 2019. Natural question generation with reinforcement learning based graph-to-sequence model. arXiv preprint arXiv:1910.08832.
Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming
In cases when the grammaticality is extremely low like 1 or 2, the relevance score will also tend to be low. Otherwise, we assume that minor grammatical variations can be ignored while gauging relevance.
Zhou, and Hsiao-Wuen Hon. 2019. Unified language model pre-training for natural language understanding and generation. arXiv preprint arXiv:1905.03197.
Xinya Du, Junru Shao, and Claire Cardie. 2017. Learning to ask: Neural question generation for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1342-1352.
Michael Flor and Brian Riordan. 2018. A semantic role-based approach to open-domain automatic question generation. In Proceedings of the Thirteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 254-263.
Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson Liu, Matthew Peters, Michael Schmitz, and Luke Zettlemoyer. 2018. AllenNLP: A deep semantic natural language processing platform. arXiv preprint arXiv:1803.07640.
Michael Heilman and Noah A Smith. 2009. Question generation via overgenerating transformations and ranking. Technical Report CMU-LTI-09-013, Language Technologies Institute, Carnegie Mellon University.
Michael Heilman and Noah A Smith. 2010. Good question! Statistical ranking for question generation. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 609-617. Association for Computational Linguistics.
Tom Hosking and Sebastian Riedel. 2019. *Evaluation rewards for question generation in models: Proceedings of the North American Chapter of the Association on Computational Linguistics*.
Lauri Karttunen. 2017. Simie and phasal implicatives. In Proceedings of the First Joint Conference on Lexical and Computational Semantics-Volume 1: Proceedings of the main conference and the shared task, and volume 1: Proceedings of the Sixth International Workshop on Semantic Evaluation, pages 124-131. Association for Computational Linguistics.
Yanghoon Kim, Hwanhee Lee, Joongbo Shin, and Kyomin Jung. 2019. Improving neural question generation using answer separation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 6602-6609.
Ghader Kurdi, Jared Leo, Biban Parsia, Uli Sattler, and Salam Al-Emari. 2020. A systematic review of automatic question generation for educational purposes. International Journal of Artificial Intelligence in Education, 30(1):121-204.
Igor Labutov, Sumit Basu, and Lucy Vanderwende. 2015. Deep questions without deep understanding. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), volume 1, pages 889-898.
Roger Levy and Galen Andrew. 2006. Tregex and tsurgeon: tools for querying and manipulating tree data structures. In LREC, pages 2231-2234. Citeseer.
Jingjing Li, Yifan Gao, Lidong Bing, Irwin King, and Michael R Lyu. 2019. Improving question generation with to the point context. arXiv preprint arXiv:1910.06036.
Yuri Lin, Jean-Baptiste Michel, Erez Lieberman Aiden, Jon Orwant, Will Brockman, and Slav Petrov. 2012. Syntactic annotations for the C++ books ngram corpus. In Proceedings of the 2012 system demonstrations, pages 9-174. Association for Computational Linguistics.
David Lindberg, Fred Popovich, John Jesbit, and Phil Winne. 2013. Generating natural language questions to superlearn on-line. In Proceedings of the 14th European Workshop on Natural Language Generation, pages 105-114.
Bang Liu, Haojie Di Niu, Haolan Chen, and Lincheng He. 2023. Asking questions the human eye: Scalability question-answer generation from text analysis. In Proceedings of The Web Conference 2023, https://doi.org/10.1007/s10074-023-0068-z.
Liu, Mingjun Zhao, Di Niu, Kunfeng Lai, Yacheng He, Haojie Wei, and Yu Xu. 2019. Learning to generate questions by learning what not to generate. arXiv preprint arXiv:1902.10418.
Marie-Catherine de Marneffe, Timothy Dozat, Natalia Silveira, Katri Haverinen, Filip Ginter, Joakim Nivre, and Christopher D. Manning. 2014. Universal Stanford dependencies: A cross-linguistic typology. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC-2014), pages 4585-4592, Reykjavik, Iceland. European Languages Resources Association (ELRA).
Karen Mazidi and Rodney D Nielsen. 2014. Linguistic considerations in automatic question generation. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), volume 2, pages 321-326.
Karen Mazidi and Rodney D Nielsen. 2015. Leveraging multiple views of text for automatic question generation. In International Conference on Artificial Intelligence in Education, pages 257-266. Springer.
Karen Mazidi and Paul Tarau. 2016. Infusing NLU into automatic question generation. In Proceedings of the 9th International Natural Language Generation conference, pages 51-60.
George A Miller. 1998. WordNet: An electronic lexical database. MIT press.
Jack Mostow and Wei Chen. 2009. Generating instruction automatically for the reading strategy of self-questioning. In AIED, pages 465-472.
Preksha Nema, Akash Kumar Mohankumar, Mitesh M Khapra, Balaji Vasan Srinivasan, and Balaraman Ravindran. 2019. Let's ask again: Refine network for automatic question generation. arXiv preprint arXiv:1909.05355.
Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. MS MARCO: A human generated machine reading comprehension dataset. arXiv preprint arXiv:1611.09268.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. arXiv preprint arXiv:1904.01038.
Martha Palmer. 2009. Semlink: Linking propbank, verbnet and framenet. In Proceedings of the generative lexicon conference, pages 9-15. GenLex-09, Pisa, Italy.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pages 311-318. Association for Computational Linguistics.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000 questions for machine comprehension of text. _Viv print arXiv:1606.05250_.
Vasile Rus, Brendan Wyse, Paul Lukek, Mihai Llanan, Svetlana Stoyanchev, and Christian Moldovan. 2012. A detailed account of the first question generation shared task evaluation challenge. *Dialgue & Discourse*, 3(2):177-2.
Karin Kipper Soder. 20 VerbNet: A broad-coverage, comprehensive lexicon. Ph.D. thesis, University of Pennsylvania
Linfeng Song, Zuo Wang, and Wael Hamza. 2017. A unified query-based generative model for question generation and question answering. arXiv preprint arXiv:1709.01058.
Xingwu Sun, Jing Liu, Yajuan Lyu, Wei He, Yanjun Ma, and Shi Wang. 2018. Answer-focused and position-aware neural question generation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3930-3939.
Liling Tan. 2014. Pywsd: Python implementations of word sense disambiguation (wsd) technologies [software]. https://github.com/alvations/pywsd.
Duyu Tang, Nan Duan, Tao Qin, Zhao Yan, and Ming Zhou. 2017. Question answering and question generation as dual tasks. arXiv preprint arXiv:1706.02027.
Luu Anh Tuan, Darsh J Shah, and Regina Barzilay. 2019. Capturing greater context for question generation. arXiv preprint arXiv:1910.10274.
Tong Wang, Xingdi Yuan, and Adam Trischler. 2017. A joint model for question answering and question generation. arXiv preprint arXiv:1706.01450.
Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. 2020. Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. arXiv preprint arXiv:2002.10957.
Dongling Xiao, Han Zhang, Yukui Yu Sun, Hao Tian, Hua Wu, and Haifeng Wang. 2017 Ernie-gen: An enhanced multi-flow premaining an fine-tuning framework for natural language generation. arXiv preprint arXiv:2001.11314.
Ziang Xie, Guillaume Piaulat, Stanley Xie, Andrew Y Ng, and Dan Jurafsky. 2018. Nosing and denoising natural language: Diversacktranslation for grammar correction. In Proceedings of the 2018 Conference of the Nor American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 9-628.
Yu Yan, Weizhen Qi, Yeyun Gong, Dayiheng Liu, Jian Duan, Jiusheng Chen, Ruofei Zhang, and Ming Zhou. 2020. Prophetnet: Predicting future n-gram for sequence-to-sequence pre-training. arXiv preprint arXiv:2001.04063.
Xuchen Yao and Yi Zhang. 2010. Question generation with minimal recursion semantics. In Proceedings of QG2010: The Third Workshop on Question Generation, pages 68-75. CiteSeer.
Xingdi Yuan, Tong Wang, Caglar Gulcehre, Alessandro Sordoni, Philip Bachman, Saizheng Zhang, Sandeep Subramanian, and Adam Trischler. 2017. Machine comprehension by text-to-text neural question generation. In Proceedings of the 2nd Workshop on Representation Learning for NLP, pages 15-25, Vancouver, Canada. Association for Computational Linguistics.
Shiyue Zhang and Mohit Bansal. 2019. Addressing semantic drift in question generation for semi-supervised question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2495-2509.
Yao Zhao, Xiaochuan Ni, Yuanyuan Ding, and Qifa Ke. 2018. Paragraph-level neural question generation with maxout pointer and gated self-attention networks. In Proceedings of the 2018 Conference on
Empirical Methods in Natural Language Processing, pages 3901-3910.
Qingyu Zhou, Nan Yang, Furu Wei, Chuanqi Tan, Hangbo Bao, and Ming Zhou. 2017. Neural question generation from text: A preliminary study. In National CCF Conference on Natural Language Processing and Chinese Computing, pages 662-671. Springer.
Qingyu Zhou, Nan Yang, Furu Wei, and Ming Zhou. 2018. Sequential copying networks. In Thirty-Second AAAI Conference on Artificial Intelligence.
Potential Short Answer (Named Entities)	Question Template	Sample Fact	Generated Question
Location	Where mainAux subj otherAux verb obj modifiers?	The event was organized at Times Square.	Where was the event organized?
Person	Who mainAux subj otherAux verb obj modifiers? Whom mainAux obj otherAux verb modifiers	WestWorld brought back the life of the roboticist Craig Smith.	Whom did WestWorld bring back the life of?
Date	When mainAux subj otherAux verb obj modifiers?	Donald Trump won the elections in the year 2016	When did Donald Trump win the elections?
Number	How many mainAux subj otherAux verb obj modifiers?	A thousand will not be enough for the event.	How many will not be enough for the event?
Phone Number	At what number mainAux subj otherAux verb obj modifiers?	The pizza guy can be reached at +91-748-728-781	At what time the number can the pizza guy be reached?
Duration	For how long mainAux subj otherAux verb obj modifiers?	Lauren would be staying in the ! for around 10 minutes	How long would Lauren be staying at the hut?
Organization	Which organization mainAux subj otherAux verb obj modifiers?	Deepak joined the ! firm, the Unite Nons.	Which organization did Deepak join?
Table 7: SRL arguments which contain a named entity are further elaborated as a short answer "for around 10 minutes" rather than only the named entity span "10 minutes". SRL arguments are highlighted in blue.
Figure 6: Questions generated by each set of heuristics for one sentence which are further sent for back-translation.
Figure 7: The MTURK template uses for collecting responses for measuring question relevance and grammatical- ity.