SlowGuess/ACL_2020_OCR huggingface dataset:
Kaustubh D. Dhole
Amelia Science
RnD, IPsoft
New York, NY 10004
kdhole@ipsoft.com
Christopher D. Manning
manning@stanford.edu
Question Generation (QG) is fundamentally a simple syntactic transformation; however, many aspects of semantics influence what questions are good to form. We implement this observation by developing Syn-QG, a set of transparent syntactic rules leveraging universal dependencies, shallow semantic parsing, lexical resources, and custom rules which transform declarative sentences into question-answer pairs. We utilize PropBank argument descriptions and VerbNet state predicates to incorporate shallow semantic content, which helps generate questions of a descriptive nature and produce inferential and semantically richer questions than existing systems. In order to improve syntactic fluency and eliminate grammatically incorrect questions, we employ back-translation over the output of these syntactic rules. A set of crowd-sourced observations shows that our system can generate a larger number of highly grammatic and relevant questions than previous QG systems and that back-translation drastically improves grammaticality at a slight cost of generating irrelevant questions.
Automatic Question Generation (QG) is the task of generating queries that answer pairs from a declarative sentence. It has direct use in education and generating engagement, where a system automatically generates questions about passages that someone has read. A more recent secondary use is for automatic generation of questions as a data augmentation approach for training Question Answering (QA) systems. QG was initially approached by syntactic rules for question-generation, followed by some form of statistical ranking of goodness, e.g., (Heilman and Smith, 2009, 2010). In recent years, as in most areas of NLP, the dominant approach has been neural network generation (Du et al., 2017),
Figure 1: The S/L structure is leveraged to invoke a template, and simple re-configuration of the modifying arguments is performed.
in particular using a sequence-to-sequence architecture which exploits the data in the rapidly growing number of large QA data sets.
Previous rule-based approaches suffer from a significant lack of variety in the questions they generate, sticking to a few simple and reliable syntactic transformation patterns. Neural architectures provide a pathway to solving this limitation since they can exploit QA datasets to learn the broad array of human question types, providing the usual neural network advantages of a data-exploiting, end-to-end trainable architecture. Nevertheless, we observe that the quality of current neural QG systems is still lacking: The generated questions lack syntactic fluency, and the models lack transparency and an easy way to improve them.
We argue that in essence QG can be governed by simple syntactic "question transformations" - while the implementation details vary, this is in accord with all major linguistic viewpoints, such as Construction Grammar and Chomskyan Generative Grammar, which emphasize grammatical rules and the existence of finite ways to create novel utterances. However, successful, fluent question generation requires more than just understanding syntactic question transformations, since felicitous questions must also observe various semantic and
This paper was retracted. For more information, see https://aclanthology.org/2020.acl-main.69.
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 752-765 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
pragmatic constraints. We approach these by making use of semantic role labelers (SRL), previously unexploited linguistic semantic resources like VerbNet's predicates (Figure 2) and PropBank's rolesets and custom rules like implications, allowing us to generate a broader range of questions of a descriptive and inferential nature. A simple transformation commonly used in rule-based QG is also displayed in Figure 1.
Figure 2: VerbNet Predicate Question Generation. Detailed intermediate steps are described in Figure 3.
We evaluate our QG framework, Syn-QG against three QG systems on a mixture of Wikipedia and commercial text sentences outperforming existing approaches in grammaticality and relevance in a crowd-sourced human evaluation while simultaneously generating more types of question. We also notice that back-translated questions are grammatically superior but are sometimes slightly irrelevant as compared to their original counterparts. The Java code is publicly available at https://bitbucket.org/kaustubhdb/te SYN-
With the advent of large-scale QA datasets (Rajpurkar et al., 2016; Niyon et al., 2016), recent work in QG (Duan et al., 2017; Zhou et al., 2017) has primarily focused on training sequence-to-sequence and sentence-based architectures. Dong et al. (2019) fine-tuned the question generation task by taking advantage of a large pre-trained language model. Success in reinforcement learning has inspired teacher-student frameworks (Wang et al., 2017; Tang et al., 2017) treating QA and QG as complementary tasks and performing joint training by using results from QA as rewards for the QG task. Yuan et al. (2017); Hosking and Riedel (2019); Zhang and Bansal (2019) used evaluation metrics like BLEU, sentence perplexity, and QA probability as rewards for dealing with exposure bias.
Chen et al. (2019) trained a reinforcement learning based graph-to-sequence architecture by embedding the passage via a novel gated bi-directional graph neural network and generating the question via a recurrent neural network. To estimate the positions of copied words, Liu et al. (2019) used a graph convolution network and convolved over the nodes of the dependency parse of the passage. Li et al. (2019) jointly modeled OpenIE relations along with the passage using a gated-attention mechanism and a dual copy mechanism.
Traditionally, question generation has been tackled by numerous rule-based approaches (Heilman and Smith, 2009; Mostow and Chen, 2009; Yao and Zhang, 2010; Lindberg et al., 2013; Labutov et al., 2015). Heilman and Smith (2009, 2010) introduced an overgenerate rank approach that generated multiple questions in a rule-based tree transformations of the constitutive parse of a declarative sentence and then linked them using a logistic-regression rank with manually designed features. Yao and Zhang (2010) described transformations of Minna and Recursion Semantics representations guaranteeing grammaticality. Other transformations have been in the past defined in terms of ten notes (Mazidi and Nielsen, 2014, 2015; Mazidi and Ito, 2016; Flor and Riordan, 2018), or experimentally performed (Heilman and Smith, 2009) by searching tree patterns via Tregex, followed by their manipulation using Tsurgeon (Levy and Andrew, 2006). Kurdi et al. (2020) provide a comprehensive summary of QG, analysing and comparing approaches before and after 2014.
Vis-à-vis current neural question generators, rule-based architectures are highly transparent, easily extensible, and generate well-formed questions since they perform clearly defined syntactic transformations like subject-auxiliary inversion and WH movement over parse structures whilst leveraging fundamental NLP annotations like named entities, co-reference, temporal entities, etc.
However, most of the existing rule-based systems have lacked diversity, being mostly focused on generating What-type and boolean questions and have mainly exploited parse structures which are not semantically informed. Mazidi and Tarau (2016); Flor and Riordan (2018) use Dependency, SRL, and NER templates but do not handle modalities and negation in a robust manner. Moreover, there is plenty of availability of core linguistic resources like VerbNet and PropBank, which provide
further unique ways to look at sentences and ask questions differently besides the generally well-established dependency and SRL parses.
3 Syn-QG
Syn-QG is a rule-based framework which generates questions by identifying potential short answers in 1) the nodes of crucial dependency relations 2) the modifying arguments of each predicate in the form of semantic roles 3) named entities and other generic entities 4) the states of VerbNet's thematic roles in the form of semantic predicates and 5) PropBank roleset specific natural language descriptions. Each of the five heuristics works independently, generating a combined set of question-answer pairs, which are eventually back-translated. We describe each of these five sources.
3.1 Dependency Heuristics
Dependency trees are syntactic tree structures, wherein syntactic units in the form of words are connected via directed links. The finite verb is considered as the structural root of the tree, and all other syntactic units are either directly (nsubj, doj etc.) or indirectly (xcomp, iobj, etc.) dependent of this finite verb.
We present rules over such dependency units annotated according to the Universal Dependents (UD) format (de Marneffe et al. 2014). An extract dependency structures, we use a parser on Gardner et al. (2018).
We make use of Pro Bank's predica- argument structure (SRL) for visual extraction of the verb headed by a select fewpendency nodes which can serve as answers. These labels treat the clause as a combination of a subject, an object, the head verb and other non-verb arguments. The clause is further refined with modal auxiliaries and negations if found around the verb. Finally, we make use of a set of predefined handwritten templates, a few of which are described in Table 1.
In each of the templates, we convert What to Who/Whom, When or Where depending on the named entity of the potential answer and do to does or did according to the tense and number of the subject to ensure subject-verb agreement. The pseudo code is described in Algorithm 2 of the Appendix.
3.2 SRL Heuristics
While dependency representations are perhaps the most popular syntactic method for automatically extracting relationships between words, they lack sufficient semantic detail. Being able to answer "Who did what to whom and how, why, when and where" has been a central focus in understanding language. In recent decades, shallow semantic parsing has been a prominent choice in understanding these relationships and has been extensively used in question generation (Mazidi and Tarau, 2016; Flor and Riordan, 2018).
PropBank-style frames provide semantically motivated roles that arguments around a verb play. Moreover, highly accurate semantic role labeling models are being developed owing to corpora like PropBank and FrameNet. We take advantage of the SRL model of Gardner et al. (2018) for extracting the roles of each verb in the sentence.
Algorithm 1 SKHeuristics
$$
\begin{array}{l} \{S R L \dots R L _ {s} \} \leftarrow , R L (w _ {0} \dots w _ {n}) \\ l o o r \dot {=} 0, u n _ {1} \dot {=} s: \\ i f S R L _ {j} \text {c o n t a i n} A _ {0} \text {o r} A _ {1} \text {a n d a t l e a s t} 1 A _ {m} \\ \mathbf {\Pi} ^ {\mathrm {e n}} \\ \left\{A _ {0} \dots A _ {C A U}, A _ {T M P} \right\} \leftarrow S R L _ {j} \\ l o o _ {P} \quad_ {x} \in S R L _ {j} i f A _ {x} = m o d i f i e r: \\ s u b j \leftarrow A _ {0} \\ A _ {x} ^ {-} \leftarrow \sum \left(A _ {3}, A _ {4}, \dots A _ {T M P} - A _ {x}\right) \\ v e r b \leftarrow \{A _ {v}, m o d a l s, n e g a t i o n \} \\ t e m p l a t e \leftarrow m o d i f i e r _ {t y p e} \leftarrow A _ {x} \\ Q A \leftarrow t e m p l a t e (s u b j, A _ {x}, v e r b, A _ {x} ^ {-}) \\ \begin{array}{c} \text {c l o s e ;} \end{array} \\ \end{array}
$$
We succinctly describe the steps taken in Algorithm 1. We first filter out all the predicates which have an Agent or a Patient and at least one other modifier like Extent, Manner, Direction, etc. These modifiers would serve as our short answers. We make use of a set of predefined handwritten templates described in Table 2, which rearrange the arguments within the fact to convert it into an interrogative statement depending on the modifier.
In Figure 1, the predicate "won" is modified by a Patient "New Mexico", an Agent "Obama", an Extent modifier "by a margin of  $5\%$  and a Temporal modifier "in 2008". For Extent as a short answer, we fill a pre-defined template "By how much mainAux nsubj otherAux verb obj modifiers?" to get the above question-answer pair. We keep the order of arguments as they appear in the original
Potential Short Answer (Dependencies)	Question Template	Sample Fact	Generated Question
subject (nsubj)	Wh mainAux otherAux verb obj modifiers?	Ricky Ponting accepted captainacy during Australia's golden era.	Who accepted captainacy during Australia's golden era?
direct object(dobj)	Wh mainAux nsubj otherAux verb modifiers?	In monsoon, India receives large amounts of rain that can cause flooding.	What does India receive in monsoon?
open clausal complement (xcomp)	Wh mainAux nsubj modifiers?	The Sheriff did not try to eat the apples while the outlaws were fasting.	What did the Sheriff not try while the outlaws were fasting?
copula (cop)	How would you describe nsubj?	Comets are leftovers from the creation of our solar system about 4.5 billion years ago.	How would you describe comets?
Table 1: A few templates to describe the construction of questions. Different word units are shown in unique colors to describe the filling of the template. All the short answers are highlighted in blue.
sentence. The templates are described in Table 2.
3.3 Named Entities, Custom Entities, and Hypernyms
We create separate templates when any numbered SRL argument contains common named entities like Person, Location, Organization etc. Like Flor and Riordan (2018), we add specific rules in the form of regexes to address special cases to differentiate between phrases like For how long an Till when instead of a generic When question type. Some of the templates are described in T 7 in the Appendix. The approach is described in Algorithm 3 in the Appendix.
We also use WordNet (Mille 1998) hypnms of all potential short answers are replace that with the bigram Which hypernym. So, for a sentence like "Hermione plays badminton at the venue", we generate a session "Which sport does Hermione play the venue?" For computing the hypernym, we use the sense disambiguation implementation of an. While supersenses do display a richer lexical variety, sense definitions don't always fit well.
3.4 Handling modals and auxiliaries
During explicit inversion of the verb and arguments around it via our templates, we tried to ensure that the positions of auxiliaries are set, and negations are correctly treated. We define a few simple rules to ensure that.
- When there are multiple auxiliaries, we only invert the first auxiliary while the second and
further auxiliaries remain, 'they' are just before the main verb.
- We make the question auxiliary finite and agree with the subject.
- We ensure that the object is kept immediately after the verb.
For passi cases, subj-verb-object is changed to verb by-subj.
Handling Factualness via Implicature
Previous rule-based approaches (Mazidi and Tarau, 2016; Flor and Riordan, 2018) have used the NEG dependency label to identify polarity. But such an approach would suffer whenever polarities would be hierarchically entailed from their parent clauses in cases like "Picard did not fail to X" where the entailed polarity of "X" is, in fact, positive. Moreover, in one-way implications like "Bojack hesitated to X", it would be best not to generate a question for unsure cases since it is open-ended if Bojack did or did not X. A similar example is displayed in Figure 5. For each verb representing a subordinate clause, we compute its entailed truth or falsity from its parent clause using the set of one-way and two-way implicative verbs, and verb-noun collocations provided by Karttunen (2012). For example, the two-way implicative construction "forget to X" entails that "X" did not happen, so it would be wrong to ask questions about "X". Karttunen (2012) provides simple implications in the form of 92 verbs and phrasal implications in the form of 9 sets of verbs and 8 sets of nouns making 1002 verb-noun collocations. The entailed polarity of a
Potential Short Answer (Verb Arguments)	Question Template	Sample Fact	Generated Question
Locative (LOC)	Where mainAux nsubj otherAux verb obj modifiers ?	Americans eat about 100 acres of pizza each day, with about 3 billion pizzas sold annually in the USA.	Where do about 3 billion pizzas sell annually ?
Manner (MNR)	How mainAux nsubj otherAux verb obj modifiers ?	Young Sheldon was caught unaware as the liquid was oozing out of the chamber in a zig-zag fashion.	How was the liquid oozing out of the chamber?
Purpose (PNC and PRP)	For what purpose mainAux nsubj otherAux verb obj modifiers ?	Collectively, South African women and children walk a daily distance equivalent to 16 trips to the moon and back to fetch water.	For what purpose do South African women and children walk a daily distance equivalent to 16 trips to the moon and back collectively ?
Cause (CAU)	Why mainAux nsubj otherAux verb obj modifiers ?	Since the average faucet releases 2 gallons of water per minute, you can save up to four gallons of water every morning by turning off the tap while you brush your teeth.	What can you save up to four gallons of water by turning off the tap while you brush your teeth every morning ?
Temporal (TMP)	When mainAux nsubj otherAux verb obj modifiers ?	Stephen Hawking once in June 2009 threw a party for time-travel but he announced the party the next day.	I did Stephen Hawking throw a party for time - travelers ?
Till when mainAux nsubj otherAux verb obj modifiers ?	Princess Sita traveled the whole town until the end of summer.	When did Stephen Hawking announce the party ?
Extent (EXT)	By how much mainAux nsubj otherAux verb obj modifiers ?	Mexico was won by Obama by a marg. of 5% in 2008.	Till when did Princess Sita travel the whole town ?
By how much was New Mexico won by Obama in 2008?
Table 2: The templates of temp. direction, etc., etc. are leveraged to ask questions about different modifiers. Answer fragments are highlighted in blue. In passive cases like the last example, we change the template order from subj-verb-object to obj-by-subj.
clause can be either TRUE or FALSE. For FALSE clauses, we often generate a boolean question with a NO answer. For UNSURE clauses, we do not generate any question. For TRUE clauses and verbs and conjunctions not present in the above set, we rely on the NEG label.
3.6 VerbNet Predicate Templates
While SRL's event-based representations have permitted us to generate questions that talk about the roles participants of an event play, we exploit VerbNet's sub-event representation to ask questions on
how participants' states change across the time frame of the event. In Figure 2, the event murder (VerbNet class murder-42.1) results in a final state in which the participant Julius Caesar is in a not-alive state.
Each class in VerbNet (Schuler, 2005; Brown et al., 2019) includes a set of member verbs, the thematic roles used in the predicate-argument structure, accompanied with flat syntactic patterns and their corresponding semantic predicates represented in neo-Davidsonian first-order-logic formulation. These semantic predicates bring forth a temporal sequencing of sub-events tracking how participants' states change over the course of the event. The advantage is to be able to ask questions
<sup>1</sup>Unsure clauses appear in one-way implicatives when it's unclear if the clause is true or false under either an affirmative or a negative parent clause.
Figure 3: VerbNet Predicate Question Generation. All the predicates of the two sub-events  $e_4$  and  $e_5$  (HAS_POSSESSION) would be considered since  $e_3$  possesses a process-oriented predicate TRANSFER. COST is the predicate of the main event E.
bearing a surface form different from the source sentence but which are driven by reasoning rather than just being paraphrastic. For example, in the sentence, "Brutus murdered Julius Caesar", the event murder-42.1 entails a final state of "death" on the Patient participant not being alive at the end of the event. So, we construct a template "ma Aux the Patient otherAux not alive?" Similarly, event pay-68-1 results in a final state in with the Recipient "Perry" has possession of "$100" and the Agent "John" has possession of "un car", against which we define the templates as shown in Figure 3.
We formulate a set of questions: boolean type and which-type questions asking specifically about these states. We create templates to VerbNet's stateful predicates like as_location, has Possession, has information, eem, has_state, cost, desire, harmed, has organization role, together, social interaction, authority relationship, etc. which are present in  $64.4\%$  of the member verbs in VerbNet<sup>2</sup>. We outline a few of the templates in Table 3.
During inference time, we first compute the Verb-Net sense, the associated thematic role mapping,
and syntactic frame (along with the predicates) with the help of Brown et al. (2019)'s parser. VerbNet's predicates are governed by the sub-events in which they occur. Although VerbNet's representation lays out a sequence of sub-events, no sub-event is explicitly mentioned as the final one<sup>3</sup>. We choose all the predicates of those sub-events which are preceded by other sub-events which possess at least one process-oriented predicate.<sup>4</sup>
3.7 PropBank Argument Descriptions
PropBank rolesets' course-grained annotation of verb-specific argument definitions ("killer", "payer", etc.) to represent semantic roles offers robustly specific natural language descriptions to ask questions about the exact roles participants play. Nonetheless, not all descriptions are suitable to be utilized directly in rigid templates. So we incorporate back-translation to 1) generate a grammatical errors propagation from incorrect parsing and template restrictions, and (2) eliminate rarely used Prop-Bank descriptions that generate highly probable questions.
While previous work in rule-based QG has used SRL templates and WordNet senses to describe the roles arguments around a verb play, previous SRL templates have always been verb-agnostic, we believe there is a great deal of potential in PropBank descriptions. Moreover, WordNet supersenses do not always give rise to acceptable questions. On manual evaluation, question relevance decreased after incorporating templates with WordNet supersenses. Instead, we make use of PropBank's verb-specific natural language argument descriptions to create an additional set of templates. VerbNet senses have a one-to-one mapping with PropBank rolesets via the SemLink project (Palmer, 2009). We hence make use of Brown et al. (2019)'s parser to find the appropriate PropBank roleset for a sentence.
However, we observed that a lot of PropBank descriptions were noisy and made use of phrases which would be unarguably rare in ordinary parlance like "breather" or "truster". To eliminate such descriptions, we computed the mean Google N-gram probabilities (Lin et al., 2012) of all the PropBank phrases in the timespan of the last 100
3 or a sub-event, which is an outcome of a process
4Out of 174 VerbNet predicates, we manually categorize 84 predicates like HAS_LOCATION, HAS POSSESSION as stateful predicates and the remaining ones like DESCRIBE, TRANSFER, etc. as process-oriented predicates.
${}^{2}$  Out of 4854 member verbs,there are 3128 members whose syntactic frame contains at least one of these predicates.
Triggering Predicate and Thematic Arguments	Question Template	Sample Fact & VerbNet Predicate	Generated Question
HAS_POSSESSION (Asset,Recipient)	Who has Asset ? Recipient	Robert paid $100 to Mary for the cycle. HAS_POSSESSION(Mary,$100)	Who has $100 ? Mary
HARMED (Patient)	What is harmed ? Patient	The terrorists bombed the building. HARMED(the building)	What is harmed ? the building
NOT_ALIVE (Patient)	Is Patient alive ? No.	According to epics, Vishnu killed the demon Kaitabh. NOT_ALIVE (the demon Kaitabh)	Is the demon Kaitabh alive ? No.
Table 3: VerbNet predicate templates (simplified) along with sample questions with the thematic roles highlighted. A question is created from the concept of "being alive" which is not synonymous with but is an outcome of "killing".