Patent Publication Number: US-11023684-B1

Title: Systems and methods for automatic generation of questions from text

Description:
This application claims priority to U.S. Provisional Application No. 62/644,766 filed on Mar. 19, 2018, entitled “System for Automatic Generation of Questions from Text;” U.S. Provisional Application No. 62/645,474 filed on Mar. 20, 2018, entitled “System for Automatic Generation of Questions from Text;” and U.S. Provisional Application No. 62/648,464 filed on Mar. 27, 2018, entitled “Semantic Role-Based Approach to Open-Domain Automatic Question Generation,” the entireties of which are herein incorporated by reference. 
    
    
     FIELD 
     The technology described herein relates generally to question generation and more specifically to automatically generating questions from text using a semantic role-based approach. 
     BACKGROUND 
     It is often desirable to formulate questions about an excerpt of text, especially in the educational sector when assessing a student&#39;s reading or language skills. But, manual generation of questions from text is a time-consuming and inefficient task. As such, automatic question generation (AQG) is an important and challenging research area in natural language processing. AQG systems can have particularly beneficial education applications, such as reading comprehension assessments, intelligent tutoring, dialogue agents, and instructional games. 
     SUMMARY 
     In accordance with the teachings herein, computer-implemented systems and methods are provided for automatically generating questions from text. An embodiment of a method for automatically generating one or more questions from text comprises receiving text including one or more sentences and parsing a sentence from the text, wherein the sentence comprises a predicate and one or more arguments associated with the predicate. The method further comprises assigning semantic role labels to the one or more arguments associated with the predicate and automatically generating one or more questions relating to the predicate. Questions generation is based on the assigned semantic role labels, wherein each answer to the one or more questions is one of the one or more arguments associated with the predicate. 
     A computer-implemented system for automatically generating one or more questions from text is further described herein, wherein the system comprises one or more data processors and one or more computer-readable storage mediums encoded with instructions for commanding the one or more data processors to execute steps that include the aforementioned method. A non-transitory computer-readable storage medium is further described herein, wherein the storage medium comprises instructions for which when executed cause a processing system to execute steps comprising the aforementioned method. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG. 1  is a block diagram depicting the generation of questions from text in a computer-implemented environment. 
         FIG. 2  is a block diagram depicting the processing of text to generate questions in a computer-implemented environment. 
         FIG. 3  is a flow diagram depicting an example method for generating questions for a single predicate. 
         FIG. 4  is a flow diagram depicting an example method for generating questions for one or more predicates and one or more sentences. 
         FIG. 5  depicts results of a comparison study between two automatic question generation systems. 
         FIG. 6  depicts a computer-implemented environment wherein users can interact with an automated question generation system. 
         FIGS. 7A, 7B, and 7C  depict example systems for use in implementing an automated question generation system. 
     
    
    
     DETAILED DESCRIPTION 
     Computer-implemented systems and methods are provided herein for automatically generating questions from text. The systems and methods utilize a semantic role-based approach to open-domain automatic question generation. 
       FIG. 1  depicts a computer-implemented environment  100  for question generation via an automated question generation (AQG) engine  102 . In embodiments, text  104  is input into the AQG engine  102 . Text  104  can be one sentence, more than one sentence, an excerpt from an entire body of text, an entire body of text, etc. The AQG engine  102  then automatically outputs one or more generated questions  106  corresponding to the inputted text  104 . Generated questions  106  are typically either constituent questions (i.e., wh-questions: who, what, when, where, why) or yes-or-no questions. For example, if the sentence “John is a painter” is input into the AQG engine  102 , the output may include “What is John?” (i.e., wh-question) and/or “Is John a painter?” (i.e., yes-or-no question). Embodiments of the AQG engine can be used to generate questions from reading passages to gauge a student&#39;s reading comprehension, to conduct formative assessments, to test language development, to engage in intelligent tutoring, or to perform various other evaluations. In embodiments, the AQG engine can formulate questions for all types of sentences, including copular sentences (e.g., sentences with a “linking verb,” such as forms of be, become, feel, or seem). 
       FIG. 2  depicts data flow through an embodiment of an AQG engine  202  in a computer-implemented environment  200 . In the depicted embodiment, one or more sentences are introduced into the AQG engine  202  as text  204 . Text  204  is received by a parsing unit  206 . In embodiments where text  204  comprises more than one sentence, parsing unit  206  extracts a single sentence from text  204  for initial processing. The single sentence then passes to the semantic role labeling (SRL) system  208 . 
     SRL system  208  isolates a predicate (e.g., the part of a sentence that contains the verb and gives information about the subject) of the sentence and assigns labels to the arguments associated with that predicate. At its basics, SRL analyzes a sentence into who did what to whom, how and when . . . . In one embodiment, all non-auxiliary verbs in the sentence are considered predicates. Semantic roles include the generalized core arguments of verbs, i.e., A0, A1, . . . , A5, and a set of adjunct modifiers that includes AM-TMP (time), AM-LOC (location), AM-DIR (direction), AM-MNR (manner), AM-CAU (cause), AM-PNC (purpose), AM-EXT (extent), AM-DIS (discourse), AM-ADV (adverbial), AM-NEG (negation), and AM-MOD (modal verbs). An example of an SRL system is the SENNA system for Semantic Role Labeling, which produces semantic role labels according to the Propbank 1.0 specifications. It is understood that any SRL system can be utilized, and the aforementioned labels may vary accordingly. 
     In some embodiments, it is necessary to double check the SRL system  208  for errors in labeling before question generation. For instance, some SRL systems assign the A1 role to subjects instead of A0. Using the previous example “John is a painter,” an SRL system may incorrectly produce: [ A1  John] is [ A1  a painter]. Because A1 is the label in this particular embodiment reserved for direct objects, it is necessary to remap [ A1  John] to a category treated as a grammatical subject for questions generation. This type of post-processing can be completed by the system itself or aided by an external source, such as an automated correction system. 
     When SRL system  208  has labeled each argument associated with a predicate, the labeled arguments  211  and predicate  209  are passed to the questions generator  210 . In embodiments, the parsed sentence may contain more than one predicate. In such instances, SRL system  208  isolates the next predicate and labels its associated arguments, repeating the process until all predicates have been isolated and their respective arguments labeled. 
     Question generator  210  processes each predicate  209  and labeled argument  211  to formulate questions  212 . To produce a question  212 , question generator  210  selects a focal argument—the argument about which the question will be asked. The text of the chosen focal argument then becomes the expected answer to the question. In the aforementioned example where “John is a painter,” question generator  210  may select the focal argument: [ A1  a painter]. The generated question “What is John?” would have the focal argument as its answer—“a painter.” 
     In an embodiment, question generator  210  creates questions from all of the major arguments (e.g., A0, A1, . . . , A5) and the adjunct arguments AM-TMP (time), AM-MNR (manner), AM-CAU (cause), AM-LOC (location), AM-PNC (purpose), and AM-DIR (direction) mentioned previously. But it is understood that questions could be formulated for a variety of other arguments, including those identified in reference to SRL system  208  above. 
     The process of producing questions involves intricate decisions by the system. There are at least three issues: (1) selecting the appropriate question word for the semantic argument, (2) deciding on “What” versus “Who,” and (3) handling prepositions. For constituent questions, selecting the appropriate wh-word is aided by the identity of the focused argument in embodiments. Manner arguments (AM-MNR) invite “How” and location arguments (AM-LOC) invite “Where.” But it is not always so simple. Consider, for example, semantic role A4, which is often used for the “end point” of complex locative constructions. A sentence like “They can fly from here [ A4  to any country], should generate a question beginning with “Where.” A similar construction in “Antarctica doesn&#39;t belong [ A4  to any country]” should not produce a “Where” question. 
     Additionally, in some cases it is difficult to decide whether to produce a “Who” or a “What” question—particularly when evaluating subjects and direct objects. In embodiments, question generator  210  may make a rule-based decision to determine the proper question word by examining the part-of-speech of the focal argument, the presence of pronouns, a check in a large gazetteer of first and last person names (about 130K entries in the exemplar implementation), and a lookup into a list of person-denoting words (e.g., king, senator, etc.). If the argument is a whole phrase, a careful analysis is required. For example, “king of the land” requires a “Who” question, but “a hat for a lady” requires a “What” question. 
     As discussed, relying only on the SRL designation for a focal argument is too general to formulate an adequate question in some embodiments. In such instances, question generator  210  analyzes the parsed sentence and retains the preposition (when present) associated with the focal argument for the formation of question-sequences. This is well illustrated when generating wh-questions for temporal arguments (AM-TMP), for example. The AM-TMP label does not distinguish between time points, durations, and sets (i.e., repetitive temporal specifications). Therefore, it is difficult to determine whether a when-question, a how long question, or a how often question is proper for an AM-TMP argument. Consider: 
     (1) [ A0  Peter] called [ AM-TMP  on Monday]. 
     (2) [ A0  Peter] called for [ AM-TMP  six hours]. 
     (3) [ A0  Peter] called [ AM-TMP  every day]. 
     Each of the arguments are labeled AM-TMP, but their corresponding proper questions are: (1) When did Peter call?; (2) For how long did Peter call?; and (3) “How often did Peter call?” 
     Analyzing prepositions can aid in the sub-classification of temporal expressions to generate a proper question-sequence. For instance, prepositions “every” and “each” hint at how often questions, “for” hints at how long questions, while many other prepositions hint at time point descriptions requiring when-questions. As such, embodiments of question generator  210  analyze prepositions when formulating questions and retain the prepositions to be used in the questions. Prepositions may also be retained for the formation of question word-sequences for non-temporal semantic arguments. For example, by retaining the preposition in the sentence “The bird sat on the branch,” question generator  210  may generate the question “On what did the bird sit?” 
     In embodiments, question generator  210  also detects and analyzes verbal groups in the parsed sentence to aid in question generation. In an English language clause, a verbal group is the main lexical verb and its related modifiers—negation, auxiliary verbs, and modals. A sentence with multiple clauses may have several verbal groups. Question generator  210  utilizes part-of-speech and lexical patterns to identify such verbal groups and, once identified, analyzes the tense, grammatical aspect, verb negation, modality, and grammatical voice (passive/active) to formulate proper questions for the labeled arguments. 
     In some embodiments, it is necessary to link the verbal group to the verb of the detected predicate. In the presence of auxiliary verbs, SRL systems (like SENNA) produce multiple analyses for the same chunk of text, with some being systemically incorrect. The analysis can be corrected by utilizing the separately detected verbal group. This type of post-processing can be completed by the system itself or aided by an external source, such as an automated correction system. 
     In embodiments, question generator  210  may formulate yes-or-no questions for each predicate that has a finite verb (thus excluding bare- and to-infinitives and gerunds). When a sentence contains multiple predicates, multiple yes-or-no questions can be generated—one for each predicate. An example method for generating a yes-or-no question is as follows: First, the system selects from a clause all semantic role-fillers for the current predicate. Then, the sequential position of SRL arguments is rearranged, if necessary. For yes-or-no questions, the standard declarative word order (usually subject-object-verb) is preserved. Do-support (inclusion of forms of the auxiliary verb “do”) is provided when needed, based on the detection and analysis of verbal groups. Constructions that do not require do-support include copular, modals, and cases when an auxiliary be/have/do verb is already present. Additionally, adjunct arguments may be moved relative to the main verb. For example, a generated yes-or-no question for the sentence “He quickly ate” may be “Did he eat quickly?” The generated question includes the addition of do-support in the form of “did,” and changes the order of the adjunct argument “quickly” in relation to the main verb “ate.” 
     In embodiments, yes-or-no questions may be posed in positive mode. An analysis of the verbal group of the predicate will provide question generator  210  information about the explicit negation of the main verb, including contracted negation, such as “didn&#39;t” and “couldn&#39;t.” In these instances, the question generation process may avoid transferring the negation into the question, while simultaneously registering that the correct answer is flipped from “yes/no” to “no/yes.” For example, from “Johnny didn&#39;t know the song,” a question generator  210  generating positive mode questions may derive “Did Johnny know the song?” (Answer: “No”); and for the copula “The tea isn&#39;t sweet enough,” question generator  210  generating positive mode questions may derive “Is the tea sweet enough?” (Answer: “No”). 
     After question generator  210  formulates questions  212  for each focal argument of each predicate in the sentence, parsing unit  206  extracts the next sentence (when present) from text  204  for question generation. This process continues until one or more questions have been generated for all sentences of text  204 . 
       FIG. 3  depicts an example method for automatically generating questions from text. At  300 , the system receives the text. As mentioned above, the text can include one or more sentences. Generally, each sentence comprises at least one predicate and one or more arguments associated with the predicate. 
     At  302 , a sentence is parsed from the text. When the text only comprises one sentence, that sentence is selected. When the text comprises more than one sentence, a single sentence from the text is parsed for initial processing. Then, the remaining sentences are analyzed individually in succession. After a single sentence is parsed, semantic role labels are assigned to each argument associated with a predicate in that sentence in the manner described above in reference to  FIG. 2  ( 304 ). Then, at  306 , one or more questions are automatically generated relating to the predicate. As discussed above, question generation is based on the semantic role labels assigned to the predicate&#39;s associated arguments. 
       FIG. 4  depicts an example method for automatically generating questions for each predicate of each sentence of text. At  400 , the system receives the selected text. At  402 , a single sentence is parsed from the text, as described above in reference to step  302  of  FIG. 3 . After a single sentence is parsed, semantic role labels are assigned to each argument associated with a predicate in that sentence in the manner described above ( 404 ). Then, at  406 , one or more questions are automatically generated for the predicate based on the semantic role labels assigned to its associated arguments. 
     Next, at  408 , the system determines if there is more than one predicate in the parsed sentence. When more than one predicate is present, the process repeats beginning at  404 —semantic role labels are assigned to each argument associated with the next predicate and questions are generated relating to that next predicate. The process continues in this fashion until questions are generated for each predicate of the parsed sentence. When only one predicate is present in a sentence, or after every predicate of the sentence is processed, the system then determines if there is more than one sentence in the text ( 410 ). When the text contains more than one sentence, the process repeats from  402  with the parsing of a next sentence from the text. The next sentence is processed using the same method as the first sentence described above. The method continues to repeat in this pattern until questions are generated relating to each predicate of each sentence of the text. 
     An evaluation of AQG systems and methods indicate that the SRL-based system described herein is superior in many respects to a system utilizing a neural network for question generation. The neural network system chosen for comparison was an LSTM (long short term memory)-based system trained over a large corpus of question-answer pairs from Wikipedia. Given an input sentence, the system generates questions based on the encoded input and what the model has learned about plausible question generation content and form from the training data. 
     In the comparison study, each system&#39;s ability to produce good questions for some pre-selected amount of text, focusing only on the question-generation capabilities, was evaluated. Five expository texts were selected for input into the system, comprising a total of 171 sentences. Both the neural and the SRL-based systems were tasked to generate questions for each of the 171 sentences. 
     The SRL-based system generated at least one question for 165 sentences and failed to provide output for 6 sentences. Overall, the SRL-based system generated 890 questions and averaged 5.4 questions for each sentence. There are two reasons for this abundance, both described in more detail above: First, the system attempts to generate a yes-or-no question for each predicate in each sentence. Thus, it generated  236  yes-or-no questions. Next, the system attempts to generate a constituent question for almost each argument of each predicate in each sentence. Thus, the system generated  654  constituent questions. The neural system generated one question for each of 170 sentences (and failed for one sentence). All questions generated by the neural system resembled constituent questions. 
     In total 1,060 questions were automatically generated for evaluation across both systems. The questions were annotated by linguistic experts and rated on three scales: Grammar, Semantics, and Relevance. The grammar and semantic scales were five-point scales, ranging from (1) severely mangled/nonsensical to (5) well-formed/adequate. The relevance scale was a four-point scale, ranging from (0) too mangled to judge relevance to (3) the question is about the sentence. 
     To estimate the quality of the generated questions, the average ratings for three groups of questions were compared: (1) yes-or-no and (2) constituent questions from the SRL-based system (SRL-YNQ and SRLCQ), and (3) questions from the neural system (NN). ANOVA analyses were conducted for each of the three rating scales and the total score across each question group. 
     Results depicting the average rating for each group of questions are presented in  FIG. 5 . As shown in the table, yes-or-no questions generated by the SRL-based system (SRL-YNQ) (n=236), were rated significantly higher than constituent questions from the SRL-based system (SRL-CQ) (n=654), which, in turn, were rated significantly higher than questions from the neural system (NN) (n=170). All comparisons were statistically significant (p&lt;0.001), except for SRL-CQ vs. NN on grammar. 
     Next, the automatically generated questions were analyzed based on their potential to be useful. A potentially useful question was determined to be one that has a reasonably good grammar (rating≥4), is semantically sensible in context (rating≥4) and is relevant to the information conveyed in the text (rating≥2). The criteria was analyzed with two measures: First, the proportion of questions having a total rating≥10 was determined. Among the SRL-YNQ questions, 81% were deemed potentially useful, compared to 64% among SRL-CQ questions, and 29% among questions generated by the neural network. A second, more stringent measure required that a question meet the criteria above on each of the three scales, i.e. Grammar≥4, Semantics≥4, and Relevance≥2. With this measure, the proportion of potentially useful questions was 71% for SRL-YNQ questions, 50% for SRL-CQ questions, and 15% for the neural network-generated questions. 
     The comparison study indicates that the herein described SRL-based system can generate a relatively high percentage of questions that are potentially usable as-is in an application, achieving good ratings for grammaticality, semantic coherence, and relevance. The SRL system is able to generate particularly high quality yes-or-no questions, as demonstrated by the strong scores from the human raters. Another strength demonstrated by the SRL-based system is the ability to systematically generate multiple constituent questions by focusing on each argument of a predicate in a clause, as described in detail above. 
     The methods and systems described herein may be implemented using any suitable processing system with any suitable combination of hardware, software and/or firmware, such as described below with reference to the non-limiting examples of  FIGS. 5, 7A, 7B, and 7C . 
       FIG. 6  depicts at  600  a computer-implemented environment wherein users  602  can interact with a system  604  hosted on one or more servers  606  through a network  608 . The system  604  contains software operations or routines. The users  602  can interact with the system  604  through a number of ways, such as over one or more networks  608 . One or more servers  606  accessible through the network(s)  608  can host system  604 . It should be understood that the system  604  could also be provided on a stand-alone computer for access by a user. 
       FIGS. 7A, 7B, and 7C  depict example systems for use in implementing a system. For example,  FIG. 7A  depicts an exemplary system  700  that includes a standalone computer architecture where a processing system  702  (e.g., one or more computer processors) includes a system  704  being executed on it. The processing system  702  has access to a non-transitory computer-readable memory  706  in addition to one or more data stores  708 . The one or more data stores  708  may contain first data  710  as well as second  712 . 
       FIG. 7B  depicts a system  720  that includes a client server architecture. One or more user PCs  722  access one or more servers  724  running a system  726  on a processing system  727  via one or more networks  728 . The one or more servers  724  may access a non-transitory computer readable memory  730  as well as one or more data stores  732 . The one or more data stores  732  may contain first data  734  as well as second data  736 . 
       FIG. 7C  shows a block diagram of exemplary hardware for a standalone computer architecture  750 , such as the architecture depicted in  FIG. 7A , that may be used to contain and/or implement the program instructions of system embodiments of the present invention. A bus  752  may serve as the information highway interconnecting the other illustrated components of the hardware. A processing system  754  labeled CPU (central processing unit) (e.g., one or more computer processors), may perform calculations and logic operations required to execute a program. A non-transitory computer-readable storage medium, such as read only memory (ROM)  756  and random access memory (RAM)  758 , may be in communication with the processing system  754  and may contain one or more programming instructions. Optionally, program instructions may be stored on a non-transitory computer-readable storage medium such as a magnetic disk, optical disk, recordable memory device, flash memory, or other physical storage medium. Computer instructions may also be communicated via a communications signal, or a modulated carrier wave, e.g., such that the instructions may then be stored on a non-transitory computer-readable storage medium. 
     A disk controller  760  interfaces one or more optional disk drives to the system bus  752 . These disk drives may be external or internal floppy disk drives such as  762 , external or internal CD-ROM, CD-R, CD-RW or DVD drives such as  764 , or external or internal hard drives  766 . As indicated previously, these various disk drives and disk controllers are optional devices. 
     Each of the element managers, real-time data buffer, conveyors, file input processor, database index shared access memory loader, reference data buffer and data managers may include a software application stored in one or more of the disk drives connected to the disk controller  760 , the ROM  756  and/or the RAM  758 . Preferably, the processor  754  may access each component as required. 
     A display interface  778  may permit information from the bus  752  to be displayed on a display  770  in audio, graphic, or alphanumeric format. Communication with external devices may optionally occur using various communication ports  772 . 
     In addition to the standard computer-type components, the hardware may also include data input devices, such as a keyboard  774 , or other input device  776 , such as a microphone, remote control, pointer, mouse and/or joystick. 
     This written description describes exemplary embodiments of the invention, but other variations fall within scope of the disclosure. For example, the systems and methods may include and utilize data signals conveyed via networks (e.g., local area network, wide area network, internet, combinations thereof, etc.), fiber optic medium, carrier waves, wireless networks, etc. for communication with one or more data processing devices. The data signals can carry any or all of the data disclosed herein that is provided to or from a device. 
     The methods and systems described herein may be implemented on many different types of processing devices by program code comprising program instructions that are executable by the device processing system. The software program instructions may include source code, object code, machine code, or any other stored data that is operable to cause a processing system to perform the methods and operations described herein. Any suitable computer languages may be used such as C, C++, Java, etc., as will be appreciated by those skilled in the art. Other implementations may also be used, however, such as firmware or even appropriately designed hardware configured to carry out the methods and systems described herein. 
     The systems&#39; and methods&#39; data (e.g., associations, mappings, data input, data output, intermediate data results, final data results, etc.) may be stored and implemented in one or more different types of computer-implemented data stores, such as different types of storage devices and programming constructs (e.g., RAM, ROM, Flash memory, flat files, databases, programming data structures, programming variables, IF-THEN (or similar type) statement constructs, etc.). It is noted that data structures describe formats for use in organizing and storing data in databases, programs, memory, or other non-transitory computer-readable media for use by a computer program. 
     The computer components, software modules, functions, data stores and data structures described herein may be connected directly or indirectly to each other in order to allow the flow of data needed for their operations. It is also noted that a module or processor includes but is not limited to a unit of code that performs a software operation, and can be implemented for example as a subroutine unit of code, or as a software function unit of code, or as an object (as in an object-oriented paradigm), or as an applet, or in a computer script language, or as another type of computer code. The software components and/or functionality may be located on a single computer or distributed across multiple computers depending upon the situation at hand. 
     It should be understood that as used in the description herein and throughout the claims that follow, the meaning of “a,” “an,” and “the” includes the plural reference unless the context clearly dictates otherwise. Also, as used in the description herein and throughout the claims that follow, the meaning of “in” includes “on” unless that context clearly dictates otherwise. Finally, as used in the description herein and throughout the claims that follow, the meanings of “and” and “or” include both the conjunctive and disjunctive and may be used interchangeably unless the context expressly dictates otherwise; the phrase “exclusive or” may be used to indicate a situation where only the disjunctive meaning may apply. 
     The invention has been described with reference to particular exemplary embodiments. However, it will be readily apparent to those skilled in the art that it is possible to embody the invention in specific forms other than those of the exemplary embodiments described above. The embodiments are merely illustrative and should not be considered restrictive. The scope of the invention is reflected in the claims, rather than the preceding description, and all variations and equivalents which fall within the range of the claims are intended to be embraced therein.