言語処理学会 第25回年次大会 発表論文集 (2019年3月)

Anaphora Reasoning Question Generation Using Entity Coreference
Kimihiro Hasegawa1 , Takaaki Matsumoto2 , and Teruko Mitamura2
12Kobe University
Carnegie Mellon University

ljnjzbo417@gmail.com, {tmatsumo, teruko}@andrew.cmu.edu

Abstract

S15:

We propose an approach for Anaphora Reasoning Question Generation from a plain text: generating a question which needs anaphora resolution to answer. This is one type of Multiple
Sentence Reasoning Question, a paragraph-level
question which needs more than one sentence ina context to answer. We apply our system to
Wikipedia articles and, based on our evaluation,
our system generates more Anaphora Reasoning
Question compared to the current state-of-the-art
neural question generation model which intends
to generate a paragraph-level question by around30%.

S16:
S17:
S18:
Q15:
Q16a:
Q16b:
Q16b*:
Q18:

When Andrew goes home after baseball, he likes
to eat a snack.
He eats carrots and bananas.
If he is a good boy, his mom, Mrs. Smith, sometimes gives him milk and cookies.
Afterwards, Andrew finishes his homework.
Who likes to eat a snack?
Who eats carrots and bananas?
What does he eat?
What does Andrew eat?
Who finishes his homework?

Figure 1: Target Examples.1

1 Introduction
Reading Comprehension (RC), understanding what atext says, is one fundamental area in Natural Language
Processing (NLP), and it relates to other areas of NLP,
such as Dialogue, Question Answering, and MachineTranslation. Although many studies in those areas havebeen done towards processing a single sentence as themain target, recently a context is drawing attention fromresearchers in those areas. Since the meaning of a sentence changes depending on a context, reading a context
plays a crucial role in understanding even a single sentence. Along with this direction, there is a study about aparser, called Abstract Meaning Representation (AMR),
by O’Gorman et al. (2018). They made a corpus for building a parser by retrieving the meaning of sentences andsentence relationship in a context and representing themin the AMR format.
While a machine utilizes a parser to understand a context, humans can deepen their understanding on a context
by using questions, especially when a question represents
a relationship between sentences. Through the process oftackling questions which need more than one sentence toanswer and checking their correct answers, humans cancomprehend the relationships between sentences better.
Hence, question-answer pairs can be a rich source of sentence relationship: relationships between sentences in acontext can be extracted from paragraph-level questions.

Motivated by this idea, we propose a method to generate Anaphora Reasoning Question (ARQ) using entity
coreference without human annotation. The overview ofour system is the following: 1) find a sentence with atleast one pronoun which refers to a specific entity in another sentence in plain text. 2) transform the sentence intoa question. 3) if the pronoun still exists in the question,then replace the pronoun with the specific entity that thepronoun refers to. For instance, in Figure 1, S16 contains a pronoun, ”He”, which refers to ”Andrew” in S15.
When we transform S16 into questions, we get Q16a andQ16b. Since Q16b has the pronoun, ”he”, which refers to”Andrew”, we replace the pronoun with ”Andrew”. Then,we get Q16b* after the replacement. In order to answer
Q16a or Q16b*, not only S16 but also S15 are necessary
to figure out what ”He” refers to in this context. We will
explain the detail of our pipeline in Chapter 3.
We conduct an evaluation of the generated questions by
our system to check what proportion of them are acceptable ARQ in terms of Grammatical Correctness, Answer
Existence, and The Number of Necessary Sentences. Forcomparison, we evaluate questions generated by a neuralQuestion Generation (QG) model, CorefNQG, by Du andCardie (2018), which also intends to generate questions
from more than one sentence. We will describe the detail
1 S15, 16, 17, 18 are extracted from a paragraph about children stories

in the MCTest by Richardson et al. (2013). Entities in italics, Andrew,he, He, etc, refer to Andrew. Q15, Q16a, Q16b, Q16b*, Q18 are generated questions by our system.

― 1105 ―Copyright(C) 2019 The Association for Natural Language Processing.
All Rights Reserved.

of our evaluation metrics and the results in Chapter 4.

2 Related Works
In the situation that neural network models have been
gaining attention from many NLP fields, there have been
increasing demands for rich and large-scale questionanswer-pair datasets. Rajpurkar et al. (2016) created
Stanford Question Answering Dataset (SQuAD) which
consists of more than 100k question-answer pairs withvarious types of reasoning, which encourages researchers
in NLP communities to build neural network models.
This dataset is created by posing a paragraph fromWikipedia to crowd workers and asking them to create
questions about the paragraph. According to the paper,
Multiple Sentence Reasoning Question (MSRQ), whichneeds more than one sentence to answer, accounts foraround 13%. Using a similar method, Khashabi et al.
(2018) created Multi-Sentence Reading Comprehension(MultiRC) corpus from selected paragraphs in various
sources: News articles, Wikipedia articles, and Fictions,where one of the Fictions is from MCTest by Richardson
et al. (2013). The MultiRC corpus consists of paragraphs,
questions, and several choices for answer-options, whereMSRQ accounts for around 60% according to the paper.
Corresponding to these datasets creation by humanannotation, which gets expensive and time-consuming
as the scale of a dataset becomes large, there havebeen researches on Question Generation by machines.
Two approaches are chosen for machine QG: neural
QG and rule-based QG. Du and Cardie (2018) built
a neural network model incorporating with coreference
knowledge for paragraph-level question generation. The
model, CorefNQG, generates a dataset with over 1
million question-answer pairs from 10,000 top-ranking
Wikipedia articles. Whereas, Heilman and Smith (2010)
built a rule-based QG model with overgenerating andscoring, which transforms a single sentence into multiple types of questions. Satria and Tokunaga (2017) built
a rule-based QG model which split a sentence with nonrestrictive clause into two independent sentences, generating reference questions with multiple answer choices.
Also, there is close research with ours: Araki et al.
(2016) proposed an approach for rule-based QG from ahuman-annotated text utilizing specific inference steps
over multiple sentences. They generate not only ARQ
but also event coreference questions and paraphrase questions.

3 Approach
In this section, we delineate the detail of our pipeline
for generating Anaphora Reasoning Question. Figure 2
shows an overview of our pipeline. A box is a data onthat step, and an arrow is a process to get the next data.
Paragraphs below explains each process, an arrow in Figure 2.

Paragraph Selection: We retrieve articles from thedataset generated by CorefNQG, in order to make a comparison between CorefNQG and our system. Referringto the condition of article selection used in MultiRC, i.e.,
the number of sentences in an article is more than 5 andless than 19, we select articles which have more than 7
and less than 16 sentences. When an article is very short,the probability of detecting entity coreference gets low.
In addition, when an article is very long, the accuracy ofentity coreference detection decreases. In general, it isdifficult to find entity coreference between two entities
when those two locate far away from each other.
Entity Corefernce Detection and Sentence Segmentation: In order to get entity coreference clusters, we apply to those selected articles NeuralCoref2 , which detects
entity coreference in an article using a neural network, extended on Spacy3 . To split an article into sentences, we
use Spacy’s sentence segmentation which uses its dependency parse to determine sentence boundaries. As a supplement to the tool, we also use Segtok4 which is a patternbased segmentation tool.
Entity Coreference Selection: Among the detectedclusters, we pick clusters based on the most representative entity in a cluster: we discard a cluster if themost representative entity is either a pronoun or determiner/possessive determiner + noun/noun phrase. Thisis because the target of our system is a cluster which has
a proper noun as the most representative entity.
Source Sentence Selection: For each candidate entity
in a selected cluster, we check the original sentence where
the candidate entity comes from. We discard a sentence
when it contains the most representative entity because
a question generated from that type of sentence does not
require multiple sentences to answer. For instance, suppose we generated a question from S15 in Figure 1 to
ask what “he” refers to. One generated question could
be Q15, but this question can be answered by looking at
S15 even though it asks about a pronoun which refers toa proper noun.
Question Generation: We apply the Heilman’s question generation tool5 to generate questions from each selected sentence. The tool could generate both Yes/Noquestions and Wh/How questions, but we only generate Wh/How questions considering that the dataset byCorefNQG contains only Wh/How questions.
Entity Coreference Resolution: The Heilman’s toolgenerates multiple questions from one sentence. In somequestions, a pronoun which can be replaced with the mostrepresentative entity is replaced with an interrogative. In
other questions, where a target pronoun remains, we replace the pronoun with the most representative entity utilizing Spacy’s dependency parser: checking if a remain-― 1106 ―2 https://github.com/huggingface/neuralcoref
3 https://spacy.io/
4 https://github.com/fnl/segtok
5 https://github.com/sumehta/question-generation

Copyright(C) 2019 The Association for Natural Language Processing.
All Rights Reserved.

Datasetparagraphselection
Paragaphw/qa pairs
entity coreference detection
Entity Coreference Clusters

sentence
segmentationquestionsource sentence
generation
selection
Target Sentences
List of Sentences

entity coreference
resolution
Resolved
Questions
Questionsquestion selection

entity coreference
selection
Selected Clusters

Anaphora Reasoning Question

Figure 2: Pipeline of Our Anaphora Reasoning Question Generation.
<Grammatical Correctness>
check if the question is syntactically well-formed.
1 (best): the question has no grammatical error.
2: the question has 1 or 2 grammatical errors.
3 (worst): the question has 3 or more grammatical errors.
<Answer Existence>
check if the answer to the question can be inferred from the article.
1 : the answer to the question can be inferred from the article.
2 : the answer to the question cannot be inferred from the article.
<The Number of Necessary Sentences>check how many sentences are required to answer the question. This
metric is used only when Answer Existence is 1.

Figure 3: Evaluation Metrics.

ing pronoun has the same head in the question’s dependency as that in the original sentence’s dependency. For
instance, in Figure 1, “He” is replaced with “Who” inQ16a, while “he” still remains in Q16b. In order to makeQ16a more specific and clear, we replace “he” with “Andrew” by checking if “he” in Q16a has the same headdependent relationship with “eat” as the relationship between “He” and ”eats” in S16.
Question Selection: Even after Entity Coreference
Resolution is performed, some questions still have pronouns. In order to make a question less ambiguous, wediscard a question with pronouns by utilizing Spacy’s
part-of-speech tag. Suppose we get Q18 from S18 in Figure 1. The pronoun, “his”, makes Q18 ambiguous, because “his” could refer to another boy’s name in the context.

4 Experiment
We conducted a human evaluation on 400 questionanswer pairs derived from 57 articles: 200 QA pairs by
our system from 40 out of 57 articles, and 200 QA pairs
by the neural QG system from 57 out of 57 articles. Basedon the evaluation metrics by Araki et al. (2016), our metrics for evaluating generated questions are GrammaticalCorrectness, Answer Existence, and The Number of Necessary Sentences. The details of our metrics are describedin Figure 3.

Paragraph: The flower may consist only of these parts, as in (3)willow, where each flower comprises only (4) a few stamens or twocarpels. ... The individual members of these surrounding structures are known as (5) sepals and petals (or tepals in flowers suchas ”Magnolia” where sepals and petals are not distinguishable fromeach other). The outer series (calyx of sepals) is usually (6) green andleaf-like, and functions to protect the rest of the flower, especiallythe bud. (S1) The
inner series (corolla of petals) is, in general, white:::::::::::::::::::::::::::::::::::orbrightly colored, and is more delicate in structure. (S2) It:::::::
functions:::::::::::::::::::::::::::::::::
to attract insect or bird pollinators. ...
::::::::::::::::::::::
Questions by Our System:
Q1a: What is the inner series more delicate in?
Q1b: What is, in general, white or brightly colored?
Q2a: What functions to attract insect or bird pollinators?
Q2b: What the outer series (calyx of sepals) functions to attract ?
Questions by CorefNQG:
Q3: In what language is the flower of the flower located ?
Q4: What does each flower have ?
Q5: What are the individual members of these surrounding structures called ?
Q6: What is calyx of sepals ?

Figure 4: Example Questions by Our System andCorefNQG. 6

4.1 Results
Table 1 shows the result of the human evaluation
on generated questions about Grammatical Correctness(GC), Answer Existence (AE) and The Number of Necessary Sentences (Num) by one evaluator. As for GC,CorefNQG is better than our system, though the sum ofGC 1 and 2 of our system is the same as that of CorefNQG.
From the evaluation of AE, our system generates more answerable questions than the neural model by about 25%.
About Num, we label “None” for a question whose AE is
2. Our system generates questions which need more than
one sentence than the neural model by around 30%.

4.2 Analysis
Since our system uses multiple tools to generate ARQ,
the errors of each tool accumulate, which results in mistakes in the final results. Especially, Entity Coreference
detection and Heilman’s QG tool are the main reasons,for instance, in Figure 4, Q2b is generated from S2 with
6 Wavy lines are sentences used to generate questions by our system.
Words in italics are answers picked by CorefNQG.

― 1107 ―Copyright(C) 2019 The Association for Natural Language Processing.
All Rights Reserved.

Models
Our System
CorefNQG

Grammatical Correctness
1 (best)23 (worst)
115
68
17127
56
17Answer Existence
1 (Exist)
2 (Not Exist)
159 (0.795)
41110 (0.55)
90

The Number of Necessary Sentences
12> None
96 (0.48)
62 (0.31) 141107 (0.535) 3 (0.015) 090

Table 1: Evaluation Results.
replacing “It” with “the outer series (calyx of sepals)”,
which is a fault of entity coreference detection. As it isseen in Q1b, Heilman’s QG tool tends to generate notaccurate questions from a complex-structured sentence,such as a sentence with a restrictive clause or a parenthesis. Whereas, as it is seen in Q3, CorefNQG tends togenerate questions with using an identical phrase multiple times. As for AE, one possible reason why our systemgenerates more answerable questions than CorefNQG isthat our system transforms sentences into questions by replacing an entity with an interrogative, for instance, ”It”
with ”What” in Q2a. Whereas, CorefNQG uses phrases
in the context to generate questions, which results in
an inconsistency between an interrogative in a question
and the rest of it, represented in Q3: ”what language”
for ”flower” is ”located”. According to our evaluation,
ARQ, even MSRQ, is rarely found in generated questions
by CorefNQG, which does not correspond closely to theevaluation results by Du and Cardie (2018): 36.42% examples of the original test set to train the neural model
require coreference knowledge to answer.

5 Conclusion and Future Work
This paper presents a rule-based approach without human annotation to generate a question which needs to look
more than one sentence to answer. Specifically, we generate a question from a sentence with a pronoun whichrefers to a proper noun in a context in the way that thequestion asks the pronoun. Our experiment shows that
our system outperforms the state-of-the-art neural QG
model which intends to generate MSRQ.
Possible future work includes the improvement of entity coreference detection and to widen the variety ofquestions by generating other types of questions which
also need more than one sentence to answer, such as
event coreference questions and paraphrase questions.
Although we employ all questions generated from one
sentence except a question with a pronoun, we need amethod to pick one question from them. Based on theidea that a question could represent a relationship between
sentences, our next step would be to extract sentence relationship in MSRQ from a corpus, such as MultiRC, inordet to improve machine Reading Comprehension. Inaddition, although in this study one evaluator judged thegenerated questions, we can employ more evaluators andcompare the results to each other, in order to minimize thesubjective opinions of evaluators.

References
Jun Araki, Dheeraj Rajagopal, Sreecharan Sankaranarayanan,Susan Holm, Yukari Yamakawa, and Teruko Mitamura. Generating questions and multiple-choice answers using semantic analysis of texts. In COLING, 2016.
Xinya Du and Claire Cardie. Harvesting paragraph-level
question-answer pairs from wikipedia. In Association for
Computational Linguistics (ACL), 2018.
Michael Heilman and Noah A Smith. Good question! statisticalranking for question generation. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 609–617. Association for Computational Linguistics, 2010.
Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam
Upadhyay, and Dan Roth. Looking beyond the surface:achallenge set for reading comprehension over multiple sentences. In Proceedings of North American Chapter of theAssociation for Computational Linguistics (NAACL), 2018.
Tim O’Gorman, Michael Regan, Kira Griffitt, Ulf Hermjakob,Kevin Knight, and Martha Palmer. Amr beyond the sentence: the multi-sentence amr corpus. In Proceedings of the27th International Conference on Computational Linguistics, pages 3693–3702. Association for Computational Linguistics, 2018. URL http://aclweb.org/anthology/C18-1313.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy
Liang. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on
Empirical Methods in Natural Language Processing, pages
2383–2392, Austin, Texas, November 2016. Association forComputational Linguistics. URL https://aclweb.org/anthology/D16-1264.
Matthew Richardson, Christopher J.C. Burges, and Erin Renshaw. Mctest: A challenge dataset for the open-domain machine comprehension of text. In Proceedings of the 2013
Conference on Empirical Methods in Natural Language Processing, pages 193–203. Association for Computational Linguistics, 2013. URL http://aclweb.org/anthology/
D13-1020.
Arief Yudha Satria and Takenobu Tokunaga. Automatic generation of english reference question by utilising nonrestrictive
relative clause. In CSEDU, 2017.

― 1108 ―Copyright(C) 2019 The Association for Natural Language Processing.
All Rights Reserved.