言語処理学会 第25回年次大会 発表論文集 (2019年3月)

Combination of Statistical and Neural Approaches
in the Japanese Question Generation System
Lasguido Nio
Koji Murakami
Rakuten Institute of Technology, Rakuten Inc.
{lasguido.nio, koji.murakami}@rakuten.com
1 Abstract
Question generation is the task of generating a natural question from a given input sentence. This often requires hand-crafted templates or sophisticated
NLP pipelines which require extensive labor and expertise to morphologically analyze the sentences andcreate the NLP framework. In order to simplify theselabors, contrastive experiment between two types ofsequence learning: statistical-based machine translation and attention-based sequence neural network havebeen done. Going further we investigate, compares,
and observe the combination of both approaches forquestion generation task. Combination of both approaches is done based on the voting mechanism. Thisway we aim to overcome both approaches and establish a system that excels in terms of content quality
and fluency according to a subjective human test.

2 Introduction
Language generation is attractive because its applications involve a vast amount of domains ranging
from dialogue systems [14, 10], reading comprehension[15, 11, 5], to Frequently Asked Questions (FAQ) generation [18]. Question generation seems to be simple
yet not many scientists explore it. Creating a goodquestion is not a trivial task, and it is much harder tocreate a deep question rather than a factoid question.
While factoid questions require an explicit memory response, deep questions require more profound thinking
and recall [8]. In this study, we will limit ourselves toquestions of factoid type that can be answered explicitly but are tedious to create.
Conventional approaches on question generation
tasks relied either on rule-based approaches [19], orcomplex NLP pipelines [8]. Recent works have started
to address this task with recent neural network-based
machine learning algorithms [20, 5, 12]. However, someissues still remain. In the majority of the previous
works, there are two common challenges: the first is thecomplexity of analyzing and building a pipeline that isable to generate a natural and relevant question, andthe second is data collection. Many researchers utilize
crowdsourcing which saves time when it comes to statistical approach, but more expensive. Here we aim todesign the experiment that is both eﬃcient and robust

in generating natural and relevant questions.
To address the above-mentioned challenges we utilize end-to-end machine learning approaches in orderto learn the sentence-question pattern automatically.
The utilization of machine learning enables us to do
soft-matching [3, 13]. This feature helps with the robustness of the model, enabling it to deal with question
patterns that are not available in the training dataset.
Recent success of neural machine translation (NMT)
technology [2] promises us a high-performance translation result. However, there have also been some reportsof traditional statistical machine translation (SMT) approaches that boast better performance, especially incertain low-resource conditions [6]. Inspired by thisfinding, we utilize these two popular machine translation paradigms in our question generation task.
The power of NMT lies within the attention mechanism and bi-directional architecture properties [9]. The
attention mechanism allows us to selectively focus onparts of the source sentence during translation, while
the bi-directional architecture enables us to capture information regarding long-term dependency structure ofsentences. On the other hand, SMT gains an advantage from phrase-alignment that is statistically calculated during the learning process [7], this alignment isbasically a mapping rule that is learned via the trainingdata.
In this work we make the following contributions:• To our knowledge, we are the first to utilize endto-end machine learning translation techniques forquestion generation on Japanese sentences. And weare among the first to employ a deep sequence-tosequence learning approach to generate questions.
• We propose a simple, but eﬀective way to performsystem combination between two popular machinetranslation paradigm in the Japanese question generation. Experimental results demonstrate that our
combined system shows promise for overcoming theshortcomings of each approach.

3 Question Generation
We formulate our question prediction task as follows:
Given an input context sentence c, we aim to generatea natural question q. Both input and output can be asequence of arbitrary length [c1 , ..., c|c| ] and [q1 , ..., q|q| ].

― 1379 ―Copyright(C) 2019 The Association for Natural Language Processing.
All Rights Reserved.

This question generation task can be defined as:q̂ = arg max P (q|c)(1)

qwhere q̂ is the system best-generated question, andP (q|c) is the conditional probability of the predictedquestion sequence q, given the input c. Here, we aim
to maximize P (q|c) over all possible q. This conditional
probability can be likened to a translation model in astatistical sequence learning approach, or a conditional
log-likelihood in neural sequence learning approach.
3.1 Rule Based Question Generation
The Japanese language falls under the class of agglutinative languages. Verbal expression is fundamentally
located at the end of a sentence, and it can be added
to various words such as nouns, particles, and auxiliary verbs with conjugations at the ending of the word
stem. If we convert an aﬃrmative sentence to yes-noquestion form, we need to add an appropriate sentence
ending particle. For generating a wh-type question, we
need not only that particle, but also need to use aninterrogative word.
We analyzed the sentences which end with the following part-of-speech: verbs, adjectives, particles, andauxiliary verbs; we will also consider a few nouns
that function similarly to adjectives, verbs, or auxiliary verbs. While creating rules, only “KA(か)” wasused as the sentence ending particle in interrogative expressions, because this is the most popular and widelyused with any part-of-speech. An interrogative expression and an edit flag (use or delete) of each 3 words
are annotated to all sentences by the human annotator. If several sentences have the same interrogative
expression and share the same words, they are merged.
Finally, we obtained approximately 1,700 converting
rules.
To generate wh-type questions, we apply our abovedescribed rules and randomly replace only one wordwith “how” or “what”. The words which can be replaced are (1) a noun in an objective case, (2) an adjective, or (3) a verb.

recursive architecture and global attention mechanism.
The bi-directional architecture enables the model tolearn in both a forward and backward context. The
attention mechanism allows the model to put emphasis on a certain part of the sentence, imitating the way
humans think to solve a task.
Taking into account the question generation task formulation on Equation 1, we can factorize the conditional probability p(q|c) as
P (q|c) =|q|∏

P (qt |c, q1..t−1 ).

(3)

t=1

Where qt are word candidates that combine to give
output question q, we treat the conditional probability P (q|c) as a product of word-level prediction. The
probability of qt is predicted based on the input context
sentence c, and all the words that have been previously
generated q1..t−1 .
Furthermore, we can break down the word-level conditional probability into
P (qt |c, q1..t−1 ) = sof tmax(Wq tanh(Wb [bt ; at ])). (4)Where bt portrays the bi-directional recursive network
state variable at the time step t, at is the attention
based encoding of input sentence context c at decoding
time step t, and Wq and Wb are the parameters to belearned.
3.4 Vote Mechanism
We devise a simple voting mechanism based on how
close the output between the proposed approaches tothe question language model in the term of perplexity. The question language model is obtained from thequestion training dataset. Combination approach (denoted by com) obtained question with the lowest perplexity from both proposed approaches. More detailsof this mechanism can be seen in Fig. 1.

3.2 Statistical Question Generation
Here we utilize phrase-based statistical machinetranslation (SMT) to capture the pattern between context sentence input and question output. We treat the Figure 1: Overview of our question voting mechanism.
sentence-question pair as a parallel corpus to train the
4 Datasets
translation model, which is based on the noisy chanThe sentence-question pairs dataset that we connel model. Considering the general question generation
structed in this study is based on the user-merchanttask, we reformulate Equation 1 with Bayes rule asreview pages [16] on the Rakuten Japan website1 . Weq̂ = arg max P (c|q)PLM (q).
(2) collect the review data from the selected wine genreq
products from 2013 to 2017. Overall we obtain 241,794
This way we can obtain the language model PLM (q) reviews and segment it into 673,963 sentences.
and separate the translation model P (c|q).
From there, we run our rule-based question generation on all of the review sentences, and employ anno3.3 Neural Question Generation
Here we generate the question with the neural ma- tators to check and correct the generated questions. In
chine translation (NMT) technique. In contrast with the end, we managed to gather around 30k data points.
the SMT, the power of NMT lies on the bi-directional― 1380 ―1 https://www.rakuten.co.jp

Copyright(C) 2019 The Association for Natural Language Processing.
All Rights Reserved.

YN
WH

SMT
91.70
75.66

SMT CMP
64.25
81.75

NMT
38.30
66.45

NMT+PRE
33.38
69.74

NMT CMP
42.32
71.70

NMT CMP+PRE
42.66
73.37

COM
71.19
81.77

RULE
71.51
83.53

Table 1: Automatic evaluation result with BLEU score given various approaches.
We split this dataset into two diﬀerent categories yn
and wh, which consecutively portray yes-no and howwhat type question, and then further randomly subdivide each of these categories into training, development, and test sets.

YN
WH

Training Development Test
15,755300
300
14,571
600
300
Table 2: Dataset statistics.

Total
16,355
15,471

However, due to the limited amount of data, we only
take 300 pairs of sentence-question as a test set. The
results of our experiments will later be presented in thecontext of this test set.
The details of this dataset are provided in Table 2.
There are around 15k training set for each yn and whcategory, making total 30k dataset for training. We
train the model separately with the diﬀerent dataset
for each yn and wh. Furthermore, we are also employing domain adaptation technique, making a compoundmodel that trained from both yn and wh dataset.

5 Experimental Evaluation
In this paper, we employ two types of approaches:statistical question generation, and neural question
generation. We employ Moses2 toolkit and OpenNMT
system 3 for SMT and NMT implementation, respectively.
During SMT training, we employ GaCha filtering[17] to remove noisy sentence level alignment, with the
GaCha filtering threshold set to 0.8. In our experiment,
the SMT model is denoted by smt.
As for the NMT training, we employ the Japanese
Wikipedia4 dataset provided by Polyglot Project [1] toenrich the model word embedding. Using this dictionary, we learned the word representation with FastText[4]. The NMT model that utilizes this embedding is indicated by +pre.
In the NMT-based approach, we follow the same configuration used by Du et al. [5]. Therefore, the model
denoted by nmt is treated as the state-of-the-art baseline.
Going a step further, we build a compound modelusing a combined dataset. This model is denoted by
cmp (smt cmp, nmt cmp, and nmt cmp+pre). This
compound model is trained using a compound dataset(yn and wh). In contrast, the non-compound models(smt, nmt, and nmt+pre) are trained with the corresponding training and test sets. For example, in a
non-compound model, if we evaluate the model withthe yn dataset, we train the model only with the yn
dataset. From this compound model, we would like toassess how the model works given a complex and noisy
training set.
Not to be confused with the compound model, here
we also introduce the voting mechanism that combines
both SMT and NMT approaches (denoted by com).
This approach combines the best model on both SMT
and NMT setup. Here we utilize SRILM5 toolkit tohelp us build the language model.
At the end of this experiment, we evaluated our system response both automatically by calculating thegenerated response q̂ with the question in reference
database q, and subjectively by giving out a survey
to humans.
5.1 Automatic Evaluation
Automatic evaluation of our models are done by
the BLEU-4 metric. This metric calculates how well
the generated question compares to the reference question. The results of the BLEU-4 evaluation is presented
above in Table 1.
5.2 Subjective Human Evaluation
Next, the human evaluation studies are performed
to measure the quality of questions. We conduct theevaluation on rule-based approach (rule), both SMT
approaches (smt, smt cmp), the best model in NMT
approaches according to the automatic evaluation results (nmt cmp+pre), and the combination vote approach com. We apply two metrics, content quality
and fluency, as defined by the Japanese Patent Oﬃce6 .
Content quality indicates grammaticality and consistency with the reference, and the results of this evaluation can be seen in Fig. 2. Fluency focuses on only
readability of the generated questions, with the resultsof this is shown in Fig. 3. If a generated questionis neither of interrogative form or question type, conflict happens between yn and wh, and so both scoresshould be lower.
Here, we randomly sampled 50 sentence-question
pairs for yn and wh from these two cases, and evaluated by asking two Japanese native speakers to rate
the pairs in terms of the metrics above on a 1-5 scale(5 for the best).
5.3 Discussion
Automatic evaluation results (Table 1) shows that
the SMT approach (smt and smt cmp) in general performs better than NMT (nmt and nmt cmp) and therule-based rule approach. Except on the wh type2 http://www.statmt.org/moses
3 http://opennmt.net
4 http://ja.wikipedia.org

5 http://www.speech.sri.com/projects/srilm
6 http://www.jpo.go.jp/shiryou/toushin/chousa/pdf/
tokkyohonyaku_hyouka/01.pdf― 1381 ―Copyright(C) 2019 The Association for Natural Language Processing.
All Rights Reserved.

Figure 2: Subjective evaluation result on content quality.

Figure 3: Subjective evaluation result on question fluency.
dataset, where the rule-based approach performs a little bit better. However, during the subjective evaluation, we found out that the content quality and fluencyof the SMT approach is much better. The machinelearning’s soft-matching feature allows the model tobecome more flexible in generating natural questions,while the rule-based generated question looks like acanned question.
Overall the combination vote system not showing significant improvement in BLEU metrics. However, on
both quality and fluency subjective evaluation, we cansee that com approach perform on par and/or significantly better compared to others. It indicates that thevoting mechanism manages to select a more naturaland comprehensible question.

6 Conclusion
A data-driven automatic question generation approach for Japanese text is presented. Experimental
evaluation shows that the statistical question generation model achieves the best performance on Japanese
text in both automatic and subjective evaluation. Furthermore, a combination system that votes the output between SMT and NMT is also introduced. This
combination system performs better in both subjective evaluation metrics, overcoming the shortcomingsof each approach. As future work, implementing thisquestion generation to the real FAQ generation problem can be a promising future direction.

References

17th Conference on Computational Natural Language Learning (2013), CoNLL ’13, Association for Computational Linguistics, pp. 183–192.
[2] Bahdanau, D., Cho, K., and Bengio, Y. Neural machine translation by jointly learning to align and translate. In Proceedingsof the 3rd International Conference on Learning Representations (2015), ICLR ’15.
[3] Bengio, Y., Ducharme, R., Vincent, P., and Janvin, C. A neuralprobabilistic language model. The Journal of Machine Learning Research 3 (2003), 1137–1155.
[4] Bojanowski, P., Grave, E., Joulin, A., and Mikolov, T. Enriching word vectors with subword information. Transactionsof the Association for Computational Linguistics 5 (2017),
135–146.
[5] Du, X., Shao, J., and Cardie, C. Learning to ask: Neuralquestion generation for reading comprehension. In Proceedingsof the 55th Annual Meeting of the Association for Computational Linguistics (2017), ACL ’17, Association for Computational Linguistics, pp. 1342–1352.
[6] Koehn, P., and Knowles, R. Six challenges for neural machinetranslation. In Proceedings of the 1st Workshop on Neural Machine Translation (2017), Association for Computational Linguistics, pp. 28–39.
[7] Koehn, P., Och, F. J., and Marcu, D. Statistical phrasebased translation. In Proceedings of the 2003 Conference of
the North American Chapter of the Association for Computational Linguistics on Human Language Technology (2003),
NAACL ’03, Association for Computational Linguistics, pp. 48–
54.
[8] Labutov, I., Basu, S., and Vanderwende, L. Deep questions
without deep understanding. In Proceedings of the 7th International Joint Conference on Natural Language Processing(July 2015), IJCNLP ’15, Association for Computational Linguistics, pp. 889–898.
[9] Luong, T., Pham, H., and Manning, C. D. Eﬀective approaches
to attention-based neural machine translation. In Proceedingsof the 2015 Conference on Empirical Methods in Natural Language Processing (2015), Association for Computational Linguistics, pp. 1412–1421.
[10] Mostafazadeh, N., Misra, I., Devlin, J., Mitchell, M., He, X.,
and Vanderwende, L. Generating natural questions about animage. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (2016), ACL ’16, Association for Computer Linguistics, pp. 1802–1813.
[11] Nguyen, T., Rosenberg, M., Song, X., Gao, J., Tiwary, S.,
Majumder, R., and Deng, L. Ms marco: A human generated MAchine Reading COmprehension dataset. arXiv preprint
arXiv:1611.09268 (2016).
[12] Nio, L., and Murakami, K. Intelligence is asking the right question: A study on japanese question generation. In Proceedingsof the IEEE Spoken Language Technology Workshop (2018).
[13] Nio, L., Sakti, S., Neubig, G., Yoshino, K., and Nakamura,S. Neural network approaches to dialog response retrieval andgeneration. IEICE Transactions 99-D, 10 (2016), 2508–2517.
[14] Piwek, P., Hernault, H., Prendinger, H., and Ishizuka, M.
T2D: Generating dialogues between virtual agents automatically from text. In Proceedings of the 7th International Conference on Intelligent Virtual Agents (2007), IVA ’07, Springer
Berlin Heidelberg, pp. 161–174.
[15] Rajpurkar, P., Zhang, J., Lopyrev, K., and Liang, P. Squad:
100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (2016), EMNLP ’16, Association for
Computational Linguistics, pp. 2383–2392.
[16] Shinzato, K., and Oyamada, Y. What do people write in reviewsfor sellers? investigation and development of an automatic classification system. Journal of Natural Language Processing 25,
1 (2018).
[17] Tan, L., and Pal, S. Manawi: Using multi-word expressions
and named entities to improve machine translation. In Proceedings of the 9th Workshop on Statistical Machine Translation(2014), Association for Computational Linguistics, pp. 201–206.
[18] Tang, D., Duan, N., Qin, T., and Zhou, M. Question answering and question generation as dual tasks. arXiv preprint
arXiv:1706.02027 (2017).
[19] Y., C., and Hasan, S. A. Towards automatic topical question
generation. In Proceedings of the 24th International Conference on Computational Linguistics (2012), COLING ’12, Association for Computational Linguistics, pp. 475–492.
[20] Yang, Z., Hu, J., Salakhutdinov, R., and Cohen, W. Semisupervised qa with generative domain-adaptive nets. In Proceedings of the 55th Annual Meeting of the Association for
Computational Linguistics (2017), ACL ’17, Association for
Computational Linguistics, pp. 1040–1050.

[1] Al-Rfou, R., Perozzi, B., and Skiena, S. Polyglot: Distributed
word representations for multilingual nlp. In Proceedings of the― 1382 ―Copyright(C) 2019 The Association for Natural Language Processing.
All Rights Reserved.