言語処理学会 第24回年次大会 発表論文集 (2018年3月)

Deep Learning Method to Extract Implicit Keywords
for Historical Essay Questions
Y uanzhi Ke†1 Kotaro Sakamoto†2†3 M adoka Ishioroshi†3 Hideyuki Shibuki†2M asaf umi Hagiwara†1 T atsunori M ori†2 N oriko Kando†3†4†1Keio U niversity †2 Y okohama N ational U niversity†3N ational Institute of T echnology †4 SOKEN DAI
E-mail: {enshika8811.a6, hagiwara}@keio.jp, {sakamoto, shib, mori}@forest.eis.ynu.ac.jp,{ishioroshi, noriko}@nii.ac.jp

1Introduction

motivated us to explore machine learning-based
methods.

AI and deep learning technology have been successful for reading comprehension [1] and shown eﬀectiveness for multi-choice questions [2, 3, 4]. However, it isstill challenging to answer the essay questions that require the examinees to write an essay for the question.
Especially, the historical essay questions in the entry
tests of universities usually provide few materials andrequire the examinees to search the related knowledge in their brain and organize them into an essay.
In such questions, stories are not provided togetherlike those in the reading comprehension task. The
machines need to recall some additional related keywords that are not in the question text and search an
external knowledge base. Moreover, historical terms
have much less frequency which makes them hard tolearn. Thirdly, because of the large knowledge base,
it is expensive to build an end-to-end neural networkmodel like that for reading comprehension.
In our works, we pay attention on the extraction of
the implicit keywords for the retrieval of the related
knowledge. Implicit keywords here refer to the keywords that are the scoring points but do not appear
in the question text. They are important to answer
essay questions and also help the machine get related
knowledge. It has the following challenging points:

3. The keywords are the historical events and entities, which are infrequent. It is challenging totrain their embeddings.
For the issues above, we propose a bi-directional
LSTM model with attention mechanism to extract
and rank the implicit keywords, a data augmentation method and a learning controlling method toimprove the training of the embeddings. To validate our method, we compared the performance ofthe proposed methods with the conventional methods in the task of implicit keyword extraction. Ourachievements include,
1. Our method outperformed TF-IDF, BM25, andElman network.
2. We proposed a novel method to augment thedata and showed that it is eﬀective to improve
the performance.
3. We have shown that controlling the learning
rate for diﬀerent words by their frequencies iseﬀective for a more reasonable ranking.

2Baselines

1. The implicit keywords are not in the question 2.1 TF-IDF and BM25text.
The hypothesis of the two methods is that the key2. The relationship of the question and the an- words have higher TF-IDF score [5] in the related
swer is hard to be defined by rules. Rule-based texts. In the TF-IDF-based method, at first, the syssystems are very limited for the task, which tem searches the knowledge base for the related texts― 560 ―

Copyright(C) 2018 The Association for Natural Language Processing.
All Rights Reserved.

The attention mechanism is to compute a logistic
regression for each output of the LSTM units as its
coeﬃcient of importance. Denote at as the attentionof yt ,at = σ (Wa [yt ] + ba ) .
(2)
The final output prediction for each class is,z=n∑(at · yt ) ,(3)t)(exp Wpi z + bip( ′)Pr (ŷ = i|s) = ∑ii′i′ ∈E exp Wp z + bp .

(4)

Fig. 1. Architecture of the neural network for ranking in our system.

Here i is one of the keywords in the keyword candidate set I.

by the TF-IDF score, then ranks the content words bythe TF-IDF scores of the words. Finally, the system
outputs the top-ranked words that are not includedin the question text.
The BM25-based method is similar to the TF-IDFbased method, but the Okapi BM25 [6] algorithm isused instead of TF-IDF.

3.2

2.2

Elman Network

Elman network is a simple recurrent neural network (RNN) model. It has been reported to be eﬀective to extract the implicit keywords [7]3Methodology

3.1

Bi-directional RNN of LSTM
with Attention For Ranking

Learning Rate Controlling

The frequencies of the words in the answers are inevitably unbalanced. The corresponding parametersof the more frequent words are updated more times.
As the proposed model is deep, it is easy to be overfitted by the unbalanced data. To balance the training
progress for diﬀerent words, we control the learning
rate for each word according to its frequency in thedataset. Formally, denote α as the global learning
rate, T F (ik ) as the frequency of keyword candidate
ik , the learning rate αk for keyword candidate ik isdefined as the follows,
αk =α.
T F (ik )(5)

Hence, the parameters for the frequent words aretrained more slower while those for the rare ones aretrained more faster.

The architecture of the proposed neural network
is shown in Fig. 1. Formally, denote the vectors of 3.3 Data Augmentation
words in the input text as x1 , x2 , x3 , ...xt , ...xn , where
In Japanese, the subjects are often omitted. In then is the length of the input. The output of eachglossary included in our corpus to train the word emLSTM unit yt is the follows:
beddings, when the subject is always shown as thetitle, the descriptions of most documents do not conft = σ (Wf [yt−1 , xt ] + bf ) ,tain the subject. It makes the machine diﬃcult toit = σ (Wi [yt−1 , xt ] + bi ) ,make use of them to build relationship between theγ̃t = tanh (Wc [yt−1 , xt ] + bc ) ,(1) subject and the description. To help the machine
γt = ft ∗ γt−1 + it ∗ γ̃t ,to recognize the relationship between them, we madeot = σ (Wo [yt−1 , xt ] + bo ) ,a script to concatenate the subject word and eachsentence in the description for each document in theyt = ot ∗ tanh (γt ) ,glossary.
where, σ (·) and ∗ are the element-wise sigmoid function and multiplication operator.

― 561 ―

Copyright(C) 2018 The Association for Natural Language Processing.
All Rights Reserved.

4Experiments

4.1

PurposeTable 1. The Performance Measured by MAP@K.
“DA” refers to data augmentation. “LC” refers tolearning rate controlling.

To validate the eﬀectiveness of our proposed method,
we compared its performance for the implicit keyword
extraction task with the baselines. We investigated
the ranking scores of the candidate keywords ranked
by diﬀerent methods, and compare them with thegold standards. Besides, in order to validate the dataaugmentation method alone, we also applied it to Elman network and compared it with the vanilla Elmannetwork.

4.2

Setup

The knowledge resource we used was 3 history textbooks, 1 glossary (7,038 terms included), and 6 exercise books (1,205 questions includes). The textbooks
include “世界史 A”, “新選世界史 B” and “世界史 B”
published by Tokyo Shoseki. The glossary is “世界史B 用語集改訂版” published by Yamakawa Shuppansha. The exercise books include “詳説世界史論述問題集” published by Yamakawa Shuppansha, “実力をつける世界史１００題改訂第３版” and “段階式世界史論述のトレーニング” published by Z-kai, “判る解ける書ける世界史論述” published by Kawai Publishing, “大学入試世界史 B 論述問題が面白いほど解ける本” published by KADOKAWA/Chukei, and “世界史論述練習帳 new” published by Parade. The text
book and the glossary were employed to pretrain theword embeddings, with data augmentation describedin Section 3.3. The question and answers pairs in theexercise books were used to train the ranking models.
For the test set, we used the first question on world
history of the admission test of Tokyo University from2000 to 2011 (totally 12 questions).

4.3

Method@5@10

Baseline(TF-IDF) [7]

0.522

0.433

Baseline(BM25) [7]

0.493

0.425

Baseline(Elman net) [7]

0.594

0.572

Proposed(Elman net+DA)

0.712

0.643

Proposed(BiLSTM+DA)

0.732

0.651

Proposed(BiLSTM+DA+LC)

0.729

0.675

Table 2. The average hits with/without learning
rate controlling. “DA” refers to data augmentation.
“LC” refers to learning rate controlling.
Method@5@10

Proposed(BiLSTM+DA)

2.083

3.667

Proposed(BiLSTM+DA+LC)

2.333

4.08

5

Results and Discussion

5.1

Results

The results are shown in Table 1. The proposed
method achieved the best performance. The Elmannetwork with data augmentation outperformed thebaseline Elman network. Learning rate controlling
improved the MAP@10 of the RNN of LSTM, butfailed to improve the MAP@5 of it. It is an interesting result, hence we additionally compared the average hits of the keywords with and without learning rate controlling as shown in Table 2. We findthat even though there is not significant diﬀerencebetween MAP@5 with or without learning rate controlling, the learning rate controlling brings more hitsof correct keywords at top 5 and at top 10.

MetricBecause we want to investigate whether the methods are able to rank the correct keywords as highas possible, we use Mean Average Precision at K(MAP@K) [8] as the metric. To observe whether thekeywords are gathered at the top, and for ease ofcomparison between the proposed methods and thereported performance of the baselines [7],5.2Discussion

The results show that the proposed method is moreeﬀective than the conventional ones. The improvement of Elman network brought by data augmentation in the results indicates that data augmentationis very powerful to improve the performance. The
improvement of the average hits by learning rate controlling shows that it is eﬀective to make the system
to highly rank more correct keywords.

― 562 ―

Copyright(C) 2018 The Association for Natural Language Processing.
All Rights Reserved.

Acknowledgement

Fig. 2. An output by the proposed method.

The authors would like to thank IR-Advanced Linguistic Technologies, Inc. for providing us the gold
standards, and the students in Yokohama National
University for helping us computerize the datasets.

ReferencesFig. 3. An output by Elman net.
As a qualitative validation of the bi-direction RNNof LSTM model, we also compared the outputs by
the proposed model with the others. Fig. 2 and Fig. 3
show examples of the outputs by the proposed modeland the Elman network respectively. We can see thatthe output by the proposed method is full of meaningful content words of historical entities and events,
while Elman network outputs some frequent but notcritical words such as “的”, “一方” and “後”. Moreover, we can see that the scores of the critical keywords such as “オスマン帝国” and “英仏” are higher.
The findings indicate that the bi-directional RNN ofLSTM can achieve better rankings.

6Conclusions

For learning and ranking the implicit keywords forhistorical essay questions, we proposed a bi-directional
RNN of LSTM, a data augmentation method anda learning rate controlling trick. The experimental results indicate that the bi-directional RNN ofLSTM can oﬀer better rankings, the data augmentation method obviously improves the performance andthe learning rate controlling makes the bi-directional
RNN model extract more critical keywords.
We believe that there is still room for improvement. For example, transfer learning is expected tohelp the scenes where the data is limited. We are going to explore such learning technology for this task
in the future. Besides, the tunning of the parameters of the stochastic gradient descending optimizer
was diﬃcult for the objective of the proposed model.
We also would like to explore other suitable objectivefunctions and optimizers for the task.

[1] Karl Moritz Hermann, Tom Koisk, EdwardGrefenstette, Lasse Espeholt, Will Kay, Mustafa
Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. In Proceedingsof the 28th International Conference on NeuralInformation Processing Systems, (2015).
[2] Hideyuki Shibuki, Kotaro Sakamoto, Yoshionobu Kano, Teruko Mitamura, Madoka Ishioroshi, Kelly Y. Itakura, Di Wang, TatsunoriMori, and Noriko Kando. Overview of theNTCIR-11 QA-Lab Task. NTCIR-11, (2014).
[3] Hideyuki Shibuki, Kotaro Sakamoto, Madoka
Ishioroshi, Akira Fujita, Yoshionobu Kano, Tatsunori Mori, and Noriko Kando. Overview of theNTCIR-12 QA Lab-2 Task. NTCIR-12, (2015).
[4] Hideyuki Shibuki, Kotaro Sakamoto, MadokaIshioroshi, Yoshinobu Kano, Teruko Mitamura,
Tatsunori Mori, and Noriko Kando. Overviewof the NTCIR-13 QA Lab-3 Task. NTCIR-13,(2017).
[5] Hans Peter Luhn. A statistical approach tomechanized encoding and searching of literaryinformation. IBM Journal of research and development, 1(4), 309-317, (1957).
[6] Stephen E. Robertson, Steve Walker, Susan
Jones, Micheline Hancock-Beaulieu, and Mike
Gatford. Okapi at TREC-3. Proceedings of theThird Text REtrieval Conference (TREC 1994),(1994).
[7] 阪本浩太郎, 陸宇傑, 福原優太, 渋木英潔, 石下円香, 森辰則, 神門典子. 大学入試世界史論述問題における非指定重要語句生成に関する検討. 言語処理学会第 23 回年次大会, (2017).
[8] Nick Pentreath, Manpreet Singh Ghotra, andRajdeep Dua. Machine Learning with Spark.
Packt Publishing Ltd., 109-113, (2015).

― 563 ―

Copyright(C) 2018 The Association for Natural Language Processing.
All Rights Reserved.