情報処理学会研究報告
IPSJ SIG Technical Report

Captioning Events in Tourist Spots by Neural
Language Generation
Mai Nguyen1,a)

吉野 幸一郎1,2,b)

鈴木 優1,3

中村 哲1

Abstract: We present an application that captions events in tourist attractions by summarizing various
information sources in natural language descriptions. The system is divided into two parts: what-to-say,which summaries information into structured data, and how-to-say, which produces natural language captions from input meaning representation. In what-to-say, information from several information sources,
such as infrared sensors and social media, are extracted into a semantic frame. In how-to-say, we utilized
semantically-conditioned long short-term memory neural networks to generate natural language captions forgiving information to users in an understandable way. An empirical evaluation of the system shows thequality of generated text across five automated metrics. The generated sentences are used in the application
system for helping visitors. The subjective evaluation shows the usefulness of the proposed system.

1. Introduction

erally, we propose to divide the generation process into
two parts: what-to-say, how-to-say. What-to-say extracts

Tourism has become a promising sector in the world.

contextual information from various information sources

These days, tourists decide their travel plan based on

into a meaning representation (MR). How-to-say gener-

many factors of tourist spots such as other user reviews,

ates a textual output from the MR based on a neural lan-their congestion, and popularity. However, many infor-

guage generation (NLG) model, particularly semantically-mation sources are changing every moment; it makes theconditioned Long Short-term Memory (SC-LSTM) [1].

decision making diﬃcult. Generally, the data available

An example is shown in Figure 1. We also combine thefor tourist consists of (i) continuous variables (e.g., con-

NLG with beam search and a n-best list reranker to sup-gestion, etc.) and (ii) discrete variables (e.g., events, thepress irrelevant information in the output.

information in social media, etc.). Hence, eﬃciently presenting these types of data is crucial for informed travel
decision making. In some circumstances, giving a textual

name[Fushimi Inari],
crowded[high],MRtime[festival days]

caption about a tourist spot is more likely to help travel-

recommended[no]ers for making better decisions since the text summary is
one of the easiest ways to understand information.
In this study, we address the task of generating textual

It is a good idea to avoid Fushimi Inari
Reference

during festival days as it is extraordinarily crowded.

descriptions about tourist spots to support user decision
making. The data of each Point of Interest (POI) such

図 1 MR とそれに付与されたレファレンスの例．MR は key-value

as congestion or events are observed from multiple dataのペアの組で表現され，レファレンスは MR に記載された内sources such as oﬃcial website, infrared sensors. Gen-

容を説明する観光スポットに関する説明文．

Fig. 1 An example MR and reference. MR is a set of key value
1

2
3
a)
b)

奈良先端科学技術大学院大学 先端科学技術研究科
生駒市高山町 8916-5
科学技術振興機構, さきがけ
現在、岐阜大学 工学部 所属
nguyen.quynh mai.nm9 at is.naist.jp
koichiro at is.naist.jp

c 2019 Information Processing Society of Japan⃝

pairs, a reference is a caption in tourist spot describes
the MR.

We have constructed an application for helping travelers with information of POIs, which changes moment by1

情報処理学会研究報告
IPSJ SIG Technical Report

moment, and integrated the NLG system as an interface

dation feed. TripAdvisor *1 has developed a system con-to access information. We build a personalized recom-

sisting of reviews from users. The recommendations are

mendation feed in pushing function to notify the latest

generated based on the stored reviews; however, the real-

information of POIs, places that travels are interested in,time changes are not considered in the application. We re-with generated descriptions.

alize a system that can describe real-time events in tourist

We conducted two experiments: the quality checking
of generated sentences and human subjective evaluation
about pushing notification. The first experiment is run ona crowd-sourced dataset in the context of an application

spots by utilizing generation based captioning and a realtime data update system.

3. System Architecture

describing sightseeing spots in Japan. We show that the

The goal of our system is building a system that can de-proposed generation system increases the quality of gener-

scribe what happens in tourist spots to users in real-time.

ated text across five diﬀerent automated metrics (BLEU,

We built a system that receives a variety of information

NIST, METEOR, ROUGE, and CIDEr) over rule-basedsources and then generates a description to notify users.

baseline models. In order to assess the subjective perfor-

Our system consists of three modules: back-end system in-mance of our system, an experiment with users using acluding event extraction (what-to-say), natural language

mobile application integrated our NLG system was con-generation (how-to-say), and pushing notification server.

ducted. The result shows the benefits of our proposed

An overview of the system was given in Figure 2 and its

system toward users during their trip.

components are described as follows.

We introduce the overall architecture of the proposed• Back-end system stores data observed from vari-system and its components, such as pushing notifications

ous data source such as infrared sensors, social me-that use the generated outputs and neutral language gen-dia, user’s application. The back-end system also ex-eration in Section 3. Section 4 details our experiments oftracts values from these data to fill states defined in

the generation generation system with automatic evalu-

a meaning representation. Once the back-end sys-

ation, Section 5 describes experiments in real fields. We

tem receives a notification request, the system sends
a pushing to specified mobile applications.

summarize related works by using smartphone applicationin Section 2 and oﬀer conclusions in Section 6.

• Pushing notification server asks POIs update to
the back-end system, summarizes updates into MR,

2. Related works

sends a generation request to the neural language gen-

The task of generating text from has structured dataeration system, and sends a pushing request to thebeen investigated in many domains such as weather fore-

back-end system with the generated sentence.

cast [2] [3], navigation assistance [4], sports [5]. Tradition-• Neural language generation system generates aally, approaches to such studies used handcrafted rules [2]sentence according to the MR given by the pushingor are templates. Despite its robustness and adequacy,

notification server.

the approach does not easily scale to large other domains
and remains repetition issue.

Our proposed parts of the system include two main components: i) pushing notification server and ii) the natural

Systems based on neural network (neural-based) at-

language generator integrated into the server. We apply atempt to learn generation directly from data, which en-

character-based version of semantically-conditioned Long

ables the system for generating descriptions more natu-

Short-term Memory proposed by Wen et al. [1] as ourrally, removes the dependency on the predefined rules.

generator, which has a gate to control the generated se-

Neural-based models have also been successfully applied

mantics. Some details are updated to fit our domain. A

to various data-to-text generation tasks. More recently,

pushing notification server is a bridge element to encapsu-

there has been some work on neural-based models forlate various information into one MR once it received newgenerating a description such as reviews from product at-

information related to attractions. The MR is sent to thetributes [6] and comments on markets [7]. However, thegenerator to produce a description in natural language.

model of Wen et al. [1] takes only a short time to pro-

The generated caption is sent in the push request to theduce text and easier for building and extending to otherback-end server and then notify on a mobile application

domains.
There also been some work on personalized recommen-c 2019 Information Processing Society of Japan⃝*1

www.tripadvisor.com

2

情報処理学会研究報告
IPSJ SIG Technical Reportto corresponding users.

MeasureDescription

BLEU

Calculate precision as ratio of cor-

3.1 Pushing Notification

rectly generated n-grams.

Pushing notifications are an important feature, retain
NIST

and re-engage users and monetize on their attention. To

Extension of BLEU score, weights then-grams informativeness.

enhance user’s experience, we integrate pushing notifica-

We usedfour-gramtion server with our natural language generator in our

METEOR

Based on the harmonic mean andtarget application, which delivers comprehensive, up-to-

computes F-measuredate information about POIs. On the other hand, the

ROUGE-L

Recall-based measure which compares
the longest common subsequence

server also takes responsibility for the what-to-say part,
CIDEr

which summarizes information about POIs.

Based on TF-IDF scoring, compares
similarity of n-grams to majority of

We designed the pushing notification server with two
main functions.

We usedfour-gramreferences

First, the server checks the back-end

database with a specific period. If the system detects any

表 1 各自動評価手法の説明．

newly updated information about POIs hold by users, theTable 1 Short descriptions of automated metrics.

system will produce input MRs corresponding to new information, generate descriptions from the MRs by using

4.1 Data collection

the neural language generation system and send push request within the report to the back-end system.

We collected data that has pairs of a MR (frame) andits captioning sentence. Each MR includes an unordered
list of slot (or attribute) and value pairs. The captioning

3.2 Neural Language Generation
We utilized Semantically Conditional Long Short-term
Memory Network (SC-LSTM) proposed by (Wen et al.,sentence is a natural language description corresponding
to the MR, possibly consisting of several sentences. An
example of the corpus is shown in Figure 1

2015b) as our generator, which has gates to control thegenerated semantics.

For data collection, we prepared a set of MRs containing seven attributes. The particular attributes used are

We give a brief overview of SC-LSTM model and re-provided in Table 2 with their data types. The order of at-

fer to the original paper [1] for an in-depth description.

tributes in MR was randomly selected and then fed to the

The model extends the original LSTM [8] cell by adding

crowd workers. Workers were shown each MR and asked

a control cell in charge of sentence planning as a frame

to enter an appropriate utterance in natural English cor-that has some slot-values. The frame is the input of the

responding to the MR. The crowd workers are not primed

SC-LSTM, converted to a one-hot encoded MR vector,

by ordering used.

which represents the value for each property. This cellmanipulates the MR-vector during the generation process

Attribute

Data Type

Example value

to ensure the information is fully rendered in the utter-

nameverbatim string

Kyoto Tower

ance. The cell acts as a forget gate keeping track of whatevent

verbatim stringcherry blossom, gion festival

stateEventdictionary

happening, finished

crowdeddictionary

high, low, average

recommended

booleanYes / Notime

enumerable

now, holidays

popularboolean

Yes/Noinformation should be retained for future time steps anddiscards the others.

4. Experiments of Generation
The experiment was designed to investigate whether

表 2 ドメインが持つスロットとその取り得る値．

our neural language generation system can generate aTable 2 Domain attributes and attribute types.

high-quality description of certain spots in the tourist domain. To train the generator, we first collected data froma human via a crowd-sourcing platform. To assess the

We used Figure-eight *2 platform to set up our anno-

eﬀectiveness of our generation system, we employ vari-tation process and to access an online workforce. Aroundous reference-based metrics on the test set: BLEU [9],3300 utterances were collected from crowd workers. After

ROUGE-L [10], METEOR [11], NIST [12], CIDEr [13]processing and grouping each reference according to its

score. Table 1 shows description of the metrics above.

*2

c 2019 Information Processing Society of Japan⃝

https://www.figure-eight.com/

3情報処理学会研究報告
IPSJ SIG Technical Report

図 2 提案システムの概要．Pushing notification server は Backend system に保管された情報を MR へと変換し，Neural language generator へ送信する．Generator の出力はイベントの説明としてサーバに送られる．

Fig. 2 Architecture of the proposed system. Pushing notification server summarize information into MR vector as input to the generator. The output of generator issent to main server to notify to users as notification of event description.

combined attributes, we obtained 283 distinct MRs. On

references are unseen in the training dataset and used for

average, each MR consists of 4 slots, and there are 11.6

automatic evaluation of the generator. We implement ourdiﬀerent RF for each MR.

character-based version of SC-LSTM in the TensorFlow
frame-work [14]. During training, we train models with

4.2 Experiment settings

Adam optimizer [15] to minimize the cross-entropy as the

Since our model is working on a character level, we treat

loss function. A similar dropout method was adopted byeach reference as a sequence of character, where each char-[16] who use the same dropout mask for inputs, outputs,

acter is represented as a 1-hot content vector. To avoid

and hidden layers at each time step. Based on a few pre-data sparsity, we substituted some properties value suchliminary experiments, we use a hidden state of size 1024

as name and event to named entity markers. For example,

for LSTM cell with a batch size of 256 and a maximum

name and event values are replaced with tokens ’X-name’sequence length of 256.

and ’X-event’ respectively.
表 3

MR およびレファレンス文の総数．

Table 3 Data split by number of references and number of

4.3 Results
We compared our generator with a rule-based generator
and reported the scores for the automatic evaluation, in-MRs.
Training

Validation

Testingcluding the metrics BLEU, ROUGE-L, METEOR, NIST,

REF

2800

200

296

and CIDEr scores by comparing with references in the

MR

283

80

80

testing set. Table 4 shows that our model performs better
than the rule-based method; the neural-based generator
archives almost the best scores. In particular, our model

The collected data is split into a training, validation,and testing set in the ratio 85%:6%:9% (refer to Table 3
for the data-spit in details). The validation and test data
are multi-references; the validation set has, on average, 8.1
references for each MR. A separate test set with previously

outperformed the baseline model with an improvement of
0.2 points in BLEU, 0.36 in NIST, 0.47 in CIDEr. In
terms of fluency, most of the captions generated by ourgenerators show that the models learn to produce fluent
sentences in domain style. In terms of informativeness,

unseen combinations of attributes contains 80 MRs and its

c 2019 Information Processing Society of Japan⃝4

情報処理学会研究報告
IPSJ SIG Technical Report

We collected two groups of experiment participants: 12
students and 10 ordinary people who have never been to
Arashiyama area. We conducted two experiments for eachgroup. Any participants are requested to use the application during their travels and visit at least three POIs byconsidering application suggestions and their preference,from 10 AM to 3 PM.
5.2 Experimental Results
After the experiment, we requested them to evaluate
the usefulness of the pushing function of the applicationin 5 degree as follows.
• 5: Very Satisfied• 4: Rather Satisfied
図 3

提案システムの画面．

Fig. 3 Screen shots of the proposed application system.

• 3: Satisfied• 2: Not Satisfied• 1: Not Satisfied at All

while some outputs show enough information once the in-

The number of participants selected each score is shown

put is short, some outputs are not entirely informative:

in Table 6. From the result, about 60% of participants are

missing or incorrect information (see Table 5).

satisfied by the application. However, the remaining peo-

5. System Evaluation in Real Field
We conducted experiments to evaluate the overall system in the real field with users by informing information
updates. Once the system received new information related to the user interests, the system notifies them tople are not satisfied. In questionnaires from users, someusers said that the number of pushing was too much, andthe timing was not satisfactory. We need to improve thispoint in future systems.

6. Conclusion and Future Workusers as pushing of description in natural language. ThisWe presented a real-time system for generating a de-section describes the details of the experimental settings

scription of tourist attractions that refers to multiple re-and results.

sources. Our automatic showed that our generation model
produces fluent text overall, enhances the quality of tex-

5.1 Experiment Setting

tual description in the tourist domain. In term of hu-

As described in Figure 2, information of any POIs andman evaluation, our system could support decision mark-users were written on the backend system by their recog-

ers during their trip. Finally, a number of potential lim-

nition system asynchronously. Our pushing notification

itations need to be considered. First, the neural-based

server checked the backend database in every five min-language generators have diﬃculty capturing long-term

utes. If the system found any updates about POIs held

structure. Second, the user’s personalized travel feed no-by the user or POIs recommended by the system, the sys-

tify them too often, we need to consider the pushing tim-

tem generated sentences and sent a push request to theing. In the future, we would like to investigate these latterbackend system. We did not control the time of pushingaspects in this field.

in our experiment; the pushing notification was sent immediately when the backend system received the pushing

参考文献

request from the pushing notification server.

[1]

We conducted the experiment at Arashiyama area in
Kyoto. 100 POIs were selected preliminary and registeredon our application system, as shown in Figure 3. Congestion of each POI is recognized by infrared sensors or human observers in three degrees (not crowded, moderately
crowded and highly crowded).

c 2019 Information Processing Society of Japan⃝[2]

Tsung-Hsien Wen, Milica Gasic, N. M. P. S. D. V. andYoung, S.: Semantically Conditioned LSTM-based Natural Language Generation for Spoken Dialogue Systems,
Proceedings of 2015 Conference on Empirical Methods
in Natural Language Processing(EMNLP), pp. 1711–
1721 (2015).
Sripada, S. G., Reiter, E. and Davy, I.: SUMTIMEMOUSAM: Configurable Marine Weather Forecast Generator (2003).

5

情報処理学会研究報告
IPSJ SIG Technical Report
表 4

各自動評価手法の結果．

Table 4 Results of the automatic metrics.
SystemBLEU

NISTMETEOR

ROUGE-L

CIDEr

rule-based

0.41

5.67

0.36

0.67

2.23

our generator

0.43

6.03

0.37

0.64

2.70

表 5 テストセットから提案法により生成されたキャプションの例（赤字は誤った情報の出力）．

Table 5 Sample of event caption generated by our generator system from the test set.
Errors are marked in color(incorrect information)
#
1
2
3
4
5

Example of descriptions about tourist spots in Japan
name[yodo river park], event[cherry blossoms], state[happening], recommended[no]cherry blossoms is upcoming in yamaga castle park , you should not go there .
name[Yoyogi Park], crowded[no], time[now], recommended[no]Yoyogi Park is not crowded right now, but it is not recommended to visit.
name[Akihabara], crowded[low], time[festival days], recommended[no]Akihabara is slightly crowded during festival days, so it is not recommended to visit.
name[kyoto imperial palace], event[aoi festival], state[happening], crowded[low], recommended[yes]Aoi festival is happening in kyoto imperial palace, it is medium crowded, you should go there.
name[Shibuya], crowded[high], time[morning], recommended[yes]Shibuya is extremely crowded in the morning, but it is recommended to visit.

表 6 主観評価の結果．質問の解答番号に回答した人数を示している．

Table 6 Results of subjective evaluation. Numbers of answers
associated to the question for pushing function are[7]

shown.
Experiment

Answers

Ratio of satisfied

5

4

3

2

1

Students

1

1

5

1

4

58%

Ordinary people

2

1

3

3

1

60%

[8]
[3]

[4]

[5]

[6]

Belz, A.: Automatic Generation of Weather Forecast
Texts Using Comprehensive Probabilistic Generationspace Models, Nat. Lang. Eng., Vol. 14, No. 4, pp. 431–
455 (online), DOI: 10.1017/S1351324907004664 (2008).
Dale, R., Geldof, S. and Prost, J.-P.: CORAL: Using Natural Language Generation for Navigational
Assistance, Proceedings of the 26th Australasian Computer Science Conference - Volume 16, ACSC ’03,
Darlinghurst, Australia, Australia, Australian Computer Society, Inc., pp. 35–44 (online), available from⟨http://dl.acm.org/citation.cfm?id=783106.783111⟩(2003).
Liang, P., Jordan, M. and Klein, D.: Learning Semantic Correspondences with Less Supervision, Proceedings of the Joint Conference of the 47th Annual
Meeting of the ACL and the 4th International Joint
Conference on Natural Language Processing of the
AFNLP, Suntec, Singapore, Association for Computational Linguistics, pp. 91–99 (online), available from⟨https://www.aclweb.org/anthology/P09-1011⟩ (2009).
Dong, L., Huang, S., Wei, F., Lapata, M., Zhou,
M. and Xu, K.: Learning to Generate Product Reviews from Attributes, Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long
Papers, Valencia, Spain, Association for Computa-

c 2019 Information Processing Society of Japan
⃝

[9]

[10]

[11]

[12]
[13]

[14]

tional Linguistics, pp. 623–632 (online), available from⟨https://www.aclweb.org/anthology/E17-1059⟩ (2017).
Murakami, S., Watanabe, A., Miyazawa, A., Goshima,K., Yanase, T., Takamura, H. and Miyao, Y.: Learning to Generate Market Comments from Stock Prices,
Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1:
Long Papers), Vancouver, Canada, Association for Computational Linguistics, pp. 1374–1384 (online), DOI:
10.18653/v1/P17-1126 (2017).
Hochreiter, S. and Schmidhuber, J.: Long Short-Term
Memory, Neural Comput., Vol. 9, No. 8, pp. 1735–1780(online), DOI: 10.1162/neco.1997.9.8.1735 (1997).
Kishore Papineni, Salim Roukos, T. W. and Zhu, W.:
Bleu: a method for automatic evaluation of machinetranslation, Proceedings of the 40th annual meeting on
association for computational linguistics, p. 311318.
Lin, C.-Y.: Rouge: A package for automatic evaluationof summaries, Text summarization branches out: Proceedings of the ACL-04 workshop (2004).
Denkowski, M. and Lavie, A.: Meteor universal: Language specific translation evaluation for any target language, Proceedings of the ninth workshop on statisticalmachine translation., pp. 376–380 (2014).
Doddington, G.: Automatic evaluation of machine translation quality using n-gram coccurrence statistics.
Ramakrishna Vedantam, C. L. Z. and Parikh, D.: Cider:
Consensus-based image description evaluation, Proceedings of the IEEE conference on computer vision andpattern recognition, pp. 4566–4575 (2015).
Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A.,
Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard,
M., Kudlur, M., Levenberg, J., Monga, R., Moore, S.,
Murray, D. G., Steiner, B., Tucker, P., Vasudevan, V.,
Warden, P., Wicke, M., Yu, Y. and Zheng, X.: TensorFlow: A system for large-scale machine learning, 12th
USENIX Symposium on Operating Systems Design and6

情報処理学会研究報告
IPSJ SIG Technical Report

[15]
[16]

Implementation (OSDI 16), pp. 265–283 (2016).
Kingma, D. P. and Ba, J.: Adam: A Method for Stochastic Optimization, arXiv e-prints (2014).
Gal, Y. and Ghahramani, Z.: A Theoretically Grounded
Application of Dropout in Recurrent Neural Networks,
arXiv e-prints, p. arXiv:1512.05287 (2015).

c 2019 Information Processing Society of Japan⃝

7