Captioning Events in Tourist Spotsby Neural Language Generation
Mai Nguyen1, Koichiro Yoshino1,2, Yu Suzuki1,3, Satoshi Nakamura1
1Nara Institute of Science and Technology
2Japan Science and Technology Agency
3Gifu University

14 June 20192019 @ Mai Nguyen, AHC Lab, NAIST

1

Outline• Motivations• Related works• System Architecture• Experiments• Conclusion

14 June 20192019 @ Mai Nguyen, AHC Lab, NAIST

2/23

Motivations
Fushimi Inari is very crowded, you
should go there in early morning

Language
Generation

Natural Language Generation (NLG) will be a good interface to output info. for user
• User can understand information immediately• Giving textual description is one of the easiest ways to present information
Data sources
14 June 20192019 @ Mai Nguyen, AHC Lab, NAIST

3/23

Objectives
The task of generating textual descriptions about tourist spots to support user decision
making in tourist domain:
Important factors:
1. Informativeness2. Naturalness

14 June 20192019 @ Mai Nguyen, AHC Lab, NAIST

4/23

Related works
The task of generating text from data has
been investigated in many domains
1. Weather forecast (Belz et al., 2008)
2. Navigation assistance (Dale et al., 2003)
3. Sports (Liang et al., 2009)
4. Market Comments (Murakami et al., 2017)
NLG approaches:
Rule-based: Define a set of rules to map frames to NL
Pros: simple, error-free, easy to control
Cons: time-consuming, poor scalability=> Neural-based NLG

14 June 2019

Pros: Access millions of traveler reviews
Cons: Out-of-date information

2019 @ Mai Nguyen, AHC Lab, NAIST

5/23

This paper introduces …
• A system for producing textual description about tourist attractions
ü Integrate a neural language generationü Summarize a multiple data resources such as infrared sensor, social
media, user’s check-in
ü coupling with backend-server, we developed a real-time application
with up-to-date information.

14 June 20192019 @ Mai Nguyen, AHC Lab, NAIST

6/23

NLG Pipeline

How to say

What to say

Language model

User’s check in

Social media

Language Generator

Congestion
Sentences
Semantic representation
14 June 20192019 @ Mai Nguyen, AHC Lab, NAIST

7/23

System Architecture
Neural language
generator

How-to-say

Pushing notification server

Backend system

Mobile application

What-to-sayData sources

14 June 20192019 @ Mai Nguyen, AHC Lab, NAIST

8/23

What-to-say• Backend server: extract and summarize data from several information
sources• Infrared sensor, Twitter, user’s check-in• Pushing server:• Check new-update information from backend with a specific time
• Create a semantic expression, N-hot vector of frames that have slot valuesexpression.
• Send the pushing request to the back-end server

14 June 20192019 @ Mai Nguyen, AHC Lab, NAIST

9/23

How-to-say, Neural Language Generation• Utilize semantically controlled LSTM,
proposed by [Wen et al, 2015]• Core Idea: using a gate mechanism tocontrol the generated semantics(dialog act/slots)• Works well in limited domain, if the“what to say” is decided
14 June 20192019 @ Mai Nguyen, AHC Lab, NAIST

10/23

Experiments
The target application is a system that describes comprehensive, up-todate information about tourist attraction by NLG
Evaluate usefulness of the
Evaluate generated
system by traveler using thesentences by automatic
application
metrics

Collect data

14 June 2019

Build NLG
models2019 @ Mai Nguyen, AHC Lab, NAIST

Integrate NLG
in the realtime system

11/23

Dataset
Attribute

Data Type

Example Value

Data Sources

nameverbatim string

kyoto tower

POISeventverbatim stringcherry blossom, gion
festival

official websites

state_event

dictionary

happening, finishedofficial websites

crowdeddictionary

high, low, average

infrared sensors

recommended

booleanyes/norecommendation system

time

enumerable

now, holidays

popularbooleanyes/nousers check-in

List of possible attributes in the our dataset, data sources and examples
14 June 20192019 @ Mai Nguyen, AHC Lab, NAIST12/23

Data collection
1. Corpus: A pair of meaning representation (MR) and corresponding references
pairs in the tourist domain.
• Meaning representation (MR): a set of pairs key-value pairs• Reference: natural language utterance describing MRMR

name=[Fushimi Inari], crowded=[high], time=[festival days], recommended=[no]Reference

It is a good idea to avoid Fushimi Inari during festival days because it is extraordinary
crowded.

2. Human annotation: workers were recruited via crowd sourcing service.
3. Results: ~ 3300 data points collected in English

14 June 20192019 @ Mai Nguyen, AHC Lab, NAIST

13/23

Pre-process dataset• Remove useless instances: which provide not enough information(Informativeness)• Grammar correction: detected by Grammarly and fixed manually, ~ 15%.
(Naturalness)• Delexicalization: some properties such as name and event to avoid data sparsity.
"name[maizuru park], event[cherry blossoms], event_state[happening], crowdedness[2]""maizuru park is not very crowded, the cherry blossoms festival is currently held there."

X-name is not very crowded , the X-event festival is currently held there .

14 June 20192019 @ Mai Nguyen, AHC Lab, NAIST

14/23

Evaluation of NLG• Corpus is split into training, validation, test set in the ratio 85%:6%:9%• Training:Training

Validation

Test

REF

2800

200296

MR283

8083

• character-based version of SC-LSTM (hidden size 1024, batch size 256)
• Adam optimizer,• Dropout,• Beam search: beam size = 10

• Baseline: rule-based• Metrics: BLEU, NIST, METEOR, ROUGE-L, CIDEr
14 June 20192019 @ Mai Nguyen, AHC Lab, NAIST

15/23

Result• Automatic metrics
SystemBLEU

NISTMETEOR

ROUGE-L

CIDEr

Rule-based

0.41

5.67

0.36

0.67

2.23

Our generator

0.43

6.03

0.37

0.64

2.70

Improvement:
ü BLEU: 0.02
ü NIST: 0.36
ü METEOR: 0.01
ü CIDEr: 0.47

14 June 20192019 @ Mai Nguyen, AHC Lab, NAIST

16/23

Examples of generated text
N. of slots
345

5

Example of description generated corresponding to MR
name[nanzenji temple], event[autumn leaves], popular[yes]Autumn leaves in nanzenji temple is popular.
name[Yoyogi Park], crowded[no], time[now], recommended[no]Yoyogi Park is not crowded right now, but it is not recommended to visit.
name[yanagidani kannon], event[autumn leaves], state[happening], crowded[high],recommended[no]Autumn leaves is happening in yanagidani kannon, it is extremely crowded, you should not gothere.
name[kyoto imperial palace], event[aoi festival], state[happening], crowded[low],recommended[yes]Aoi festival is happening in kyoto imperial palace, it is medium crowded, you should go there.

The NLG model can produce fluent text and would often repeat the same sentence structure
multiple times
14 June 20192019 @ Mai Nguyen, AHC Lab, NAIST

17/23

Results analysis
Human Evaluation - in term of informativenessNo. slots

Total of MR

Number of correct
generated text

Percentage

318

1794.4%

4

54

49

90.7%

5115

45.4%

Total

83

7085.54%

Neural LG have difficulty capturing long-term structure MRs

14 June 20192019 @ Mai Nguyen, AHC Lab, NAIST

18/23

Experiment in Real Field• Place: Arashiyama area, Kyoto• Participants• 12 students• 19 ordinary people• Task descriptions• Use the application during their trip
• Visit at least 3 POIs from 10 AM to3 PM• Answer the questionnaire
14 June 20192019 @ Mai Nguyen, AHC Lab, NAIST

19/23

Application

14 June 20192019 @ Mai Nguyen, AHC Lab, NAIST

20/23

Experimental Results
Evaluate usefulness of the pushing function in range 1-5:Answers

Experiment

Ratio of satisfied

543

2

1Students

115

1458%

Ordinarypeople

2133160 %14 June 20192019 @ Mai Nguyen, AHC Lab, NAIST• 5: Very satisfied• 4: Rather Satisfied• 3: Satisfied• 2: Not Satisfied• 1: Not Satisfied at all

21/23

Conclusions• Our generation model produces fluent text overall, work smoothly on
real-time application.
• In term of usefulness, our system could support decision-marking fortourists.
• Limitations:• Neural LG have difficulty capturing long-term structure because the training
data is quite small• The application’s notification was too much for travelers.

14 June 20192019 @ Mai Nguyen, AHC Lab, NAIST

22/23

Reference
1. Tsung-Hsien Wen, Milica Gasic, N. M. P. S. D. V. and Young, S.: Semantically Conditioned LSTM-based Natural Language
Generation for Spoken Dialogue Systems, Proceedings of 2015 Conference on Empirical Methods in Natural Language
Processing(EMNLP), pp. 1711-1721 (2015).
2. Sripada, S. G., Reiter, E. and Davy, I.: SUMTIMEMOUSAM: Congurable Marine Weather Forecast Generator (2003)
3. Belz, A.: Automatic Generation of Weather Forecast Texts Using Comprehensive Probabilistic Generation-space Models,Nat. Lang. Eng., Vol. 14, No. 4, pp. 431-455 (online), DOI: 10.1017/S1351324907004664 (2008).
4. Dale, R., Geldof, S. and Prost, J.-P.: CORAL: Using Natural Language Generation for Navigational Assistance, Proceedings ofthe 26th Australasian Computer Science Conference - Volume 16, ACSC ‘03, Darlinghurst, Australia, Australia, AustralianComputer Society, Inc., pp. 35{44 (online), available from (2003)
5. Liang, P., Jordan, M. and Klein, D.: Learning Se-mantic Correspondences with Less Supervision, Proceedings of the Joint
Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language
Processing of the AFNLP, Suntec, Singapore, Association for Computational Linguistics, pp. 91-99 (2009).

14 June 20192019 @ Mai Nguyen, AHC Lab, NAIST

23/23