言語処理学会 第25回年次大会 発表論文集 (2019年3月)

Vision Mediated Story Generation
Hong Chen Rapael Shu

Noriki Nishida

Hideki Nakayama

東京大学 情報理工研究科{chen, raphael, nishida, }@nlab.ci.i.u-tokyo.ac.jp
nakayama@ci.i.u-tokyo.ac.jp

1 Introduction
Story generation is the task of automatically generating
a complete story.For this task, the dataset is especially important. Looking back to our childhood, after we could
understand each word’s meaning, it still took a long time
for us to achieve the ability to write a good story. According to this observation, a large dataset is undoubtedly
necessary. However, unfortunately, the story data is hard
to be obtained. So in this work, we propose a unsupervised method to train the story generation model without
any story data.
On the other hand, diﬀerent from the novel, the storybooks for children contain many pictures and we callthem the picture-book. It means that the image information would help children to understand the context of thestory. The case of picture-books tells us that visual information might play a significant role in a story. For
another example, when people start writing a story, theywill imagine the pictures of the next event in their minds
and then write down what happened in the image. So in
this view, we state that the image information is indispensable in the open story generation task.
To summary our contribution:

Furthermore, [6] used event as mediation to generatestory. They first extracted the event from the sentence,then predicted the next event and transfered the event sentence into the story sentence as the next generated sentence. [3] utilised diﬀerent attention mechanics learningword, sentence and paragraph hierarchically. They also
created their own dataset Writing Prompt Dataset to train
their model.

3Method

To overview our model, its input is a topic sentence
and it contains three part. The first part is retrieving visual representation with a given topic. The next one ispredicting a sequence of the visual representations. The
last one is recovering these visual representations into thesentences. Thus, we can get a sequence of narratives. Fig.
1 shows the overview of our model.

• It is the first time to solve the open story generation
problem in the visual space.

図 1: The overview of our entire method.

• We advise an unsupervised approach to generate astory.

3.1

2 Related work
There exists a large number of researches utilising deeplearning. The simplest way is to use a sequence to sequence model.

Retrieve image

The objective of this stage is to retrieve a desirable image. [2] embeded the text space into the visual space.
They passed three types of sentence representations into
the network and got the representation in the visualspace. Three representations are Bag of Word (BoW)[4], Word2Vec [7] and Gated Recurrent Unit (GRU) [1].

― 751 ―

Copyright(C) 2019 The Association for Natural Language Processing.
All Rights Reserved.

When retrieving the image, they calulated the cosine sim-3.3

ilarity with each image in the database and found the image with the maximum value as retrieved image. In thiswork, we borrow their model to retrieve the image. Fig. 2
shows the network. The Flickr30k dataset is used in thistraining step.

Generate sentence

The purpose of this stage is to generate a story-stylesentence. To achieve this goal, we separate it into twostages. The first is generating captions with conventional
approaches and the second is transferring the captions
into the story-style sentences.
Togenerate the caption for the image, we borrow

the NIC model [8]. We also produce a model to transfer
the caption to the sentence. This model is the same as theNIC model but with diﬀerent input. The NIC model’s input is the representation of the image and this model’sinput are the concatenation of the caption and the image representation. The caption representation is the last
ouput of GRU whose input is the word embeddings of thecaption. The image representation is the substraction of
the current image representation and the previous image
representation. Fig. 3 shows the caption to story-style
図 2: This network embeds the text space into the visual

sentence model. The dataset that we used for generat-space.

ing caption is the MSCOCO dataset. For transferring thecaption to sentence, we use the corresponded the caption
and story sentence pairs in the VIST dataset.

3.2 Predict next image
The predicted next image should contain close semantic meaning to the current one. This regularisation ensures the coherence of our generated story. We experiment three traditional methods in this stage: K nearest
neighbour search, Multilayer Perceptron with MSE loss
and Multilayer Perceptron with Pairwise Ranking loss.
MethodTop 1

Top 5

Top 10NNS

9.33%

18.96%

22.47%

MLP(MSE)

0.70%

3.30%

5.84%

MLP(Pairwise)

0.50%

1.30%

3.10%

図 3: The caption to story-style sentence model. The in-put combine the caption feature and the image feature.
The GRU model produces the caption feature. The im-

表 1: The performance of three methods in predicting thenext image test on VIST test dataset.

age feature is the subtraction of the feature of the current
image and the previous image.

From the Table 2, K nearest neighbour search performs
the best. Therefore, we use K nearest neighbour search

4Experiment

4.1

Topic Coherence Score

in this stage. In the test phase, we use the entire images
in the VIST dataset as the next image candidates.

[5] produced a network to evaluate the score of the coherence. Inspired by their idea, we propose a model to― 752 ―Copyright(C) 2019 The Association for Natural Language Processing.
All Rights Reserved.

produce a topic coherence score for our task. Fig. 4of this evaluation.

shows the Topic Coherence model.

Score(D) =1 ∑ NumberOfUniques(d)|D| d∈D TopicInconsistency (d)NumberOfUniques (d) = len(set( flatten (d)))[()]
TopicInconsistency (d) = Ewi ,wi ∈d: distance wi , w j(3)NumberOfUniques shows how many diﬀerent words arein the story and TopicInconsistency is the mean value ofthe distance of all possible word pairs in the story. Inshort, we collect all the words appeared in the gener-

図 4: Topic Coherence model

For training this model, we use the story in VIST

ated story. The number of the word collections is theNumberOfUniques while TopicInconsistency calculates

dataset as training data. The stories in this dataset has

the distances for all the possible word pairs in the collec-

high topic coherence, so we give one as their label. Ontions. The higher score means the better performance.

the other side, we counterfeit two kinds of stories withno topic coherence. 1) We randomly choose five sentences from the sentence corpus, compose them into a4.3

Resultstory. As for the objective function, We use the margin

We use the Topic Coherence Score and the Consistency

ranking loss. 2) We randomly duplicate one sentence inand Diversity Score to evaluate the model. We randomly

the story. Both of the faked data are labeled zero for train-

generate 100 stories using the first sentence of the story ining.

VIST test data as input sentence in both our method andthe baseline method. From the results, we can see thatloss = margin + score( f ake) − score(real)(1)

We test the trained topic coherence model with 1000 story
in test data and 1000 fake data. These fake data use thewe get higher score than the baseline in Table 2. Fig. 5
present some examples using our method and the baselinemodel.

first sentence in a real story as the start sentence and ex-Method

Ours

Baseline

GroudTruth

Model based

0.43710.38550.7234

Counting based

1.882

0.945

2.99

Human1.8521.480

2.108

tend four randomly chosen sentences from sentence corpus. We get 86.45% in the test data.
1∑I(score(reali ) > score( f akei ))n i=0nAcc =(2)This result shows this topic coherence model has ability
to distinguish the bad story and the good story.

表 2: The Model based score is the Topic Coherence

Score and the Counting based score stands for the Consistency and Diversity Score. We also propose the human

4.2 Consistency and Diversity Score

preference which is a three-grade marking system.

The second evaluation method that we produce is acounting-based method. We assume that a good story
should both satisfy the topic consistency and the word diversity. Based on this assumption, we introduce the func-5tion to evaluate the story. Equation 3 shows the function

Future WorkAs for the future work, we present two possible waysof improving the quality of the story generation. First, as
we said, it is possible to add visual reasoning in the image― 753 ―

Copyright(C) 2019 The Association for Natural Language Processing.
All Rights Reserved.

Input: the catering was well done .
Ground Truth: they were many diﬀerent dishes for all of our tastes . we were able to try many types of flavors . the day
went really well at the oﬃce party . good music and food brought every one closer together .
Baseline: everyone had a great time at the ceremony . there were a lot of people waiting for the ceremony to start . they
were all very proud of being there . we had a great time .
Ours: we all had a great time eating and drinking and celebrating with everyone . we all had a great time at the party andeating some delicious food . we all had a great time and we learned a lot about it and the food . we had a great time at theparty .
Predicted image:

図 5: Exmaples compared with our method and the baseline method and human annotated story.

prediction part. With the KNN method we used in thispaper, the reasoning part is not concerned, which makes
the event flow becomes not clear. For the second one, Thedataset we used is not good enough for generating a story
with good quality. Our goal is to generate a story withany topic. However, in the training dataset, if it does notcontain the all kinds of story, for testing, we also can notgenerate the story with such style. Thus, we need a betterdataset.

[5] Alice Lai and Joel Tetreault. Discourse coherence in thewild: A dataset, evaluation and methods. arXiv preprint
arXiv:1805.04993, 2018.
[6] Lara J Martin, Prithviraj Ammanabrolu, Xinyu Wang,William Hancock, Shruti Singh, Brent Harrison, andMark O Riedl.
Event representations for automatedstory generation with deep neural nets. arXiv preprint
arXiv:1706.01331, 2017.
[7] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado,and Jeﬀ Dean. Distributed representations of words andphrases and their compositionality. In Advances in neural
information processing systems, pp. 3111–3119, 2013.
[8] Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. Show and tell: A neural image caption generator. In Proceedings of the IEEE conference on computer
vision and pattern recognition, pp. 3156–3164, 2015.

6 Acknowledgements
This work was supported by JSPS KAKENHI Grant
Number 16H05872. The members in AIST also produced
useful advices for this work.

参考文献[1] Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, andYoshua Bengio. Empirical evaluation of gated recurrent
neural networks on sequence modeling. arXiv preprint
arXiv:1412.3555, 2014.
[2] Jianfeng Dong, Xirong Li, and Cees GM Snoek. Predicting visual features from text for image and video caption
retrieval. IEEE Transactions on Multimedia, 2018.
[3] Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchicalneural story generation. arXiv preprint arXiv:1805.04833,
2018.
[4] Zellig S Harris. Distributional structure. Word, Vol. 10, No.
2-3, pp. 146–162, 1954.

― 754 ―

Copyright(C) 2019 The Association for Natural Language Processing.
All Rights Reserved.