Contextualize, Show and Tell: A Neural Visual Storyteller

We present a neural model for generating short stories from image sequences, which extends the image description model by Vinyals et al. (Vinyals et al., 2015). This extension relies on an encoder LSTM to compute a context vector of each story from the image sequence. This context vector is used as the first state of multiple independent decoder LSTMs, each of which generates the portion of the story corresponding to each image in the sequence by taking the image embedding as the first input. Our model showed competitive results with the METEOR metric and human ratings in the internal track of the Visual Storytelling Challenge 2018.

section_name: Introduction
paragraphs: Over the past few years, generating text from images and videos has gained a lot of attention in the Computer Vision and Natural Language Processing communities and several related tasks have been proposed, such as image labeling, image and video description and visual question answering. In particular, prominent results have been achieved in image description with various deep neural network architectures, e.g. BIBREF1 , BIBREF2 , BIBREF3 , BIBREF0 . However, the need of generating more narrative texts from images which may reflect experiences, rather than just listing objects and their attributes, has given rise to tasks such as visual storytelling BIBREF4 . This task is about generating a story from a sequence of images. Figure 1 shows the difference between descriptions of images in isolation and stories for images in sequence.
In this paper, we describe the deep neural network architecture we used for the Visual Storytelling Challenge 2018. The problem to solve in this challenge can be stated as follows: Given a sequence of 5 images, the system should output a story related to the content and events in the images. Our architecture is an extension of the image description architecture presented by BIBREF0 . We submitted the generated stories to the internal track of the Visual Storytelling (VIST) Challenge, which were evaluated using the METEOR metric BIBREF5 as well as human ratings.
section_name: Previous work
paragraphs: The work by BIBREF6 presented probably the first system for generating stories from an album of images. This early approach involved the use of the NYC and Disney datasets mined from blog posts by the authors.
The visual storytelling task and dataset were introduced by BIBREF4 . This was the first dataset specifically created for visual storytelling. They proposed a baseline approach which consists of a sequence to sequence model, where the encoder takes the sequence of images as input and the decoder takes the last state of the encoder as its first state to generate the story. Since this model produces stories with generic phrases, they used decode-time heuristics to improve the generated stories.
 BIBREF7 presented a multi-task model that performs album summarization and story generation. Even though the model achieved state-of-the-art scores on the VIST dataset with different metrics, some of the sample stories presented in the paper are incoherent.
section_name: Model
paragraphs: Our model extends the image description model by BIBREF0 , which consists of an encoder-decoder architecture. The encoder is a Convolutional Neural Network (CNN) and the decoder is a Long Short-Term Memory (LSTM) network, as presented in Figure 2 . The image is passed through the encoder generating the image representation that is used by the decoder to know the content of the image and generate the description word by word. In the following, we describe how we extended this model for the visual storytelling task.
section_name: Encoder
paragraphs: The model's first component is a Recurrent Neural Network (RNN), more precisely an LSTM that summarizes the sequence of images. At every timestep $t$ the network takes as input an image $I_i$ where $i\in \lbrace 1,2,3,4,5\rbrace $ from the sequence. At time $t=5$ , the LSTM has encoded the 5 images and provides the sequence's context through its last hidden state denoted by $h_e^{(t)}$ . The representation of the images was obtained through Inception V3.
section_name: Decoder
paragraphs: The decoder is the second LSTM network that uses the information obtained from the encoder to generate the sequence's story. The first input $x_0$ to the decoder is the image for which the text is being generated. The last hidden state from the encoder $h_e^{(t)}$ is used to initialize the first hidden state of the decoder $h_d^{(0)}$ . With this strategy, we provide the decoder with the context of the whole sequence and the content of the current image (i.e. global and local information) to generate the corresponding text that will contribute to the overall story.
Our model contains five independent decoders, one for each image in the sequence. All the 5 decoders use the last hidden state of the encoder (i.e. the context) as its first hidden state and take the corresponding image embedding as its first input. In this way, the first decoder generates the sequence of words for the first image in the sequence, the second decoder for the second image in the sequence, and so on. This allows each decoder to learn a specific language model for each position of the sequence. For instance, the first decoder will learn the opening sentences of the story while the last decoder the closing sentences. The word embeddings were computed using word2vec BIBREF8 .
Our proposed architecture is presented in Figure 3 . For each image in the sequence, we obtain its representation $\lbrace e(I_1),...,e(I_5)\rbrace $ using Inception v3. The encoder takes the images in order, one at every timestep $t$ . At time $t=5$ , we obtain the context vector through $h_e^{(t)}$ (represented by $\mathbf {Z}$ ). This vector is used to initialize each decoder's hidden state while the first input to each decoder is its corresponding image embedding $e(I_i)$ . Each decoder generates a sequence of words $\lbrace p_1,...,p_{n}\rbrace $ for each image in the sequence. The final story is the concatenation of the output of the 5 decoders.
section_name: Methodology
paragraphs: The generated stories were evaluated using both automatic metrics and human ratings. The automatic evaluation was performed by computing the METEOR metric BIBREF5 on a public test set and a hidden test set. The former is a set of $1,938$ image sequences and stories taken from the test set of the VIST dataset BIBREF4 . The latter consists of new stories generated by humans from a subset of image sequences of the public test set.
Human ratings of the stories were collected from crowd workers in Amazon Mechanical Turk. Only 200 stories were selected from the hidden test set for this evaluation. The crowd workers evaluated each story on a Likert scale with respect to 6 aspects: a) the story is focused, b) the story has good structure and coherence, c) would you share this story, d) do you think this story was written by a human, e) the story is visually grounded and f) the story is detailed. The crowd workers were also asked to evaluate stories generated by humans for comparison purposes.
section_name: Results
paragraphs: Table 1 shows the METEOR scores by our model in the public and hidden test set of the Visual Storytelling Challenge 2018. Table 2 presents results of the human evaluation. Our model achieved competitive METEOR scores in both test sets and performed well in the human evaluation.
An evaluation over the complete VIST test set was also performed and the results are shown in Table 3 .
Our model obtained the highest scores with the METEOR and BLEU-3 metrics but lagged behind the model by BIBREF7 with the ROUGE and CIDEr metrics.
Figure 4 shows some sample stories generated by our model from the public test set of the Visual Storytelling Challenge 2018. Although some of the generated stories are grammatically correct and coherent, they tend to contain repetitive phrases or ideas. We can also observe that some stories are not nearly related to the actual content of the images or include generic phrases like This is a picture of a store. These limitations of our model reflected on the ratings of the visually grounded and detailed aspects of the human evaluation.
section_name: Conclusions and Future Work
paragraphs: Our visual storyteller incorporates a context encoder and multiple independent decoders to the image description architecture by BIBREF0 to generate stories from image sequences. Having an independent decoder for each position of the image sequence, allowed our visual storyteller to build more specific language models using the context vector as its first state and the image embedding as its first input. In the internal track of the Visual Storytelling Challenge 2018, we obtained competitive METEOR scores in both the public and hidden test sets and performed well in the human evaluation.
In the future, we plan to explore the use of an attention mechanism or a bidirectional LSTM to cope with repetitive phrases within the same story.
section_name: Acknowledgments
paragraphs: This research is supported by the PAPIIT-UNAM research grant IA104016. Diana González Rico is supported by CONACYT.

Questions and Answers:
question: What model is used to encode the images?
answers: answer: unanswerable: no
extractive_spans: a Convolutional Neural Network (CNN)
evidence: Our model extends the image description model by BIBREF0 , which consists of an encoder-decoder architecture. The encoder is a Convolutional Neural Network (CNN) and the decoder is a Long Short-Term Memory (LSTM) network, as presented in Figure 2 . The image is passed through the encoder generating the image representation that is used by the decoder to know the content of the image and generate the description word by word. In the following, we describe how we extended this model for the visual storytelling task.
answer: unanswerable: no
extractive_spans: LSTM
evidence: The model's first component is a Recurrent Neural Network (RNN), more precisely an LSTM that summarizes the sequence of images. At every timestep $t$ the network takes as input an image $I_i$ where $i\in \lbrace 1,2,3,4,5\rbrace $ from the sequence. At time $t=5$ , the LSTM has encoded the 5 images and provides the sequence's context through its last hidden state denoted by $h_e^{(t)}$ . The representation of the images was obtained through Inception V3.
question: How is the sequential nature of the story captured?
answers: answer: unanswerable: no
extractive_spans: we provide the decoder with the context of the whole sequence and the content of the current image (i.e. global and local information) to generate the corresponding text that will contribute to the overall story
evidence: The decoder is the second LSTM network that uses the information obtained from the encoder to generate the sequence's story. The first input $x_0$ to the decoder is the image for which the text is being generated. The last hidden state from the encoder $h_e^{(t)}$ is used to initialize the first hidden state of the decoder $h_d^{(0)}$ . With this strategy, we provide the decoder with the context of the whole sequence and the content of the current image (i.e. global and local information) to generate the corresponding text that will contribute to the overall story.
answer: unanswerable: no
extractive_spans: The encoder takes the images in order, one at every timestep $t$ . At time $t=5$ , we obtain the context vector through $h_e^{(t)}$ (represented by $\mathbf {Z}$ ). This vector is used to initialize each decoder's hidden state while the first input to each decoder is its corresponding image embedding $e(I_i)$ . Each decoder generates a sequence of words $\lbrace p_1,...,p_{n}\rbrace $ for each image in the sequence. 
evidence: Our proposed architecture is presented in Figure 3 . For each image in the sequence, we obtain its representation $\lbrace e(I_1),...,e(I_5)\rbrace $ using Inception v3. The encoder takes the images in order, one at every timestep $t$ . At time $t=5$ , we obtain the context vector through $h_e^{(t)}$ (represented by $\mathbf {Z}$ ). This vector is used to initialize each decoder's hidden state while the first input to each decoder is its corresponding image embedding $e(I_i)$ . Each decoder generates a sequence of words $\lbrace p_1,...,p_{n}\rbrace $ for each image in the sequence. The final story is the concatenation of the output of the 5 decoders.
question: Is the position in the sequence part of the input?
answers: answer: unanswerable: no
yes_no: no
answer: unanswerable: no
yes_no: yes
evidence: The model's first component is a Recurrent Neural Network (RNN), more precisely an LSTM that summarizes the sequence of images. At every timestep $t$ the network takes as input an image $I_i$ where $i\in \lbrace 1,2,3,4,5\rbrace $ from the sequence. At time $t=5$ , the LSTM has encoded the 5 images and provides the sequence's context through its last hidden state denoted by $h_e^{(t)}$ . The representation of the images was obtained through Inception V3.
question: Do the decoder LSTMs all have the same weights?
answers: answer: unanswerable: no
yes_no: no
evidence: Our model contains five independent decoders, one for each image in the sequence. All the 5 decoders use the last hidden state of the encoder (i.e. the context) as its first hidden state and take the corresponding image embedding as its first input. In this way, the first decoder generates the sequence of words for the first image in the sequence, the second decoder for the second image in the sequence, and so on. This allows each decoder to learn a specific language model for each position of the sequence. For instance, the first decoder will learn the opening sentences of the story while the last decoder the closing sentences. The word embeddings were computed using word2vec BIBREF8 .
answer: unanswerable: no
yes_no: no
evidence: Our model contains five independent decoders, one for each image in the sequence. All the 5 decoders use the last hidden state of the encoder (i.e. the context) as its first hidden state and take the corresponding image embedding as its first input. In this way, the first decoder generates the sequence of words for the first image in the sequence, the second decoder for the second image in the sequence, and so on. This allows each decoder to learn a specific language model for each position of the sequence. For instance, the first decoder will learn the opening sentences of the story while the last decoder the closing sentences. The word embeddings were computed using word2vec BIBREF8 .

Figure and Table Captions:
caption: Figure 1: Examples of stories for images in sequence (above) and image descriptions in isolation (below) from the VIST dataset (Huang et al., 2016).
caption: Figure 2: Show and Tell architecture. Image reproduced from (Vinyals et al., 2015).
caption: Figure 3: Proposed sequence to sequence architecture.
caption: Table 2: Human evaluation of stories generated by our visual storyteller, compared to stories generated by humans.
caption: Table 3: Automatic evaluation on the VIST dataset. A comparison between the baseline (Huang et al., 2016), (Yu et al., 2017) and ours.
caption: Figure 4: Sample stories generated by our visual storyteller, compared to generated by humans.