Talk : Generating Narrative Paragraph for Photo Stream via Bidirectional Attention Recurrent Neural Networks

Automatic generation of natural language description for individual images (a.k.a. image captioning) has attracted extensive research attention. In this paper, we take one step further to investigate the generation of a paragraph to describe a photo stream for the purpose of storytelling. This task is even more challenging than individual image description due to the difficulty in modeling the large visual variance in an ordered photo collection and in preserving the longterm language coherence among multiple sentences. To deal with these challenges, we formulate the task as a sequence-tosequence learning problem and propose a novel joint learning model by leveraging the semantic coherence in a photo stream. Specifically, to reduce visual variance, we learn a semantic space by jointly embedding each photo with its corresponding contextual sentence, so that the semantically related photos and their correlations are discovered. Then, to preserve language coherence in the paragraph, we learn a novel Bidirectional Attention-based Recurrent Neural Network (BARNN) model, which can attend on the discovered semantic relation to produce a sentence sequence and maintain its consistence with the photo stream. We integrate the two-step learning components into one single optimization formulation and train the network in an end-to-end manner. Experiments on three widely-used datasets (NYC/Disney/SIND) show that the proposed approach outperforms state-of-the-art methods with large margins for both retrieval and paragraph generation tasks. We also show the subjective preference of the machine-generated stories by the proposed approach over the baselines through a user study with 40 human subjects. Introduction Generating a human-level narrative from an ordered photo stream, in this research we refer to as “visual storytelling”, presents a fundamental challenge to both computer vision and natural language processing areas. This is challenging because it requires not only the full understanding of each photo in a stream as well as the relation among different photos, but also a sophisticated mechanism to generate a natural paragraph from the perspective of language coherence. Existing researches have focused more on generating natural descriptions for a single photo. In this research we take ∗This work was performed when Yu Liu was visiting Microsoft Research Asia as a research intern. Copyright c © 2017, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. Image Captions: A group people is play ball. A man is playing a ball. A group of people is having dinner around a table. Storytelling: A group of people gather as a team. The ball game begins. After the ball game, the team winds down with some food. Figure 1: The difference between storytelling and image captions in isolation is the coherence among sentences, i.e. the green term “ball game” and the blue term “the team” are introduced from previous photos with corresponding color. [Best viewed in color] one step further to investigate the problem of paragraphing a photo stream. We consider this task as a sequenceto-sequence learning problem, where the input is a photo stream (with order) while the output is a sentence sequence, each corresponding to one photo. It is a totally different issue from conventional sequenceto-sequence problems. The closely related research to visual storytelling is image and video captioning, where the Recurrent Neural Networks (RNNs) are usually employed for generating a single sentence from a given image or video clip (Venugopalan et al. 2015a)(Donahue et al. 2015), (Venugopalan et al. 2015b), (Pan et al. 2016). However, the task of visual storytelling is more challenging due to the difficulty of modeling the large visual variance in an ordered photo collection and preserving the long-term language coherence among multiple sentences. First, at the input side, photo stream usually has significantly large visual variance, such as the example in Figure 1. Most existing models for sequence learning with deep structure (e.g., RNN) are not designed to deal with such long-term dependency with large visual variance. Second, at the output side, visual storytelling requires much more complicated textual form in coherent and consistent paragraph. As the storytelling in the Figure 1, the overlapping of semantics between sentences often results in complex relational structure of a story. To deal with the above two challenges, we formulate this task as a sequence-to-sequence learning problem and propose a novel joint learning model by leveraging the semantic coherence in a photo stream. First, we learn a common semantic space by jointly embedding each image with its corresponding sentence using semantic embedding in (Kiros, Salakhutdinov, and Zemel 2014) to alleviate visual variance problem. Semantic learning is important in tasks like multimedia representation and search (Mei et al. 2014). As other semantic learning works (Fu et al. 2014; 2015b; 2015a; Wang et al. 2016), we assume that the semantics of visual and text in telling the same story are relevant (Liu, Mei, and Chen 2016). Our idea is to learn a semantic space where the related photo-sentence pair can be close enough to reflect the same semantics. As a result, semantically related photos with large visual gap can be bridged by the neighborhood relation in the semantic space. Hence the visual variance is reduced and semantics are learned. For example, as in Figure 1, since it can be observed in the story dataset that “dinner/picnic” almost always follow a “ball game,” we learn that these two visually different events should be semantically closed to each other. Moreover, a semantic relation matrix (coherence matrix), can be further identified by distance measure in this space. The coherence matrix is important to describe the semantic structure of the story, and will be used in an attention scheme to guide the training our sequence model for narrative paragraph generation. Second, we propose a Bidirectional Attention-based Recurrent Neural Network (BARNN) to use the coherence matrix to enforce the sentence-to-sentence coherence. In our model, as the convention, the memory of each recurrent timestep encodes the deep feature (semantic in our case) of one photo/sentence in the sequence. Existing research (Park and Kim 2015) has proposed to use bidirectional RNN (BRNN) to capture sentence-level transition, and only considers the adjacent memories in the sequence. However, as we have shown in Figure 1, stories usually consist of much more complex structure, where the semantic of arbitrary timesteps in the sequence can be related. In this structure, each semantic memory may be contributed from arbitrary timesteps. For the example shown in storytelling of Figure 1, the term “the team” and “ball game” in sentence 1 and 2 can contributes to the prior knowledge “after the ball game, the team. ” in sentence 3. To this end, we propose to design a framework with attention mechanism that integrates the memory semantics from various timesteps, via connections called skip. For this purpose, We design a novel recurrent unit by granting classic GRU with skip connections to allow this attention scheme. Thus the unit is called skip-GRU (sGRU). Note that the coherence matrix plays a role here of guiding the attention in how much information to contribute. Therefore, learning this BARNN can enable us to coherently model the sequence of photos. Note that we combine the learning of BARNN and previous semantic embedding into one objective function and train in end-to-end manner. The contributions of the paper can be summarized as: • The inherent challenges of visual storytelling are addressed by training an cross-modality embedding model, which can overcome the large visual variance in photo stream and represent the semantic of an underlying story. • A novel BARNN framework with a new-designed skip gated recurrent unit (sGRU) is proposed to leverage implicit semantic relation in order to enforce the coherence of predicted sentences. • Extensive experiments on the three storytelling datasets (NYC, Disney (Park and Kim 2015) and SIND (Huang et al. 2016)) have been carried out, and superior performance over the state-of-the-art with large margins in both retrieval and generation tasks has been obtained.