Year: 2021

Abstract:
Sequential text-to-image retrieval, a.k.a. Story-to-images task, requires semantic alignment with a given story and maintaining global coherence in drawn image sequence simultaneously. Most of the previous works have only focused on modeling how to follow the content of a given story faithfully. This kind of overfitting tendency hinders matching structural similarity between images, causing an inconsistency in global visual information such as backgrounds. To handle this imbalanced problem, we propose a novel image sequence retrieval framework that utilizes scene graph similarities of the images and a dual learning scheme. Scene graph describes high-level information of visual groundings and adjacency relations of the key entities in a visual scene. In our proposed retriever, the graph encoding head learns to maximize embedding similarities among sampled images, giving a strong signal that forces the retriever to also consider morphological relevance with previously sampled images. We set a video captioning as a dual learning task that reconstructs the input story from the sampled image sequence. This inverse mapping gives informative feedback for our proposed retrieval system to maintain global contextual information of a given story. We also suggest a new contextual sentence encoding architecture to embed a sentence in consideration of the surrounding context. Through extensive experiments, Our proposed framework shows better qualitative and quantitative performance with Visual Storytelling benchmark compared to conventional story-to-image models.

text: Sequential text-to-image retrieval, a.k.a. Story-to-images task, requires semantic alignment with a given story and maintaining global coherence in drawn image sequence simultaneously. Most of the previous works have only focused on modeling how to follow the content of a given story faithfully. This kind of overfitting tendency hinders matching structural similarity between images, causing an inconsistency in global visual information such as backgrounds. To handle this imbalanced problem, we propose a novel image sequence retrieval framework that utilizes scene graph similarities of the images and a dual learning scheme. Scene graph describes high-level information of visual groundings and adjacency relations of the key entities in a visual scene. In our proposed retriever, the graph encoding head learns to maximize embedding similarities among sampled images, giving a strong signal that forces the retriever to also consider morphological relevance with previously sampled images. We set a video captioning as a dual learning task that reconstructs the input story from the sampled image sequence. This inverse mapping gives informative feedback for our proposed retrieval system to maintain global contextual information of a given story. We also suggest a new contextual sentence encoding architecture to embed a sentence in consideration of the surrounding context. Through extensive experiments, Our proposed framework shows better qualitative and quantitative performance with Visual Storytelling benchmark compared to conventional story-to-image models.

heading: 1 INTRODUCTION
text: Visual content and textual description are in synergytic relations, have advantageous on delivering information and being a proper expression means in real world communications. In this sense, successful cross-modal learning requires neural networks to deeply comprehend semantic relations between corresponding visual concepts and their textual descriptions. Examples include learning joint representation(Li et al., 2015; Ma et al., 2021; Pan et al., 2013), text generation from visual depiction(Farhadi et al., 2010; Kulkarni et al., 2013; Socher et al., 2014), and image or video retrieval from text queries(Kim et al., 2015; Zitnick et al., 2013). Such multi-modal tasks build on common embedding space where semantically associated visual and text embeddings are jointly mapped into similar location. Hence, the key of multi-modal learning is understanding semantic relationship between distinct modality representations.
In this sense, image-text retrieval have been widely explored as a one of the core tasks in the multimodal learning. The performance of such retrieval system has been measured by examining how the retrieved samples are semantically aligned with given query. In addition, most of them have been trained on singleton image-text pair dataset(e.g., COCO(Lin et al., 2015)). These requirement and condition enforces the existing retrieval system to output the sample which is the most semantically fitted to the query. In other words, the performance of the system is dominantly determined by the level of overfitting tendency of the output. The more output sample is relational to the input, the more performance increases. This comes with potential arguments in sequential cross-modal learning since it only cares inter relationship with certain modality input at given time step, while neglecting the intra relations of output modality. Consequently, when we qualitatively analyze output samples, we can observe some incoherencies between output samples.
In sequential text-to-image, many of conventional approaches attempt to maximize the visual coherence with given set of descriptions. Though they largely improved modeling a cross-modal coherency between text and image, many of them neglect scenery coherence among images. In other words, since the system focuses on reflecting objects, optical discrepancies frequently occur between neighboring images. For example, peripheral visual cues like background objects, weather, and location changes as the story proceeds. We view this problem as an optimization problem whose goal is alleviating the one-sided overfitting to a certain modality. To address this, we have to consider an additional method which can teach the retriever to also count similarity in the other modality.
To overcome the suggested issue, we should handle two main challenges. First, we also have to measure how much are two successive landscapes similar, that is, we need an organized representation that includes spatial information of each photo. Second, even if an image sequence was chosen considering scenery similarity, the curated image sequence must be still capable of narrate given input story. In this paper, we address aforementioned challenges with scene graph structure and dual learning framework.
Scene graph(Johnson et al., 2015) is a graph representation that includes abstract summaries of the objects and their relationships within an image. The objects are represented as nodes, and the relations between them is usually represented as bidirectional edges. Due to it’s canonicalized description, scene graphs are very effective structured representation to easily figuring out global information of both images and language. We generate scene graph of every image in training set to provide extra spatial information for the retriever. Hence, we need to also develop a encoding module to process scene graphs. In this paper, we attach GCN(graph convolutional network)(Kipf & Welling, 2016) as a scene graph encoding head of the retriever which extracts compressed representation from nodes and edges from given scene graph. By drawing the scene graphs of retrieved image sequence and passing to the following encoding head, model can compute the extent of their similarities in graph embedding level.
Dealing with second challenge, we utilized dual learning framework which first introduced by Xia et al. (2016). In dual learning, two agents are involved where one agent solves primal task and the other solves dual task. For example, Xia et al. (2016) set English-to-French translation as a primal, and French-to-English as a dual. This cyclic loop gives each agent to learn error feedback from each other. We set story-to-images as a primal task, and images-to-story, which can be regarded as video captioning, be a dual task. The main purpose of leveraging dual framework is providing informative error signal from dual task to the primal retriever system. The signal will teach the retriver to pick well-curated photos which become the input for the dual agent. In overall, dual task contributes to enhance the qualitative confidence for an output of the retriver.
Figure 1 depicts our aforementioned approaches in a single architecture. We experiment our approaches to the most popular visual storytelling benchmark: VIST(Huang et al., 2016). Specific configuration of VIST will be explained in section 4.
In a nutshell, out key contributions are as follows:
1. We propose end-to-end sequential text-to-image retriever which achieves semantic coherency in both inter and intra modalities, which resolves overfitting on single modality
2. We explore visual storytelling, which is non-trivial task that requires balance in semantic dependencies between distinct modalities.

heading: 2 RELATED WORKS

heading: 2.1 IMAGE RETRIEVAL
text: Much cross-modal retrieval research has dealt with learning a latent space that jointly embeds images and sentences into the same metric space. Especially, image-caption retrieval focuses on matching the most relevant image(s) from a database with a given text query(Babenko et al., 2014). As retrieval-based system finds output candidates in a pre-structured database, it is more advantageous in overall likeliness in output qualities and shows less variation in sample mean quality. However, most of multi-modal retrieval systems deals with mapping a single instance. Few retrieval works have been explored to retrieve sequential outputs for structured queries. Some previous retrieval systems ranked images based on visual phrases(Sadeghi & Farhadi, 2011), or multi-attribute descriptions(Siddiquie et al., 2011). Kim et al. (2015) first proposed a ranking system to retrieve image sequences from natural language paragraphs. Recently, Chen et al. (2019) proposed visual segment matching framework to improve the output coherency and storyboard creation tool for how retrieval system can be applied to practical field application. Nevertheless, none of which considered the semantic arrangement with previously sampled instances.

heading: 2.2 SCENE GRAPHS
text: A scene graph depicts the contents of an image in the form of graph structure. Graph nodes represent objects, their attributes, and the relationshipo among them. As a scene graph provide visual information in abstractive level, it has been proven to be effective in a range of visual comprehension tasks such as image retrieval(Yoon et al., 2020), image or video captioning(Chen et al., 2020; Hong et al., 2020), visual question answering(Damodaran et al., 2021), and image generation(Johnson et al., 2018). A number of applications utilizing scene graph information have been widely spread after a large-scale scene graph annotations of real world images revealed from Visual Genome dataset(Krishna et al., 2017). In representation learning context, there have been many recent works focusing on learning intermediate representations of scene graphs. Those works suggest scene graph representation as an useful compressed information for downstream applications. Raboh et al. (2020) proposed differentiable scene graphs, which can be trained end-to-end with reasoning supervision. Maheshwari et al. (2021) constructed semantically rich representation through ranking loss(Karpathy & Fei-Fei, 2015; Kiros et al., 2014) coupled with triple sampling strategy in image retrieval task. The closest related work (Yoon et al., 2020) proposed experimental approach that leverages similarities of scene graph embedding for image-to-image retrieval task.

heading: 2.3 DUAL LEARNING
text: The application of dual learning was first proposed by (He et al., 2016a) to relieve the burden for preparing paired training data of English-to-French translation. The key idea of dual learning is setting a primal and a dual task in a domain translation task. Learning source-to-target(primal) and target-to-source(dual) mappings simultaneously gives ..... Especially, such a mutual reinforcing mechanism have shown effective results on generation tasks in unsupervised settings. (Luo et al., 2019)The advanced image-to-image translation GANs(Yi et al., 2017; Zhu et al., 2020) have shown competitive performance with unlabeled data by leveraging primal-dual relation to guarantee stable domain translation performance without daunting the qualities of the generated images. In our work, we set an image sequence retrieval from given story as a primal task and regenerating the original story from chosen images as a dual. The informative error signals from reconstruction(dual) task enforces the retrieval agent to choose more ’thoughtful’ inputs for dual agent.

heading: 3 APPROACH

heading: 3.1 PROBLEM STATEMENT
text: Let S = {si}Ni=1 is a story comprises N text descriptions, where each description composed of a single or a few sentences. For the simplicity and avoid annotation confusion, consider a single description as a single sentence. The goal of the task is retrieving an image sequence I = {ij}Nj=1 that is semantically aligned to a story S and each image ij is descriptive photo of the description si=j . Hence, the retrieval system F : {si}Ni=1 7−→ {ij} j j=1 should map given story to the most probable image sequence without losing visual coherency. In this paper, we set I and S to have a same cardinality to easily compare the matching results by one-to-one. We left one-to-many retrieval as our future work.

heading: 3.2 IMAGE SEQUENCE RETRIEVAL VIA GRAPH SIMILARITIES
text: Contextual Sentence Encoder. We give a story for the direct input to our retrieval system. A story describes given image sequence(video) as a set of natural language descriptions. Many of previous text-image retrieval system receives a single text query as an input for their text encoders. If we try to process a story with traditional text encoders, contextual connection between each description will be break inevitably since there’s no other way to encode a story without recurrently injecting the description. In consequence, feature representation of each description will be separately located even in text-image joint embedding space. To encourage each description to be encoded in homogeneous way, each embedding must imply contextual information of preceding text. In other words, the desired text encoder in story-to-images retrieval system must be context-recognizable. Thus, we need to implement a novel story encoder for the task. Inspired by Chen et al. (2019), we suggest C.S.E(Contextual Sentence Encoder) which is suitable for sequential text-to-image retrieval.
?? describes the overall architecture of C.S.E. The key idea of C.S.E is considering structured hierarchical relation between text and it’s tokens. To extract a dense representation of the description, the tokens in the description should also contain relevant information about other surrounding texts.
C.S.E efficiently encodes each description si ∈ S, considering other surrounding descriptions in S. C.S.E consists of bottm Bi-LSTM layer, intermediate attention layer, and final Bi-LSTM layer with global average pooling head. Let a single description in S as si = {wi1 , wi2 , ..., wiL} , which is a sequence of L token wit s.t. wit ∈ R|V |1 ≤ t ≤ L. At time step t, the forward hidden state of the bottom Bi-LSTM layer −→ hit receives e i t ∈ Rdemb an embedding of eit, and the last hidden state −→ hit .
The backward hidden state at time t ←− hit receives arguments in the same way. The final hidden state of at time t is a concatenation of ←− hit and −→ hit . This can be summarized as
−→ hit = −−−−→ LSTM (−−→ hit−1,Wew i t ; −→ Wh, −→ bh ) ←− hit = ←−−−− LSTM ( ←− hit+1,Wew i t ; ←− Wh, ←− bh
) hit = [−→ hit ; ←− hit ] (1)
We can regard hit as a compreseed representation of token wit. To embed the context of surrounding descriptions, we set hit as a query of Bahdanau attention(Bahdanau et al., 2016) module.
To compute textual coherency with the description si which includes query wit, we select other descriptions except si as candidates for keys, and their hidden representations as keys. We pass all sk ∈ S, 1 ≤ k ≤ N, s.t.k 6= i to pre-trained sentence encoder(Kiros et al., 2014) and extract a set of hidden representations {k1, .., ki−1, ki+1, .., kN}. Then, the value vector of attention vit is computed as a weighted sum of keys and queries:
vit = ∑
1≤j 6=i≤N
hitαij (2)
The attention weight αij is a softmax score of attention layer with tanh activation computed by
αij = exp (eij)∑Tx k=1 exp (eik) eij = v > a tanh ( Wah i t + Uakj
) (3) where v>a , Wa, and Ua are learnable parameters of above attention layer. As v i t is a value vector from hit and {k1, .., ki−1, ki+1, .., kN}, we can regard vit dense token representation which embeds contextual semantic relationships with other surrounding descriptions in a single story.
Now we pass the concatenation of vit and h i t as a new input for the second Bi-LSTM layer. We denote the hidden state of the second RNN-based layer as g, and the second layer goes same progress to extract the tth token representation of description si :
−→ git = −−−−→ LSTM (−−→ git−1, [ hit ; v i t ] ; −→ Wg, −→ bg ) ←− git = ←−−−− LSTM (←−− git+1, [ hit ; v i t ] ; ←− Wg, ←− bg
) git = [−→ git ; ←− git
] ri = avgpool ([ gi1 ; ... ; g i L ]) (4)
Finally, we average-pool along time steps 1 ≤ t ≤ L to get the final contextual representation of the description si.

heading: 3.3 SCENE GRAPH GENERATION AND EMBEDDING
text: Scene Graph Generation. A scene graph is an abstract representation of the visual contents of an image(Johnson et al., 2015). Formally, we define scene graph G of an image I as G = {O,R} a set of object nodes O, and a set of relationship between two certain nodes R. Every relationship rk ∈ R is represented by a triplet of nodes (subjective, predicate, objective), explaining dynamical association between two nodes. Predicate is represented as undirected edge. We treat GloVe(Pennington et al., 2014) embedding of the label name as a feature representation of all nodes in O and all edges in R. Specific configuration will be explained in section 4. The combination of constituents of scene graph varies in related works(Chen et al., 2020; Hong et al., 2020; Yoon et al., 2020; Damodaran et al., 2021; Johnson et al., 2015; Krishna et al., 2017; Raboh et al., 2020).For example, some works(Yoon et al., 2020; Ashual & Wolf, 2019) also included a set of attributes
of objects A as another constituent of scene graph. In this work, we .... to relieve computational burden.
Generating scene graph G structure from an image I is equivalent to parsing an object detection result of a target image. We used Faster R-CNN(Ren et al., 2015) as an underlying detector. For each image I , the detector predicts a set of region proposals B = b1, b2, ..., bn. Each proposal bk ∈ B comes with bounding box feature representation and probabilities for corresponding object label. Building on these information, we applied recently proposed method(Zellers et al., 2018) for our scene graph generator. In detail, we utilized VG(Visual Genome) dataset(Krishna et al., 2017) configuration to assign proper predicate for constructing R. Predicate label is predicted based on frequency prior knowledge from VG. In overall retrieval pipeline, we generate scene graphs from primarily picked image sequence I for given story S and pass to the graph encoding head layer to compare the scenery similarities via computing similarity scores of their graph embeddings.
Encoding Scene Graphs via GCN. In order to encode a scene graph with end-to-end fashion, we need a suitable neural architecture that can operate directly on graph-structured data. We apply Graph Convolutional Network(Kipf & Welling, 2016) as our main graph encoding module since it’s learning ability on graph representation have been proved in many related works(Johnson et al., 2018; Yoon et al., 2020; Kipf & Welling, 2016; Chen et al., 2020).

heading: 3.4 DUAL LEARNING WITH VIDEO CAPTIONING
text: We adopt dual learning framework for enhancing a contextual coherency of output image sequence. The primal task can be represented as F : S −→ I with proposed retriever F . We denote G as a dual agent, s.t. G : I −→ S. G can be also regarded as visual storyteller, which generates figurative and consistent narrative for successive images. As F recurrently selects I , we primarily considered image captioning(Karpathy & Fei-Fei, 2015) as a dual. However, since each description in a single story narrates same circumstance, reconstructed descriptions will be semantically isolated from global context. On this, the output format must be a paragraph-level captions. Hence we regard dual task as a video captioning problem, to generated semantically aligned sentences while keeping global context. We construct video captioning module based upon the proposed architecture of GLAC Net(Kim et al., 2018).

heading: 3.5 TRAINING OBJECTIVES
text: image Sequence Retrieval. Let fT ( · ; θT ) a textual encoder parameterized by θT , which is C.S.E in this paper. hT (S) = fT (S; θT ) ∈ RdT is a dense representation of an input description S which embeds contextual relations with surrounding S in a story. Similarly, let hV(I) = fV(I; θV) ∈ RdV be a feature representation from pre-trained image encoder(e.g., VGG19(Simonyan & Zisserman, 2014), ResNet152(He et al., 2016b)) before the last FC layer when given an input image I . We map hT andhV into joint embedding space through following linear transformation.
φT (S ;WT , θT ) = W > T hT (S) ∈ Rde
φV(I ;WV , θV) = W > V hV(I) ∈ Rde
(5)
With linear operatorsWT ∈ RdT ×de andWV ∈ RdV ,×de , we now can do vector computation across different modalities. We measure the similarity of two distinct modality representation through cosine-similarity base score function, defined as
s(s, i) = sim(φT (s), φV(i)) (6)
where sim(a,b) = a·b‖a‖‖b‖ . To retrieve an image which semantically matches current input description and morphologically similar with previous image, we jointly use hinge ranking losses(Faghri et al., 2017; Chechik et al., 2010; Frome et al., 2007) between correct matches and other wrong ones and graph embedding similarity. At time step t, the step-wise loss `t is
`(s, i) = ∑ s̃ max(∆− s(s, i) + s(s̃, i), 0)
+ ∑ ĩ max(∆− s(s, i) + s(s, ĩ), 0)
+ 1− s(ψit−1 , ψit)
with margin ∆, negative description s̃ for i and negative image sample ĩ for s. A hinge ranking loss, a.k.a triplet ranking loss, directs the retriever F to minimize the first and the second term via choosing the closest counterpart i or s than to any unmatched samples ĩ or s̃ by margin ∆. At the same time, the last term of `t contributes F to search a photo that is structurally similar to an earlier image. The total loss for image retrieval is Lretrieval = ∑N t `(st, it).
Video Captioning. After selecting aligned images as a sequence I = {i1, ..., iN}, we input I and a story S as a target ground-truth text, training video caption generator G to reconstruct S. We use cross-entropy loss
Lcaption = − N∑ l=1 V∑ v=1 yvil log p v il (7)
where v ∈ {1, ..., V } is an index of vocabulary set. pvil is predicted probability for i− th token, and yvil is the target token.
Overall Objective. The overall objective is the sum of image sequence retrieval loss and video captioning loss. The total objective is as follows:
Ltotal = Lretreival + Lcaption (8)

heading: 4 EXPERIMENTS

heading: 4.1 EXPERIMENTAL SETUP
text: Dataset. We evaluate the proposed method on the VIST training set for the training, and evaluate story-to-images retrieval on VIST test set. VIST includes total 210,819 unique photos within 10,117 Flickr albums. There are two data type in VIST, DII(description-in-isolation) and SIS(story-insequence) respectively. The DII data only contain pairs of single sentence and sinle image, while SIS contains pairs of story and images for training. For our task, we only use SIS type. A single story in SIS consists of five successive images with the corresponding captions. After excluding broken images, we finally use 40,071 stories for training, 4,998 stories for validation and 5,055 stories for test set.
Baselines. We compare our approach with conventional text-to-image retrieval baselines. Primarily we adopt VSE++(Faghri et al., 2017), which exploits the idea of hard negative mining(Dalal & Triggs, 2005; Felzenszwalb et al., 2009) for learning visual-semantic embeddings for cross-modal retrieval. We also apply variant of VSE++, denoted as VSE0 which uses hinge-based triplet ranking loss(Karpathy & Fei-Fei, 2015; Kiros et al., 2014; Socher et al., 2014). Since both of them are single entry retrieval model, we also compare our retrieval system with existing sequential text-to-image model, CNSI(Ravi et al., 2018). CNSI is a global visual semantic matching model that utilizes precomputed modality feature as an encoder. Lastly, we conduct ablation study to examine the power of contextual text encoding, which is implemented by comparing our suggested retriever and the retriver without CSE.
Metrics. We useRecall@K(k = {1, 5, 10}) for main evaluation metric for VIST. For each description in the story, we retrieve top-K image predictions and measure the total percentage of sentence descriptions whose ground-truth images are whether ranked in the top-K predictions. Hence, the desired retrieval system maximizes recall at top-K. Also, we jointly evaluate on common retrieval metrics including median rank (MedR).
Hyperparameters. The target parameters are included in CSE, graph encoding head, and video captioning module. We unified optimizers for each module with Adam, setting distinct initial learning rates. In order, we set 0.0002, 0.0001, and 0.001. We only decay the learning rate for CSE, keeping initial learning rate for the first 15 epochs and then lower the learning rate to 0.00002 for remaining epochs. We set a minibatch size as 32, all parameters are trained for 30 epochs. We employ 300-dimensional GloVe as a feature representations of nodes and relations in a scene graph.

heading: 4.2 QUANTITATIVE RESULTS
text: Table 1 shows story-to-images retrieval performance on the VIST testing set. Overall, We observe sequential retrieval system(CNSI, Ours) performs better than single entry retrieval models(VSE0,
VSE++). In story-to-images setting, we could easily expect this kind of result since the formers(CNSI, Ours) sequentially embeds features from a given story compared to the latters(VSE0, VSE++). Besides, we presume usage of hard negatives for objective function for retrieval can bring positive effect for increasing the retrieval performance through comparing the results of VSE series. CNSI yields the best performance among baselines. Because CSE is one of the main contributors to maintain global semantic context in a story, our suggested retriever without CSE shows comparable performance with VSE series. Overall, our suggested pipeline outperforms the baselines. We empirically observe that leveraging scene graph similarity and dual framework helps gives better predictions in retrieved images. Nevertheless, there are not dramatic increases in performances. We assume that even the design choices of architecture for the main system pretty differs a lot, the differences in recall percentages of top-K output samples are relatively trivial. We leave evaluation on larger K, and other visual storytelling benchmark for our future work.

heading: 4.3 QUALITATIVE ANALYSIS
text: Figure 3 depicts samples of predicted images for when given a random story in the test set. We can observe that VSE++ does not maintain global context in both images and text. CNSI shows better result, still less incoherent. Compared to others, our retrieval system provides better visual descriptions. Nevertheless, we conclude there’s a lot of room to develop the performance in qualitative way.

heading: 5 CONCLUSION
text: In this paper, we introduced a new story-to-images retrieval framework that can alleviate potential pitfall of sampling visually incoherent images from a database. Our main technical contributions include (1) utilizing scene graph similarities with prior sample and (2)apply video captioning as a dual task. Our suggested framework shows superior performance in VIST benchmark compared to conventional text-to-image retrieval works.

Review summaries:
The authors propose a method for a sequence-to-sequence retrieval task. The idea of the task is given a set of sentences describing steps in a coherent story, the model needs to retrieve a single image per each sentence to illustrate that story. Unlike standard cross-modal retrieval where the goal is to just retrieve the image from a single caption, ignoring context, here the goal is to retrieve an image that is consistent with the retrieved visual story so far (i.e. each step in the story should be visually consistent across the retrieved images).  The authors method is designed to address these unique aspects of the problem. First, the authors encode a given sentence from the story using a contextual embedding computed using surrounding captions. They use this contextual sentence embedding to perform cross-modal retrieval of images. After doing this, they also take the previously chosen image and extract its scene graph. They essentially then compute a distance using both the scene graph embedding (from prior retrieved images) and the sentence embedding to determine the image at that time step. Finally, the authors take the set of retrieved images to generate captions and then measure the consistency of the generated captions with the story.  The authors present preliminary experimental results showing that the proposed method has very slight gains on the VIST (Visual Storytelling) dataset. They also present a qualitative result showing a retrieved "visual story."

The paper focuses on the topic of sequential text-to-image retrieval, where the goal is to match text sequences and images. Specifically, authors propose to use scene graph representation of images and learn to maximize graph embedding similarities among sampled images, so that morphological relevance with previously sampled images is considered during the process. Dual learning task is also constructed by calculating a video captioning loss.   Experiments are conducted on VIST dataset as for story-to-images retrieval, which shows the strength of the proposed method compared with some baseline models. It performs around 0.34 % better than works published before year 2018. 

This paper proposes a new story to images retrieval system (or sequence-of-texts to sequence-of-images retrieval). It consist of: (1) the Contextual Sentence Encoder (CSE) that embeds description text in each story taking advantage of the context, that is each description embedding feature also absorbing information from other description in the same story, (2) the scene graph generation and embedding module to encode visual contents of images, and (3) a dual learning task that tries to regenerate the original story from image sequence retrieval result. The proposed system outperforms other method on the VIST testset.