Graph Similarities and Dual Approach for Sequential Text-to-Image Retrieval
Authors: Keonwoo Kim
Sihyeon Jo
Seong-Woo Kim
Venue: ICLR 2022 Submitted (2022)
Keywords: sequential text-to-image retrieval
story-to-image retrieval
scene graph embedding
dual learning

Abstract: Sequential text-to-image retrieval, a.k.a. Story-to-images task, requires semantic alignment with a given story and maintaining global coherence in drawn image sequence simultaneously. Most of the previous works have only focused on modeling how to follow the content of a given story faithfully. This kind of overfitting tendency hinders matching structural similarity between images, causing an inconsistency in global visual information such as backgrounds. To handle this imbalanced problem, we propose a novel image sequence retrieval framework that utilizes scene graph similarities of the images and a dual learning scheme. Scene graph describes high-level information of visual groundings and adjacency relations of the key entities in a visual scene. In our proposed retriever, the graph encoding head learns to maximize graph embedding similarities among sampled images, giving a strong signal that forces the retriever to also consider morphological relevance with previously sampled images. We set a video captioning as a dual learning task that reconstructs the input story from the sampled image sequence. This inverse mapping gives informative feedback for our proposed retrieval system to maintain global contextual information of a given story. We also suggest a new contextual sentence encoding architecture to embed a sentence in consideration of the surrounding context. Through extensive experiments, Our proposed framework shows better qualitative and quantitative performance with Visual Storytelling benchmark compared to conventional story-to-image models.

Under review as a conference paper at ICLR 2022
GRAPH SIMILARITIES AND DUAL APPROACH FOR SEQUENTIAL TEXT-TO-IMAGE RETRIEVAL
Anonymous authors
Paper under double-blind review
ABSTRACT
Sequential text-to-image retrieval, a.k.a. Story-to-imagen task, requires semantic alignment with given story and maintaining global coherence in drawn image sequence simultaneously. Most of the previous works have only focused on mod- eling how to follow the content of a given story faithfully. This kind of overfit- ting tendency hinders matching structural similarity between images, causing an inconsistency in global visual information such as backgrounds. To handle this imbalanced problem, we propose novel image sequence retrieval framework that utilizes scene graph similarities of the images and a dual learning scheme. Scene graph describes high-level information of visual groundings and adjacency relations of the key entities in a visual scene. our proposed retriever, the graph encoding head learns to maximize embedding similarities among sampled images, giving a strong signal that forces the retriever to also consider morphological rel- evance with previously sampled images. We learning task that reconstructs the input story from the sampled image sequence. This inverse mapping gives informative feedback for our proposed retrieval sys- tem to maintain global contextual information of given story. We also suggest new contextual sentence encoding architecture to embed sentence in consider- ation of the surrounding context. Through extensive experiments, Our proposed framework shows better qualitative and quantitative performance with Visual Sto- rytelling benchmark compared to conventional story-to-image models.
video captioning as dual
INTRODUCTION
Visual content and textual description are in synergytic relations, have advantageous on delivering information and being proper expression means in real world communications. In this sense, successful cross-modal learning requires neural networks to deeply comprehend semantic relations between corresponding visual concepts and their textual descriptions. Examples include learning joint representation(Li ct al., 2015; Ma et al., 2021; Pan et al., 2013), text generation from visual Eepiction(Farhade) et al., 2010; Kulkarni al., 2013; Socher et al., 2014), and image video retrieval from text queries(Kim) (iim et al., 2015; Zitnick et al., 2013). Such multi-modal tasks build on common embedding space where semantically associated visual and text embeddings are jointly mapped into similar location. Hence, the key of multi-modal learning is understanding semantic relationship between distinct modality representations.
In this sense, image-text retrieval have been widely explored as a one of the core tasks in the multi- modal learning. The performance of such retrieval system has been measured by examining how the retrieved samples are semantically aligned with given query. In addition, most of them have been trained on singleton image-text pair dataset(e.g.. COCO(Lin et al., 2015)). These requirement and condition enforces the existing retrieval system to output the sample which is the most semantically fitted to the query. In other words, the performance the system is dominantly determined by the level of overfitting tendency of the output. The more output sample is relational to the input, the more performance increases. This comes with potential arguments in sequential cross-modal learning since it only cares inter relationship with certain modality input at given time step. while neglecting the intra relations of output modality. Consequently. when we qualitatively analyze output samples, can observe some incoherencies between output samples.

---

Under review as a conference paper at ICLR 2022
Images Retriever
Images
Story
Crewntied
1. Up atcoe bell goas.
diannt work,
Now diecus, soccer and yo aro of stuck the root
5g
5g
Docdee
Dosotor
Caption Generator
Figure I: An Overall illustration of GD framework
In sequential text-to-image, many of conventional approaches attempt maximize the visual co- herence with given set of descriptions. Though they largely improved modeling a cross-modn coherency between text and image, many of them neglect scenery coherence among images. In other words, since the system focuses on reflecting objects, optical discrepancies frequently occur between neighboring images. For example, peripheral visual cues like background objects, weather, and location changes as the story proceeds. We view this problem as an optimization problem whose goal is alleviating the one-sided overfitting certain modality. To address this, have consider an additional method which can teach the retriever to also count similarity in the other modality.
To overcome the suggested issue, we should handle two main challenges. First, we also have to measure how much are two successive landscapes similar, that is, we need an organized representa- tion that includes spatial information of each photo. Second, even if an image sequence was chosen considering scenery similarity, the curated image sequence must be still capable of narrate given input story. In this paper, we address aforementioned challenges with scene graph structure and dual learning framework.
Scene graph(Johnson et al., 2015) is a graph representation that includes abstract summaries of the objects and their relationships within an image. The objects are represented as nodes, and the relations between them usually represented as bidirectional edges. Due to it's canonicalized de- scription, scene graphs are very effective structured representation to easily figuring out global in- formation of both images and language. We generate scene graph of every image in training set to provide extra spatial information for the retriever. Hence, need to also develop encoding module to process scene graphs. In this paper, we attach GCN(graph convolutional network) (orr))((jjj)) (Kipf scene graph encoding head of the retriever which extracts compressed repre- sentation from nodes and edges from given scene graph. By drawing the scene graphs of retrieved image sequence and passing to the following encoding head, model can compute the extent of their similarities in graph embedding level.
& Welling, 2016)
Dealing with second challenge, we utilized dual learning framework which first introduced by Xia et al. (2016). In dual learning, two agents are involved where one agent solves primal task and the other solves dual task. For example, Xia et al. (2016) set English-to English-to-Fremi to-French translation as primal, and French-to- -ingggllihh as a dual. This cyclic loop gives each agent to learn error feedback from each other. We set pory-to-imagee as primal task, and images-to-story, which can regarded as video captioning, be dual task. The main purpose of leveraging dual framework providing informative error signal from dual task to the primal retriever system. The signal will teach the retriver to pick well-curated photos which become the input for the dual agent. In overall, dual task contributes to enhance the qualitative confidence for an output of the retriver.
Figure 1 depicts our aforementioned approaches in single architecture. We experiment our ap- proaches to the most popular visual storytelling benchmark: VIST(Huang et al., 2016). Specific configuration of VIST will be explained in section 4.
2

---

Under review as a conference paper at ICLR 2022
In nutshell, out key contributions are as follows:
1. We propose end-to-end sequential text-to-image retriever which achieves semantic co- herency in both inter and intra modalities, which resolves overfitting on single modality
2. We explore visual storytelling, which is non-trivial task that requires balance in semantic dependencies between distinct modalities.
RELATED WORKS
2.1
IMAGE RETRIEVAL
Much cross-modal retrieval research has dealt with learning latent space that jointly embeds im- ages and sentences into the same metric space. Especially, image-caption retrieval focuses on match- ing the most relevant image(s) from a database with given text query(Babenko et al., 2014). As retrieval-based system finds output candidates in pre-structured database, it is more advantagcous in overall likeliness in output qualities and shows less variation in sample mean quality. However, most of multi-mode retrieval systems deals with mapping a single instance. Few retrieval works have been explored to retrieve sequential outputs for structured queries. Some previous retrieval systems ranked images based on visual phrases(Sadeghi & Farhadi, 2011), or multi-attribute de- scriptions(Siddiquee et al., 2011). Kim et al. (2015) first proposed a ranking system to retrieve image sequences from natural language paragraphs. Recently, Chen et al. (2019) proposed visual segment matching framework to improve the output coherency and storyboard creation tool for how retrieval system be applied practical field application. Nevertheless, none of which considered the semantic arrangement with previously sampled instances.
2.2
SCENE GRAPHS
A scene graph depicts the contents of an image in the form of graph structure. Graph nodes rep- resent objects, their attributes, and the relationshipo among them. As a scene graph provide visual information in abstractive level, it has been proven to be effective in range of visual compre- hension tasks such as image retrieval(Yoon et al., 2020), image or video captioning(Cher 2020; Hong et al., 2020), visual question unswerigg(Damodaan tion(Johnson et al., 2018). number of applications utilizing scene graph information have been widely spread after large-scale scene graph annotations of real world images revealed from Visual Genome dataset(Krise (Krishna et al., 2017). In representation learning context, there have been many recent works focusing learning intermediate representations of scene graphs. Those works sug- gest scene graph representation as an useful compressed information for downstream applications. Raboh et al. (2020) proposed differentiable scene graphs, which can be trained end-to-end with rea- soning supervision. Maheshwari et al. (2021) constructed semantically rich representation through ranking (arrpatyy & Fei-Fei, 2015; Kiros et al., 2014) coupled with triple sampling strategy in image retrieval task. The closest related work (Yoon et al., 2020) proposed experimental approach that leverages similarities scene graph embedding for image-to-image retrieval task.
et al., et al., 2021), and image genera-
2.3 DUAL LEARNING
The application of dual learning was first proposed by (He et al., 2016a) to relieve the burden for preparing paired training data of English-to-Frened translation. The key idea of dual learning setting a primal and a dual task in a domain translation task. Learning source-t--target( and (arget-to-source(duaa) mappings simultaneously gives .... Especially, such a mutual reinforcing mechanism have shown effective results generation tasks in unsupervised settings. (Luo et al., 2he9)The advanced image-to-image translation GANs(Yi et al., 2017; Zhu et al., 2020) have shown competitive performance with unlabeled data by leveraging primal-dual relation guarantee stable domain translation performance without daunting the qualities of the generated images. In our work, set image sequence retrieval from given story primal task and regenerating the original story from chosen images as a dual. The informative error signals from reconstruction((dua) task enforces the retrieval agent choose more 'thoughtful" inputs for dual agent.
3

---

Under review as a conference paper at ICLR 2022
Avg Pool
(Pre-trained)
Story
Description
Bertance Enscaler
BILSTM
BILSTM
BILSTM
Attention
Description 2
Sonterce Ericodar
Keys
Query
BiLSTM
BILSTM
BILSTM
Description k
Embedding
Embedsing
Description N
Faccoi
Token1
Token2
Token L
Description k
Figure 2: An illustration of overall architecture of Contextual Sentence Encoder
APPROACH
PROBLEM STATEMENT
Let S = {8)) is a story comprises N text descriptions, where each description composed of single or few sentences. For the simplicity and avoid annotation confusion, consider single description as a single sentence. The goal of the task is retrieving an image sequence I {i that semantically aligned story S and each image iz is descriptive photo of the description Si=j. Hence, the retrieval system F {sss/= - { should map given story to the most probable image sequence without losing visual coherency. In this paper, we set I and S to have same cardinality to easily compare the matching results by one-to-one. We left one-to-many retrieval as our future work.
3.2 IMAGE SEQUENCE RETRIEVAL VIA GRAPH SIMILARITIES
Contextual Sentence Encoder. We give story for the direct input to our retrieval system. A story describes given image sequence(video) as of natural language descriptions. Many of previous text-image retrieval system receives a single text query as an input for their text encoders. If we try to process story with traditional text encoders, contextual connection between each description will be break inevitably since there's 's no other way to encode a story without recurrently injecting the description. consequence, feature representation each description will be separately located even in text-image joint embedding space. To encourage each description to be encoded in homo- geneous way, each embedding must imply contextual information of preceding text. In other words, the desired text encoder in story-to-images retrieval system must be context-recognizabl Thus, need to implement a novel story encoder for the task. Inspired by Chen et al. (2019), we suggest C.S.E C.S.E(CCntextul Sentence Encoder) which is suitable for sequential text-to-image retrieval.
?? describes the overall architecture of C.S.E. The key idea of C.S.E is considering structured hier- archical relation between and it's tokens. To extract dense representation of the description, the tokens the description should also contain relevant information about other surrounding texts.
C.S.E efficiently encodes each description 8; S. considering other surrounding descriptions in S. C.S.E. consists of bottm Bi-LSTM layer, intermediate attention layer, and final Bi-LSTM layer with global average pooling head. Let single description in S 8; {wi Wigs w;,,,,,,,] sequence of L token Wi, s.t. Wi, RIV S L. At time step i, the forward hidden state of the + bottom Bi-LSTM layer b receives of Redm embedding of ef, and the last hidden state h
w;,} } which is
4

---

Under review as a conference paper at ICLR 2022
The backward hidden state at time hig receives arguments in the same way. The final hidden state of at time & a concatenation of hig and 14- This can be summarized as
h LSTM
(hi W. Who bn.)
h LSTM
< Mi41 Wewf:
Wh, bn
(1)
h2
We can regard bf as a compreseed representation of token Wit. To embed the context of surrounding descriptions, set hj query of Bahdanau attention(Bahdanau al., 2016) module.
To compute textual coherency with the description 8; which includes query Wit, we select other descriptions except 8; as candidates for keys, and their hidden representations keys. We pass all 86 E S,1 < & I s.t. * to pre-trained sentence encoder(Kiros et al., 2014) and extract set of hidden representations {k: {l:".., k'-1, k++1, ,.., k Then, the value vector of attention of is computed as a weighted sum of keys and queries:
Σ
Maag
ISiTiSN
The attention weight ai is softmax score of attention layer with tanh activation computed by
exp (Riy)
0s Σ exp (Cak)
Cig va tanh (Wahi + Uakj)
where vg W., and Ua are learnable parameters of above attention layer. As of is value vector from h; and {k' .... k-1 1++1 ,..,, }: we can regard of dense token representation which embeds contextual semantic relationships with other surrounding descriptions in single story.
Now we pass the concatenation of of and h as new input for the second Bi-LSTM layer. We denote the hidden state of the second RNN-based layer g, and the second layer goes same progress to extract the th token representation of description 8i
g. = LSTM
9 [h] og W by)
LSTM
g [ht: wi] W b
9
1:
(4)
gi gi sé
avgpool ([g) :..; gt.l)
Finally, average-pont pooe along time steps 1 < t < L to get the final contextual representation of the description 8;.
3.3 SCENE GRAPH GENERATION AND EMBEDDING
Scene Graph Generation. A scene graph is an abstract representation of the visual contents of an image(Johnson et al., 2015). Formally, we define scene graph G of an image I as G = {0, R} a set of object nodes O, and set of relationship between two certain nodes R. Every relation- ship 'R. is represented by triplet of nodes (subjective, predicate, objective). explaining dynamical association between two nodes. Predicate is represented as undirected edge. We treat VePeennnnnn et al., 2014) embedding of the label name feature representation of all nodes in O and all edges R. Specific configuration will be explained in section 4. The combination of constituents of scene graph varies in related works(Chen et al., 2020; Hong et al., 2020; Yoon et al., 2020; Damodaran et al., 2021; Johnson et al., 2015; Krishna et al., 2017; Raboh al., 2020) example, some works(Yoon al., 2020; Ashual & Wolf, 2019) also included set of attributes
5

---

Under review as a conference paper at ICLR 2022
of objects A as another constituent of scene graph. In this work, we ... to relieve computational burden.
Generating scene graph 9 structure from an image / is equivalent to parsing an object detection result of a target image. We used Faster R-CNN(Ren et al., 2015) as an underlying detector. For each image I, the detector predicts set of region proposals B = bj, bt,,b,,....b. bn. Each proposal by E B comes with bounding box feature representation and probabilities for corresponding object label. Building on these information, we applied recently proposed method(Zellers al., 2018) for our scene graph generator. In detail, we utilized VG(Visual Genome) dataset(Krishna et al., 2017) configuration to assign proper predicate for constructing R. Predicate label is predicted based on frequency prior knowledge from VG. In overall retrieval pipeline, we generate scene graphs from primarily picked image sequence for given story S and pass to the graph encoding head layer to compare the scenery similarities via computing similarity scores of their graph embeddings.
Encoding Scene Graphs via GCN. In order to encode a scene graph with end-to-end fashion, we need suitable neural architecture that can operate directly on graph-structured data. We apply Graph Convolutional Network(Kipf & Welling, 2016) as our main graph encoding module since it's learning ability on graph representation have been proved in many related works(Johnson et al., 2018; Yoon et al., 2020; Kipf& & Welling, 2016; Chen et al., 2020).
3.4 DUAL LEARNING WITH VIDEO CAPTIONING
We adopt dual learning framework for enhancing a contextual coherency of output image sequence. The primal task can be represented as F : S - with proposed retriever F. We denote G as visual storyteller, which generates
dual agent, s.t. G I - S. G' can be also regarded figurative and consistent narrative for successive images. As F recurrently selects 7, we primarily considered image captioning(Karpanl
& Fei-Fei, 2015) as a dual. However, since each description in a single story narrates same circumstance, reconstructed descriptions will be semantically isolated from global context. On this, the output format must be a paragraph-level captions. Hence we regard dual task as a video captioning problem, to generated semantically aligned sentences while keeping global context. We construct video captioning module based upon the proposed architecture of GLAC Net(Kim et al., 2018).
3.5 TRAINING OBJECTIVES
image Sequence Retrieval. Let :0T textual encoder parameterized by θr, which is C.S.E in this paper. (S) = fr( RdT is a dense representation of an input description S which embeds contextual relations with surrounding S in a story. Similarly, let hy(()) = fv(1;0)) E R be feature representation from pre-trained image encoder(e.g.. VGG PGG19(Simonyan & Zisserman, 2014), 52 et al., 2016b)) before the last FC layer when given an input image I. We map hr andhy into joint embedding space through following linear transformation.
Φ WT 0T) W. lTT E Rd.
(5)
dv 0v) W (T) E Rd.
With linear operators different modalities. We measure the similarity of two distinct modality representation through cosine-similarity base score function, defined as
Wy Rd and Wy RRd..xd, we now can do vector computation across
s(8 sim(dd(s)) (i())
where sim a. b) a.b [. To retrieve an image which semantically matches current input descrip- tion and morphologically similar with previous image, we jointly use hinge ranking losses(Faghri et al., 2017; Chechik et al., 2010; Frome et al., 2007) between correct matches and other wrong ones and graph embedding similarity. At time step I, the step-wise loss & is
((s,
max A - sss + (), Σ
8
Σ mar(4 A - s((, + ss. 7).
+
+1 1 s(Wi,,, 1hx))

---

Under review as a conference paper at ICLR 2022
with margin A, negative description š for and negative image sample i for A hinge ranking loss, a.k.a triplet ranking loss, directs the retriever F to minimize the first and the second term via choosing the closest counterpart i or s than to any unmatched samples or by margin A. At the same time, the last term of & contributes F to search photo that is structurally similar to an carlier image. The total loss for image retrieval is Lvetrienat Σ. ((22,
Video Captioning. After selecting aligned images as a sequence I = {i1,-.,,N' we input I and story S as target ground-truth text, training video caption generator G reconstruct S. We use cross-entropy loss
N
V
Leaption
Va log ph
[=1 o=1
where a E {L.... V} an index of vocabulary set. Pl is predicted probability for i th token, and Vy is the target token.
Overall Objective. The overall objective is the sum of image sequence retrieval loss and video captioning loss. The total objective is as follows:
Liotal Leetreival + cappiion
(8)
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Dataset. We evaluate the proposed method the VIST training set for the training, and evaluate story-to-images retrieval on VIST test set. VIST includes total 210,819 unique photos within 10,117 and SIS(story-in-
Flickr albums. There two data type in VIST, pllidesrriptioninniioaaaai sequence) respectively. The DII data only contain pairs of single sentence and sinle image, while SIS contains pairs of story and images for training. For our task, we only use SIS type. A single story in SIS consists of five successive images with the corresponding captions. After excluding broken images, we finally use 40,071 stories for training, 4,998 stories for validation and 5,055 stories for test set.
Baselines. We compare our approach with conventional text-to-image retrieval baselines. Primarily we adopt (Fahria et al., 2017), which exploits the idea of hard negative mining(Dala) & Triggs, 2005; Felzenszwalb et al., 2009) for learning visual-semantic embeddings for cross-modal retrieval. We also apply variant of VSE++, denoted as VSEO which uses hinge-based triplet ranking loss(Karpathy & Fei-Fei, 2015; Kiros et al., 2014; Socher et al., 2014). Since both of them are single entry retrieval model, model, CNSI(Ravi et al., 2018). CNSI is global visual semantic matching model that utilizes pre- computed modality feature of contextual text encoding, which is implemented by comparing our suggested retriever and the retriver without CSE.
also compare
retrieval system with existing sequential text-to-image
encoder. Lastly, conduct ablation study to examine the power
Metrics. We use Recall@ (Kc {1, 5, 10)) for main evaluation metric for VIST. For each descrip- tion in the story, we retrieve top-K image predictions and measure the total percentage of sentence descriptions whose boound-truth images are whether ranked in the top-K predictions. Hence. the desired retrieval system maximizes recall top-K. Also, we jointly evaluate on common retrieval metrics including median rank (MedR).
Hyperparameters. The target parameters are included in CSE, graph encoding head, and video captioning module. We unified optimizers for each module with Adam, setting distinct initial learn- ing rates. In order, we set 0.0002, 0,0001, and 0.001. We only decay the learning rate for CSE, keeping initial learning rate for the first 15 epochs and then lower the learning rate to 0.00002 for remaining epochs. We set a minibatch size as 32, all parameters are trained for 30 epochs. We employ 300-dimensional GloVe feature representations of nodes and relations in scene graph.
4.2 QUANTITATIVE RESULTS
Table shows story-to-imane to-ima retrieval performance on the VIST testing set. Overall, We observe sequential retrieval Syst, Ours) performs better than single entry retrieval VSdels(VSFO.
7

---

Under review as a conference paper at ICLR 2022
Table 1: story-to-images retrieval performance the VIST testing percentage(%).
All scores reported in
Method
R@D
R@5
R@ 10
Med
VSEO
11.25
12.27
12.31
11.74
VSE++
12.28
12.29
13.27
12.57
CNSI
13.01
13.99
14.27
13.77
Ours(w/o CSE)
10.39
11.48
12.10
11.50
Ours
13.35
14.07
14.40
13.98
VSE++), In story-to-images setting, could easily expect this kind of result since the form- ers(CNSI, Ours) sequentially embeds features from given story compared to the latters VSEO, VSE++). Besides, presume usage of hard negatives for objective function for retrieval can bring positive effect for increasing the retrieval performance through comparing the results of VSE series. CNSI yields the best performance among baselines. Because CSE is of the main contributors to maintain global semantic context in a story, our suggested retriever without CSE shows compa- rable performance with VSE series. Overall, our suggested pipeline outperforms the baselines. We empirically observe that leveraging scene graph similarity and dual framework helps gives better predictions in retrieved images. Nevertheless, there are not dramatic increases in performances. We assume that even the design choices of architecture for the main system pretty differs lot, the dif- ferences in recall percentages of top-K output samples are relatively trivial. We leave evaluation on larger K, and other visual storytelling benchmark for our future work.
Graduation day has finally arrived. All the students startd filing in. He was thrilled finally be graduation. Everyone posed for pictures outside the venue. And capp the day; we all hung out at the pool
GT
VSE++
CNSI
Ours
Figure 3: Qualitative comparison predicted images from test set sample.
4.3 QUALITATIVE ANALYSIS
Figure 3 depicts samples of predicted images for when given random story in the test set. We can observe that VSE++ does not maintain global context both images and text. CNSI shows better result, still less incoherent. Compared others, retrieval system provides better visual descrip- tions. Nevertheless, we conclude there's a lot of room to develop the performance qualitative way.
8

---

Under review as a conference paper at ICLR 2022
5 CONCLUSION
In this paper, we introduced a new story-to-images retrieval framework that can alleviate potential pitfall of sampling visually incoherent images from database. Our main technical contributions include (1) utilizing scene graph similarities with prior sample and (2)apply video captioning as dual task. Our suggested framework shows superior performance in VIST benchmark compared to conventional -magee retrieval works.
REFERENCES
Oron Ashual and Lior Wolf. Specifying object attributes and relations in interactive scene genera- tion, 2019.
Artem Babenko, Anton Slesarev, Alexandr Chigorin, and Victor Lempitsky. Neural codes for image retrieval. In European conference on computer vision, pp. 584-599 Springer, 2014.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate, 2016.
Gal Chechik, Varun Sharma, Uri Shalit, and Samy Bengio. Large scale online learning of image similarity through ranking. Journal Machine Learning Research, 11(3), 2010.
Shizhe Chen, Bei Liu, Jianlong Fu, Ruihua Song, Qin Jin, Pingping Lin, Xiaoyu Qi, Chunting Wang, and Jin Zhou. Neural storyboard artist: Visualizing stories with coherent image sequences. In Proceedings of the 27th ACM International Conference Multimedia, pp. 1236-2245 2019.
Shizhe Chen, Qin Jin, Peng Wang, and Qi Wu. Say you wish: Fine-grained control of image caption generation with abstract scene graphs, 2020.
Navneet Dalal and Bill Triggs, Histograms of oriented gradients for human detection. In 2005 IEEE computer society conference on computer vision and pattern recognition (CVPR'05), volume 1, pp. 886-893, Icee, 2005.
Vinay Damodaran, Sharanya Chakravarthy, Akshay Kumar, Anjana Umapathy, Teruko Mitamura, Yuta Nakashima, Noa Garcia, and Chenhui Chu. Understanding the role of scene graphs in visual question answering, 2021.
Fartash Faghri, David J Fleet, Jamie Ryan Kiros, and Sanja Fidler. Vse++: Improving visual- semantic embeddings with hard negatives. arXiv preprint 1rXiv:170..05612.220 2017.
Ali Farhadi, Mohsen Hejrati, Mohammad Amin Sadeghi, Peter Young, Cyrus Rashtchian, Julia Hockenmaier, and David Forsyth. Every picture tells story: Generating sentences from images. In European conference on computer vision, pp. 15-29. Springer, 2010.
Pedro F Felzenszwalb, Ross B Girshick, David McAllester, and Deva Ramanan. Object detection with discriminatively trained part-based models. IEEE transactions on pattern analysis and ma- chine intelligence, 32(9): 667--1655 2009.
Andrea Frome, Yoram Singer, Fei Sha, and Jitendra Malik. Learning globally-consiseent local dis- tance functions for shape-based image retrieval and classification. In 2007 IEEE 11th Interna- tional Conference on Computer Vision, pp. 1-8. IEEE, 2007.
Di He, Yingce Xia, Tao Qin, Liwei Wang, Nenghai Yu, Tie- Liu, and Wei-Ying Ma. Dual learning for machine translation. Advances in neural information processing systems, 29:820- 828, 2016a.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016b.
Xudong Hong, Rakshith Shetty, Asad Sayeed, Khushboo Mehra, Vera Demberg, and Bernt Schiele. Diverse and relevant visual storytelling with scene graph embeddings. In Proceedings of the 24th Conference Computational Natural Language Learning, pp. 420-430. 2020.

---

Under review as a conference paper at ICLR 2022
Ting- Hao Huang. Francis Ferraro, Nasrin Mostafazadeh, Ishan Misra, Aishwarya Agrawal, Jacob Devlin, Ross Girshick, Xiaodong He, Pushmeet Kohli, Dhruv Batra, et al. Visual storytelling. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pp. 1233-1239, 2016.
Justin Johnson, Ranjay Krishna, Michael Stark, Li-Jia Li, David Shamma, Michael Bernstein, and Li Fei-Fei. Image retrieval using scene graphs. Proceedings of the IEEE conference on com- puter vision and pattern recognition, pp. 3668-3678, 2015.
Justin Johnson, Agrim Gupta, and Li Fei-Fei. Image generation from scene graphs. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1219-1228, 2018.
Andrej Karpathy and Li Fei-Fei. Deep visual-semantic alignments for generating image descrip- tions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp-
3128-3137 2015.
Gunhee Kim, Seungwhan Moon, and Leonid Sigal. Ranking and retrieval of image sequences from multiple paragraph queries. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 1993-2001, 2015.
Taehyeong Kim, Min-Oh Heo, Seonil Son, Kyoung- Whhng Park, and Byoung-Tak Zhang. Glac net: Glocal attention cascading networks for multi-image cued story generation. arXiv preprint arXiv:
2018.
Thomas N Kipf and Max Welling, Semi-supervised classification with graph convolutional net- works. arXiv preprint arXiv: 160:002090 2016.
Ryan Kiros, Ruslan Salakhutdinov, and Richard S Zemel. Unilying visual-semantic embeddings with multimodal neural language models. arXiv preprint 1rXiv:1411.253: 2014.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanic Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting lan- guage and vision using crowdsourced dense image annotations. International of computer vision, 123(1):32-73, 2017.
Girish Kulkarni, Visruth Premraj, Vicente Ordonez, Sagnik Dhar, Siming Li, Yejin Choi, Alexan- der C Berg, and Tamara L Berg. Babytalk: Understanding and generating simple image descrip- tions. IEEE transactions on pattern analysis and machine intelligence, 35(12):8811-2903, 2013.
Yangyan Li, Hao Su, Charles Ruizhongtai Qi, Noa Fish, Daniel Cohen-Or, and Leonidas J Guibas. Joint embeddings of shapes and images via cnn image purification. ACM transactions on graphics (TOG), 34(6):1-12 2015.
Tsung- Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pictro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dollár. Microsoft coco: Common objects in context, 2015.
Fuli Luo, Peng Li, Jie Zhou, Pengcheng Yang, Baobao Chang, Zhifang Sui, and Xu Sun. A dual reinforcement learning framework for unsupervised text style transfer. arXiv preprint 190i: 10060, 2019.
Zhiyang Ma, Wenfeng Zheng, Xiaobing Chen, and Lirong Yin. Joint embedding vqa model based on dynamic word vector. PeerJ Computer Science, 7:c353, 2021.
Paridhi Maheshwari, Ritwick Chaudhry, and Vishwa Vinay. Scene graph embeddings using relative similarity supervision. arXiv preprint arXiv:2104.02387 2021.
Sinno Jialin Pan. Zhiqiang Toh, and Jian Su. Transfer joint embedding for cross-domain named entity recognition. ACM Transactions on Information Systems (TOIS), 31(2):1-27, 2013.
Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 1532-1543, 2014.
10

---

Under review as a conference paper at ICLR 2022
Moshiko Raboh, Roei Herzig, Jonathan Berant, Gal Chechik, and Amir Globerson. Differentiable scene graphs. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1488-1497, 2020.
Hareesh Ravi, Lezi Wang, Carlos Muniz, Leonid Sigal, Dimitris Metaxas, and Mubbasir Kapadia. Show me a story: Towards coherent neural story illustration. In Proceedings of the IEEE Confer- ence on Computer Vision and Pattern Recognition, pp. 7613-7621, 2018.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-enm: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28: 91-99, 2015.
Mohammad Amin Sadeghi and Ali Farhadi. Recognition using visual phrases. IEEE, 2011.
Behjat Siddiquie, Rogerio S Feris, and Larry S Davis. Image ranking and retrieval based on multi- attribute queries. In CVPR 2011, pp. 801-808. IEEE, 2011.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1556, 2014.
Richard Socher, Andrej Karpathy, Quoc V Le, Christopher D Manning, and Andrew Y Ng. Grounded compositional semantics for finding and describing images with sentences. Trans- actions of the Association for Computational Linguistics, 2:207-218, 2014.
Yingce Xia, Di He, Tao Qin, Liwei Wang, Nenghai Yu, Tie-Yan Liu, and Wei-Ying Ma. Dual learning for machine translation, 2016.
Zili Yi, Hao Zhang, Ping Tan, and Minglun Gong. Dualgan: Unsupervised dual learning for image- to-image translation. In Proceedings of the IEEE international conference on computer vision, pp. 2849-2857, 2017.
Sangwoong Yoon, Woo Young Kang, Sungwook Jeon, SeongEun Lee, Changjin Han, Jonghun Park, and Eun-Sol Kim. Image-to-image retrieval by learning similarity between scene graphs, 2020.
Rowan Zellers, Mark Yatskar, Sam Thomson, and Yejin Choi. Neural motifs: Scene graph parsing with global context. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5831-5840, 2018.
Jun-Yan Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks, 2020.
Lawrence Zitnick, Devi Parikh, and Lucy Vanderwende. Learning the visual interpretation of sentences. In Proceedings of the IEEE International Conference Computer Vision, pp. 1681- 1688, 2013.


Reviews:
[{"review_id": "bMBlXEWrt8v", "reviewer": "Reviewer_6mhj", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n\n+ The idea of optimizing graph embedding similarities among sampled images is interesting.\n\n+ The proposed method shows some strength compared with some existing methods.\n\n+ The paper is easy to follow. \n\n\nConcerns:\n\n- The comparisons are actually unfair with existing methods. Data and annotations related to scene graph generation (e.g. Visual genome dataset) are used for training the proposed model. However, existing methods do not rely on such data and labels. Comparisons in Table 1 is unfair.\n\n- Based on unfair comparisons, the strength of the proposed model is quite limited. It achieves only 0.34 % improvements compared with CNSI which is published in year 2018. VSE++ and VSE0 are very basic baseline models in the common image-text matching setting. Why not including more recent image-text matching models rather than VSE++ into comparisons?\n\n- Experimental designs and results are not convincing. There is no ablation studies to show and evaluate the contributions of each part of the model. The complex designs here may be redundant. \n\n- Technical contributions are also limited. Using scene graph for cross-modal retrieval is not new [a]. Considering both retrieval and generation is also not new [b]. Considering both graph space and joint space is interesting but not novel enough and its effectiveness is not well supported due to the missing ablation studies.  \n\n[a] Cross-modal Scene Graph Matching for Relationship-aware Image-Text Retrieval, WACV 2020\n[b] Look, Imagine and Match: Improving Textual-Visual Cross-Modal Retrieval with Generative Models, CVPR 2018\n\n- The writing needs improvements and many typos need to be revised, e.g. Eq, L_retreival -> L_retrieval,  image Sequence Retrieval -> Image Sequence Retrieval, etc (not listing all of them). \n", "summary_of_the_paper": "The paper focuses on the topic of sequential text-to-image retrieval, where the goal is to match text sequences and images. Specifically, authors propose to use scene graph representation of images and learn to maximize graph embedding similarities among sampled images, so that morphological relevance with previously sampled images is considered during the process. Dual learning task is also constructed by calculating a video captioning loss. \n\nExperiments are conducted on VIST dataset as for story-to-images retrieval, which shows the strength of the proposed method compared with some baseline models. It performs around 0.34 % better than works published before year 2018. ", "summary_of_the_review": "The review is mainly based on the technical novelties/contributions, experimental designs, state-of-the-art comparisons, etc. Please refer to the Strength and Concerns parts in the Main Review section for more details. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "FKxe3gnf9Zg", "reviewer": "Reviewer_1pAQ", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Strengths\nThe authors are on the right track with the method design. It is clear that capturing the context of other sentences (i.e. baking the context of the entire story into a single sentence embedding) is important to ensure that the retrieved image is not just feasible for the given sentence, but also for possible future sentences. The authors try to address this by using a contextualized embedding method for each sentence. \nAdditionally, the authors identify that visual stories should also usually have some visual consistency. In the example they give with a ball being kicked on the roof, the authors show one image with a ball in the air, next it is moving higher, and eventually we see an image of a house with the ball on the roof. Thus, the authors identify that their should be some sort of continuity between the scene graphs for each textual description (as image_{t-1} shifts to image_{t}). Thus they extract scenes and impose a constraint that the retrieved image shouldn't be too far from the previously retrieved scene graph.\n\nI also like the idea of using cycle consistency (that is to say, once we retrieve a set of images from a textual story, can we try to regenerate that same story from the set of images). However, I have a lot of questions about that that I will discuss in the weaknesses section, however I think the basic premise is reasonable.\n\n# Weaknesses\nUnfortunately, the paper suffers from major weaknesses. Before getting into technicalities, the paper draft seems highly unpolished, draftlike, and incomplete. For example, Figure 1 is impossible to read and had to be zoomed to an extremely high degree to understand. Other portions of the paper are incomplete. For example, on page 3, sentences just trail off. \"Learning source-to-target(primal) and target-to-source(dual) mappings simultaneously gives......\" \nThere are numerous occasions throughout the text where there are \"placeholder\" ellipses. Also, the top of page 6 (\"In this work, we ......\"). \nThere are spelling errors throughout the text (e.g. \"compreseed\"), inconsistent capitalization of sections (e.g. page 6- image Sequence Retrieval is lower case \"i\"), and the text is otherwise poorly written, hard to follow, and in many instances too light on details to adequately understand what is being done.\n\nI will briefly mention technical critiques now. First, the authors method of using \"contextual sentence encoder\" is as far as I can tell essentially identical to the Chen et al. 2019 reference. Thus, there is no novelty on the \"CSE\" portion of this. The \"graph encoding\" portion is very light on details, but also appears to primarily rely on off-the-shelf methods methods (e.g. Faster R-CNN, other cited works - e.g. Zellers, et al.). There are no details about the GCN used, but it seems to be a standard model and I can't assess any novelty there.\nThe authors use a standard triplet loss to train the method, with an extra constraint that imposes that the retrieved image's graph embedding should be close to the prior one.\nOne shortcoming of this is that it isn't conditioned on the text at all. For example, there are many visual stories where it makes sense that each image should be consistent with its prior image. But what if there is a sudden turn in the story? What if the text says something about a different scene or location (this occurs in the pool party example in the single qualitative result). In this case, the model will be constrained to pick an image with a similar graph embedding as a previous image, but this may actually not be desirable in that case. A better strategy would be to weight the graph consistency based on some relationship of text query $t$ with $t-1$.\n\nI am also very concerned about the \"captioning\" consistency piece. The authors mention this as a critical part of their method, but there is only an inch of space devoted to it on page 7. Basically, the idea is after we select images, we want to use a video captioning method to regenerate the original story. This is a good idea in practice, but I am very concerned about this. First of all, the image retrieval method retrieves an image at each time step. The authors then say, after selecting the aligned images, they input them into a video captioning method and train it with a loss on the original story text (ground truth text). Ideally, what we would want to do is backpropagate through this captioning method, back through the image retrieval method, and through the entire system.\nHowever, I'm not sure this is possible. How can the gradients of the video captioner be backpropped back through the retrieval process? Analogous to image captioning methods, once one generates words from a captioning method, one cannot then backpropagate through the chosen words (since the word selection is non-differentiable). Here, since there is a set of images, and presumably the authors take top-1, how does the captioning method get to consider other possibilities? How is this process differentiable? It would seem some sort of reinforcement learning is necessary - but there are no details.\n\nFor experiments, the authors compare against older baselines (e.g. VSE++), when much more recent methods are available. The experiments are very underwhelming. We see the authors method for R@1 gets 13.35, while best baseline gets 13.01. Median r of authors' method is 13.98, while best baseline is 13.77. In sum, there is no significant gain over prior state of the art work. I would also argue that the authors need to do a much more rigorous experimental comparison to more related work (e.g. Batra, Vishwash, et al. \"Variational Recurrent Sequence-to-Sequence Retrieval for Stepwise Illustration.\" Advances in Information Retrieval 12035 (2020): 50.). The authors also include a single qualitative result, but it isn't clear how representative it is or that the results are even particularly meaningful. No human study is done to assess the quality of the chosen \"stories\".", "summary_of_the_paper": "The authors propose a method for a sequence-to-sequence retrieval task. The idea of the task is given a set of sentences describing steps in a coherent story, the model needs to retrieve a single image per each sentence to illustrate that story. Unlike standard cross-modal retrieval where the goal is to just retrieve the image from a single caption, ignoring context, here the goal is to retrieve an image that is consistent with the retrieved visual story so far (i.e. each step in the story should be visually consistent across the retrieved images).\n\nThe authors method is designed to address these unique aspects of the problem. First, the authors encode a given sentence from the story using a contextual embedding computed using surrounding captions. They use this contextual sentence embedding to perform cross-modal retrieval of images. After doing this, they also take the previously chosen image and extract its scene graph. They essentially then compute a distance using both the scene graph embedding (from prior retrieved images) and the sentence embedding to determine the image at that time step. Finally, the authors take the set of retrieved images to generate captions and then measure the consistency of the generated captions with the story.\n\nThe authors present preliminary experimental results showing that the proposed method has very slight gains on the VIST (Visual Storytelling) dataset. They also present a qualitative result showing a retrieved \"visual story.\"", "summary_of_the_review": "In sum, the authors tackle the problem of visual storytelling by proposing a method that captures sentence context and prior chosen image scene structure for sequence to sequence retrieval. The task is interesting, however, the authors' method lacks technical novelty and key components of the method are unclear. The paper is highly draft-like, with figures unreadable, portions unwritten, and writing errors throughout. The experiments show slight gains of the authors' method, but it isn't clear if these gains are significant and there is a lack of comparison to recent relevant research. \nAt present, the paper is not ready for publication and needs substantial work.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "9KOaY54ka3H", "reviewer": "Reviewer_iXYw", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- For the some part, the paper is well written and easy to read, especially the 1st 2 sections, I felt that the motivations are strong.\n- The proposed system cleverly leverages existing techniques (CSE, graph generation & embedding, dual learning) to realize the motivations: CSE produces context informative embedding of text, and the scene graph based image retrieval helps improving global contextual consistency of retrieved images. The dual learning task is basically the reverse translation task makes sense; it is commonly seen in the literature, helping the forward translation task more consistent.\n\nWeaknesses:\nI have the following troubles understanding the proposed system:\n- (1) The images retriever part is figure 1 is not clear (or was not explained at all). It seems that it performs retrieval of 1 image at a time; at each iteration, there is a 1st retrieval phase and a 2nd retrieval phase? I wasn't able to piece them together by just looking at the figure.\n- (2) 1 of the motivation of the proposed is that existing works \"neglect scenery coherence among images\", I was expecting that something in section 3 would explicitly address that? Section 3.3 is just describing some existing work that is being used here, not connecting everything together, or maybe the authors have not finish writing this section?\n- (3) It's not clear what is the goal of the retrieval loss in section 3.5 where VGG/ResNet is being used as image encoder. I'm guessing they are used to train the CSE only, and the pretrained VGG/ResNet is never used in testing? But so which loss optimizes the graph generation and embedding part of the system?\n\nI feel that the experiments are not extensive enough\n- (4) no ablation study, this is the biggest weakness of this work I think. It is good to study & validate each components effect, whether they are leveraged from existing works or newly proposed. Have we tried a different text embeder, or graph generation method, or graph embedding method, or ranking loss, etc. What is the effect of the additional dual learning task? I think the ours vs ours w/o CSE is very insightful comparison, but more is needed, it's not enough to just say \"We empirically observe that leveraging scene graph similarity and dual framework helps gives better predictions in retrieved images\" without convincing evidence & extensive study.\n- (5) While it is expected that the proposed would outperform the single-image-to-text methods (VSE0, VSE++), the equivalent proposed (ours w/o CSE) really under perform making me think that maybe the \"backbone\" of the proposed is not well optimized. That might be because of the lack of validations mentioned above.\n- (6) Maybe more qualitative analysis, depict both cases where it does well and cases where it doesn't.\n\ntypos: ?? in page 4, last paragraph in page 3, first line of page 6, ", "summary_of_the_paper": "This paper proposes a new story to images retrieval system (or sequence-of-texts to sequence-of-images retrieval).\nIt consist of: (1) the Contextual Sentence Encoder (CSE) that embeds description text in each story taking advantage of the context, that is each description embedding feature also absorbing information from other description in the same story, (2) the scene graph generation and embedding module to encode visual contents of images, and (3) a dual learning task that tries to regenerate the original story from image sequence retrieval result.\nThe proposed system outperforms other method on the VIST testset.", "summary_of_the_review": "This work is interesting but underdeveloped.\n\nI think the proposed system passes the bar as a novel method. While most of its component are from existing works, the integration and design of the whole system for the story-to-images retrieval task is a significant technical contribution.\n\nHowever, the experiment & ablation study are not extensive enough. Writings seem unfinished last few sections of the paper.\n\nSo even though this work is very promising, I'd recommend not accept the paper at the current state. The authors are encouraged to spend more time polishing it.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewers unanimously vote for rejection for several reasons. First, the draft is incomplete and difficult to read. Second, one of the proposed methods (contextual sentence encoder) appears the same as past work, while the other proposed method (graph encoding) is difficult to interpret from what is written. Third, the draft is missing comparisons with recent work, and some included comparisons may be unfair due to data conditions. No author response was provided. The reviewer consensus is that this draft is underdeveloped, and not yet ready for submission or publication.
Decision: Reject