Graph Similarities and Dual Approach for Sequential Text-to-Image Retrieval

Sequential text-to-image retrieval , a.k.a . Story-to-images task , requires semantic alignment with a given story and maintaining global coherence in drawn image sequence simultaneously . Most of the previous works have only focused on modeling how to follow the content of a given story faithfully . This kind of overfitting tendency hinders matching structural similarity between images , causing an inconsistency in global visual information such as backgrounds . To handle this imbalanced problem , we propose a novel image sequence retrieval framework that utilizes scene graph similarities of the images and a dual learning scheme . Scene graph describes high-level information of visual groundings and adjacency relations of the key entities in a visual scene . In our proposed retriever , the graph encoding head learns to maximize embedding similarities among sampled images , giving a strong signal that forces the retriever to also consider morphological relevance with previously sampled images . We set a video captioning as a dual learning task that reconstructs the input story from the sampled image sequence . This inverse mapping gives informative feedback for our proposed retrieval system to maintain global contextual information of a given story . We also suggest a new contextual sentence encoding architecture to embed a sentence in consideration of the surrounding context . Through extensive experiments , Our proposed framework shows better qualitative and quantitative performance with Visual Storytelling benchmark compared to conventional story-to-image models . 1 INTRODUCTION . Visual content and textual description are in synergytic relations , have advantageous on delivering information and being a proper expression means in real world communications . In this sense , successful cross-modal learning requires neural networks to deeply comprehend semantic relations between corresponding visual concepts and their textual descriptions . Examples include learning joint representation ( Li et al. , 2015 ; Ma et al. , 2021 ; Pan et al. , 2013 ) , text generation from visual depiction ( Farhadi et al. , 2010 ; Kulkarni et al. , 2013 ; Socher et al. , 2014 ) , and image or video retrieval from text queries ( Kim et al. , 2015 ; Zitnick et al. , 2013 ) . Such multi-modal tasks build on common embedding space where semantically associated visual and text embeddings are jointly mapped into similar location . Hence , the key of multi-modal learning is understanding semantic relationship between distinct modality representations . In this sense , image-text retrieval have been widely explored as a one of the core tasks in the multimodal learning . The performance of such retrieval system has been measured by examining how the retrieved samples are semantically aligned with given query . In addition , most of them have been trained on singleton image-text pair dataset ( e.g. , COCO ( Lin et al. , 2015 ) ) . These requirement and condition enforces the existing retrieval system to output the sample which is the most semantically fitted to the query . In other words , the performance of the system is dominantly determined by the level of overfitting tendency of the output . The more output sample is relational to the input , the more performance increases . This comes with potential arguments in sequential cross-modal learning since it only cares inter relationship with certain modality input at given time step , while neglecting the intra relations of output modality . Consequently , when we qualitatively analyze output samples , we can observe some incoherencies between output samples . In sequential text-to-image , many of conventional approaches attempt to maximize the visual coherence with given set of descriptions . Though they largely improved modeling a cross-modal coherency between text and image , many of them neglect scenery coherence among images . In other words , since the system focuses on reflecting objects , optical discrepancies frequently occur between neighboring images . For example , peripheral visual cues like background objects , weather , and location changes as the story proceeds . We view this problem as an optimization problem whose goal is alleviating the one-sided overfitting to a certain modality . To address this , we have to consider an additional method which can teach the retriever to also count similarity in the other modality . To overcome the suggested issue , we should handle two main challenges . First , we also have to measure how much are two successive landscapes similar , that is , we need an organized representation that includes spatial information of each photo . Second , even if an image sequence was chosen considering scenery similarity , the curated image sequence must be still capable of narrate given input story . In this paper , we address aforementioned challenges with scene graph structure and dual learning framework . Scene graph ( Johnson et al. , 2015 ) is a graph representation that includes abstract summaries of the objects and their relationships within an image . The objects are represented as nodes , and the relations between them is usually represented as bidirectional edges . Due to it ’ s canonicalized description , scene graphs are very effective structured representation to easily figuring out global information of both images and language . We generate scene graph of every image in training set to provide extra spatial information for the retriever . Hence , we need to also develop a encoding module to process scene graphs . In this paper , we attach GCN ( graph convolutional network ) ( Kipf & Welling , 2016 ) as a scene graph encoding head of the retriever which extracts compressed representation from nodes and edges from given scene graph . By drawing the scene graphs of retrieved image sequence and passing to the following encoding head , model can compute the extent of their similarities in graph embedding level . Dealing with second challenge , we utilized dual learning framework which first introduced by Xia et al . ( 2016 ) . In dual learning , two agents are involved where one agent solves primal task and the other solves dual task . For example , Xia et al . ( 2016 ) set English-to-French translation as a primal , and French-to-English as a dual . This cyclic loop gives each agent to learn error feedback from each other . We set story-to-images as a primal task , and images-to-story , which can be regarded as video captioning , be a dual task . The main purpose of leveraging dual framework is providing informative error signal from dual task to the primal retriever system . The signal will teach the retriver to pick well-curated photos which become the input for the dual agent . In overall , dual task contributes to enhance the qualitative confidence for an output of the retriver . Figure 1 depicts our aforementioned approaches in a single architecture . We experiment our approaches to the most popular visual storytelling benchmark : VIST ( Huang et al. , 2016 ) . Specific configuration of VIST will be explained in section 4 . In a nutshell , out key contributions are as follows : 1 . We propose end-to-end sequential text-to-image retriever which achieves semantic coherency in both inter and intra modalities , which resolves overfitting on single modality 2 . We explore visual storytelling , which is non-trivial task that requires balance in semantic dependencies between distinct modalities . 2 RELATED WORKS . 2.1 IMAGE RETRIEVAL . Much cross-modal retrieval research has dealt with learning a latent space that jointly embeds images and sentences into the same metric space . Especially , image-caption retrieval focuses on matching the most relevant image ( s ) from a database with a given text query ( Babenko et al. , 2014 ) . As retrieval-based system finds output candidates in a pre-structured database , it is more advantageous in overall likeliness in output qualities and shows less variation in sample mean quality . However , most of multi-modal retrieval systems deals with mapping a single instance . Few retrieval works have been explored to retrieve sequential outputs for structured queries . Some previous retrieval systems ranked images based on visual phrases ( Sadeghi & Farhadi , 2011 ) , or multi-attribute descriptions ( Siddiquie et al. , 2011 ) . Kim et al . ( 2015 ) first proposed a ranking system to retrieve image sequences from natural language paragraphs . Recently , Chen et al . ( 2019 ) proposed visual segment matching framework to improve the output coherency and storyboard creation tool for how retrieval system can be applied to practical field application . Nevertheless , none of which considered the semantic arrangement with previously sampled instances . 2.2 SCENE GRAPHS . A scene graph depicts the contents of an image in the form of graph structure . Graph nodes represent objects , their attributes , and the relationshipo among them . As a scene graph provide visual information in abstractive level , it has been proven to be effective in a range of visual comprehension tasks such as image retrieval ( Yoon et al. , 2020 ) , image or video captioning ( Chen et al. , 2020 ; Hong et al. , 2020 ) , visual question answering ( Damodaran et al. , 2021 ) , and image generation ( Johnson et al. , 2018 ) . A number of applications utilizing scene graph information have been widely spread after a large-scale scene graph annotations of real world images revealed from Visual Genome dataset ( Krishna et al. , 2017 ) . In representation learning context , there have been many recent works focusing on learning intermediate representations of scene graphs . Those works suggest scene graph representation as an useful compressed information for downstream applications . Raboh et al . ( 2020 ) proposed differentiable scene graphs , which can be trained end-to-end with reasoning supervision . Maheshwari et al . ( 2021 ) constructed semantically rich representation through ranking loss ( Karpathy & Fei-Fei , 2015 ; Kiros et al. , 2014 ) coupled with triple sampling strategy in image retrieval task . The closest related work ( Yoon et al. , 2020 ) proposed experimental approach that leverages similarities of scene graph embedding for image-to-image retrieval task . 2.3 DUAL LEARNING . The application of dual learning was first proposed by ( He et al. , 2016a ) to relieve the burden for preparing paired training data of English-to-French translation . The key idea of dual learning is setting a primal and a dual task in a domain translation task . Learning source-to-target ( primal ) and target-to-source ( dual ) mappings simultaneously gives ..... Especially , such a mutual reinforcing mechanism have shown effective results on generation tasks in unsupervised settings . ( Luo et al. , 2019 ) The advanced image-to-image translation GANs ( Yi et al. , 2017 ; Zhu et al. , 2020 ) have shown competitive performance with unlabeled data by leveraging primal-dual relation to guarantee stable domain translation performance without daunting the qualities of the generated images . In our work , we set an image sequence retrieval from given story as a primal task and regenerating the original story from chosen images as a dual . The informative error signals from reconstruction ( dual ) task enforces the retrieval agent to choose more ’ thoughtful ’ inputs for dual agent . 3 APPROACH . 3.1 PROBLEM STATEMENT . Let S = { si } Ni=1 is a story comprises N text descriptions , where each description composed of a single or a few sentences . For the simplicity and avoid annotation confusion , consider a single description as a single sentence . The goal of the task is retrieving an image sequence I = { ij } Nj=1 that is semantically aligned to a story S and each image ij is descriptive photo of the description si=j . Hence , the retrieval system F : { si } Ni=1 7−→ { ij } j j=1 should map given story to the most probable image sequence without losing visual coherency . In this paper , we set I and S to have a same cardinality to easily compare the matching results by one-to-one . We left one-to-many retrieval as our future work .

Summary: The authors propose a method for a sequence-to-sequence retrieval task. The idea of the task is given a set of sentences describing steps in a coherent story, the model needs to retrieve a single image per each sentence to illustrate that story. Unlike standard cross-modal retrieval where the goal is to just retrieve the image from a single caption, ignoring context, here the goal is to retrieve an image that is consistent with the retrieved visual story so far (i.e. each step in the story should be visually consistent across the retrieved images).  The authors method is designed to address these unique aspects of the problem. First, the authors encode a given sentence from the story using a contextual embedding computed using surrounding captions. They use this contextual sentence embedding to perform cross-modal retrieval of images. After doing this, they also take the previously chosen image and extract its scene graph. They essentially then compute a distance using both the scene graph embedding (from prior retrieved images) and the sentence embedding to determine the image at that time step. Finally, the authors take the set of retrieved images to generate captions and then measure the consistency of the generated captions with the story.  The authors present preliminary experimental results showing that the proposed method has very slight gains on the VIST (Visual Storytelling) dataset. They also present a qualitative result showing a retrieved "visual story."