ARTEMIS: Attention-based Retrieval with Text-Explicit Matching and Implicit Similarity

1 INTRODUCTION . When using an image search engine , should the user provide a detailed textual description or a visual example ? These two image retrieval tasks , respectively studied as cross-modal search ( Wang et al. , 2016 ) and visual search ( Philbin et al. , 2007 ) in the computer vision community , have been the topic of extensive studies . However , limiting search queries to a single modality is restrictive . On the one hand , text proposes an accurate but only partial depiction of the desired result , as it is impossible to provide an exhaustive textual description . On the other hand , visual queries are richer but a lot more ambiguous : how should the system guess which characteristics the user wants to maintain for the target image ? The task of image search with free-form text modifiers reduces this ambiguity by allowing a dual query composed of an example image ( the reference image ) and a textual description ( the text modifier ) which explains how the reference image should be modified to obtain the desired result ( the target image ) . This task is illustrated in Figure 1 . From this formulation , a challenging research question arises ; How should the two facets of this dual query be leveraged and combined ? First methods to answer this question directly borrowed from the visual question answering literature ( Kim et al. , 2016 ; Santoro et al. , 2017 ) . Now , the standard approach for this task consists in fusing the features of the two components of the query into a single representation , so it can be compared to the representation of any potential target image ( Vo et al. , 2019 ; Chen & Bazzani , 2020 ) . Among the current strategies , some resort to rich external information ( Chen et al. , 2020 ; Liu et al. , 2021 ) while others rely on multi-level visual representations ( Chen et al. , 2020 ) or heavy cross-attention architectures ( Hosseinzadeh & Wang , 2020 ; Chawla et al. , 2021 ; Liu et al. , 2021 ) . Departing from these strategies , we draw inspiration from two fields related to the task , cross-modal and visual search , and we advocate for a combination of the two components of the query , in a way that also takes into account their specific relationship with the target image . To this end , we design two independent modules , one for each modality . Our Explicit matching ( EM ) module measures the compatibility of potential target images with the properties mentioned in the textual part of the query , as a form of cross-modal retrieval . Simultaneously , our Implicit similarity ( IS ) module considers the relevance of the target images with respect to the properties of the reference image implied by the textual modifier . Additionally , we note that the text modifier provides some insights about what should change ( as an explicit request from the user ) , but also about what should be preserved in the target image with respect to the reference image ( i.e . what is not mentioned in the text modifier ) . Consequently , each of our modules uses a light-weight attention mechanism , guided by the text modifier which helps selecting the characteristics of the target image to focus on for retrieval . This results in our proposed model , named Attention-based Retrieval with Text-Explicit Matching and Implicit Similarity ( ARTEMIS ) , which jointly learns these two modules , their respective attention mechanisms , and the way they should be combined , in a unified manner . In summary , our contribution is twofold . First , we suggest a new way to look at the task of image search with free-form text modifiers , where we draw inspiration from – and combine – the two fields of cross-modal and visual retrieval . Second , we propose a computationally efficient model based on two complementary modules and their respective text-guided light-weight attention layers , where each handles a modality of the query and both can be trained jointly . We experimentally validate ARTEMIS on several datasets and show that , despite its simplicity in terms of architecture and training strategy , it consistently outperforms the state of the art . 2 RELATED WORK . Image retrieval with purely visual queries has been extensively studied and is typically referred to as instance-level image retrieval ( Philbin et al. , 2007 ) . Most approaches either learn global image representations for efficient search ( Perronnin et al. , 2010 ; Radenović et al. , 2016 ; Gordo et al. , 2016 ) or extract local descriptors for a costly yet robust matching ( Arandjelović & Zisserman , 2012 ; Tolias et al. , 2016 ; Noh et al. , 2017 ) . Current state-of-the-art models use both global and local descriptors jointly and perform both strategies sequentially ( Cao et al. , 2020 ; Tolias et al. , 2020 ) . The most standard retrieval scenario involving text , cross-modal retrieval , uses a textual description of the image of interest as a query . A common approach is to learn a joint embedding space and modality-specific encoders producing global representations in that shared space . Then , the task boils down to computing distances between those representations ( Frome et al. , 2013 ; Wang et al. , 2016 ; Faghri et al. , 2018 ) . The most recent works focus on cross-attention between text segments and image region features ( Lee et al. , 2018 ; Li et al. , 2019 ; 2020 ) . However , these models do not scale , and they require to score each potential text-query target-image pair independently . Miech et al . ( 2021 ) address this issue by distilling a cross-attention architecture into a dual encoder one . Despite similarities between these two tasks and the one of image search with text modifiers , the first baselines for the latter initially adapted visual question answering ( VQA ) approaches ( Kim et al. , 2016 ) . To reason with features from different modalities , Santoro et al . ( 2017 ) use a sequence of MLPs , whereas FiLM ( Perez et al. , 2018 ) injects text features into the image encoder at multiple layers , altering its behavior through complex modifications . Current approaches address this task by composing the two query elements into a single joint representation that is then compared to any potential target image feature . As such , in TIRG ( Vo et al. , 2019 ) , the reference image and the “ relative captions ” ( or text modifiers ) are fused through a gatingresidual mechanism , and the text feature acts as a bridge between the two images in the visual representation space . Anwaar et al . ( 2021 ) use an autoencoder-based model to map the reference and the target images into the same complex space and learn the text modifier representation as a transformation in this space . Lee et al . ( 2021 ) and Chawla et al . ( 2021 ) both propose to disentangle the multi-modal information into content and style . Chen & Bazzani ( 2020 ) resort to image ’ s descriptive texts as side information to train a joint visual-semantic space , and then train a TIRG model on top . The alignment of visual and textual features for regularisation is reused in VAL ( Chen et al. , 2020 ) which also inserts a composite transformer at many levels of the visual encoder to preserve and transform the visual content depending on the text modifier ; the model is then optimized with a hierarchical matching objective . Hosseinzadeh & Wang ( 2020 ) also align image and text through a cross-modal module , but they use region proposals instead of CNN activations . Liu et al . ( 2021 ) propose a transformer-based model that leverages rich pre-trained vision-and-language knowledge for modifying the visual features of the query conditioned by the text . We note that these last methods are reminiscent of the heavy cross-attention models for cross-modal retrieval and share the same limitations with respect to scaling . In contrast to most methods described above , ARTEMIS does not compose modalities into a joint global feature for the query ( Vo et al. , 2019 ; Chen & Bazzani , 2020 ; Lee et al. , 2021 ) , it does not compute costly cross-attention involving the target image ( Hosseinzadeh & Wang , 2020 ; Chawla et al. , 2021 ) and does not extract multi-level visual representations ( Chen et al. , 2020 ) . Instead it leverages the textual modifier in simple attention mechanisms to weight the dimensions of the visual features , emphasizing the characteristics on which the matching should focus . This results in a model with a manageable amount of parameters to learn during training , efficient inference on the query side , and leading to state-of-the-art results . About attention . We note that the term attention has been heavily used in the past , referring to light-weighting mechanisms ( Xu et al. , 2015 ; Li et al. , 2017 ; Kang et al. , 2018 ; Wang et al. , 2019 ) similarly to us , or to more complex weighting mechanisms ( Lu et al. , 2016 ) . This differs from the more recent definition of attention introduced by Vaswani et al . ( 2017 ) , which became standard in many approaches tackling the cross-modal retrieval ( Lee et al. , 2018 ; Li et al. , 2019 ; 2020 ) and VQA ( Bai et al. , 2018 ) tasks . 3 PROPOSED METHOD . This section describes ARTEMIS , our proposed approach for the task of image search with freeform text modifiers . In this setting , queries are bimodal pairs , composed of a reference image Ir and a text modifier Tm . Such queries are used to retrieve any relevant image It from a gallery of images . Let φ ( . ) and θ ( . ) be the visual and textual encoders respectively . We denote the resulting L2normalized features of the reference image r , of the text modifier m , and of the target image t , i.e . r = φ ( Ir ) ∈ RHI , m = θ ( Tm ) ∈ RHT , and t = φ ( It ) ∈ RHI . Two complementary views of the task . Both the textual and the visual cues are important to find relevant target images . Consider for example the query in Figure 1 , where the dress ’ length , among other style components from the visual example , and the colour requested in the text modifier are combined to retrieve relevant results . The reference image provides strong visual context and displays some semantic details that should be shared with the target image , to the extent that the text does not specify their alteration . In other words , by the absence of an explicit modification request of some properties , the modifying text “ validates ” the associated visual cues from the reference image : those visual cues can be used as direct query for the target image . Hence , part of the task boils down to image-to-image retrieval ( visual search ) , implicitly conditioned on the text . On the other hand , the textual cue explicitly calls for modifications of the reference image . It was even observed that , in some cases , users tend to directly describe characteristics of the target im- age , without explicitly stating the discrepancy between the reference and the target image1 ( c.f . the analysis presented for FashionIQ in Table 2 of Wu et al . ( 2021 ) , for CIRR in Table 1 of Liu et al . ( 2021 ) ) . Therefore the text , which describes mandatory details for the target , can be directly used to query for the target image , as is common in text-to-image retrieval . Yet , due to the specificities of the task , we go beyond the usual text-to-image retrieval . Proposed approach . We propose to revisit text-to-image and image-to-image retrieval to tackle the task at stake , which is at the intersection of both . Standard strategies , for both , boil down to directly comparing the global features of the query ( image or text ) and of the potential targets , producing two independent , modality-specific , and not necessarily immediately compatible scoring strategies . In particular , directly comparing the representations of the reference and the target images is inaccurate as the two images should differ , based on the modifying text . Likewise , comparing directly the representations of the modifying text and the target image is undesired since part of the target image should stay similar to the reference image . Here , we reconcile the two tasks and enhance them by designing a model which is trained to match i ) the characteristics of the reference image with the ones of the target image and ii ) the requirements provided by the text with properties of the target image . Based on the intuition above , in both cases we use the text as an attention mechanism to select the visual cues which should be emphasized during matching . The first objective is carried out by the Implicit Similarity module ( Section 3.1 ) , which draws from visual search , and the second by the Explicit Matching module ( Section 3.2 ) , inherited from crossmodal retrieval . These two modules respectively output the IS and the EM scores , which are simply summed into one as a form of late fusion . The modules are jointly trained through this final score ( Section 3.3 ) . The full architecture of the model is illustrated in Figure 2 .

Summary: The paper presents a method for the task special kind of multi-modal retrieval: image search with free-form text modifiers, where image is used as a query and accompanied text specifies the differences with respect to the given query image which target image should satisfy. It is practically important problem for which several benchmark datasets exist. The paper proposes to use the representation of text modifier to attend (modulate) both to query and target image representations, and to use these modulated representations to obtain image and text matching scores, which are then averaged to obtain final score for each target image, as displayed in Figure 2. The proposed model architecture is conceptually simple and lightweight, allowing thus scalable retrieval system, while outperforming several recent baselines on three challenging benchmarks: Fashion IQ, Shoes and CIRR dataset. The paper also presents qualitative results of attention via Grad-CAM.