ARTEMIS: Attention-based Retrieval with Text-Explicit Matching and Implicit Similarity
Authors: Ginger Delmas
Rafael S. Rezende
Gabriela Csurka
Diane Larlus
Venue: ICLR 2022 Poster (2022)

Abstract: An intuitive way to search for images is to use queries composed of an example image and a complementary text. While the first provides rich and implicit context for the search, the latter explicitly calls for new traits, or specifies how some elements of the example image should be changed to retrieve the desired target image. Current approaches typically combine the features of each of the two elements of the query into a single representation, which can then be compared to the ones of the potential target images. Our work aims at shedding new light on the task by looking at it through the prism of two familiar and related frameworks: text-to-image and image-to-image retrieval. Taking inspiration from them, we exploit the specific relation of each query element with the targeted image and derive light-weight attention mechanisms which enable to mediate between the two complementary modalities. We validate our approach on several retrieval benchmarks, querying with images and their associated free-form text modifiers. Our method obtains state-of-the-art results without resorting to side information, multi-level features, heavy pre-training nor large architectures as in previous works.

Published as a conference paper at ICLR 2022
ARTEMIS: ATTENTION-BASED RETRIEVAL WITH TEXT-EXPLICIT MATCHING & IMPLICIT SIMILARITY
Ginger Delmas
Rafael S. Rezende
Gabriela Csurka
Diane Larlus
NAVER LABS Europe
ABSTRACT
An intuitive way to search for images is to use queries composed of an example image and complementary text. While the first provides rich and implicit con- text for the search, the latter explicitly calls for new traits, specifies how some elements of the example image should be changed to retrieve the desired target image. Current approaches typically combine the features of each of the two el- ements of the query into single representation, which can then be compared the ones of the potential target images. Our work aims at shedding new light on the task by looking at it through the prism two familiar and related frameworks: text-to-image and image-to-image retrieval. Taking inspiration from them, we ex- ploit the specific relation of each query element with the targeted image and derive light- weight attention mechanisms which enable to mediate between the two plementary modalities. We validate our approach on several retrieval benchmarks, querying with images and their associated free-form text modifiers. Our method obtains state-of-the-arr results without resorting to side information, multi-level features, heavy pre-training nor large architectures as in previous works. Our code is available at https: ///gtthubb com/ naver/ waavee/rrttee aartemis.
INTRODUCTION
When using an image search engine, should the user provide detailed textual description visual example? These two image retrieval tasks, respectively studied as cross-modal search (Wang et al., 2016) and visual search (Philbin al., 2007) the computer vision community, have been the topic of extensive studies.
However, limiting search queries to a single modality is restrictive. On the one hand, text proposes an accurate but only partial depiction of the desired result, as it is impossible to provide exhaustive textual description. On the other hand, visual queries are richer but a lot more ambiguous: how should the system guess which characteristics the user wants to maintain for the target image? The task of image search with free-form text modifiers reduces this ambiguity by allowing dual query composed of example (called reference) image, and a textual description (called text modifier) which explains how the reference image should be modified to obtain the desired result (the target image). This task is illustrated in Figure 1, From this formulation, challenging research question arises: How should the two facets of this dual query be leveraged and combined?
The first methods answer this question directly are borrowed from the Visual Question Answering literature (Kim et al., 2016; Santoro et al., 2017). The most standard approach for this task consists in fusing the features of the two components of the query into a single representation, so it can be compared to the representation of any potential target image (Vo et al., 2019; Chen & Bazzani, 2020). Among the current strategies, some resort to rich external information (Chen et al., 2020; Liu et al., 2021) while others rely on multi-level visual representations (Chen et al., 2020) heavy cross-attention architectures (Hosseinzadeh Wang, 2020; Chawla et al., 2021; Liu et al., 202 l).
Departing from these strategies, we draw inspiration from two fields related to the task, cross-modal and visual search, and we advocate for a combination of the two components of the query taking into account their specific relationship with the target image. Additionally, we note that the text modifier provides some insights about what should change (as an explicit request from the user), and what should be preserved in the target image with respect the reference image (i.e. what is

---

Published as a conference paper at ICLR 2022
search
ARTEMIS
Ir cream in culor
Cross-modal search
Figure I: Image search with free-form text modifiers (shown in yellow) enriches the visual query with text in natural language. The task differs from visual search (using only the reference image, shown in red), and cross modal search (using only the textual cue, shown in green). Yet, these can be seen as complementary aspects of the studied task. Our method leverages both modalities to look for target images that i) explicitly match (EM) the characteristics mentioned in the text, and ii) bear some resemblance with the reference image using text-guided implicit similarity (IS).
not mentioned in the text modifier). All these observations lead to the design of two independent modules, one for each modality, that are eventually combined.
Our Explicit Matching (EM) module measures the compatibility of potential target images with the properties mentioned in the textual part of the query, as a form of cross-modal retrieval. Simultane- ously, our Implicir Similarity (IS) module considers the relevance of the target images with respect to the properties the reference image implied by the textual modifier.
Additionally, each of our modules ifier which helps selecting the characteristics of the target image to focus on for retrieval. This results in proposed model, named A Attention-based Retrieval with Text-Explicin Matching and Implicit Similarity (A ARTEMIS), which jointly learns these two modules, their respective attention mechanisms, and the way they should be combined, in a unified manner.
light- attention mechanism, guided by the text mod-
In summary, our contribution is twofold. First, we suggest new way to look at the task of image search with free-form text modifiers, where draw inspiration from and combine - the two fields of cross-noe moddll and visual retrieval. Second, we propose a computationally efficient model based on two complementary modules and their respective text-guided light-weight attention layers, where each handles one modality of the query. These modules are trained jointly.
We experimentally validate ARTEMIS on several datasets and show that, despite its simplicity in terms of architecture and training strategy, it consistently outperforms the state of the art.
2 RELATED WORK
Image retrieval with purely visual queries has been extensively studied and typically referred to as instance-level image retrieval (Philbin et al., 2007). Most approaches either learn global image representations for efficient search (Perronnin al., 2010; Radenović et al., 2016; Gordo al., 2016) or extract local descriptors for a costly yet robust matching (Arandjelović & Zisserman, 2012; Tolias al., 2016; Noh et al., 2017). Current sfate-of oottee-f-thheaee the-art models use both global and local descriptors sequentially, using the latter to refine the initial global ranking (Cao et al., 2020; Tolias et al., 2020).
The most standard retrieval scenario involving text, cross-modal retrieval, uses textual description of the image of interest query. A common approach is to learn joint embedding space and modality-specific encoders producing global representations
that shared space. Then, the task boils down to computing distances between those representations (Frome et al., 2013; Wang et al., 2016; Faghri et al., 2018). The most recent works focus on cross-attention between text segments and image region features (Lee al., 2018; Li et al., 2019; 2020). However, these models do not scale, and they require to score each potential text-query target-image pair independently. Miech al. (2021) address this issue by distilling a cross-attention architecture into dual encoder one.
2

---

Published as a conference paper at ICLR 2022
Despite similitudes between these two tasks and the one of image search with text modifiers, the first baselines for the latter initially adapted visual question answering (VQA) approaches (Kim et al., 2016). To reason with features from different modalities, Santoro et al. (2017) use sequence of MLPs, whereas FiLM (Perez et al., 2018) injects text features into the image encoder at multiple layers, altering its behavior through complex modifications.
Current approaches address this task by composing the two query elements into a single joint repre- sentation that is compared to any potential target image feature. As such, in TIRG (Vo al., 2019), the reference image and the "relative captions" (or text modifiers) are fused through a gating-residual mechanism, and the text feature acts as a bridge between the two images in the visual representation space. Anwaar et al. (2021) use an images into the same complex space and learn the text modifier representation as a transformation in this space. Lee et al. (2021) and Chawla et al. (2021) both propose to disentangle the multi-modal information into content and style. Chen & Bazzani (2020) resort to image's s descriptive texts as side information to train joint visual-semantic space, training a TIRG model on top. The alignment of visual and textual features for regularisation is reused in VAL (Chen et al., 2020) which also inserts composite transformer at many levels of the visual encoder preserve and transform the visual content depending on the text modifier; the model is then optimized with a hierarchical matching objective. Hosseinzadeh & Wang (2020) also align image and text through cross-modal module, but they use region proposals instead of CNN activations. Liu et al. (2021) propose a transformer- based model that leverages rich pre-trained iision-and-lanuuee features of the query conditioned by the text. We note that these last methods are reminiscent of the heavy cross-attention models and share the same limitations with respect to scaling.
seeencoder-basee model to map the reference and the target
knowledge for modifying the visual
In contrast to most methods described above, ARTEMIS does not compose modalities into a joint global feature for the query (Vo et al., 2019; Chen & Bazzani, 2020; Lee et al., 2021), does not compute costly cross-attention involving the target image (Hosseinzadeh & Wang, 2020; Chawla et al., 2021) and does not extract multi-level visual representations (Chen et al., 2020). Instead leverages the textual modifier in simple attention mechanisms to weight the dimensions of the visual representation, emphasizing the characteristics on which the matching should focus. This results in model with manageable amount of parameters to learn, efficient inference on the query side, and that leads to sate-offthee-s results.
The term attention has been heavily used in the past, referring to light- weighting mechanisms (Xu et al., 2015; Li et al., 2017; Kang et al., 2018; Wang et al., 2019) similarly to us, or to more complex weighting mechanisms (Lu et al., 2016). This differs from the more recent definition of attention introduced by Vaswani et al. (2017), which became standard in many approaches tackling the cross- modal retrieval (Lee al., 2018; Li et al., 2019; 2020) and VQA (Bai et al., 2018) tasks.
PROPOSED METHOD
This section deseribes ARTEMIS, our proposed approach for the task of image search with free- form text modifiers. In this setting, queries are bimodal pairs, composed of reference image I, and a text modifier T Such queries are used to retrieve any relevant image I, from a gallery of images.
Let o(.) and ((.) be the visual and textual encoders respectively. We denote the resulting L2- normalized features of the reference image r, of the text modifier m, and of the target image t, i.e. = (T,)) E RH m ((Tm) E RH, and t ((T,) E RR.
Two complementary views of the task. Both the textual and the visual cues are important to find relevant target images. Consider for example the query in Figure 1. The reference image provides strong visual context and displays some semantic details e.g.. the dress' length, among other style components- that should be shared with the target image, to the extent that the text does not specify their alteration. In other words, by the absence of an explicit modification request of some propertics, the modifying text implicitly "validates" the visual cues from the reference image associated to those properties, and these cues can be used to directly query for the target image. Hence, part of the task boils down to image-to-img -m-ge retrieval (visual search), implicitly conditioned on the text modifier.
On the other hand, the textual cue explicitly calls for modifications of the reference image. was even observed that, in some cases, users tend to directly describe characteristics of the target image,
3

---

Published as a conference paper at ICLR 2022
without explicitly stating the discrepancy between the reference and the target image' (c.f. the anal- ysis presented for FashionlQ in (Wu et al., 2021) and for CIRR in (Liu et al., 202 Therefore the text, which describes mandatory details for the target, can be directly used to query for the target im- age, as is common in text-to-image retrieval. Yet, due to the specificities of the task, we go beyond the usual text-to-image retrieval.
Proposed approach. We propose to revisit text-to-image and saangeto-image retrieval to tackle the task at stake, which is at the intersection of both. Standard strategies, for both, boil down to directly comparing the global feature of the query (image or text) to those of the potential targets, producing two independent, moodaiitysspecife
and not necessarily immediately compatible scoring strategies. In particular, directly comparing the representations of the reference and the target images is inaccu- rate the two images should differ, based on the modifying text. Likewise, directly comparing the representations of the modifying text and the target image is insufficient and potentially detrimental since part of the target image should stay similar to the reference image.
Here, we reconcile the two tasks and enhance them by designing a model which is trained to match i) the characteristics of the reference image the ones of the target image and ii) the requirements provided by the text with properties of the target image. Based on the intuitions above, in both cases we use the text design an attention mechanism that selects the visual cues which should be emphasized during matching.
The first objective is carried out by the Implicit Similarity module (Section 3.1), which draws from visual search, and the second one by the Explicit Matching module (Section 3.2), inherited from cross-modal retrieval. These two modules, jointly trained, respectively output the IS and the EM scores, which are simply summed into one, as a form of late fusion (Section 3.3). The full architec- ture of the model is illustrated in Figure 2.
3.1 IMPLICIT SIMILARITY
In the first module, the textual cue m guides the comparison between the target and the reference images; it is used to define an attention mechanism Ass(.) R"r - 00, 1]"", taking input, which predicts the importance of the visual representation dimensions when comparing the target to the reference image. The IS score is then computed as:
sis(r, m, t) = (A)s(m)) @ r Ars (m) o t)
(1)
where @ is the pointwise product and (.I-) represents the cosine similarity between the two L.2- normalized terms. Ars(.) is two-layer MLP separated by a ReLU, and followed by softmax.
3.2 EXPLICIT MATCHING
This second module captures how well the target image matches the textual cue. As the target image t and the textual modifier m belong to different modalities, we first project m into the visual space with a learned linear transformation T(.) RH RH In addition, since the textual modifier should not act on all the characteristics of the image ( the target image should remain partially similar the reference image), we rely on a second attention mechanism, AgM(.) Hr + [0, 1 to emphasize the dimensions of the image feature to focus on for matching.
Hence, the EM score is computed as:
SEM (m,, t) (T(m) (m)) @ t).
where T.)) consists in a fully connected layer. Note that (.s and (gM(() share the same ar- chitecture but not their weights, as the role they give to is different. Indeed, intuitively AgM(() (.) focuses on what is mentioned by the textual modifier while Ars(.) looks for what is not. Put dif- ferently, AEM (-) "up-weights" the dimensions in the image feature that correspond to the semantic information provided by the textual cue, whereas (.s "down-weights" these dimensions to rather focus on the shared characteristics between the reference and target images.
Not having the reference image explicitly referred to in the text does not mean is useless (c.f. first view).
4

---

Published as a conference paper at ICLR 2022
Attention
eeference
image
fr
CNN Backbon:
Viead monder
Implicit Similarity
cargen Ballage
Poletwee product
Shared weighe
Ais
Final score
Cante denlarty
modiler
dog, aud T' grats, make one dog black"
A Eng
Explicit
Nomalced vecaral aepessnaation
GLOVE
LSTM BXGRU
Matching (EM)
Tr
Figure 2: Illustration of the proposed architecture. Our model takes triplet (reference image I,, target image 7t, text modifier T..) as input and feed each element to its respective encoder. The text feature then used multiple ways (including two attention mechanisms) to compute the implicit similarity (IS) and the explicit matching (EM) scores. These two complementary retrieval scores are summed to produce the final score which is used to rank potential target images.
3.3 TRAINING PIPELINE
Given a minibatch of B training samples = {(Ir. Tm., [x)) where It corresponds to an image that is annotated correct target for the query (4r, Tm.J, we train our model using the batch-based classification (BBC) loss:
1
exply s(rs: m., t()] i=1 E, exply s(r,, m., tni)
LABC (B)
log
(3)
B
- tx) is the "compatibility" score of the triplet (i in [1, Bia For ARTEMIS, this compatibility score is the sum of the EM and IS scores, respectively obtained by each of the modules presented above:
where y is learnable temperature parameter and s(r,, mj
s(r,, mj tR) SEM (m, tx) mr,, m,,tu).
(4)
The BBC loss, proposed as InfoNCE in the self-supervised learning literature (Oord et al., 2018), was introduced for image retrieval with text modifiers by Vo et al. (2019) and its efficacy for the task was confirmed by Lee et al. (2021). contrast to the triplet and the soft triplet losses which rely on negative mining strategies to obtain optimal results, the BBC loss considers all the negative samples mini-batch, and therefore simultancously learns from easy and hard negatives.
Encoders. Our approach generic and can accommodate for diverse visual encoders á and textual encoders 0. We present results with several combinations, for the sake of comparison with prior work. We typically use GloVe word embeddings (Pennington et al., 2014) followed by either a LSTM (Hochreiter & Schmidhuber, 1997) BiGRU (Cho et al., 2014) as text encoder and an ImageNet pretrained ResNet18 or ResNet50 model (He et al., 2016) as image encoder. Further implementation details can found Appendix A.
EXPERIMENTS
4.1 EXPERIMENTAL SETTING
Datasets. We consider two datasets focusing on the fashion domain and one open-domain images, all three using human-written textual modifiers in natural language. The Fashion IQ dataset (Wu et al., 2021) is composed of 46.6k training images and around 15.5k images for both the validation and test sets. There are 18k training queries, and 12k queries per evaluation split, covering three fashion categories: women's tops (toptee), women's 's dresses (dress) and men's 's shirts
5

---

Published as a conference paper at ICLR 2022
Table 1: Ablative study of the different components of our method. Overall 1d12ndd in black/blue.
FIQ val split
Shoes
CIRR val split
Flavors of ARTEMIS
R910
RG50
m. rank
R@10
RG50
m. rank
R355
R@50
R, 11
Image only (r [t)
4.86
12.01
917.11
28.47
53.13
41.67
30.10
75.75
20.84
Text only (m |t)
15.55
36.11
121.44
13.50
31.46
154.00
21.93
65.71
38.28
Late fusion (r + m. (t)
25.69
50.14
51.78
48.95
75.62
11.33
30.94
78.04
21.65
IS module only
6.18
16.42
449.56
32.63
57.41
31.33
32.28
77.50
21.16
EM module only
15.61
36.43
113.89
13.72
32.39
146.67
29.71
72.24
43.48
Full ARTEMIS
26.05
50.29
52.89
53.11
79.31
8.67
48.95
89.19
41.42
(shirt). The text modifier is composed of two relative captions produced by two different human annotators, exposed the same reference-targel target image pair. The Shoes dataset (Guo et al., 2018) is extracted from the Attribute Discovery Dataset (Berg et al., 2010). It consists of 10k training im- ages structured in 9k training triplets, and 4.7k test images including 1.7k test queries. The recently released CIRR dataset (Liu et al., 2021) is composed of 36k pairs of open-domain images, arranged in a 80%-10%- modifying text should only be relevant to one image pair, and irrelevant to any other image pairs containing the same reference image.
split between the train/validation/eess. The annotation process is such that the
Evaluation. We use the standard evaluation protocol for each dataset. We report Recall@ K (ROK), which is the percentage of queries for which least one of the correct ground truth item is ranked among the top K retrieved items. All reported numbers are obtained as an average of 3 runs. In Section 4.3, we additionally report standard deviations.
For FashionIQ, the standard evaluation metric is the one used for the companion challenge (Gao & Guo, 2020). It computes the R@10 and R@50 for each of the 3 fashion categories and averages these six results. For Shoes, we report the R@1, R@10 and R@50 as well as their average. For CIRR, following Liu et al. (2021), we also report Recall @K, which the set of candidate target images is restricted to images semantically similar to the correct target image. According to Liu et al. (2021), this metric is less sensitive to false-negatives.
Cross-validation Since the test split annotations of FashionIQ became available only very recently, previous works only report results the validation set, and resort ad hoc strategies to set their prper-parameters and early-stop criteria. In order to compare our method to previously published approaches, we provide results on both the test and validation sets by performing bidirectional cross-valls validaaiio between them: we run our experiments for fixed number of epochs and, at the end of each epoch, we evaluate our model on both sets. We select the best performing model on the validation set to report results on the test set and vice-versa. When no validation split is available, as for Shoes, we report the results on the last checkpoint. For CIRR, we select the best checkpoint on the validation split to evaluate on their test server.
4.2 ABLATION STUDY
We first conduct ablative study the validation set of all datasets to evaluate the influence of several design choices in our architecture. For FashionIQ and Shoes, we additionally report median rank of the correct target. For CIRR, report the R@5 and R, @1 in addition of R@50, since they best capture the model capabilities according to Liu et al. (2021). Results are presented in Table 1. All experiments in this section use ResNet50 and BiGRU as visual and textual encoders, trained as described in the Appendix $ A. We separate our ablation studies into two parts; first we look at several baselines which do not use any attention mechanism, then we challenge the different module choices by training them independently.
Baselines. Image-only and text-only baselines use only one of the queries elements to retrieve the target. Their compatibility scores can be written as Samg (r, m, t) = (r {) and Stze (r. m. t) = (m t), respectively. We also report the late fusion of the query embeddings. which can be seen as an attention-free version of ARTEMIS. In this case, the compatibility score becomes sty m, t) (r m t).

---

Published as a conference paper at ICLR 2022
Table 2: Fashion IQ, official validation set. We report the challenge metric (CM) and individual R@K scores. 9 means our re-implementation.
denotes the use of additional side information (e.g. extra captions from other datasets) at train time. Unless mentioned otherwise, each method uses Resnet50 and LSTM as visual and textual backbones, respectively. Overall 1 121/2nd in black/blue
Method
CM
RMX1
Ro050
Dress
Shirt
Toptoe
Mean
Dress
Shirt
Topleo
Mean
IVSM' (Chen Razzoni, 2020)
19:27
10.70
12.00
13.00
11.90
25,90
27.10
26.90
26.63
ComposeAB (Anwwar al. 2021)
29.40
TCIR (Chawla la et al.,2022
29.51
19.33
14.47
19.73
17.84
43.52
35.47
44.56
41.18
CIRPL ANT (1.in et 2021)
25.17
14.38
13.64
16.44
14.82
34.66
33.56
38, 34
35.52
CIRPLANT (Lin et 2022)
30:20
17.45
17.53
21.64
18.87
40.41
38.81
45.38
41.53
TIRG' (RN50 + -.TMM)
36.16
24.19:
20.13:aas
26.05
23,46-131
50.42
43.56
s2.57
48.85-10m
TIRG' (RN50 BiGRU)
23.80
19,90=amm
15.82
23,17
18.61.1
42,14
51 64
47. 18
(Ches
2020)
21.12
25.64
49.49
VAL (Chen et al., 2020)
35.38
22.53
22.38
27.53
24.15
14.00
44.15
51.68
16.61
CoSMo (Loo et oll, 2021)
31,26
21.39
16.90
21.32
19.87
44.45
37.49
46.02
42.65
ARTEMIS (RN18 LSTM) (ours)
34. 10100
20.35.44 25.23
23.36,0m
22.98
48 64
43.67as4
46,97aam
46, 43
ARTEMIS (RN50 LSTM) (ours)
36.51 eou4s
27.34
21.05
24.9
51.71
44.18
49.87
48.59
ARTEMIS (RNIS BiGRU) (ours)
34.75 sou
24.84
20.40
23.63 vore
22.95
49.00
43.22
46.54 47.39
ARTEMIS (RN50 BiGRU) (ours)
38.17
27.16-
21.78:133
29.00:qu
26,05:133
52.4
43.6/
54,83
50.29
Table 3: Fashion IQ, test set. We report the challenge metric (CM) and individual R@K scores. 1 means our re-implementation. denotes the use of additional side information (e.g, attributes) at training time. Overall 1 1d in black/blue.
Method
CM
ASCTD
Dress
Toptce
Shirt
Topteo
Mean
Transf, Dialog (Wa al., 2021)
21.57
12.45
11.05
11:24
11.58
35.21
28 22.99
30.45
31.55
Transf, Dialogg (Wa et al., 20223
22.05
13.39
11.03
11.74
12.05
35.56
29.03
31.52
32.04
TIRG' (RNS0 LSTM)
15.52
25.66.1.14
20.35-4a5
24.64
23.55
49.81
43.36
49.31s810
TIRG' (RN50 + BiGRU)
34.56
25.01
19.18=1s
24/13awe
22.97
49.17sau
42.48
48.61
16.75suu
ARTEMIS (RN18 + LSTM) (ours)
35.05
23.57
20
25, 262016
22.97
48 15gan
42.67a5-1
50.57g3
33.13
ARTEMIS (RN50 LSTM) (ours)
36.96
26.60
20.69.114
27.11
24.80
51.45
43.13
52.76
49.1
ARTEMIS (RNI8 BiGRU) (ours)
23.92
20.19...1
25.30
23.30
48.7 (8.71
42.01
$1.54:a5
47.42 47.42.015
ARTEMIS (RN50 BIGRU) (ours)
37.43
28.13-0025
21,43-422
25.91:00
25,16-111
51.66
45.22
50.41
19.10
The performance of these baselines changes according to the nature of each dataset. For FashionIQ, where 68% of the modifiers make direct reference to the target image (Wu et al., 2021), the text- only baseline outperforms the image-only one. For Shoes, based on visual inspection, we conjecture that the reference-targe target image pairs are, on average, more similar than for FashionIQ, making the reference image more crucial for many queries. For CIRR, while querying only with the reference image offers a strong baseline, the text modifier is more discriminative when ranking on the restricted set of very similar images. This is measured by the R, Q1 metric. Note that the late fusion baseline produces strong results for both FashionIQ and Shoes, but barely improves upon the image-only baseline for CIRR, which highlights the importance of the attention mechanisms.
Independent modules. We evaluate the IS and EM independently as separate retrieval models. Their corresponding compatibility scores are given by Equations and 2, respectively. We observe that IS and EM perform differently for different datasets, and expected, follow the previous ob- servations made for the image-only and text-only baselines. We see that both our modules slightly outperform their corresponding baseline. The full ARTEMIS performs best for all three datasets. This highlights the synergy between IS and E.M, supporting the motivations extensively discussed in Sections 3.1 and 3.2, but also the fact that the method is robust to different types of datasets/ queries: those where the visual part is more important as well as those where the textual part is preponderant.
4.3 COMPARISON WITH THE STATE OF THE ART
We now compare ARTEMIS to state-of-the-an art approaches on each benchmark. We also report results for our implementation of TIRG (Vo et al., 2019), a well-established composition network baseline, with the same training pipeline as ARTEMIS, as the original paper does not provide results on these more recent datasets.
7

---

Published as a conference paper at ICLR 2022
Table 4: Shoes dataset. means our re-implementation. denotes the use of additional information (e.g. extra-captions. attributes) training time. All reported methods use ResNet50 as backbone. Overall 1d in black/blue.
Method
R@1
R@00
R@50
( R /3
TIRG (RN50-LSTM)
15,52
+..
76.49+142
46.89
TIRG (RN50-BiGRU)
14,46
47.51
75.174032
45.71 +.050
VAL (Chen et al., 2020)
16.49
49.12
73.53
46.38
VAL" (Chen et al., 2020)
17.18
51.52
75.83
48.18
CoSMo (Lee al., 2021)
16.72
48.36
75.64
46.91
ARTEMIS (RN50-LSTM) (ours)
17.60
51.05+0..1
76,85+0a1
48.50
ARTEMIS (RN50-BiGRU) (ours)
18.72+023
53.11+0.77
79.31+019
50.38
Table 5: CIRR dataset, test set. Recall@ K and Recallsubeet @K (according to Liu et al. (2021),
Recall @1 best assess fine-grained reasoning ability). Gray background is directly compa- rable to our results, since it relies on a model pre-trained on a very large set of image-caption pairs.
Overall 1st "222d in black/blue.
Method
Recalla
Recall
K
10
K 50
K
K
K
CIRPLANT" (init, OSCAR)
19.55
52.55
68.39
92.38
39.20
63.03
79.49
45.88
CIRPLANT
15.18
43.36
60.48
87.64
33.81
$6,99
75.40
38.59
TIRG! (BIGRU)
10.01+014
38.31
54.59
84.69ewx
37.36-006
59.31
72.51+010
37,84+227
ARTEMIS (BIGRU) (ours)
16.96
46.10=0ms
61.31+1
87.73-002
39.99
62.20
75.67+075
43.05=031
Results on Fashion IQ are reported Table 2 and Table 3 for the validation and the test sets, respectively. Since the test split annotations were unavailable until very recently (only Wu et al. (2021) published results on that split), most published approaches report results on the validation set. In both cases best ARTEMIS (RN50- BiGRU) outperforms the state of the art.
As most published works (Chen et al., 2020; Lee et al., 2021) use LSTM as text backbone and ResNet50 image backbone, also report results for ARTEMIS in this setting and show that we remain competitive, outperforming VAL by an average of 1.1% % over the challenge metric. Furthermore, we also report results with ResNet18 compare several recent works (Anwaar et al., 2021; Chen & Bazzani, 2020), who prioritize a faster and smaller backbone. We observe that, for smaller image backbone, LSTM seems to perform better then BiGRU. This is also true for our re-implementation of TIRG.
Results on Shoes presented in Table As the Shoes dataset does not contain a dedicated validation set, we reuse the exact same training procedure and hyperparameters as for FashionIQ. ARTEMIS obtains state-of-the-an results with both LSTM and BiGRU encoders, confirming our approach's 's ability to generalize.
CIRR results are reported in Table As recommended by Liu al. (2021), we use pre-extracted 52 features for the images. This affects our training pipeline as the image backbone can- not be finetuned and we cannot apply any image transformation. CIRPLANT obtains competitive results only when its multi-layer cross- transformer is initialized with the weights from the Os- car model (Li et al., 2020) trained on 6.5 million image-caption pairs. When compared to methods trained on the same data, ARTEMIS outperforms CIRPLANT on all metrics, in spite of a simpler ar- chitecture. In particular, our Rt @ result suggests, according to Liu al. (2021), that ARTEMIS has great fine-grained reasoning ability.
4.4 QUALITATIVE RESULTS
In Figure we present a few heatmap examples obtained with the Grad-CAM (Selvaraju et al., 2017) algorithm. Specifically, the image activations at the last convolutional layer of the CNN are reweighed with the pooled gradients of the same layer, after back mack-propagation of the coefficients contributing the most the EM and IS scores. It appears clearly that the IS module attends to the
8

---

Published as a conference paper at ICLR 2022
Query
Target
Query
Target
Query
Target
(end] is lighter color and has a looser fft
are gray with the sides
have laces, Velcro closure
EM
IS sarrre coeffciint
saae coefficient
cocificient
AS
IS
Figure 3: Visualisation of image parts contributing the most to the EM and IS scores, for queries from the Fashion IQ and Shoes datasets where the correct target is ranked first. An EM component is presented on the target image. The bottom two rows show two relevant components for IS, applicd to both the source and the target images (see Appendix D for details).
Hide part
the bocy the dag
More and huidfed on the white doth.
Figure 4: Qualitative examples of image-text queries of CIRR and its Top-6 retrieved results. A green frame denotes the ground-truth target (see Appendix D for more examples).
visual cues that are shared between the reference and the target images, such as the shirt's collar and bottom, the sole shape the heel, or the color and the shoe opening. In a complementary way, the EM module addresses the image parts that are the most related to the caption: the color, the shirt's fit, the slits on the side or the shoe's closure.
In Figure 4, we present some query examples from the CIRR dataset along with the Top-6 retrieved images. In both examples, we see that the reference image contributes a strong semantic context, and that the
text modifier provides important information to distinguish the correct target image (framed in green) from other very similar images. Additional qualitative examples and further analysis available in the Appendix $ D.
CONCLUSION
We proposed ARTEMIS, new method for image search with free-form text modifiers. It combines two modules, each focusing on one of the modalities of the query. The Explicit Matching module assesses how potential targets fit the textual modifier while the Implicit Similarity module compares potential target images to the reference image, assisted by the text. Both modules contain a light- weight attention mechanism which uses the text to guide the retrieval process in way that is specific to each modality of the query. Extensive experiments on several benchmarks validate our approach and show that consistently reach state-of-the-art performance.

---

Published as a conference paper at ICLR 2022
REFERENCES
Muhammad Umer Anwaar, Egor Labintcev, and Martin Kleinsteuber. Compositional learning of image-text query for image retrieval. In Proc. WACV, pp- 1140-1149, January 2021.
Relja Arandjelović and Andrew Zisserman. Three things everyone should know to improve object retrieval. In Proc. CVPR, pp. 2911 2911-2918. IEEE, 2012.
Yalong Bai, Jianlong Fu, Tiejun Zhao, and Tao Mei. Deep attention neural tensor network for visual question answering. In Proc. ECCV, pp. 20-35, 2018.
Tamara L Berg, Alexander C Berg, and Jonathan Shih. Automatic attribute discovery and character- ization from noisy web data. In Proc. ECCV, pp. 663-676, 2010.
Bingyi Cao, Andre Araujo, and Jack Sim. Unifying deep local and global features for image search. In Proc. ECCV, pp. 726-743. Springer, 2020.
Pranit Chawla, Surgan Jandial, Pinkesh Badjatiya, Ayush Chopra, Mausoom Sarkar, and Balaji Krishnamurthy. Leveraging style and content features for conditioned image retrieval. In Proc. CVPR Worshops, pp. 3978-3988 June 2021.
Yanbei Chen and Loris Bazzani. Learning joint visual semantic matching embeddings for language- guided retrieval. In Proc. ECCV, 2020.
Yanbei Chen, Shaogang Gong, and Loris Bazzani. Image search with text feedback by visiolinguistic attention learning. Proc. CVPR, pp. 3001-3011, 2020.
Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Hol- ger Schwenk, and Yoshua Bengio. Learning phrase representations using mnn encoder-decoder for statistical machine translation. In Proc. EMNLP, pp. 1724-1734, 2014.
Sanghyuk Chun, Scong Joon Oh, Rafael S Rezende, Yannis Kalantidis, and Diane Larlus. Proba- bilistic embeddings for cross-modal retrieval. Proc. CVPR, 2021.
Mostafa Dehghani, Anurag Arnab, Lucas Beyer, Ashish Vaswani, and Yi Tay. The efficiency mis- nomer. arXiv preprint orXiv:2 10. 12894, 2021.
Fartash Faghri, David J Fleet, Jamie Ryan Kiros, and Sanja Fidler. VSE++ Improving visual- semantic embeddings with hard negatives. In Proc. BMVC, 2018.
Andrea Frome, Greg Corrado, Jonathon Shlens, Samy Bengio, Jeffrey Dean, Mare' Aurelio Ranzato, and Tomas Mikolov. Devise: A deep visual-semantic embedding model. In Proc. NeurIPS, 2013.
Yupeng Gao and Xiaoxiao Guo. Fashion iq challenge. In Proc. CVPR Worshops, 2020. URL http://eee....eeeeee////eaaaaaaa
atttps:/ssites google com/ www/wwww/cccccccctcco.....cc...t www.rrratiie22222/aaah.....o
Albert Gordo, Jon Almazán, Jerome Revaud, and Diane Larlus. Deep image retrieval: Learning global representations for image search. In Proc. ECCV, 2016.
Xiaoxiao Guo, Hui Wu, Yu Cheng, Steven Rennie, Gerald Tesauro, and Rogerio Feris. Dialog-based interactive image retrieval. Proc. NeurlPS, pp. 676-686, 2018
Xintong Han, Zuxuan Wu, Phoenix Huang, Xiao Zhang, Menglong Zhu, Yuan Li, Yang Zhao, and Larry Davis. Automatic spatially-aware fashion concept discovery. Proc. ICCV, pp. 1463-1471, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. Proc. CVPR, pp. 770-778, 2016.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Mehrdad Hosseinzadeh and Yang Wang. Composed query image retrieval using locally bounded features. In Proc. CVPR, pp. 3596-3605, 2020.
10

---

Published as a conference paper at ICLR 2022
Jeff Johnson.
Rethinking floating point for deep learning. arXiv preprint arXiv:1811.0171111
Guoliang Kang. Liang Zheng, Yan Yan, and Yi Yang. Deep Adversarial Attention Alignment for Unsupervised Domain Adaptation: The Benefit of Target Expectation Maximization. In Proc. ECCV, 2018.
Jin-Hwa Kim, Sang-Woo Lee, Donghyun Kwak, Min-Oh Heo, Jeonghee Kim, Jung-Wor Ha, and Byoung- Tak Zhang. Multimodal residual learning for visual QA. In Proc. NeurIPS, pp. 361-369, 2016.
Kuang-Huei Lee, Xi Chen, Gang Hua, Houdong Hu, and Xiaodong He. Stacked cross attention for image-text matching. In Proc. ECCV, pp. 201-216, 2018.
Seungmin Lee, Dongwan Kim, and Bohyung Han. Cosmo: Content-style modulation for image retrieval with text feedback. In Proc. CVPR, pp. 802-812, June 2021.
Kunpeng Li, Yulun Zhang, Kai Li, Yuanyuan Li, and Yun Fu, Visual semantic reasoning for image- text matching. In Proc. ICCV, pp. 4654 4662, 2019.
Shuang Li, Tong Xiao, Hongsheng Li, Bolei Zhou, Dayu Yue, and Xiaogang Wang. Person Search with Natural Language Description. In Proc. CVPR, 2017.
Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang. Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. Oscar: seeantesemantics aligned pre-training for vision-language tasks. In Proc. ECCV, pp. 121-137. Springer, 2020.
Zheyuan Liu, Cristian Rodriguez-Opazo, Damien Teney, and Stephen Gould. Image retrieval on real-life images with pre-trained rision-and-languaae models. Proc. ICCV, 2021.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. Proc. ICLR, 2017.
Jiasen Lu, Jianwei Yang, Dhruv Batra, and Devi Parikh. Hierarchical question-image co-attention for visual question answering. Advances neural information processing systems, 29:289-297, 2016.
Antoine Miech, Jean-Baptiste Alayrac, Ivan Laptev, Josef Sivic, and Andrew Zisserman. Thinking fast and slow: Efficient text-to-visual retrieval with transformers. In Proc. CVPR. pp. 9826-9836, 2021.
Kevin Musgrave, Serge Belongic, and Ser-Nam Lim. A metric learning reality check. In Proc. ECCV, pp. 681-699, 2020.
Hyeonwoo Noh, Andre Araujo, Jack Sim, Tobias Weyand, and Bohyung Han. Large-scale image retrieval with attentive deep local features. In Proc. ICCV, pp. 3456-3465, 2017.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint arXiv:1807.03748. 2018
Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In Proc. EMNLP, pp. 1532-1543, 2014.
Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville, Film: Visual reasoning with a general conditioning layer. In Proc. AAAI, 2018.
Florent Perronnin, Yan Liu, Jorge Sánchez, and Hervé Poirier. Large-scale image retrieval with compressed Fisher vectors. In Proc. CVPR, 2010.
James Philbin, Ondfej Chum, Michael Isard, Josef Sivie, and Andrew Zisserman. Object retrieval with large vocabularies and fast spatial matching. In Proc. CVPR, 2007.
Filip Radenović, Giorgos Tolias, and Ondřej Chum. CNN image retrieval learns from BoW: Unsu- pervised fine-tuning with hard examples. In Proc. ECCV, 2016.
Filip Radenović, Giorgos Tolias, and Ondiej Chum. Fine-tuning CNN image retrieval with no human annotation. PAMI, 1655-16656 2018
11

---

Published as a conference paper at ICLR 2022
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. IJC,, 115(3):211-255 2015.
Adam Santoro, David Raposo, David G Barrett Mateusz Malinowski, Razvan Pascanu, Peter Battaglia, and Timothy Lillicrap. A simple neural network module for relational reasoning. In Proc. NeurlPS, pp. 4967-4976, 2017.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based local- ization. Proc. ICCV, pp. 618-626, 2017.
Yale Song and Mohammad Soleymani. Polysemous visual-semantic embedding for cross-modal retrieval. In Proc. CVPR, pp. 1979-1988, 2019.
Giorgos Tolias, Yannis Avrithis, and Hervé Jégou. Image search with selective match kernels: ag- gregation across single and multiple images. IJC,, 116(3):247-261, 2016.
Giorgos Tolias, Tomas Jenicek, and Ondřej Chum. Learning and aggregating deep local descriptors for instance-level recognition. In Proc. ECCV, pp. 460-477. Springer, 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proc. NeurIPS, pp. 5998- 6008, 2017.
Nam Vo, Lu Jiang, Chen Sun, Kevin Murphy, Li-Jia Li, Li Fei-Fei, and James Hays. Composing text and image for image retrieval-an empirical odyssey. In Proc. CVPR, pp. 6439-6444 6448, 2019.
Liwei Wang, Yin Li, and Svetlana Lazebnik. Learning deep structure-preserving image-text embed- dings. In Proc. CVPR, pp. 5005-5013, 2016.
Ximei Wang, Liang Li, Weirui Ye, Mingsheng Long, and Jianmin Wang. Transferable Attention for Domain Adaptation. Proc. AAAI, 2019.
Hui Wu. Yupeng Gao, Xiaoxiao Guo, Ziad Al-Halah, Steven Rennie, Kristen Grauman, and Rogerio Feris. Fashion IQ: new dataset towards retrieving images by natural language feedback. In Proc. CVPR, pp. 1307-11317, 2021.
Kelvin Xu, Jimmy Ba, Jamie R. Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. Show, Attend and Tell: Neural Image Caption Generation with Visual Attention. In Proc. ICML, 2015.
12

---

Published as a conference paper at ICLR 2022
APPENDIX
This document provides additional results that complement the main paper. Its content includes some implementation details ( A), our experiments on an additional dataset ($ B), an efficiency study of our method in comparison to other approaches ($ C), and some extra qualitative results for ARTEMIS ($ D).
A IMPLEMENTATION DETAILS
Texts are pre-processed to replace special characters by spaces and to remove all other characters than letters. We observe that using spelling correction and lemmatization does not improve the results. The text modifier is first tokenized into words, and encoded using 300-dimensional Vl embeddings (Pennington et al., 2014). Either a BiGRU (Cho et al., 2014), or a LSTM (Hochreiter & Schmidhuber, 1997) (followed by average pooling and fully-connected layer, in Chen et al. (2020)) are learned from scratch on top of the word embeddings to produce sentence-level feature. For Fashion IQ (Wu al., 2021), the only dataset for which each reference/target image pair is provided with two relative captions (used as text modifiers) from two different annotators, we follow previous works (Lee et al., 2021) and use the concatenation of the two captions (in both orders) for training and evaluation. This means that we use about 12k queries fr the validation split.
To maintain the aspect ratio and avoid losing information about the top and the bottom of the clothing items, images are first padded with white pixels to obtain a square. They are then resized to 256 x 256 pixels and augmented using random horizontal flip and a random crop of size of 224 x 224 pixels during training. For inference, images are resized to 256 x 256 and center cropped to 224 x 224. The image features are obtained using a CNN with a GeM pooling (Radenović et al., 2018), which is widely used in the instance-level image retrieval community instead of the usual average pool- ing. The pooling layer is followed by a newly learned fully- cully-connected layer. Our experiments employ either ResNet18 or ResNet50 architecture (He et al., 2016). initialized sakovsky et al., 2015). The only exceptions are the experiments on CIRR, for which not all images available to download. Liu et al. (2021) provide pre-extracted ResNet152 features (outputs of the average pooling layer) that we use as a replacement of the image backbone. As consequence, for all experiments on CIRR, the image backbone is not fine-tuned and we do not apply any image transformation.
ImageNet (Rus-
Following Song& & Soleymani (2019), we freeze the base encoders during the first 8 epochs to pre- train the sentence encoder, as well as the EM and IS modules. Then, we train our model end-to-end for 50 epochs. Our training pipeline uses Adam optimizer (Loshchilov & Hutter, 2017), a batch size of 32 and an initial learning rate of 5 10-4 with decay of 0.5 every 10 epochs. The dimension of both the image and the textual embeddings is set to Hy H, = 512.
B EXPERIMENTS ON FASHION20OK
In the literature (Vo et al., 2019; Chen et al., 2020; Lee et al., 2021), the Fashion200K dataset (Han al., 2017) is used as dataset for image search with text modifiers. The dataset is composed of 201k in-shop fashion article images, split into 172k train images and 29k test images. From these test images, Vo et al. (2019) provides list of 31k image pairing which is used to produce candidate queries for evaluation. The text modifiers, in contrast to the other presented datasets, are not in natural language: they are automatically generated from the set of image attributes following the "replace att. X by all. y" template. In our work, we focus on free-form human annotations, which the common point between the three datasets presented the main paper. However, we here compare ARTEMIS to the other published works on this dataset, for reference.
In order to better handle the large train set of Fashion200K, we follow VAL (Chen et al., 2020) and CoSMo (Lee et al., 202 1), and we train our model with larger mini-batch of 128 for 100 epochs end-to-end. While Chen et al. (2020); Chen & Bazzani (2020) use MobileNet-vl as image backbone, follow Vo et al. (2019); Lee et al. (2021) and use ResNet18. As Lee et al. (2021) pointed out, though the difference in model architecture may play a role in overall results, both models have almost identical scores on both ImageNet Topl and Tops error rates.
13

---

Published as a conference paper at ICLR 2022
Table 6: Fashion200K dataset results. i means results of our re-implementation. denotes results published by Vo et al. (2019). denotes the use of additional information (e.g. extra captions) at training time. For our results, we report the average of 3 runs. Overall 1 13"2""1""" in black/blue/red.
Method
R
R@10
R@50
E R@K)
FiLM (Perez al., 2018)
12.9
39.5
61.9
38.1
MRNI (Kim et al., 2016)
13.4
40.0
61.9
38,4
Relationship' (Santoro et al., 2017)
13.0
40.5
62.4
38.6
TIRG (Vo et al., 2019)
14.1
42.5
63.8
40.1
TIRG (RNI8+LSTM)
20.8
50.7
69.9
47.1
TIRG (RNI8+BIGRU)
20.2.10.4
49,4
69,8
46.5106
LBF (Hosseinzadch Wang, 2020)
17.8
48.4
68.5
44,9
VAL (Chen et al., 2020)
21.2
49.0
68.8
46.3
VAL* (Chen et al., 2020)
22.9
50.8
72.7
48.8
CoSMo (Lee al., 2021)
23.3
50.4
69.3
47.7
JVSM" (Chen & Bazzani, 2020)
19.0
52.1
70.0
47.0
ARTEMIS (ours) (RN18+L.STM)
21.5=00
51.1
70.5
47.7
ARTEMIS (ours) (RN18+BiGRU)
20.2
49.3
69.3
46.2+00
Interestingly, while ARTEMIS performed better with a BiGRU as text encoder for all other datasets, best results for Fashion200K obtained with a LSTM. We observe the same for our imple- mentation TIRG, and more generally for all our experiments that use a ResNer18.
We observe Table 6 that VAL* obtains the best performance, thanks to the extra captions used to pre-train their auxiliary regularizer. The same side information was also leveraged by Chen & Baz- zani (2020). Excluding models trained with this extra data, both ARTEMIS and CoSMo outperform VAL, and our approach obtain the best results in 2 out of the 4 evaluation metrics.
C COMPLEXITY AND EFFICIENCY STUDY
In the main paper, we extensively discuss our method's performance terms of accuracy. on several benchmarks. This section provides a complementary study that compares ARTEMIS' complexity with the complexity of several approaches. Since no single measure is enough to assess the com- plexity of model (Dehghani al., 2021), we compare the different models on three metrics:
Number of parameters. We measure the number of trainable parameters of the full mod- els. Many works use this metric as a proxy to model complexity and memory footprint.
Number of multiply-add operations. The number (MAC) is theoretical measure that allows to compare the number of operations needed by the model to perform a forward pass, independently report the number of GMACs for the forward pass of triplet (I,., (I,.TTm: Tm, It), where reference and target images are both size 224 224, and the modifier text is composed of 20 tokens (in average, CIRR sentences contain 11.3 words and FashionIQ 5.3, according Table in Liu et al. (2021)).
multiply-accumular operations
the hardware (Johnson, 2018). We
Latency time. Latency usually refers to the required time to perform an inference forward pass, either for batch of images the full dataset. It is an informative speed metric for real-time systems that require user input, so it is well suited to the retrieval task. We measure the time each method needs to process all queries, all target images, and all query- target scores of the FashionIQ validation split. Note that 12k queries are involved, since we do the inference for the concatenation of the two captions both orders: 'capl and cap2', 'cap2 and capl', following Lee et al. (2021) (see details in Appendix $A). All latency times are measured on the same GPU NVIDIA T4.
We report results for ARTEMIS, TIRG, VAL and CoSMo. For fair comparison, all reported numbers are obtained for ResNet50 and LSTM as image and text encoder respectively. Since these encoders important source of complexity in the models, also evaluate the encoders alone,
14

---

Published as a conference paper at ICLR 2022
Table 7: Efficiency comparison. All models use Resnet50 as image encoder and LSTM as text encoder. In red, we give the added values compared to our reference point which only considers the encoders. Latency is computed over the 12k queries of the FashionIQ validation set.
Method
MACs (G)
Parameters (M)
Latency FIQ (s)
Reference (only RN50 LSTM encoders)
8.35
31.65
114.09
ARTEMIS
8.35 (+0.001)
32.96
(+1.31)
122.87 (+8.78)
TIRG (Vo et al., 2019)
8.35 (+0.002)
33.75 (+2.10)
118.20 (+4.11)
VAL (Chen et al., 2020)
8.39 (+0.040)
59.48
(+27.83)
CoSMo (Lee et al., 2021)
10.45 (+2.110)
79.32 (+47.73)
133.84 (+19.75)
to provide a reference point. For this reference, we estimate the corresponding latency by evaluating the late-fusion baseline described in Section 4.2. We report our results in Table 7. We make the following observations:
ARTEMIS and TIRG both add very little complexity compared to the reference. Similarly to TIRG (Vo et al., 2019), our approach adds only few operations and parameters to the base encoders used to compute r, m and t. Both methods' parameters mostly come from their fully- connected layers. TIRG uses more parameters than ARTEMIS because its gating mechanisms takes the concatenated vector [r, m] as input instead of just the text embeddings m, doubling the number of required connections with respect ARTEMIS.
ARTEMIS and TIRG add some reasonable extra-latency compared to the reference. Both ARTEMIS and TIRG add little latency time to the encoder reference point. TIRG is more efficient due to its simple pass on the target representation: pre-computed can be directly compared to the composite query representation by simple cosine similarity. ARTEMIS also leverages pre- computed f, but re-weights its dimensions using Ais(m) and AgM (m) then 1-2-nrmmaizes the result before comparing the query features, with cosine similarity well. This double use of the target explains ARTEMIS slightly longer latency time.
VAL has a higher computational complexity. We were unable to produce the same experiments on the publicly available code of VAL to compute the latency. For the complexity measurements, we can attest to an increase in the number of GMACs and parameters that is more significant than for both ARTEMIS and TIRG. The latency is likely to increase similarly, as VAL inserted composite transformers at different levels of the visual encoder, and compares representations coming from the concatenation of average- pooleed features at low-, mid- and high-level (i.e. features of size 2048 + 1024 + 512).
CoSMo adds both extra-complexity and extra-latency. CoSMo, similarly TIRG, uses a com- position module to fuse r and m. and does single pass on each target. However, its compo- sition module, combining a disentangled non-local block for content modulation and text-guided affine transformations for style modulation, is more complex than ARTEMIS. Furthermore, while ARTEMIS and TIRG introduce little complexity, CoSMo adds around 25% multiply-accumulatie operations, and more than doubles the number of trainable parameters.
D QUALITATIVE RESULTS
Additional ranking results. We provide qualitative results for the FashionIQ (Wu al., 2021), the Shoes (Guo et al., 2018; Berg et al., 2010) and the CIRR (Liu et al., 2021) datasets in Figures 5, 6, 7, and 9. They illustrate the model's ability to properly encode and reason on general concepts. In the case of FashionIQ, it includes the length of sleeves and dresses, the patterns (floral, stripes, dots, plaid) or the logo color (different from the background color) and the looseness clothing items. As for Shoes it captures shininess, wool/suede material, and the presence of laces, of fur, of Velcro closures, or of a buckle, for instance. The CIRR dataset is open-domain, and we notice that the model is able to reason The model is also able to deal with more complex information, such as dolphins (Figure 6, second row) different renderings of the same logo (Figure 6, third row). Eventually, observe that the
wide variety of semantic concepts (from trees to the people's youth).
15

---

Published as a conference paper at ICLR 2022
model gathers pieces of information from the reference image when needed (style and cut, length or size. collar form, sleeve length, pattern or logo, sole shape, animal breed, sculpture etc.). While the model relies on the text modifier to propose coherent images, it also resorts to the reference image to fill in the gaps.
Extensive studies of the qualitative results (Figure 10) reveal that the model has trouble understand the negative formulations: in the third row of Figure 10, the proposed toptees remain colorful; in the second row, the ranked clothing items are still loose and sheer; the last row shows a mix of shoes with laces and other with straps. For FashionIQ. it likely due to the fact that only 3.5% of the queries use negation, according to semantic analysis carried out in the companion paper (Wu et al., 2021). Supposedly, there are not enough examples at training time for the model to learn the associated semantic.
Limitations.
Besides, the captions were written by different human annotators. They sometimes present a sub- jective judgment (e.g. "is cuter", "more manly") lack essential details to discriminate better the target among other images in the gallery. We give a few illustrative examples of such cases in Figure 10. For the top query, the specific required color is not mentioned, hence the various guesses taken by the model.
One limitation inherent the task is well illustrated by the blue dress example in Figure 5, the boot example in Figure 8 or the bird example in Figure 9: very often, many candidate target images would be acceptable answers ROK K is a more reliable metric when K is bigger as it's more lenient to plausible wrong predictions. Other works (Musgrave et al., 2020; Chun et al., 2021) have pointed out the limitations of relying on RåK for retrieval tasks when K is too small, and suggested other metrics. This is also limitation that Liu al. (2021) tries to address with their R. @K metric, by annotating images while aware of possible false negatives.
given query, beyond the one annotated as ground-truth. Hence believe
Additional heatmaps on examples extracted from the FashionIQ dataset (Wu et al., 2021) are pre- sented in Figure 11. Similarly, Figure 12 presents results on the Shoes (Guo et al., 2018; Berg et al., 2010) dataset.
The most important parts of the image with regard to the computation of each score, Explicit Match- ing (EM) and Implicit Similarity (IS), are highlighted red. The coefficients of EM and interme- diary results (i.e. before completing the scalar product with the sum) are referred as "components". The heatmaps are obtained by backpropagating to the last convolutional layer of the CNN some of the components that contribute the most to the EM and IS scores. We show the influence of same IS component both on the reference and the target images, since both images are involved in the computation of the IS score. Conversely, EM is based only on the target image, with regard to the text modifier.
It emerges from the IS visualizations that the model is able to draw the parallel between the two images on several concepts. For clothing items, it can be the shape of the collar, the sleeve length or the cut of the dress. For shoe items, it includes the presence of heels laces, the shoe opening or the sole shape. In complementary way, the EM module addresses the image parts that are the most related the caption: for example, focuses on the pattern, the color or decoration requirements.
Eventually, these visualizations help to understand what semantic parts of the image are taken into account by the model to evaluate the relevance of candidate target image for a given query.
16

---

Published as a conference paper at ICLR 2022
longer sleeves and has stripes [and] has higher neckline with sleeves
short blue neck sleeveless dress [and] has sleeves and solid brighter color blue
orange and less revealing [and] is orange ruffled.
is brown with long sieeves and neck [and] has more red
Figure 5: Qualitative results for queries from the "dress" category of the FashionIQ dataset (Wu et al., 2021). We show the six top ranked images for each query. The expected targets are indicated with green border. It appears that the model is able reason several concepts (color, pattern, ruffles, sleeve length, form of the neck...), and to use the reference image to infer unspecified prop- erties in the text modifier (style, length, tightness...)
17

---

Published as a conference paper at ICLR 2022
navy blue with yellow lettering [and] Is navy in color
N7PD
E
CTM
Has dolphins on a light blue tee shirt [and] is lighter
has more realistic picture [and] has 3d printed logo
plaid long sleeve turquoise [and] has plaid patter and longer
sleeves
Figure 6: Qualitative results for queries from the "shirt" category the FashionIQ dataset (Wu et al., 2021). We show the six top ranked images for each query. The expected targets are indicated with a green border. appears that the model is able to reason on several concepts (differrnnt color for the clothing background and the logo, graphic, style, sleeve length...), and to use the reference image to infer unspecified properties the text modifier (kind, style, form of the collar..).
18

---

Published as a conference paper at ICLR 2022
The shirt is white with flwwers [and] is lighter
has longer sleeve with polka dols [and] is orange and has buttons and longer sleeves
Is brighter in color and has horizontal stripes [and] is red striped instead of blue.
The shirt short sleeved and light green color, [and] soft green colored
Figure 7: Qualitative results for queries from the "toptee" category of the FashionIQ dataset (Wu et al., 2021). We show the six top ranked images for each query. The expected targets are indicated with green border. It appears that the model is able reason on several concepts (color, pattern, presence of buttons, sleeve length, form of the neck..),, and to use the reference image to infer unspecified properties in the text modifier (kind, style, length, form of the collar..).
19

---

Published as a conference paper at ICLR 2022
is blue
is more shiny
sneakers with colorful trims
have fur
are leather with buckle
green, with bow
are suede, with buckles
are brown suede with laces
have Velcro closures
gray wool with floral design
Figure 8: Qualitative results for queries from the Shoes dataset (Guo al., 2018; Berg al., 2010). It appears that the model is able to reason several concepts (kind, color, style, shininess, fur, buckle, laces, Velcro closures...), and to use the reference image to infer unspecified properties in the text modifier (kind, style, presence of heels or laces, shapes of the sole and of the opening, etc.).
20

---

Published as a conference paper at ICLR 2022
two bottles are identical, and in the middle one pepsi bottle
Dog sits rock the water.
Younger girls and different pajamas
Dog runs through the snow.
Focus on two birds with orange and green bodies perched on branch.
Take far away shot showing trees behind scene
Another angle and gorilla's mouth open
More distant pandas and one with his face tumed to the left.
Figure 9: Qualitative results for queries from the CIRR dataset (Liu al., 2021). We show the six top ranked images for each query. The expected targets are indicated with a green border. It appears that the model is able reason on several concepts (plurality, background/foreground, natural elements. viewpoint...), and to use the reference image to infer unspecified properties in the text modifier (e.g. the animal breed).
21

---

Published as a conference paper at ICLR 2022
lighter [and] is more colorful
print and is less sheer [and] is not loose and more floral.
not colorful [and] is more revealing
have laces, Velcro straps
Qualitative results showing some limitations, for queries from FashionIQ (Wu et al., 2021) (the first rows) and Shoes (Guo et al., 2018; Berg et al., 2010) (last row). For any of these examples, the ground truth not included by the model the top ranking of potential targets. This is either due to a lack of information (first row) or the non-understanding of negation (last 3 rows).
Figure 10:
22

---

Published as a conference paper at ICLR 2022
Query
Target
Query
Target
Query
Target
EM
EM
EM
is long [and] is more langer sleeves
less fitted and green fand] lighter in
[and] is lighler color and
shorter sleeves
loaser fit
IS
coefiicient
same coeflidint
same coefficient
Query
Target
Query
Target
Query
Target
EM
EM
EM
polke and beit
has net sheer sleeves [and] transparent
aimiler in color with fewer letters [and] has zane
(and] has
ZAND
polka dot print belted at the waist
IS
ooefiidiet
same coeffident
Same Speehiiae
ZAND
Figure 1I: Visualisation of image parts contributing the most to the EM and IS scores, for queries from the FashionIQ dataset (Wu et al., 2021) where the correct target is ranked first. For each of the six blocks, we provide the heatmaps of an EM component on the target image, and of two relevant IS components, applied both the reference and target images. The EM heatmaps highlight some of the connections made by the model between the text modifier and the target image. The IS heatmaps display some detected similarities between the two images. Both EM and IS help to evaluate the relevance of the target image with regard to the query.
23

---

Published as a conference paper at ICLR 2022
Query
Target
Query
Target
Query
Target
EM
EM
EM
closed loafers
gray with slits on sides
have Jaces, nor a Velcro closure
IS
coeffiiiert
same coeffidint
coeffiiient
Query
Target
Query
Target
Query
Target
EM
EM
EM
have small aliver accents
is gray suede with a folded
are black decoration
IS
IS
same coeffident
same Samnieee endeent
saree coeeffiziio
Figure 12: Visualisation of image parts contributing the most to the EM and IS scores, for queries from the Shoes dataset (Guo al., 2018; Berg et al., 2010) where the correct target is ranked first. For each of the six blocks, we provide the heatmaps of an EM component on the target image, and of two relevant
components, applied both the reference and the target images. The EM heatmaps highlight some of the connections made by the model between the text modifier and the target image. The IS heatmaps display some detected similarities between the two images. Both EM and help to evaluate the relevance of the target image with regard to the query.
24


Reviews:
[{"review_id": "wWKNQv0MEuA", "reviewer": "Reviewer_pgma", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- Strengths: \n\n  1. It is somewhat interesting to consider the image retrieval with text modifiers in a unified framework, and the strategy of directly combining image-image similarity and image-text similarity seems easy but effective.\n\n  2. This paper is well presented, with clear organization and detailed visual demonstration in experiments.\n\n- Weaknesses: \n\n  1. The attention mechanism has been widely utilized for image-text matching/VQA such as [1,2, 3], and the proposed element-wise product in IS and EM may not provide many insights. \n  [1] Person Search with Natural Language Description, CVPR 2017.\n  [2] Hierarchical Question-Image Co-Attention for Visual Question Answering, NIPS 2016.\n  [3] Deep Attention Neural Tensor Network for Visual Question Answering, ECCV 2018.\n\n  2. The explanations on the proposed IS and EM module are not clear enough. It would be better to provide more detailed explanations on the insights/function/effect of the specifically designed architecture.\n\n  3. Another concern might be the framework itself. Different from CoSMo which learns feature embeddings for fast retrieval, this paper requires to inference each query-target-text triplet online, which would be much slower. It would be better to give more comparisons on efficiency and performance with different frameworks.\n\n", "summary_of_the_paper": "This paper aims to tackle the problem of image search with free-form text modifiers and proposes to combine the implicit similarity and explicit matching score. The implicit similarity module utilizes the textual modifier as a filter to compute the similarity between the query and the target image, and the explicit matching module computes the similarity between the text and the filtered target image. This work combines the image-text similarity and text-guided image-image similarity in a unified framework, and experiments on the Fashion IQ, Shoes and CIRR dataset demonstrates the effectiveness of the proposed strategy. ", "summary_of_the_review": "Overall, this paper proposes the ARTEMIS for image-search with free-form text modifiers. Although simple and effective, the methodology may not be novel enough: it is more like a simple combination of matching scores and does not provide many insights.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "l5rF1JQDYt9", "reviewer": "Reviewer_L8FA", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n-------\n\n* Extremely well written and well motivated\n* A simple but novel idea/architecture\n* Comprehensive experiments showing the proposed approach achieves state-of-the-art performance on 3 datasets\n* The ablations clearly show the utility of each of the parts of the already simple architecture\n\nSuggestions and Questions\n-----\n\n* How were heat maps shown picked? Were the images and queries picked at random, and then the heat maps visualized? Or visualized first and then picked after?\n* I think the attention mechanism (such as $\\mathcal{A}_{IS}\\(.\\)$) used here isn't really an attention mechanism in the true sense. It seems a little closer to gating the different image and text embeddings based on the modifier text than attending over them (which would use the modifier text _and_ the image, for example, to generate the attention maps).\n\nMinor comments\n-----\n\n* Page 2: \"conciliates\" -> \"conciliate\" (the word itself sounds incorrect in this context though, did you mean combine?)", "summary_of_the_paper": "This paper focuses on the task of image retrieval using a sample image combined with text modifiers. It does this by the use of two modules that compute the implicit similarity (image-image similarity) and explicit matching (image-text similarity) scores. The modules computing both these scores use attention mechanisms so as to be conditioned on the text modifier. The scores are summed for the final retrieval. The efficiency of this architecture is shown quite clearly by comparing it with other state-of-the-art architectures on 3 different datasets. Ablations and qualitative analysis further highlights the utility of the architecture and its different components.\n\n", "summary_of_the_review": "Based on all the strengths (extremely well written and well motivated; clear; a simple but novel architecture; comprehensive experiments and good ablations; great results), I'd recommend this paper be accepted.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "cF76bhUr6Fn", "reviewer": "Reviewer_5uBZ", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n- The motivation and exposition of the problem and the solution is very clear.\n- The paper proposes simple yet effective and practically applicable method.\n- The experimental evaluation is solid, ablation experiments are through and limitations are discussed, albeit in supplementary material \n\n**Weaknesses**\n- Not sure how much the improvement is significant, if results are reported as average over three runs why there is no variance?\n- Explicit complexity analysis would help understand the advantage of the method against more computationally complex (e.g. cross-attention) methods.\n- “We select the best performing model on the validation set to report results on the test set and vice-versa” is not comparable to other reported methods since they didn’t have access to test set. The comparison with TIRG is valid, assuming that the protocol was the same for both ATERMIS and TIRG.", "summary_of_the_paper": "The paper presents a method for the task special kind of multi-modal retrieval: image search with free-form text modifiers, where image is used as a query and accompanied text specifies the differences with respect to the given query image which target image should satisfy. It is practically important problem for which several benchmark datasets exist. The paper proposes to use the representation of text modifier to attend (modulate) both to query and target image representations, and to use these modulated representations to obtain image and text matching scores, which are then averaged to obtain final score for each target image, as displayed in Figure 2. The proposed model architecture is conceptually simple and lightweight, allowing thus scalable retrieval system, while outperforming several recent baselines on three challenging benchmarks: Fashion IQ, Shoes and CIRR dataset. The paper also presents qualitative results of attention via Grad-CAM. \n", "summary_of_the_review": "The paper presents a simple method for important problem that outperforms recently proposed methods on three challenging benchmarks. The paper is clearly written, backed with experimental evaluation and ablations, where it shows good performance.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewers viewed the proposed approach to image retrieval as both simple and effective, the manuscript as well written and well motivated, and the presented experiments as relatively compressive (spanning three datasets). There was some discussion about novelty -- all reviewers viewed the approach as simple, but one had concerns about the approach's novelty relative to past work. This concern, as well as some concerns about experimental evaluation, were adequately addressed in author response.
Decision: Accept (Poster)