Candidate Set Re-ranking for Composed Image Retrieval with Dual Multi-modal Encoder
Authors: Zheyuan Liu
Weixuan Sun
Damien Teney
Stephen Gould
Venue: Accepted by TMLR ()

Abstract: Composed image retrieval aims to find an image that best matches a given multi-modal user query consisting of a reference image and text pair. Existing methods commonly pre-compute image embeddings over the entire corpus and compare these to a reference image embedding modified by the query text at test time. Such a pipeline is very efficient at test time since fast vector distances can be used to evaluate candidates, but modifying the reference image embedding guided only by a short textual description can be difficult, especially independent of potential candidates. An alternative approach is to allow interactions between the query and every possible candidate, i.e., reference-text-candidate triplets, and pick the best from the entire set. Though this approach is more discriminative, for large-scale datasets the computational cost is prohibitive since pre-computation of candidate embeddings is no longer possible. We propose to combine the merits of both schemes using a two-stage model. Our first stage adopts the conventional vector distancing metric and performs a fast pruning among candidates. Meanwhile, our second stage employs a dual-encoder architecture, which effectively attends to the input triplet of reference-text-candidate and re-ranks the candidates. Both stages utilize a vision-and-language pre-trained network, which has proven beneficial for various downstream tasks. Our method consistently outperforms state-of-the-art approaches on standard benchmarks for the task. Our implementation is available at https://github.com/Cuberick-Orion/Candidate-Reranking-CIR.

Published in Transactions on Machine Learning Research (01/2024)
Candidate Set Re-ranking for Composed Image Retrieval with Dual Multi-modal Encoder
Zheyuan Liu
sheyuen.luu@anu. Www.anu. edu.ou
Australian National University
Weixuan Sun
Australian National University
Damien Teney
domien.teney
Idiap Research Institute Australian Institute for Machine Learning (AIML)
Stephen Gould
seephen.gould@annee
Australian National University
Reviewed on OpenReview: https: openreview. net/ net/ forum? id- JAwemcupp.
Abstract
Composed image retrieval aims to find an image that best matches a given multi-modal user query consisting of a reference image and text pair. Existing methods commonly pre-compute image embeddings over the entire corpus and compare these to a reference image embedding modified by the query text at test time. Such a pipeline very efficient at test time since fast vector distances can be used to evaluate candidates, but modifying the reference image embedding guided only by short textual description be difficult, especially independent of potential candidates. An alternative approach is to allow interactions between the query and every possible candidate, i.e., teapeencceeee---adddaatttt triplets, and pick the best from the entire set. Though this approach is more discriminative, for large-scale datasets the computational cost is prohibitive since pre-computation of candidate emboddings no longer possible. We propose combine the merits of both schemes using a two-stage model. Our first stage adopts the conventional vector distancing metric and performs fast pruning among candidates. Mcanwhile, second stage employs dual-cncoder architecture, which effectively attends to the input triplet of cettemmce-ee--canniie candidates. Both stages utilize a vision-and-laguugge pre-trained network, which has proven beneficial for various downstream tasks. Our method consistently outperforms state-of-the- art approaches on standard benchmarks for the task. Our implementation is available at http::///ggthbbbcc/o//rrrrtatrte
and re-ranks the
1 Introduction
The task of composed image retrieval (CIR) aims at finding a candidate image from large corpus that best matches user query, which is comprised of reference image and modification sentence describing certain changes. Compared to conventional image retrieval setups such text-based (Li et al.) 2011
content-based (Tong & Chang 2001} retrieval, the incorporation of both the visual and textual modalities enables users to more expressively convey the desired concepts, which is useful for both specialized domains such as fashion recommendations (Wu et al.
2021 Han et. al. 2017 and the more general case of searching
over open-domain images (Liu et al. 2021
Couairon et al. 2022 Delmas et al. 2022
Existing work (Vo 2019 Dodds et al.
2020
Chen al. 2020a Baldrati al. 2022a on CIR. mostly
adopts the paradigm of separately embedding the input visual and textual modalities, ies, followed by model

---

Published in Transactions on Machine Learning Research (01/2024)
"Show the living room with fireplace."
"Change to white dog the ground."
Easy negatives with irrelevant contents
Visually similar hard negatives
Pre-fiitered to remove
Subsequently re-ranked along with the ground truth
Reference > Target
Figure 1: An illustration of the two-stage scheme, with easy negatives pre-filtered out, and the remaining candidates re-ranked.
that acts as an image feature modifier conditioned on the text. The modified image feature is finally compared against features of all candidate images through vector distances (i.e., cosine similarities) before yielding the most similar one as the prediction (sce Figure lefft.. The main benefit of such a. pipcline the inference cost. For a query consisting of reference image IR and modification text t, to select the best matching candidate, the model shall exhaustively assess triplets (Tn, I, Tc) for every image Ic 'D where D is the candidate image corpus. Such an exhaustive pairing of (JR,t) t) and le results in large number of query-candidate pairs. In the pipcline discussed above, all candidate images are individually pro-embedded, and the comparison with the joint-embedding of (TR, 1) is through the cosine similarity function that is elficient to compute even at a large scale. We point out that such a pipeline presents a trade-off between the inference cost and the ability to exercise explicit query-candidate reasoning. In essence, the candidate images are only presented to the model indirectly through computing the cosine similarity and the loss function, resulting in the model having to estimate the modified visual features from text inputs in its forward path.
To this end, we propose a solution that exhaustively classifies the (/R, (c) triplets, which enables richer query-candidate interactions - thus, achieving an appreciable performance gain while still maintaining a reasonable inference cost. We observe that for CIR, easy and hard negatives can be distinctly separated, as the nature this task dictates that the ground truth candidate be visually similar to the reference, otherwise it would be trivial to study a modification (Liu al., 2002 Couairon et al. 2022). We then further deduce that a group of hard negatives exist, which is likely benefit from fine-grained multi-modal reasoning, illustrated in Figure [ This motivates two-stage method, where we first filter all candidates to reduce their quantity. Since the goal at this stage is to remove easy negatives, low-cost vector distance (i.e., cosine similarity) based pipeline would suffice. We then re-rank the remaining candidates with explicit text-image matching on each possible triplet. Granted, such a process is more computationally intense but is empirically beneficial for reasoning among hard candidates. With the pre-fitteing in place, we are able to limit the overall inference time within an acceptable range. The main focus of this paper is on the second stage.
Note that our two-stage pipeline relates the inference scheme of image-text retrieval (Lin al. 2014h in recent vision-and-langune language pretrained (VLP) networks (Li et al. 2021 2022 Specifically, Li et al. (2021) propose to first compute feature similarities for all image-text pairs, then re-rank the top-k candidates through joint image-text encoder via the image-text matching (TTM) scores, which grcatly speeds up the inference compared Lo previous VLP networks that require computing ITM scores for all image-text pairs (Chen el al.) 2020b Li et al., 2020). Here, we arrive at a similar two-stage scheme but for the task of CIR. We also note that our mothod, although sharing similar philosophy and is based on VLP networks, not a direct replica of what is done in the image-text retrieval tasks discussed above. With the unique input triplets of (JR. t, Ic), novel model architectures are required for efficient interactions among the three features of two modalities.
2

---

Published in Transactions on Machine Learning Research (01/2024)
In summary, contribution is a two-stage method that combines the efficiency of the existing pipeline and the ability to assess fine-grained query-candidate interactions through explicit pairing. We base our design on VLP models while developing task-specific architectures that encourage interactions among input entities. Our approach significantly outperforms existing methods on datasets of multiple domains.
2
Related Work
The task of image retrieval traditionally accepts input in the form of either an image (Tong & Chang 2001
or text (Zhang et al. 2005 Li et al. (011). The aim is to retrieve an image whose content is the most similar to the input one, or respectively, best matches the textual description. Vo et al. (2019) propose composed image retrieval (CIR), which takes as input both modalities, using an image as a reference while text as a modifier.
Current approaches address this task by designing models that serve as a reference image modifier conditioned text, essentially composing the input modalities into one joint representation, which is compared with features of candidates through cosine similarity. Among them, TIRG (Vo et al. 2019) uses gating mechanism along with residual connection that aims at finding relevant changes and preserving necessary information within the reference respectively. The outputs of the two paths are summed together to produce the final representation. [Anwaar et al. (2021) follow a similar design but pre-encode the inputs separately and project them into a common embedding space for manipulation. Hosscinzadeh Wang (2020) propose to adopt regional features as in visual question answering (VQA) (Anderson et al.) 2018)) instead of convolutional features. Likewise, WWen et al. (2021) develop global and local composition networks to better fuse the modalities. VAL (Chen et al. 2020aa introduces a transformer network to jointly encode the input modalities, where the hicrarchical design encourages multi-layer matching. MAAF (Dodds et al. 2020) adopts the transformer network differently by pre-processing the input into sequences of tokens to be concatenated and jointly attended. Yang et al. [2021] designs a joint prediction module on top of VAL that highlights the correspondercces between reference and candidate images. Notably, the module only used in training as it is intractable to apply it to every possible pair of reference and candidate images during inference.
CIRPLANT (Liu et al. 2021 proposes to use a. pre-trained vision-and-languagge (VLP) transformer to
modify the visual content, alongside CLIP4CIR (Baldrati et al. 2022a b). BLIP4CIR (Liu et al. 2024 and
CASE (Levy et al. 2023).
DCNet (Kim et al., 2021 introduces the composition and correction networks, with the latter accepting a reference image with a candidate target image and asscssing their relevancy. This, on first look, suggests an exhaustive reference-candiddlt pairing. Though, inference cost. limits the interaction of a pair of reference and candidate images to simple operations i... element-wise product and subtraction with single-layer multi-layer perceptron (MLP). ARTEMIS (Delmas et al.) 2022) is the first to introduce model that scores each pair of query and candidate image, which separates it apart from an image modifierrbased pipeline. However, inference cost still confines such scoring to cosine similarities between individually pre-encoded modalities. In contrast to existing approaches, our method is in two stages. We do not seek to modify image features in an end-to-end manner. Instead, we pre-filter the candidates and focus more on re-ranking the remaining hard negatives. The re-ranking step is formatted as a scoring task based on contrastive learning, which is natural for VLP networks trained with similar objectives.
We note that the concept of two-stage scheme is not new for conventional image-text document retrieval. Indeed, re-ranking selected list of candidate images via e.g., k-nearest neighbors (Shen et al. 2012) query expansion techniques (Chum et al.] 2007) has been widely studied. More recent and related work on VLP models (Li el al. 2021 2022 2023) propose to first score the similarities between image and text features, then re-rank the top-k pairs via a multi- multi-modal classifier. This aligns nicely with the two pre-training objectives, namely, imago-text contrastive and image-text matching. To the best of our knowledge, we are the first apply such a two-stage scheme CIR. We contribute by designing an architecture that reasons over the triplet of (JR,4, Tc), which differs from the conventional retrieval tasks discussed above.
3

---

Published in Transactions on Machine Learning Research (01/2024)
t- -Pointy ear dog
Tokenize
gdaand
Passed Canditite Re-ranknng 4
Tokenize
PPointy ear dog:
V 344p
Embed
Encoder
Embed
Encoder- X
Encoder-
Embed
pre-gndoessed
esrectt [CLS]
peeprroeeeed
shared weights
extract [C115] tokin
pro processed
Concat
Project
Embed
Passed Candidate Re-aankng
In training contrastive ming
ranking by
cosine distances
top it fitered set of candidatos
MLP
Project
In inference scoring
cand dates ebb
seract [CLS, taker
Stage-1 Candidate Filtering
Stage-Il
Candidate Re-ranking
Figure 2: Overall training pipeline. In both stages, we freeze the image encoders (dashed fillings), as detailed in Section 4.1.3. (Left) Candidate filtering model, which takes as imput the tokenized text and cross-attends it with the reference image. The output is the sequential feature 2t, where we extract the [CLS] token as the summarized representation of the query q= (/n,4) 1) to compare its similarity with features of T/- (Right) Candidate re-ranking model with duall-nncoder architecture. Stacked elements signify that we exhaustively pair up each candidate 1 among the selected top-K with the query q for assessment. Note that the two encoders take in different inputs for cross cross-attention. The output [CLS] tokens are concatenated and passed for producing logit. Note that the two stages are two separate models and not jointly trained.
3 Two-stage Composed Image Retrieval
The task of CIR be defined follows. Let Tn be some reference image and t be piece of text describing change to the image. Then given query consisting of the pair q= t), the aim of CIR is to find the best match, i.e,, the target image /T, among all candidates in large image corpus D. In this work, we propose a two-stage model where we first filter the large corpus to obtain a smaller set of candidate images relevant to the query (soe Scction 3.1 and then re-rank to obtain an ordered list of target images (sce Section 3.2
For both steps, we base our designs on the vision-and-language pre-trained (VLP) network BLIP [Li et al. 2022), though other VLP models might be used. BLIP consists of an image encoder and a text encoder. The image encoder is vision transformer (Dosovitskiy et al., 2021 that accepts input raw image and produces the spatial image features by slicing the image in patches and flattening them in sequence. A global image feature is also represented as prepended special [CLS] token. The text encoder can operate in three modes. When configured as a uni-modal encoder, it takes in a sequence of tokenized words from a text sequence and outputs the sequential features with a [CLS] token summarizing the whole text, as in BERT (Devlin et al., 2019). Optionally, the text encoder can be configured as multi-modal encoder, where a cross-attention (CA) layer is inserted after each self-attention (SA) layer. As shown in Figure 3 the CA layer accepts the sequential output. of the image encoder and performs image-text altention. The output of which is passed into the feed-for forward (FF) layer and is the same length as the input text sequence. The transformer-basee text encoder accepts inputs of varied lengths while sharing the same token dimension d as the output of the image encoder. In this paper, we denote the features of an arbitrary image (resp. input text)) as o (resp. w) and its length as I.p (resp. I. We note that a decoder mode is also available in BLIP for generative tasks (e.g., image captioning (Anderson et al. 2018 ) though it is not used in this work.
3.1 Candidate Filtering
The first stage of our approach aims to filter out the majority of candidates leaving only a few of the more difficult candidates for further analysis in the second step. Shown in Figure 2 (left), we adopt the BLIP text encoder in its multi-modal mode such that it jointly embeds a given query = (AR. t) into a sequential output, which we denote as 2, E R xd Note that the output sequence 2, is of length Ing as the input text, a characteristic that further exploited in the second stage model. We extract the feature of the [CLS] token in 4 as single d-dimensional vector and compare it to pre-computed [CLS] embeddings of all candidate
4

---

Published in Transactions on Machine Learning Research (01/2024)
Encoder- 2
IT
Passed to the next trensformer SA layer
SA
B
CA
FF
Merge
2
SA
CA
FF
Pessed to the next trensformer SA layer
Encoder-
IT'
Figure 3: Details of the transformer layer in our dual-encoder architecture. Here, we take the first layer as an example. SA: Self-attention layer, CA: Cross- attes-attention layer, FF; Feed-forward layer. @: element-wise addition for residual connections. All modules in the figure are being trained. Dashed fillings on FF suggest, weight-sharing.
images 1 E D via cosine similarity. Here, the pre-computed candidate embeddings are independent of the query texl, which is weakness that we address in our second stage model. Since BLIP by default projects the [CLS] token features into d = 256 for both image and text, the comparison can be efficiently done through the cosine similarity function.
A After training the candidate filtering model, we select the top- K candidates (for each query) for re-ranking the second stage. Here we choose K to be sufficiently large so that the ground-truth target is empirically observed to be within the selected images for most queries. We term the percentage value of queries with ground truth within the top-K as ground truth coverage in the following sections. Empirically, we find that setting K to 50 or 100 gives a good trade-off between recall and inference cost in the second stage. Details on the ablation of K across datasets are discussed in Section 4.3.2
We note that concurrent to our work, CASE Levy et al. 2023 adopts a similar approach as our candidate filtering model, in that it uses BLIP for reference image-text fusion. We point out that both our filtering model and CASE use BLIP in one of its originally proposed configurations without architectural changes, and hence, unsurprising and could be viewed as natural progression for the task. Meanwhile, our second-stage re-ranking sets us apart from this concurrent work.
3.2 Candidate Re-ranking
The second stage re-ranks the filtered set of candidates. Since this set is much smaller than the entire corpus we can afford richer, more expensive approach. To this end, we introduce a novel dual-encoder design for this subtask inspired by the BLIP architecture proposed for NLVR (Suhr et al. 2019).
As shown in Figure 2 (right), our two encoders run in parallel as two branches to serve separate purposes, one to encode t with 14, (Encoder-t) and the other to encode JR with 14 (Encoder-2s). Internally, they exchange information via dedicated merging layers. Specifically, Encoder-t takes as input the tokenized t, which is then embedded into a sequential feature of size 4 xd Meanwhile, Encoder-z, accepts as input 2, E R xd from the previous stage, which a surrogate of JR (further discussed below). Since models of the two stages are trained separately, here we can pre-compute ž, for each query of q = (IR, () using the first-stage candidate filtering model. We note that for given query, the lengths of 2 and the embedded t are always identical, as the output of text encoder (i.e., the candidate filtering model) shall retain the dimension of the input coming through the SA layers (see Figure 3) This characteristic makes merging the outputs of the two encoders within each transformer layer straight.orward, which is discussed further below.
We use default 12-layer transformer for each encoder. Within cach transformer layer, both encoders cross-attend the inputs introduced above with the sequential feature of an arbitrary 14. The intuition is to allow 1r to separately attend to the two elements in each query q for relevancy, namely t and JR. For
5

---

Published in Transactions on Machine Learning Research (01/2024)
Encoder-t, the two entities entering the cross-attention (CA) layer are self-explanatory. For Encoder-24, however, opt for using 2 as a surrogate of In. The main reason is the GPU memory limit, it is intractable to perform image-image cross attention between lR and 14 with the default L. = 577 during training. Although spatial pooling can be used to reduce the length the input In sequence, we empirically find inferior, polentially due to the loss of information pooling, which discussed further in Section 4.3 As an alternative, 2t can be viewed as an embedding that contains sufficient In information and readily available, since has been pre-computed in the previous stage from the query pair q. A bonus of using 2 is that can easily merge the cross-attended features, it shares the same dimensionality as t at all times. Empirically, we confirm that our design choices yield better results.
Figure 3 depicts the internal of the transformer layer of the re-ranking model. As illustrated, we merge the outputs of the CA layers from the two encoders in each transformer layer. Specifically, given the outputs of the encoders after the CA layers, the merging performed as an average pooling the first six layers, and concatenation followed by simple MLP in the last six layers. The merged feature is then passed into a residual connection, followed by the FF layers. Regarding weight-sharing across layers in each encoder, we opt for having separate weights of SA and CA layers within each encoder, while sharing the weights of FF layers to account for the different inputs passing through the SA and CA layers. We point out that due to the residual connections (Figure 3 the outputs of the two encoders after the final transformer block are different in values, even though the FF layers are of the same weights.
We formulate the re-ranking as scoring task among the set of candidate images score the true target higher than all other negative images. For each sequential output from either encoder, we extract the [CLS] token the front as the summarized feature. We then concatenate the two [CLS] outputs from two encoders and use two-layer MLP as the scorer head, which resembles the BLIP image-text matching (TTM) pre-training task [i et al. 2022
setup.
3.3
Training Pipeline
3.3.1
Candidate Filtering
Our filtering model follows the contrastive learning pipeline (Radford et al. 2021 with a batch-based
classification loss (Vo et al., 2019) commonly adopted in previous work {Liu al. 2024 Levy et al., 2023
Specifically, in training, given a batch size of B, the features of the i-th query ( t¹) with its ground-truth
target IT formulate the loss
exp X (fo(
B
Lpittering
Σ
log
B
(1)
B
Σ expp[x . (To(T
1=1
where fo is the candidate filtering model parameterized by 0, À is learnable temperature parameter following Radford et al.) 2021), and x((;,) is the similarity kernel as cosine similarity.
In inference, the model ranks all candidate images 1t for each query via the same similarity kernel s(s.). We then pick the top-K list for each query for the second-stage re-ranking.
3.3.2 Candidate Re-ranking
The re-ranking model is trained with similar contrastive loss discussed above. Specifically, for cach (16 t², 11) triplet, we extract the predicted logit and contrast against all other [ 14) with i + j. essentially creating (B negatives for each positive triplet. The loss formulated as:
exp
(T d. 1+))
B
Litorankiig
log
B
(2)
B
Σ
fr t H.14)
i=1
exp
6

---

Published in Transactions on Machine Learning Research (01/2024)
where fy is the candidate re-ranking model parameterized by y.
Note that in training, we randomly sample negatives within the same batch to form triplets. Therefore, the choice of K does not affect the training process. We empirically find this yielding better performance than training only on the top-K negatives, with the benefit of not relying on a filtered candidate list for training. Incidentally, it also more efficient, as we do not need to independently load negatives for each query. During inference, the model only considers, for each query, the selected top-K candidates and ranks them by the predicted logits.
4 Experiments
4.1
Experimental Setup
4.1.1 Datasets
Following previous work, we consider two datasets different domains. Fashion-IQ (Wu et al. 2021 is a dataset of fashion products in three categories, namely Dress, Shirt, and Toptee, which form over 30k triplets with 77k images. The annotations are collected from human annotators and overall concise. CIRR (Luu et al. 2021 is proposed to specifically study the fine- gra-graindd visiolinguistic cues and implicit human agreements. It contains 36k pairs of queries with human geneaateee annotations, where images often contain rich object interactions (Suhr et al. 2019
4.1.2 Evaluation Metrics
We follow previous work report our results in Recall@K, that is the percentage of queries whose true target is ranked to be among the top-K' candidates. For Fashion-IQ, we assess the performance with Recall@10 and 50 on each category. Such choices of K values account for the possible false negatives in the candidates. On CIRR, we report 1ecall@1, 5, 10, and 50. We additionally record Recall @K, where the candidates are limited to pre-defined set of five with high similarities. The set of five candidates contains no false negatives, making this metric more suitable to study fine-grained reasoning ability.
For Fashion-IQ, we report results on the validation split, as the ground truths of the test. split remain nonpublic. For CIRR, we report our main results on the test split obtained from the evaluation server
4.1.3 Implementation Details
We adopt the standard image pre-processing and model configurations of BLIP encoders (Li et al. 2022 Except for image padding, which we follow Baldrati et al. (202aa) with a padding ratio of 1.25. Image resolution set to 384 x 384. We initialize the image and text encoders with the BLIP w/ ViT-B pre-trained weights. In both stages, we freeze the ViT image encoder and only finetune the text encoders due to the GPU memory limits.
For all models in both stages, we use AdamW (Loshchilov & Hutter 2019 with an initial learning rate of 2 x 10-5, weight decay of 0.05, and cosine learning rate scheduler (Loshchilov & Hutter 2017) with its minimum learning rate set to O.
For candidate filtering (first stage) model, we train with a batch size of 512 for 10 epochs on both Fashion- IQ and CIRR. For candidate re-ranking (sccond stage) model, we reduce the batch size 16 due to the GPU memory limit, it requires exhaustively pairing up queries with each candidate. For Fashion-IQ, we train the re-ranking model for 50 epochs, for CIRR, we train it for 80 epochs.
All experiments are conducted on a single NVIDIA A100 80G with PyTorch while enabling automatic mixed precision (Micikevicius et al. 2018). We base our implementation on the BLIP codebas
1 Both datascts are publicly released under the MIT Liccuse, which allows distributions and academic usages. httpp::/eiirr.ce cecs. edu. au/test, process/ https://gitthb..u hwt.h//a.....mmaaeeseerree//..
7

---

Published in Transactions on Machine Learning Research (01/2024)
Dress
Shirt
Toptee
Average
Avg.
Methods
Re5110
R4350
It510
Re030
It310
Re550
R@10
Ro650
Metric
MRN (iim
34.33
36.33
15.44
34.28
24.86
2
FILM Peres
2018
14.23
33.34
15.04
34.09
17.30
37.68
15.52
35.04
25.26
3
TIRG Vo 0l Su
2019
14.87
31.66
18.26
37. .9.89
19.08
39.62
17.10
37.39
27.10
Relationship Switorr
15.44
18.33
38
21.10
44.77
18.29
40.49
29.39
CIRPLANT
31.66
13.61
33.56
1-
38.31
14.82
35.52
25.17
6
CIRPLANT OSCCR
Lib al
2021
17.45
40.41
17.53
38.81
21 21.64
45.38
18.87
41.53
30.20
7
VAL W/GloVe
Cless ei Il.
2122k
22.53
44.00
22.38
44.15
27.53
51.68
24.15
46.61
35.40
CurlingNet
15.97
31.36
DCNes Kim al,
2021
28.95
566.07
23,95
17.30
30..4
58.
27.78
5:3.89
40.81
10
CoSMo Lee et BL.
2021
2021
25.64
50.30
24.90
19.
29.21
57.46
26 58
52.31
39. 19:45
MAAF Dodils
23.8
21.3
27.9
53.6
48.8
ERTS DDemns
2012
25.68
51.25
28.59
55.06
21.57
44.13
25.25
50.08
37.67
13
SAC w/ w/BEiCT Uandin es al.
2122
26.52
51.01
28.02
51.86
32.70
61.23
29.08
54.70
41.89
14
AMC (Zhu et al. 2023
31.73
59.25
30.67
59.08
36,21
66.06
32 .7
61.61
47:25
CLIPACIR Baldralt
20226
33.81
58.
39.99
00:45
65.37
38.32
61.74
50.03
16
BLIPACIR Lan dt all 2774
42.09
67.33
41.76
04.28
46.61
70.32
43.40
67.31
55.40
17
FAME-ViL. [Tan of
2123
42.19
67 38
47.64
68.79
50.09
73.07
46.55 06.84
69.75
58.29
(Levy
47.77
48.48
70.23
48.79
70.68
59.74
Ours F
43.78
67 38
45,04
67.47
499 62
72.62
46.15
69.15
57.65
20
Ours Ry00
18.14
71.34
50.15
71.25
35.23
76.80
51.17
73.13
62.15
Table 1: Fashion-IQ, validation split. We report Average Metric (Rocallay@@11+ Reell @50) as in (Wu et al. 2021). Rows 1-2 are cited from (Wu et al. 2021). t: Methods trained with additional data in a multi- task setup. F (shaded) denotes candidate filtering model, R denotes candidate re-ranking model with the top-K filtered results from F. For Fashion-IQ we use top-100, which has ground truth coverage of 77.24%, 75.86% and 81.18% for Dress, Shirt and Toptee calegories respectively. Best numbers (resp. second-best) are in black (resp. underlined).
results obtained
Recalli K
Recallsuted &K
Avg-
Methods
K
K
K 10
K 50
K
K
K
Metric
1
TIRG
Vo et al
2019
11.61
48.37
61.088
90.113
22.67
41.97
65.14
35.52
TIRG+
Last ODV
Voret
2019
11.04
35.68
51.27
29
23.82
64.55
29.75
MAAF
Dodds
2072
33,03
61.60
27.01
MAAF HERL (Dodds
ml
2020
10.12
33
48.01
80.57
22.04
42.41
62.14
27.57
MAAF IT
Dodds ct
2012
9.90
32.85
48.83 83.83
80.27
21.17
42.04
60:91
27.02
MAAF RP
68
27.37
CIRPLANT
Liu
2021
15.18
13.36
60.48
87.64
33.88
36,19
75.40
38.59
8
CIRPLANT w W/OSAA
Lim ét
2021
19.55
52.55
68.39
92.38
39.20
63.03
79:19
45.88
ARTEMIS Delmas ot
2022
16.96
39.99
75.67
43.05
10
CLIP4CIR Buutrnt
22222b
38.53
69.98
81,86
95.93
68.19
85.64
94.17
69.09
11
BLIP4CIR Liu ot ml 2024
40.15
73.08
83.88
96.27
72.10
88.27
95.93
72.59
12
CASE (Levy et al.
45.00
79.11
87.25
97.57
75.88
910.58
96.00
77.50
13
CASE Pre-LaSCo.Ca
Levy
19.35
88, 75
90.37
95.71
78.25
14
Ours F
44.70
76.59
86.43
97.18
76.02
80. 02.992
95, 6-4
75.81
15
Ours R.30
50.55
81.75
89.78
97.18
80.04
91.90
96.58
80.90
Table CIRR, test split. We pick our best-performing model on the validation split and submit results online for testing. We report the Average Metric RRecall@5+Reccllsuueee @1)/2 as in (Liu et al. 2021). Rows 1-8 are cited from (Liu et al. 2021). t: Methods trained with additional pre-training data. F (shaded) denotes candidate filtering model, R denotes candidate re-ranking model with results obtained on the top-K filtered results from F. For CIRR we use top-50, which has a ground truth coverage of 97. 97..8% Best numbers (resp. second-best best) are in black (resp. underlined).
4.2 Performance Comparison with State-of-the-art
4.2.1 Results on Fashion-IQ
Table I compares the performance on Fashion-IQ. We note that our re-ranking model (row 20) outperforms all existing methods consistently across three categorics. Impressively, the performance increase notable when compared to CASE (row 18), a method that also uses BLIP encoders. This suggests that our two-stage design, particularly the explicit query-specific candidate re-ranking, is beneficial to the task.
8

---

Published in Transactions on Machine Learning Research (01/2024)
Dress
Shirt
Toptee
Average
Avg.
Methods
R@10
R@55
R@10
RØ50
R@10
R/350
R810
R
Metric
Ours Rio0
48.14
71.34
50.15
71.25
55.23
76,80
51.17
73.13
62.15
w/o. Encoder-t
41.75
68.17
42.64
68.11
47.83
74.35
44.07
70.21
57.14
w/oo Encoder-2,
37.48
66.83
39.16
65.75
46.25
72.62
40.96
68.40
54.68
u: Refers
46.55
71.24
47.84
70.07
54.36
76.83
49.39
72.38
60.98
w. Refees Spacial-6x6
48.04
71.10
48.04
70.31
54.82
76.39
50.30
72.60
61.45
Full-MLP merge
47.00
70.80
47.79
69.63
54.61
76.54
49.80
72.32
61.06
Dual Fred-forward
44.67
69.71
46.37
69.97
52.83
76.39
46.96
72.02
59.99
Table 3:
Ablation studies on Fashion-IQ, validation split. We report Average Metric (Recallavg @10+Recallayg ( Shaded is our candidate re-ranking model as in Table [ row 20. Rows 1 2 ablate the utility of the dual-encoder design. Rows 3-4 examine the difference between using 2n and the reference image features in Encoder-2, in Figure 2 (right). row 4, we choose REfCLS Spatial-Gx6 to showcase the performance under minimum information loss with pooling, as the hardware cannot accommodate a longer sequence of image input. Rows 5 6 test architectural designs. Best results in black.
Recall@K
Recallsubeee'
Avg. Metric
Methods
K
K =
K = 10
K = 50
K=
K K= 2
K = 3
Ours R30
53.24
83.11
90.03
97.08
80.44
92.54
97.01
81.78
w/o. Encoder-t
46.21
79.57
80.33
97.08
75.25
89.72
95.72
77.41
2 w/o. Encoder-2,
43.51
75.25
84.24
97.08
80.34
91.68
96.72
77.79
3
wr. RefeLs Spatial-6xG
52.98
83.23
90.48
97.08
79.55
90.06
96.70
81.39
4 Full-MLPP merge
52.91
82.64
90.15
97.08
79.60
96.89
81.12
Dual Feed-forward
54.20
82.80
90.39
97.08
80.22
91.89
96.53
81.51
Table 4:
We report the Average Metric (Recall@5++ Recalll-eeeellbbbeeeee @1)l Experiments conducted following Table candidate re-ranking model as in Table 2 row 15. Note the values differ for the test and validation splits. Best results are in black.
Ablation studies on CIRR, validation split.
3 on Fashion-IQ. Shaded is
Regarding our first stage filtering model (row 19), achieve a performance slightly behind CASE. As discussed in Section 3. we share a similar, default BLIP-based (Li et al.) 2022 architecture and training pipeline as CASE. Upon examining the ablation studics by Levy et al. (2023), we conjecture that the lower performance is mainly because we adopt a different loss and do not finetune the ViT image encoder alongside due to hardware limits We note that our first stage model primarily intended to obtaín the top-K filtered candidate list, and that the main focus of our work the re-ranking model outperforms all existing methods by a large margin.
4.2.2 Results on CIRR
Table 2 compares the performance on CIRR. Overall, we observe a similar trend in performance increase as in Fashion-IQ, IQ. This includes the performance comparison between our filtering model (row 14) and CASE (Levy) et al.] 2023) (row 12), as discussed above. We notice that our re-ranked results (row 15) outperform all previous methods, including models that are based on BLIP and pro-trained on additional data of large scales (row 13). This demonstrates that our design more effectively harnesses the information within the input entities than existing work through explicit cuery-candidate reasoning.
We unable reproduce or assess their results the time of writing the code for CASE is not publicly released.
9

---

Published in Transactions on Machine Learning Research (01/2024)
Dress
Shirt
Toptee
Average
Avg.
Methods
RØ10
R@50
R@10
RØ50
R@10
R@50
R0
R0550
Metric
F (candidate filtering)
43.78
67.38
45.04
67.47
49.62
72.62
46.15
69.15
57.65
Rso
48.24
67 .8
50.15
67.47
54.56
72.62
50.98
69.15
60.07
Rz0
48.
09.66
50.59
69.09
54.87
75.83
51.20
71.52
61.36
Rgo
48.14
70.90
50.15
70.76
55.07
76.13
51.12
72.60
61.86
Ri00
48.14
71.34
50.15
71.25
55.23
76.80
51.17
73.13
62.15
R150
48.
71.49
50.20
71.49
55.28
77.41
51.19
73.46
62.33
R200
47.89
71.44
50.15
71.00
55.38
77.41
51.14
73.28
62.21
Table 5: Fashion-TQ,, validation split. Rx= Ablation on K value in candidate re-ranking. F (row 1): the candidate filtering model (attached as a reference to compare against), as Table ! row 19. Shaded: our choice of K when reporting the main results in Table row 20.
Recall@
Recallsubad &K
Avg.
Methods
K
K
K
10
K
50
K
K
2
K
Metric
(candidate filtering)
46.83
78.59
88.04
97.08
76.11
90.65
96.05
77.53
2
Rao
53.03
83.16
90.62
95.26
80.44
92.54
97.01
81.80
3
R4o
52.98
82.83
90.29
96.27
80.44
92.54
97.01
81.64
4
Rss
53.24
83.11
90.03
97.08
80.44
92.54
97.01
81,78
5
R,00
52.88
82.92
90.07
97.90
80.44
92.54
97.01
81.68
6 R,500
52.91
82.85
90.05
98.04
80.44
92.54
97.01
81.65
Table 6: CIRR, validation split. Rx: Ablation K value in candidate re-ranking. F (row 1): the candidate filtering model (altached as reference to compare against), as in Table 2 row 14. Shaded: our choice of K when reporting the main results in Table 2 row 15. Note the values differ for the test and validation splits.
4.3 Ablation Studies
4.3.1 Key Design Choices
In Table 8 we several variants of the re-ranking model to verify our design choices. We report performance the Fashion-IQ validation split for all experiments.
We begin with assessing the necessity of our dual-encoder setup, as shown in Table 3 rows 1 and 2. Given that & is obtained from both the text t and the reference image JR (Figure 2 left), one might question if explicit text information, ic., Encoder-t, is still noeded in the re-ranking step. To this end, in row 1, we demonstrato that the performance would decrease significantly if Encoder-L is removed from the setup. Subsequently, in row 2, we validate that Encoder-2, also crucial the performance, which suggests that the reference image information embedded in 2, is meaningful for the prediction. However, this alone does not fully justify our scheme of using Recall what was discussed in Section our motivation for adopting 2₁ instead of the embedding of Jn. is that GPU memory consumption prohibits direct image-image cross-attention, unless certain spatial pooling is applied to the reference image. To validate that using ,, is a better alternative to the pooling methods, in rows 3 and 4, we show that the performance decreases when replacing 2 with such pooled features.
surrogate of JR. thus allowing interactions between the reference and candidate images.
We additionally show two variants related our design choices. Row replaces the first six merging layers from the average pooling to MLP, while row 6 removes the weight-sharing of the FF layers. We note consistent performance decrease in both cases.
Table 4 shows the ablation studies conducted on CIRR, which complements the same set of experiments performed on Fashion-IQ. We find that both the text and reference image play an important role in CIRR (rows and 2) as in Fashion-IQ, and our choice of using 2; as a surrogate for it is validated (row 3). Regarding the ablated design choices of the architocture (rows and 5), we notice they bear a slightly smaller impact on CIRR than on Fashion-IQ, but our design still yields better overall performance in the Recallsubsee and Average Metric.
10

---

Published in Transactions on Machine Learning Research (01/2024)
Interestingly, when combining the results shown in rows 1 and 2, we discover that Encoder-t, ie., explicit text information, is more important to the Recallsubect metric, while Encoder-z,, along with the embedded reference image, is more crucial to the global Recall metric. This is thought to be caused by the fact that Recallsubset only considers a selected group of five candidates of high similaritics, hence the information within the reference image contributes less to the retrieval. Meanwhile, the need of using visual cues in the reference image to exclude negatives remains vital to the global Recall as the candidates are not pre-sclected per their visual similarities in this scenario. Indeed, previous (Liu et al., 2021) as well concurrent work (Levy et al, 2023
Vaze et al. 2023 observe a similar case in their ablation studies.
4.3.2 K values in Candidate Re-ranking
Tables 5 and 0 study the effect of varying K on the re-ranking model. Recall that K is the number of samples taken from the first stage to be re-ranked. As discussed in Section 3.3 K only affects inference but not training.
Given that increasing K effectively increases the ground truth coverage (sec definition in Section 3.1 and detailed statistics in Section A.2), one could reasonably expect that a larger K yields higher performance. However, we note that a higher K would also lead to more negative candidates, which potentially impact the performance should the re-ranking model fails to properly rank them. To this end, trade-off exists between the two contributing factors.
For Fashion-TQ, we see general trend of performance increase while increasing K, except for when K = 200 (Table 5 row 7). For CIRR (Table 0 we observe minor performance differences when varying K, which is unsurprising given the high ground truth coverage in general (see Table 2 caption). Consequently, we could only notice a clear trend in Recall@50. Here, we note that K does not affect validating on Recallsuboet: as such a metric only concerns five pre-determined candidates per query.
As discussed in Section 3.1 we have not handpicked K per training instance. Instead, for every dataset, we globally select K based on the empirical balance between the ground truth coverage and inference cost.
4.4 Inference Time
One obvious limitation of our method is the inference time of the re-ranking model, as it requires exhaustive pairing of the query and top-K candidates. Several factors contribute to the case, including the size of the validation/Les split of the dataset, choice of K, as well as the general length of the input lext, which affects the efficiency of the attention layers within the transformer. We observe that compared to a traditional method, in this case, our filtering model, the inference time of the re-ranking step is increased by approximately eight times on Fashion-IQ and 35 times on CIRR. Qualitatively, with our default choices of K in Tables 1 and [ it takes around 0.10 seconds to infer on query for Fashion-IQ and 0.11 seconds for CIRR, which amounts to approximately 10 and 7.5 minutes for the validation split of the two datasets respectively. See Section
vector distancing-base
A.3 for detailed analysis on inference time us. the choice of K.
We note that our focus of this work is on achieving higher performance through model architectural design and better use of input information, and is not optimized for applications where processing speed is more important than retrieval accuracy. This in line with most existing work on vision-and- languuug such as BLIP (Li et al. 2022), ; where the task of image-text retrieval is also studied in non real-time fashion. We point out that the additional cost results in significant increase in performance around 5% absolute in average Recall compared to the pre-filtered results (Tables 1 and 2 between the last two rows).
4.5 Qualitative Results
We present several retrieved results on CIRR in Figure 4 where we show the pipcline of filtcring followed by re-ranking. For (a) and (b), we note that explicit text-candidate pairing can be more beneficial in cases where new elements are added (i.e., "trees", "two people" and "cat"), as the re-ranking model can readily identify concepts within each candidate, and assess its correlations between the text. We specifically point to (b), where with initial filtering, only one candidate among the top-6 contains cat the text describes. After re-ranking, three candidates with cat are brought forward, with the true target ranked the first. Tn
11

---

Published in Transactions on Machine Learning Research (01/2024)
(a) Put trees belind the horse drawn carriage with people riding it.
(b) Change the dog to durker color and
add cat.
(c) Take the picture from library.
(d) Be a same breed dog with puppy running.
Figure Qualitative examples on CIRR. For cach sample, we showcase the query (left) with the filtered top-6 candidates (row F), followed by the re-ranked top-6 results (row R). True targets are green frames. We demonstrate three cases where re-ranking brings the true target forward (a-c), and one failure case (d).
(c), we show that our re-ranking model is also effective at recognizing global visual concepts such scenes (i.e., library). Finally, we list failure case of the re-ranking in (d), where observe that our re-ranking model fails to associate the concept of "with" with having two entities simultancously in the image. As a result, the lop-3 re-ranked candidates each only pictures a single puppy rumning. We additionally point out that the filtering module is effective at removing easy negatives. As shown Figure (row F) on each sample, the top-ranked candidates are already picturing similar objects or scenes as the reference. Qualitative examples on Fashion-IQ are demonstrated in Appendix A..I, A detailed quantitative analysis on the effect of re-ranking presented in Section A.4 which complements discussions in this section.
5 Conclusion
We propose two-stage method for composed image retrieval, which trades off the inference cost with model that cxhaustively pairs up queries with each candidate image. Our filtering module follows prior work and uses vector distances to quickly generate a small candidate set. We then design dual-encoder architecture to assess the remaining candidates against the query for their relevancy. Both stages of our method are designed based on the existing vision-and-langee langgaagge pre-trained model. We experimentally show that our approach consistently outperforms existing methods on two popular benchmarks, Fashion-IQ and CIRR.
References
Peter Anderson, Xiaodong He, Chris Buchler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answering. In IEFE Conference
12

---

Published in Transactions on Machine Learning Research (01/2024)
Computer Vision and Pottern Recognition, 2018.
Muhammad Umer Anwaar, Egor Labintcev, and Martin Kleinsteuber. Compositional learning of imago-text query for image retrieval. In Proceedings of the IBBE/CVF Winler Conference on Applicalions of Computer Vision (WACV), January 2021.
Alberto Baldrati, Marco Bertini, Tiberio Uricchio, and Alberto Del Bimbo. Effective conditioned and composed image retrieval combining clip-based features. In Proceedings of the IREE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2022a.
Alberto Baldrati, Marco Bertini, Tiberio Uricchio, and Alberto Del Bimbo. Conditioned and composed image retrieval combining and partially fine-tuning clip-based features. In Proceedings of the IEEE/CW Conference on Computer Vision and Pottern Recognition (CVPR) Workshops, 2022b.
Yanbei Chen, Shaogang Gong, and Loris Bazzani. Image search with text fcodback by visiolinguistic attention learning. In IEEE Conference on Compuler Vision and Pallern Recognition, 2020a.
Ven-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In European Conference on Computer Vision, 2020b.
Ondrej Chum, James Philbin, Josef Sivie, Michael Tsard, and Andrew Zisserman. Total recall: Automatic query expansion with a generative feature model for object retrieval. In 2007 IEEE 11th International Conference on Compuler Vision, 2007.
Guillaume Couairon, Matthijs Douze, Matthieu Cord, and Holger Schwenk. Embedding arithmetic of multimodal queries for image retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4958, 2022.
Ginger Delmas, Rafael Sampaio de Rezende, Gabriela Caurka, and Diane Larlus. Artemis: Attention-based retrieval with text-explicit matching and implicit similarity. In International Conference on Learning Representations, 2022.
Jacob Devlin, Ming-Was Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North Americon Chapler of the Associalion for Compulational Linguistics: Human Language Technologies, Volume (Long and Short Papers), 2019.
Eric Dodds, Jack Culpepper, Simao Herdade, Yang Zhang, and Kofi Boakye. aadallity-gnosstee attention fusion for visual search with text feedback, 2020. ai preprint arXiv:2007.0014: [cs.CV].
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference Learning Representations, 2021.
Xiao Han, Xiatian Zhu, Licheng Yu, Li Zhang, Yi-Zhe Song, and Tao Xiang. Famo-vil: Multi-tasking vision-language model for heterogeneous fashion tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2669 2680, 2023.
Xintong Han, Zuxuan Wu, Phoenix X Huang, Xiao Zhang, Menglong Zhu, Yuan Li, Yang Zhao, and Larry S Davis. Automatic spatially-aware fashion concept discovery. In IEFE International Conference on Computer Vision, 2017.
M. Hosseinzadeh and Y. Wang. Composed query image retrieval using locally bounded features. In IEEE Conference on Compuler Vision and Pallern Recognition, 2020.
S. Jandial, P. Badjatiya, P. Chawla, A. Chopra, M. Sarkar, and B. Krishnamurthy. Sac: Semantic attention composition for text-conditioned image retrieval. In 2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2022.
13

---

Published in Transactions on Machine Learning Research (01/2024)
Jin- Kim, Sang- Woo Lee, Donghyun Kwak, Min-Oh Heo, Jeonghee Kim, Jung- Woo Ha, and Byoung-Tae Tak Zhang. Multimodal residual learning for visual qa. In Advances in newral informalion processing systems, 2016.
Jongseok Kim, Youngjae Yu, Hoeseong Kim, and Gunhee Kim. Dual compositional learning in interactive image retrieval. Proceedings of the AAAI Conference on Artificial Intelligence, 2021.
Seung-Min Lee, Dongwan Kim, and Bohyung Han. Cosmo: Content-style modulation for image retrieval with text feedback. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
Matan Levy, Rami Ben-Ari, Nir Darshan, and Dani Lischinski. Data roaming and early fusion for composed image retrieval, 2023. ar preprint arXiv:2303.09 0094299930999999/ [cs.CV].
Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafqq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. Advances in neural information processing systems, 2021.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International Conference on Machine Learning, 2022.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In International Conference on Machine Learning, 2023.
Wen Li, Lixin Duan, Dong Xu, and Ivor Wai-Hung Tsang. Text-based image retrieval using progressive multi-instance learning. In IEEE International Conference on Computer Vision, 2011.
Xiujun Li, Xi Yin, Chunyuan Li, Xiaowei Hu, Pengchuan Zhang, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, Yejin Choi, and Jianfeng Gao. Oscar: Object-semantics aligned pre-training for vision-language tasks. In European Conference on Computer Vision, 2020.
Tsung-Y Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. In European Conference Computer Vision, 2014.
Zheyuan Liu, Cristian Rodriguez-Opazz, Damien Teney, and Stephen Gould. Image retrieval real-life images with pre-trained vision-and-languaae models. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 2125-2134 2021.
Zheyuan Liu, Weixuan Sun, Yicong Hong, Damien Teney, and Stephen Gould. Bi-directional training for composed image retrieval via text prompt learning. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 5753-5762, 2024.
Ilya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. In International Conference on Learning Representations, 2017.
Nya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019.
Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. Mixed precision training. In International Conference on Learning Representations, 2018.
Ethan Perez, Florian Strub, Harm de Vries, Vincent Dumoulin, and Aaron C. Courville. Film: Visual reasoning with a general conditioning layer. In Proceedings of the AAAI Conference on Artificial Intelligence, 2018.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krucgor, and Tlya Sutskever. Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning, 2021.
14

---

Published in Transactions on Machine Learning Research (01/2024)
Adam Santoro, David Raposo, David G Barrett, Mateusz Malinowski, Razvan Pascanu, Peter Battaglia, and Timothy Tillicrap. A simple neural network module for relational reasoning. Tn Advances in neural information processing systems, 2017.
Xiaohui Shen, Zhe Lin, Jonathan Brandt, Shai Avidan, and Ying Wu. Object retrieval and localization with crantally-oonstrained similarity measure and k-nn re-ranking. In 2012 IEEE Conference on Computer Vision and Pottern Recognátion, 2012.
Alane Suhr, Stephanie Zhou,
Ally Zhang, Iris Zhang, Huajun Bai, and Yoar Artzi A corpus for reasoning about natural language grounded in photographs. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019.
Simon Tong and Edward Chang. Support Vector Machine active learning for image retrieval. In Proceedings of the Ninth ACM International Conference on Multimedia, 2001.
Sagar Vaze, Nicolas Carion, and Ishan Misra. Genecis: A benchmark for general conditional image similarity. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, 2023.
Nam Vo, Lu Jiang, Chen Sun, Kevin Murphy, Li-Jia Li, Li Fei-Fei, and James Hays. Composing text and image for image retrieval an empirical odyssey. In IEEE Conference on Computer Vision and Pattern Recognition, 2019.
Haokun Wen, Xuemeng Song, Xin Yang, Yibing Zhan, and Liqiang Nic. Comprehensive linguistic-visnal composition network for image retrieval. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2021.
Hui Wu, Yupeng Gao, Xiaoxiao Guo, Ziad Al-Halah, Steven Remnie, Kristen Grauman, and Rogerio Feris. Fashion iq: A new dataset lowards relrieving images by natural language feedback. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11307-11317, 2021.
Yuchen Yang, Min Wang, Wen gang Zhou, and Houqiang Li. Cross-modal joint prediction and alignment for composed query image retrieval. Proceedings of the 29th ACM International Conference on Multimedia, 2021.
Youngjae Yu, Seunghwan Lee, Yuncheol Choi, and Gunhee Kim. Curlingnet: Compositional learning between images and text for fashion iq data, 2020. arXiv preprint arXiv:2003. [cs. CV..
Chen Zhang, Joyce Y. Chai, and Rong Jin. User term feedback in interactive text-based image retricval. Proceedings of the 28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, 2005.
Hongguang Zhu, Yunchao Wei, Yao Zhao, Chujjie Zhang, and Shujuan Huang. Ame: Adaptive multi-expert collaborative network for text-guided image retricval. ACM Transactions on Multimedia Computing, Communications, and Applications, 2023.
A Appendix
A.1 Qualitative Examples on Fashion-IQ
Here, we provide qualitative examples on Fashion-IQ in Figure 5 which tells similar story as in CIRR. We show cases where re-ranking brings the ground truth forward in (a-d), along with failure cases in (e-f). Note the sometimes ambiguous and abstract human annotations when describing fashion style changes (c.g., "more patriotic" in a), whose ground truth target can be difficult estimate purely based on the query. In these conditions, explicitly reasoning over candidate images is particularly beneficial.
15

---

Published in Transactions on Machine Learning Research (01/2024)
(n) Is more patriotic rather than religious and darker blue with more colorful graphic. Ground truth initially ranked 11ch
(b) shiny and silver with shorter sleeves
(e) length slenved purple and red dress and is longer.
Ground truth initially ranked Sth
(d) more contrast
and is sleeveless
lighter
color.
(c) Is shorter and bold and is orange.
(f) Has short sleeves and white shiny and is white.
Figure 5: Qualitative examples on Fashion-IQ. For each sample, we showcase the query (left) with the filtered top-6 candidates (row F), followed by the re-ranked top-6 results (row R). Each query comes with two sentences of annotations which is joined by "and". True targets are green frames. For examples with ground truth initially ranked beyond the top-6, we report their rankings below the annotation, as in (a) and (c).
16

---

Published in Transactions on Machine Learning Research (01/2024)
30
stirt
diess topite
98
Ground Inuth corenge (36
96
70
94
92
90
88
Figure 6: Ground truth coverage (in percentage) us. K for candidate re-ranking, Left: Fashion-IQ validation
split with statistics of the three categories. Right: CIRR validation split. The ranges of K follow Tables
and 0
1000
Interenee time (000)
800
600
600
Inferene
400
200
300
190
130 140 150
Figure Inference time (in seconds) us. K for candidate re-ranking. Left: Fashion-IQ validation split. Right: CIRR validation split. The ranges of K follow Tables and
A.2 Analysis on Ground Truth Coverage vs. K
In Figure we illustrate the relationship between the ground truth coverage and the choice of K, which complements discussions in Section 4.3.2
A.3 Analysis Inference Time K
In Figure i we study the relationship between the choice K and the inference time, which complements
discussions in Section 4.4 We anticipate their relationship to be roughly linear. Here, we point out that
regardless of the choice of K, there exist certain fixed overheads (such model checkpoint loading). However, they only account for small percentage of the total time. Therefore, the overall trend remains linear, demonstrated in the figure.
A.4 Quantitative Analysis on Candidate Re-ranking
Here, we perform quantitative analysis of the effect of the re-ranking model. In Table we report the average rankings of the largets in the validation split before and after the re-ranking (lermed the initial and re-ranked rankings), along with their differences for both datasets. Globally, we observe that the targets are more highly ranked on average after re-ranking, which corresponds the higher Recall K performance Tables and 2 We notice that on Fashion-IQ, the targets are on average brought forward by in the indexes depending the category, which is significant. improvement. Meanwhile, CIRR, the absolute
17

---

Published in Transactions on Machine Learning Research (01/2024)
Initial ranking
Re-ranked ranking
Difference (A)
Percentage decrease (%)
Fashion-IQ Dreas
28.74
22.25
6.49
22.58
Fashion-IQ Shirt
26,96
20.42
6.54
24.26
Fashion-IQ Toptee
25.81
18.
7.07
27.39
CIRR
5.32
4.09
1.22
23.12 12
Table 7: The average initial and re-ranked rankings for targets in the validation splits of Fashion-1Q and CIRR. For Fashion-IQ, report detailed statistics of the three categories. Initial ranking: rankings after the first stage filtering model. Re-ranked ranking: rankings after the second stage re-ranking model. Difference (A): the difference between the two stages, where a positive value suggests that the targets are brought forward by the re-ranking model, which is favoured. Percentage decrease: calculated as (difference initial ranking) x100%. We consider the top-200 candidates per query for this analysis.
value is smaller. However, note that the initial rankings of targets in CIRR are also much higher, leaving less room for improvement compared to Fashion-IQ. Overall, the percentage decrcase in the target indexes on both datasets is over 20%, which validates that our re-ranking model is extremely effective.
Figure S further investigates the effectiveness of our re-ranking model, particular, we wish to answer the question "How much benefits does the re-ranking stage bring to queries of different levels of difficulty?". To this end, we examine, for each dataset (and category in Fashion-IQ), the average difference (A) in target indexes before and after re-rank us. their initial rankings. We first observe that most of the A values positive, suggesting that the targets are brought forward through re-ranking. This corroborates with the discussions above for Table 7 To take the analysis to finer level, we point to cases where the initial rankings of the targets are extremely low (i.e., their indexes are high, which puts them on the right- hand side of the histograms). They are presumed hard cases, as the first stage filtering model failed at ranking the targets high. We note that our re-ranking model can oflen significantly improve the larget rankings by over or even 100 in indexes under such circumstances, as demonstrated by the high A values. This further confirms the strength of re-ranking. Interostingly, we also point out that the standard Recall@ metric may not fully capture the benefit of the re-ranking in such cases, mostly because target that is initially ranked extremely low may not be brought forward sufficiently enough, even if its index is improved by 50 or 100. This is especially true for CIRR where the metric considers Recall@5. Nevertheless, the improvements in such challenging cases still reveal the offectiveness our re-ranking model.
18

---

Published in Transactions on Machine Learning Research (01/2024)
(a)
(b)
150
100
Avergee
Avange
-50
-50
Nomber of samples
samples
100
108
10a
10
m M M
10a
INY
MAN
100
50
150
200
100
125
1.50
175
200
Initial ranking (index)
Initial rankiing (index)
(c)
(d)
100
100
Avenge
-50
Numbber of samples
of sumples
10P
102
10a
10a
m
1
mm
10"
10a
Www
10P
Initial ranking (index)
Initial ranking (index)
Figure 8: The initial rankings of targets us. the difference in average ranking (noted as Average A) before and after the re-ranking stage. A posilive Average value suggests that the targels are brought forward by the re-ranking model, which favoured. Likewise for a negative value. In each plot, we also show the number of queries counted for every initial ranking. (a) Fashion-IQ Dress. (b) Fashion-IQ Shirt. (c) Fashion-IQ Toptee. (d) CIRR. Analysis performed on the validation splits. We consider the top-200 candidates per query for this analysis, in Table 7
19


Reviews:
[{"review_id": "wl1UQblhnb", "reviewer": "Reviewer_tcq4", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\n1. Extensive experiments with detailed analysis of ablation studies are conducted.\n2. Pre-filtering to balance the inference cost and performance. \n\nWeakness / Concerns:\n\nMy primary concern revolves around the fact that this architecture, without incorporating supplementary structures for text, appears to solely compute text similarity. In simpler terms, I am apprehensive that this model might encounter difficulties with ambiguous sentences. For example, if we input an image of a yellow cat along with the text 'change to a white cat,' what outcomes could we anticipate?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a two-stage method to find an image that best matches a reference image and text pair. The first stage filters the large corpus to obtain hard negatives, while the second stage, a scoring task, employs a dual-encoder for re-ranking the hard negatives. This achieves a balance in inference cost via the pre-filtering. The effectiveness of the proposed method is validated on two datasets in different domains.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Could you provide more experiments regarding my concern? \n2. The choice of $K$ contributes to the inference time of the re-ranking model. Could you provide more information about the relationship between the number of hard negatives $K$ and the inference time? Additionally, how can we generally choose a suitable $K$?\n\nMinor:\n\nIn the first paragraph of sec4.3.2：‘Table 5 and Table 6 studies’ <-study", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Z3uP2EFcYH", "reviewer": "Reviewer_iJWP", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n1. The idea of leveraging two-stage pipeline is interesting.\n\n2. The experiment results are strong.\n\nWeaknesses:\n1. As shown in the experiments, the proposed method is much slower than previous methods. While it is true that it significantly improves the performance, I am afraid that such a method would not be practical in real-life scenarios.\n\n2. According to Sec. 4.1.3 the text encoder is optimized during training. In this way, how can the embedding $z_t$ be pre-computed?\n\n3. It is not clear that how the model is optimized. Is the candidate filtering model used in the training of stage2 model? If so, how are they alternatively updated? \n\n4. Ideally the encoder-t and encoder-$z_t$ should reflect the similarity between candidate image and $I_R$ and $t$ individually. I wonder if the authors can provide any experiments to explain the information learned in the dual encoder.\n\n5. I wonder if it is necessary to conduct an experiment to show the sanity of candidate filtering. For example, what is the probability that the ground truth will be contained in the filtered set?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper mainly targets composed image retrieval. The authors point out the problem of the previous methods that compare pre-computed image embeddings to the query text-engaged image embedding. To solve the problem they propose a two-stage pipeline that includes a candidate filtering stage, which filters out large proportion of candidate images, and a re-ranking stage which computes two candidate-aware embeddings with two separate branches. The proposed method is evaluated on two datasets including Fashion-IQ and CIRR, on which the results reflect the effectiveness of the method.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The authors can refer to the weaknesses for the problems that need to be solved. Moreover, I suggest the authors summarize the main notations in the overview in Sec.3 for better understanding.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "2WDJ15bqJP", "reviewer": "Reviewer_HV9g", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n\n**Motivation:** The paper's motivation is commendable. Designing a reliable metric for single-stage retrieval methods that can simultaneously perform coarse-grained filtering and fine-grained ranking is a challenging task. Thus, the addition of a specialized re-ranking stage is a natural and beneficial choice.\n\n**Clarity:** The paper is well-written and effectively communicates its key points. Figures 2 and 3 are of high quality and successfully convey essential information.\n\n**Performance Improvement:** The proposed method demonstrates a significant performance improvement. For instance, in Table 1, it outperforms the single-stage baseline (Ours F) by approximately 4.5%. Additionally, it surpasses all other publications listed in the table.\n\n**Ablation Analysis:** The authors have invested substantial effort in conducting ablation experiments to analyze their design choices thoroughly.\n\n**Weaknesses:**\n\n**Input Limitation:** It appears that the method requires both an input image and a text description, limiting its real-world applicability. It would be ideal if the proposed method could handle scenarios where only text inputs are provided, enhancing its versatility.\n\n**Runtime Overhead:** The second stage introduces a substantial runtime overhead, resulting in a 5% performance gain. This trade-off may not be favorable, considering the speed overhead of 8-35x. Given this large runtime overhead, simpler alternatives such as ensembling might outperform the baseline significantly. Moreover, the proposed method's scalability to real-world datasets and applications may be hindered by this runtime overhead.\n\n**Component Originality:** While the paper is easy to understand and well-written, it appears that most of its components are derived from existing methods. For example, the authors acknowledge that the objective is similar to and inspired by BLIP's ITM loss, raising questions about the novelty of the approach.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this research paper, the authors investigate the challenge of composed image retrieval. Traditional approaches involve computing multi-modal embeddings for both query and target images, subsequently returning candidates with the smallest cosine distance to the query. However, the authors argue that augmenting this process with a second re-ranking stage can significantly enhance performance.\n\nThe key insight lies in reformulating the re-ranking task as a scoring problem, wherein true-positive candidates are assigned higher scores, while other candidates receive lower scores. Extensive evaluations conducted on Fashion-IQ and CIRR datasets demonstrate that the proposed method yields substantial performance improvements over single-stage baselines. It is worth noting that this enhanced performance comes at the expense of relatively higher computational runtime, ranging from 8 to 35x.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "It would be great if the authors could respond to my comments on \"Weaknesses\".", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper underwent a thorough review process involving multiple experts. Unfortunately, the reviewing process took longer than initially anticipated for certain reasons. Notably, two out of three reviewers express positive support for the paper. Overall, the authors delve into a compelling task and set a new benchmark for it. The majority of reviewers are in agreement that the paper's claims are substantiated by experimental results, a pivotal factor leading to the Assistant Editor's recommendation for acceptance.

 However, reviewers did raise concerns about the paper's perceived lack of novelty for ICLR and the efficiency of the proposed algorithm. These critiques are seen as valuable feedback, and addressing them could be a focus for future work.
Decision: Accept (poster)