Image2Sentence based Asymmetrical Zero-shot Composed Image Retrieval
Authors: Yongchao Du
Min Wang
Wengang Zhou
Shuping Hui
Houqiang Li
Venue: ICLR 2024 spotlight (2024)
Keywords: zero-shot
composed image retrieval
asymmetrical
Primary area: representation learning for computer vision, audio, language, and other modalities

Abstract: The task of composed image retrieval (CIR) aims to retrieve images based on the query image and the text describing the users' intent. 
Existing methods have made great progress with the advanced large vision-language (VL) model in CIR task, however, they generally suffer from two main issues: lack of labeled triplets for model training and difficulty of deployment on resource-restricted environments when deploying the large vision-language model. To tackle the above problems, we propose Image2Sentence based Asymmetric zero-shot composed image retrieval (ISA), which takes advantage of the VL model and only relies on unlabeled images for composition learning. In the framework, we propose a new adaptive token learner that maps an image to a sentence in the word embedding space of VL model.  The sentence adaptively captures discriminative visual information and is further integrated with the text modifier. An asymmetric structure is devised for flexible deployment, in which the lightweight model is adopted for the query side while the large VL model is deployed on the gallery side. The global contrastive distillation and the local alignment regularization are adopted for the alignment between the light model and the VL model for CIR task.  Our experiments demonstrate that the proposed ISA could better cope with the real retrieval scenarios and further improve retrieval accuracy and efficiency.

Published as a conference paper at ICLR 2024
IMAGE2SENTENCE BASED ASYMMETRIC ZERO-SHOT COMPOSED IMAGE RETRIEVAL
Yongchao Du', Min Wang?* Wengang Zhou 2, Shuping Hui', Houqiang Li CCAS Key Laboratory of Technology in GIPAS, University of Science and Technology of China Insttittte of Artificial Intelligence, Hefei Comprehensive National Science Center {ycdu2020, huisp @maall wuscc.c...... edu, en, wangmine iai uutu.uutuuu. cn, (zhwg, Liha]oustee edu. cn
ABSTRACT
The task of composed image retrieval (CIR) aims to retrieve images based on the query image and the text describing the users' intent. Existing methods have made great progress with the advanced large vision-language (VL) model in CIR task, however, they generally suffer from two main issues: lack of labeled triplets for model training and difficulty of deployment on resource-restricted environments when deploying the large vision-language model. To tackle the above problems, we propose Image2Sentence based Asymmetric zero-shot composed image re- trieval (ISA), which takes advantage the VI. model and only relies on unlabeled images for composition learning. In the framework, we propose a new adaptive token learner that maps an image to sentence the word embedding space of VI. model. The sentence adaptively captures discriminative visual information and is further integrated with the text modifier. An asymmetric structure is devised for flexible deployment, in which the lightweight model is adopted for the query side while the large VL model is deployed on the gallery side. The global contrastive distillation and the local alignment regularization are adopted for the alignment between the light model and the VI. model for CIR task. Our experiments demon- strate that the proposed ISA could better cope with the real retrieval scenarios and further improve retrieval accuracy and efficiency.
INTRODUCTION
Image retrieval is a crucial task for computer vision, and plays a fundamental role in a variety of applications, e.g., product retrieval Dong et al. (2022); Zhan et al. (202T); Liu et al. (2016); Zhou al. (2010), landmark retrieval Radenović et al. (2018). and person re-identification Zheng et al. (2016) Gheisssri et al. (2006). Composed image retrieval (CIR) is presented as more challenging task that requires accurate understanding of the users' intentions. CIR allows multi-modal queries and aims to retrieve images that are both visually similar to the given query image and match with the text modifier, which enables users to conduct more precise and flexible search. However, to learn the composition ability, it requires labeled triplets of reference image. target image and text modifier Liu et al. (2021) Wu et al. (021). which are time-consuming and expensive to collect.
To eliminate the limitation of data scarcity, zero-shot composed image retrieval (ZSCIR) is pro- posed in Saito et al. (2023): ;: Baldrati et al. (2023a), which converts the CIR task to text-to-image retrieval by mapping the image into single word and combining it with text modifier. However, they all adopt symmetric retrieval setting based on CLIP models RRadford et al. (2021). i.e., using large models to extract both query and database features, and fail to consider the real-life application where the model may be deployed on a eesource-limites platform such as mobile device, etc. More- over, the mapping is implemented by an MLP that projects image feature to a single word vector, ie., representing image as single word in the word embedding space. This may be insufficient as a faithful representation for an image, and the heterogeneity of word embedding space and visual feature space may further dampen the discriminative information of image feature after projection, thus constraining its potential.
"Correponning authors: Min Wang and Wengang Zhou

---

Published as a conference paper at ICLR 2024
To address the above issues, we propose Image2Sentence based Asymmetric ZSCIR that utilizes different models to extract query and database features. Specifically, a lightweight model is adopted for the query side, which is usually user's terminal device with limited computation resources; the large VI. model is employed for the gallery side, which is usually deployed on a cloud platform. Asymmetric structures often suffer from two core issues: inferior representation capability and misalignment between the lightweight model and the larger model. To this end, we propose an adaptive token learner realize the alignment between the lightweight visual model and the large visual language model, which could further improve the representation discrimination ability of the lightweight model. Specifically, the lightweight model extracts the visual feature map of an input image, then the adaptive token learner extracts visual tokens from visual feature map, and maps them to more informative sentence the word embedding space, By adaptively focusing on more dis- tinctive visual patterns, our adaptive token learner serves as a semantic filter that automatically filters out trivial background noise and preserves discriminative visual information. The mapped sentence is learned to cope with the visual feature from the large visual encoder under the global contrastive distillation (GCD), as a constraint to build up semantic alignment between the large model and the lightweight model. Therefore during inference, the mapped sentence is directly concatenated with the modifier text as a caption to search for images. To better learn the image2sentence mapping and retain more visual information, we further leverage the local alignment regularization (LAR) to help the mapped sentence capture more distinctive visual concepts, and to be closer to the real caption that faithfully describes the image. Our asymmetric framework allows for more flexible deployment while also enhancing performance compared with symmetric setting, and outperforms the state-of-the-ar methods on three benchmarks.
2 RELATED WORK
2.1 COMPOSED IMAGE RETRIEVAL
CIR is the task of retrieving images that correspond to the reference image with modifier text, and has been investigated in several domains, e.g., natural scene Baldrati et al. (2023a); Liu et al. (2021). fashion domain Dodds et al. 2002 Lee et al. (2021): Wu et al. (2021). ) and synthetic domain Gu et al. (003)); Vo et al. (2019). Supervised learning serves as stepping stone in the progression of CIR, and is first introduced in Vo et al. (2019), in which meticulously annotated (reference image, text modifier, target image) triplets are adopted for training. Supervised methods mainly follow the pipeline that merges the features of reference image and text modifier to align with the target image feature, as a result, their innovations generally revolve around jointly capturing the information from both visual and textual modality Vo et al. (2019); Lee et al. (202I): [Cenn et al. (2020). Moreover, some methods further improve the performance by adopting multimodal foundation model Baldrati
let al.] (2021 2022a b. 2023b); Levy et al. (2023) or incorporating pretraining- finetuning strategy
Dodds et al. 2022 Han al. 2022 2023
However, the above supervised methods heavily rely on the triplet training dataset, which involves great difficulty in collecting and labeling in large scale. A new task of zero-shot composed image retrieval (ZSCIR) (Saioo et al. (2023) is proposed to liberate the model learning from meticulous labllt-labeled data, and relies only on image-text pairs. [aato et al. (2023) propose to use MLP to convert CLIP visual fcature to a single word in the word embedding space, and adopt a cycle contrastive loss for training: Baldrati et al. (2023a) propose a two-stage framework including text inversion and distillation to convert image into text domain. However, the above methods rely on symmetric retrieval, in which heavy models are utilized to extract features for both query and database, and are difficcult to deploy in practical retrieval application scenarios. We propose a ZS- CIR framework, which enables flexible deployment on resource-limited scenarios by adopting the lightweight model. Different from previous works, instead of converting image single word, we propose the adaptive token learner to filter out more discriminative visual information, which is further mapped to more informative sentence. Apart from global contrastive loss, a dense local alignment regularization is applied on mapped sentences to preserve the visual information during conversion. Our work shows its superiority in both retrieval accuracy and deployment flexibility.
2.2 ASYMMETRIC RETRIEVAL
In recent times, the adoption of expansive deep models has greatly improved the multi-mode com- prehension ability Jia et al. (2021): Li et al. (2021 2022 2023); |Radford et al. (202T). However,
2

---

Published as a conference paper at ICLR 2024
Training framework
Adaptive token learner
A= f0
Adgetive soken k
Add & noem Rod frevent
LAR loos
Zest
1
1
larga Test Enosskr
Light Eacoder
Matching Encoder
Encealer
1
t
0 vt
Add
photo
(M
Auguaii
Figure 1: Left: our training framework. Large visual encoder, large text encoder and matching encoder of pre-trained foundation model are frozen during training. The framework is trained with global contrastive distillation (GCD) and local alignment regularization (LAR). Right: adaptive token learner. Adaptive token learner learns to adaptively select salient visual patterns and map them to a sentence in the word embedding space of text encoder with attention mechanism.
large pre-trained models usually consume more resources computing and memory, which is not desirable for some resource-constrainee scenarios such image retrieval. To tackle this inconve- nience, recently many works Budnik Avrithis (202I); Wu et al. (2022a): Duggal et al. (202I); I): Wu et al. (2022b) have been proposed to research into asymmetric image retrival, in which the query (user) side adopts a lightweight model such as Howard et al. (2017) 2019); Liu et al. (2023); Mehta & Rastegari (202T); (Sandler et al.] (2018); Tan & Le (2019) while the gallery (server) side takes a large model. To our knowledge, our proposed framework is the first work to adopt asymmetric retrieval in CIR task, and could even outperform its symmetric counterpart.
3 METHOD
The overview of our framework is shown in Figure[] Our framework integrates a pre-trained vision- language (VL) model, comprising large visual encoder and large text encoder. These encoders are specifically designed to align visual and textual modalities within unified feature space. Ad- ditionally, a matching encoder is incorporated ensures fine-grained alignment and interpretation between different modalities. We freeze the pa- rameter of VL model maintain the alignment between visual and textual modality during training. Given an image I, a lightweight visual encoder is applied to extract visual feature map F of augmented I, and new adaptive token learner utilized to map the visual feature map series of sentence tokens U. U is then concatenated with prompt such as "a photo of" to form a full sentence, which is fed into the large text encoder to obtain textual feature Large visual encoder is utilized to extract the image feature v and feature map F" of I. Global contrastive distillation is applicd to align v and t while local alignment regularization is applied on and F".
enhance image-text matching capabilities, which
3.1 LIGHT VISUAL ENCODER AND ADAPTIVE TOKEN LEARNER
As shown in Figure D we apply data augmentation to input image I and feed it into the light visual encoder, to extract visual feature map FLE e R HxW XC It is common understanding that light models are less discriminative compared with the large models. Therefore, we introduce an image
innovative adaptive token learner. Instead of representing conceptualizes the image as a sentence, which significantly enhances visual discrimination within the word embedding space. Each image element is treated an individual narrative component, allowing for richer and more detailed representation in the embedding space. Specifically, we map each pixel fLE of FLE to one of semantic groups using convolutions, and generate L attention a conditioned on the F FLE The spatial attention maps are computed as cxp(w(") f1.E)
single word, method
maps A = {a()) ....)al)))
Ele exp(w( fI.E)
where W = (1) [ E L are the parameters of the L convolutions. Spatial attention applied adaptively discover discriminative visual patterns, and obtain L visual tokens Z [z(1] 2(4)
3

---

Published as a conference paper at ICLR 2024
H.W Σ A=L,e=1
H.W Σ
20
p(a
h
Shie p(a
=
(1) h,
(2)
A=
Since each visual token is learned separately without considering their mutual relationship, self- attention utilized to model the interaction among visual tokens to obtain the refined tokens Z,
Zont = + softmax/ I. (.( (Z,,) (2Q,))
33)
Z, Zout o Z W.i W
where Qr, K,, V, are the parameters of query, key and value, and W11, W2 are the parameters of fewarrorward network with a(-) activation. To obtain the L sentence tokens that involve filtered visual semantics, we adopt the cross-attention to adaptively select the salient visual patterns from the feature map FLE, FLE is first flattened sequence FLE" and the sentence tokens are calculated as
=ff.f. HH.W E "
Ze Z, Zstfmmmxx,, ((FLE' (Z.) ) U = [Z. + (Z.W W2 P:
FLE' Ves
where Qc, Ke, V. are the parameters of query, key and value, and Wal Wa are the parameters of feed-forward network with (-) activation. W, Rxxd= are the parameters of fc layer to project
the output from ferd-forward network to L sentence tokens U (, u( R 1xdd., where
dw the dimension of word embedding space. In sum, the adaptive token learner incorporates the spatial attention mechanism to filter noisy information by selecting more prominent visual patterns, and map them to the word embedding space as a sentence.
3.2 LEARNING WITH PRE-TRAINED FOUNDATION MODEL
Given sentence tokens U, we concatenate them with the predefined prompt to form full sentence, and feed it into the large text encoder obtain the textual feature t. During training, is constrained to be similar to the corresponding image feature v extracted from the large visual encoder within batch B. The normalized image-to-text and text-to-image similarity are calculated as
explufti/t Σ exply (//T)² JEB
exp(t? v./T)
pd
12T
p T21
(5)
Σ exp(t; u,/T)²
JEB
where t denotes the temperature parameter. Suppose 12T T21 denote the ground-truth one-hot similarity, the global contrastive distillation is computed as the cross-entropy H between p and y
LOOD = 2 Es (p 127 H(p"21 y T2
y
(6)
To further enhance the semantic alignment of sentence tokens with visual information, local align- ment regularization is applied to harness sentence tokens to be more like the caption that faithfully describes the image content in the word embedding space. Suppose that the word embedding space is shared across the text encoder and matching encoder, therefore the local alignment regularization will help to generate better sentence tokens, and improve the global semantic alignment between and v. Specifically, we feed sentence tokens together with visual feature map F" into the matching encoder to obtain the probability pLAR The local alignment regularization is calculated as
LLAR = Ep LAR y L.AR)),
(7)
where y LAR denotes the binary one-hot vector that represents the paired label. Therefore, the final loss could be computed as
C LoOD + CLAR-
(8)
With the knowledge from the pre-trained foundation model, our framework could acquire composi- tion ability by training unlabeled images.
4

---

Published as a conference paper at ICLR 2024
Feetback
Image + Text
User
Large Enceder
on grass and down on
photo
Upload
lying
that is grass and has one gat lving down
tokon lonee
image
Large Visual Encoder
Adaptive
Mobile device
Cloud
Light Encoder
server
The inference workflow of our framework. The light encoder and the adaptive token learner are deployed on mobile device, to extract sentence tokens from the query image and con- catenate them with the prompt and text modifier as the composed query. Pre-trained foundation model deployed on the cloud server, and the large text encoder receives the uploaded composed query to extract the text feature for retrieval, and the large visual encoder is used to extract features of database images. The retrieval results are returned the user side.
Figure 2:
3.3 INFERENCE
In practical application scenarios, the memory and computing resources should be included in our consideration. Therefore, as shown in Figure the lightweight visual encoder and the adaptive token learner are deployed on mobile devices, while the large visual and text encoder are deployed on the server side. During the inference phase, the The system then converts this query image into sentence tokens, a process that encapsulates the visual content into textual format. This conversion is pivotal for the subsequent retrieval process, as it enables the system to interpret and process the visual data in a linguistically contextualized manner. Specifically,
inputs query image along with text modifier.
concatenate the sentence tokens and pre-defined prompts together with text modifier as the composed query: [prompt] U1 " uz that [text modifier]. It uploaded to the cloud server retrieve the images from the database. The large text encoder is utilized extract the composed query feature during online inference, and the large visual encoder is only utilized to extract image features from the database offline.
EXPERIMENTS
In this section, we conduct series of experiments evaluate our ISA on the task of zero-shot composed image retrieval. Three datasets are utilized to verify the effectiveness of our method, including CIRR Liu et (202I), FashionlQ Wu et (202T) and CIRCO Baldrati et al. (2033a).
Liu
4.1 IMPLEMENTATION DETAIL.
We adopt the BLIP model Li et al. 2022 foundation model. CC3M Sharma et al. Since BLIP visual encoder is frozen during training, image feature v and visual feature map Fe could be ex- tracted and stored in advance. For the adaptive token learner, the token length L is set as 6, and two hidden dimensions of feed-forward are set as 256 and 512, respectively. AdamW optimizer with 3e-4 learning rate is adopted, and the framework is trained for 20 epochs with 5 epochs of linear warm-up and 15 epochs of cosine nealing. The proposed method is implemented on open source Pytorch framework on a server with 4 NVIDIA GeForce RTX 3090 GPU, with the batch size 320.
pre-trained on 129M image-text pairs the multi-modal 2018) is adopted as the training set in our framework.
Query model
Gallery model
FLOPS(G)
PARAM(M)
ABS
%
ABS
%
BLIP VE
BLIP VE
16.86
100.0
87.0
100.0
ElicientNet BO
0.43
2.6
5.3
6.1
EfficientNet B2
EllicienVVT M2
BLIP VE
0.21
5.7
MobileNet V2
0.35
2.1
3.5
4.0
MobileVit
1.42
5.5
Table 1: FLOPS and parameters of different light models
with adaptive token learner in our work, absolute (ABS) and relative (%) to the BLIP visual encoder (BLIP VE).
5

---

Published as a conference paper at ICLR 2024
We first compare with three simple baselines which directly extract the visual and text features using large foundation model: (i) Image-only, using only visual features for both query and database; (ii) Text-only, using textual features for query and visual features for database; (iii) Image + Text, simply merging the visual and textual features with addition for query. while using the visual features for database. For the state-of-the-art ZSCIR methods, Pic2word (Saioo al. (2d SEARLE Baldrati et al. (2023a) are included. Since Pic2word and SEARLE are all based on ViT-L/14 CLIP model, we re-implement their methods on BLIP with the open-source codes for fair comparison. Two lightweight models are mainly used as the lightweight visual encoder for our ISA: EfficientNet B2 & Le (2019) and Efficient M2 et al. (2023), as representatives of two main categories of deep architectures (CNN and transformer). Since CLIP models do not include the pre-trained matching encoder that share the word embedding space with the text encoder, we only report our results based on BLIP. We also compare ISA with symmetric baseline, i.e., frozen BLIP visual encoder is adopted to obtain the visual feature map, followed by the same adaptive token learner to obtain the sentence tokens.
Recall
Gallery model
Query model
Method
Avg
R
R@5
R 10
R@
CLIP VE
Image-only Text-only
7,40
23.60
34,00
57.40
30.60
20.90
44.80
56.70
79.10
50.36
CLIP
VIT-L/14
CLIP VE
Image + Text
12.40
36.20
49.10
78.20
43.97
CLIP VE
Pic2 Saito et 2023
23.90
51.70
65.30
87.80
57.17
CLIP VE
SEARLE Baldrat al.
2025a
24.22
52.41
66.29
88.63
57.88
BLIP VE
Image-only Text-oni only
7.52
21.83
31.21
$2,94
28.37
26.48
50.46
61.22
82,26
55,10
BLIP VE
Image + Text
8.19
24.19
34.41
57.81
31.15
BLIP
BLIP VE
Pic2word Saito al.
2023
26.70
53.16
64.10
84.36
57.08
BLIP VE
SEARLE Baldran at.
2025a
29.27
54.86
66.57
86.16
59.21
BLIP VE:
Ours(Sym) Ours(Asy) Ours(Asy)
29.68
58.72
70.79
90.33
62,38
EfficientNet B2
30.84
61.06
73.57
92.43
64.48
ViT M2
29.63
58.99
71.37
91.47
62.87
Table 2: Results of zero-shot composed image retrieval on CIRR test set. The best result for each column is shown in bold fonts. Sym: use BLIP visual models to both extract features for query and database images. Asy: use different visual models to extract features for query and database images.
Gallery model
Query model
Method
mAP@ K
Avg
mAP@5
mAP@10
mAP@25
CLIP VE
Image-only Text-only
1.69
2.18
2.78
3.22
2.47
2.67
3.38
3.69
VIT-L/14
CLIP
CLIP VE
Image Text
3.60
4.40
5.59
6.21
4.95
CLIP
Pic Word Salto
8.43
9.11
10.15
10.73
CLIP VE
SEARLE Baldrah et al
2021a
10.18
11.03
12.72
13.67
11.90
BLIP VE
Image-only Text-only
1.28
1.48
1.88
2.13
1.69
4.01
4.37
4,79
5.13
4.58
BLIP VR
Image Text
1.39
1.72
2.22
2.51
1.96
BLIP
PicZword Saito
8.69
9,36
BLIP VR
SEARL .E Rainrrl ot ml 20Z1A
10.65
11.34
12.40
13.02
11.85
BLIP
Ours(Sym)
EfficientNet B2
Ours(Asy)
11.33
12.25
13:42
13.97
12.74
EllicientVi
Ours(Asy)
9.82
11.61
12.09
Table 3: Results of zero-shot composed image retrieval on CIRCO test set. The best result for each column is shown in bold fonts. Sym: use BLIP visual models to both extract features for query and database images. Asy: use different visual models extract features for query and database images.
4.2 COMPARISON WITH THE SFATE-OF-THEE-AR METHODS
We compare different methods on the test set of CIRR and CIRCO while on the validation set of FashionIQ, and the final results are computed as the average of 3 runs on different random seeds for training. As shown in Table I the lightweight models are more efficient in parameters and computations
computational complexity compared with BLIP visual encoder. The retrieval performance on three
respectively. Our method could outperform simple baselines (Image-only, Text-only and Image Text) by a large margin. From the results of Pic2word and SEARLE, we find that the methods based on BLIP are generally better than those based on CLIP for ZSCIR. Our EfficientNet B2-based ISA could outperform Pic2word and SEARLE on three benchmark datasets, while EfficientViT M2-based ISA achieves better results on FashionIQ and CIRR and is comparable with SEARLE on CIRCO. Compared with our symmetric retrieval in
benchmark datasets is reported Table 2 3

---

Published as a conference paper at ICLR 2024
Gallery model
Query model
Method
Dress
Shirt
Top&tee
Average
Ro:10
R.530
KorT0
K.050
10550
CLIP VE
Inisge-nnly
5,40
13.90
9.90
20.80
5.30
17.70
7.90
17.50
Test-only
13.60
29.70
18.90
31.80
19.30
37.00
17.30
32,90
CLIP VIT-L/M4
CLIP VE
Image Text
16.30
33.60
21.00
34.50
22.20
39.00
19.80
35,70
CLIP VE
Plr2Word Satto St
2023
20.00
40120
26.20
43.60
27.90
47.40
24.70
43.70
CLIP VE
SEARLE Batratr ct
20334
20.32
43,18
27.43
45.68
29.32
50.17
25,69
46.34
BLIP VE
Imaage-nlyy Text-only
4.07
11.35
7.07
16.39
6.88
13.67
6.01
13.67
35.05
BLIP VE
Image + Test
5.16
13.58
8.15
18.89
7.50
16.62
7.03
16.36
BLIP
BLIP VE
BLIP VE
SEARLE BAARLEBBaaaee Baldral ot al.
Pic2word Salto
20235
21.35
22.11
41.79
12.68
27.51
29.72
46.01
48.53
31.03
29.12
52.57
27.62
47.56
BLIP
OursiSym)
24.69
30.79
50.08
33.91
29.79
49.19
EfficientNet B2
Ours(Asy)
25.33
46.26
30.03
48.58
33.45
53.80
29.60
49,54
EfficientViT
Ours(Asy)
25.48
45.51
48.68
32.94
54.31
29.35
Table Results of zero-shot composed image retrieval on FashionIQ validation set. The best result for each column is shown in bold fonts. Sym: use BLIP visual model to both extract features for query and database. Asy: use different visual models to extract features for query and database. Ours (Sym), our ISA could achieve superior performance, which could be attributed to ISA con- taining more training parameters compared with Ours (Sym) (lightweight model adaptive token learner v.s. adaptive token learner) during training. Meanwhile, ISA consumes much less compu- tation resources during online retrieval and is more flexible deploy for real application scenarios, demonstrating the effectiveness of our framework.
4.3 ABLATION STUDY
For ablation study, we mainly adopt the validation set CIRR and CIRCO for evaluation, and the results for FashionlQ included in the appendix.
4.3.1 TRAINING WITH DIFFERENT MAPPING NETWORK AND NUMBER OF TOKENS
To verify the effectiveness of our adaptive token learner, we compare it with variant that adopts ML.P as the mapping network, which maps the image feature to single word following Baldrati et al. (2023a): Saito et al. (202)). The results are shown in Table which demonstrate the consistent superiority for both EfficientNet B2 and EfficientViT M2. These results prove the effectiveness of our adaptive token learner, which maps the visual feature map to multiple sentence tokens per image, and helps to adaptively select more discriminative visual patterns from the query image to generate more informative sentence tokens. It is notable that the variant MLP (single token) could be seen as Pic2word with LAR and outperform Pic2word, thus proving the effectiveness of LAR.
Query model
Mapping network
R81
RG5
CIRR
R810
RG500
Avg.
mAP@5
mAP@ 10
CIRCO
mAP@ 25
mAP@ 50
Avg
EficientNett B2
ATL
31.48
63.42
77.27
93.16
66.33
13.19
13.83
15,20
15.85
14.52
Elicienteet B2
MLP (single token)
30.18
59.46
71.30
89.33
62.57
8.28
8.67
9.62
10.00
9,14
EllicietVV M2
ATL
31.26
62.31
75.70
92.38
65.41
12.24
12.75
13,80
14.27
13.27
EfficienVVI M2
MLP (single token)
30.32
60.25
71.49
89.57
62.90
7.19
7.73
8.47
9.09
8.12
Table 5: Results of different mapping networks with BLIP on CIRR and CIRCO validation set. ATL denotes the proposed adaptive token learner. The best results are shown bold fonts.
For our adaptive token learner, show the performance of our method with different token lengths L on three datasets in Figure 3 We average the recall or precision values for each dataset to obtain the overall evaluation criterion. We find that the average performance improves with the increase of token length for both lightweight visual encoders, however excessive large token length leads to performance degradation. We visualize the attention map of adaptive token learner and the retrieval results in Figure 4 and discover that with excessive large token lengths, some sentence tokens may tend to focus on the background or the trivial patterns, which would likely introduce noise to degrade the retrieval performance.
4.3.2 TRAIN WITH DIFFERENT QUERY MODELS
In order to investigate the influence of different lightweight encoders, we conduct experiments with different light visual models and pre-training parameters. shown in Table 6 We could that EfficientNet B2 shows the best overall performance in CNN, while Efficient ViT M2 is proved to be the best in transformer. This could be attributed to the superior network design and larger capacity of
7

---

Published as a conference paper at ICLR 2024
FashionIQ
CIRR
CIRCO
16
Avernge sereees
Arcine ScOrES
66
Avenge soores
38
36
62
34
loken length senience
loken length of sentence
loken length of senience
Figure 3: The average and standard deviation of performance scores three validation sets.
attention map
L= attention map
Retrieval results
Retrieval results
has dog a different breed and shows jolly roger
is shot with the same style and most of the benches are empty
10, attention map
10, attention map
Retricval results
Retrieval results
Figure 4: Attention map of sentence tokens for different token lengths and retrieval results. Query image and text modifier the left side. Ground-truth target images marked in green boxes.
parameters. Moreover, better initialization weights, i.e., adopting weights pre-trained on ImageNet Deng et al. (2009). could obtain better performance compared with random initialization. This reveals the possibility of adopting different lightweight models to flexibly meet various efficiency and retrieval accuracy requirements, and improve the performance with better pre-trained weights.
CIRR
CIRCO
Query model
Avg
Avg
R
R@P
R@ 10
6
mAP@5
mAP@
mAP@25
mAP@50
EfficientNet BO
32.02
64.57
76.12
92.29
66.25
11.11
12.04
13.14
13.67
12.49
EfficientNet Bn*
27.82
58.38
71.16
90.07
61.86
9.5
9.96
10.6
11.02
10.27
EfficientNet
31.48
63.42
77.27
93.16
66.33
15.85
EffcieniNet B2*
29.47
60.13
74.05
91.27
63,73
11.16
11.57
12.56
13.13
12.11
EMticietViT M2
31.26
62.31
75.70
92.38
65.41
12.24
12.75
13.80
14.27
13.27
EfficientViT
28.56
59.46
71.11
89.38
62.13
10.19
10.79
11.48
11.94
11.10
MobileNet V2
74.27
12.09
MobileNet V2*
29.47
60. 14
72
90.07
62.95
8.55
9.16
10.04
10.39
9.54
MobileVTT V2
27.63
58.66
70.85
88.68
61.46
9.44
9.68
10.49
10.96
10.14
V2*
9.54
Table 6: Results different query models with BLIP on CIRR/CIRCO validation set. * denotes adopting random initialization for light query model. The best results are shown in bold fonts.
4.3.3 SYMMETRIC AND ASYMMETRIC RETRIEVAL
To verify the effectiveness of asymmetric framework, we conduct additional experiments on sym- metric setting, which adopts the same model extract both query and database features. Specif- ically, for the symmetric retrieval in the second, 4-th and 6-th rows of Table ? query and gallery models are both updated during training. Therefore, the network should both learn to generate the image feature, and map the image to the sentence tokens in the word embedding space. We find that they would drastically degrade the performance. This could be attributed to the confusion of multi- task learning for image feature and sentence tokens, since image feature space and word embedding space are heterogeneous semantic spaces. Moreover, we also carry out additional experiments that adopting BLIP VE as both query model and gallery model, and only query model is updated during
8

---

Published as a conference paper at ICLR 2024
training avoid the multi-task confusion. Therefore, BLIP VE is only trained generate sentence tokens. As revealed in the result of BLIP VE i. there is no breaking down for learning. however is still outperformed by EfficientNet B2- b2-based ISA. This could be explained by the fact that BLIP VE is much heavier than light models as shown in T and could be trained only with reduced hatch size, thus degrading the effectiveness of contrastive and LAR loss. The above results reveal the importance of pre-trained alignment for different modalities in ZSCIR, but the BLIP visual encoder could be replaced with lightweight model for efficiency.
Query model
Gallery model
CIRR
Avg
CIRCO
Avg
R@11
R@5
R 10
R@:50
mAP@5
mAP@11 10
mAP@
mAP@50
EfficientNet B2
BLIP VF.
31.48
63.42
77.27
93.16
66.33
13.19
13.83
15:20
15,85
14.52
Efficienteet
EfficientNet B2
0.61
0.74
1.02
3.97
0.00
0.02
0.08
0.06
ViTicien M2
BLIP VE
31.26
62.31
75.70
92.38
65.41
12.24
12.75
13,80
14.27
13.27
Efficien/VTT M2
EfficienVVT M2
0.29
0.37
0.79
2.68
1,03
0.00
0.01
0.04
0.11
0.04
BLIP VE 1
BLIP VE
30.93
62.09
74.67
92.90
65.15
12.60
13:20
14.56
15.07
13.86
BLIP
BLIP
0.49
0.02
0.03
0.08
0.07
Table 7: Comparison with symmetrie retrieval on the validation set of CIRR and CIRCO, BLIP VE denotes the BLIP visual encoder. t denotes updating the query BLIP VE while preserving the pre-trained BLIP VE in gallery side during training for maintaining the multi-modality alignment. The best results are shown in bold fonts.
4.3.4 LEARNING WITH DIFFERENT LOSS CONSTRAINTS
In our framework, we mainly apply global contrastive distillation (GCD) and local alignment regu- larization (LAR) for learning. We verify the losses by dropping each of them separately. To further validate LAR, we compare it with the text regularization, i.e., LLM TR proposed in Baldrati et al. 2023a). Specifically, LLM TR is realized following the pipeline that utilizes a classification model to label each training image and an LLM to do sentence- sentence-making. A cosine loss is applied to constrain the sentence tokens to be similar the L.-M-made sentence text regularization. As reported in Ta- it is obvious that both losses are crucial for training, since dropping either of them would lead to consistent worse performance. Text regularization from LLM-made sentences would be helpful but only restricted to CIRCO, however it needs extra classification model and LLM model which leads to a much more complex pipeline. Our LAR is simple but more effective on three benchmarks.
ble [
Query model
Variants
CIRR
CTRCO
Avg
R@1
Rep
R@:0
RP:50
MAPo3
mAP@10
mAPD25
MAPP530
GCD + LAR
31,48
63.42
77.27
93.16
66.33
13.19
13,83
15.20
15.85
14.52
EfficientNet H2
GCD
29.36
60.18
73,90
91.13
63.77
10.16
10.98
11.88
12.41
11.43
LAR
27.02
56.67
68.91
88.18
60.20
7.56
7.68
8.45
8.94
8.16
GCD + -LM TR
29.43
60.18
73.21
91.32
63.54
11.07
11.35
12.59
13:17
12.05
31.26
62.31
75.70
92.38
65.41
12.75
13,80
14:27
13.27
EfficientVTT M2
GCD
26.87
55.58
69.00
89.38
60.01
9.48
9.84
10.95
11.38
10.41
LAR
24.35
50.49
84.00
55.53
6.00
6.50
GCD L.M TR
27.61
56.49
70.03
89.76
60.97
10.18
10.69
11.48
12.04
11,10
Table 8: Results of different losses on CIRR and CIRCO validation set. GCD: global contrastive dis- tillation; LAR: local alignment regularization; LLM TR: text regularization with the sentences made by LLM. The best and second-best results are highlighted in bold and underlined fonts, respectively.
CONCLUSION
In this paper, we focus on new but more practical scenarios of asymmetric zero-shot composed im- age retrieval, which eliminates the requirement of meticulously--abeled training triplets and is flex- ible for deployment on a resource-constraing platform. Specifically. lightweight model is adopted for the query side while large visual encoder of foundation model is adopted for the gallery side. Adaptive token learner is proposed as the semantic filter to automatically select discriminative se- mantic groups from image and map them to sentence, maintaining the fidelity of visual semantics within the word embedding space. By directly concatenating the mapped sentence with text modi- fier, we are able to compose the multi-modal query to achieve composed retrieval. Global contrastive distillation is applied to align the light model with large foundation model, and local alignment regularization is further utilized to help the mapped sentence better align with the real caption that faithfully describes the image. Extensive experimental results demonstrate that our framework could both achieve better performance and improve retrieval efficiency with deployment fiexibility.

---

Published as a conference paper at ICLR 2024
6 ACKNOWLEDGEMENT
This work was supported by the National Natural Science Foundation of China under Contract 62102128 and 62021001. It was also supported by the cluste built by MCC Lab of Information Science and Technology Institution of USTC.
REFERENCES
Alberto Baldrati, Marco Bertini, Tiberio Uricchio, and Alberto Del Bimbo. Conditioned image retrieval for fashion using contrastive learning and clip-based features. ACM Multimedia Asia, pp. 1-5. 2021.
Alberto Baldrati, Marco Bertini, Tiberio Uricchio, and Alberto Del Bimbo. Conditioned and com- posed image retrieval combining and partially fine-tuning clip-based features. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4959-4965 2022a.
Alberto Baldrati, Marco Bertini, Tiberio Uricchio, and Alberto Del Bimbo. Effective conditioned and composed image retrieval combining clip-based features. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 21466-221111 21474, 2022b.
Alberto Baldrati, Lorenzo Agnolucci, Marco Bertini, and Alberto Del Bimbo. Zero-shot composed image retrieval with textual inversion. arXiv preprint arXiv:2303. 15247, 2023a.
Alberto Baldrati, Marco Bertini, Tiberio Uricchio, and Alberto del Bimbo. Composed im- age retrieval using contrastive learning and task-oriented clip-based features. arXiv preprint arXiv:2308. / 1485, 2023b.
Matcusz Budnik and Yannis Avrithis. Asymmetric metric learning for knowledge transfer. In Pro- ceedings of the IEEE/VV Conference on Computer Vision and Pattern Recognition, pp. 8228- 8238, 2021.
Yanbei Chen, Shaogang Gong, and Loris Bazzani. Image search with text feedback by visiolinguistic attention learning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 3001-3011, 2020.
Ginger Delmas, Rafael Rezende, Gabriela Csurka, and Diane Larlus. Artemis: Attention-based retrieval with text-explicit matching and implicit similarity. In International Conference on Learn- ing Representations, 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proceedings the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 248-255, Icee, 2009.
Eric Dodds, Jack Culpepper, and Gaurav Srivastava. Training and challenging models for text- guided fashion image retrieval. arXiv preprint arXiv:2204.11066 2022.
Xiao Dong, Xunlin Zhan, Yangxin Wu, Yunchao Wei, Michael C Kampffmeyer, Xiaoyong Wei, Minlong Lu, Yaowei Wang, and Xiaodan Liang. M5product: Self-harmonized contrastive learn- ing for e-commercial multi-modal pretraining. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 21252-21262, 2022.
Rahul Duggal, Hao Zhou, Shuo Yang, Yuanjun Xiong, Wei Xia, Zhuowen Tu, and Stefano Soatto. Compatibility-awar heterogeneous visual search. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10723-10732, 2021.
Niloofar Gheissari, Thomas B Sebastian, and Richard Hartley. Person reidentification using spa- tiotemporal appearance. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, volume 2, pp.
IEEF, 2006.
Geonmo Gu, Sanghyuk Chun, Wonjae Kim, HeeJae Jun, Yoohoon Kang, and Sangdoo Yun. Compodiff: Versatile composed image retrieval with latent diffusion. arXiv preprint arXiv:2303.11911 2023.
10

---

Published as a conference paper at ICLR 2024
Xiao Han, Licheng Yu, Xiatian Zhu, Li Zhang, Yi-Zhe Song, and Tao Xiang. Fashionvil: Fashion- focused vision-and-languagg representation learning. In Proceedings of the European Conference Computer Viston, pp. 634 651. Springer, 2022.
Xiao Han, Xiatian Zhu, Licheng Yu, Li Zhang, Yi-Zhe Song, and Tao Xiang. vile-vil: Multi- the IEEE/CVF
tasking Aision-languag model for heterogeneous fashion tasks. In Proceedings the Conference on Computer Vision and Puttern Recognition, pp. 2669-2680, 2023.
Andrew Howard, Mark Sandler, Grace Chu, Liang-Chich Chich Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukum Zhu, Ruoming Pang, Vijay Vasudevan, et al. Searching for mobilenetv3. In Pro- ceedings of the IEEE/CVF International Conference Computer Vision, pp. 1314-1324, 2019.
Andrew Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andrectto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: 1704.0486, 2017.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning, pp. 4904 4916. PMLR, 2021.
Seungmin Lee, Dongwan Kim, and Bohyung Han. Cosmo: Content-style modulation for image retrieval with text feedback. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 802-812, 2021.
Matan Levy, Rami Ben-Ari, Nir Darshan, and Dani Lischinski. Data roaming and early fusion for composed image retrieval. arXiv preprint (xX1:22333.04299, 2023.
Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven
Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. Advances in Neural Information Processing Systems, 34:9694 9705, 2021.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre- training for unified vision-language understanding and generation. In International Conference Machine Learning, pp. 12888-12900. PMLR, 2022.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. arXiv preprint 1rXiv:2301.12597 2023.
Victor Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Y Zou. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. Ad- vances in Neural Information Processing Systems, 35: 15:17612-17625. 2022.
Xinyu Liu, Houwen Peng. Ningxin Zheng, Yuqing Yang, Han Hu, and Yixuan Yuan. Efficientvit: Memory efficient vision transformer with cascaded group attention. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14420-14430, 2023.
Zheyuan Liu, Cristian Rodriguez-Opazo. Damien Teney, and Stephen Gould. Image retrieval on real-life images with pre-trained vision-and-language models. In Proceedings of the ЛEEF/CVF International Conference Computer Vision, pp. 2125-2134, 2021.
Ziwei Liu, Ping Luo, Shi Qiu, Xiaogang Wang. and Xiaoou Tang. Deepfashion: Powering robust clothes recognition and retrieval with rich annotations. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1096-1104, 2016.
Sachin Mehta and Mohammad Rastegari. Mobilevit: weigh, general purposs,, and mobile- friendly vision transformer. arXiv preprint 1rXiv:21 178, 2021.
Filip Radenović, Giorgos Tolias, and Ondrej Chum. Fine-tuning enn image retrieval with no human annotation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(7):155--1688 2018.
11

---

Published as a conference paper at ICLR 2024
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference Machine Learning, pp. 8748-8763. PMLR, 2021.
Kuniaki Saito, Kihyuk Sohn, Xiang Zhang. Chun-Liang Li, Chen-Yu Yu Lee, Kate Saenko, and Tomas Pfister. Pic2word: Mapping pictures to words for zero-shot composed image retrieval. In Pmo- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19305- 19314, 2023.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chiel Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4510-4520 2018.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pp. 2556-223 2565, 2018
Mingxing Tan and Quoe Le. Efficientnet: Rethinking model scaling for convolutional neural net- works. In International Conference on Machine Learning, pp. 6105-61 61 14. PMLR, 2019.
Nam Vo, Lu Jiang, Chen Sun, Kevin Murphy, Li-Jia Li, Li Fei-Fei, and James Hays. Composing text and image for image retrieval-an empirical odyssey. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6439-6448, 2019.
Hui Wu, Yupeng Gao, Xiaoxiao Guo, Ziad Al-Halah, Steven Rennie, Kristen Grauman, and Roge- rio Feris. Fashion iq: A new dataset towards retrieving images by natural language feedback. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp-
11307-11314 -1377, 2021.
Hui Wa, Min Wang, Wengang Zhou, and Houqiang Li. A general rank preserving framework for asymmetric image retrieval. The Eleventh International Conference on Learning Representa- tions, 2022a.
Hui Wu, Min Wang. Wengang Zhou, Houqiang Li, and Qi Tian. Contextual similarity distillation for asymmetric image retrieval. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp.
2022b.
Xunlin Zhan, Yangxin Wu, Xiao Dong, Yunchao Wei, Minlong Lu, Yichi Zhang, Hang Xu, and Xi- aodan Liang. ProductIm: m: Towards weakly supervised instance-level product retrieval via cross- modal pretraining. In Proceedings of the IEEE/CVF International Conference on Computer Vi- sion, pp. 11782-11791, 2021.
Liang Zheng, Yi Yang, and Alexander G Hauptmann. Person re-identification: Past, present and future. arXiv preprint arXiv:1610.02900 2016.
Wengang Zhou, Yijuan Lu, Houqiang Li, Yibing Song, and Qi Tian. Spatial coding for large scale partial duplicaee web image search. In Proceedings of the 18th ACM International Conference on Multimedia, pp. 511-520, 2010.
A APPENDIX
In the appendix, we first present detailed descriptions for the three evaluation datasets.
FashionIQ Wu et al. (002 is fashion domain dataset, and it includes 77684 fashion images which are divided into three different categories: Dress, Toptee, and Shirt. The training set comprises
18000 triplets. and in total 46609 images. The validation set includes 15537 images and 6017 triplets. Each query image in the validation set has only one target image. It also has a test set but without ground-truth labels, thus the test set is not used for evaluation.
12

---

Published as conference paper at ICLR 2024
CIRR Liu et al. 2022 is dataset of natural domain, and includes 21552 real-life images. It involves 36554 triplets, and are randomly assigned in 80% for training, 10% for validation and 10% for test. Each query image in the validation and test set has only target image.
CIRCO Baldrati et al. (2023a) is collected from MSCOCO and consists of a validation set and test set, which includes 220 and 800 query images, respectively. Each query image has text modifier and more than one target images. The database involves 120k images.
We report the comparison with the state-of-the totte-of-th-aatt methods CIRR and CIRCO validation sets in Table A and 10] respectively. Accompanied with Table[land{] our method demonstrates consistent superiority over the simple baselines (Image-only; Text-only and Image Text) and the state-of-the- art methods of Pic2word and SEARLE.
Recall K
Gallery model
Query model
Method
Average
R@1
R@5
R@
R@50
Image-only
7.06
25.52
35.64
60.18
32.10
Text-only
21.19
46.14
57.43
78.74
50.88
CLIP VIT-1114
Image Text
10.79
34.37
47.05
75.27
41.87
Pic2Word
22.54
51.69
64.76
86.19
56,30
SEARLE
25.07
55.20
68.52
90.24
59.76
Image-only
6.72
23.13
32.82
55.66
29.58
Text-only
25.38
51.02
61.76
81.68
54.96
Image Text
7.58
25.90
36.28
61.13
32.72
Pic2word
27.89
58.15
69.42
87.53
60.75
BLIP
SEARLE
28.03
60.74
71.41
88.82
62.25
Ours(Sym)
30.21
61.52
74.09
90.81
64.16
EfficientNet B2
Ours(Asy)
31.48
63.42
77.27
93.16
66.33
EfficientViT M2
Ours(Asy)
31.26
62.31
75.70
92.38
65.41
Table 9: Results of zero-shot composed image retrieval on CIRR validation set.
Gallery model
Query model
Method
Average
mAP@5
mAP@ 10
m5P@25
mAP@50
Image-only
1.85
2.24
3.11
3.57
2.69
Text-only
2.78
3.06
3.72
3.95
3.38
CLIP VIT-L/14
Image Text
4.71
5.74
6.88
7.44
6.19
Pic2Word
7.50
8.31
9,45
10.11
8.84
SEARLE
11.02
12.12
13.62
14.43
12.80
Image-only
1.86
2.18
2.71
2.97
2,43
Text-only
5.32
5.76
6.29
5.85
Image Text
2.04
2.63
3.18
3.49
2.84
Pic2Word
7.91
8.27
9,05
9.56
8.70
BLIP
SEARLE
11.52
11.74
12.91
13.56
12.43
Ours(Sym)
11.03
13.17
13.81
12.51
EfficientNet B2 EfficientViT M2
Ours(Asy)
13.19
13.83
15.20
15.85
14.52
Ours(Asy)
12.24
12.75
13.80
14.27
13.27
Table 10: Results of zero-shot composed image retrieval on CIRCO validation set.
We also report the results of ablation study FashionlQ dataset.
Table II shows adopting different mapping networks during training, which demonstrates consistent superiority for both EfficientNet B2 and EfficientViT M2.
Table 12 shows utilizing different query models for ZSCIR, which shows the superiority of lightweight models with pre-training initialization.
Table 13 shows the comparison of symmetric and asymmetric retrieval on FashionIQ dataset, and shows consistent superiority of our asymmetric structure.
13

---

Published as a conference paper at ICLR 2024
Dress
Shirt
Top&tee
Average
Query model
Mapping network
R@ 10
R@:00
R@ 10
R@
6 10
R@
R 10
EfficientNer B2
transformer
25.33
46.26
30.03
48.58
33.45
53.80
29.60
49.54
AfficientNer B2
MI.P (single token)
23.65
43.83
28.46
47.74
31.16
51.96
27.76
47.84
Vif
transfomner
25.48
45.51
29.64
48.68
32.94
54.31
29.35
49.50
EfficientViT M2
MLP (single token)
24.39
44.27
29.05
47.45
30.75
53.19
28.06
48.30
Results of different mapping networks with BLIP on FashionIQ validation set. The best results are shown in bold fonts.
Table 11:
Dress
Shirt
Top&tee
Average
Query model
R@
10
R@50
@@@ 10
R@50
R@10
R@ 50
R @ 10
R@
EfficientNet BO
25.19
45.56
28.95
48.04
32.53
52.78
28.89
48.79
EfficientNet BO*
23.05
44.17
27.43
45.14
29.98
51.76
26.82
47.02
EfficientNet B2
25.33
46.26
30.03
48.58
33.45
53.80
29.60
49.54
EfficientNet B2*
25.04
44.22
28.31
46.76
31.26
53.24
28.20
48.07
Vff M2
25.48
45.51
29.64
48.68
32.94
54.31
29.35
49.50
EfficientViT M2
24.14
43.63
27.63
45.58
31.72
53.24
27.83
47.48
MobileNet V2
24.34
45.51
28.95
47.35
32.02
54.11
28.44
48.99
MobileNet V2*
24.59
44.47
27.48
45.78
31.51
53.19
27.86
47.81
Vob V2
22.16
40.46
26.35
43.23
27.84
49.01
25.45
44.23
Mobile VIT V2"
21.52
39.41
25.66
42.89
27.54
48.70
24.91
43.67
Table 12:
Results of different query models with BLIP FashionIQ validation set. denotes adopting random initialization for light query model. The best results are shown bold fonts.
Table 14 demonstrates the influence of different losses on FashionIQ, which further verifies the effectiveness of GCD and LAR losses.
To better understand the performance of zero-shot methods, we compare our methods with the
pervised methods that adopt the labeled-triplets for training, including CIRPLANT et al. (202T). OCCEMIS/Delmal al. (00T)) and ELIP4CirBaldaaal et al.) (20226). We report the results on CIRR and fashionIQ, Since CIRCO does not include training set so that there is no supervised results for it. From Table 15 and T6 our method could surpass CIRPLANT and ALTEMIS, which shows the effectiveness of leveraging pre-trained BLIP model for zero-shot learning. Although there is still gap compared with CLIP4Cir, our method is getting closer with the supervised state-of-the-art method in contrast to previous zero-shot methods, while achieving deployment flexibility with the asymmetric framework.
We visualize the results of the attention map of sentence tokens with the text modifier, and to com- pare them with the spatial attention of adaptive token learner together with the retrieval results. Some good retrieval results shown in Figure 5 We could see that there some tokens that attend to both concepts of query images and concepts of text modifier. In the up-right example, the third and
Dress
Shirt
Toptee
Average
Query model
Gallery model
R@
R@050
@@:
R@500
R
R@
R@
R@500
EfficientNet
BLIP VE
25.33
46.26
30.03
48.58
33.45
53.80
29.60
49.54
EfficientNet
BLIP VE
2.08
0.64
2.01
1.63
0.67
EfficientViT M2
BLIP VE
25.48
45.51
29.64
48.68
32.94
54.31
29.35
49.50
EfficientViT M2
BLIP VE
0.69
1.59
1.08
2.16
1.22
2.04
1.00
BLIP VE
BLIP VE
24.89
45.36
30.18
49.07
35.65
55.12
30.24
49.85
BLIP VE
BLIP VE
2.13
5.7
1.86
4.47
6.12
2.13
5.43
Table 13: Comparison with symmetric retrieval on the validation set of FashionIQ. BLIP VE denotes
the BLIP visual encoder. t denotes updating the query BLIP VE while preserving the pre- trained BLIP VE in gallery side during training for maintaining the multi modalttyy alignment. The best results shown in bold fonts..
14

---

Published as a conference paper at ICLR 2024
Dress
Shirt
Top&tee
Average
Query model
Variants
R@ 10
R@ 50
R@ 10
R 50
R@ 10
R@ 50
R@ 10
30
GCD LAR
25.33
46.26
30.03
48.58
33,45
53.80
29.60
49.54
GCD
23.90
45.66
27.33
45.14
31.31
52.12
27.51
47.64
EfficientNet B2
LAR
22.76
42.69
26.15
46.16
29.68
50.59
26.20
45.81
GCD + LLM TR
24.19
45.17
27.92
44.90
31.06
51.56
27.72
47.21
GCD + LAR
25.48
45.51
29.64
48.68
32.94
54.31
29.35
49.50
GCD
23.35
44.03
26.79
44.80
30.65
50.99
26.93
46.61
EfficientViT M2
LAR
13.73
28.46
21.49
38.13
24.68
43.50
19.97
36.70
GCD LLM TR
23.80
44.03
26.94
45.00
30.49
50.94
27.08
46.66
Table 14: Results of different losses on FashionIQ validation set. GCD: global contrastive distil- lation; LAR: local alignment regularization; LLM TR: text regularization that adopts the sentences made by LLM. The best and second-best results are highlighted in bold and underlined fonts, re- spectively.
Recall K
Supervision
Method
Avg
R@1
R@5
R@ 10
R@
CIRPLANT
19.60
52.60
68.40
92.40
58.25
Supervised
ALTEMIS
16.96
46.10
61.31
87.73
53.03
CLIP4Cir
33.60
65.40
77.40
95.20
67.90
Pic2Word(CLIP)
23.90
51.70
65.30
87.80
57.17
SEARLE(CLIP)
24.22
52.41
66.29
88.63
57.88
Pic2word(BLIP)
26.70
53.16
64.10
84,36
57.08
Zero-shot
SEARLE(BLIP)
29.27
54.86
66.57
86.16
59.21
Ours(Asy, EN B2)
30.84
61.06
73.57
92.43
64.48
Ours(Asy, EVT M2)
29.63
58.99
71.37
91.47
62.87
Table 15: Comparsion with supervised methods on CIRR test set. The best and second best results are shown bold and underline fonts, respectively.
the 6-th tokens focus on "cat" and "fruits" respectively, and the 4-th and 6-th are paying attention to the concepts in the text modifier, ie., "vegetables" and "bench", respectively. These examples could extract distinctive visual concepts and interact with the text modifier correspondingly. thus achieving good retrieval performance.
We also present some failure cases in F The failure retrieval could be attributed the wrong associated concept, rare words, and abstract concepts. For the example in the middle, although the 4-th token catches the concept of "woman" and attends to the "phone", there is difficulty associating "them" in the text modifier with "dog", i.e., the 6-th attends to "them" but fail to catch the concept
Dress
Shirt
Top&tee
Average
Supervision
Method
R@10
R@50
R@10
R@50
R@:10
R@50
R@
R@50
CIRPLANT
17.45
40.41
17.53
38.81
21.64
45.38
18.87
41.53
Supervised
ALTEMIS
27.20
52.40
21.80
43.60
29.20
54.80
26.10
50.30
CLIP4Cir
30.30
54.50
37.20
55.80
39.20
61.30
35.60
57.20
Pic2Word
20.00
40.20
26.20
43.60
27.90
47.40
24.70
43.70
SEARLE(CLIP
20.32
43.18
27.43
45.68
29.32
50.17
25.69
46.34
Pie2word( BLIP))
21.35
42.68
27.51
46.01
29.12
49.33
25.99
46.00
Zero-shot
SEARLE(BLIP)
22.11
41.79
29.72
48.53
31.03
52.37
27.62
47.56
Ours(Asy, EN B2)
25.33
46.26
30.03
48.58
33.45
53,80
29.60
49.54
Ours(Asy, EVT M2)
25.48
45.51
29.64
48.68
32.94
54.31
29.35
49.50
Table 16:
Comparsion with supervised methods on FashionIQ validation set. The best and second best results shown in bold and underline fonts, respectively.
15

---

Published as a conference paper at ICLR 2024
of "dog" in the visual feature map. For the example in the bottom, it is difficult for the model to grasp the abstract concept "pattern" and "less of a V-neck", thus leading to wrong retrieval results.
We present more visualization comparison of different token lengths in Figure 7 and 8 Generally, to achieve accurate retrieval, the pseudo sentence tokens need to accomplish two aspects: extracting effective visual information and interacting with textual information. When the token length is small, there would be drop of performance due to the lack of capacity to extract sufficient visual information. As shown in Figure 7 query model with small token length performs worse even if the attention maps are reasonable. When the token length increases for too long, on one hand, some tokens may focus on trivial and noisy visual patterns, as shown in Figure 4 on the other hand, these tokens may interact incorrectly with the text modifier, and may impact the correct token- text interaction. shown Figure [ the excessive tokens associate the visual pattern with incorrect text information, potentially interfering with the interaction of multi-modal information. Even if there are accurate associations between other sentence tokens and text modifier, this interference would still degrade the retrieval accuracy.
To verify the efficiency of our framework, we report the latency on both query side and cloud side on CIRCO vailidation set shown in Table 17 For query side, we report the inference latency for different query models. We find that lightweight encoders are generally twice faster than large visual encoder (BLIP VE). For cloud side, since the gallery models are the same (BLIP TE & VE), the retrieval latency is the same.
To further verify the effectiveness of adaptive token learner, carry out experiments for other three variants of mapping networks:
(I) simple tokenizer (ST): only using the block described in Eq. and Eq. in section 3.1 in the manuscript.
(II) adaptive token learner without cross-attention (ATL w/o CA): including the blocks described in Eq.
(3.
(III) adaptive token learner without cross-attention (ATL w/o SA): including the blocks described in Eq.
(4)
We report the results on EfficientNet B2 below. From the table, it could be that with more components added to the mapping network, the performance gradually improves on three benchmark datasets. The simple tokenizer divides the semantics in different groups, which preserves more visual information than MLP. The self-attention models the mutual relationship between different visual tokens and cross-attention filters out noisy information, enhancing the pseudo sentence tokens to be more robust and refined visual representations in word embedding space. Therefore, the full version of adaptive token learner (ATL) achieves the best performance.
Query inference
Retrieval latency (ms, per query)
Query model
Gallery model
Total retrieval latency
latency (ms, per query)
EfficientNet
3.07
BLIP VE
3.58
6.65
EfficientNet
3.24
BLIP
3.58
EfficientViT M2
3.27
BLIP VE
3.58
6.85
MobileNet V2
3,03
BLIP VE
3.58
6.61
MobileVIT V2
3,40
BLIP VE
3.58
6.98
BLIP VE
6.53
BLIP VE
3.58
10.11
CLIP VIT-L/14
24.44
CLIP VIT-L/14
3.60
15.38
Table 17: Inference latency on query side and gallery side for different query and gallery models.
16

---

Published as a conference paper at ICLR 2024
Components
Recall @
Mapping network
Avg
ST
SA CA
R
R@5
R 10
R@50
MLP ST
30.18
59.46
71.30
89.33
62.57
29.94
61.68
73.71
91.92
64.31
ATL w/o CA ATI w/ SA ATL
29.85
61.61
74.48
91.53
64.37
30.35
62.13
75.95
91.89
65.08
31.48
63.42
77.27
93.16
Table 18: Ablation study on variants of mapping networks CIRR validation set. Best results are shown bold fonts.
Components
mAP@K
Mapping network
Avg
ST
SA
CA
MAP@
mAP@5
mAP@ 10
mAP@50
MLP ST
8.28
8.67
9.62
10.00
9.14
10.33
11.38
11.79
10.85
ATL CA ATL w/o SA ATT.
10.46
10.94
12.03
12.52
11.49
11.52
12.15
13.83
13.58
12.77
13.19
13,83
15.20
15.85
14.52
Table 19: Ablation study on variants of mapping networks on CIRCO validation set. Best results are shown bold fonts.
Components
Dress
Shirt
Top&tee
Average
Mapping network
ST
SA CA
R@10
R@50
R@ 10
R 50
R@10 10
R@50
R@ 10
R@:50
MLP
23.65
43.83
28.46
47.74
31.16
51.96
27.76
47.84
ST ATL w/o CA ATL w/o ATL
23.45
43.58
28.80
45.78
31.57
52.32
27.94
47.23
24.00
45.86
27.33
45.29
31.62
$2.27
27.65
47.80
24.15
46.71
28.16
46.22
32.46
52.37
28.26
48.43
25.33
46.26
30.03
48.58
33.45
53,80
29.60
49.54
Table 20: Ablation study on variants of mapping networks on FashionIQ. Best results are shown in bold fonts.
17

---

Published as a conference paper at ICLR 2024
has of them and a beer instead
5
of a coke
Bism
boor
has more boxes and they are on the grass
Scaie
has a
#
# 3
bench and
vegetables instead of fruits
black classy dress with white collar and is more formal
Bock cina aay dras with whene color
mone berma
Figure 5: Some good retrieval examples and the attention map of sentence tokens visual feature map and with modifier text. Ground-truth target images are marked in green boxes.
18

---

Published as a conference paper at ICLR 2024
Is redder
and has
more garnishes
edsh
has only
# 6
one of them, the woman talking on the phone
mom
Une women
Is less revealing and has pattern and less of a V- neck
Figure 6: Some failure retrieval case and the attention map of sentence tokens on visual feature map and with modifier text. Ground-truth target images are marked in green boxes.
19

---

Published as a conference paper at ICLR 2024
L= 6, retrieval results & attention map
only one and the photo has a body of water in the background
proto
L= 2, retrieval results & attention map
1000
1000
beskgrrocn
L= 6, retrieval results & attention map
# 6
Is green, has a different shape and is seen from a closer look
green
and
from
ditairce
L= 2, retrieval results & attention map
oreen
und
2000
hom
Figure 7: More comparison results of different token lengths (2 v.s. 6). Query image and text modifier are shown on the left, and the attention map of sentence tokens with text modifiers and on visual feature map are shown the right. Ground-tret -rutt target images are marked in green boxes.
20

---

Published as a conference paper at ICLR 2024
L= 6, retrieval results & attention map
#
No wsth
Has red color and is only one, the photo has rocks in the background
L= 10, retrieval results & attention map
L= 6, retrieval results & attention map
L= 10, retrieval results & attention map
Is green, has different shape and is from closer look
Figure 8: More comparison results of different token lengths (6 v.s. 10). Query image and text modifier are shown the left, and the attention map of sentence tokens with text modifiers and on visual feature map are shown on the right. Ground- -ruth target images are marked in green boxes.
21


Reviews:
[{"review_id": "ljoI7SCM0D", "reviewer": "Reviewer_ZKF9", "summary": "This paper proposes an image2sentence based asymmetric framework for zero-shot composed image retrieval tasks. In particular, a lightweight visual encoder and a consequent adaptive token learner are designed to effectively extract the visual features from query images for the mobile side. By doing so, the learned features could be generated as a good visual prompt as with the text intent for conventional LLM to deal with image retrieval tasks. In addition, a local alignment regularization term is added to further improve the training. The experiments conducted on several benchmark datasets verify the effectiveness of the proposed method compared with existing SOTA ones.", "questions": "Please check the weaknesses above.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper is of good written quality that makes the readers easy to follow. The logic, the notion expression, and the experiments are all very clear.\n\n2. The asymmetric design is interesting and this design has been proven an efficient way to deal with resource-limited circumstances.\n\n3. The experiments conducted are very convincing to support the contribution claimed by the authors. The properties of the proposed method are well demonstrated in the ablation study.", "weaknesses": "1. It could be better to discuss more about the number of token selections in detail. According to Fig 3, it seems the performance is a bit sensitive to the number of tokens used in the proposed method. Then, a more detailed discussion of this observation with a visual example of the same query but a different number of tokens could help to better demonstrate the impact brought by the tokens.\n\n2. It could be good to add a discussion of the relationship between the proposed adaptive token learner and the similar approach used in the following papers [1,2]. They have a similar structure, a discussion would help to better locate the position of the token learned in this work.\n\n [1] Wu, Hui, et al. \"Learning token-based representation for image retrieval.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 36. No. 3. 2022.\n [2] Locatello, Francesco, et al. \"Object-centric learning with slot attention.\" Advances in Neural Information Processing Systems 33 (2020): 11525-11538.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FcDWJQNTXF", "reviewer": "Reviewer_WHDg", "summary": "The paper introduces a novel approach to composed image retrieval (CIR), emphasizing on the challenges associated with composed image retrieval that needs understanding of both visual and textual data. To address data scarcity in CIR, the authors introduce a new task paradigm named \"zero-shot composed image retrieval\" (ZSCIR) that transforms image retrieval to a text-to-image format, allowing for a more intuitive mapping between images and descriptive text. However, the methods presented face challenges with large-scale models which are not suitable for deployment on resource-constrained platforms, such as mobile devices. To mitigate this, the authors propose an asymmetric approach, termed Image2Sentence based Asymmetric ZSCIR, that uses different models for query and database extraction. This method utilizes a lightweight model for the user's device and a heavier model for cloud processing. The core of this approach is an adaptive token learner which converts visual features into textual tokens, thus enhancing the representation. The proposed framework was tested on various benchmarks, demonstrating its efficiency and effectiveness compared to existing state-of-the-art methods.", "questions": "1.  How does the ZSCIR approach compare in performance and efficiency with state-of-the-art image retrieval methods that don't employ a text-to-image asymmetry? Are there scenarios where a traditional symmetric approach might outperform ZSCIR?\n\n2. In terms of training data apart from the image augmentation, did you employ any data augmentation techniques to enhance the performance and robustness of the ZSCIR model? Furthermore, how did you ensure the diversity and representativeness of the descriptive texts used in the system?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper addresses the challenges in Composed Image Retrieval by introducing the zero-shot composed image retrieval (ZSCIR). This method offers a fresh perspective on image retrieval by transforming it to a text-to-image format, thereby providing a more direct linkage between descriptive text and its corresponding image.\n\n2. The introduction of an adaptive token learner, which effectively translates visual features into textual tokens, stands out as a major strength. This conversion mechanism is pivotal in enhancing the representation of images and ensuring that the retrieval process is both accurate and efficient. The adaptive nature of the learner means that it can adjust and improve over time, potentially leading to even better retrieval results in the future.", "weaknesses": "1. While the adaptive token learner is a strength in terms of converting visual features to textual tokens, there's a risk that the system could become overly reliant on this component. If the learner fails or encounters unanticipated scenarios, it might compromise the effectiveness of the entire retrieval process.\n\n2. Introducing an asymmetric text-to-image retrieval approach, while innovative, adds an extra layer of complexity to the system. This might present challenges in terms of maintainability, debugging, and further development of the system.\n\n3. The transformation of the retrieval problem from image-to-image to text-to-image inherently assumes that the descriptive texts are of high quality and detailed. Any inaccuracies or vagueness in the text could lead to inefficient or incorrect image retrievals.\n\n4. The paper presentation is not very attractive. It is difficult to understand the novelties / contributions after reading the introduction of the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BHtq8sUKKo", "reviewer": "Reviewer_K4T5", "summary": "This paper introduces an asymmetric zero-shot composed image retrieval framework. The asymmetric retrieval pipeline is established using a lightweight model for query images and a large foundation model for gallery images, enabling feature extraction. Composed image retrieval is achieved by concatenating the sentence representation mapped from the image with a text modifier. To align the features extracted by the lightweight model and the large foundation model, two techniques, namely global contrastive distillation and local alignment regularization, are proposed. Extensive experiments and an ablation study conducted on benchmark datasets have demonstrated the effectiveness of the proposed method.", "questions": "t-SNE visualizations could be shown to illustrate the differences between the different methods. This would provide a more intuitive understanding of the feature distributions and separability.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. An asymmetric zero-shot composed image retrieval framework is proposed. \n2. Global contrastive distillation and local alignment regularization techniques are proposed to align features from different models.\n3. Extensive experiments are conducted.", "weaknesses": "1. The clarity of the writing, particularly in the methods section, requires improvement.\n\n2. For image-only retrieval, could you provide results using the DINO-V2 and MoCo-V3 pretrained models? The CLIP model is typically used for content matching between image and text features.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CnPMtigEHp", "reviewer": "Reviewer_zR31", "summary": "This paper aims at the task of composed image retrieval (CIR) which retrieves images by providing a multimodal query such as a query image and additional text which describes the user's further query intention. Usually, such task is resolved by aligning the multimodal query and gallery features with vision-and-language pretraining and finetuning techniques. The authors argue that existing methods are not feasible to mobile applications due to expensive computational costs by forwarding large multimodal foundation models on mobile devices. The proposed solution is adopting a lightweight model to process the query while still maintaining the large foundation model for the gallery side. In order to bridge the representation gap between the lightweight and large model, the adaptive token learner is proposed to map an image to a sentence in the language model space. Finally, the authors verify their contributions on three evaluation benchmarks.", "questions": "[1] Figure 1 could be further improved. Specifically, the pink rectangle and trapezoid could be shrunk and the blue trapezoids could be enlarged. As a result, it is much easier to quickly grasp the idea at first glance for readers. \n\n[2] Table 5 provides a viriant mapping network which is actually a MLP. Are there other options?\n\n[3] Figure 3 reveals a fact that larger token lengths could degrade the retrieval performance. The authors attribute such fact to the background noise or the trivial patterns. Are there any deeper insights or visualization analysis?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "[1] Overall, this paper is well written and easy to follow. \n\n[2] I like the motivation of this work since deploying original large foundation model is almost impossible in mobile applications. Different from pruning or distilling such heavy models, the authors proposed the lightweight encoder with a tunable adaptive token learner. The idea behind is borrowed from the LLM-Adapters. \n\n[3] The proposed modules are technically sound and the experimental resutls are sufficient. The training resources are extremly friendly with 4 RTX 3090 GPUs.", "weaknesses": "[1] Since this work is a retrieval task, it is important to report how the retrieval performance varies as the gallery size scales up. I understand that the three evalutation datasets are standard benchmarks. However, it would make the contribution more solid if millions distractors could be involved in the gallery, although most existing SOTAs didn't report such results. \n\n[2] The inference time should be reported including in the query side and in the cloud side.\n\n[3] Some minor issues are listed in the next part.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)