Composed Image Retrieval with Text Feedback via Multi-grained Uncertainty Regularization
Authors: Yiyang Chen
Zhedong Zheng
Wei Ji
Leigang Qu
Tat-Seng Chua
Venue: ICLR 2024 poster (2024)
Keywords: Multi-modal learning;
Primary area: representation learning for computer vision, audio, language, and other modalities

Abstract: We investigate composed image retrieval with text feedback. Users gradually look for the target of interest by moving from coarse to fine-grained feedback.  However, existing methods merely focus on the latter, i.e., fine-grained search, by harnessing positive and negative pairs during training. This pair-based paradigm only considers the one-to-one distance between a pair of specific points, which is not aligned with the one-to-many coarse-grained retrieval process and compromises the recall rate. 
In an attempt to fill this gap, we introduce a unified learning approach to simultaneously modeling the coarse- and fine-grained retrieval by considering the multi-grained uncertainty. 
The key idea underpinning the proposed method is to integrate fine- and coarse-grained retrieval as matching data points with small and large fluctuations, respectively.
Specifically, our method contains two modules: uncertainty modeling and uncertainty regularization. 
(1) The uncertainty modeling simulates the multi-grained queries by introducing identically distributed fluctuations in the feature space. 
(2) Based on the uncertainty modeling, we further introduce uncertainty regularization to adapt the matching objective according to the fluctuation range.
Compared with existing methods, the proposed strategy explicitly prevents the model from pushing away potential candidates in the early stage and thus improves the recall rate.  On the three public datasets, i.e., FashionIQ, Fashion200k, and Shoes, the proposed method has achieved +4.03%, + 3.38%, and + 2.40% Recall@50 accuracy over a strong baseline, respectively.

Published as a conference paper at ICLR 2024
COMPOSED IMAGE RETRIEVAL WITH TEXT FEEDBACK VIA MULTI-GRAINED UNCERTAINTY REGULARIZA-
TION
Yiyang Chen 1.2* Zhedong Zheng Wei Ji
Leigang Qu' Tat-Seng Chua
Sea-NExT Joint Lab, National University of Singapore 2 Tsinghua University
3 Faculty of Science and Technology, and Institute of Collaborative Innovation, University of Macau
ABSTRACT
We investigate composed image retrieval with text feedback. Users gradually look for the target of interest by moving from coarse to fine-grained feedback. How- ever, existing methods merely focus on the latter, i.e., fine-grained search, by har- nessing positive and negative pairs during training. This pair-based paradigm only considers the one-to-one distance between a pair of specific points, which is not aligned with the to-etoymnny coarse- coarse-grained retrieval process and compromises the recall rate. In an attempt to fill this gap, we introduce unified learning ap- proach to simultaneously modeling the coarse- and fine-gratd ane-grained retrieval by con- sidering the multi qulti--raiied uncertainty. The key idea underpinning the proposed method is to integrate fine- and coarse-grainet retrieval as matching data points with small and large fluctuations, respectively. Specifically, our method contains two modules: uncertainty modeling and uncertainty regularization. (1) The uncer- tainty modeling simulates the muult--gained queries by introducing identically dis- tributed fluctuations in the feature space. (2) Based on the uncertainty modeling, we further introduce uncertainty regularization adapt the matching objective ac- cording to the fluctuation range. Compared with existing methods, the proposed strategy explicitly prevents the model from pushing away potential candidates in the early stage, and thus improves the recall rate. On the three public datasets, i.e. FashionIQ, 200hion200k, and Shoes, the proposed method has achieved +4.03% +3.38%, and 40% Recall @ 50 accuracy over a strong baseline, respectively.
INTRODUCTION
Despite the great success of recent deeply-learned retrieval systems (Radenović et al., 2018; Zheng et al., 2017a; 2020b), obtaining accurate queries remains challenging. Users usually cannot express clearly their intentions at first glance or describe the object of interest with details at the very be- ginning. Considering such an obstacle, the retrieval system with feedback is preferable since it is similar to the human recognition process, which guides the users to provide more discriminative search conditions. These conditions are used to narrow down the search scope effectively and effi- ciently, such as "/ want similar shoes but with red color". In this work, without loss of generality, we study a single-round real-world scenario, i.e., composed image retrieval with text feedback. This task is also named text-guided image retrieval. Given one query image and one feedback, the re- trieval model intends to spot the image, which is similar to the query but with modified attributes according to the text. Such an ideal text-guided image retrieval system can be widely applied to shopping and tourism websites to help users find the target products/ attractive destinations without the need to express clear intention at the start (Liu et al., 2012). Recent developments in this field are mainly attributed to two trends: 1) the rapid development of deeply-learned methods in both computer vision and natural language processing communities, e-g., the availability of pre-trained large-scale cross-modality models, such as CLIP (Radford et al., 2021); and 2) the fine-grained met- ric learning biometric recognition, such as triplet loss with hard mining (Hermans et al., 2017; Oh Song et al., 2016), contrastive loss (Zheng et al., 2017b), and infoNCE loss (Oord et al., 2018; Han et al., 2022), which mines one-to-one relations among pair- or triplet- inputs.
Work done during an internship NUS. Correspondence to zhedongzheng www.een.muummmmuuum.mmmm.

---

Published as a conference paper at ICLR 2024
However, one inherent problem remains: how to model the coarse -raained retrieval? The fine- grained metric learning in biometrics de facto is designed for strict one-to-one fine-grained match- ing, which is not well aligned with the mulli-grained image retrieval with text feedback. As shown in Figure la, we notice that there exists multiple true matchings or similar candidates. If we still apply pair-wise metrie learning, it will push away these true positives, compromising the training on coarse-grained annotations.
As attempt to loosen this restriction, we introduce a matching scheme model an uncertainty range, which is inspired by the human retrieval process from coarse-grained range to fine-grained range. As shown in Figure 1b, we leverage uncertain fluctuation to build the gulti-grained area in the representation space. For the detailed query, we still apply one-to-one matching as we do in biometric recognition. On the other hand, more common cases are one-to-many matching. We conduct matching between one query point and point with an uncertain range. The uncertain range a feature space, including multiple potential candidates due to the imprecise query images or the ambiguous textual description. Jointly considering the two kinds of query, we further introduce a unified uncertainty-beet
approach for both one-to-one matching and one-to-many matching, in the multi-graine image retrieval with text feedback. In particular, the unified uncertainty- approach consists of two modules, i.e., uncertainty modeling and uncertainty regularization. The uncertainty modeling simulates the real uncertain range within a valid range. The range is estimated based on the feature distribution within the mini-batch. Following the uncertainty modeling, the model learns from these noisy data with different weights. The weights are adaptively changed ac- cording to the fluctuation range as well. In general, we will not punish the model, if the query is ambiguous. In this way, we formulate the fine-graines matching with the caase-graiine matching in one unified optimization objective during training. Different from only applying one-to-one match- ing, the uncertainty regularization prevents the model from pushing away potential true positives, thus improving the recall rate. Our contributions are as follows.
We pinpoint training/test misalignment in real-world image retrieval with text feedback, specifically between fine- -ra-grr metric learning and the practical need for coarse- coarse-grained inference. Traditional metric learning mostly focuses on one-to-one alignment, adversely impacting one-to-many coarse-grained learning. This problem identification underscores the gap we address in our approach.
We introduce a new unified method to learn both fine- and coarse-grained matching dur- ing training. In particular, we leverage the uncertainty-based matching, which consists of uncertainty modeling and uncertainty regularization.
Albeit simple, the proposed method has achieved competitive recall rates, i.e., 61.39%, 79. 84% and 70.2% Recall 50 accuracy on three large-scale datasets, i.e., FashionIQ, Fash- ion200k, and Shoes, respectively. Since our method is orthological to existing methods, it can be combined with existing works to improve performance further.
RELATED WORK
Composed Image Retrieval with Text Feedback. Traditional image retrieval systems utilize one relevant image as query (Philbin et al., 2007; Zheng et al., 2020a). It is usually challenging to acquire such an accurate query image in advance. The multimodal query involves various input queries of different modalities, such as image and text, which eases the difficulty in crafting query and provides more diverse details. In this work, we focus on image retrieval with text feedback, also called text-guided image retrieval. Specifically, the input query contains a reference image and textual feedback describing the modifications between the reference image and the target image. The critical challenge is how to compose the two modality inputs properly (He et al., 2022; Yang et al., 2016; Saito et al., 2023; Shin et al., 2021), and align visual and language space (Radford et al., 2021; Norelli et al., 2022; Saito et al., 2023). The existing methods (Lee et al., 2021; Baldrati al., 2022) usually extract the textual and visual features of the query separately through the text encoder and image encoder. These two types of features are composited as the final query embeddings to match the visual features of the target image. Generally, there are two families of works on image retrieval with text feedback based on whether using the pre-trainne trained model. The first line of work mainly studies how to properly combine the features of the two modalities (Qu et al., 2021; Han et al., 2022). Content-Stye Modulation (CosMo) (Lee et al., 2021) proposes an image- compositor containing two independent modulators. Given the visual feature of the reference image,
2

---

Published as a conference paper at ICLR 2024
Hi, find me a sleeveless dress for party.
Found sleeveless dresses for party like these.
P+
Coarse grained Retrieval
One-lo-one Matching
One-to-many Matching
orange with black
Black
think the third one great, but want red retro Style, preferably with tie belt and bow collar.
Dante
Just
How about this ane?
Fine -ra ined Retrieval
ft
Great, exactly want.
(a)
Figure 1: (a) The typical retrieval process contains two steps, i.e, the coarse-grainet retrieval and fine-grained retrieval. The soarse-grained retrieval hamesses the brief descriptions imprecise query images, while the fine-grained retrieval requires more details for one-to-one mapping. The existing approaches usually focus optimizing the strict pair-wise distance during training, which is different from the one-to-many coarse-graines test setting. Overwhelming one-to-one metric learning compromises the model to recall potential candidates. (b) Our intuition. We notice that there exist two typical matching types for the fine- and coarse-grained retrieval. Here we show the difference between one-to-one matching (left) and one-to-many matching (right).
the content modulator first performs local updates on the feature map according to text features. The style modulator then recovers the visual feature distribution with the original feature mean and std for matching. Furthermore, CLVC-Net (Wen et al., 2021) introduces two fine-grained compositors: a local-wise image-based compositor and a global-wise text-based compositor. Following the spirit of mutual learning (Zhang et al., 2018), two compositors are learned from each other considering the prediction consistency. With the rapid development of the big model, another line of work is to leverage the model pre-trained on large-scale cross-modality datasets, and follow the pretrain- finetune paradigm. For instance, CLIP4Cir (Baldrati et al., 2022) applies CLIP (Radford et al., 2021) the initial network to integrate text and image features, and adopts two-stage training strategy to ease the optimization. Similar to CLIP, CLIP4Cir fine-tunes the CLIP text encoder and CLIP visual encoder for feature matching in the first stage. In the second stage, the two encoders are fixed and non-linear compositor is added to fuse the text and visual feature in an end-to-end manner. Taking one step further, Zhao al. (2022) introduce extra large-scale datasets, e.g., FACAD (Yang et al., 2020), FashionGen (Rostamzadeh et al., 201 (), for pretraining. Different from these existing works, in this work, we do not focus on the network structure or pretraining weights. Instead, we take closer look at the multi-grained matching, especially the coarse-grained one-to-many matching during training. We explicitly introduce the uncertainty range to simulate the intra-class jittering.
Uncertainty Learning. With the rapid development of data-driven methods, the demands on the model reliability rise. For instance, one challenging problem still remains how to measure the "con- fidence" of prediction. Therefore, some researchers resort to uncertainty. Kendall & Gal (2017) divide the uncertainty into two major categories, i.e., epistemic uncertainty and aleatoric uncertainty. The former epistemic uncertainty denotes model uncertainty that the models, even trained on the same observation (dataset), learn different weights. The typical work of this direction is Bayesian networks (Jordan al., 2007; Gal & Ghahramani, 2016), which does not learn any specific weights but the distribution of weights. In a similar spirit, Monte Carlo Dropout (Gal & Ghahramani, 2016) proposed to simulate the Bayesian networks during inference, randomly dropping the network weights. Another family of works deals with the inherent data uncertainty, usually caused by device deviations or ambiguous annotations. This line of uncertainty approaches has been developed in many fields, including image retrieval (Warburg et al., 2021), image classification (Postels et al., 2021), image segmentation (Zheng & Yang, 2021), 3D reconstruction (Truong et al., 2021), and person re-id (Yu et al., 2019; Zhang et al., 2022). In terms of representation learning, (Chun et al., 2021; Pishdad et al., 2022) directly regress the mean and variance of the input data and use the probability distribution similarity instead of the cosine similarity. Similar to (Chun et al., 2021; Pishdad et al., 2022), Oh et al. (2018) use Monte Carlo sampling to obtain the averaged distribution similarity. Besides, Warburg et al. (2021) directly consider the loss variance instead of the feature
3

---

Published as a conference paper at ICLR 2024
Conpoositoe
Coome-grained
"is short sleeved and pink
Text
Lu
Fine-grained
Matching
N(1,6;)
Image Encoder
Gaussian
Generation
f
Linfo
share weights
Re.Pt
Tretwooo
Image Encader
f
h
Augmenter
The overview of our network. Given the source image La and the text T, for modifica- tion, we obtain the composed features by combining f7 and f! via compositor. The compositor contains a content module and a style module. Meanwhile, we extract the visual features ft of the target image I, via the same image encoder the source image. Our main contributions uncertainty modeling via augmenter, and the uncertainty regularization for coarse matching. (1) The proposed augmenter applies feature-level noise to ft. yielding ft with identical Gaussian Noise (11, and (N,,, de), respectively. Albeit simple, it is worth noting that the augmented feature ft simulates the intra-class jittering of the target image, following the original feature distribution. (2) The commonly used InfoNCE loss focuses the fine- grained one-to-one mapping between the original target feature f+ and the composited feature fs. Different from InfoNCE loss, the proposed method harnesses the augmented feature f, and fa to simulate the one-to-many mapping, consider- ing different fluctuations during training. Our model applies both the fine- fine-grained matching and the proposed coarse- saaree-rrined uncertainty regularization, facilitating the model training.
Figure 2:
the
variance. In particular, they re-formulate the original triplet loss to involve the consideration of loss variance. Another line of works (Chang et al., 2020; Dou et al., 2022) directly adds noise to features simulate the uncertainty. The variance also from the model prediction like (Chun et al., 2021; Pishdad et al., 2022) and a dynamic uncertainty-based loss is deployed as (Kendall & Gal, 2017; Zheng & Yang, 2021). Similarly, in this work, we also focus on enabling the model learning from multi-grained annotations, which can be viewed as an annotation deviation. Differently, there are two primary differences from existing uncertainty- -based works: (1) We explicitly introduce Gaus- sian Noise to simulate the data uncertainty in terms of the coarse-grained retrieval. For instance, we simulate the "weak" positive pairs by adding intra-class jittering. (2) We explicitly involve the noisy grade in optimization objectives, which unifies the coarse- and fine-grained retrieval. If we add zero noise, we still apply the strict one-to-one metric learning objective. we introduce more noise, give the network less punishments. It is worth noting that unifying the fine- and coarse coarse-grained retrieval explicitly preserve the original representation learning, while preventing over wer-fitting to am- biguous coarse-gee coarse-grained samples in the dataset.
3 METHOD
3.1 PROBLEM DEFINITION
We show the brief pipeline in Figure 2. In this paper, we do not pursue sophisticated network structure but new learning strategy. Therefore, we mainly follow existing works (Lee al., 2021; Chen et al., 2020) to build the network for fair comparison, and more structure details are provided in Implementation Details. Given triplet input, i.e., one source image input J,, one text sentence T, and the target image 4g the model extracts the visual feature fo of L., the target feature ft of 1, and the textual feature f2 Tf .. Then we leverage a simple compositor to combine the source visual feature f! and the source text feature f as the composed feature fs- We intend to increase the cosine similarity between f. and fi in the representation space. During test time, we extract f. as the query feature to find the most similar f. in the candidate pool.
We mainly consider the uncertainty in the triplet annotations. As shown in Figure la, the dataset usually contains multi-grained matching triplets, considering descriptions and input images. If the
4

---

Published as a conference paper at ICLR 2024
description is ambiguous the source image is relatively imprecise, the system should return multi- ple candidate images. In the training phase, if we still apply strong one-to-one matching supervision on such triplet, it will significantly compromise the model recall potential true matches. There- fore, in this work, we mainly focus on the uncertainty in the triplet matching. not specified, we deploy uncertainty denote matching uncertainty.
3.2 UNCERTAINTY MODELING
We introduce a noise augmenter to simulate the intra-class jittering. As shown in Figure 1b, instead of striet one-to-one matching, we impel the model to focus on one-to-many matching. Therefore, we need to generate the jittering via augmenter. The augmenter directly works on the final represen- tation space. In particular, the augmenter adds Gaussian Noise of the original feature distribution to the target features ft. The mean Mr and standard deviation o, of the Gaussian Noise calculated from the original feature ft. We then apply the generated Gaussian noise to the whitened feature. Therefore, the final jittered feature fe can be formulated as follows:
(1) where a and 3 are the noisy vectors with the same shape as the input target feature, a ~ o1), 3 ~ N(,, ot), and ft is whitened feature f. = fr-M We apply the element-wise multi- plication to re-scale the imput feature, so the Gaussian noise mean of a is set as 1, which different from B. It is worth noting that, in this way, make the feature fluctuate in a limited degree, which is close to the original distribution.
= f. +
3.3 UNCERTAINTY REGULARIZATION
The existing methods usually adopt InfoNCE loss (Goldberger et al., 2004; Movshovitz-Atiaa al., 2017: Snell et al., 2017; Gidaris & Komodakis, 2018; Vo et al., 2019a; Lee et al., 2021) which can be viewed kind of batch-wise classification loss. It can be simply formulated as:
1 Σ log Linto (Ss, fc) = B 1=1
exp (N (fz, fi))
E/=1 exp K ff. ff)
Given the composed feature fa and the target feature fe of a mini-batch with B samples, InfoNCE loss maximizes the self-similarity K. (fz, fi)) while minimizing the similarity with other samples
(( A j) in the batch. We adopt the cosine similarity K, which is ((f(, -
fi.ei
We note that InfoNCE loss merely focuses on the one-to-one fine-grained matching. In this work, intend unify the fine- and coarse-grained matching. Inspired by the aleatoric uncertainty, we propose the uncertainty regularization. The basic InfoNCE loss is a special case of our loss. Given two types of features f. and fs, our uncertainty regularization can be defined as follow:
L (Ja, f1.0)))) Linto (fa. f...) + logo²
(3)
20²
If the a is a constant, our C, regresses to a weighted Linfo- The main difference from the InfoNCE loss is that adaptively tune the optimization objective according the jittering level in fc. If the jittering fluctuation is large (i.e., large σ), the weight of first-term InfoNCE loss decreases. In contrast, if the feature has limited changes, the regularization is close the original InfoNCE loss.
To optimize the multi multi-grained retrieval performance, we adopt a combination of the fine- -neigrai loss Lint and the proposed uncertainty regularization L. Therefore, the total loss is follows:
Lioral Cuu (J.. f..o.) + (1 - 7) Linto (fa, ft),
where 7 is a dynamic weight hyperparameter to balance the ratio of the fine- and coarse-grained retrieval. o, is the standard deviation of ft- If we ignore the constant term, which does not generate backward gradients, we could rewrite this loss in a unified manner as:
Liotal = =7f (Jar oe. + (1 (1-7) C. Je. St. V2
(5)
During training, gradually decrease the coarse coarse-grained learning, and increase the fine- fineinrieeed carrent.epoch where "o is the initial weight. By setting the total_epoch
learning by annealing 7 exp(-70
5

---

Published as a conference paper at ICLR 2024
exponential function, we ensure Y [0,1]. 1]. y O, only consider the fine- fine-grained retrieval as existing works (Lee et al., 2021). If y = 1, we only consider the coarse-gres -aaneeg one-to- one-to-many matching between fluctuated features.
Discussion. 1). What is the advantage of uncertainty regularization in the feature learning? The uncertainty regularization is to simulate the one-to-many matching case, which leaves space for multiple ground-truth candidates. It successfully invades the model to over-fitting the strict one- to-one matching as in biometric recognition. As shown in the experiment, the proposed method significantly surpasses other competitive methods in terms of the recall rate, especially Recall @ 50. 2). How about using uncertainty regularization alone? Why adopt the dynamic weight? Only coarse matching, which is easy to converge, leads the model to miss the challenging fine- the-grained matching, even if the description is relatively detailed. Therefore, when the model converges with the "easy" graiiee matching during training, we encourage the model to focus back on the fine-grained matching again. The ablation study on ? in Section 4 also verifies this point. 3). Extensibility to the Ensembled Dataset. For instance, in the real-world scenario, one common challenge is how to learn from the ensembled dataset. The ensembled dataset may contain fine- grained text descriptions as well as coarse -aase-grained attribute annotations, like key words. For such a case, the proposed method could, in nature, facilitate the model to learn from multi multi-grained data. The experiment on the three subsets of FashionIQ verifies the scalability of the proposed method.
4 EXPERIMENT
Implementation Details. We employ the pre-trained models backhone: ResNet-50 (He et al., 2015) on ImageNet as the image encoder and RoBERTa (Liu et al., 2019) as the text encoder. In par- ticular, we adopt the same compositor structure in the existing work CosMo for fair comparison, which contains two modules: the Content Modulator (CM) and the Style Modulator (SM). The CM uses Disentangled Multi-modal Non-local block (DMNL) to perform local updates to the reference image feature, while the SM employs multimodal attention block to capture the style information conveyed by the modifier text. The outputs of the CM and SM are then fused using simple con- catenation operation, and the resulting feature is fed into fully connected layer to produce the final image-text representation. SGD optimizer (Robbins & Monro, 1951) is deployed with a mini-batch of 32 for 50 training epochs and the base learning rate is 2 x 10-2 following Lee et al. (2021). We apply the one-step learning rate scheduler to decay the learning rate by 10 at the 45th epoch. We empirically set w11 = 1, W22 = which control the scale of augmenter generates Gaussian noise, and the initial balance weight 70 = 1. During inference, we extract the composed feature fa to calculate the cosine similarity with the feature of gallery images. The final ranking list is generated according to the feature similarity. Reproducibility. The code is based on Pytorch (Paszke et al., 2019). and annoymous code is at !. We will make our code open-source for reproducing all results.
Datasets. Without loss of generability, we verify the effectiveness of the proposed method on the fashion datasets, which collect the feedback from customers easily, including FashionIQ (Wu et al., 2021), Fashion200k (Han et al., 2017) and Shoes (Guo et al., 2018). Each image in these fashion datasets is tagged with descriptive texts as product description, such as "similar style t-shirt but white logo print". FashionIQ. We follow the training and test split of existing works (Chen et al., 2020; Lee et al., 2021). Due to privacy changes and deletions, some links are out-of-the-date. We try our best to make up for the missing data by requesting other authors. As result, we download 75,384 images and use 46,609 images in the original protocol for training. Shoes. Shoes (Guo et al., 2018) crawls 10,751 pairs of shoe images with relative expressions that describe fine-grained visual differences. We use 10,000 samples for training and 4,658 samples for evaluation. Fashion200k. Fashion200k (Han et al., 2017) has five subsets: dresses, jackets, pants, skirts, and tops. We deploy 172,000 images for training on all subsets and 33,480 test queries for evaluation. Evaluation met- ric. Following existing works, we report the average Recall Recall@ 10, and Recall 55 of all queries.
Comparison with Competitive Methods. We show the recall rate on FashionIQ in Table includ- ing the three subsets and the average score. We could observe four points: (1) The method with uncertainty modeling has largely improved the baseline in both Recall@ 10 and @ aceu- racy, verifying the motivation of the proposed component on recalling more potential candidates.
ht pes / //tg///tttttt this wow.oonoo de Caw--hheennnerreeeeeeee Suncerta int.... ret rievs)

---

Published as a conference paper at ICLR 2024
Table 1: Results on FashionIQ. The best performance is in bold. Here we show the recall rate
R@K, which denotes Recall@K. K. Average denotes the mean of R@ K on all subsets. It is worth
noting that our method with one ResNet-50 is competitive with CLIP4Cir (Baldrati al., 2022) of
4x ResNet-50 in R R050. We train models with different initializations as model ensembles. Note
that Fashion ViL introduces extra datasets for training.
Method
Visual Backbone
Dress
Shart
Topice
Average
R&10
RG550
R
R00
10
RG00
30
MRN (Kim al. 2016)
ResNet-152
12.32
32.18
13.88
34.33
18.11
36.33
13.44
34.28
FiLM (Perez et al, 2018)
ResNet-50
14.23
33,34
15.04
34.09
17.30
37.68
15.52
35.04
TIRG (Vo al1,9)) 2019b)
ResNet-17
14.87
34.66
18.26
37.89
19.08
39.62
17.40
37.39
Pic2Word (Saito et al. 2023)
ViT-1/14
20.00
40.20
26.20
43.60
27.90
47.40
24.70
43.70
(Chen et al., 2020)
ResNet-50
21.12
42.19
21.03
43.44
25.64
49.49
22.60
45.04
ARTEMIS (Delmas et 2022)
ResNet-50
27.16
52.40
21.78
29,20
43.64
CoSMo (Lee et al., 2021)
ResNet-50
25.64
50.30
24.90
49.18
29.21
57.46
26.58
52.31
DCNet (Kim et al., 2021)
ResNet-50
28.95
56.07
23.95
47.30
30.44
58.29
27.78
$3.89
Fashion ViL (Han
2022)
ResNet-50
28.46
54.24
22.33
46.07
29.02
57.93
26.60
$2.74
VaL (Han et ul., 2022)
ResNet-50
33,47
59.94
25.17
50.39
34,98
60.79
31.21
57.04
Baseline
ResNet-50
24.80
52.35
27.70
55.71
33.40
63.64
28.63
57.23
Ours
ResNet-50
30.60
$7.46
31.54
58.29
37.37
68.41
33.17
61.39
CLVC-Net (Wen et al. 2021)
ResNct-50x2
29.85
56.47
28.75
54,76
33.50
64.00
30.70
58.41
Ours
ResNet-50x2
31.25
58.35
31.69
60.65
39.82
71.07
34.25
63.36
CLIP4Cir (Baldruti et ul., 2022)
ResNet-50x4
31.63
56.67
36.36
58.00
38. 19
62, 42
35.39
59.03
Ours
ResNet 50x4
32.61
61.34
33,23
62.55
41.40
72.51
Table 2: Results on Shoes and Fash- ion200k. We mainly compare ours with single model-based methods (the best performance in red, the second-best in orange). Similar to the phenomenon on FashionIQ. could observe that the proposed method improves the recall rate over baseline, which is aligned with our intuition on multi- gellii-graned matching.
Method
R@1
RGr 10
Shoes
RG:30
RBT
Fushion2200
RGF10
RG:50
MRNOKm 2010)
11.74
41.70
67.01
13.4
44100
61.9
FiLM/Parcz 2018)
10.19
38.89
68. 68.30
12.9
39.5
61.9
TIRG/V 20196)
12.6
45.45
69.39
14.1
42.5
63.8
VAL 2020)
49.12
73.53
49.0
68.8
CoSMotLec et al., 2021)
16.72
48.36
75.64
25,3
50.4
69.3
DCNet(Kim al., 2021)
53.82
79.33
46.9
67.6
ARTEMIS at. 2022)
18.72
53.11
79.31
51.1
70.5
Baseline
15.36
49148
76.46
19.5
46.7
67.8
Ours
18.41
53.63
79.84
52.1
70.2
Especially for Recall @ 50, the proposed method improves the accuracy from 37 57.23% to 61.39%. (2)
Comparing with the same visual backbone, i.e., one ResNet-50, the proposed method has arrived a competitive recall rate in terms of both subsets and averaged score. (3) The proposed method with one ResNet-50 competitive with the ensembled methods, such as CLIP4Cir (Baldrati et al., 2022) with 4x ResNet-50. In particular, ours with one ResNet-50 has arrived at 61.39% Recall (0, sur- passing CLIP4Cir 559.0% Recall@50) 50). (4) We train our model ensemble with different initializa- tion, and simply concatenate features, which also surpasses both CLVC-Net (Wen et al., 2021) and CLIP4Cir (Baldrati et al., 2022). We observe similar performance improvement on Shoes and Fash- ion200k in Table 2. (1) The proposed method surpasses the baseline, yielding 18.41% Recall @ 1, 53.63% Recall@ @ 10, and 79.84% Recall 50 on Shoes, and 21.8% Recall@ 1, 52. 1% Recall 10, and 70.2% Recall @ 50 on Fashion200k. Especially, in terms of Recall @ 50 accuracy, the model with uncertainty regularization has obtained +3.38% and +2.40% accuracy increase on Shoes and Fash- ion200k, respectively. (2) Based on one ResNet-50 backbone, the proposed method is competitive with ARTEMIS (Delmas et al., 2022) in both Shoes and Fashion200k, but ours are more efficient, considering that ARTEMIS needs calculate the similarity score for every input triplets. (3) Our method also achieves better Recall @ 50 79.84% than CLVC-Net (Wen et al., 2021) 79.47% with 2x ResNet-50 backbone, and arrives at competitive Recall 10 accuracy on Fashion200k.
Can Dropout replace the uncertainty regularization? No. Similar to our method, the dropout function also explicitly introduces the noise during training. There are also two primary differences: (1) Our feature fluctuation is generated according to the original feature distribution, instead of a fixed drop rate in dropout. (2) The proposed uncertainty regularization adaptively punishes the net- work according to the noise grade. To verify this point, we compare the results between ours and the dropout regularization. In particular, we add dropout layer after the fully connected layer of the text encoder, which is before the compositor. For fair comparison, we deploy the baseline (Cinfo) without modeling uncertainty for evaluation and show the results of different dropout rates in Ta- ble 3a. We observe that the dropout does not facilitate the model recalling more true candidates. No matter whether the drop rate is set as 0.2 or 0.5, the performance is close to the baseline without the dropout layer. In contrast, only using the uncertainty regularization (L.,) improves Recall from

---

Published as a conference paper at ICLR 2024
Table 3: Ablation studies on the Shoes dataset. The average is (R@ 10+R@50)/2 (a) Impact of dropout. We compare the impact of the dropout rate and the proposed uncertainty regularization Lu- We could observe that dropout has limited impacts on the final retrieval performance. In contrast, the proposed Lu shows significant recall accuracy boost. (b) Parameter sensitivity of wT, w2. The left side of the table shows the R Ks of different values of w when we fix w2 = 1. Meanwhile, the right side of the table shows the @@ Ks of different values of Wy when we fix W1 = 1. W1 W22 has the best performance and the same results on two sides. (c) The influence of the initial balance weight to the model. (d) Ablation Study on static 7 as constant.
(a)
Unceriainly
Drop Rate
K
5
Average
70
R
R@:50
Average
Baseline (emts)
49.48
76.46
62.97
0.1
50,80
78.64
64.72
Cints Dropout
0.2
49.74
76.37
63.06
0.5
51.20
79.12
65,76
Cuta Dropout
0.5
49.00
75.83
62.42
1
53.63
79.84
66.74
Dou et (2022)
50.14
77.89
64.01
2
49.66
77.63
63.65
Augment Source Feature f!
52.20
77.75
64.98
3
50.83
77.55
64.19
Only Ca
50.83
77.41
64.12
5
50.14
76.32
63.23
Ours (Cinto C.))
53.63
79.84
66.74
10
49.91
76.43
63.17
(b)
(buseline)
49,48
76.46
62,97
Scale
W/
I. 122
(d)
R
Average
R
R@
Average
48.42
75.57
62.00
51.03
79.10
65.07
R
R@:00
Average
0.2
51.35
76.35
63.85
50.49
79.15
64.82
0.2
30.90
63.12
47.01
51.03
78.84
64.94
50.69
77.75
64,22
0.5
41.87
73.11
57.49
0.8
41.87
72.39
57.13
51.95
65.45
50.43
78.38
64.41
53.63
79.84
66.74
53.63
79.84
66.74
48.71
76.80
62.76
49.83
78.41
64.12
51.29
78.06
64.68
48.20
76.83
62.52
48.91
76.23
62.57
48.60
77.18
62.89
10
48.71
76.80
62.76
47.02
74.71
60.87
76.46% to 77.41%. Combining with the fine-grained matching baseline, the uncertainty regulariza- tion even improves Recall 10 by +4.15% accuracy. Besides, we re-implement (Dou et al., 2022) by adding two extra branches to regress the feature mean and variance for loss calculation. It has achieved 50.14% Recall@ 10 and 77.89 % Recall@50, which is inferior to our method.
Impact of the noise fluctuation. We study the impact of the noise fluctuation in the uncertainty modeling. In particular, we change the noise scale in Eq. by adjusting the scale of a and B. The modified formulation is as follows:
(6) where a' (11, w1or and 3' E N(,, W27). W1, uP2 are the scalars to control the noise scale. We show the impact of different noise scales in Table 3b. We fix one of the w to and change the another parameter. We could observe two points: (1) As we expected, the identical noise setting (wi = w2 1) achieves the best performance. This is because such noise is close to the original feature distribution and simulates the fluctuation in the training set. (2) The experiment results also show that uncertainty regularization training data contains lots of noise, the network is still robust and achieves reasonable performance. It is attributed uncertainty regularization that punishes the network according to the noise grade.
a' + 8¹
tolerate large amplitude noise changes. Even if the
Parameter sensitivity of the balance weight 7. As shown in Eq. 4, y is dynamic weight to help balance the fine- and coarse-grained retrieval. During training, we encourage the model to gradually pay more attention the fine-grained retrieval. According to 7 exp(-20- carrent-epach ), we total .epoch
different initial values of 7o to change the balance of the two loss functions. We evaluate the model on the Shoes dataset and report results in Table 3c. If Yo is close to 0, the model mostly learns the uncertainty loss on the coarse-grained retrieval, and thus recall rate still high. In contrast, if to is close to +o0, the model only focuses on the fine- fine-grained learning, and thus the model converges to the baseline. Therefore, when deploying the model to the unseen environment, To = 1 can good initial setting. Besides, results with fixed are shown in Table 3d. The constant uncertainty loss drives the model to focus on coarse coarse-grained grained matching, resulting in low recall rates as well.
Qualitative visualization. We show the top-5 retrieval results FashionIQ. Fashion200k, and Shoes in Figure 3. (I) Compared with the baseline, our model captures more fine-grained keywords, like "shiner". (2) The proposed method also recalls more candidate images with consistent style. also reflects that the proposed method is robust and provides a better user experience, since most websites display not only 1 but also top-5 products.
Training Convergence. As shown Table 4a, the baseline model (blue line) is prone to overfit all labels, including the graieed triplets. Therefore, the training loss converges quickly.
8

---

Published as a conference paper at ICLR 2024
lishow
multiontio black
Tarki Shand
Quaryt Dann Haans
Squelina
Dut
He Sinii
1. Fashaark
IB. Fastiona00k
Figure 3: Qualitative image retrieval result on FashionlQ. Fashion200k and Shoes. We mainly compare the top-5 ranking list of the proposed method with the baseline. (Please zoom in.)
Table 4: (a) The training loss of the proposed method and baseline. The baseline model is prone to over-fit fit all triplets in one-to-one matching style. In contrast, the proposed method would converge one non-zero constant. (b) Compatibility to existing methods, i.e., Clip4Cir (Baldrati al., 2022), on FashionlQ. *: We re-implemented the method with one ResNet50, and add our method. We also adopt state-of-the-ar cross- modality backbone Blip (Li et al., 2022), and add our method on the Shoes dataset. We observe that the proposed method could further improve the performance.
(a)
(b)
FashionIQ
R@
R@50
18-
Beseliee (yo
Ours (L, ma!
Clip4Cir
32.36
56.74
1.0
ClipACir* + Ours
34.19
59.23
0.6
0.8
Shoes
R 10
R@50
0.4
Blip eL al. 2022)
45.93
70.21
0.
Blip (Li et al. 2022) Ours
48.40
75.29
Epoch
In contrast, our method (orange line) also converges but does not force the loss to be zero. Because we provide the second term in uncertainty regularization Eq. 3, which could serve as a loose term.
Only Coarse Retrieval Evaluation. We design an interesting experiment only to consider query pairs with ambiguous coarse descriptions, i.e., less than 5 words in the FashionIQ dress dataset (about 6,246 of 10,942 queries). Compared with the baseline model, our method improves 1.46% Recall @1 10 rate and 3.34% Recall@50 rate. The result verifies that model can significantly improve coarse retrieval performance over baseline.
Augment the source feature f!_ We modify and add feature augments to the source image, but the Recall @ 10 rate decreases 1, 1.43% and the Recall @ 50 rate decreases 2.09% on Shoes (see Table 3a). due to the conflict with the one-to-many matching. If conduct the source feature augmentation, it will become to many-to-one matching. As the visual intuition in Figure 1b, is better to apply such augmentation on the target feature instead.
Complementary to other works? Yes. We re-implement the competitive method Clip4Cir (Bal- drati et al., 2022) in Table 4b, and show our method is complementary, further improving the recall. Similarly, we also adopt a state-of-the-arr cross-modality backbone Blip (Li et al., 2022), and add our method the Shoes dataset. We observe that the proposed method could further improve the performance of about 5% on both Recall @ 10 and Recall@:50
5 CONCLUSION
In this work, we provide an early attempt at a unified learning approach to simultaneously modeling coarse- and fine-grained retrieval, which could provide a better user experience in real-world re- trieval scenarios. Different from existing one-to-one matching approaches, the proposed uncertainty modeling explicitly considers the uncertainty fluctuation in the feature space. The feature fluctuation simulates the one-to-mant many matching for the coarse-grained retrieval. The multi-gran multi-grained uncertainty regularization adaptively modifies the punishment according to the fluctuation degree during the en- tire training process, and thus can be combined with the conventional fine-grained loss improve the performance further. Extensive experiments verify the effectiveness of the proposed method on the composed image retrieval with text feedback, especially in terms of recall rate. In the future, will continue to explore the feasibility of the proposed method on common -bjecct retrieval datasets, and involving the knowledge graph (Liu et al., 2020; Sun et al., 2021).

---

Published as a conference paper at ICLR 2024
REFERENCES
Alberto Baldrati, Marco Bertini, Tiberio Uricchio, and Alberto Del Bimbo. Conditioned and com- posed image retrieval combining and partially fine-tuning clip-based features. In CVPR Work- shops, 2022.
Jie Chang.
Zhonghao Lan, Changmao Cheng, and Yichen Wei. Data uncertainty learning in face recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recog- nition, pp. 0010-5719999 2020.
Yanbei Chen, Shaogang Gong, and Loris Bazzani. Image search with text feedback by visiolinguistic attention learning. In CVPR, June 2020.
Sanghyuk Chun, Seong Joon Oh, Rafael Sampaio De Rezende, Yannis Kalantidis, and Diane Larlus. Probabilistic embeddings for cross moosl retrieval. In CVPR, 2021.
Ginger Delmas, Rafael S Rezende, Gabriela Csurka, and Diane Larlus, Artemis: Attention-based retrieval with text-explicit matching and implicit similarity. In ICLR, 2022.
Zhaopeng Dou, Zhongdao Wang, Weihua Chen, Yali Li, and Shengjin Wang. Reliability-awan prediction via uncertainty learning for person image retrieval. In ECCV, 2022.
Yarin Gal and Zoubin Ghahramani. Dropout bayesian approximation: Representing model uncertainty in deep learning. In ICML, pp. 1050-1059. PMLR, 2016.
Spyros Gidaris and Nikos Komodakis. Dynamic few-shot visual learning without forgetting, In CVPR, June 2018
Jacob Goldberger, Geoffrey E Hinton, Sam Roweis, and Russ R Salakhutdinov. Neighbourhood components analysis. In NeurlPS, 2004.
Xiaoxiao Guo, Hui Wu, Yu Cheng, Steven Rennie, Gerald Tesauro, and Rogerio Feris. Dialog-based interactive image retrieval. In NeurlPS, 2018. URL http: //tp://pappeaa/ee/ httpsccc/ppper/ 7348-dialsd waw.--iiaooo---dd----aeee-
iawassddiinteaactiveee www.tteerraiiieeeeeeeeeeeeeeeeee image-retrieval...... pdf.
Xiao Han, Licheng Yu, Xiatian Zhu, Li Zhang, Yi-Zhe Song, and Tao Xiang. Fashionvil: Fashion- focused vision-andd-laguugee representation learning. In ECCV, 2022.
Xintong Han, Zuxuan Wu, Phoenix X. Huang, Xiao Zhang, Menglong Zhu, Yuan Li, Yang Zhao, and Larry S. Davis. Automatic spatially-aware fashion concept discovery. In ICCV, 2017. doi:
10.11099/CCV.207..666
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. CVPR, 2015.
Xuchai He, Diji Yang, Weixi Feng. Tsu-Jui Fu, Arjun Akula, Varun Jampani, Pradyumna Narayana, Sugato Basu, William Yang Wang, and Xin Eric Wang. CPL: Counterfactual prompt learning for vision and language models. In Proceedings of the 2022 Conference Empirical Methods in Natural Language Processing, 2022.
Alexander Hermans, Lucas Beyer, and Bastian Leibe. In defense of the triplet loss for person re- identification. arXiv: /00 077377 2017.
Michael Jordan, Jon Kleinberg, and Schölkopf Bernhard. Bayesian networks and decision graphs. Springer New York, 2007.
Alex Kendall and Yarin Gal. What uncertainties do we need bayesian deep learning for computer vision? In NeurIPS, 2017. URL https: : http://pccceee....ppeeppp neurips,..... pctpppppppppp.////////p// file/2650d6083 005.:44440055522222222222e2
Pww.bb--ppppr..p.......p.. .pdf.
Jin-Hwa Kim, Sang-Woo Lee, Donghyun Kwak, Min-Oh Heo, Jeonghee Kim, Jung- Woo Ha, and Byoung- Zhang. Multimodal residual learning for visual qa. In
NeurIPS, 2016. URL https://proceeddl. neurips........... www.eiis....ppppepeeeeee..eeeeee cc/papper/216//11e
pdf.
10

---

Published as a conference paper at ICLR 2024
Jongseok Kim, Youngjae Yu, Hoeseong Kim, and Gunhee Kim. Dual Compositional Learning in Interactive Image Retrieval. In AAAI, 2021.
S. Lee, D. Kim, and B. Han. Cosmo: Content-style modulation for image retrieval with text feed- back. In CVPR, 2021.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre- training for unified vision-language understanding and generation. In International Conference Machine Learning, pp. 12888-12900 PMLR, 2022.
Shaoteng Liu, Jingjing Chen, Liangming Pan, Chong- Wah Ngo, Tat-Seng Chua, and Yu-Gang Jiang. Hyperbolic visual embedding learning for zero-shot recognition. In CVPR, 2020.
Si Liu, Zheng Song, Meng Wang, Changsheng Xu, Hanqing Lu, and Shuicheng Yan. Street-to-shop: Cross-scenario clothing retrieval via parts alignment and auxiliary set. In ACM Multimedia, pp- 1335-1336, 2012.
2012.
Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. urXivv/907, 1692, 2019.
Yair Movshovitz-Attias, Alexander Toshev, Thomas K. Leung, Sergey loffe, and Saurabh Singh. No fuss distance metric learning using proxies. In ICCV, 2017.
Antonio Norelli, Marco Fumero, Valentino Maiorca, Luca Moschella, Emanuele Rodolà, and Francesco Locatello. Asif: Coupled data turns unimodal models to multimodal without train- ing. arXiv preprint arXiv:2210.01738 2022.
Scong Joon Oh, Kevin Murphy, Jiyan Pan, Joseph Roth, Florian Schroff, and Andrew Gallagher. Modeling uncertainty with hedged instance embedding. 1rXiv:1810.003/9, 2018.
Hyun Oh Song. Yu Xiang, Stefanie Jegelka, and Silvio Savarese. Deep metric learning via lifted structured feature embedding. In CVPR, 2016.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv: 1807.08000000 003448 2018
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In NeurIPS. 2019.
Ethan Perez, Florian Strub, Harm de Vries, Vincent Dumoulin, and Aaron C. Courville, Film: Visual reasoning with general conditioning layer. In AAAI, 2018.
James Philbin, Ondrej Chum, Michael Isard, Josef Sivic, and Andrew Zisserman. Object retrieval with large vocabularies and fast spatial matching. In CVPR, pp. 1-8. IEEE, 2007.
Leila Pishdad, Ran Zhang, Konstantinos G Derpanis, Allan Jepson, and Afsaneh Fazly, Uncertainty- based cross-modal retrieval with probabilistic representations. arXiv:2204.09266 2022.
Janis Postels, Mattia Segù, Tao Sun, Lue Van Gool, Fisher Yu, and Federico Tombari. On practicality of deterministic epistemic uncertainty. CoRR, abs/2107.000 2021. URL https:
Leigang Qu, Meng Liu, Jianlong Wu, Zan Gao, and Liqiang Nie. Dynamic modality interaction modeling for image-text retrieval. ACM SIGIR, 2021.
Filip Radenovié, Giorgos Tolias, and Ondrej Chum. Fine-tuning enn image retrieval with no human annotation. IEEE transactions on pattern analysis and machine intelligence, 41(7):1655-1668, 2018.
11

---

Published as a conference paper at ICLR 2024
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021.
Herbert Robbins and Sutton Monro. A Stochastic Approximation Method. The Annals of Math- ematical Statistics, 22(3):400 407, 1951. doi: 1,11214//mm//17777955555 URL https:
//doi. /214/ aom////o///////////////// 11177295787
Negar Rostamzadeh, Seyedarian Hosseini, Thomas Boquet, Wojeiech Stokowiec, Ying Zhang, Christian Jauvin, and Chris Pal. Fashion-gen: The generative fashion dataset and challenge.
arXiv:1806.083/7 2018.
Kuniaki Saito, Kihyuk Sohn, Xiang Zhang, Chun-Liang Li, Chen- Yu Lee, Kate Saenko, and Tomas Pfister. Pic2word: Mapping pictures to words for zero-shot composed image retrieval. arXiv preprint wrXiv:2302.03084 2023.
Minchul Shin, Yoonjae Cho, Byungsoo Ko, and Geonmo Gu. Rtie: Residual learning for text and image composition using graph convolutional network. arXiv preprint arXiv:2104.03015. 2021.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In NeurlPS, 2017. URL httpps:/pppooeeddggsss... wewp://oooee.ii.....i........... neurips wttpcc/ppppppp//ceeeeep cb8da676 6 7 6 F28.2282222 Fwe.ee22222eeeeeee 2000000100000000eeeee0e-----a200 pdf.
Yifan Sun, Yuke Zhu, Yuhan Zhang, Pengkun Zheng, Xi Qiu, Chi Zhang, and Yichen Wei. Dynamic metric learning: Towards a scalable metric space to accommodate multiple semantic scales. In CVPR, 2021.
Prune Truong, Martin Danelljan, Radu Timofte, and Luc Van Gool. Pdc-net+: Enhanced proba- bilistic dense correspondence network. CoRR, MMt/2109913912, 2021. URL https: org/ abs/ 2109.111.2 13912.
Nam Vo, Lu Jiang, Chen Sun, Kevin Murphy, Li-Jia Li, Li Fei-Fei, and James Hays. Composing text and image for image retrieval-an empirical odyssey. In CVPR, 2019a.
Nam Vo, Lu Jiang, Chen Sun, Kevin Murphy, Li-Jia Li, Li Fei-Fei, and James Hays. Composing text and image for image retrieval-an empirical odyssey. CVPR, 2019b.
Frederik Warburg, Martin Jørgensen, Javier Civera, and Søren Hauberg. Bayesian triplet loss: Un- certainty quantification in image retrieval. ICCV, pp. 12138-12148, 2021.
Haokun Wen, Xuemeng Song, Xin Yang. Yibing Zhan, and Liqiang Nie. Comprehensive linguistic- visual composition network for image retrieval. In ACM SIGIR, SIGIR '21, 2021. ISBN 9781450380379. doi: 00.1145//40483333666997. URL https:/ /tte::/ddci.ooo ooci.org/li /00 ort/10.111////// 34C4835. 3462967.
Hui Wu, Yupeng Gao, Xiaoxiao Guo, Ziad AI-Halah, Steven Rennie, Kristen Grauman, and Rogerio Feris. The fashion dataset: Retrieving images by combining side information and relative natural language feedback. CVPR, 2021.
Xuewen Yang. Heming Zhang, Di Jin, Yingru Liu, Chi-Hao Wu, Jianchao Tan, Dongliang Xie, Jue Wang, and Xin Wang. Fashion captioning: Towards generating accurate descriptions with semantic rewards. In ECCV, pp. 1-17. Springer, 2020.
Zichao Yang, Xiaodong He, Jianfeng Gao, Li Deng, and Alex Smola. Stacked attention networks for image question answering. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 21-29, 2016.
Tianyuan Yu, Da Li, Yongxin Yang, Timothy Hospedales, and Tao Xiang. Robust person re- identification by modelling feature uncertainty. In ICCV, pp. 552-561, 2019. doi: 10.1109/ ICCV.2019.0006.
Xinyu Zhang, Dongdong Li, Zhigang Wang, Jian Wang, Eruii Ding, Javen Qinfeng Shi, Zhaoxiang Zhang, and Jingdong Wang. Implicit sample extension for unsupervised person re-identification. In CVPR, 2022.
12

---

Published as a conference paper at ICLR 2024
Ying Zhang, Tao Xiang, Timothy M. Hospedales, and Huchuan Lu. Deep mutual learning. In CVPR, pp. 4320 4328, 2018. doi: 109//V/// 2018......01000400 8.00444.
Yida Zhao, Yuqing Song, and Qin Jin. Progressive learning for image retrieval with hybrid-modality queries. In Proceedings of the 45th International ACM SIGIR Conference Research and De- velopment in Information Retrieval, SIGIR '22, pp. 1012-1021, New York, NY, USA, 2022. Association for Computing Machinery. ISBN 9781450387323, doi: 10.1455/377495.3532207
URL https: / wwg/oo.g./..ooooooo.oooooo 1......111533474955555 0tt.///.........1111.1.3........ 35320447,
Liang Zheng, Yi Yang, and Qi Tian. Sift meets cnn: A decade survey of instance retrieval. IEEE
Iransactions on pattern analysis and machine intelligence, (0(5):1224110 2017a.
Zhedong Zheng and Yi Yang. Rectifying pseudo label learning via uncertainty estimation for domain adaptive semantic segmentation. International Journal of Computer Vision (IJCV), 2021. doi:
doi: 0. 1007 s11263- 02000000--00020 01395-y.
Zhedong Zheng, Liang Zheng, and Yi Yang. A discriminatively leamned cnn embedding for person reidentification. ACM transactions on multimedia computing, communications, and applications (TOMM), 14(1):1-20, 2017b.
Zhedong Zheng, Yunchao Wei, and Yi Yang. University- 1652: A multi-view multi-source bench- mark for drone-based geo-localization. In ACM MM, 2020a.
Zhedong Zheng, Liang Zheng, Michael Garrett, Yi Yang, Mingliang Xu, and Yi-Dong Shen. Dual- path convolutional image-text embeddings with instance loss. ACM Transactions Multimedia Computing, Communications, and Applications (TOMM), 16(2):1-23, 2020b.
13


Reviews:
[{"review_id": "Xsql52rvUx", "reviewer": "Reviewer_oWB8", "summary": "This paper proposes a novel approach to simultaneously model the coarse- and fine-grained retrieval by using multi-grained uncertainty, which contains uncertainty modeling and uncertainty regularization. The corresponding experiments validate the effectiveness of the proposed method on three public datasets.", "questions": "None", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1.The authors propose a new learning strategy in this paper, which is well-written and easy to follow.\n2.This paper provides comprehensive experimental analysis and ablation discussion.", "weaknesses": "1.\tThe novelty is not well explained, please explain the difference from existing methods. The authors have to provide convincing proof to show why their proposed learning strategy is significantly different from the former methods. In Table 4(b), the author has provided a very simple discussion and experiment. However, it would be beneficial to include additional comparison methods and provide a more detailed discussion.\n2.\tIn the method section of this paper, it is important to highlight the unique aspects of your proposed learning strategy. While the uncertainty modeling approach presented is easy to follow, it should be acknowledged that its simplicity may also be considered a limitation in terms of novelty.\n3.\tThe comparison baseline of this paper is not sufficiently novel, consider exploring alternative methods that may be more unique. Recent three years of state-of-the-art methods should be included in the comparison experiment section to validate the effectiveness of the proposed method.\n4.\tAs mentioned, the main contribution of this paper includes three points. The description of the first contribution is not clear enough.\n5.\tThe tables in this paper are informative, but several of them require further standardization and uniformity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "E0AdSW2sCN", "reviewer": "Reviewer_FQ9t", "summary": "This paper identifies a training/test misalignment between the fine-grained metric learning and the demand on coarse-grained inference in the real-world image retrieval with text feedback. This paper introduces a new unified method to learn both fine- and coarse-grained matching during training. In particular, we leverage the uncertainty-based matching, which consists of uncertainty modeling and uncertainty regularization.", "questions": "1. Highlight the technical contributions.\n2. Provide more qualitative results and analyses.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The application is interesting, matching source image and text with target image.\n2. The paper is overall well-written and easy to follow.2\n3. The designed method is reasonable.\n4. The experimental results look good.", "weaknesses": "1. The technical novelty and contribution are limited. It is trivial that adding noise to features can enhance the robustness. The other components in this method are trivial.\n2. The qualitative results are insufficient. The authors should provide more qualitative results, and provide in-depth analyses on the cases where the proposed method outperforms or underperforms the baselines.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vxA9czyZ11", "reviewer": "Reviewer_cCoz", "summary": "In this paper, the authors studied the image text composed retrieval in the field of fashion, aiming to prevent the model from prematurely excluding the correct candidate results in the early retrieval stage, and improve the recall of the retrieval task. To solve this problem, this paper proposed a composed retrieval model based on multi-granularity uncertainty regularization, which models coarse-grained and fine-grained retrieval simultaneously by considering multi-granularity uncertainty. The experiments showed that compared with existing methods, the proposed model can significantly improve the retrieval accuracy. This paper has strong practical significance and clear research motivation. However, there are also some problems in writing, which need to be strengthened.", "questions": "please see the weaknesses.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "a.\tThis paper found that in the process of image text combination query in the real world, the multi-round interaction process inevitably includes coarse-grained retrieval and fine-grained retrieval, and the traditional fine-grained metric learning method cannot meet the requirements of coarse-grained reasoning.\nb.\tIn this paper, a new multi-granularity uncertainty regularization method is proposed. By uncertainty modeling and uncertainty regularization, fine-grained and coarse-grained matching can be learned during training. By controlling noise levels, the proposed method can be reduced to one-to-one metric learning, which is an extension of the traditional method.\nc.\tIn this paper, a large number of experiments are carried out on three public data sets, and the experiments showed that the proposed method can improve the recall rate of existing methods.\nd.\tSince the proposed method is orthogonal to traditional methods, it can be combined with existing works to further improve their performance.", "weaknesses": "a.\tFigure 1.a and section 3.1 did not clearly explain why there is uncertainty in triplet matching, and it is recommended to give examples of situations where the description is unclear or the source image is inaccurate.\nb.\tSince this paper mainly built the network using existing works [1][2], but did not describe their implementation, it is recommended to provide a more detailed explanation of the details of the network.\nc.\tThere are some colloquial problems in the language of this paper, and some expressions need to be improved. It is suggested to polish the language.\n\nReference\n[1]\tLee S, Kim D, Han B. Cosmo: Content-style modulation for image retrieval with text feedback[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021: 802-812.\n[2]\tChen Y, Gong S, Bazzani L. Image search with text feedback by visiolinguistic attention learning[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020: 3001-3011.\n\n\nTypos and minors\nThere are non-standard punctuation marks, such as P8, the third line of \"Parameter sensitivity of the balance weight\", etc. It is recommended to check the whole paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)