Easy Regional Contrastive Learning of Expressive Fashion Representations
Authors: Daiqing Qi
Handong Zhao
Sheng Li
Venue: NeurIPS 2024 poster (2024)
Keywords: vision-language models
contrastive learning
visual fashion representations
Primary area: machine_vision
TL;DR: We proposed a simple yet effective contrastive learning framework for fashion domain

Abstract: When learning vision-language models (VLM) for the fashion domain, most existing works design new architectures from vanilla BERT with additional objectives, or perform dense multi-task learning with fashion-specific tasks. Though progress has been made, their architecture or objectives are often intricate and the extendibility is limited.
By contrast, with simple architecture (comprising only two unimodal encoders) and just the contrastive objective, popular pre-trained VL models (e.g., CLIP) achieve superior performance in general domains, which are further easily extended to downstream tasks.
However, inheriting such benefits of CLIP in the fashion domain is non-trivial in the presence of the notable domain gap. Empirically, we find that directly finetuning on fashion data leads CLIP to frequently ignore minor yet important details such as logos and composition, which are critical in fashion tasks such as retrieval and captioning.
In this work, to maintain CLIP's simple architecture and objective while explicitly attending to fashion details, we propose $E^2$: Easy Regional Contrastive Learning of Expressive Fashion Representations.
$E^2$ introduces only a few selection tokens and fusion blocks (just 1.9\% additional parameters in total) with only contrastive losses. Despite lightweight, in our primary focus, cross-modal retrieval, $E^2$ notably outperforms existing fashion VLMs with various fashion-specific objectives.
Moreover, thanks to CLIP's widespread use in downstream tasks in general domains (e.g., zero-shot composed image retrieval and image captioning), our model can easily extend these models  from general domain to the fashion domain with notable improvement.
To conduct a comprehensive evaluation, we further collect data from Amazon Reviews to build a new dataset for cross-modal retrieval in the fashion domain.

E2: Easy Regional Contrastive Learning of Expressive Fashion Representations
Daiqing Qi University Virginia Charlottesville, VA 22904 daiqing qi@virginia. edu
Handong Zhao Adobe Research San Jose, CA 95110 hazhao@adobe.. com
Sheng Li University of Virginia Charlotteswillle, VA 22904 shengli@virginia edu
Text-to-image Rebrieval
Image-to-Tee Songe-to-Testt Retrieval
$5.4 CLP-FT JCML'2I
55.2 CUP-FT, IOML
Retrieved Image
Image
Deseripties:
block
+81
henale odustable coavee. Cutegory: BACKPACKS, Brand Gozzi Compoation Leather
+7.
cift and hom Appliui black andweble enrrr Sub Categury: SWEATSHANTS Brand
shoutder pit-bcclc
Hasus, Composaionn cotton..
Zero
Retrieval
Fashion
Reference
Retroved Image
ReVV JCLR'24
Reference
Generated
DeCop, ICER
Modilynng Text prppir anows on.it
+3.4
Description short slowe cotton Jeseyt- akot wiits. cownisss collar patch pocket or chost cutestt sievee lonal solcing Brand Socal Sub- Catagory: I sbits Compostion body prlyerve bitconttio polywttr
+4.6
CIRRVI +Ours
DaCop+Ours
14:
Capbec, EMNLP/22
14.8
CopDac+Ours
Figure I: Model performance downstream tasks. The first row shows the direct application of our model cross-modal retrieval. The second row shows that, when replacing the CLIP encoders with ours in existing SOTA models in zero-shot text-guided image retrieval and fashion image captioning, their results can further improved notably. (We show R@ results for cross cross-mode modal retrieval, average R @ 10 for text-guided image retrieval, and @@4 for image captioning.
Abstract
When learning vision dision-language models (VLM) for the fashion domain, most exist- ing works design new architectures from vanilla BERT with additional objectives, perform dense multi-task learning with esshion-speciff tasks. Though progress has been made, their architecture or objectives are often intricate and the extendibil- ity is limited. By contrast, with simple architecture (comprising only two unimodal encoders) and just the contrastive objective, popular pre-trained VL models (e.g., CLIP) achieve superior performance in general domains, which are further easily extended to downstream tasks. However, inheriting such benefits of CLIP in the fashion domain is non-trivial in the presence of the notable domain gap. Empir- ically, we find that directly finetuning on fashion data leads CLIP to frequently ignore minor yet important details such logos and composition, which critical in fashion tasks such as retrieval and captioning. this work, to maintain CLIP's simple architecture and objective while explicitly attending fashion details, propose E2: Easy Regional Contrastive Learning of Expressive Fashion Represen- tations. E2 introduces only few selection tokens and fusion blocks (just 1.9% additional parameters in total) with only contrastive losses. Despite lightweight, in our primary focus, cross-mas modall retrieval, E² notably outperforms existing fashion VLMs with various fashion -sphiiii objectives. Moreover, thanks to CLIP's widespread use downstream tasks in general domains (e.g., zero-shot composed image retrieval and image captioning), our model can easily extend these models from general domain the fashion domain with notable improvement (Fig. D To conduct comprehensive evaluation, we further collect data from Amazon Reviews build new dataset (Fig. for cross-mout moosl retrieval in the fashion domain.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

General Domain
Fashion Domain
Description: Long siaeve danim shirt in dark nany. Spread collar Button closure at tront tcaturing rib knit trim strped blun, white and black, Patch pocket and leather polic
C
Sincle-buttoon button chest.
Sitver-tone
Tonal slitching
Category: SHIRTS.
Brand: Rat Simons.
Season: FW22016 (FallWiiter)
Composition: 96% cutton, 4% polyurothane.
girl climbs huge man riding bike on hill.
CLIP-FT
Ours
Figure 2: Illustration of domain gap and attention results CLIP and E², General domain data often consist of a short caption which describes few objects in an image, while fashion data come with description and meta information (tag entities) single product. CLIP: Image Tokens with maximum attention values (in each attention head) with the global token are marked yellow. E²: Selected image tokens by selection tokens during the second stage are colored. Blue: Brand. Orange: Season, Red: Sub-category. Green: Composition.
1
Introduction
There has been a long research line for vision-language learning in general domain 13 34
23
26
40
52
[53]
Recently, cross-modal retrieval in fashion domain is receiving increasing
attention 10
27
12
56
13
38
22
14 12 Most existing fashion vision-language models (VLM)
12
38
14
T0
12 56 design new vision-language models based vanilla BERT [6].
Existing studies commonly train BERT-based models with Masked Language Modeling (MLM), (Image-Text Matching) ITM, Image-Text Contrastive Learning (ITC), Masked Image Modeling (MIM) to their specific architectures. Various techniques tailored to fashion domain have been proposed, including learning extra fashion- specific tasks, attention mechanism [38, or additional modules for fashion feature learning [14]. FashionSAP (14] and FAME- ViL [33 perform a fashion- specific multi-task learning with various fashion tasks in addition to cross modall retrieval, such as category recognition. However, the learning of existing BERT-based or multi-task fashion VLMs is often intricate with their complex architecture additional fashion-specific objectives. By contrast, with simple architecture that comprises only two unimodal transformer encoders, and single contrastive learning objective, contrastive sagguage-imagee pre-training models such as CLIP (34] exhibit outstanding performance from cross-modal retrieval to a wide range of downstream 42 20 33.. Motivated by the simplicity and effectiveness of CLIP, we look forward to learning a model that inherits such benefits of CLIP in the fashion domain, which could be simple, lightweight while highly effective. While directly finetuning CLIP (FashionCLIP (4) is an intuitive solution, it is deficient in presence of the notable domain shift (55). Consequently, Ma et al., (27) uses CLIP backbone and improves it by using additional text encoders to mitigate the word ambiguity in fashion language. However, they ignore the uniqueness of visual learning in fashion domain.
tasks 124
Different from data in general domain, product images and descriptions in fashion domain are unique in several aspects. As illustrated in Fig. in general domain, an image contains only a few distinctive objects, and text descriptions are more concise and general. However, in fashion domain, a product image usually includes only one foreground object but with rich details. Besides, the fashion text often provides group of metadata (tag entities) [6. 36). such Composition, Brand, Description, Sub-category, etc. When directly finetuning CLIP on fashion data, we find it tends to give more attention regions that are closer Consequently. it misses details associated with tag entities such as composition and logo, which represent more of local view and are critical for distinguishing visually similar items.
global view, e.g., dark regions in dark clothing (Fig. and 14).
To quantify the capability of visual representation in recognition of product details, we design an exploratory entity classification task based linear probing: fitting linear classifier image embeddings from different models respectively, with tag entities used as labels. Results in Fig. 3 show that, the classifier which learns from fine-tuned CLIP (CLIP-FT) embeddings are less effective, indicating that CLIP-FT embeddings contain less entity-related information, i.c., CLIP-FT is ineffective in extracting relaty information from fashion images. Details are available in Appendix where we provide a more comprehensive analysis.

---

① AmaazonFashionn (Ours)
Fashion
Dataset Statistics
Description: Fveedhwetty Mather Groem Dress 14 Black and While
Description: Shortslewss black Singleshaatter gold Adymmetrcc seam walst aik livin in beige Toral sttshing. Calegory: LONG DRESSES Brand: Stwks Mecarthey Season PW2008 Compostton Rody 70's cotton 30% nyfon. Lining: 100% sik.
Dataset
(Dur))))
?
Image-Text Paris
1.3M
28.8K
Brand Count
38211
570
Brand: Eive-Priity Pritt
Product Count
544713
67686
Category: Clothing
Avg. Desc. Length
11.43
92
Figure 4: Comparison of FashionGen [36] and AmazonFashion (Ours), which is notably different in size, diversity, language style and image scope (close-up shots of products are included). We compare it with more datasets 7 [5], and show more details (e.g., clustering [52] in Appendix D.
Then it is natural ask: is it possible to have a CLIP image encoder that learns richer visual representations towards tag en- tities without eroding its learnt knowledge?
CLP CLPY
Towards this end, we propose model: E²: Easy Regional Contrastive Learn- ing of Expressive Fashion Representations. Without modifying the Vision Transformer (ViT) g structure of the image encoder, E learns richer representations with the guidance of region contrastive loss. Meanwhile, it explicitly pays more attention to details in images by selecting most relevant image patch tokens that contain rich detailed information and fusing them with selection tokens during the forward propagation in ViT. Our contributions are follows:
Brand
Season
Sub-category
Compostioo
Figure 3: Linear Probing Results on FashionGen. The more informative embeddings are, the higher accuracy a classifier obtains.
(1) We first reveal that the image encoder of pre-trained CLIP is biased towards visually dominant regions of product and consequently ignores small but critical details when directly fine-tuned on fashion domain. Furthermore, we present the first simple yet effective solution to this problem by allowing the model to learn more fine- mne-grained visual representations towards tag entities. (2) Motivated by the observation in (1), we propose a concise regional contrastive learning framework comprising only two unimodal encoders and just contrastive learning objectives, while it learns richer visual representations. (3) We conduct extensive experiments with existing benchmark datasets, including our new benchmark dataset (Fig. 4 for fashion cross-modal retrieval, which differs from the popular benchmark FashionGen (36) in notably larger size, a wider variety of brands and products, more concise and general descriptions, and more diverse image scopes, making it more challenging and more practical. (4) Inheriting the benefits of the widely used CLIP, our model can easily extend models from general domain to the fashion domain with notable improvement (Fig in downstream tasks (e.g., zero-shot composed image retrieval and image captioning).
2 Related Work
Text and Image Matching. Text and image matching aims to semantically align the text and image. Various BERT- based vision language models (VLM) 40 towards this end. Different from previous studies, CLIP 64 introduces large-scale vision language pre-training framework, which learns from 400 million image-text pairs with contrastive learning. Despite of the simplicity of its structure, CLIP is surprisingly effective in visual-language learning
26
41
25
14
12 [0] are proposed
and various downstream tasks 24 42 20
Fashion Cross-modal Retrieval. Different from general domain, the fashion data involves large amount of Nomain-specitfe information with richer details than data from general domain, such as the brand, material, texture, composition, various of minute design differences, etc. Besides, the fashion text is composed of description and a group of tag entities (meta information), which describes the meta information of products, such as its sub-category, season, brand, to name few. Wang et al. 455 enhances the task with knowledge graph [77 B7 31]. A line of research pre-trains BERT-based that are tailored for fashion data. Fame- ViL. [33 and FashionSAP [14 further formulate multi-task learning framework with extra fashion-specific objectives. Different from existing BERT-based models, Chia et al. and Ma et al. [27] are built upon powerful CLIP.
models [00 (6) 12 14 12
While Chia et al. 4 directly finetune CLIP for continual learning [2] with fashion data, Ma et al. 27
3

---

improve its language learning with additional text encoders to diminish word ambiguity. However, they do not consider the visual discrepancy. To better learn from fashion images, which contain richer detailed information, we improve CLIP from the perspective of visual learning, enabling its image encoder to learn more fine-grat lineggrined representations for better image-text aligoment.
Zero-shot Text-guided Image Retrieval. In Text-guided Image Retrieval (TGIR), also known as compositional image retrieval (CIR), users perform interactive dialogue to refine given query image toward retrieving specific items. Classic models often employ custom models that project text-image pairs into common embedding space. With the advance of VL foundation models (e.g.. CLIP), interest in CIR has surged, especially in zero-shot settings without task-specific models [20]. We show that when combined with E", which learns more fine-grained representations towards fashion product details, their performances can be further improved.
aims to generate image/video captions without human-annotated data. Different from above works, built upon CLIP, DeCap [24] and CapDec [30] use text-only data train a decoder from seratch. We show that when combined with E², which learns more fine-grained representations towards fashion product details, their performances can be further improved.
Zero-shot Image Captioning. Zero-shot captioning 3 46
We provide a detailed discussion on our innovation compared with existing works in Appendix G
Methodology
3.1 Contrastive Language-Image Pre-training
Instead of learning from predicting fixed set of predetermined object categories, i.e., the classification task, CLIP (Contrastive Lingaage-maaee Pre-training) (34) directly learns visual representations from raw text, and is trained on 400 million image-text pairs with contrastive learning. Specifically, given a batch of N image-text pairs {(I, T.) images and texts are encoded as d-dimensional embeddings by the image encoder (-) and the text encoder h ht"(.). Denote the image embedding and text embedding as 2 and z!" respectively. During training, CLIP learns image-text matching from of N x N possible combinations by maximizing the similarity scores of N matched pairs while minimizing the scores of the rest N2 NN mismatched pairs. The cosine similarity score of an image-text pair is calculated as z @ zT (i,j E {1, ..., N)). In practice, CLIP optimizes cross-entropy loss over the N X N similarity scores matrix, namely contrastive loss, denoting as:
Lemtra (((2/1=7)) =))) L12t + LT21,
exp(=]] x(/TT) where L12r = Ei=1 log - exp /T/)) and Lr21 N EN log >X (xp (x/T)/
T is the temperature scalar. CLIP proves its effectiveness on various tasks in general domain. However, when adapting it to fashion domain, the fashion data due to its uniqueness in richer details and more compact layout. To overcome the challenges, we make full use of tag entities and learn more fine- -raiind representations with explicit (1) token fusion and selection, and (2) region contrastive learning.
particularly difficult for CLIP to effectively learn from
3.2 Regional Contrastive Learning of Fashion Representations
We present details of our framework, Easy Regional Contrastive Learning of Expressive Fashion Representations (F) (F²) in Fig. 5 The core idea of F is "Easy" and "Fxpressive": it inherits the (1) simple design of CLIP with only few inserted fusion blocks and selection tokens in the vision encoder, and (2) the simple learning objective: only contrastive learning objectives are used. Yet E more effective than existing large fashion VLMs with various objectives.
In the following, we first describe the overall framework of E and then introduce the selection tokens, which are key elements for our (1) token fusion and selection, and (2) region contrastive learning. After that, we explain each component and finally summarize the whole training process.
Framework. We build E² upon CLIP 34 where we keep its text encoder unchanged and facilitate its image encoder with proposed selection tokens, fusion blocks and region contrastive loss. Similar to CLIP, given an image-text pair, F² learns one global embedding for each for the calculation of similarities for contrastive learning.

---

Contrastive Ioe
Fread Selection Telans
Region Contractiv Ioe
Seleeeed Image
CLIP Text Encoder
Raych Tokens
Tranformer Layer (-1)
(Selection)
Imas lokera
Fusion Block
wey Speni
Tag Entities
Image Feoch Tokene
Selection Tolens
bg
Layers (-3)
Fusion Block
Selectin tolens
(hoell
Sub
und
Mr4 Selcction tokors
Fusion Block
Pesting
Pooking
Pooling
Transformer Layers (+8)
Separated Average Pooling
Figure 5: (a) Framework. E contains an image encoder and text encoder. The image encoder consists CLIP transformer layers with inserted fusion blocks, where selection tokens update themselves with most relevant image tokens. After the second stage of token fusion, only the global image token and selection tokens are kept as input for the last transformer layer. Selection tokens further learn epeity-ppecific knowledge with region contrastive loss. (b) The Architecture of Fusion Block. In fusion block, each selection token selects one most relevant image patch tokens and update itself with the averaged embedding of itself and the selected token. (c) Visualization. Logo is covered by blue masks. Front zipper area indicates its season (Fall/Winter 2016) and left front area with sleeves suggest the sub-category (shirts). Note that the figure is only for illustration, in experiments, we consider C = 4 tag entities and assign S = 2 selections for each.
Given image, it is first split into P non-overlapping patches, which are linearly projected into a latent space, denoted as {p.)--- In addition to P image tokens, we further propose of selection tokens categorized into C categories, with S tokens for each sub-category, denoted as [x. For instance, for the image encoder in Fig. 5 C 3 and S 2, The two blue, red and yellow selection tokens are associated with the tag entity Brand, Sub-category and Season, respectively. To capture detailed information in image patches, each selection token updates itself with one most relevant image token in fusion block. Note that selection tokens are supervised by the region contrastive loss. We will discuss it later in this section.
Selection Tokens.
Multi-Stage Image Token Fusion and Selection. To learn more fine tineggrined visual representations with the help of tag entities in fashion language, we use a group of selection tokens to iteratively select most informative images tokens and merge themselves with the selected ones. In this way, selection tokens contain rich information about the details of their associated tag entities. For more fine-grained interactions, we drop less informative image tokens and only keep the global embedding and selection tokens as the input of the last transformer layer. As shown in Fig. 5 we perform multi-stage token fusion and selection to obtain the final global embedding. Given an image, we first obtain P image tokens {p.)-1- Then they are concatenated with set of selection tokens { ) and input to the image encoder, where we perform a multi-stage token fusion and selection. In each stage, input tokens sequentially go through few transformer layers and fusion block. Formally, suppose there are L stages, and during the 1-th stage, we denote the input tokens as {p!) and CxS The information propagation with each group of transformer layers is performed:
S
(p))
( Cx) Transffrmer(((p]) ( (x1)
where [:] means concatenation. Then the obtained image and selection tokens are fed to the fusion block, where each selection token selects one most related image token and updates the embedding of itself with the selected image token:
(3) After that, the output tokens serve as input tokens for stage (+1 if it is not the last stage. After obtaining the output tokens from the last stage L (via Eq, 3 with (=L), we only keep the global embedding pe (+1 and selection tokens (**+1) ),x as the input to the last transformer layer. To enrich the global embedding with fine-graines features related to given tag entities, we enforce it to
(p) +1 (
CIx Je = FusionBlocck (8)
5

---

focus on interacting with informative selection tokens by dropping less relevant image tokens to avoid their distractions. This step is critical for the image encoder to effectively learn more fine-grained visual representations with tag entities. As shown in Fig. 5 a). the last transformer layer is applied on the kept tokens to obtain p/+1 Finally, the global image embedding z is obtained by applying MLP to it:
(4) Fusion Block. In fusion blocks, selection tokens are fused with their most relevant image patch tokens to enrich themselves with entity-specific (e.g.. brand, composition, etc.) visual information. Given a group of image patch tokens {p]).=r. denoting as a matrix P', and a group of selection tokens {s]] denoting as a matrix S we use selection tokens as queries to select the most relevant image tokens, which we call hard atlention. For each selection token, it is updated by averaging itself with the selected image patch token. Specifically, denoting Q', K and V are linear projections of S and P respectively: Q' WGS K P and V W.P', Attention weight matrix A' is calculated by:
Po
1++1 Transformer((p] I { { 1E= and x' = MLP(P!+*)
exp (Q,.K
K + 7.)
A
(5)
Ek=1 exp Qi K + 7/k
where y is the i.i.d random sample drawn from the Gumbel(O, 1) distribution. To explicitly select one most similar image patch token for each selection token, we reformulate the attention weight matrix by turning each row A.: into a one-hot representation with assigning one to the term with highest similarity score and zero to the rest. As the argmar operation is not differentiable, use the gumbel-softmax and straight- through trick 8
48
AL. soe-hot(ann) (argmax A + Al.. - sto)(A,,),
(6)
where the operator stop stops the gradients propagation. With one-hot vectors in the attention weight matrix A', each selection token can pick up one corresponding image patch token and update itself with the selected one via:
1t S V A'.
(7)
In fusion blocks of different stages, selection tokens constantly select most relevant image tokens explicitly and enrich the representations of themselves with selected token embeddings for better selection and fusion in the next stage.
At the last stage, the output of the fusion block only contains selection tokens, which already contain rich fine-graing grained information about associated tag entities (i.e., brand, composition, season and categories, etc.). They are input the last transformation layer with the global image token, that the global token can effectively interact with them without distractions from irreverent patch tokens. In this way, the global token better captures details of fashion images.
Region Contrastive Learning with Selection Tokens. As each selection token is associated with tag entity and it aims to select most relevant image tokens in fusion blocks, selection tokens are further supervised with the region contrastive loss. Specifically, assume we have group of selection tokens (xx, which are categorized into C categories, with S tokens for each sub-category. For better illustration this subsection, we reformulate them as { where E { ('brand', "composition' .... (C categories in total) and s E 21, 2..,, S}. After going through the last transformer layer, each set of selection tokens { ===1 with the same associated sub-category e is fed to its corresponding average pooling layer (Fig. [5 rright), that selection tokens from the same sub-category are formed to a more comprehensive and informative single pooled embedding $ fimol as:
8 Final = AvgPool( {vgg e E ("brand",...)
(8)
Similar to the contrastive learning process with global image embeddings and global text embeddings introduced in section 3.1 each & Fimal learns to match the corresponding ground truth tag entity z((e) via the region contrastive loss by replacing the global image embedding with the corresponding pooled selection token in [
Cregion (1(27 E(() T(s)
f ) L12T + Cr21-
The region contrastive loss explicitly aligns each iinat with the corresponding ground truth tag entity T(e) for sub-category With region contrastive loss, pooled selection tokens are distinguishable

---

Model
RGl R R@10 R
Image Text
RG5 R@ MeanR@1
Imrge
Model
R@1
Image
R
R
10
RG
Text Image
R
R@/10
Summ
ALBEF
22
41.7
$1.0
46.2
CLI--PPT
M
15.7
43.1
60.0
16.6
43.3 59,4
238.1
Syn:Mask 3X
55.4
64.0
FashionBen
46.3
46.4
251.1
FashionSAP 14
54.3
77.3
83.2
62.8
83.9
90.5
451.8
58.5
KaleidoBert
56
27.9 60.0
68.3
33.8
60.6
68.5
319.1
FashionVill
12
42.8
71.5
80.6
51.3
75.4
84.7
406.5
47.1
FLD-VLP
28
64.3
86.8
93.5
58.7
84.5
91.6
479.7
FashionCLIP
54.3
85.5
92.8
54,4
85.9
92.3
463.2
54.4
FAME-VIL
65.9
91.9
97.2
62.9
87.4
93.5
498.8
HIUP-FT
59
55.5
86.1
93.2
55.8
86.5
93.1
470.2
55.7
HIUP-FF
14
68.1
95.0
98.5
65.3
93.6
98.2
518.7
CLIP-FT
55.2
92.9
55.4
92.9
CLIP-FT
93.2
515.9
Ours
62.8 89.7 95.3
64.5
90.1
95.5
497.9
63.7
Ours
76.2 96.8
99.1
73.1
95.3
99.1
538.6
Table 1: Retrieval performances (full evaluation) on Table 2: Retrieval performances (sample-106 100 Fashion-Gen Gent. (*) denotes results from our implemen- evaluation) Fashion- Gen. (*) denotes re- tation. As the code and pre-trained model of FILIP sults from our implementation. are not released yet, we implemented the model and initialize with pre-trained CLIP weights.
towards tag entities, which means learnable selection tokens can effectively select and fuse themselves with tag entity-specific image patch tokens given fashion images. Finally, the global image embedding learns more fine-grained representation by: (1) interactions with these selection tokens in transformer layers, and (2) explicit token selection process before the last transformer layer, which filters less relevant patch tokens and helps the global embedding to concentrate more on certain regions.
Remark. Some semantic segmentation models, e.g., Seg [39] and GrT (88 also involve additional tokens. But the usage is notably different. As their motivation is to group image tokens to larger objects, each image token is assigned to an additional token, where softmax applied to additional tokens. Because our motivation is to pay more attention to details of an object. each additional (selection) token selects a most relevant image token, where softmax is applied to image tokens. We carefully discuss the differences in detail in Appendix H
4 Experiments
Datasets. For a fair comparison, we first evaluate our model on the benchmark dataset Fashion- Gen [36), following existing works [10] 56 12 27]. Besides, we also collect text descriptions and product images in fashion domain from Amazon Reviews (2 and build large-scale fashion dataset which contains 1.3M image-text pairs, where we use 910K and 390K pairs for training and test, respectively. It is more challenging dataset as the text descriptions are briefer and more general than FashionGen. We refer this dataset as AmazonFashion. More details are available in Appendix D
Settings. E is initialized from the pre-trained CLIP [34] (ViT-B-32 (9) We use two selection tokens for each tag entity (S 2). as Fig. shows. We present the validation of this choice in later experiments. More detailed configuration and hyper-parameter setting are available in Appendix A
Evaluation. Our model is evaluated in various downstream tasks. For retrieval tasks, we perform two kinds of evaluations. The positive candidate is the ground truth item, while the negative can- didates are either 100 randomly sampled unmatched items (referred as sample- 100 evaluation) or all unmatched items in the dataset (full evaluation). Following 10 27), the evaluation metries are Rank @1 1, Rank@5 Rank @ 10 and Sum 100 @ + Rank 5 Rank@ 10).
Note that full evaluation, which is suggested by latest works 27 12 is more challenging and practical and consistent with practical retrieval tasks. We compare with basclines with full evaluation unless they not compatible with full evaluation, where run Sample- 100 evaluation instead.
4.1 Comparison with State-of-the-Art
Cross Model Retrieval. For full evaluation on FashionGen, we compare our model with existing fashion VLMs and finetuned CLIP-based models. To compare with more baselines which are not evaluation. Because
compatible with full evaluation, following [0. [56]. we also do a Sample- CLIP-FT and FILIP-FT outperform other baselines by a large margin, we compare E² with the two closest baselines on AmazonFashion with full evaluation. Table F²2 notably outperforms other basclines in full evaluation, manifesting the effectiveness of paying more attention certain regions
7

---

Deccriptio: Cotton and Sen-ddedwwwwen incuers
Description Guu leaher in Carry handle at
bluck Theee podeet abyling
Iwdn eduarabii shoulde straps pr--buréle fastering
Signetue ertching novite back Tond staching cate-ara
nool
Srin
Gueri [r)
Howdfer
Swine Lauaent
Fendi
foce. Tworway sip catagery:
M
Magiss Compositioe
sollens
DX
Compositior Leather
(b)
Kara
Guod
Jimnr
Choe
MeCartrey
Stella
Sophie
hulme
Figure 6: T2I retrieval examples. For each example, the query text is displayed on the left. The first row is the top-5 retrieval result by E² and the second row is the result by CLIP-FT. Ground truth is boxed in green. For each product image, its Composition (a) or Brand (b) is marked below. For top-1 retrieved item, we mark its tag entity red if it is not consistent with the query text.
that contain rich information about tag entities. Table Sshows shows our model also outperforms baselines in sample-100 evaluation by a large margin. On the more challenging large-scale AmazonFashion dataset, E² consistently achieves better performance than competing baselines (Table 3)
In Fig. 6 we present examples of retrieval results by E² and CLIP to illustrate in which cases CLIP fails to differentiate minor differences towards visually similar products. In Fig. Ga). both CLIP and E² retrieved visually similar black trousers. While E² ranks the ground truth sample first, CLIP ranks visually similar product higher without considering its composition. Fig. ((b) shows E² ranks the Gucci backpack first by recognizing its logo at face while CLIP ranks the Kara backpack first incorrectly. More visualizations are available in Appendix B
Zero-shot Image Captioning (ZS-IC). Built upon CLIP, E² can be easily extended to popular SOTA CLIP-based zero-shot image captioning models [24 [2). DeCap [24) and CapDec [30] share similar high-level ideas: during training, language decoder trained reconstruct the text input, where the CLIP text encoder serves as the encoder. During inference, with the CLIP image encoder, given image is first encoded generate captions. We train the decoder on text-only data from FashionGen but with our E² as the backbone encoder instead of CLIP. Results are shown in Tab. 4 As learnt features by E² contain more fine-grained information towards product specifications, generated captions with E² better matches ground truth captions, leading to high scores. We show the example Fig 7 Vanilla DeCap tends to make more mistakes towards details, such as brand, composition. While our improved DeCap with E² constantly yields better results.
CLIP feature, which later fed to the pre-trained decoder to
Zero-shot Text-guided Image Retrieval (ZS-TGIR). CIReVL [20 exploits pre-trained vision- language models (CLIP) alone with an LLM for ZS-TGIR without training. Similar the case in ZS-IC, when replacing the CLIP encoders with E², CL is easily extended to fashion domain, consequently the model performance notably improves in Tab. 8
Remark. EI-CLIP 277 and fine-grained CLIPs 44 are not necessarily our baselines, as our focuses
different and orthogonal to each other. Still we have a comparison demonstrate the uniqueness and advantages of our model. Detailed results and discussions are presented in Appendix H
4.2 Ablation and Further Analysis
To study the effectiveness of each module and how much each group of selection tokens contribute to the our model, we design ablation studies from the two perspective: architecture and selection tokens. Architecture. We first ablate fusion blocks, where the input and output of each transformer layer are
Image to Text
Text to Image
Model
B@4
Model
SumR
R@1 R@S R@10 R@ 1 R65 R@10
CapDec (30), EMNLP'22
14.23
5.14
14.98
15.02 17.87
FILIP
49
6.2
17.8
25.8
6.2
18.1
25.9
100.0
CapDec w/ g2
19.13
10.13
18.91
18.77 23.82
CLIP-FT
3
6.1
17.7
25.8
6.2
17.9
25.8
99.5
DeCap 24, ICLR'23
16.88
16.21
19.42
E (Ours)
7.5
20.3
28.4
7.4
20.2
28.4
112.2
DeCap E2
21.35
12.18 20.93 20.60 25.03
Table 3: Retrieval performances (full evaluation) on AmazonFashion. (*) denotes results from our implementation.
Table 4: Zero-shot image captioning results on FashionGen with BLEU@4 (B), CIDEr (C), METEOR (M), ROUGE (R).
8

---

Ground Trush
DeCap ICLR23
DeCop Our
Desciption Long sleve french terry hood in
Longslevee Dowstring in hood Hultodoor gronic printed in here. Treal
Descrition: Long alevee French pulkose
Descipione Long cleove franch teryy hood in Drowsting
when Rbb init creanedi orloo, cutts, and
ithood Mubroo or gopnic printed st from and embroddend redandings embondeed stirningg
logo entmaateeeeed hore hem tndt porneciin at Locke pop Catogory H5
Nultinnkrr grapiir pertedin sonee block.
sicevs Ri
Brand: Commic Garpora Pla.
Simoro
Ziups
Compastion sior cotton.
Compostian cortinn
Compas cotion
Seacon: S52018
Seasor 552017
Seavos: S01018
Figure 7: Zero-shot Image Captioning examples. We mark the wrong captions red, and mark a caption green if it exactly matches the ground truth.
Method
Re 10
Shirt
R455 R 10 Re50 Re 10 RD Re 10 Re50
Dress
Toptee
Average
Model
R@J 1 R
Image to Text
R
R@1
Text to Image
R@5 R@:10
Full method
62.8
89.3
95.3
64.5
90.1
95.5
PALAVRA JCCV/23
21.49
37.05
17.25
35,94
20.55
38.76
19.76
37.25
SEARLE ECCV122
24.44
41.61
18.54
39.51
25.70
46.46
22,89
42.53
wlo Fusion Blocks
Select. Tokens
58.8
55.2
88.7
85.7
94.9
58.5
88,9
94,9
CIReVL. ICLR224
28.36
47.84
25.29
46.36
31.21
53.85
28.29
49.35
Region
56.3
CIReVI. w/ E (Ours)
32,02 50.73
28.47
49.39
34.72
31.73
52.25
w/ FILIP backed
62.1 89.8
95.9
62.9
9044
95.9
Figure 8: Results Fashion-IQ dataset
Figure 9: Ablations FashionGen.
all of the image batch tokens and selection tokens. Without the explicit selection and fusion process in the fusion blocks, the selection tokens, which are supervised with region contrastive loss and carry rich information towards tag entities, are still interacting implicitly with image patch tokens and the global token in ViT layers. In this way, we assume that, even without fusion blocks, the global token still learns richer information about tag entities and lead to better retrieval performance. Then we are curious to see how much the explicit token selection and fusion process benefit the learning process. in addition to the potential improvements from this implicit interaction in transformer layers. Table 9 shows both modules are critical. We also ablate our region contrastive loss (RCL).
We also study how each group of selection tokens contribute to the model performance. While all groups of selection tokens improve the model performance, their contributions are different. While Composition and Brand are more helpful, Season and Sub-category contribute slightly less. One potential reason is that they are easier to be visually distinguished than product texture and small logos, which especially require our fusion and selection process. We have detailed discussion on their effectiveness in Appendix and on parameter-effiiency in Appendix !
Group of Selection Tokens.
Parameter Sensitivity. We also examine the impact of batch size, which has a significant influence on the performance of contrastive learning. E² consistency surpasses CLIP-FT across range of batch sizes, with greater improvement over CLIP-FT as batch size decreases. We present detailed results and analysis in Appendix C
Number of Selection tokens. In experiments, we use two selection tokens for each tag. It is reasonable as we not selecting and fusing image patches in raw pixel space, instead, we conduct it with image patch (token) embeddings within ViT layers in contextual embedding space, where each token contains rich context/neighbor information. In fact, one token can already represent large area if considering its neighbour information. Our choice of two tokens empirically enough for all tags. We quantitatively validated this choice in Appendix
5 Limitations
Built upon the pre-trained foundation VL model (e.g., CLIP), F. could be bottlenecked by the qualify of its large-scale pre-training in general domain. Besides, as our regional contrastive learning with fusion blocks and selection tokens explicitly aligns regions in the input image with the corresponding tag entities from text, intuitively, the quality of the provided tag entities can influence our model performance, Although in most cases, fashion data contain such information (e.g., brand, composition etc.), it is still limitation if only datasets with poor tag entities are available for finetuning.
Conclusion
In this paper, we propose a simple yet effective framework on learning fashion representations, and first emphasize the importance of learning fine-grained visual representations when applying CLIP to fashion domain. To achieve it, we further propose F²2 with selection tokens and region contrastive loss to enforce extra attention to details. Experimental results prove the effectiveness of E

---

Acknowledgement
The work is in part supported by the National Science Foundation under Grants 21S-2316306 and CNS-2330215, and a gift from Adobe.
References
[1] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Micch, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716-23736, 2022.
[2] Alberto Baldrati, Lorenzo Agnolueci, Marco Bertini, and Alberto Del Bimbo. Zero-shot composed image retrieval with textual inversion. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 15338-15347, 2023.
[3] Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale mmage-text pre-training to recognize long-tail tail visual concepts. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 3558 3568, 2021.
[4] Patrick John Chia, Giuseppe Attanasio, Federico Bianchi. Silvia Terragni, Ana Rita Magalhäes, Diogo Goncalves, Ciro Greco, and Jacopo Tagliabue. Contrastive language and vision learning of general fashion concepts. Scientific Reports,
Nov 2022.
[5] Niv Cohen, Rinon Gal, Eli A Meirom, Gal Chechik, and Yuval Atzmon. "this is my unicorn, fluffy": Personalizing frozen vision-lane langugge representations. In European conference on computer vision, pages 558-577. Springer. 2022.
[6] Jacob Devlin, Ming-Wo Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirec- tional transformers for language understanding. arXiv preprint 1rXiv:1810.04805, 2018.
[7] Xiao Dong, Xunlin Zhan, Yangxin Wu, Yunchao Wei, Michael C Kampffmeyer, Xiaoyong Wei, Minlong Lu, Yaowei Wang, and Xiaodan Liang. MSproduct: Self-harmonized contrastive learning for e-commercial multi-modal pretraining. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 21252-21262. 2022.
[8] Xuanyi Dong and Yi Yang. Searching for robust neural architecture in four gpu hours. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 1761-1770, 2019.
[9] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words:
Transformers for image recognition at scale, arXiv preprint XrXiv:2010.1 19929, 2020.
[10] Dehong Gao, Linbo Jin, Ben Chen, Minghui Qiu, Peng Li. Wei, Hu. and Wang. Fashionbert: Text and image matching with adaptive loss for cross-modal retrieval. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2251-22601 2020.
[11] Shijie Geng. Jianbo Yuan, Yu Tian, Yuxiao Chen, and Yongfeng Zhang. Hiclip: Contrastive language-imas pretraining with hierarchy-awe aware attention. ICLR, 2023.
[12] Xiao Han, Licheng Yu, Xiatian Zhu, Li Zhang, Yi-Zhe Song, and Tao Xiang. Fashionvil: Fashion-focusel vision-and asiion-andlanggugee representation learning, In European Conference Computer Vision, pages 634-651. Springer, 2022.
[13] Xiao Han, Xiatian 7hu, Licheng Yu, Li Zhang, Yi-Zhe Song, and Tao Xiang. Fame-vil: Multi-tasking vision-language model for heterogencous fashion tasks. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pages 2669-2680. 2023.
[14] Yunpeng Han, Lisai Zhang, Qingcai Chen, Zhijian Chen, Zhonghua Li, Jianxin Yang, and Zhao Cao. Fashionsap: Symbols and attributes prompt for fine-grained fashion vision-language pre-training. In CVPR, 2023.
[15] David Hardoon, Sandor Szedmak, and John Shawe- Shawe-Taylor. Canonical correlation analysis: An overview with application to learning methods. Neural computation, 16(12):263--266 2004.
[16] Ruining He and Julian McAuley. Ups and downs: Modeling the visual evolution of fashion trends with one-class collaborative filtering. In proceedings of the 25th international conference on world wide web, pages 507-517, 2016.
10

---

[17] Yuncheng Hua, Yuan-Fang Li, Guilin Qi, Wei Wu, Jingyao Zhang, and Daiqing Qi. Less is more: Data- efficient complex question answering knowledge bases. Journal of Web Semanticx, 63:100612, 2020.
[18] Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In European Conference on Computer Vision, pages 709-727. Springer, 2022.
[19] Andrej Karpathy and Li Fei-Fei. Deep visual-semantic alignments for generating image descriptions. In Proceedings of the IEFE conference on computer vision and pattern recognition, pages 3128-3137, 2015.
[20] Shyamgopal Karthik, Karsten Roth, Massimiliano Mancini, and Zeynep Akata. Vision-by-language for training-free compositional image retrieval. arXiv preprint 10.0099,,, 2023.
[21] Juncheng Li, Xin He, Longhui Wei, Long Qian, Linchao Zhu, Lingxi Xie. Yueting Zhuang. Qi Tian, and Siliang Tang. Fine-grained semantically aligned -aiounlanguage pre-training. NIPS, 2022.
[22] Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. Advances in neural information processing systems, 34:9694 9705, 2021.
[23] Kunpeng Li, Yulun Zhang, Kai Li, Yuanyuan Li, and Yun Fu. Visual semantic reasoning for image-text matching. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4654-4662, 2019.
[24] Wei Li, Linchao Zhu, Longyin Wen, and Yi Yang. Decap: Decoding clip latents for zero-shot captioning via text-only training. International Conference on Learning Representations, 2023.
[25] Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong. Furu Wei, et al. Oscar: Object-semantic aligned pre-training for -siiunla tasks. In European Conference on Computer Vision, pages 121-137. Springer, 2020.
[26] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lec. Vilbert: Pretraining task- agnostic visiolinguistic representations for lan-auugee tasks. Advances in neural information processing systems, 32, 2019.
[27] Haoyu Ma, Handong Zhao, Zhe Lin, Ajinkya Kale, Zhangyang Wang, Tong Yu, Jiuxiang Gu, Sunav Choudhary, and Xiaohui Xie. Ei-clip: Entity-aware interventional contrastive learning for e-commerce cross-modal retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1851-18061, 2022.
[28] Suvir Mirchandani. Licheng Yu, Mengjiao Wang. Animesh Sinha, Wenwen Jiang, Tao Xiang, and Ning Zhang. Fad-vlp: Fashion vision-and language pre-training towards unified retrieval and captioning. arXiv preprint ar Xiv:22 10. 15028, 2022.
[29] Jianmo Ni, Jiacheng Li, and Julian McAuley. Justifying recommendations using distantly- labeled reviews and fine-grained aspects. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLPP) pages 188-197, 2019.
[30] David Nukrai, Ron Mokady, and Amir Globerson. Text-only training for image captioning using noise- injected clip. Findings of the Association for Computational Linguistics: EMNLP, 2022.
[31] Daiqing Qi. Handong Zhao, Xiaowei Jia, and Sheng Li. Revealing an overlooked challenge in class- incremental graph learning.
[32] Daiqing Qi. Handong Zhao, and Sheng Li. Better generative replay for continual federated learning, 2023.
[33] Daiqing Qi, Handong Zhao, Aidong Zhang, and Sheng Li. Generalizing unseen domains via text-guided augmentation. In European Conference on Computer Vision (ECCV), 2024.
[34] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748-8763. PMLR, 2021.
[35] Yongming Rao, Wenliang Zhao, Guangyi Chen, Yansong Tang, Zheng Zhu, Guan Huang, Jie Zhou, and Jiwen Lu. Denseclip: language-guided dense prediction with context- aware prompting. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 18082-18 18091, 2022.
11

---

[36] Negar Rostamzadeh, Seyedarian Hosseini, Thomas Boquet, Wojeiech Stokowiec, Ying Zhang, Christian Jauvin, and Chris Pal. Fashion-gen: The generative fashion dataset and challenge. arXiv preprint urXiv:1806.088) 180118806.0.1.1.. 2018.
[37] Heng-Shiou Sheu, Zhixuan Chu, Daiqing Qi, and Sheng Li. Knowledge-guided article embedding refinement for session-based news recommendation. IEEE Transactions on Neural Networks and Learning Systems, 33(12):7921 -92229977999997... 2022.
[38] Chull Hwan Song, Taebaek Hwang, Jooyoung Yoon, Shunghyun Choi, and Yeong Hyeon Gu. Syne- mask: Synchronized attentional masking for fashion-centric vision-language pretraining. arXiv preprint arXiv:2404.00 156.4060.1166..... 2024.
[39] Robin Strudel, Ricardo Garcia, Ivan Laptev, and Cordelia Schmid. Segmenter: Transformer for semantic segmentation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 7262-7272 2021.
[40] Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. Vl-bert: Pre-training of generic visual-linguistir representations. arXiv preprint arXiv:/908.000 08530, 2019.
[41] Hao Tan and Mohit Bansal. Lxmert: Learning cross-modality encoder representations from transformers. arXiv preprint arXiv: /00. 074900 2019.
[42] Yoad Tewel, Yoav Shalev, Idan Schwartz, and Lior Wolf. Zerocap: Zero-shot image-to-tex generation for visual-semantic arithmetic. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17918-17921 2022.
[43] Hugo Touvron, Matthieu Cord, Mathijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International conference on machine learning, pages 10347-10357. PMLR, 2021.
[44] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
[45] Xiaodan Wang, Chengyu Wang. Lei Li, Zhixu Li, Ben Chen, Linbo Jin, Jun Huang. Yanghua Xiao, and Ming Gao. FashionKLIP. Enhancing E-commerce image-text retrieval with fashion multi-modal conceptual knowledge graph. In Sunayana Sitaram, Beata Beigman Klebanov, and Jason D Williams, editors, Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume 5: Industry Track), pages 149-158, Toronto, Canada, July 2023. Association for Computational Linguistics.
[46] Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. Simvlm: Simple visual language model pretraining with weak supervision. The Tenth International Conference on Learning Representations, 2022.
[47] Hui Wu, Yupeng Gao, Xiaoxiao Guo, Ziad Al-Halah, Steven Rennie, Kristen Grauman, and Rogerio Feris. Fashion iq: new dataset towards retrieving images by natural language feedhack. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 1 11307-11317, 2021.
[48] Jiarui Xu, Shalini De Mello, Sifci Liu, Wonmin Bycon, Thomas Breuel, Jan Kautz, and Xiaolong Wang. Groupvit: Semantic segmentation emerges from text supervision. In Proceedings of the IELE/CVF Conference on Computer Vision and Pattern Recognition, pages 18134-18144, 2022.
[49]
Lewei Yao, Runhui Huang, La Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. Filip: fine-grained interactive language-image pre-training. ICI.R, 2022.
[50] Andy Zeng, Maria Attarian, Brian Ichter, Krzysztof Choromanski, Adrian Wong, Stefan Welker, Federico Tombari, Aveck Purohit, Michael Ryoo, Vikas Sindhwani, ct al. Socratic models: Composing zero-shot multimodal reasoning with language. arXiv preprint 2XX::2204000598 2022.
[51] Xunlin Zhan, Yangxin Wa, Xiao Dong, Yunchao Wei, Minlong Lu, Yichi Zhang, Hang Xu, and Xiaodan Liang. Productl m: Towards weakly supervised instance-level product retrieval cross-modal pretraining. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pages 11782-11791, 2021.
[52] Handong Zhao, Zhengming Ding, and Yun Fu. Multi-view clustering via deep matrix factorization. In Thirty-first AAAI conference on artificial intelligence, 2017.
[53] Handong Zhao, Hongfu Liu, and Yun Fu. Incomplete multi-modt modal visual data grouping. In ICCAI pages 2392-2398, 2016.
12

---

[54] Yiwu Zhong, Jianwei Yang, Pengchuan Zhang, Chunyuan Li, Noel Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, al. Regionclip: Region-based language-image pretraining. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 16793-
16803, 2022.
[55] Ronghang Zhu, Dongliang Guo, Daiqing Qi. Zhixuan Chu, Xiang Yu, Sheng Li. survey of trustworthy representation learning across domains. ACM Trans. Knowl. Discov. Data, apr 2024. Just Accepted.
[56] Mingchen Zhuge, Dehong Gao, Deng- Deng-Ping Fan, Linbo Jin, Ben Chen, Haoming Zhou, Minghui Qiu, and Ling Shao. Kaleido-bert, Vision- languae pre-training on fashion domain. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 2647-12655, 2021.
13

---

A Experimental Settings
For FashionGen, following [27 10], the model is trained for 20 epochs. The weight decay set to le - 4, and the learning rate is set to 5e - 5 with the cosine annealing learning rate decay scheduler applied. As the selection tokens are randomly initialized, to match them with the pre-traines CLIP model, we first freeze other parameters and train selection tokens with an initial learning rate 5e 4 for 5 epochs. The default batch size is 64. Configurations are the same with amazonFashion, while the epoch is set to 10.
B Visualization
We provide more retrieval results in Fig. 15 and Fig. 16] For each example, the first row is the top-5 retrieval result by E² and the second row is the result by CLIP. Ground truth is boxed in green.
To intuitively see what selected by selection tokens, we visualize selected tokens the last stage in Fig. [4] Results show our selection tokens are effective in finding regions that contain detailed information about the tag entities, e.g., brand, composition. In the last transformer layer, only selection tokens and global token are kept (Eq. 4) (image tokens are dropped) to allow the latter effectively learns fine-grained representations from the former. We also visualize what the global token of CLIP focuses by marking image tokens with the maximum attention values with it for each attention head (12 in total). Note the overlap exists, i.e., one image token may have the maximum attention value with the global token in multiple attention heads. Results show CLIP is ineffective in finding tag entity-related information and sometimes focuses on totally unrelated regions. It is observed that visually dominant areas which are closer to the global view of the product, e.g., dark areas in a dark product in Fig. 14 (g). (i), (j), tend to win higher attention scores with the global token. Consequently, regions with detailed information such as logos are overlooked. One potential reason is that, pre-training data on general domain often consist of a simple caption and a few very distinctive objects (Fig. 2). which enables CLIP to capture features better from high-level view. In this case, focusing more on a global view of objects instead of a local view helps the model to differentiate large and distinctive object in image. However, when it comes fashion domain, where multiple details of a single product are required to be aware of, CLIP fails because CLIP still tries to capture the global view of an object and ignores local view, even through has already been finetuned. By contrast, with our model, Brands are all picked. Composition prefers solid color areas, where material more clear. Sub-category is reflected by sleeves, collar and shoulder. Sleeves and front zippers reveal the Season.
C Parameter Sensitivity
We also investigate the impact of batch size, which can be substantial to contrastive learning. Results in Tab. show that E consistency outperforms CLIP-FT across different batch sizes. Our findings indicate that E² consistency surpasses CLIP-FT across a range of batch sizes, with E² showing greater improvement over CLIP-FT batch size decreases. Notably, it is especially helpful in scenarios where hardware limitations impose constraints on batch size, where E²'s superiority becomes more evident.
Image to Text
Text to Image
Batch Size
Model
SumR
R0
R
R8
16
CLIP-PTT
29.9
63.0
76.7
30.7
63.7
76.7
340.7
E² (Ours)
42.0
75.7
86.3
42.2
76.5
87.0
409.7
CLIP-FT
42.7
77.0
87.0
43.6
77.1
87.1
414.6
32
E (Ours)
92.7
92.9
462.6
CLIP-FTI FT
55.2
85.7
92.9
55.4
86.2
12.9
468.2
15 (Qurs)
62.8
89.7
95.3
64.5
90.1
95.5
497.9
CLIP-FT E (Ours)
88.2
93.7
88.7
93.9
189.7
128
66.9
91.3
95.9
66.6
91.3
95.8
507.7
Table 5: Influence of batch size. We evaluate the performance of F and CLIP-FT under different batch size settings on FashionGenn
14

---

Ever-Pratty Elegant Formal
Evening Dress US Supphire Blue
Brand: Ever Pretty
Category Clathing
Package Dimensions: 8.28 1.5 inches Shipping
BORRKPPOT
Elegant Floor Length Groom
Brand: Ever-Protty
Category: Cluthing
Package Dimensions: 12.1x 9 inches
Shipping Weight ounces
DPKITNA
Title: Babyroom
boys' Long Sleeve 100% Cotton Pajame G7163
Brand: Babyroom
Category: Clathing
Package Dimensions: 8.1 6.9 1.6
Shipping Weight cunces
BO11DWWSSG
(d)
Title: Women's Autumn Winter Stand Collar Ultraligtt Down Jacket Black Medium.
Brand: Demetory
Glothing B011DV
AmazonFashine example
cress black, Single-shoulder, Zipper accent at front in gold--tna. Asymmattic saam at waist, Silk lining beige. Toral siiching.
Description:
Category: LONG DRESSES Brand: Stells McCartey FW2016
Composition: Body: cotton, Lining
FashionCien example
Figure Examples from AmazonFashion (blue) and FashionGen (yellow).
15

---

D Datasets
FashionGen 36 is the existing benchmark dataset for fashion cross-domain retrieval. Existing
works [12 10 27 [6] evaluate the performance of their model on fashion cross-domain retrieval
with this dataset.
Fashion ViL. [12 uses two dataset: FashionGen [36 and FashionlQ [47 for fashion domain retrieval. The critical difference is: FashionIQ 47 is for text-guided image retrieval (TGIR), special type of image retrieval problem, while FashionGen [36] is for cross-modal retrieval. Similar to FashionGen,
AmazonFashion built for the task of cross-modal retrieval.
Compared with M5Product (7] and ProductIM [5T], ours has two distinctive advantages: (a) Larger fashion data size: Product M is designed for cosmetics and groceries, not for fashion products. In all 6.3M data in M5Product, 0.5M is in fashion domain, while ours L.3M. (b) 3.1 times conciser caption in average, leading to more practical case: in practice, users usually search products online with short queries, instead a long paragraph of detailed text FashionGen style. Our dataset better matches this more challenging and practical case.
Dataset
Image-Tees Text Pairs Brand Count Product Count Avg. Desc. Len.
FashionGinn
29.6K
370
67866
AmazonFashion (Ours)
1.M
544713
Table 6: Data statistics of AmazonFashion and FashionGen. We show the number of image-tex pairs, brand count, product count and the average length of the image descriptions.
Hello ACERIST
DC Comiia Farum Neveltie
Haney Devidson
Under Armour
Casio
Jordan
9.9 0%.9%
2. 2.7%
2.7%
Wins
3.8%
3.8%
U.S. Polo Assn.
4.4%
Amazon Collection
23.5%
4.4%
JUTW
4.6%
5.0%
Old Glery
5.2%
6.2%
5.4%
Coach
Michael Kors
Invicta
The North Face
Figure 11: Top 20 most frequent brands in FashionGen.
D.1 FashionGen
contains 67, 666 different fashion products. Each product has one text description with one to six images from different angles. There are 260, 480 and 35, 528 image-text pairs for training and testing,
16

---

6000
AaszonFashion
FashionGen
5000
Frequency
Sunn Leanert
3000
2000
1000
100
200
300
400
500
Brand Index
Figure 12: Brand frequency of Amazon and FashionGen.
respectively. Each text description contains overall description along with several tag entities: Brand, Sub-category, Season and Composition.
D.2
AmazonFashion
We show several examples from AmazonFashion and FashionGen in Fig. T0 highlight our differ- ences between these two benchmarks. We also present detailed statistics of two datasets in Tab. and Fig 12 In general, AmazonFashion is closer the practical cross-model retrieval scenario.
Tab. 6 shows that AmazonFashion is different from FashionGen in notably larger size (1.3M), wider variety of brands and products (67 and 8 times more, respectively), more concise and general descriptions (0.6 times shorter), and more diverse image scopes. These differences make AmazonFashion both more challenging and closer to real-world scenarios, where query languages from users are often more general and more concise, meanwhile, the number candidate products is usually notably greater.
We also present the brand frequency plot in Fig 12 For each dataset, we selected the top 200 brands with the highest frequency of occurrence and plotted them in descending order. Note that for the AmazonFashion plot, we excluded the brand amazon collection, which has frequency of 19, 714, significantly higher than the other brands. Both datasets exhibit a long-tail distribution in terms of brand frequency, which accurately reflects real-world scenarios.
Besides significant differences in data statistics, images in AmazonFashion also exhibit substantial variations. While FashionGen only provides regular views of product from different angles such as Fig. [O]e), AmazonFshion includes more diverse images, such product pictures from different domains in Fig. [0 a). and high-resolution images that showcase intricate product details in Fig. 10 c). It is close to practical scenarios where available product images website often vary in styles. It also brings new challenges as better adaption to images different scales and styles is excepted. Thus, the AmazonFashion holds significant value for researches on cross-modal retrieval in the fashion domain, and we encourage its use in future work.
D.3 Model Results
We report results of representative CLiP-based models, including ours on AmazonFashion in Tab.
D.4 Brand Details
We show the distribution of the frequency of top 20 most frequent brand in Fig II
17

---

Image Text
Text Imuge
Model
SumR
R@5
R@10
R@5
FILIP-PT
49
6.2
17.8
25.8
6.2
18.1
25.9
100.0
CLIP-FT
64
6.1
17.7
25.8
6.2
17.9
25.8
99.5
1" (Ours)
7.5
20.3
28.4
7.4
20.2
28.4
112.2
Table 7: Retrieval performances (sample- 100 100 evaluation) on AmazonFashion. (*) denotes results from our implementation.
E
Linear Probing
E.1 Experiment
Linear probing is a popular technique to evaluate the quality of learned embeddings of a model: fitting a linear classifier the image embeddings of the model. Typically, the better the model is trained, the higher performance a classifier achieves, because a well-trained model tends to give more informative embeddings, from which classifier learns better.
To evaluate how well CLIP, CLIP-FT, FILIP-FT and E² are able to learn entity-specific knowledge from images, perform four classification tasks. More specifically, for each task, train linear classifier towards one of four tag entities (i.e., to classify: Brand, Sub-category, Season and Composition respectively. We use the Adam optimizer and train each classifier for 20 epochs with learning rate = 0.001. Results are shown in Fig.
image embeddings of CLIP [4), CLIP-FT (34), FILIP-FT (49), and F.2
DUP
CLP-TT
2:0
RUP-TT
Curs
E:4
Brand
Sub-celegory
Composition
Figure 13: Linear probing results on FashionGen. The more informative embeddings are, the higher accuracy classifier obtains.
E.2
Analysis
Results show classifiers trained with E² embeddings consistently outperform classifiers trained with CLIP, CLIP-FT and FILIP-FT embeddings, indicating that E² embeddings contain more information about tag entities. This can be attributed to our proposed token fusion, token selection, and region contrastive loss, which explicitly forces the model to pay special attention these details.
Besides, delve deeper into the linear probing results, which further reveal several interesting facts on how each tag entity impacts the model performance. Fig, [ (Brand) indicates that while fine-tuning can enhance CLIP's understanding of product brands, the extent of improvement is comparatively small when contrasted with the significant progress achieved through E², By comparison. Fig. 13 (Season) shows fine-tuning is effective for CLIP learn about produce season. However, when comes to Fig. 13 (Sub-category) fine-tuning almost yields no benefits. We assume that fine-tuning can only help learn easy classes such as bag, jeans, shirts, etc., while E² can help difficult classes. In this case, because the vanilla pre-trained CLIP is already effective in recognizing high-level concepts, fine-tuning fails to yield further improvements. E² significantly benefits the learning of product categories by learning more about difficult classes. In Fig. T3 (Composition). although the relative magnitudes are similar that Fig. [3] (Brand), the absolute differences among four models are much smaller, which indicates effectively learning about product composition is a harder task.
18

---

F
Ablation
To study how each group of selection tokens contribute to the model performance, we train and evaluate E² without one group of selection tokens associated with one lag entity at time and report the result in Table 8 While all groups of selection tokens improve the model performance, their contributions are different. Selection tokens associated with Brand and Composition
more helpful. As Fig. [3| and Fig. [ paying special attention to the brand and composition of a product helps model better differentiate visually very similar items. By contrast, Season and Sub-category contribute less as they are easier to be visually distinguished than product texture and logos, which harder discriminate and especially requires our fusion and selection process with the selection tokens.
Image Text
Text Image
Model
SumR
R@l
R@5
R@
R@1
R@5
R@10
E² (Ours)
62.8
89.7
95.3
64.5
90.1
95.5
497.9
w/ Composition
59.4
88.4
94.2
61.8
89.9
95.3
488.9
w/o Sub-category
61.2
89.7
95.0
60.8
89.6
95.1
491.3
w/o Brand
58.3
88.3
95.0
58.0
88.5
94.7
482.7
w/o Season
60.0
88.9
94.4
59.8
88.6
95.0
486.6
Table 8: Ablation study (selection tokens) on FashionGen.
G
Innovation
In this section, we discuss our differences from related works, and our innovation (motivation and technical contribution).
G.1 Technical Innovation
The design of selection and fusion mechanism with fusion blocks and selection tokens is our new technical contribution, which notably improves the performance. Typical CLIP-based adaption models (e.g.. RegionCLIIP [54]. FILIP 49). DenseCL.IP (35 FashionCIIp [4] are built upon
CLIP without modifying its inner structure. However, we break into the CLIIP encoder by introducing tag-aware fusion blocks. Specifically, the above models, including FashionSAP [4] do not explicitly align image patch tokens within ViT layers with text information, which we prove to be critical in fashion domain where the details do matter. To this end, our tag-aware selection tokens select and fuse image patch tokens. They differ from typical new additional tokens (VPT [8]] DeiT MK)), which are simply added to input layers and structurally similar to the CLS token. Our adaptation of CLIP is new and first tailored for fashion scenarios with tag entities, where general CLIP adaptatin models are hard to handle (more details in Appendix H)
G.2 Motivation
Our highlight is not improving the CLIP vision encoder in general domain, existing works such as FILIP [49 do. Instead, we are among the first to adapt CLIP to fashion domain. When combined with SOTA CLIP--bsed zero-shot image captioning models, their performance can be obviously improved.
Most recently, several works [49 21 11 start to improve language image pre-training with semantic alignment. FILIP 49 and LOUPE 21 learn better representations with semantic alignment of patch tokens. HiCLIP 11 learns to discover hierarchy in data. Group ViT [48] also forms semantic segments hierarchically, but it only focuses on semantic segmentation. Our motivation differs from above methods in that, while they learn representations of larger objects via grouping smaller regions (i.e., image tokens), we focus on certain regions with detailed information of an object, e.g., its brand and composition. Besides, they focus on designing models for better image-text matching generally without considering the uniqueness of fashion data. Our E² pays additional attention to image details associated with tag entities for better visual learning (more details in Appendix Hy
19

---

Image to Text
Text to Image
Model
SumR
R@]
R@5
R@ 10
R@]
R@5
R@ 10
EI-CLIP
59.4
88.4
94.2
61.8
89.9
95.3
488.9
EI-CLIP w/o
56.2
86.8
93.6
56.4
87.2
93.5
473.7
E (Ours)
62.8
89.7
95.3
64.5
90.1
95.5
497.9
Table 9: Retrieval performances (full-caddidate) on FashionGen.
H
Discussion
Differences from Segmentation Models. Some semantic segmentation models [39 48 also involve additional tokens, including GroupViT [48]. We are very different from each other in: (1) motivation, (2) selection technical details, and (3) overall framework. In summary, group tokens in GroupViT serve as "collectors", thus image tokens are iterated to be allocated a group token. By contrast, in E², the process is opposite: selection tokens iterated to select one most relevant image token. We do attention with different objects therefore are not interchangeable. Group cannot select the most relevant image tokens, while ours does not support grouping of them. Our framework design is also significantly different. Group adopts an iterative approach, where grouped tokens are consecutively fed inputs to transformer layers, introducing new group tokens each stage. In contrast, our approach utilizes image tokens along with a fixed number of selection tokens as inputs. To sum up, due different tasks/objectives, GiT and E are very different in both technical details and the overall framework.
Comparisons with EI-CLIP. While we facilitate the image encoder, EI-CLIP [27] improves the text encoding process by reducing word ambiguity in tag entities with additional text encoders. Thus we are orthogonal to each other. Besides, EI-CLIP uses additional text encoders and during inference, tag entities must be separately encoded by the encoders. contrast, E.2 does not require separated tag entities as text input. In our model, same to existing works 10 56 and CL.PP, tag entities are concatenated with the description as one string, which is input to the CLIP text encoder. This is more practical and especially helpful in cases where tag entities mixed with descriptions and separating them infeasible. We also evaluate EI-CLIP under this setting, which follows existing works [10, [56] and ours (w/o E. in Table D). Although EI-CLIP has additional encoders and requires separated tag entities during inference, is still inferior to ours according to Table When evaluating under the same setting, the gap is more notable.
Differences from Fine-grained CLIPs. Several recent works 44 11 21 propose better match image-text pairs with semantic alignment or hierarchical grouping. While they learn representations of larger objects via grouping smaller regions (image tokens), we focus on finding certain regions with detailed information of part of an object (e.g., its brand, season, composition), motivated by the uniqueness of fashion data. We achieve this by designing better way to exploit tag entities, i.c., introducing selection tokens associated with these entities, while they seek for a general way to better align an image-text pair without considering the of characteristics of fashion data. Thus our work is orthogonal to theirs and we can build E² based on their pre- rre-trained models. We choose is the most representative one, based on which makes it extendable others. Because codes and pre-trained models of mentioned CLIP variants are not released, we are not able to evaluate of them. However, as FILIP 49 shares the same structure with CLIP, we implement and initialize it with pre-trained CLIP model and finetune it on our datasets. We also use FILIPP as the backbone model in our ablation study.
CLIP in this work
I Analytical Experiments
In this section, we explore 1. how the number of selection tokens can influence our model performance and 2. the parameter efficiency of our model.
20

---

I.1 Selection Tokens
In Tab. 10 we provide different choices of selection tokens (②. (-Q). Two tokens for each tag are better than one (3), and comparable to more tokens (OO)), or combination of different numbers (0). It reasonable we are not selecting and fusing image patches raw pixel space, instead, we conduct it with image patch (token) embeddings within ViT layers in contextual embedding space, where each token contains rich context/neighbor information. E.g., in Fig. 7 in paper, one token can already represent large area if considering its neighbour information. Our choice of two tokens is empirically enough for all tags, including season, as quantitatively validated Tab.
1.2 Parameter Efficiency
We compare the amount of learnable parameters of ours (default setting in paper is shaded in green) and most competing baselines (0) in Tab. IO Compared to CLIP-FT (10). we only involve minimal new parameters fusion blocks: (1) three projection matrices: W W and W RR 768xdw and (2): 8 selection tokens E x66 where dw = 768 in paper. We have comparable parameters with CLIP-based methods and significantly fewer parameters than (2), while achieving notably better results over them. We also tried different dw and numbers of selection tokens (G-O). and our original setting (2) achieves competing results with very few new parameters.
Model
SC
Params (Total)
Params (Token)
SumR
Ours (dw 128)
2
2
2
86.6M
6.1K
494.3
Ours (dw=768)
2
2
2
87.7M
K.
497.9
Ours (204))04)
2
2
2
94.8M
6.1K
498.2
Ours (6w=)68)
87.7M
3.1K
493.8
Ours (dw=768)
4
4
4
87.7M
12K
497.9
Ours 776)
8
8
8
87.7M
24K
498.3
Ours (76))
16
16
16
16
87.7M
48K
498.2
Ours (6w=
8
4
87.7M 8
18K
497.9
EI-CLIP
473.7
CLIP-FT
468.2
FILIP-FT
86M
470.2
FashionSAP
120M
451.8
Table 10: Results on FashionGen with full-candidate evaluation. C: Composition, S: Season, B: Brand, SC: Sub-category. Numbers are the quantity of corresponding selection tokens. Total number of learnable parameters in vision encoder: Params (Total). We also list parameters of selection tokens (part of the total Params): Params (Token).
J Broader Impacts
There is no special societal impact of our work performed.
21

---

PROM
PROM
(a)
(c)
(8)
(h) i
IVENE
IVENCH
(d)
(i)
(I)
Srand
Scapor:
Cacegory
composition
Xaximal Accention
Figure 14: Left: E² Token Selection Examples. We marked selected image patch tokens during the second
stage. Each tag entity is associated with two selection tokens. Note that tokens may overlap when image patch token is selected by multiple selection tokens. Right: CLIP Attention Visualization. We marked image tokens with the maximum attention values (in each attention head) with the global token. Note that, the "Category" in the figure refers to "Sub-category".
22

---

Catagony: T-SHIRTS Brand farco Season: 5520 Compostiin 100% cotinn
Dencriptin:: Eheach colerblocioe Browettng click antrodennd
Nulle aeshiccappiiccooe and riboo ain setoed rod and cllvetaa fort kit culla und hem, Drospes shoulbers. Catagery:
Description: Buubed catiiin shoulder bag grw Cunh than shou ader stoer Logs plagus and outh chain tnsued bros Siivetone testtaae Theal etiiiigg Appers Ehhweghh a heigh v 207uich Carngory" S0 ouL CFF BAGS Brand Saini Leuten Season. FW/017 Composiloon
Deseription: Nylon duille beg black, Tonal lostt timu Imoughout Singls robol kouteer caryy hancle kup Datecaadoo and computineniin und pockst locp do doauro Louthen pulli aspered uochet, and pulch poslot uniil-irne Brand Gibenchy Sesson FM000
mmtetor horticaee. mitici Entegory:
Figure 15: T2I retrieval examples. For each example, the query text is displayed on the top. The first row is the top-5 retrieval result by E and the second row is the result by CLIP. Ground truth is boxed in green.
23

---

Descriptio: Fitee short lnst seten- whit,, Conamedi dlliding Category: Brand
Deacrption: Tapered janna in danc slue festurnng mictied pine cwandye, Fading cerreetng and whtakeeing throupheet pooppocketsppll g. ietthel kgg bock 6.5 Catagory: JEANS Brand: Guod Sassor: 192201
edge
Buton-ly Corrze sttching
Corpostion: 100% collee
Description:
Long paralled colton ust ostion an coletledadin white, Skgnature bain'
Spent color, Conswaee bution disaue frvet Bingiebuuton born ruffs, Tooal dliching Cotegery: SHIRTS Boad Net
Description: Shoo canun cotine jenuey heblit while lonit conumede oslor Gevtic prinnw hark
back turs. Toral allvergg Concosled buteen caure et fron, bu butoon barsi Tonal staching Gategory: Composilier
SET2
Figure 16: T2I retrieval examples. For each example, the query text is displayed on the top. The first row is the top-5 retrieval result by E² and the second row is the result by CLIP. Ground truth is boxed in green.
24

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our main contribution are presented at the end the introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We use section to discuss the limitation.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers penalize honesty concerning limitations.
will be specifically instructed
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
25

---

Justification:
We do not involve theoretical proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We introduced the implementation details in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
26

---

Answer: [Yes]
Justification:
We will provide open access to both data and source code with sufficient
instructions.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm ec/
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https for more details.
//nips.cc/pp wwwppbblccgguiicuceccccceoeooopp hetggudddeeeoeeubbinninn
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
The implementation detail is presented in the paper and the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[NA]
Justification:
We follow previous works to present the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
27

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
is mentioned in the implementation details.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httttp////arrrpss..c//b///////i
Answer: [Yes]
Justification:
We follow the guidelines of Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
It is discussed in the appendix.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
28

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our model has little negative impact on society.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: They are all public.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
29

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
It introduced in the paper.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
No crowdsourcing is involved.
Guidelines:
The answer NA means that the paper does involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: None them are involved.
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
30


Reviews:
[{"review_id": "5mGbfKzL71", "reviewer": "Reviewer_Mfk5", "summary": "This work proposes a new approach to learn improved and more expressive visual representations for fashion-specific tasks. Prior works have proposed to learn fashion representation either by complex multi-task objectives or fine-tuning strong pretrained visual features such as CLIP. However, such methods fail to learn strong features that can distinguish between fashion-specific attributes/entities that are often available in image descriptions. Authors propose E^2 which add new layers over CLIP visual encoder along with selection tokens for different entities (such as brand, composition). They propose to attend these selection token with the image tokens such as each selection token can select a single image token. They use contrastive loss to supervise the network training and also use separate contrastive losses to supervise each of the selection tokens (supervision is provided from the categories). Authors have shown strong improvements over SOTA in image retrieval as well as zero-shot captioning task highlight the efficacy of the proposed representations. They also show some qualitative results to show the benefits of the proposed approach.", "questions": "- Why did the authors use hard-attention for the selection tokens. Was there a difference in performance if doing soft-attention? The visualization of the selection tokens in figure14 is not showing much difference (the attention seems to be clustered around a few points). This leads to the question if the selection tokens are interpretable\n\n- Why did the authors choose only one fashion dataset from prior works (other is proposed in this paper). Why weren't other datasets considered?\n\n- What is the benefit of using >1 selection tokens for each sub-category?\n\n- Would this work also generalize to other domains. If yes, what is needed to achieve this (e.g. detailed description + specific attributes)", "limitations": "Yes", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The core idea of using selection tokens to attend to specific fashion attributes is interesting. Moreover, building it on top of CLIP can lead to better generation.\n- The idea of gumbel-softmax with straight-through gradient trick for learning the hard attention is also interesting\n- Authors have shown strong improvements over SOTA methods in both captioning and retrieval tasks. The ablation experiments also show improvement due to the proposed modules.\n- Authors did provide additional details in the appendix such as detailed ablation studies, additional results on selection tokens", "weaknesses": "- Authors could have done a better job at presenting their approach and highlighting the core novelty (use of fusion blocks with selection tokens) in the introduction and abstract.\n- Although the paper was easy to follow- the presentation could have been improved. For example the figures were often small to read.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your efforts in reviewing this paper and giving valuable suggestions!\n\n## W1\n\nThanks for the suggestion! In our current version, we have focused more on discussing the motivation and the benefits of our approach, including analytical discussions, in the Introduction and Abstract. Following your suggestion, we will introduce more technical novelty of our approach at the beginning in the final version.\n\n## W2\n\nWe appreciate your suggestion and apologize for the inconvenience caused by the small fonts due to limited space. In our final version, we will better arrange the format to allow for an improved layout, with better fonts in proper size.\n\n## Q1\n\nWe used the hard-attention because we wanted to explicitly select most relevant and informative image patch tokens while dropping less relevant ones, to enable the (merged) selection tokens containing **tag entity (or attributes)-specific knowledge**. In the Introduction, we show that such knowledge is critical to fashion tasks, and therefore we choose hard attention.\n\nMoreover, hard-attention could also lead to better performance (cross-modal retrieval on FashionGen with full-candidate evaluation):\n\n\n| Method          | I2T         |   I2T    |   I2T    | T2I    |  T2I   |  T2I   | SumR  | MeanR@1 |\n|-----------------|-----------------------|-------|-------|-----------------------|-------|-------|-------|---------|\n|                 | R@1                   | R@5   | R@10  | R@1                   | R@5   | R@10  |   -    |    -     |\n| Soft-attention  | 61.9                  | 88.5  | 94.8  | 63.4                  | 89.4  | 94.8  | 492.8 | 62.6    |\n| Hard-attention  | **62.8**              | **89.3** | **95.3** | **64.5**              | **90.1** | **95.5** | **497.9** | **63.7** |\n\nIn Figure 14, we are not doing a comparison between hard-attention and soft-attention. Instead, we actually show: (1) the selected tokens in our model and (2) the tokens that have the maximal attention scores with the global token in the transformer in **vanilla CLIP-FT model**. For (2), the colored tokens (with high attention scores) are intuitively the ones that the CLIP-FT model gives more importance during inference.\n\nBy the comparison, we want to show that our selected tokens can have a better coverage of important attributes in fashion domain.\n\nFor example, we can find that:\n\n(1) Vanilla CLIP-FT tends to give more importance to image patch tokens of a solid color (or pure color) in fashion images. For instance, black areas in (g), (i), (j), and white areas in (k), (l).\n\n(2) By contrast, our selection tokens capture informative image patch tokens: E.g., 'Brand' tokens (blue) accurately capture the printed logo in (a), (b), (c), (d), (e) and (f). \n\nWe have detailed explanation in Appendix B (Line 472).\n\nThe quantitative and qualitative results demonstrate that our selection tokens are effective and interpretable.\n\n## Q2\n\nIn fact, in addition to FashionGen, we also included Fashion IQ [1] in Table 8, a fashion benchmark for the Text-guided Image Retrieval (TGIR) task. We evaluated the specific zero-shot TGIR task on this benchmark. We will add details of this dataset into the Section 4 (Datasets) for better clarity in our final paper.\n\nWe evaluate the cross-modal retrieval and fashion captioning task with FashionGen because it is the most commonly used cross-modal retrieval benchmark in Fashion domain (e.g., it is used by ALBEF, SyncMask, FashionSAP, FashionViL, FashionBert, KaleidoBert, FaD-VLP, FAME-ViL, etc.). Both recent works (FashionSAP, SyncMask, FaD-VLP, FAME-ViL) and representative works (FashionBert, KaleidoBert) evaluate the cross-modal retrieval task with only FashionGen. We did not use other fashion datasets for cross-modal retrieval because there was not another widely-used and well-established benchmark for this task.\n\n## Q3\n\nIntuitively, with more selection tokens for each sub-category, we could select more image patch tokens, and merge them with each selection token. In Appendix I, we analyzed how different numbers of selection tokens for each sub-category can affect model results in detail (quantitative results are available Table 10). Empirically, using 2 selection tokens for each sub-category achieves a good balance between accuracy and efficiency.\n\n## Q4\n\nConceptually \"yes\". As our model learns to attend to important attributes/regions with additional fusion blocks and selection tokens, our method could be applicable to other data/domains as long as specific attributes are provided.\n\nIn fashion domain, the specific attributes are the existing tag entities, in other domains (such as medical images), we could directly use similar existing attributes (if any) and apply our model.\n\nIf there are no such attributes in existing data immediately, one intuitive solution could be collecting useful attributes for existing data first (manually or using other models). For example, in contrastive learning for medical images [2,3,4,5], extracted entities [6] from image descriptions could be used as additional attributes.\n\nFinally, we would like to kindly remind that, fashion domain itself is an emerging topic [7-15], and our effective contrastive learning of fashion representations would benefit a series of downstream tasks such as cross-modal retrieval, fashion captioning, text-guided retrieval, recommendation, etc. \n\nAlthough our method also demonstrates insights in finetuning large-scale pre-trained vision-language models on any paired (image-text) data with additional attributes, and could be potentially applied to other domains (such as medical images), the primary focus of this work is still fashion representation learning [8,10,12,13,14,15]. \n\nThank you again for your thorough review of this paper and your valuable suggestions. We hope the responses could address your concerns. If you have any further questions, we look forward to having further discussions with you!\n\n# Reference\n(available in \"Author Rebuttal by Authors\")", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vVsVrIegop", "reviewer": "Reviewer_tu2V", "summary": "The paper discusses adaptation of CLIP for fashion domain w/ an emphasis on importance of learning fine-grained visual representations. The authors propose E^2 with selection tokens and region contrastive loss to enforce extra attention.", "questions": "none", "limitations": "not mentioned", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "Emphasis on importance of learning fine-grained visual representations with selection tokens and region contrastive loss to enforce extra attention. Good analysis is performed in the experiments.", "weaknesses": "Minor comment: the paper could be made a bit easier to understand for the general ML audience attending NeurIPS", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your  valuable suggestions and the recognition of the analysis and experiments in our study!\n\nIn our final version, we will make the paper more understandable for the general ML audience in NeurIPS. Specifically, we will carefully introduce the background of our problem, and how our specific approach is derived from general machine learning principles.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "T80w1OPKS4", "reviewer": "Reviewer_Kwtu", "summary": "This paper proposes a framework to train CLIP models more adaptable to fashion domain (shopping item images and structured descriptions, i.e, “tags”, such as brand, composition etc.). The new framework (“E2”) outperforms other common methods by a large margin on a wide range of tasks, by 1) adding fusion blocks in image encoder to select more representative patches for tags, and 2) adding tag specific contrastive losses.", "questions": "It’s common to have the same tags from different images (such as the same brand of different images of t-shirts, shoes…). This could lead to many false-negatives in contrastive training, and may explain why the fusion block is necessary. Have you identified such an issue and considered any solution for this (such as “soft” contrastive labels)?\n\nIn Figure 5, it would be more clear if the locations of a/b/c can be explicitly stated, such as “a - Left, b - Right, c - Middle”.\n\nWhat is the token length of text encoder?\n\nIn  Parameter Sensitivity section, is the batch size’s unit thousand (otherwise looks too small for contrastive learning)?", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The proposed method is simple yet effective. It demonstrates the effectiveness of the region level visual patch selection and fusion, which also makes sense for applications beyond the fashion domain.\n\nThe performance improvement is large and consistent on diverse VL tasks.\n\nIt’s easy to replace CLIP components with E2 in any VLM. This paper also shows its effectiveness beyond retrieval by replacing the CLIP in DeCap.", "weaknesses": "The efficiency evaluation is missing. It’s unclear how the training/inference speed is affected by the newly added components in CLIP.\n\nBesides retrieval and captioning, visual question answering is another common task for vision language modeling. It would be more solid if we add another eval on CLIP based VQA models.\n\nThe impact is mostly limited in the fashion domain. It would be more impactful if more applications would be found for the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your efforts in reviewing this paper and giving valuable suggestions!\n\n## W1\n\nThanks for your helpful suggestion! One of the highlights of our model is being lightweight with minimal additional cost.\n\nWe compare the training and inference speed with the vanilla CLIP as follows:\n\nUnder the same environment and configuration, with a single A100 (40GB), the running time is summarized in the table:\n\n\n\n| Model        | Training Time (per epoch) | Inference Time |\n| ------------ | ------------------------- | -------------- |\n| CLIP         | 231.86s (3 m 52 s)        | 17.55s         |\n| $E^2$ (Ours) | 250.60s (4m 10 s)         | 17.57s         |\n\nWe report the averaged training time of one epoch on FashionGen (20 epochs in total for full finetuing). For inference, we report the running time for single inference with full-candidate evaluation on FashionGen (390K candidate samples).\n\nThe above table shows that we have only around 8% additional training time (19 seconds longer per epoch) compared to CLIP, while the inference time is almost the same. \n\n## W2\n\nThanks for the very good suggestion, and an interesting direction as future work. We currently do not have the evaluation on VQA because, although there are a lot of available well-established VQA benchmarks in the general domain, there is not a standard VQA benchmark or dataset in the fashion domain. FashionVQA [16] is the most related work, unfortunately neither datasets nor models are publicly available.\n\nTherefore, if we want to evaluate CLIP-based VQA models (and our model) in the fashion domain, we need to at least build a fashion VQA benchmark first (and possibly another fashion dataset for simple finetuning), which is beyond the scope of this work.\n\nBecause our primary focus is not benchmarking fashion VQA, we currently do not evaluate VQA. We mainly focus on the well-defined and well-established fashion tasks in existing works (such as ALBEF, SyncMask, FashionSAP, FashionViL, FashionBert, KaleidoBert, FaD-VLP, and FAME-ViL).\n\n## W3\n\nThanks for the great advice. As our model learns to attend to important attributes/regions with additional fusion blocks and selection tokens, our method could be easily applicable to other domains as long as specific attributes are provided (like tag entities in fashion domain).\n\nIn fashion domain, the specific attributes are the existing tag entities, in other domains (such as medical images), we could directly use similar existing attributes (if any) and apply our model.\n\nIf there are no such attributes in existing data immediately,  one intuitive solution could be collecting useful attributes for existing data first (manually or using other models).  For example, in contrastive learning for medical images [2,3,4,5], extracted entities [6] from image descriptions could be used as additional attributes.\n\nFinally, we would like to kindly remind that, fashion domain itself is an emerging topic [7-15], and our effective contrastive learning of fashion representations would benefit a series of downstream tasks such as cross-modal retrieval, fashion captioning, text-guided retrieval, recommendation, etc. \nAlthough our method also demonstrates insights in finetuning large-scale pre-trained vision-language models on any paired (image-text) data with additional attributes, and could be potentially applied to other domains (such as medical images), the primary focus of this work is still fashion representation learning [8,10,12,13,14,15]. \n\n## Q1 & Q4\n\nThanks for this insightful question. For question 4, the batch size's unit is not thousand. It is understandable that in large-scale pre-training in general domain (e.g., 400M samples in CLIP/OpenCLIP[17] pre-training), large batch sizes (16K, 32K, or 88K) are used. \nSuch large batch sizes are the “global batch size”, which is accumulated from local batch sizes in thousands of individual GPUs. For instance, OpenCLIP [17] maximizes the local batch size per GPU (86 to 88 per GPU) and uses around one thousand GPUs to have a global batch size of 86K to 88K.\nWhen adapting the pre-trained CLIP to a specific domain (FashionGen) with obviously fewer data and limited GPU resources, a batch size from 16 to 128 is typically used in existing works (FashionViL, FashionSAP, FaD-VLP, and FAME-ViL). We follow them to set similar batch sizes for fair comparison.\n\nThis could also explain the question 1. We understand that there might be false-negatives in contrastive training in a batch, because different images could have the same tag. However, as the commonly used batch size for the in-domain finetuning on fashion data is relatively small (16 to 128), in practice, false-negatives are not common in a single batch. Therefore, we currently just ignore their marginal influence and do not include a specific design for it. \n\nHowever, we agree that involving a specific solution for this would be a bonus. Potential solutions could include adopting different sampling strategies or adjusted objectives  (e.g., Khosla et al. [18] propose “supervised contrastive” (fig. 2 in their paper) to contrast the set of all samples from the same class as positives against the negatives from the rest of the batch)..\n\n## Q2 \n\nThanks for your advice. It is very helpful for improving the clarity of this figure.\n\n## Q3\n\nThe default limit for the input text length of the CLIP text encoder we used is 77\n\nThank you again for your thorough review of this paper and your valuable suggestions. We hope the response could address your concerns. If you have any further questions, we look forward to more discussions with you!\n\n# Reference\n\n(available in \"Author Rebuttal by Authors\")", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SSPjxgVUbb", "reviewer": "Reviewer_LcUx", "summary": "This paper focused on adapting CLIP-based VLMs to the fashion domain. The motivation is that directly finetuning CLIP models on fashion data will lead to insufficient learning of entity-related details like logos and composition. To tackle this challenge, this work proposed a region-based contrastive loss by inducing selection tokens to capture information on tag entities like brand, sub-category, and season, and further applied region contrastive loss on these token embeddings. Also, this work collected a new dataset called AmazonFashion which consists of briefer and more general text descriptions compared to the previous FashionGen benchmark. Experiments on several benchmarks validated the effectiveness of the proposed method.", "questions": "I have no further questions for the authors.", "limitations": "The authors have adequately addressed the limitations.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "\\+ The contributions of this work to the fashion domain are significant in terms of method and dataset.\n\nFirst, this work introduced selection tokens of tag entities and region contrastive loss to adapt CLIP models to the fashion domain and learn more fine-grained visual representations. The motivation that directly finetuning CLIP models on fashion data will lead to insufficient learning of entity-related details is reasonable and the solution makes sense.\n\nSecond, this paper collected a new dataset called AmazonFashion which consists of briefer and more general text descriptions compared to the previous FashionGen benchmark. The new dataset would serve as a new benchmark for future research on the fashion domain.\n\n\\+ The paper is well-organized and written. It is easy to follow the idea and understand the key contributions of the work.\n\n\\+ Experiments are conducted on Fashion-Gen and AmazonFashion datasets and evaluated on cross-model retrieval, zero-shot image captioning, and zero-shot text-guided image retrieval tasks to validate the effectiveness of the proposed method. The method can be applied to various base models like CapDec and DeCap, which shows the generalization of the method.", "weaknesses": "\\- The proposed method works well for the fashion domain but is not evaluated on general domains for other fine-grained recognition or captioning tasks. This is not a reason to reject the paper, but limits the scope of the work. This work is a good application work and proposed a reasonable solution, but the insights are not very strong for other domains or tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your efforts in reviewing this paper and giving valuable suggestions!\n\nWe appreciate your recognition of the effectiveness and rationale of our work. While our method consistently outperforms existing works and exhibits simplicity and efficiency in fashion domain, it also demonstrates some insights in finetuning large-scale pre-trained vision-language models on paired (image-text) data with additional attributes.\n\nOur design could be viewed as a general vision-language contrastive learning method for paired image-text data with attributes. As our model learns to attend to important attributes/regions with additional fusion blocks and selection tokens, our method could be applicable to other data/domains as long as specific attributes are provided.\n\nIn fashion domain, the specific attributes are the existing tag entities, in other domains (such as medical images), we could directly use similar existing attributes (if any) and apply our model.\n\nIf there are no such attributes in existing data immediately,  one intuitive solution could be collecting useful attributes for existing data first (manually or using other models).  For example, in contrastive learning for medical images [2,3,4,5], extracted entities [6] from image descriptions could be used as additional attributes.\n\nFinally, we would like to kindly remind that, fashion domain itself is an emerging topic [7-15], and our effective contrastive learning of fashion representations would benefit a series of downstream tasks such as cross-modal retrieval, fashion captioning, text-guided retrieval, recommendation, etc. \n\nAlthough our method also demonstrates insights in finetuning large-scale pre-trained vision-language models on any paired (image-text) data with additional attributes, and could be potentially applied to other domains (such as medical images), the primary focus of this work is still fashion representation learning [8,10,12,13,14,15]. \n\nThank you again for your thorough review of this paper and your valuable suggestions. We hope the response could address your concerns. If you have any further questions, we look forward to more discussion with you!\n\n# Reference\n\n(available in \"Author Rebuttal by Authors\")", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
## Reference

[1] Wu et al., Fashion IQ: A new dataset towards retrieving images by natural language feedback, CVPR 2021

[2] Irvin et al., Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. AAAI 2019

[3] Johnson et al., Mimic-cxr, a de-identified publicly available database of chest radiographs with free-text reports. Scientific data

[4] Rahman et al., Exploring the effect of image enhancement techniques on covid-19 detection using chest x-ray images. Computers in biology and medicine, 2021

[5] Shih et al., Augmenting the national institutes of health chest radiograph dataset with expert annotations of possible pneumonia. Radiology: Artificial Intelligence, 2019

[6] Wang et al., MedCLIP: Contrastive Learning from Unpaired Medical Images and Text, EMNLP 2022

[7] Liu et al., Arbitrary Virtual Try-on Network: Characteristics Representation and Trade-off between Body and Clothing, ICLR 2023

[8] Bai et al., Cross-Domain Product Representation Learning for Rich-Content E-Commerce, ICCV 2023

[9] Baldrati et al., Multimodal Garment Designer: Human-Centric Latent Diffusion Models for Fashion Image Editing, ICCV 2023

[10] Pal et al., FashionNTM: Multi-turn Fashion Image Retrieval via Cascaded Memory, ICCV 2023

[11] Karras et al., DreamPose: Fashion Image-to-Video Synthesis via Stable Diffusion, ICCV 2023

[12] Han et al., FAME-ViL: Multi-Tasking Vision-Language Model for Heterogeneous Fashion Tasks, CVPR 2023

[13] Jiao et al., Learning Attribute and Class-Specific Representation Duet for Fine-grained Fashion Analysis, CVPR 2023

[14] Han et al., FashionSAP: Symbols and Attributes Prompt for Fine-grained Fashion Vision-Language Pre-training, CVPR 2023

[15] Song et al., SyncMask: Synchronized Attentional Masking for Fashion-centric Vision-Language Pretraining, CVPR 2024

[16] Wang et al., FashionVQA: A Domain-Specific Visual Question Answering System, CVPRW 2022.

[17] Cherti et al., Reproducible scaling laws for contrastive language-image learning, CVPR 2023

[18] Khosla et al., Supervised Contrastive Learning, NeurIPS 2020

Meta-review: The reviewers generally appreciated the proposed approach, which is specifically designed for visual recognition in the fashion domain, where fine-grained visual representations are crucial for strong performance. They acknowledged that the core idea of using selection tokens to attend to specific fashion attributes is interesting and that the paper provides ample and convincing empirical evidence to support its effectiveness. While there was some concern about the limited scope of the paper's application to the fashion domain, the reviewers agreed that this alone should not be a reason for rejection, as the paper in its current state merits acceptance. After carefully reading the reviews and the rebuttal, this meta-reviewer concurs with the reviewers' unanimous recommendation for acceptance.
Decision: Accept (poster)