Easy Regional Contrastive Learning of Expressive Fashion Representations

Authors: daiqing  qi
handong  zhao
raf  simons
stella  mccartney
haerfest  saint
laurent  kara
fendi  kara
]  gucci
sophie  hulme
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 06:30:39

Abstract: When learning vision-language models (VLM) for the fashion domain, most existing works design new architectures from vanilla BERT with additional objectives, or perform dense multi-task learning with fashion-specific tasks. Though progress has been made, their architecture or objectives are often intricate and the extendibility is limited.
By contrast, with simple architecture (comprising only two unimodal encoders) and just the contrastive objective, popular pre-trained VL models (e.g., CLIP) achieve superior performance in general domains, which are further easily extended to downstream tasks.
However, inheriting such benefits of CLIP in the fashion domain is non-trivial in the presence of the notable domain gap. Empirically, we find that directly finetuning on fashion data leads CLIP to frequently ignore minor yet important details such as logos and composition, which are critical in fashion tasks such as retrieval and captioning.
In this work, to maintain CLIP's simple architecture and objective while explicitly attending to fashion details, we propose $E^2$: Easy Regional Contrastive Learning of Expressive Fashion Representations.
$E^2$ introduces only a few selection tokens and fusion blocks (just 1.9\% additional parameters in total) with only contrastive losses. Despite lightweight, in our primary focus, cross-modal retrieval, $E^2$ notably outperforms existing fashion VLMs with various fashion-specific objectives.
Moreover, thanks to CLIP's widespread use in downstream tasks in general domains (e.g., zero-shot composed image retrieval and image captioning), our model can easily extend these models  from general domain to the fashion domain with notable improvement.
To conduct a comprehensive evaluation, we further collect data from Amazon Reviews to build a new dataset for cross-modal retrieval in the fashion domain.

Main paper: 
 introduction: 
 There has been a long research line for vision-language learning in general domain [15, 34, 19, 23, 26, 40, 52, 53] . Recently, cross-modal retrieval in fashion domain is receiving increasing attention [10, 27, 12, 56, 13, 4, 38, 22, 14, 12] . Most existing fashion vision-language models (VLM) [12, 38, 14, 10, 12, 56] design new vision-language models based on vanilla BERT [6] .Existing studies commonly train BERT-based models with Masked Language Modeling (MLM), (Image-Text Matching) ITM, Image-Text Contrastive Learning (ITC), or Masked Image Modeling (MIM) to their specific architectures. Various techniques tailored to fashion domain have been proposed, including learning extra fashion-specific tasks, new attention mechanism [38] , or additional modules for fashion feature learning [14] . FashionSAP [14] and FAME-ViL [13] perform a fashionspecific multi-task learning with various fashion tasks in addition to cross-modal retrieval, such as category recognition. However, the learning of existing BERT-based or multi-task fashion VLMs is often intricate with their complex architecture or additional fashion-specific objectives. By contrast, with simple architecture that comprises only two unimodal transformer encoders, and a single contrastive learning objective, contrastive language-image pre-training models such as CLIP [34] exhibit outstanding performance from cross-modal retrieval to a wide range of downstream tasks [24, 42, 20, 33] . Motivated by the simplicity and effectiveness of CLIP, we look forward to learning a model that inherits such benefits of CLIP in the fashion domain, which could be simple, lightweight while highly effective. While directly finetuning CLIP (FashionCLIP [4] ) is an intuitive solution, it is deficient in presence of the notable domain shift [55] . Consequently, Ma et al., [27] uses CLIP as backbone and improves it by using additional text encoders to mitigate the word ambiguity in fashion language. However, they ignore the uniqueness of visual learning in fashion domain.Different from data in general domain, product images and descriptions in fashion domain are unique in several aspects. As illustrated in Fig. 2 , in general domain, an image contains only a few distinctive objects, and text descriptions are more concise and general. However, in fashion domain, a product image usually includes only one foreground object but with rich details. Besides, the fashion text often provides a group of metadata (tag entities) [16, 36] , such as Composition, Brand, Description, Sub-category, etc. When directly finetuning CLIP on fashion data, we find it tends to give more attention to regions that are closer to a global view, e.g., dark regions in dark clothing (Fig. 2 and 14 ). Consequently, it misses details associated with tag entities such as composition and logo, which represent more of a local view and are critical for distinguishing visually similar items.To quantify the capability of visual representation in recognition of product details, we design an exploratory entity classification task based on linear probing: fitting a linear classifier on image embeddings from different models respectively, with tag entities used as labels. Results in Fig. 3 show that, the classifier which learns from fine-tuned CLIP (CLIP-FT) embeddings are less effective, indicating that CLIP-FT embeddings contain less entity-related information, i.e., CLIP-FT is ineffective in extracting entity-related information from fashion images. Details are available in Appendix E, where we provide a more comprehensive analysis. Figure 4 : Comparison of FashionGen [36] and AmazonFashion (Ours), which is notably different in size, diversity, language style and image scope (close-up shots of products are included). We compare it with more datasets [7, 51] , and show more details (e.g., clustering [52] ) in Appendix D. (1) We first reveal that the image encoder of pre-trained CLIP is biased towards visually dominant regions of a product and consequently ignores small but critical details when directly fine-tuned on fashion domain. Furthermore, we present the first simple yet effective solution to this problem by allowing the model to learn more fine-grained visual representations towards tag entities. (2) Motivated by the observation in (1), we propose a concise regional contrastive learning framework comprising only two unimodal encoders and just contrastive learning objectives, while it learns richer visual representations. (3) We conduct extensive experiments with existing benchmark datasets, including our new benchmark dataset (Fig. 4 ) for fashion cross-modal retrieval, which differs from the popular benchmark FashionGen [36] in notably larger size, a wider variety of brands and products, more concise and general descriptions, and more diverse image scopes, making it more challenging and more practical. (4) Inheriting the benefits of the widely used CLIP, our model can easily extend models from general domain to the fashion domain with notable improvement (Fig 1 ) in downstream tasks (e.g., zero-shot composed image retrieval and image captioning). 
related work: 
 Text and Image Matching. Text and image matching aims to semantically align the text and image. Various BERT-based vision-language models (VLM) [40, 26, 41, 25, 14, 12, 10] are proposed towards this end. Different from previous studies, CLIP [34] introduces a large-scale vision language pre-training framework, which learns from 400 million image-text pairs with contrastive learning. Despite of the simplicity of its structure, CLIP is surprisingly effective in visual-language learning and various downstream tasks [24, 42, 20] .Fashion Cross-modal Retrieval. Different from general domain, the fashion data involves large amount of domain-specific information with richer details than data from general domain, such as the brand, material, texture, composition, various of minute design differences, etc. Besides, the fashion text is composed of description and a group of tag entities (meta information), which describes the meta information of products, such as its sub-category, season, brand, to name a few. Wang et al. [45] enhances the task with knowledge graph [17, 37, 31] . A line of research pre-trains BERT-based models [10, 56, 12, 14, 12] that are tailored for fashion data. Fame-ViL [13] and FashionSAP [14] further formulate a multi-task learning framework with extra fashion-specific objectives. Different from existing BERT-based models, Chia et al. [4] and Ma et al. [27] are built upon powerful CLIP. While Chia et al. [4] directly finetune CLIP for continual learning [32] with fashion data, Ma et al. [27] improve its language learning with additional text encoders to diminish word ambiguity. However, they do not consider the visual discrepancy. To better learn from fashion images, which contain richer detailed information, we improve CLIP from the perspective of visual learning, enabling its image encoder to learn more fine-grained representations for better image-text alignment.Zero-shot Text-guided Image Retrieval. In Text-guided Image Retrieval (TGIR), also known as compositional image retrieval (CIR), users perform interactive dialogue to refine a given query image toward retrieving specific items. Classic models often employ custom models that project text-image pairs into a common embedding space. With the advance of VL foundation models (e.g., CLIP), interest in CIR has surged, especially in zero-shot settings without task-specific models [20] . We show that when combined with E 2 , which learns more fine-grained representations towards fashion product details, their performances can be further improved.Zero-shot Image Captioning. Zero-shot captioning [3, 46, 1, 42, 50, 24, 30] aims to generate image/video captions without human-annotated data. Different from above works, built upon CLIP, DeCap [24] and CapDec [30] use text-only data to train a decoder from scratch. We show that when combined with E 2 , which learns more fine-grained representations towards fashion product details, their performances can be further improved.We provide a detailed discussion on our innovation compared with existing works in Appendix G.3 Methodology i and z T i , respectively. During training, CLIP learns image-text matching from of N × N possible combinations by maximizing the similarity scores of N matched pairs while minimizing the scores of the rest N 2 -N mismatched pairs. The cosine similarity score of an image-text pair is calculated as z I i ⊙ z T i , (i, j ∈ {1, 2, ..., N }). In practice, CLIP optimizes a cross-entropy loss over the N × N similarity scores matrix, namely contrastive loss, denoting as:EQUATIONwhereL I2T = -1 N N i=1 log exp(z I i •z T i /τ ) N j=1 exp(z I i •z T j /τ ), andL T 2I = -1 N N i=1 log exp(z T i •z I i /τ ) N j=1 exp(z T i •z I j /τ ), τ is the temperature scalar. CLIP proves its effectiveness on various tasks in general domain. However, when adapting it to fashion domain, it is particularly difficult for CLIP to effectively learn from the fashion data due to its uniqueness in richer details and more compact layout. To overcome the challenges, we make full use of tag entities and learn more fine-grained representations with explicit (1) token fusion and selection, and (2) region contrastive learning.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: We use a section to discuss the limitation.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
regional contrastive learning of fashion representations: 
 We present details of our framework, Easy Regional Contrastive Learning of Expressive Fashion Representations (E 2 ) in Fig. 5 . The core idea of E 2 is "Easy" and "Expressive": it inherits the (1) simple design of CLIP with only a few inserted fusion blocks and selection tokens in the vision encoder, and (2) the simple learning objective: only contrastive learning objectives are used. Yet E 2 is more effective than existing large fashion VLMs with various objectives.In the following, we first describe the overall framework of E 2 , and then introduce the selection tokens, which are key elements for our (1) token fusion and selection, and (2) region contrastive learning. After that, we explain each component and finally summarize the whole training process.Framework. We build E 2 upon CLIP [34] , where we keep its text encoder unchanged and facilitate its image encoder with proposed selection tokens, fusion blocks and region contrastive loss. Similar to CLIP, given an image-text pair, E 2 learns one global embedding for each for the calculation of similarities for contrastive learning. Selection Tokens. Given an image, it is first split into P non-overlapping patches, which are linearly projected into a latent space, denoted as {p i } P i=1 . In addition to P image tokens, we further propose a set of selection tokens categorized into C categories, with S tokens for each sub-category, denoted as {s i } C×S i=1 . For instance, for the image encoder in Fig. 5 , C = 3 and S = 2, The two blue, red and yellow selection tokens are associated with the tag entity Brand, Sub-category and Season, respectively. To capture detailed information in image patches, each selection token updates itself with one most relevant image token in fusion block. Note that selection tokens are supervised by the region contrastive loss. We will discuss it later in this section. Multi-Stage Image Token Fusion and Selection. To learn more fine-grained visual representations with the help of tag entities in fashion language, we use a group of selection tokens to iteratively select most informative images tokens and merge themselves with the selected ones. In this way, selection tokens contain rich information about the details of their associated tag entities. For more fine-grained interactions, we drop less informative image tokens and only keep the global embedding and selection tokens as the input of the last transformer layer. As shown in Fig. 5 , we perform a multi-stage token fusion and selection to obtain the final global embedding. Given an image, we first obtain P image tokens {p i } P i=1 . Then they are concatenated with a set of selection tokens {s i } C×S i=1 and input to the image encoder, where we perform a multi-stage token fusion and selection. In each stage, input tokens sequentially go through a few transformer layers and a fusion block. Formally, suppose there are L stages, and during the l-th stage, we denote the input tokens as {p l i } P i=1 and {s l i } C×S i=1 . The information propagation with each group of transformer layers is performed:EQUATIONwhere [; ] means concatenation. Then the obtained image and selection tokens are fed to the fusion block, where each selection token selects one most related image token and updates the embedding of itself with the selected image token:EQUATIONAfter that, the output tokens serve as input tokens for stage l+1 if it is not the last stage. After obtaining the output tokens from the last stage L (via Eq. 3 with l=L), we only keep the global embedding p L+1 0 and selection tokens {s L+1 i } C×S i=1 as the input to the last transformer layer. To enrich the global embedding with fine-grained features related to given tag entities, we enforce it to focus on interacting with informative selection tokens by dropping less relevant image tokens to avoid their distractions. This step is critical for the image encoder to effectively learn more fine-grained visual representations with tag entities. As shown in Fig. 5 (a), the last transformer layer is applied on the kept tokens to obtain pL+1 0 . Finally, the global image embedding z I is obtained by applying a MLP to it:pL+1 0 = Transformer([p L+1 0 ; {s L+1 i } C×S i=1 ]), and z I = MLP( pL+1 0).(4) Fusion Block. In fusion blocks, selection tokens are fused with their most relevant image patch tokens to enrich themselves with entity-specific (e.g., brand, composition, etc.) visual information. Given a group of image patch tokens {p l i } P i=1 , denoting as a matrix P l , and a group of selection tokens {s l i } C×S i=1 , denoting as a matrix S l , we use selection tokens as queries to select the most relevant image tokens, which we call hard attention. For each selection token, it is updated by averaging itself with the selected image patch token. Specifically, denoting Q l , K l and V l are linear projections of S l and P l respectively: Q l = W q S l , K l = W k P l and V l = W v P l . Attention weight matrix A l is calculated by:EQUATIONwhere γ i is the i.i.d random sample drawn from the Gumbel(0, 1) distribution. To explicitly select one most similar image patch token for each selection token, we reformulate the attention weight matrix by turning each row A i,: into a one-hot representation with assigning one to the term with highest similarity score and zero to the rest. As the argmax operation is not differentiable, we use the gumbel-softmax and straight-through trick [8, 44, 48] :EQUATIONwhere the operator stop stops the gradients propagation. With one-hot vectors in the attention weight matrix A l , each selection token can pick up one corresponding image patch token and update itself with the selected one via:EQUATIONIn fusion blocks of different stages, selection tokens constantly select most relevant image tokens explicitly and enrich the representations of themselves with selected token embeddings for better selection and fusion in the next stage.At the last stage, the output of the fusion block only contains selection tokens, which already contain rich fine-grained information about associated tag entities (i.e., brand, composition, season and categories, etc.). They are input to the last transformation layer with the global image token, so that the global token can effectively interact with them without distractions from irreverent patch tokens. In this way, the global token better captures details of fashion images. Region Contrastive Learning with Selection Tokens. As each selection token is associated with a tag entity and it aims to select most relevant image tokens in fusion blocks, selection tokens are further supervised with the region contrastive loss. Specifically, assume we have a group of selection tokens {s i } C×S i=1 , which are categorized into C categories, with S tokens for each sub-category. For better illustration in this subsection, we reformulate them as {s c s } where c ∈ {'brand', 'composition', ...} (C categories in total) and s ∈ {1, 2, ..., S}. After going through the last transformer layer, each set of selection tokens {s c s } S s=1 with the same associated sub-category c is fed to its corresponding average pooling layer (Fig. 5 right), so that selection tokens from the same sub-category are formed to a more comprehensive and informative single pooled embedding s c f inal as:EQUATIONSimilar to the contrastive learning process with global image embeddings and global text embeddings introduced in section 3.1, each s c f inal learns to match the corresponding ground truth tag entity z T (c) via the region contrastive loss by replacing the global image embedding with the corresponding pooled selection token in Eq. 1:EQUATIONThe region contrastive loss explicitly aligns each s c f inal with the corresponding ground truth tag entity z T (c) for sub-category c. With region contrastive loss, pooled selection tokens are distinguishableText to Image SumR MeanR@1 R@1 R@5 R@10 R@1 R@5 R@10 ALBEF [22] 41 Text to Image SumR R@1 R@5 R@10 R@1 R@5 R@10 CLIP-VPT [18] Remark. Some semantic segmentation models, e.g., Seg [39] and GroupViT [48] also involve additional tokens. But the usage is notably different. As their motivation is to group image tokens to larger objects, each image token is assigned to an additional token, where softmax is applied to additional tokens. Because our motivation is to pay more attention to details of an object, each additional (selection) token selects a most relevant image token, where softmax is applied to image tokens. We carefully discuss the differences in detail in Appendix H.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [NA]Answer: [Yes] Justification:We will provide open access to both data and source code with sufficient instructions.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experiments: 
 Datasets. For a fair comparison, we first evaluate our model on the benchmark dataset Fashion-Gen [36] , following existing works [10, 56, 12, 27] . Besides, we also collect text descriptions and product images in fashion domain from Amazon Reviews [29] and build a large-scale fashion dataset which contains 1.3M image-text pairs, where we use 910K and 390K pairs for training and test, respectively. It is a more challenging dataset as the text descriptions are briefer and more general than FashionGen. We refer this dataset as AmazonFashion. More details are available in Appendix D.Settings. E 2 is initialized from the pre-trained CLIP [34] (ViT-B-32 [9] ) We use two selection tokens for each tag entity (S = 2), as Fig. 5 shows. We present the validation of this choice in later experiments. More detailed configuration and hyper-parameter setting are available in Appendix A. Evaluation. Our model is evaluated in various downstream tasks. For retrieval tasks, we perform two kinds of evaluations. The positive candidate is the ground truth item, while the negative candidates are either 100 randomly sampled unmatched items (referred as sample-100 evaluation) or all unmatched items in the dataset (full evaluation). Following [10, 27] , the evaluation metrics are Rank@1, Rank@5, Rank@10 and SumR=100(Rank@1 + Rank@5 + Rank@10).Note that full evaluation, which is suggested by latest works [27, 12] , is more challenging and practical and consistent with practical retrieval tasks. We compare with baselines with full evaluation unless they are not compatible with full evaluation, where we run Sample-100 evaluation instead.Cross Model Retrieval. For full evaluation on FashionGen, we compare our model with existing fashion VLMs and finetuned CLIP-based models. To compare with more baselines which are not compatible with full evaluation, following [10, 56] , we also do a Sample-100 evaluation. Because CLIP-FT and FILIP-FT outperform other baselines by a large margin, we compare E 2 with the two closest baselines on AmazonFashion with full evaluation. Table 1 shows E 2 notably outperforms other baselines in full evaluation, manifesting the effectiveness of paying more attention to certain regions that contain rich information about tag entities. Table 2 shows our model also outperforms baselines in sample-100 evaluation by a large margin. On the more challenging large-scale AmazonFashion dataset, E 2 consistently achieves better performance than competing baselines (Table 3 ).In Fig. 6 , we present examples of retrieval results by E 2 and CLIP to illustrate in which cases CLIP fails to differentiate minor differences towards visually similar products. In Fig. 6 (a), both CLIP and E 2 retrieved visually similar black trousers. While E 2 ranks the ground truth sample first, CLIP ranks visually similar product higher without considering its composition. . Fig. 6(b ) shows E 2 ranks the Gucci backpack first by recognizing its logo at face while CLIP ranks the Kara backpack first incorrectly. More visualizations are available in Appendix B.Zero-shot Image Captioning (ZS-IC). Built upon CLIP, E 2 can be easily extended to popular SOTA CLIP-based zero-shot image captioning models [24, 30] . DeCap [24] and CapDec [30] share similar high-level ideas: during training, a language decoder is trained to reconstruct the text input, where the CLIP text encoder serves as the encoder. During inference, with the CLIP image encoder, a given image is first encoded to a CLIP feature, which is later fed to the pre-trained decoder to generate captions. We train the decoder on text-only data from FashionGen but with our E 2 as the backbone encoder instead of CLIP. Results are shown in Tab. 4. As learnt features by E 2 contain more fine-grained information towards product specifications, generated captions with E 2 better matches ground truth captions, leading to high scores. We show the example in Fig 7 . Vanilla DeCap tends to make more mistakes towards details, such as brand, composition. While our improved DeCap with E 2 constantly yields better results.Zero-shot Text-guided Image Retrieval (ZS-TGIR). CIReVL [20] exploits pre-trained visionlanguage models (CLIP) alone with an LLM for ZS-TGIR without training. Similar to the case in ZS-IC, when replacing the CLIP encoders with E 2 , CIReVL is easily extended to fashion domain, consequently the model performance notably improves in Tab. 8. [27] and fine-grained CLIPs [49] are not necessarily our baselines, as our focuses are different and orthogonal to each other. Still we have a comparison to demonstrate the uniqueness and advantages of our model. Detailed results and discussions are presented in Appendix H.To study the effectiveness of each module and how much each group of selection tokens contribute to the our model, we design ablation studies from the two perspective: architecture and selection tokens. Architecture. We first ablate fusion blocks, where the input and output of each transformer layer areImage to Text Text to Image SumR R@1 R@5 R@10 R@1 R@5 R@10 FILIP-FT* [49] 6.2 17.8 25.8 6.2 18.1 25.9 100.0 CLIP-FT [34] 6 Method Shirt Dress Toptee Average R@10 R@50 R@10 R@50 R@10 R@50 R@10 R@50 PALAVRA [5] , ICCV'23 21. 49 Image to Text Text to Image R@1 R@5 R@10 R@1 R@5 R@10 Full method 62.8 89. all of the image batch tokens and selection tokens. Without the explicit selection and fusion process in the fusion blocks, the selection tokens, which are supervised with region contrastive loss and carry rich information towards tag entities, are still interacting implicitly with image patch tokens and the global token in ViT layers. In this way, we assume that, even without fusion blocks, the global token still learns richer information about tag entities and lead to better retrieval performance. Then we are curious to see how much the explicit token selection and fusion process benefit the learning process, in addition to the potential improvements from this implicit interaction in transformer layers. Table 9 shows both modules are critical. We also ablate our region contrastive loss (RCL).Group of Selection Tokens. We also study how each group of selection tokens contribute to the model performance. While all groups of selection tokens improve the model performance, their contributions are different. While Composition and Brand are more helpful, Season and Sub-category contribute slightly less. One potential reason is that they are easier to be visually distinguished than product texture and small logos, which especially require our fusion and selection process. We have detailed discussion on their effectiveness in Appendix F, and on parameter-efficiency in Appendix I.Parameter Sensitivity. We also examine the impact of batch size, which has a significant influence on the performance of contrastive learning. E 2 consistency surpasses CLIP-FT across a range of batch sizes, with greater improvement over CLIP-FT as batch size decreases. We present detailed results and analysis in Appendix C.In experiments, we use two selection tokens for each tag. It is reasonable as we are not selecting and fusing image patches in raw pixel space, instead, we conduct it with image patch (token) embeddings within ViT layers in contextual embedding space, where each token contains rich context/neighbor information. In fact, one token can already represent a large area if considering its neighbour information. Our choice of two tokens is empirically enough for all tags. We quantitatively validated this choice in Appendix I. 
limitations: 
 Built upon the pre-trained foundation VL model (e.g., CLIP), E 2 could be bottlenecked by the qualify of its large-scale pre-training in general domain. Besides, as our regional contrastive learning with fusion blocks and selection tokens explicitly aligns regions in the input image with the corresponding tag entities from text, intuitively, the quality of the provided tag entities can influence our model performance. Although in most cases, fashion data contain such information (e.g., brand, composition etc.), it is still a limitation if only datasets with poor tag entities are available for finetuning. 
conclusion: 
 In this paper, we propose a simple yet effective framework on learning fashion representations, and first emphasize the importance of learning fine-grained visual representations when applying CLIP to fashion domain. To achieve it, we further propose E 2 with selection tokens and region contrastive loss to enforce extra attention to details. Experimental results prove the effectiveness of E 2 .Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: The implementation detail is presented in the paper and the appendix.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [NA]Justification: We follow previous works to present the results.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: It is mentioned in the implementation details.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: It is discussed in the appendix.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification:Our model has little negative impact on society.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: They are all public.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
sheng li: 
 University of Virginia Charlottesville, VA 22904 shengli@virginia.edu Figure 1 : Model performance on downstream tasks. The first row shows the direct application of our model in cross-modal retrieval. The second row shows that, when replacing the CLIP encoders with ours in existing SOTA models in zero-shot text-guided image retrieval and fashion image captioning, their results can be further improved notably. (We show R@1 results for cross-modal retrieval, average R@10 for text-guided image retrieval, and B@4 for image captioning. )When learning vision-language models (VLM) for the fashion domain, most existing works design new architectures from vanilla BERT with additional objectives, or perform dense multi-task learning with fashion-specific tasks. Though progress has been made, their architecture or objectives are often intricate and the extendibility is limited. By contrast, with simple architecture (comprising only two unimodal encoders) and just the contrastive objective, popular pre-trained VL models (e.g., CLIP) achieve superior performance in general domains, which are further easily extended to downstream tasks. However, inheriting such benefits of CLIP in the fashion domain is non-trivial in the presence of the notable domain gap. Empirically, we find that directly finetuning on fashion data leads CLIP to frequently ignore minor yet important details such as logos and composition, which are critical in fashion tasks such as retrieval and captioning. In this work, to maintain CLIP's simple architecture and objective while explicitly attending to fashion details, we propose E 2 : Easy Regional Contrastive Learning of Expressive Fashion Representations. E 2 introduces only a few selection tokens and fusion blocks (just 1.9% additional parameters in total) with only contrastive losses. Despite lightweight, in our primary focus, cross-modal retrieval, E 2 notably outperforms existing fashion VLMs with various fashion-specific objectives. Moreover, thanks to CLIP's widespread use in downstream tasks in general domains (e.g., zero-shot composed image retrieval and image captioning), our model can easily extend these models from general domain to the fashion domain with notable improvement (Fig. 1 ). To conduct a comprehensive evaluation, we further collect data from Amazon Reviews to build a new dataset (Fig. 4 ) for cross-modal retrieval in the fashion domain.38th Conference on Neural Information Processing Systems (NeurIPS 2024). 
 
 Appendix: For FashionGen, following [27, 10] , the model is trained for 20 epochs. The weight decay is set to 1e -4, and the learning rate is set to 5e -5 with the cosine annealing learning rate decay scheduler applied. As the selection tokens are randomly initialized, to match them with the pre-trained CLIP model, we first freeze other parameters and train selection tokens with an initial learning rate 5e -4 for 5 epochs. The default batch size is 64. Configurations are the same with AmazonFashion, while the epoch is set to 10.We provide more retrieval results in Fig. 15 and Fig. 16 . For each example, the first row is the top-5 retrieval result by E 2 and the second row is the result by CLIP. Ground truth is boxed in green.To intuitively see what is selected by selection tokens, we visualize selected tokens in the last stage in Fig. 14 . Results show our selection tokens are effective in finding regions that contain detailed information about the tag entities, e.g., brand, composition. In the last transformer layer, only selection tokens and global token are kept (Eq. 4) (image tokens are dropped) to allow the latter effectively learns fine-grained representations from the former. We also visualize what the global token of CLIP focuses by marking image tokens with the maximum attention values with it for each attention head (12 in total). Note the overlap exists, i.e., one image token may have the maximum attention value with the global token in multiple attention heads. Results show CLIP is ineffective in finding tag entity-related information and sometimes focuses on totally unrelated regions. It is observed that visually dominant areas which are closer to the global view of the product, e.g., dark areas in a dark product in Fig. 14 (g), (i), (j), tend to win higher attention scores with the global token. Consequently, regions with detailed information such as logos are overlooked. One potential reason is that, pre-training data on general domain often consist of a simple caption and a few very distinctive objects (Fig. 2 ), which enables CLIP to capture features better from a high-level view.In this case, focusing more on a global view of objects instead of a local view helps the model to differentiate large and distinctive object in an image. However, when it comes to fashion domain, where multiple details of a single product are required to be aware of, CLIP fails because CLIP still tries to capture the global view of an object and ignores local view, even through it has already been finetuned. By contrast, with our model, Brands are all picked. Composition prefers solid color areas, where material is more clear. Sub-category is reflected by sleeves, collar and shoulder. Sleeves and front zippers reveal the Season.We also investigate the impact of batch size, which can be substantial to contrastive learning. Results in Tab. 5 show that E 2 consistency outperforms CLIP-FT across different batch sizes. Our findings indicate that E 2 consistency surpasses CLIP-FT across a range of batch sizes, with E 2 showing greater improvement over CLIP-FT as batch size decreases. Notably, it is especially helpful in scenarios where hardware limitations impose constraints on batch size, where E 2 's superiority becomes more evident.Text to Image SumR R@1 R@5 R@10 R@1 R@5 R@10 FashionGen [36] is the existing benchmark dataset for fashion cross-domain retrieval. Existing works [12, 10, 27, 56] evaluate the performance of their model on fashion cross-domain retrieval with this dataset.FashionViL [12] uses two dataset: FashionGen [36] and FashionIQ [47] for fashion domain retrieval. The critical difference is: FashionIQ [47] is for text-guided image retrieval (TGIR), a special type of image retrieval problem, while FashionGen [36] is for cross-modal retrieval. Similar to FashionGen, our AmazonFashion is built for the task of cross-modal retrieval.Compared with M5Product [7] and Product1M [51] , ours has two distinctive advantages: (a) Larger fashion data size: Product1M is designed for cosmetics and groceries, not for fashion products. In all 6.3M data in M5Product, 0.5M is in fashion domain, while ours is 1.3M. (b) 3.1 times conciser caption in average, leading to a more practical case: in practice, users usually search products online with short queries, instead of a long paragraph of detailed text in FashionGen style. Our dataset better matches this more challenging and practical case. We show several examples from AmazonFashion and FashionGen in Fig. 10 to highlight our differences between these two benchmarks. We also present detailed statistics of two datasets in Tab. 6 and Fig . 12 . In general, AmazonFashion is closer to the practical cross-model retrieval scenario.Tab. 6 shows that AmazonFashion is different from FashionGen in notably larger size (1.3M), a wider variety of brands and products (67 and 8 times more, respectively), more concise and general descriptions (0.6 times shorter), and more diverse image scopes. These differences make AmazonFashion both more challenging and closer to real-world scenarios, where query languages from users are often more general and more concise, meanwhile, the number of candidate products is usually notably greater.We also present the brand frequency plot in Fig . 12 . For each dataset, we selected the top 200 brands with the highest frequency of occurrence and plotted them in a descending order. Note that for the AmazonFashion plot, we excluded the brand amazon collection, which has a frequency of 19, 714, significantly higher than the other brands. Both datasets exhibit a long-tail distribution in terms of brand frequency, which accurately reflects real-world scenarios. Besides significant differences in data statistics, images in AmazonFashion also exhibit substantial variations. While FashionGen only provides regular views of a product from different angles such as Fig. 10(e) , AmazonFshion includes more diverse images, such as product pictures from different domains in Fig. 10(a) , and high-resolution images that showcase intricate product details in Fig. 10(ac ). It is close to practical scenarios where available product images on website often vary in styles. It also brings new challenges as better adaption to images at different scales and styles is excepted. Thus, the AmazonFashion holds significant value for researches on cross-modal retrieval in the fashion domain, and we encourage its use in future work.We report results of representative CLIP-based models, including ours on AmazonFashion in Tab. 7.We show the distribution of the frequency of top 20 most frequent brand in Fig 11 .Text to Image SumR R@1 R@5 R@10 R@1 R@5 R@10 Linear probing is a popular technique to evaluate the quality of learned embeddings of a model: fitting a linear classifier to the image embeddings of the model. Typically, the better the model is trained, the higher performance a classifier achieves, because a well-trained model tends to give more informative embeddings, from which a classifier learns better.To evaluate how well CLIP, CLIP-FT, FILIP-FT and E 2 are able to learn entity-specific knowledge from images, we perform four classification tasks. More specifically, for each task, we train a linear classifier towards one of four tag entities (i.e., to classify: Brand, Sub-category, Season and Composition) on image embeddings of CLIP [34] , CLIP-FT [34] , FILIP-FT [49] , and E 2 , respectively. We use the Adam optimizer and train each classifier for 20 epochs with learning rate = 0.001. Results are shown in Fig. 3 . Results show classifiers trained with E 2 embeddings consistently outperform classifiers trained with CLIP, CLIP-FT and FILIP-FT embeddings, indicating that E 2 embeddings contain more information about tag entities. This can be attributed to our proposed token fusion, token selection, and region contrastive loss, which explicitly forces the model to pay special attention to these details.Besides, we delve deeper into the linear probing results, which further reveal several interesting facts on how each tag entity impacts the model performance. Fig. 13 (Brand) indicates that while fine-tuning can enhance CLIP's understanding of product brands, the extent of improvement is comparatively small when contrasted with the significant progress achieved through E 2 . By comparison, Fig. 13 (Season) shows fine-tuning is effective for CLIP to learn about produce season. However, when it comes to Fig. 13 (Sub-category), fine-tuning almost yields no benefits. We assume that fine-tuning can only help learn easy classes such as bag, jeans, shirts, etc., while E 2 can help difficult classes. In this case, because the vanilla pre-trained CLIP is already effective in recognizing high-level concepts, fine-tuning fails to yield further improvements. E 2 significantly benefits the learning of product categories by learning more about difficult classes. In Fig. 13 (Composition), although the relative magnitudes are similar to that in Fig. 13 (Brand), the absolute differences among four models are much smaller, which indicates effectively learning about product composition is a harder task.To study how each group of selection tokens contribute to the model performance, we train and evaluate E 2 without one group of selection tokens associated with one tag entity at a time and report the result in Table 8 . While all groups of selection tokens improve the model performance, their contributions are different. Selection tokens associated with Brand and Composition are more helpful. As Fig. 15 and Fig. 16 show, paying special attention to the brand and composition of a product helps our model better differentiate visually very similar items. By contrast, Season and Sub-category contribute less as they are easier to be visually distinguished than product texture and logos, which are harder to discriminate and especially requires our fusion and selection process with the selection tokens.Text to Image SumR R@1 R@5 R@10 R@1 R@5 R@10 E 2 (In this section, we discuss our differences from related works, and our innovation (motivation and technical contribution).The design of selection and fusion mechanism with fusion blocks and selection tokens is our new technical contribution, which notably improves the performance. Typical CLIP-based adaption models (e.g., RegionCLIP [54] , FILIP [49] , DenseCLIP [35] , FashionCLIP [4] ) are built upon CLIP without modifying its inner structure. However, we break into the CLIP encoder by introducing tag-aware fusion blocks. Specifically, the above models, including FashionSAP [14] , do not explicitly align image patch tokens within ViT layers with text information, which we prove to be critical in fashion domain where the details do matter. To this end, our tag-aware selection tokens select and fuse image patch tokens. They differ from typical new additional tokens (VPT [18] , DeiT [43] ), which are simply added to input layers and structurally similar to the CLS token. Our adaptation of CLIP is new and first tailored for fashion scenarios with tag entities, where general CLIP-adaptation models are hard to handle (more details in Appendix H).Our highlight is not improving the CLIP vision encoder in general domain, as existing works such as FILIP [49] do. Instead, we are among the first to adapt CLIP to fashion domain. When combined with SOTA CLIP-based zero-shot image captioning models, their performance can be obviously improved.Most recently, several works [49, 21, 11] start to improve language image pre-training with semantic alignment. FILIP [49] and LOUPE [21] learn better representations with semantic alignment of patch tokens. HiCLIP [11] learns to discover hierarchy in data. GroupViT [48] also forms semantic segments hierarchically, but it only focuses on semantic segmentation. Our motivation differs from above methods in that, while they learn representations of larger objects via grouping smaller regions (i.e., image tokens), we focus on certain regions with detailed information of an object, e.g., its brand and composition. Besides, they focus on designing models for better image-text matching generally without considering the uniqueness of fashion data. Our E 2 pays additional attention to image details associated with tag entities for better visual learning (more details in Appendix H). SumR R@1 R@5 R@10 R@1 R@5 R@Differences from Segmentation Models. Some semantic segmentation models [39, 48] also involve additional tokens, including GroupViT [48] . We are very different from each other in: (1) motivation, (2) selection technical details, and (3) overall framework. In summary, group tokens in GroupViT serve as "collectors", thus image tokens are iterated to be allocated to a group token. By contrast, in E 2 , the process is opposite: selection tokens are iterated to select one most relevant image token. We do attention with different objects therefore are not interchangeable. GroupViT cannot select the most relevant image tokens, while ours does not support grouping of them. Our framework design is also significantly different. GroupViT adopts an iterative approach, where grouped tokens are consecutively fed as inputs to transformer layers, introducing new group tokens at each stage. In contrast, our approach utilizes image tokens along with a fixed number of selection tokens as inputs.To sum up, due to different tasks/objectives, GroupViT and E 2 are very different in both technical details and the overall framework.Comparisons with EI-CLIP. While we facilitate the image encoder, EI-CLIP [27] improves the text encoding process by reducing word ambiguity in tag entities with additional text encoders. Thus we are orthogonal to each other. Besides, EI-CLIP uses additional text encoders and during inference, tag entities must be separately encoded by the encoders. In contrast, E 2 does not require separated tag entities as text input. In our model, same to existing works [10, 56] and CLIP, tag entities are concatenated with the description as one string, which is input to the CLIP text encoder. This is more practical and especially helpful in cases where tag entities are mixed with descriptions and separating them is infeasible. We also evaluate EI-CLIP under this setting, which follows existing works [10, 56] and ours (w/o E. in Table 9 ). Although EI-CLIP has additional encoders and requires separated tag entities during inference, it is still inferior to ours according to Table 9 . When evaluating under the same setting, the gap is more notable.Differences from Fine-grained CLIPs. Several recent works [49, 11, 21] propose to better match image-text pairs with semantic alignment or hierarchical grouping. While they learn representations of larger objects via grouping smaller regions (image tokens), we focus on finding certain regions with detailed information of part of an object (e.g., its brand, season, composition), motivated by the uniqueness of fashion data. We achieve this by designing a better way to exploit tag entities, i.e., introducing selection tokens associated with these entities, while they seek for a general way to better align an image-text pair without considering the of characteristics of fashion data. Thus our work is orthogonal to theirs and we can build E 2 based on their pre-trained models. We choose CLIP in this work as it is the most representative one, based on which makes it extendable to others. Because codes and pre-trained models of mentioned CLIP variants are not released, we are not able to evaluate all of them. However, as FILIP [49] shares the same structure with CLIP, we implement it and initialize it with a pre-trained CLIP model and finetune it on our datasets. We also use FILIP as the backbone model in our ablation study.In this section, we explore 1. how the number of selection tokens can influence our model performance and 2. the parameter efficiency of our model. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.Justification: We follow the guidelines of Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 


Decision: The reviewers generally appreciated the proposed approach, which is specifically designed for visual recognition in the fashion domain, where fine-grained visual representations are crucial for strong performance. They acknowledged that the core idea of using selection tokens to attend to specific fashion attributes is interesting and that the paper provides ample and convincing empirical evidence to support its effectiveness. While there was some concern about the limited scope of the paper's application to the fashion domain, the reviewers agreed that this alone should not be a reason for rejection, as the paper in its current state merits acceptance. After carefully reading the reviews and the rebuttal, this meta-reviewer concurs with the reviewers' unanimous recommendation for acceptance.

Reviews: review: paper_summary: paper_summary: This work proposes a new approach to learn improved and more expressive visual representations for fashion-specific tasks. Prior works have proposed to learn fashion representation either by complex multi-task objectives or fine-tuning strong pretrained visual features such as CLIP. However, such methods fail to learn strong features that can distinguish between fashion-specific attributes/entities that are often available in image descriptions. Authors propose E^2 which add new layers over CLIP visual encoder along with selection tokens for different entities (such as brand, composition). They propose to attend these selection token with the image tokens such as each selection token can select a single image token. They use contrastive loss to supervise the network training and also use separate contrastive losses to supervise each of the selection tokens (supervision is provided from the categories). Authors have shown strong improvements over SOTA in image retrieval as well as zero-shot captioning task highlight the efficacy of the proposed representations. They also show some qualitative results to show the benefits of the proposed approach.
strength_weakness: strength_weakness: - Authors could have done a better job at presenting their approach and highlighting the core novelty (use of fusion blocks with selection tokens) in the introduction and abstract.
- Although the paper was easy to follow- the presentation could have been improved. For example the figures were often small to read.
questions: questions: - Why did the authors use hard-attention for the selection tokens. Was there a difference in performance if doing soft-attention? The visualization of the selection tokens in figure14 is not showing much difference (the attention seems to be clustered around a few points). This leads to the question if the selection tokens are interpretable

- Why did the authors choose only one fashion dataset from prior works (other is proposed in this paper). Why weren't other datasets considered?

- What is the benefit of using >1 selection tokens for each sub-category?

- Would this work also generalize to other domains. If yes, what is needed to achieve this (e.g. detailed description + specific attributes)
limitations: limitations: Yes
paper_summary: paper_summary: The paper discusses adaptation of CLIP for fashion domain w/ an emphasis on importance of learning fine-grained visual representations. The authors propose E^2 with selection tokens and region contrastive loss to enforce extra attention.
strength_weakness: strength_weakness: Minor comment: the paper could be made a bit easier to understand for the general ML audience attending NeurIPS
questions: questions: none
limitations: limitations: not mentioned
paper_summary: paper_summary: This paper proposes a framework to train CLIP models more adaptable to fashion domain (shopping item images and structured descriptions, i.e, “tags”, such as brand, composition etc.). The new framework (“E2”) outperforms other common methods by a large margin on a wide range of tasks, by 1) adding fusion blocks in image encoder to select more representative patches for tags, and 2) adding tag specific contrastive losses.
strength_weakness: strength_weakness: The efficiency evaluation is missing. It’s unclear how the training/inference speed is affected by the newly added components in CLIP.

Besides retrieval and captioning, visual question answering is another common task for vision language modeling. It would be more solid if we add another eval on CLIP based VQA models.

The impact is mostly limited in the fashion domain. It would be more impactful if more applications would be found for the proposed method.
questions: questions: It’s common to have the same tags from different images (such as the same brand of different images of t-shirts, shoes…). This could lead to many false-negatives in contrastive training, and may explain why the fusion block is necessary. Have you identified such an issue and considered any solution for this (such as “soft” contrastive labels)?

In Figure 5, it would be more clear if the locations of a/b/c can be explicitly stated, such as “a - Left, b - Right, c - Middle”.

What is the token length of text encoder?

In  Parameter Sensitivity section, is the batch size’s unit thousand (otherwise looks too small for contrastive learning)?
limitations: limitations: N/A
paper_summary: paper_summary: This paper focused on adapting CLIP-based VLMs to the fashion domain. The motivation is that directly finetuning CLIP models on fashion data will lead to insufficient learning of entity-related details like logos and composition. To tackle this challenge, this work proposed a region-based contrastive loss by inducing selection tokens to capture information on tag entities like brand, sub-category, and season, and further applied region contrastive loss on these token embeddings. Also, this work collected a new dataset called AmazonFashion which consists of briefer and more general text descriptions compared to the previous FashionGen benchmark. Experiments on several benchmarks validated the effectiveness of the proposed method.
strength_weakness: strength_weakness: \- The proposed method works well for the fashion domain but is not evaluated on general domains for other fine-grained recognition or captioning tasks. This is not a reason to reject the paper, but limits the scope of the work. This work is a good application work and proposed a reasonable solution, but the insights are not very strong for other domains or tasks.
questions: questions: I have no further questions for the authors.
limitations: limitations: The authors have adequately addressed the limitations.

Comments: comment: We sincerely thank you again for your efforts in reviewing this paper and giving valuable suggestions!

Would you mind checking our responses to see if your previous concerns/questions have been addressed? We highly value your feedback, and if you have any more questions or suggestions, we look forward to having further discussions with you.

Thanks!
Thank you for your prompt response! We are glad to hear that our explanation addressed most of your questions and that you are comfortable maintaining the positive rating. Your feedback is greatly appreciated!
Thanks for the detailed explanation, which has addressed most of my questions. I am happy to maintain my previous rating.