LRVS-Fashion: Extending Visual Search with Referring Instructions

Authors: Simon Lepage, Jeremie Mary, David Picard
Primary area: datasets and benchmarks
Keywords: Visual Search, Image Embedding, Retrieval, Dataset

Abstract:
This paper introduces a new challenge for image similarity search in the context of fashion, addressing the inherent ambiguity in this domain stemming from complex images. We present Referred Visual Search (RVS), a task allowing users to define more precisely the desired similarity, following recent interest in the industry. We release a new large public dataset, LRVS-Fashion, consisting of 272k fashion products with 842k images extracted from fashion catalogs, designed explicitly for this task. However, unlike traditional visual search methods in the industry, we demonstrate that superior performance can be achieved by bypassing explicit object detection and adopting weakly-supervised conditional contrastive learning on image tuples. Our method is lightweight and demonstrates robustness, reaching Recall at one superior to strong detection-based baselines against 2M distractors.

Decision: reject
Reviewer scores: 8, 5, 5, 3

Reviews and discussion:
<thread>
creation date:
1730272927090

thread_type:
official_review

<thread_object>
creator: reviewer_jqz1

<content>
summary:
This paper introduces a new task, Referred Visual Search, which retrieves specified products within a given image based on user requirements, and constructs a new dataset for this task. Additionally, a CLIP-based model architecture is proposed and its performance is validated on the newly created dataset.

strengths:
1.	The proposed new task has practical application value in the e-commerce domain.
2.	The illustrations are clear and detailed, facilitating understanding.

weaknesses:
1.	The significance of the proposed task needs further clarification. What distinguishes Referred Visual Search from text-guided visual localization or composed image retrieval? Can existing methods in these areas be adapted to this task? 
2.	The paper lacks novelty, as applying the CLIP architecture in the e-commerce domain has been previously explored [a-c]. The authors need to further explain the innovation of their proposed method compared to existing methods. 
3.	The experiments require further refinement. Given that the authors utilized methods from composed image retrieval for performance comparison, it is essential to validate the proposed method on the FashionIQ and Shoes datasets within this context to effectively demonstrate its efficacy.
4.	The proposed method incorporates the Grounding DINO and SAM models; thus, a discussion on the fairness of performance comparisons with other methods is warranted. Additionally, the time and space complexity of the proposed method should be compared against other representative methods to provide a comprehensive evaluation.

[a] Han Y, Zhang L, Chen Q, et al. Fashionsap: Symbols and attributes prompt for fine-grained fashion vision-language pre-training[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 15028-15038.

[b] Zhou J, Zheng X, Lyu Y, et al. E-clip: Towards label-efficient event-based open-world understanding by clip[J]. arXiv preprint arXiv:2308.03135, 2023.

[c] Han X, Zhu X, Yu L, et al. Fame-vil: Multi-tasking vision-language model for heterogeneous fashion tasks[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 2669-2680.

questions:
Please refer to the Weaknesses section and address my concerns regarding the task setup, methodological innovation, and performance of the proposed method.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730576329658

thread_type:
official_review

<thread_object>
creator: reviewer_rist

<content>
summary:
This paper proposes a new visual search task with referring instructions as the hints.
It also creates a new large public dataset, LRVS-Fashion, as the benchmark.
The authors also introduce a CLIP-like simple baseline.

strengths:
1. The efforts on collecting the benckmark is worthy of applause. This dataset could be potentially used for other fashion-related tasks.
2. The writing of this paper is good and easy to follow.

weaknesses:
1. I am not convinced on the motivation of the proposed task. I cannot fully imagine a real application scenario for the proposed Referred Visual Search. According to my experience and understanding on fashion/ads, the query image (ie, dressed human) and the target image (ie, garment, product and so on) shown in Figure 1 are normally from the same product set which provided by the advertiser (at least they share some common information, eg, brand name). In that case, we don't need to do large-scale image-base retrieval, which is implementation-wise expensive in the industry. The only case I understand is letting the user themselves upload their individual query images and do a cross-domain image retrieval. However, the target and query image used in this paper are from the same domain and thus    cannot model the real world problem, although I understand collecting cross-domain benchmark is even harder.
2. I believe the proposed RVS is just a sub-task of Composed Image Retrieval (CIR), while their relationship is not clearly discussed (between L147 and L153) and the latest CIR related works were not cited. Regarding "rather than modifying the image" in L151, I think getting the feature of the ROI of the query image (Figure 4) is also kind of modifying the query image feature.
3. The proposed model arch is okayish but not that innovative (L081) to me. I agree this can be regarded as a strong baseline but don't think this widely used arch can be claimed as innovative.
4. I think some zero-shot methods (no training needed) should be considered as baselines too, eg, multimodal LLM that can fuse the query image feature and the instructions.

questions:
Please refer to the weakness part.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730650739774

thread_type:
official_review

<thread_object>
creator: reviewer_mvge

<content>
summary:
This paper introduces Referred Visual Search (RVS) in the fashion domain. The paper makes two main contributions: (1) A new large-scale dataset called LRVS-Fashion and (2) A weakly-supervised method that outperforms detection-based approaches.

strengths:
1. The paper proposes a high-quality dataset that will be beneficial to the field
2. The paper is well-written and easy to read.

weaknesses:
1. The proposed task is like a special case for the task of composed image retrieval(CIR). However, the experiment is not compared with recent CIR methods, which makes the result not convincing. 
2. The proposed method shows high performance in LRVS-F, as shown in Table 2, nearly 99% in Cat@1. These superior performances raise the question of the proposed dataset's motivation. Is there any specific problem the dataset wants to diagnose?

questions:
see weakness
</content>
</thread_object>

</thread>
<thread>
creation date:
1730679164652

thread_type:
official_review

<thread_object>
creator: reviewer_dwga

<content>
summary:
This paper introduces the Referred Visual Search (RVS) task, addressing the challenges of image similarity search in the fashion domain, particularly in scenarios where product images need to be retrieved from query images containing multiple products or from model pictures. The authors present a large dataset, LRVS-Fashion, containing 272k fashion products and 842k images, designed for this task. They propose a novel method based on weakly-supervised conditional contrastive learning, achieving superior performance without relying on explicit cropping or segmentation. The study demonstrates the effectiveness of their approach in comparison to a variety of baselines. The paper is overall well written.

strengths:
The introduction of RVS as a task for fashion images addresses a gap in the current literature and reveals significant value for practical applications such as e-commerce systems.

The LRVS-Fashion dataset is extensive in terms of its scale, organization, and cleaning process, providing valuable resources for future research and enhancing the field's accessibility.

The proposed approach, based on conditional contrastive learning, is intuitive yet elegant. By avoiding explicit object detection or segmentation, it is efficient and easier to implement in various applications, particularly by reducing the extra stages of data collection or model design. In addition, it demonstrates impressive robustness against a large number of distractors, which is critical in real-world applications.

The paper provides a thorough literature review and extensive comparisons with existing methods. For example, it compares with compositional image retrieval methods that share a conditional learning requirement but targeting at different application scenarios. The experiment setup is clear and the overall results are convincing in terms of accuracy and efficiency.

weaknesses:
While the experimental results are solid, it could be more comprehensive if the analysis can go deeper into the impact of objects' visibility in query images considering object sizes, occlusions or viewpoints. Or, a simpler referring aspect could be segmentation mask area or its ratio to a detected object bounding boxes? One example is how the retrieval accuracy changes among shirts because they are often partially visible due to overlaid with outwears.

questions:
The study focuses primarily on fashion images and leverages LAION-5B to create the benchmark dataset. While it serves as a valuable testbed for future research, it would be interesting to discuss how to construct larger datasets in real-world scenarios and the challenges associated with generalizability to close domains such as street outfit photos that have noisy backgrounds, or even beyond fashion such as furniture, landmarks, etc. This would further strengthen the impact of this work?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732548144134

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
Dear Reviewers,

We would like to thank you for your thoughtful feedback. Addressing your questions and comments, particularly regarding the positioning with respect to Composed Image Retrieval (CIR), has significantly improved our work.

Below, we outline our conclusions based on the reviews, addressing the main strengths and weaknesses highlighted by the reviewers. We also provide a summary of the additional evaluations conducted during the rebuttal period.

### **Main Strengths**
We are pleased to see that all the reviewers agreed on the value of our task and dataset for the e-commerce and fashion fields.

> *“The LRVS-Fashion dataset is extensive in terms of its scale, organization, and cleaning process, providing valuable resources for future research and enhancing the field's accessibility.”* - Reviewer DWgA

> *“The paper proposes a high-quality dataset that will be beneficial to the field.”* - Reviewer MVgE

> *“The efforts on collecting the benchmark is worthy of applause. This dataset could be potentially used for other fashion-related tasks.”* - Reviewer risT

> *“The proposed new task has practical application value in the e-commerce domain.”* - Reviewer jqz1

Additionally, all reviewers highlighted the quality of the writing and illustrations, which facilitate a better understanding of our work.

### **Main Weaknesses**
The main concern regarding our task is its positioning relative to CIR. Reviewers MVgE and risT suggest that Referred Visual Search (RVS) can be seen as a special case of CIR, while Reviewer jqz1 requested additional clarification.

Recent work on Conditional Image Similarity (CIS) [1] enumerates CIR and RVS as two sub-tasks of CIS. Both share the same type of input (image + conditioning text) and output (embeddings), and are retrieval tasks, but their goals differ. Notably, CIR introduces a “negative similarity condition” used to edit the image, whereas RVS uses “referring instructions” to focus on a specific part of the image.

Our RVS task aligns with the unnamed “Focus Object” task described in the GeneCIS benchmark [1]. However, while they propose evaluating general-purpose models on small galleries (15 images), we use a specific fashion setting with a large gallery (2 million images) that better aligns with real world use cases.

Our work contributes to the existing CIS benchmarks, and results from CIR models confirm that the tasks are distinct.

[1] Vaze, S., Carion, N., & Misra, I. (2023). Genecis: A benchmark for general conditional image similarity. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition


### **Summary of new experimental results**

Following the reviewers' questions and comments, we conducted several new evaluations during the rebuttal period, which we have incorporated into the revised version of the manuscript.

**In-depth analysis** - Reviewer DWgA’s suggestion regarding the size of the items and their occlusion led us to the computation of per-category and per-size metrics. The results confirm a general weakness across all methods for small objects, which is particularly detrimental to certain categories (near 20% R@1 for the smallest object categories “Waist”, “Feet” and “Hands” compared to above 80% for the largest object categories “Upper Body” and “Whole Body”).

**New baselines** - we added 3 models to our baselines:
- MiniCPM-V 2.6 (suggested by Reviewer risT) - A SOTA VLM method, which highlights an interesting research direction in adapting VLM to CIS tasks, reaching 9.2% R@1.
- MagicLens - A SOTA zero-shot CIS method trained on a large scale dataset, which shows that our benchmark is much more challenging than existing CIS benchmarks, reaching 7.5% R@1.
- FAME-ViL (suggested by Reviewer jqz1) - A fashion multi-task model trained on CIR, which shows that fashion RVS differs from fashion CIR, reaching 9.3% R@1.

These new results confirm that dedicated methods, like our simple conditional ViT approach (reaching 69.3% R@1), are still necessary for RVS (see Table 2 of the revised version).

Modifications to the paper made for the revision can be accessed with the OpenReview tool (Revisions → Compare Revisions).
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper was reviewed by four experts in the field. The paper received mixed review ratings of 8553. 

Reviewer DWgA is the most positive reviewer, and they liked the proposed e2e method. The major concerns from other reviewers are:
1. The positioning of the work, being a special case of composed image retrieval (CIR)
2. The novelty and innovation of the method.

The AC read the paper, reviews, and rebuttal carefully. The ACs believed that the concerns around the positioning of the method and the technical advancement this paper brings to the research community outweigh the paper's strengths. While the paper clearly has merit, we regret that the paper cannot be recommended for acceptance at this time. The authors are encouraged to consider the reviewers' comments when revising and resubmitting the paper.