Title: Highlighting What Matters: Promptable Embeddings for Attribute-Focused Image Retrieval
Venue: NeurIPS 2025
Accepted: yes
Keywords: text-to-image retrieval

image embeddings
Abstract: While an image is worth more than a thousand words, only a few provide crucial information for a given task and thus should be focused on. In light of this, ideal text-to-image (T2I) retrievers should prioritize specific visual attributes relevant to queries. To evaluate current retrievers on handling attribute-focused queries, we build COCO-Facet, a COCO-based benchmark with 9,112 queries about diverse attributes of interest. We find that CLIP-like retrievers, which are widely adopted due to their efficiency and zero-shot ability, have poor and imbalanced performance, possibly because their image embeddings focus on global semantics and subjects while leaving out other details. Notably, we reveal that even recent Multimodal Large Language Model (MLLM)-based, stronger retrievers with a larger output dimension struggle with this limitation. Hence, we hypothesize that retrieving with *general* image embeddings is suboptimal for performing such queries. As a solution, we propose to use *promptable* image embeddings enabled by these multimodal retrievers, which boost performance by highlighting required attributes. Our pipeline for deriving such embeddings generalizes across query types, image pools, and base retriever architectures. To enhance real-world applicability, we offer two acceleration strategies: Pre-processing promptable embeddings and using linear approximations. We show that the former yields a 15\% improvement in Recall@5 when prompts are predefined, while the latter achieves an 8\% improvement when prompts are only available during inference.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces promptable image embeddings to enhance attribute-focused text-to-image (T2I) retrieval. The authors propose a new benchmark, COCO-FACET, and show that existing CLIP-like and multimodal retrieval models perform suboptimally on fine-grained attribute queries. They demonstrate that conditioning image embeddings on attribute-specific prompts substantially improves retrieval accuracy. Additionally, two acceleration strategies are presented to facilitate practical deployment.\n\nSTRENGTHS AND WEAKNESSES:\nPros:\n1. The authors clearly identify the limitations of global image embeddings and propose a well-motivated approach, which is highly relevant for T2I retrieval tasks.\n2. The introduction of the COCO-FACET benchmark is a valuable contribution, as it encourages further research into the retrieval of fine-grained visual attributes.\n\nCons:\n1. The proposed method is somewhat similar to previous works, such as CLOC and FLAIR, which also generate task-specific embeddings. The authors should more explicitly clarify the distinctions and advantages of their approach over existing methods.\n2. The method utilizes GPT-4o to generate prompts that guide image embeddings. However, since GPT-4o is itself a state-of-the-art multimodal large language model, there is a concern that GPT-4o alone might already solve the retrieval task independently, without the need for additional models like VLM2Vec. The paper does not evaluate this possibility or clarify whether GPT-4o serves solely as a prompt generator or is effectively part of the retrieval pipeline.\n3. The inclusion of acceleration strategies is a positive aspect. However, the paper lacks a detailed comparison of the actual inference time between the proposed approach and prior methods. Providing such a comparison would enhance the practical significance of the work. \n4. While the authors state that their method can handle attributes such as weather and time, supported by experimental results, the paper does not clearly explain how promptable embeddings capture these global or abstract concepts. A more detailed explanation or visualization of how these attributes are encoded, as well as an analysis of failure cases, would strengthen this point.\n\nQUESTIONS:\nPlease refer to the section outlining the weaknesses of the paper. I look forward to the authors' responses to these questions.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on the attribute-focused T2I retrieval task, where queries emphasize specific attributes of the target image. Motivated by this setup, the authors introduce the COCO-FACET benchmark, built upon the COCO dataset, to evaluate performance in this setting. They also propose a promptable embedding method with MLLM model. I believe the motivation is strong and dataset can be a contribution, but there are some concerns on assumptions that used in method\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n1. The targeted problem (attribute-focused) T2I task seems important and reasonable.\n2. The suggested dataset seems reasonable and constructed carefully to cope with intended goal. \n3. The paper is easy to follow and the related works are clearly explained.\n4. The baselines are fairly well compared with the recent ones. \n\n\n**Weaknesses**\n1. **Scalability and Assumptions of the Method**\nA key concern lies in the computational assumptions underlying the proposed method. As the authors themselves note, the promptable embeddings need to be recomputed for the entire gallery set for each query—if this is indeed how the original implementation works, it would be infeasible for large-scale gallery sets common in real-world applications. Even with the proposed linear approximation, the computational cost remains significantly higher than traditional T2I retrieval methods—potentially up to 100× more expensive for K=100.\nFurthermore, both acceleration strategies rely on assumptions that may not hold in practice, such as prior knowledge of the target category or the ability to pre-filter gallery images by category. These assumptions undermine the method’s applicability in realistic settings. The paper would benefit from an alternative approach that avoids recomputing gallery representations for each query. I think the current direction for the method would not be feasible in even a moderate size index set (gallery) setting.\n\n2. **Dataset Visualization and figure**\nMore concrete visualizations of the dataset would improve clarity and help readers better understand the proposed benchmark. Additionally, Figure 1 is potentially misleading, as the query text appears to be embedded within the gallery set, which may confuse readers about the retrieval setup.\n\nQUESTIONS:\nI wrote them above\n\nLIMITATIONS:\nI wrote them above", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a new benchmark called COCO-Facet which focuses on fine-grained attributes for text-image retrieval tasks. The retrieval task aims at difficult categories such as gestures, times, material, and counts, among others, revealing that current VLM embedding models struggle with identifying these attributes reliably. Finally, the paper proposes to use MLLM embedding models with promptable embeddings to improve the retrieval performance significantly and offers insights about accelerating the inference speed of these large models.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n- The proposed COCO-Facet dataset is well motivated, tackles an important problem of fine-grained attribute retrieval, and facilitates further development of stronger embedding models.\n- The quantitative evaluation is extensive, covering a wide range of VLMs, both native embedding models and MLLM adapted embedding models.\n- Employing promptable models is a reasonable step towards solving this task. The paper also addresses the inference speed issue by providing two possible solutions with their own trade-offs: pre-processing embeddings and linear approximation.\n\n### Weaknesses\n- **Missing comparison to existing benchmarks**\n  - The paper does not justify the introduction of this new dataset with respect to existing ones. For instance, MMVP [1] contains similar categories such as counts, color/appearance, structural characteristics, etc. A discussion around the new aspects this dataset introduces compared to existing benchmarks is important for judging its value.\n  - Comparing the proposed method improvements on related datasets could further reveal if the behavior of the improvements is similar or whether one dataset is more challenging than the other.\n- **Methods should be evaluated zero-shot**\n  - The proposed method make a lot of assumptions about prior knowledge about the inference task which is often not the case in real-world scenarios. For instance, the pre-processed embeddings contain the GT best-case prompt as part of the prompt set simplifying the problem significantly. Moreover, it seems the category of the retrieval query needs to be known at test time for the promptable embedding solutions.\n  - A more realistic setting would either test the pre-processed embeddings from COCO-Facet on another dataset, or the other way around, possibly even creating a uninformed prompt set independently.\n- **Technical contributions are limited**\n  - Since the main contribution is the dataset, I see this as a minor weakness. Nonetheless, promptable embedding are a key feature of the MLLM embedding models, although Sec. 4 makes it sounds like something innovative.\n  - Sec. 5.2 is an interesting approach, but lacks in experimental details and clarity. It seems the approach requires a base dataset of K samples per category but it is not explained how this set is chosen and how one would know the category of the inference sample. Related to the previous point, in realistic applications, training and test set might not come from the same distribution.\n\n[1] Tong et al., Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs, CVPR 2024\n\nQUESTIONS:\n1. In Sec. 5.1, how does the performance change if you do not assume knowledge about the categories, i.e., the prompt set is disjoint from the GT categories.\n2. How do you choose the (a,b) pairs for Sec. 5.2? What happens if the categories for which K samples are available is disjoint from the inference categories, i.e., novel tasks/categories at test time?\n3. From the cited related work, it seems that FLAIR [2] is the closest to a promptable embedding model that is not based on an MLLM. Since it is concurrent work, I do not view it negatively that it is missing from the comparison, but it would certainly be interesting to see how it compares in Tab. 1 and/or Tab. 2. However, it does not seem to be available in a comparable model size and is a smaller model.\n\n[2] Xiao et al., FLAIR: VLM with Fine-grained Language-informed Image Representations, CVPR 2025\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces COCO-FACET, a benchmark for evaluating text-to-image retrievers on attribute-focused queries, revealing limitations in current approaches. The authors demonstrate that both CLIP-like and MLLM-based retrievers struggle with fine-grained visual attributes, particularly with non-dominant elements. As a solution, they propose using promptable image embeddings through MLLM-based universal embedders and present two acceleration strategies to make this approach more practical.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n1. Well-constructed benchmark: The COCO-FACET dataset is methodically structured with 9,112 queries across 8 attribute types, providing a valuable resource for the community to evaluate attribute-focused retrieval capabilities.\n\n2. Comprehensive evaluation: The paper presents thorough benchmarking across 12 state-of-the-art retrieval models with varying architectures and scales, offering a clear picture of current limitations in the field.\n\n3. Effective acceleration strategies: The proposed acceleration methods demonstrate meaningful performance improvements.\n\n## Weaknesses\n\n1. Limited technical novelty: The core approach of using MLLMs to obtain multimodal representations has been explored in previous work. The paper's main contribution appears to be the benchmark itself rather than a novel technical solution to the identified problem.\n\n2. Efficiency concerns remain: While acceleration strategies are proposed, the fundamental challenge of computational cost in promptable image embedding approaches persists. The vanilla approach requires N×M forward passes, which remains impractical for large-scale retrieval systems.\n\n3. Conflicting motivations: There is a conflict between the stated goal of improving retrieval for uncommon attributes and the design of the pre-processing embeddings acceleration method, which requires predefined attribute types. This approach only works for a finite set of predetermined attributes, contradicting the motivation to handle diverse, unexpected attribute queries.\n\nQUESTIONS:\nThe underlying principle of the Linear Approximation acceleration method requires clarification. In line 299, the authors propose using Wa for retrieval, which appears to rely solely on image embeddings without a prompt. Then what is the fundamental distinction between W and standard CLIP encoders? More specifically, while W is learned to approximate the nonlinear transformation U(a,p) for a specific prompt p, it's counterintuitive that this linear operation can generalize to highlight specific attributes across the entire image pool for different queries without the prompt being present during retrieval. Further explanation of how this approximation preserves attribute-specific information would strengthen the paper.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}