Highlighting What Matters: Promptable Embeddings for Attribute-Focused Image Retrieval
Authors: Siting Li
Xiang Gao
Simon Shaolei Du
Venue: NeurIPS 2025 poster ()
Keywords: text-to-image retrieval
image embeddings
Primary area: applications
TL;DR: We build a benchmark on attribute-focused text-to-image retrieval and propose a pipeline of using promptable image embeddings for solving it, leading to performance gain.

Abstract: While an image is worth more than a thousand words, only a few provide crucial information for a given task and thus should be focused on. In light of this, ideal text-to-image (T2I) retrievers should prioritize specific visual attributes relevant to queries. To evaluate current retrievers on handling attribute-focused queries, we build COCO-Facet, a COCO-based benchmark with 9,112 queries about diverse attributes of interest. We find that CLIP-like retrievers, which are widely adopted due to their efficiency and zero-shot ability, have poor and imbalanced performance, possibly because their image embeddings focus on global semantics and subjects while leaving out other details. Notably, we reveal that even recent Multimodal Large Language Model (MLLM)-based, stronger retrievers with a larger output dimension struggle with this limitation. Hence, we hypothesize that retrieving with *general* image embeddings is suboptimal for performing such queries. As a solution, we propose to use *promptable* image embeddings enabled by these multimodal retrievers, which boost performance by highlighting required attributes. Our pipeline for deriving such embeddings generalizes across query types, image pools, and base retriever architectures. To enhance real-world applicability, we offer two acceleration strategies: Pre-processing promptable embeddings and using linear approximations. We show that the former yields a 15\% improvement in Recall@5 when prompts are predefined, while the latter achieves an 8\% improvement when prompts are only available during inference.

Highlighting What Matters: Promptable Embeddings for Attribute-Focused Image Retrieval
Siting Li
Xiang Gao III, Tsinghua University
University of Washington sitingli@cs washington. edu
x- x-gao220mails htrp2200mmmlssssssnntstaee edu.
Simon Shaolei Du University of Washington
ssduÊcs. washington, edu
Abstract
While an image is worth more than a thousand words, only a few provide crucial information for given task and thus should be focused on. In light of this, ideal text-to-image (T2I) retrievers should prioritize specific visual attributes relevant to queries. To evaluate current retrievers on handling attribute-focused queries, we build COCO-FACEE
a COCO-based benchmark with 9,112 queries about diverse attributes of interest. We find that CLIP-like retrievers, which are widely adopted due to their efficiency and zero-shot ability, have poor and imbalanced performance, possibly because their image embeddings focus on global semantics and subjects while leaving out other details. Notably, we reveal that even recent Multimodal Large Language Model (MILLM)-based, stronger retrievers with larger output dimension struggle with this limitation. Hence, we hypothesize that retrieving with general image embeddings suboptimal for performing such queries. As a solution, we propose to use promptable image embeddings enabled by these multimodal retrievers, which boost performance by highlighting required attributes. Our pipeline for deriving such embeddings generalizes across query types, image pools, and base retriever architectures. To enhance real-world applicability, we offer acceleration strategies: Pre- processscs promptable embeddings and using linear approximations. We show that the former yields a 15% improvement in Recall @ 5 when prompts are predefined, while the latter achieves 8% improvement when prompts are only available during inference.
Introduction
Images offer valuable information that facilitates problem-solving and reasoning [Zelers et al. 2019
Lu et al. 2022 2023). Although there may be abundant information in one image, especially when
depicts a complex scene with plenty of elements [Gabbay et al., 2021 Urbanek et al. 2024 Nguyen et al. 2024], usually only small part is critical to task query at time. For both performance and efficiency considerations, there has been recent effort to focus the models on key aspects of images for better visual reasoning [Wu and Xie 2023 Hu et al., 2024b OpenAl 2025]. Similarly, for text-to-image (T2I) retrieval that helps knowledge and fact checking Yasunaga et al.) 2022 Sharifymoghaddam
al. 2024). an ideal retriever should also be able select images with given attributes of interest, such as specified time, location, or object, which are not necessarily the main elements of the image.
httpp://ggthuu..oo//11s///////0 Facet
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Attibbut--oousse Image Retrieval
COCO-Facet Benchmark
9,112
Query: Find me an image that contains any car.
Captions
Answers:
Count of People
Objects
Scenes
Animals
Visual7W VisDial
Ground Truth
VLM2Vec w/o Prompt X
CLIP x
Materials
Times
Weathers
Gestures
Cosine Similarity
CLIP
w/o Prompt
w/ Prompt
100
Recall@5 (%)
MLLM-Based Universal Embedder (VLM2Vec)
80
Which objects
40
Prompt
are present in this image?
Query
29
Pre-Processed Embeddings
Linear Approximation
Figure Overview. (Above) We study the task of attribute-ocuseed text-to-image retrieval and build COCO-FACET for benchmarking various retrievers. (Below) We show that using promptable image embeddings enhances performance on such queries, and propose two acceleration strategies to improve its applicability.
Are current T21 retrievers capable of accomplishing attribute-focusel queries? We explore this question on various CLIP-like models, from CLIP [Radford et al.) 2021] to recent SigLIP2 [Tschan- nen al. 2025].
well Multimodal Large Language Model (MLLM)-based embedders like VLM2 Vec Jiang et al. [024b). Since commonly used T21 benchmarks like MSCOCO (Lin et al. 2014 and Flickr30K Young et al. 2014) only contain general queries on global alignment, new benchmark, COCO- FACET with 9,1 12 setributefoocused queries of 8 types, based on existing annotations of COCO images [in et al. 2014 Zhu al. 2016 Caesar al. 2018 Das al. 2017]. We find that current retrievers behave worse attributes other than Animals compared with general T2I retrieval on MSCOCO, and struggle more with detailed or underexplored attributes such as Time. Additionally, we discover that they fail to prioritize images with the correct-but wet---bttoo--omiine attribute over images with wrong-but-dee but-omnnnnt attribute (See an example in Figure [].
build
As embedders with various architectures and scales all fall short on such queries, we hypothesize that retrieving with general image embeddings is inefficient and suboptimal for this task. Therefore, we propose to retrieve with promptable image embeddings: We employ MLLM-based universal embedders that can process combination of images and text, and show that using the GPT-written prompt for each category as text helps to highlight key attributes in image embeddings, demonstrating improvement on harder attributes while maintaining good performance on easier ones (See the plot in Figure TT. Additionally, we design two strategies to accelerate this pipeline: Predefining potentially useful prompts and pre-pree pre-processing the promptable image embeddings, deriving linear approximation of the embedder at test time, which can be efficiently applied to the query vectors.
Our main contributions listed as follows:
Introduced in Section 3 our benchmark COCO-FACES attribute-focused T21 queries is good supplement to the current general-purposed T21 retrieval evaluation. We also provide construction pipelines, which can be utilized for building benchmarks focusing on other attributes for future research.
Section 3.2 we reveal the limitation of current CLIP-like retrievers and MLLM-based embedders on cerribute-fccuser T2I queries, which affects models with various scales, image resolutions, and output dimensions.
In Section we propose to use promptable image embedding enabled by MLLM- based universal embedders as the solution. We show that it enhances the retrieval performance and generalizes over query types, image pools, and base retriever architectures.
Section we develop two acceleration strategies for real-world usage. Our results demonstrate that the preppsscssingg technique increases Recall@ by 15% when prompts predefined, while
5

---

the linear approximation achieves an 8% improvement on Recall @ when prompts are only available at inference time.
2 Related Work
Text-to-image Retrieval has been long- research direction not only due to the real-world
need for image search, but also as an important step in general problem-solving Yasunaga et al.
2022 Hu et al., 2024a]. CLIP-like dual-encoder approaches [Radford et al. 2021 Zhai et al. 2023
Cherti et al. 2023 Li et al. 2023b] are widely- used due to their efficiency and remarkable zero-shot
performance standard T2I benchmarks like MSCOCO Lin et al. [014] and Flickr30K Young
ict al. 2014). Recent MLLM- based retrievers (Lin et al., 2024| Lan et al.]5 Zhou et al. 2024a
Jiang al. 2024b] extend the input modality to joint image-text pairs. Since these models output
image and text embeddings for retrieval, there have been benchmarks for more comprehensive
evaluation of embeddings (Wei et al.) [2024 Jiang et al. 2024b Xiao al. 2025) which include
domain- focused retrieval subsets like FashionlQ eW al.
2021J. EDIS [Liu et [2023] OVEN (Hu
al. 2023). Wiki-SS-NQ [Ma et al.) [024]. We note that these subsets either focus on the global semantics of the images or pre-select images with a single main subject, though in-the-wild images
can be visually crowded with many attributes. Visual Genome [Krishna al. 2017 considers such complexity by annotating the images with detailed descriptions for advanced visual understanding. Aoti-Atts [Patterson and Hays, 2016) is built with attribute annotations for multi-label classification but only targets main subjects like people, animals, and objects.
On the other hand, previous studies on visual grounding and reasoning have pointed out that CLIPP-lke models [Paiss et al.] 2023) Chen et al.) 2024a Tong et al. 2024) might neglect visual details. Sogi et al. 2024] shows that the retrieval performance degrades when the target objects are small. To overcome this limitation, researchers propose to use two-stage approaches [Miech et al. 2021 Geigle
al., 2022] or guidance from an LLM [Lee et al., [2024] Notably, pre-CLIP methods represented by scene graphs [Johnson et al. 2015 Schuster et al. Phamm et al., [024] might capture more visual details, but they lag behind terms of data availability and inference efficiency.
Another improvement based on CLIP-like models is to obtain task-specific image embeddings, similar to task-specific text embeddings (Su et al. 2022 Asai et al., 2022]. CLOC [Chen et al. 2024a proposes the new learning goal of promptable embeddings for better localization given spatial hints. GiVE [Li et al. [0224] and FLAIR (Xiao al. 2024 design patch-level or token-level image- text interaction mechanisms for language-informed image embeddings. Universal embedders [Wei) al. 2024 Zhang al., 2024 Zhou et al., 2024a b) leverage promptable query embeddings for the composed image retrieval task. For the retrieval targets, VLM2Vec [Jiang et al. 2024b) and E5-V (Jiang et al. 2024a] try using different texts as prompts when deriving image embeddings for some domains (e.g., news, fashion). .M--Ebee [in et al. 2024] recognizes the benefit of prompts but requires fine-tuning with domain-specific instruction. Promptable embeddings are also applied to other areas like reinforcement learning [Chen et al.) 2024b). We systematically study the promptable image embeddings for retricval targets and propose acceleration strategies.
Localized Vision-Language Models (VLMs) are motivated by similar ideas that some part of an image is of interest for given task. V* (Wu and Xie| [023) formulates the problem iterative visual search on a high-resolution image. Kosmos-2 [Peng et al., 2023) and GLIPv2 (Zhang et al.) 2022] consider language grounding by bounding boxes phrases. DAM (Lian et al. 2025 explores captioning for a given region. [Lin et al. (223) applies cropping to regions of interest, and Wang et al. similar task called Partial Scene Text Retrieval. The visual intelligence of OpenAl's o3 and o4-mini models is incorporated with simple tools like zooming and cropping to process the images for better reasoning [OpenAl 2025). , but their approach region-based, while our attributes of interest could be non-region-basee (c.g., Scene and Time).
2024 aims
3 Benchmarking T2I Retrievers on Attribute-Focusel Queries
We focus on attribute-focused T2I queries in this work. In standard, general-purpose T2I benchmarks like MSCOCO [in et al. 2014), queries are image captions that describe the main content of the image (e.g., "A black Honda motorcycle parked front of a garage.") ") but omitting other attributes like Weather, especially when they are non-dominant attributes. Hence, they cannot
3

---

Table I: Model details of T2I retrievers and their average Recall @ and Recall@5 (in percentage points) on the MSCOCO 2017 validation set and COCO-FACET. Recent MIL.M-based universal embedders (shown in the second section) outperform CLIP-like models, but all T21 retrievers exhibit performance drop on sttribue--occused queries.
Retriever
Img. Size Params (M) Output Dim.
COCO
COCO-FACET
Recall
Recall Recall @ Recall
CLIP-ViT-L/14
336
427.9
768
81.0
97.9
33.7
47.0
EVA01 ViT-g-14 14
2242
1136.4
1024
83.2
98.4
35.4
48.3
EVA02 ViT-bigE-144 14+
2242
5044.9
1024
87.9
99.2
34.2
48.8
SigLIP ViT-SO-14
3842
878.0
1152
73.2
95.5
37.8
51.9
SigLIP2 ViT-SO-14
3842
1136.0
1152
87.4
98.4
39.8
52.6
BLIP2-COCO
224²
1173.2
768
88.8
99.1
37.8
51.6
MagicLens
2242
427.6
768
87.2
99.2
40.6
56.0
E5-V
336
8355.3
4096
89.6
99.3
46.0
61.8
MM-Embed
336
8175.5
4096
93.2
99.7
42.8
58.4
MMRet-MLLMSS2
336
7566.3
4096
93.7
99.7
48.8
64.5
LLaVE-2B
3362
1945.2
1536
92.4
99.7
45.6
59.5
VLM2Vec-Phi-22
336
4146.6
3072
89.4
99.5
44.5
58.9
be used for purpose directly. A recent benchmark MMVP-VLM [Tong al. 2024
contains 270
fine-gras gine-graindd testcases but only has a small image pool (two images per query) designed for image-text matching instead of T2I retrieval.
Therefore, we construct new benchmark, COCO-FACEE for evaluating T21 retrievers on attribute-
focused queries in Section 3.1 and analyze their performance in Section 3.2 We reveal that current retrievers fall short of such queries, even though they involve less image-text matching than long MSCOCO-style captions with multiple attributes.
3.1 Benchmark Construction
We utilize the existing annotations provided by MSCOCO [Lin al. 2014), Visual7W [Zhu et al. 2016], VisDial [Das et al.) 2017], and COCO-Stuff [Caesar et al. 2018 about COCO images. In total, we collect 9,112 test cases covering eight types: Objects, Animals, and Gestures of people in the image based on MSCOCO's annotations segmentation, Scenes and Times of the day shown in the image based on Visual7W's s where- and when when-question answering. Count of People in the image based on Visual7W's how-many-ppen peoay---pole-quustton answering. Weathers based on MSCOCO, Visual7W, and VisDial annotations, AMaterials of objects or surfaces shown in the images based on COCO-Stuff. While Objects, Animals, and Scenes are
extensively studied in areas like image classification, others less-explored. Additionally, consider both regional attributes Animals, Objects, Gesturess Materials)) and global attributes Scenes, z Count of People, Times, Weathers) which require inference based the whole picture, so simple strategies like cropping zooming may not be effective.
Each test case contains a text query (e.g., "Find me an everyday image that shows the scene of the beach. "), a positive candidate (ground truth) that contains the attribute required in query (the scene the beach), and negative candidates ensure the quality of the negatives, randomly select from images that have exclusive attributes to avoid ambiguity (e.g., images that show the scene conference room). More benchmark details and generation procedure Appendix D We use the validation set MSCOCO 2017 for comparison after converting it to the same format ("Find me an everyday image that matches the given caption. "+COCO caption as the query text, and 100 candidate images).
do not contain such attribute. To
deferred to Appendix A and examples of each category shown in
3.2 Benchmark Results
We evaluate 12 state-of-the-art T21 retrievers and present the results in Table and Table 2 The first
seven rows feature the CLIP family (CLIP [Radford 202T). EVA-CLIP [Fang et al.) 2023 2024
SigLIP (Zhai et al. 2023
LigLIP2 [Tchannen et al. 2025 and BLIP2 finetuned COCO Li ct al.

---

Table 2: Recall and 5 (in percentage points) for various text-to-image retrievers by category on our COCO-FACET benchmark Animals, 1. Scenes, Objects, S Count of People, A Materials, O Times, Weathers, $: Gestures). Cells shaded in red indicate low category-specifie performance (Recall 10% or R 20%). All models struggle more on the last five attributes.
Img.
Size
Recall @
CLIP-ViT-L/4
3362
91.5
55.2
54.0
3.5
3.5
4.5
4.2
6.8
EVA01 ViT-g-14
224
93.7
56.4
58.1
2.3
2.8
4.6
2.9
7.0
EVA02 ViT-bigE-I4+
224
88.1
55.8
55.9
3.3
3.1
4.6
3.2
7.1
SigLIP ViT-SO-14
3842
92.4
57.0
63.1
4.4
3.2
4.7
3,8
7.5
SigLIP2 ViT-SO-14
3842
94.9
54.1
66.2
4.9
3.4
3.6
4.8
11.2
BLIP2-COCO
224
87.0
62.2
61.3
4.9
5,0
4.6
3.5
14.8
MagicLens
224
94.7
70.9
63.2
15.9
6.2
10.0
3.2
14.3
ES-V
336
92.7
70.4
71.2
31.4
10.5
7.5
5.2
20.1
MM-Embed
336
92.7
67.4
68.1
13.3
7.4
5.1
3.9
19.5
MMReI-MLLM-S2
3362
97.2
72.1
76.0
29.8
10.0
8.4
3.6
24.1
LLaVE-2B
336
96.3
70.9
73.1
19.4
8.8
3.0
4.5
19.2
VLM2Vec-Phi--.5>V
336
95.5
69.8
74.4
14.4
6.3
5.5
4.3
9.8
Recall
5
CLIP-ViT-L/14
3362
98.4
80.8
72.7
13.5
11.4
10.1
14.3
18.5
EVA01 ViT-g-14
224
99.0
84.9
75.6
12.1
9.6
12.4
12.8
19.1
EVA02 ViT-bigE-14+
224
98.7
83.7
75.5
15.9
10.8
12.6
14.6
19.9
SigLIP ViT-SO-14
3842
99.5
81.4
80.3
13.3
14.5
21.5
13.1
20.8
SigLIP2 ViT-SO-14
3842
99.3
79.1
82.2
18.7
12.3
12.4
13.5
25.2
BLIP2-COCO
224
97.8
85.5
75.9
19.3
15.3
15.7
14.6
33.1
MagicLens
2242
99.3
89.5
81.3
35.7
15.7
25.1
16.3
32.1
E5-V
336
98.3
91.9
89.0
60.1
25.2
18.6
16.1
35.5
MM-Embed
336
98.7
87.8
84.9
29.3
20.3
15.0
15.3
45.3
MMRRI-MMLMMS22
336²
99.9
92.4
91.6
45.2
28.3
19.7
17.6
49.7
LLaVE-2B
3362
99.4
93.6
88.2
41.3
21.5
10.9
16.3
37.0
VLM2Vec-Phi-3.55
336
99.3
90.7
90.7
36.6
18.2
12.9
19.2
27.1
2023b]) and Maniceeen [hhang et al. 2024), which use unimodal encoders. Although MagicLens accepts image+text the input through fusion module, we follow their T21 retrieval protocol and use finetuned text and image encoders without other modules. As CLIPP sensitive to text format, also conduct evaluation using standard CLIP- texts ("a photo ... "") for in Appendix B.2 but find no significant difference. The last five rows are recent MLLM- based universal embedders that accept combinations of images and texts as input naturally: E5-V [Jian et al.) 2024a , MM-Embed [Lin et al,, 2024), MMRet [Zhou et al. 2024a]. LLaVE [Lan et al. 2025 and VLM2Vec (Jiang et al. 2024b). Since targets image-only, texts (e.g., "Represent the given image." for VLM2Vec) accompanied with the images for encoding. More details about these models in Appendix
the default, general
Current retrievers struggle on our benchmark. Compared with Recall original MSCOCO, the average performance on our benchmark degrades for all retrievers across different architectures and scales. the first section, the performance among the CLIP family are close, with the most recent model, SigLIP2, scoring the highest COCO-FACET. MagicLens has the best overall performance despite its small scale, output dimension, and image resolution. In the second section, universal multimodal embedders have better performance than unimodal embedders, with MMRet-MLLMM in a relatively large scale outperforming others. Still, the performance differences are small among universal embedders, with clear gap between COCO and COCO-FACER results.
Recall on
Retrievers have imbalanced performance different attributes. Again compared with COCO, models have lower performance on attributes apart from Animals, and significantly lower on the last five attributes in Table indicating that they are largely neglected. Possible causes include
5

---

Table 3: @ecall@) I and Recall@5 for text-to-image retrieval (in percentage points) on our COCO- FACET benchmark. Promptable image embeddings yield substantial average improvements and outperform the baselines on seven out of eight attributes.
Avg.
Recall
CLIP-VIT-LI1-3336p
90.9
55.2
53.1
3.5
3.5
4.5
4.2
6.8
33.7
VLM2Vec-Phi-3..5
95.5
69.8
74.4
14.4
6.3
5.5
4.3
9.8
44.5
w/ GPT prompt
90.7
81.4
75.5
72.7
25.8
18.4
14.4
15.7
53,4
Text-Based
69.9
66.9
62.6
35.6
13.5
13.2
7.1
10.7
40.5
Recall @
CLIP-VIT-L/14-3366x
97.9
80.8
72.2
13.5
11.4
10.1
14.3
18.5
47.0
VL.M2Vec-Phi-3.5-.
99.3
90.7
90.7
36.6
18.2
12.9
19.2
27.1
58.9
w/ GPT prompt
98.7
95.9
92.0
92.1
48.8
82.4
36.5
39.3
75.5
Text-Based
90.7
86.6
81.7
60.6
37.2
39.2
24.4
26.1
60.6
reporting bias (Kamath et al.] "people murder" is more likely to appear than "people breathe" in the corpora, as in our case the Time (e.g.. morning or night) might be too obvious report-in both training data and previous evaluation. Mcanwhile, this verifies the findings of prior work on visual grounding and reasoning [Paiss al., [2023 Chen al. 2024a Tong al.) [024] in the setting of T2I retrieval, that CLIP-like models only achieve global text-image alignment, and thus their image embeddings focus on global semantics or subjects while leaving out attributes like object details and quantity.
Pominant-but-wee but--routwwoogggg attributes may be favored correct-but- but---ttooo----annt ones. We look into the failure cases of CLIP- ViT-L/14 and VLM2Vec as representative retrievers. As shown in train) Figure [ they seem rank higher the images with similar attribute (a motorcycle as the main content but not the one with the correct attribute as non-dominant elements (a car in the background). More examples can be found in Appendix D This implies that "simple" images consisting of fewer, more salient attributes may be preferred in retrieval rather than "complex" images the cost of precision.
4 Promptable Image Embeddings
When compressing images into embeddings of limited length, some visual information may be discarded, leading to low performance on relevant queries. Hence, natural approach to address this issue is (1) learning denser visual representation during pretraining, or (2) using embeddings with larger dimensions. However, recent SigLIP2 [Tschannen et al.) 2025) trained with a global-local loss for improving fine- ane-igained local semantics only slightly outperforms other CLIP-like models, and using -imendime large embeddings (like MMRet-MLLM-S9 still fails to resolve this issue, with Recall @ I lower than 10% and Recall @5 lower than 20% for some categories. Based these findings, we hypothesize that general image embeddings might be incfficient for antribute-speciffe queries.
We therefore focus on highlighting the important part in the embeddings. For this purpose, we propose to use promptable image embeddings for different textual prompts, which is enabled by recent MLLM-bases universal embedders. Although they are mainly motivated by tasks involving combinations of images and texts as queries targets (e.g., composed image retrieval), previous research (Tiang et al., 2024a finds that they can (1) deal with unseen prompts since they are based pre-trained MLLMs, and (2) accept task-specific prompts accompanied with images, such as modification in the FashionIQ dataset [Wu et al.) 2021). In Section 4.1 we formally study the promptable image embeddings and demonstrate that our designed pipeline yields a performance boost for attribute-focuse queries. Moreover, we show that it surpasses text-based T2I retrieval (Section 4.2)
conditioning image embeddings on
4.1 Method
We employ VL.M [Jiang et al. 2024b) as the base retriever for deriving promptable image embeddings in this subsection. This model is built on Phi-3.5-V [Abdin et al. 2024), which

---

80
90
Without GPT Prompt
Without GPT Prompt
With GPT Prompt
With GPT Prompt
70
00
75.5%
Recall@11 (%)
Recall@s (%6
73.0%
53.495
54.
56.1%
57.6%
61.8%
46.0%
45.6%
48.8%
60
58.
996
59.5%
40
50
30
40
MMRet-
LLaVE MLLM-SZ
Figure 2: Average retrieval performance across various base retrievers on COCO-FACET, with and without GPT-gennrated prompts. The same set of prompts brings consistent performance gain on different multimodal base retrievers.
has strong capabilities in image understanding. VeM2 generates embeddings by taking the last layer vector representation of the last token as the embedding and is fine-tuned with contrastive loss. Hence, when given a piece of text and an image, it produces a single combined embedding.
The authors previously tried using questiontimage as the input for embedding queries for the VQA tasks, while the targets are the ground truth, text-only answers. This motivates us to use questions as prompts for target images to ask about the required attributes in our case. Ideally, the resulting embeddings would contain "answers" such constructed visual questions, which is the corresponding attribute of the image. To make the pipeline automatic and extendable, we use GPT-4o [Hurst et al. 2024] to generate questions with the following template:
Write question to ask about the (Attribute Name} in image, with possible answers such as {A}, {B}, and so on. Please answer in one sentence without mentioning any answer
For example, the output for input tuple (people gesture, standing, jumping) "What gesture are the people making in this image?" Then we concatenate this question with the fixed template of the base retriever ("<limage_11 1 Represent the given image with the following question:") ") to construct the full prompt used for model input. We show that the promptable image embeddings improve the performance on the five more challenging categories, by highlighting the corresponding attributes (See T). In addition, we test human-written prompts and observe similar performance gain, and attach all prompts in Appendix B.3 We further evaluate recent non-MLLM-basse retriever with ianguage-informer image representation for highlighting relevant patches but find little improve- ment (FLAIR [Xiao et al. 2024 in Appendix C.2) This underlines the advantage of inheriting the image-text alignment in pretrained MLLMs with a flexible image focus.
VL.M2Vec-Phi-3.5-V. especially
Furthermore, the same set of prompts generalizes to other universal multimodal embedders used as base retrievers (ES-V, V. LLaVE, MMRet- MLLM-S2) (See Fig- ure D We also apply them to MM-Embed, but this model lacks the zero-shot ability on prompting and re- quires fine-tuning as mentioned in their paper. See detailed numerical results by category in Appendix [
Table 4: Recall and Recall@5 of T2I retrieval for the converted Place365 and SUN397, comparing promptable image embeddings with original image embed- dings. The improvement indicates that our pipeline generalizes effectively to these im- age pools.
Additionally, the method works on different image pools. We converted two scene classification bench- marks, Place365 [Zhou et al. 2017] and SUN397 [Xiao et al. 2010) provided in MMEB with 1,000 test cases each [Jiang et al., 2024b). J into text-to-image retrieval benchmarks in the same format, except that we use one positive candidate and 499 negative candidates for each test case. We notice that the same prompt used for Useene increases both Recall@ and Recall @ per- formance in Table
Place365 SUN397
Recall@1
VLMM2Vec-Phi-335-
30.0
57.0
w/ GPT prompt
33,9
61.9
Recall
VLM2Vec-Phi-3.5-v
58.2
85.2
w/ GPT prompt
66.7
88.9

---

4.2 Comparison with Text-Based T2I Retrieval
As we mention Section 4.1 the promptable image embeddings are likely to contain the correspond- ing attributes of the image as "answers" to the prompt. Another similar approach is to directly ask an MLLM the same question about the image and to obtain a pure-text answer. Then, we can use the text embedding of this answer as the target embedding, which is explored in previous work [Karthik] ct al. 2023). The pure-text answer is also supposed to be a dense representation of the image which contains required attributes in the prompt.
Is this text-based approach equivalent to our method? We conduct such comparison using LLaVA- 1.5 [Liu et al. [024]] as the MLLM and GRIT [Muennighoti et 2024] as the text embedder. As shown in the last row in Table 3 this text-based approach underperforms our method, and even loses to VLM22Vec-Phi-3.5
without prompt in Recall@1, 1. When checking its failure cases, we find that it suffers lot from hallucination: For instance, when asking LLa about animals' existence in an image with only feathers in container, it wrongly answers with "there are birds visible." " Besides, it cannot process the linguistic ambiguity in a pure-text answer (c.g.. "chicken" can refer to domestic animal or a type of meat). Some examples are attached in Appendix C.5 This indicates that the promptable image embedding provides more than an embedding of the pure-text answer the given visual question in the prompt.
Acceleration
While effective, the pipeline has high computational cost in real-world T2I retrieval. For the experi- ments in Section | we assume that the query type is known during the pre pre-procsssing stage for computing the promptable image embeddings. However, the query type is only known at test time, we need to take the per-query computational cost into consideration. Assume that we have N images in the pool and M text queries. Since N is typieally large, the ideal per-query cost should not grow linearly with N.
We focus primarily on embedding cost, as the maximum cosine-similarig searching step can be efficiently handled by the FAISS library [Douze et al. 20244). Let v denote the cost of computing a single image embedding, and the cost for single text embedding, using CLIP-like models. Let F represent the cost of a single forward pass through the base model of our multimodal embedder (e.g., Phi-3.5-V with CLIP vision encoder). For CLIP-like like models, the total embedding cost is No + Mt, with per-query embedding cost of t. If we stick to the original pipeline for promptable image embeddings, the total embedding cost will be No + M(NF F), leading to a per-query cost of NF + F.
To reduce the per-query cost, we explore two strategies VLM2Vec-Phi-335- in this section. (1) The first approach is straightforward: We predefine potentially useful prompts and pre-process the promptable image embeddings. (2) The second solution is to derive linear approximation of the retriever at test time. Both strategies have lower per-query embedding cost and outperform the baseline VLM2Vec- Phi-c..-h---VVV on Recall @5 of COCO-FACET.
5.1 Pro-Proces Embeddings
Since many attributes of interest can be predicted beforehand with prior knowledge of the incoming tasks, we can predefine some prompts at the pre-processing stage using our pipeline. During inference, we only need select the most suitable prompt and retrieve from the corresponding promptable image embeddings.
We test this strategy the COCO-FACET benchmark using the prompt set obtained in Section 4.1 We use GPT-4o for prompt selection at test time with the template attached in Appendix [B.4 The ground-truth prompt for each query can be selected with high accuracy low selection accuracy for the Gesture category is due to the similar prompt in Countt of People category, but find that such similar prompt other than the ground truth could also lead to correct answers, indicating some degree of error tolerance. (See Table 5))
average at test time. The
The per-query embedding cost of this strategy is F (for embedding the query text), and there is an additional cost for calling the GPT-4o API. The NF term in the embedding cost is replaced by a higher memory cost and pro-prrces time cost.
8

---

Table 5: Recall 1 and Recall @5 (in percentage points) of accelerated text-to-image retrieval with pre-processed promptable image embeddings on the COCO-FACET benchmark. The ground-truth prompt can be selected with a high accuracy for most categories.
& Avg.
Recall @
VLM2Vec-Phi-33--V
95.5
69.8
74.4
14.4
6.3
5.5
4.3
9.8
44.5
w/ selected GPT prompt
90.9
81.4
75.5
72.7
25.8
18.4
14.4
10.5
52.8
w/ gt GPT prompt
90.7
81.4
75.5
72.7
25.8
18.4
14.4
15.7
53.4
Recall@s
99.3
90.7
90.7
36.6
18.2
12.9
19.2
27.1
58.9
w/ selected GPT prompt
99.1
95.9
92.0
92.1
48.8
82.4
36.5
25.3
73.7
w/ gt GPT prompt
98.7
95.9
92.0
92.1
48.8
82.4
36.5
39.3
75.5
Selection Acc.
100
86.6
99.9
100
100
100
100
8.9
87.9
Table Recall@ and @ (in percentage points) of accelerated text-to-image retrieval with approximated promptable image embeddings (K 100) the COCO-FACET benchmark. Results are averaged over five independent runs.
x
Avg.
Recall
VLM2Vec-Phi-3.55
95.5
69.8
74.4
14.4
6.3
5.5
4.3
9.8
44.5
w/ linear aрprox.
72.1
67.2
57.1
47.5
24.3
35.7
9.0
14.2
42.5
w/ GPT prompt
90.7
81,4
75.5
72.7
25.8
18.4
14.4
15.7
53.4
Recall@5
VLM2Vec-Phi-3.5X1
99.3
90.7
90.7
36.6
18.2
12.9
19.2
27.1
58.9
w/ linear aррrox.
84.6
91.9
83.2
73.7
43.9
71.5
28.4
38.1
67.0
w/ GPT prompt
98.7
95.9
92.0
92.1
48.8
82,4
36.5
39.3
75.5
5.2 Linear Approximation at Test Time
When there are novel attributes required in the query, can we process with lower cost? We can first use the previous automatic pipeline to get prompt Assuming that the model has access to images in the same category as the query, we experiment with test-time linear approximation of the universal embedde Specifically, we denote the normalized original image embeddings without a prompt as a, and the normalized promptable image embeddings as b. Let U be the multimodal embedder and q be the normalized query embedding. We would like find matrix W with respect to such that
Wa U(a.p) =
for all a in the image pool. After deriving W, we can use Wa for retrieval, searching for the such that the dot product between Wa and q maximized.
We find this matrix W based on a small amount of (a, b) pairs. During pre-processing, we store for all images in the pool. At test time, after obtaining p. we uniformly sample K images from the candidate pools of all queries in the same category along with their and calculate U Uaa p) = b. denoting by matrices A and B. The best linear approximation then given by W BAT W can be applied the query ạ instead, since (Wa a) q a (W" q). The per-query embedding cost is thus KF + F, while the pre- pre-processing cost remains unchanged. It is worth noting that theoretically, W should be orthogonal to ensure that ||W a|l2 1 as ||b|/2 1, but in practice we find that the W directly derived from BAA and normalized after being applied to works well. Thus, strict orthogonality is not required.
We test the method on VLM2Vec- Phi-c.--i- with K -100 for each category. The results are shown in T with the error bar listed in Appendix C.3 Although the linear approximation of a MLLM- based embedder has limited expressiveness and cannot capture the nonlinear, complex
We include experiments that ablate this assumption in Appendix E

---

Animals
Scenes
Objects
Count of People
60-
70
Recall Score (86
60-
Recall Scoree (%6
Recall Score (%6
-6
Recall Scote
40
Recall
60
40
50
10
femnhll
20
10
10.
100
100
40
100
20
40
100
Sample Size
Sample Size
Sample Size
Sample Size
Materlals
Times
Weathers
Gestures
70
Bece
Recal Score (9)
Recall Scooe
50
Recall Score
20
20
lacligs Dueilre Rece
Recal Score
25
-5
26
feseire Noce
Accagg
20
40
100
20
40
30
100
20
40 Sample Size
60
100
20
40
60
50
100
Sample Size
Sample Size
Sample Size
Figure 3: Recall 1 and Recall@5 of accelerated text-to-image retrieval using approximated prompt- able image embeddings with varying sample size K in percentage points on COCO-FACET. The results are averaged over five independent runs. "Baseline" refers to using VLM22ec-Phii3355 without prompts.
cross-modal interactions, it still improves the baseline on the harder attributes and yields a better @ecall@5,
We also test smaller K for cost-flexible deployment. When we vary K from 5 to 100, both metrics show overall improvement in Figure 3 allowing tuning K based on available compute and desired latency. On XMateriale als and Times, using very small sample size (K = 5) outperforms the baseline model without prompt. On other harder attributes, K 40 is sufficient for exceeding the baseline. However, it still challenging to match the baseline on the three more common attributes Animals, Scenes, and Objects), which is a limitation of current approximation approach. We defer the actual cost analysis to Appendix C.3 and visualization to Appendix C.4
Conclusion
We introduced COCO-FACET, a benchmark to evaluate text-to-image retrieval performance on sitribute-focused queries, revealing limitations in current CLiP-like and MLLM- retrievers. General-purposee image embeddings often overlook fine-grained visual attributes critical for accurate retrieval. To address this, we propose to use promptable image embeddings on MLLM-basee universal embedders, which improve focus relevant attributes and enhance retrieval quality while being flexible, model-agnostic. We explore efficient acceleration strategies that makes it more practical for deployment. Together, work offers promising direction for building more precise and efficient T21 retrievers that can be integrated into systems for general problem-solving.
Acknowledgments and Disclosure of Funding
SSD) acknowledges the support of NSF DMS 2134106, NSF IIS 2143493, NSF IIS 2229881, Sloan Fellowship, and the AI2050 program at Schmidt Sciences.
References
M. Abdin, J. Aneja, Awadalla, A. Awadallah, A. A. Awan, N. Bach, A. Bahree, A. Bakhtiari, J. Bao, H. Behl, al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404. 14219, 2024.
A. Asai, T. Schick, P. Lewis, X. Chen, G. Izacard, S. Riedel, H. Hajishirzi, and W.-t. Yih. Task-aware retrieval with instructions. arXiv preprint 2.Xiv02211.09260, 2022.
10

---

H. Caesar, J. Uijlings, and V. Ferrari. Coco-stuff: Thing and stuff classes in context. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1209-1218, 2018.
H. Chen, L. Wang, N. Yang, Y. Zhu, Z. Zhao, F. Wei, and Z. Dou. mme5: Improving multimodal multilingual embeddings via high-quality synthetic data, arXiv preprint arXiv:2502.08466 2025.
H.-Y. Chen, Z. Lai, H. Zhang, X. Wang. M. Fichner, K. You, M. Cao,, B. Zhang. Y. Yang. and Z. Gan. Contrastive localized language-image pre-training. arXiv preprint arXiv:2410.02746, 2024a.
W. Chen, O. Mees, A. Kumar, and S. Levine. Vision-language models provide promptable represen- tations for reinforcement learning. arXiv preprini 24Mi::2402.02655/ 2024b.
M. Cherti, R. Beaumont, R. Wightman, M. Wortsman, G. Ilharco, C. Gordon, C. Schuhmann, L. Schmidt, and J. Jitsev. Reproducible scaling laws for contrastive language-image learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2818 2829, 2023.
A. Das, S. Kottur, K. Gupta, A. Singh, D. Yadav, J. M. Moura, D. Parikh, and D. Batra. Visual dialog. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 326 335, 2017.
M. Douze, A. Guzhva, C. Deng, J. Johnson, G. Szilvasy, P.-E. Mazaré, M. Lomeli, L. Hosseini, and H. Jégou. The faiss library. arXiv preprint arXiv:2401.0828, 2024.
Y. Fang, W. Wang, B. Xie, Q Sun, L. Wu, X. Wang. T. Huang, X. Wang, and Y. Cao. Eva: Exploring the limits of masked visual representation learning at scale. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 19358-19369, 2023.
Y. Fang. Q. Sun, Wang. T. Huang, X. Wang, and Cao. Eva-02: visual representation for neon genesis. Image and Vision Computing, 149:105171, 2024.
A. Gabbay, N. Cohen, and Y. Hoshen. An image is worth more than a thousand words: Towards disentanglement in the wild. Advances in Neural Information Processing Systems, 34:9216-9228, 2021.
G. Geigle, J. Pfeiffer, N. Reimers, I. Vulić, and I. Gurevych. Retrieve fast, rerank smart: Cooperative and joint approaches for improved cross-modal retrieval. Transactions of the Association for Computational Linguistics, 10:503-521, 2022.
H. Hu, Y. Luan, Y. Chen, U. Khandelwal, M. Joshi, K. Lee, K. Toutanova, and M.-W. Chang. Open-domain visual entity recognition: Towards recognizing millions of wikipedia entities. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12065-12075, 2023.
W. Hu, J.-C. Gu, Z.-Y. Dou, M. Fayyaz, P. Lu, K.-W. Chang, and N. Peng. Mrag-bench: Vision- centric evaluation for retrieval-augmenntr multimodal models. arXiv preprint arXiv:2410.08182, 2024a.
Y. Hu, W. Shi, X. Fu, D. Roth, M. Ostendorf, L. Zettlemoyer, N. A. Smith, and R. Krishna. Visual sketchpad: Sketching as a visual chain of thought for multimodal language models. arXiv preprint arXiv:2406.0944 09400, 2024b.
A. Hurst, A. Lerer, A. P. Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda, A. Hayes, A. Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.2 2024.
G. Ilharco, M. Wortsman, R. Wightman, C. Gordon, N. Carlini, R. Taori, A. Dave, V. Shankar, H. Namkoong, J. Miller, H. Hajishirzi, A. Farhadi, and L. Schmidt. Openclip, July 2021. URL https://doi. orgo 10. 5281/zenodo. 5143773 If you use this software, please cite it as below.
T. Jiang, M. Song, Z. Zhang, H. Huang, W. Deng, F. Sun, Q. Zhang, D. Wang, and F. Zhuang. E5-v: Universal embeddings with multimodal large language models. arXiv preprint arXiv:2407.12580 2024a.
7. Jiang, R. Meng, X. Yang, S. Yavuz, Zhou, and Chen. V1m2vee: Training vision- language models for massive multimodal embedding tasks. arXiv preprint arXiv:2410.05160, 2024b.
11

---

J. Johnson, R. Krishna, M. Stark, L.-J. D. Shamma, M. Bernstein, and L. Fei-Fei. Image retrieval using scene graphs. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2015.
A. Kamath, J. Hessel, K. Chandu, J. D. Hwang, K.-W. Chang, and R. Krishma. Scale can't 't overcome pragmatics: The impact of reporting bias on vision-language reasoning.
S. Karthik, K. Roth, M. Mancini, and Z. Akata. ssiion-y-language for training- compositional image retrieval. arXiv preprint arXiv:2310.0m 09291, 2023.
R. Krishna, Y. Zhu, O. Groth, J. Johnson, K. Hata, J. Kravitz, S. Chen, Y. Kalantidis, L.-J. Li, D. A. Shamma, al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123:32-73, 2017.
Z. Lan, L. Niu, F. Meng, J. Zhou, and Su. Llave: Large language and vision embedding models with hardness-weighter contrastive learning. arXiv preprint arXiv:2503.04881 2025.
S. Lee, S. Yu, J. Park, J. Yi, and S. Yoon. Interactive text-to-image retrieval with large language models: A plug-and-pla approach. arXiv preprint 1rXXv:2440603411,, 2024.
D. Li, J. Li, H. Le, G. Wang, S. Savarese, and S. C. Hoi. LAVIS: one-stop library for language- vision intelligence. In Proceedings of the 61st Annual Meeting the Association for Computational Linguistics Volume 3: System Demonstrations), pages 31-41, Toronto, Canada, July 2023a. Asso- ciation for Computational Linguistics. httpbitts//alllatoollooggoo org/
wwwp//tgg...///eee///e///// 2023 acl demo. 3
J. Li, D. Li, S. Savarese, and S. Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages -9742. PMLR, 2023b.
J. Li, J. Ma, X. Zhang, Y. Li, and J. Shi. Give: Guiding visual encoder to perceive overlooked information. arXiv preprint arXiv:2410.20100 2024.
L. Lian, Y. Ding, Ge. S. Liu, H. Mao, Li, M. Pavone, M.-Y. Liu, T. Darrell, A. Yala, et al. De- scribe anything: Detailed localized image and video captioning. arXiv preprint arXiv:2504.16072 2025.
S.-C. Lin, C. Lee, M. Shoeybi, J. Lin, B. Catanzaro, and W. Ping. Mm-embed: Universal multimodal retrieval with multimodal lims. arXiv preprint arXiv:2411.025711 2024.
T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft coco: Common objects in context. In Computer vision-ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part v 13, pages 740-755 Springer, 2014.
W. Lin, J. Chen, J. Mei, A. Coca, and B. Byrne Fine-grained late-interaction multi-modal retrieval for retrieval augmented visual question answering. Advances in Neural Information Processing Systems, 11:22800-22444 2023.
H. Liu, C. Li, L.,, and Y. Lee. Improved baselines with visual instruction tuning. In Proceedings of the JEEE/VVF Conference on Computer Vision and Pattern Recognition, pages 26296-26306, 2024.
S. Liu, W. Feng, T.-j. Fu, W. Chen, and W. Y. Wang. Edis: Entity-drived image search over multimodal web content. arXiv preprint 2335 13631, 2023.
Lu, S. Mishra, T. Xia, L. Qiu, K.-W. Chang. S.-C Zhu, O. Tafjord, P. Clark, and A. Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35:2507-2522 2022.
P. Lu, H. Bansal, T. Xia, J. Liu, C. Li, H. Hajishirzi, H. Cheng, K.-W. Chang, M. Galley, and J. Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.0223
2023.
X. Ma, S.-C. Lin, M. Li, W. Chen, and Lin. Unifying multimodal retrieval via document screenshot embedding, arXiv preprint arXiv:2406.11255 2024.
12

---

A. Miech, J.-B. Alayrae, I. Laptev, J. Sivic, and A. Zisserman. Thinking fast and slow: Efficient text- to-visual retrieval with transformers. Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 9826-982888 9836, 2021.
N. Muennighoff, S. Hongjin, L. Wang, N. Yang. F. Wei, T. Yu, A. Singh, and D. Kiela. Generative representational instruction tuning. In ICLR 2024 Workshop: How Far Are We From AGI, 2024.
D.-K. Nguyen, M. Assran, U. Jain, M. R. Oswald, C. G. Snoek, and X. Chen. An image is worth more than 16x patches: Exploring transformers on individual pixels. arXiv preprint arXiv:2406.09415, 2024.
OpenAL. Thinking with images. Accessed: 2025-05-09.
with -ittheggges// 2025.
R. Paiss, A. Ephrat, O. Tov, S. Zada, I. Mosseri, M. Irani, and T. Dekel. Teaching clip to count to ten. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3170-3180, 2023.
G. Patterson and J. Hays. Coco attributes: Attributes for people, animals, and objects. European Conference Computer Vision, 2016.
Z. Peng, W. Wang. L. Dong, Y. Hao, S. Huang, Ma, and F. Wei. Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint arXiv:2306. 0e0222366.4824, 14824, 2023.
K. Pham, C. Huynh, S.-N. Lim, and A. Shrivastava. Composing object relations and attributes for image-text matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14354-14363, 2024.
A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763 PmL.R, 2021.
S. Schuster, R. Krishna, A. Chang, I.. Fei-Fei, and C. D. Manning. Generating semantically precise scene graphs from textual descriptions for improved image retrieval. In A. Belz, L., Coheur, V. Ferrari, M.-F. Moens, K. Pastra, and I. Vulić, editors, Proceedings of the Fourth Workshop on Vision and Language, pages 70-80, Lisbon, Portugal, Sept. 2015. Association for Computational Linguisties. doi: 10. 10.18653/v 1/W1/-28111121111 URL https::/tcc.cllaa http//aaclatthoooggggg org.1ly-..g//111111111
S. Sharilymoghaddam, S. Upadhyay, W. Chen, and J. Lin. Unirag: Universal retrieval augmentation for multi-modal large language models. arXiv preprint arXiv:2405. 10311, 2024.
N. Sogi, T. Shibata, and M. Terao Object-aware query perturbation for cross-modal image-text retrieval. European Conference on Computer Vision, pages 447-464. Springer, 2024.
H. Su, W. Shi, J. Kasai, Y. Wang, Y. Hu, M. Ostendorf, W.-t. Yih, N. A. Smith, L. Zettlemoyer, and T. Yu. One embedder, any task: Instruction-finetune text embeddings. arXiv preprint arXiv:2212.09744 2022.
S. Tong, 7. Liu, Y. Zhai, Y. Ma, Y. LeCun, and S. Xie. Eyes wide shut? exploring the visual shortcomings of multimodal 1lms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9568-9578. 2024.
M. Tschannen, A. Gritsenko, X. Wang. M. F. Naeem, I. Alabdulmohsin, N. Parthasarathy, T. Evans, L. Beyer, Y. Xia, B. Mustafa, et al. Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features. arXiv preprint arXiv:2502.1423 147766 2025.
J. Urbanek, F. Bordes, P. Astolfi, M. Williamson, V. Sharma, and A. Cooero--Soianne A picture is worth more than 77 text tokens: Evaluating clip-style models dense captions. In Proceedings of the IEEE/VFF Conference on Computer Vision and Pattern Recognition, pages 26700-26709, 2024.
H. Wang, M. Liao, Z. Xie, W. Liu, and X. Bai. Partial scene text retrieval. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
13

---

C. Wei, Y. Chen, H. Chen, H. Hu, G. Zhang, J. Fu, A. Ritter, and W. Chen. Unir: Training and benchmarking universal multimodal information retrievers. In European Conference on Computer Vision, pages 387-404. Springer, 2024.
H. Wu, Y. Gao, X. Guo, Z. Al-Halah, S. Rennie, K. Grauman, and R. Feris. Fashion iq: A new dataset towards retrieving images by natural language feedback. In Proceedings of the IEEE/CVF Conference computer vision and pattern recognition, pages 11307-11317 2021.
P. Wu and S. Xie. V*: Guided visual search 2311::331..44135, 2023.
mechanism in multimodal llms. arXiv preprint
C. Xiao, I. Chung, I. Kerboua, J. Stirling, X. Zhang, M. Kardos, R. Solomatin, N. A. Moubayed, K. Enevoldsen, and N. Muennighoff. Mieb: Massive image embedding benchmark. arXiv preprint arXiv:2504. 10471, 2025.
J. Xiao, J. Hays, K. A. Ehinger, A. Oliva, and A. Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE computer society conference on computer vision and pattern recognition, pages 3485-3492. IEEE, 2010.
R. Xiao, S. Kim, M.-I. Georgescu, Z. Akata, and S. Alaniz. Flair: Vlm with fine- fine-grained language- informed image representations. arXiv preprint arXiv:2412.0356,, 2024.
M. Yasunaga, A. Aghajanyan, W. Shi, James, J. Leskovec, P. Liang, M. Lewis, L. Zettlemoyer, and W.-t Yih. Retrieval-augmeentd multimodal language modeling. arXiv preprint arXiv:2211.1256. 2022.
P. Young. A. Lai, M. Hodosh, and J. Hockenmaier. From image descriptions to visual denota- tions: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Compulational Linguistics, 2:67-78, 2014.
R. Zellers, Y. Bisk, A. Farhadi, and Y. Choi. From recognition to cognition: Visual commonsense reasoning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages
2019.
X. Zhai, B. Mustafa, A. Kolesnikov, and L. Beyer. Sigmoid loss for language image pre-training, In Proceedings of the IEEE/CVF international conference on computer vision, pages 11975-11986, 2023.
H. Zhang, Zhang, X. Hu, Y.-.. Chen, Li, X. Dai, L. Wang. Yuan, J.-N. Hwang, and J. Gao. Glipv2: Unifying localization and vision-language understanding. Advances in Neural Information Processing Systems, 35:36067-3600 -6080, 2022.
K. Zhang, Y. Luan, H. Hu, Lee, S. Qiao, W. Chen, Su, and M.-W. Chang. Magiclens: Self-supervised image retrieval with open-ended instructions. arXiv preprint arXiv:2403. 19651, 2024.
B. Zhou, A. Lapedriza, A. Khosla, A. Oliva, and A. Torralba. Places: A 10 million image database for scene recognition. IEEE Transactions Pattern Analysis and Machine Intelligence, 2017.
J. Zhou, Z. Liu, Z. Liu, S. Xiao, Y. Wang. B. Zhao, C. J. Zhang, D. Lian, and Y. Xiong. Megapairs: Massive data synthesis for universal multimodal retrieval. arXiv preprint arXiv:2412. 14475, 2024a.
J. Zhou, Z. Liu, S. Xiao, B. Zhao, and Y. Xiong. Vista: visualized text embedding for universal multi-modal retrieval. arXiv preprint arXiv:2406.0422 04292, 2024b.
Y. Zhu, O. Groth, M. Bernstein, and L.. Fei-Fei. Visual7w: Grounded question answering in images. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 4995-5004, 2016.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The contributions and scope are reflected in the paper detailed in the end of our introduction (Section I).
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitations are listed Appendix E
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
15

---

Justification:
We do not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We describe the benchmark construction process and details of our methods in Appendix A and Appendix
Section 4 Section
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
16

---

Answer: [Yes]
Justification: The link for downloading our COCO-FACET benchmark is attached in the main text, and the evaluate code is included in the supplementary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed run reproduce the results. See the NeurlPS code and data submission guidelines (https /pips.ccc/pulic/guuu guwde//ddee/ee/ Cote/ooodeeeiiseeeeeeee hwtpibbmsssiooooooooiooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We do not perform the training of any new model, and we have included all the test details in Section Section 4 Section 5 and in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: The only results with factors of variability are the linear approximation
described in Section 5.2 and the standard error reported in the Table 15
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by emror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
17

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The compute resources required is mentioned in Appendix B.1
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: The authors have reviewed the NeurIPS Code of Ethics and make sure that the research conforms with it. We perform manual checking of the images used as ground truth in COCO-FACET make sure that they do not contain inappropriate contents, and other considerations are detailed in Appendix A
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The broader impacts are described in Appendix
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
18

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification:
We do not release any new pretrained model or newly-scraped data.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We credited the creators or owners models we used for evaluation in and
Appendix and included the license and terms of use. The license and terms of use of the data are mentioned Appendix A
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
19

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, batasets.papersstt paperswi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
The new assets include the COCO-FACHE benchmark and the new methods we proposed. The documentation is attached in the code file in the supplementary material.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: No human subjects are involved for crowdsourcing.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
No human subjects are involved in the experiments.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
20

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification: We use LLM to generate the prompts and select from existing prompts, which is described in Section 4.1
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM for what should or should not be described.
21

---

A The COCO-FACET Benchmark
We provide more details on the COCO- FACET Benchmark in this section.
Building upon the annotations of MSCOCO 2017 captions Lin et al., 2014]. COCO-Stuff [Caesar
al. 2018], Visual7W [Zhu al. 2016), and Visual Dialog [Das al. 2017 7 based on the COCO images, we generate 8 new subsets for evaluating T2I retrievers on attribute-ocused retrieval tasks. Use of the COCO images must abide by the COCO anf Flickr Terms of Use The MSCOCO 2017 and COCO-Stuff annotations belong to the COCO Consortium and are licensed under a Creative Commons Attribution 4.0 License. The Visual7W annotations are under MIT License. The Visual Dialog annotations are licensed under Creative Commons Attribution 4.0 International License.
We extract images with different types of objects including laptops, bicycles and sandwiches it, with 3749 test cases in total; 8 gestures of people including standing and sitting on it, with 1176 test cases in total; and other 6 benchmarks showing the other attributes. The details of the dataset construction is introduced in Appendix A.I where the full list of categories and related statistics can be seen in Table 7
In short, we came up with the most proper division for every image in every fixed criteria. The information needed for the classification might come from the whole picture part of it. Each test case consists of a query text (c.g.. "Find me an everyday image showing some object or surface made of stone"). 99 negatives that should be divided into other categories with no intersection with the true category in the classification for the retrieval (e.g. photo of wooden floor). For each of the test cases in these provided data, the chance level performance to choose a correct image is 1 1%.
image corresponding to the text (e.g. photo of stone wall) and
A.1 Generating Datasets
We use these steps norm when generating the datasets. Details for each benchmark will be shown in Appendix A.2
1. We find proper criteria that can include sufficient images and categories.
2. Initially, we filter the captions or dialogues within the datasets to identify those containing information in specific categories.
3. Then we analyze some filtered samples and refine the rules of filtering.
4. We remove the samples that belong to multiple mutually exclusive categories under the rule.
5. Subsequently, we manually validate the positive images to minimize wrong labels and ambiguity.
6. We design the queries and randomly select negative samples from mutually exclusive categories.
A.2 Details of Sets
See Table for the category statistics.
A.2.1 Gestures of People
Idea: The position and arrangement of different parts of the human body vary significantly depending on the gesture. such as standing, sitting, or lying. Therefore, classification based on gestures should be feasible.
Categories:
"stand", "sit", "jump", "lie", "bend", "squat", "kneel", "crawl" (8 categories, 1 1176 samples in total).
Avoidance: We avoid using "squat" or "kneel" as negative samples for each other, as they be hard to distinguish (e.g., a kneeling squat).
Sampling: All samples
sourced from the val2017 split of the COCO dataset. We first filter captions using keywords such as "stand" or "sitting." Then, we refine the keywords
http:///oocdaaaastt.org www.eeroootttteeeeeeeeeeeeeeeeee and https://www flickr. com/ comm/creativecomoons
22

---

Table 7: Statistics of our COCO-FACET benchmark( Animals, Count of People. Materials, Times, Weathers,
Scenes,
Objects,
Gestures).
NUM.
NUM. Categories
Details of each category
Samples
"bird": 44, "cat": 141, "dog": 112, "horse": 88, "sheep"; 48, "cow": 56, "clephant": 79, "bear": 42, "zebra": 73, "giraffe": 80
763
10
172
>90
omitted
3849
69
omitted
"0": 131, "I": 88, "2": 106, "3": 86, "4" 42, "5"; 22, "6"; 23,
570
12
"7": 11, "8": 8, "9": 8, "10": 7, "over 10": 38
1128
5
"wood": 231, "stone": 57, "metal": 741, "paper": 68, "brick": 31
"daytime"; 588, "night": 103, "aftermoon": 24, "dusk"; 3, "morning:: 15, "sunrise": 1, "evening": 26
760
"sunny": 179, "clear": 77, "misty": 14, "overcast"; 17,
694
12
"cloudy": 138, "rainy": 50, "drizzly": 1, "stormy", 2.
"snowy": 193, "warm": 8, "cold": 14, "chilly":
"stand": 660, "sit": 386, "lic": 25, "jump": 77,
1176
8
"bend": 10, "squat": 3, "kneel": 14, "crawl":
with more precise phrases to avoid mislabeling images based on captions like "A building sits between." " Finally, we manually check the images to ensure that they meet our criteria.
A.2.2 Materials of Objects or Surfaces
Idea: Although different materials may have similar uses, they often exhibit different visual characteristics. Therefore, a classifier should be able to distinguish between them.
Categories: "wood", "metal". "stone". "brick", "paper" (5 categories, 1 1128 samples in total).
Avoidance: We avoid using "stone" "brick" negative samples for each other. This is because some images contain bricks made of stone. Since we categorize such images under "stone",, aim avoid interfering with the classification of "brick".
Sampling: We utilized the COCO-Stur Stuff annotations on val2017, extracting subcategories that fall under our target categorics. Then, we manually review the images and exclude samples that may contain multiple materials, have unclear material identification, only show a small portion of the target material.
A.2.3 Count of People
Idea: Object detection models are capable accurately locating object boundaries. There- fore, it is reasonable to infer that they can accurately count the number of people in an image.
Categories: "0", "2", "3", "4" "5", "6", '7", "8", "9", "10", "over 10" (12 categories, 570 samples in total).
Avoidance: require that the negative samples differ from the positive sample by least 3. This accounts for potential ambiguity regarding whether certain individuals should be
23

---

included or excluded, providing a margin for potential count discrepancy of one person in both positive and negative samples. Specifically. "over 10" is treated as 11 when calculating the differences.
Sampling: We primarily use the Visual7W VQA dataset our resource. We first filter questions that contain interrogative phrases like "how many people" Then, we select ques- tions with clear answers relevant to our task, either manually verify them or deduce the final answer from similar questions (e.g.. "how many people other than a person") and visually confirm.
A.2.4 Weather Conditions
Idea:
Weather conditions are relatively easy to distinguish in images showing large outdoor scene. In addition, there are many different criteria for evaluating weather, allowing substantial number of categories.
"sunny", "clear", "misty"" "overcast", "cloudy", "rainy", "drizzly", "stormy", "snowy", "warm", "cold", "chilly" (12 categories, 694 samples in total).
Categories:
Avoidance: For the first two categories ("sunny" and "clear"), avoid using them as negative samples for each other because they both describe a sky with no clouds. We also avoid using "warm" or "sunny" as negative samples as these conditions can coexist with clear or sunny sky.
Categories three through eight ("misty" to "stormy") are not used negative samples for each other, as they all describe conditions involving clouds, rain, or something that obstructs sunlight, making them difficullt to delineate clearly. Furthermore, we did not use "cold" or "chilly" as negative samples for these conditions, as was difficult to determine the precise temperature under such circumstances. Categories three through five ("misty", "overcast", "cloudy") are not used as negative samples with "warm" for similar reasons (We assume that it is generally not warm when it is raining).
Among the last four categories ("snowy", "warm", "cold", "chilly"), "snowy", "cold", and "'hilll" often co-occur and are therefore not used as positive and negative samples together.
Sampling: We first extract questions containing weather conditions from the Visual7W and VisDial datasets. Then, we record, simplify and verify the answers. addition, we deduce weather conditions from the captions in val2017 and perform a double check.
A.2.5 Time of the Scenery
Idea: The Visual 7W dataset includes "When" as a category of questions. These questions often have clear answer, thus can get the time of the scene. This is proper benchmark, since time is property of the whole image.
Categories: "daytime", "night", "evening", "afternoon", "dusk", "morning", "sunrise"(7 categories, 760 samples in total)
Avoidance: We avoid using samples from similar category as negative samples. For instance, "morning" is not a good negative sample for "sunrise".
Sampling: We filter the answers the time questions in Visual 7W. The core word used our category.
A.2.6 Scenes of the Locations or Activities
Idea: Different locations and activities have quite distinct scenes. Through the"where" questions of the Visual 7W set, we can easily get description of the scene (a noun), and repetitions hardly exist.
Categories: "beach (scene)", "beach shore", "baseball game", "Oahu", "baseball field", "baseball park", "sports arena". "outdoor eating area", "bedroom", "bathroom", "station" "train station", "railroad tracks", "backyard", "backyard patio", "zoo", "tennis court", "Broad- way". "harbor", "street", "city street", "side of the road", "mountain", "river", "safari", "grassland", "airport", "air strip", "parking lot(area)", "skate park", "open field", "field", "construction site", "classroom". "fountain", "London", "nature", "farm", "restaurant", "din- ner". "dining room", "kitchen", "kitchen being remodeled in home", "living room", "park",
24

---

"ski slope", "(ski) lodge", "sidewalk", "parlor", "boardwalk", "waterhole", "'baby shower", "press conference", "upple computers". "downtown Toronto", "outside a city", "near river". "by/near the ocean", "ocean shore", "inside home", "in room", "inside a refrigerator", "near the food", "performance", "market", "farmers market", "a man on phone room", "tourist trap", , "coffee shop". "on desk", "table", "on counter", "sky". "woods"" "birthday party", "outdoor show", "yard", "soccer field", "indoors", "in front of clock tower", "on road in front of large building". "in building", "in front of a television", "third street", "in a car", "airport runway", "intersection", "museum", "concert photography session", "inside home very close marina and the sea", "road", "Tokyo", "by the water", "on sand dune", "bakery", "motorcycle race", "house" (172 samples in total)
Avoidance: We just avoid the categories with similar meaning to be used as both positive and negative. In fact, we manually check all the negative samples to avoid conflicting with the positive sample.
Sampling: We used the "Where" questions from the Visual 7W set. The answer often used directly as the category to expand the number of categories.
A.2.7 Objects contained
The COCO dataset provides the retrieval for objects in it, and the position of the objects are located with bounding boxes. So whether the object is crucial feature in the picture is really clear. To be clear, we do not involve animals or people in this classification, for they have totally different norms.
Idea:
Categories: "bicycle", "bus", "light", "backpack", etc. (3849 samples in total)
Avoidance: The COCO set gives quite comprehensive content about the objects in the image. So we just need to avoid using the images with the same kind objects (possibly with other objects) as negative samples.
Sampling: We can just use the annotations of objects for the COCO validation set.
A.2.8 Animals Contained
Idea:
Animals have a great difference from static objects; they are classified with their appearances and actions. As a result, we create an independent benchmark.
"giraffe", "zebra", "bear", "elephant", "cow", "sheep", "horse", "dog", "cat", "bird" (10 categories, 763 samples in total)
Categories:
Avoidance: We just need to avoid the same species some close species in the negative samples.
Sampling:
We include the objects marked in the COCO validation set that are animals or similar to animals.
B Details of Retrievers
We list the details of retrievers used in our evaluation this section, including the baselines and the promptable image embeddings. The evaluation code is attached in the supplementary material for reproducibility purpose.
B.1 Information of Baselines
All evaluations of CLIP-family, MagicLens, MLLM-based universal multimodal retrievers, and variants of VeM2Vec can be done using one A6000 GPU with 48GB memory less than hours per category.
The CLIP family comprises vision-language models trained via contrastive learning on large-scale image-text pairs. CLIP [Radford et al., 2021] introduced this paradigm, enabling zero-shot transfer to various vision tasks. We use the weights
CLIP-family:
pettps://huggingee
co/
under MIT License. EVA-CLIP [Fang et al.) 2023 2024] enhances CLIP by integrating improved training techniques for better efficiency and ef- fectiveness. We access their public model weights at OpenCLIP [[lharco et al. 2021 with
openai/clip-vite
25

---

model name "EVA01-g-14" and "EVA02-E- p4 under MIT license. SigLIP (Zhai et al. 2023] replaces the softmax loss with a sigmoid loss, allowing for scalable training without the need for large batch sizes. Building upon this, SigL.P2 (Tschannen et al. 2025] incor- porates multilingual capabilities and improved semantic understanding. We use the weights at hhttps: battps://huggingffee
co/ ct/ //huggingface co/ http:/oggggl//eegggee.2 wiglipp-so400m 50400m-patch144 patphh1--hhhhhtt1 for SigLIP2 under Apache license 2.0. BLIP2 [Li et al. 2023b] fine-tuned on COCO leverages frozen image encoder and lightweight Q-Former to bridge vision and language modalities effectively. We use the @bip2_feature _el___feetettteaaaealeeeeeeeeeee extractor" provided by LAVIS [i et al., 2023aa] under BSD 3-Clause License. By default, we directly the
-atch11134 for SigI.IP and httpps
query text as text input for these models.
MagicLens: MagicLens (Zhang et al. 2024 is a self-supervised image retrieval model trained on 36.7M triplets (query image, instruction, target image). It supports open-ended instructions, enabling retrieval based on diverse semantic relations beyond visual similarity, The model employs dual-encoder architecture with shared parameters and utilizes multi-head attention pooling to generate unified embeddings. We use the weights shared in the official github repository at chttps://gtt gitbbb com/google deepmind magi cllicceenss under Apache-2.0 license. We only use their vision encoder and language encoder like CLIP, as we find that the model does not support zero-shot instructions for embeddings.
E5-V: E5-V (Jiang et al. 2024a adapts an MLLM to generate universal multimodal embeddings. Unlike traditional models trained on image-text pairs. E5-V leverages MLLM's capabilities to represent multimodal information effectively, demonstrating significant potential in various retrieval tasks. We use the model weights released at htttps://huugggngafaee...o wot.:/eeo.ooooooooooo/eoe../
MM-Embed:
MM-Embed [Lin et al. 2024 is universal multimodal retrieval model that fine-tunes MLLMs as bi-encoder retrievers across diverse datasets and tasks. It supports flexible vision-language alignment and is adaptable to both retrieval and classification tasks without the need for instruction tuning. However, we find that the model does not process zero-shot instructions well. We use the weights at https:: httpss:///ggggggggee Non Commercial 4.0.
co htt//vviiaa////ee// aaMddaaaaMMMMM Embed under Creative Commons Attribution
MMRet: MMRet [Zhou et al. 2024a is trained on MegaPairs, a massive synthetic dataset generated using vision-language models and open-domain images. It employs separate encoders for vision and language, followed by deep fusion layers for cross-modal alignment, achieving state-of-the-an performance in universal multimodal retrieval tasks. We employ the MMRRI-MLLMMS2 released at https://huggingge
Co/BAAI/GGE-VLL -LLLMLL- under the MIT license.
LLaVE: LLaVE [Lan et al. 2025 introduces hardness- hardness-weighhted contrastive learning to train large language and vision embedding models. By dynamically adjusting the learning process based on the difficulty of negative pairs, LLaVE enhances representation learning, leading to improved performance across various multimodal tasks. We use the LLaVE-2B released at https://huggingface cttifaace.cc//t// cww//hibinnaae
LLLaVeennnLLL-----L 2B under Apache license 2.0.
VLM2Vec: VLM2Vec fiang et al. 2024b transforms vision-language models into efficient multi-
modal embedders through contrastive training on the Massive Multimodal Embedding Benchmark (MMEB). supports instruction- -uideed representation generation, outperforming existing mod- els on both in-distribution and out-of-distrine -i-or-ddiitriittionn datasets. We use the VLMMVVccPhi-3.55 at httpp://hugginggee co Co/TIGER- TEEETTIGER-LLLVLLLEE Lab/VLM2Vec-FUl Full under Apache license 2.0.
B.2 Retrieval with CLIP-Style Text
Our query text is designed to suit the retrieval tasks in universal multimodal embedders such as VLM2Vec. So, a question arises when we try to evaluate with CLIP, which the recommended CLIP evaluation starts with "A photo of." We perform ablation study replacing the text with CLIP-style text for evaluation. The mechanism of substitution is shown in Table The results are shown in Table 9 where significant difference is observed.
26

---

Table 8: CLIP-style text used in our evaluation.
Original Text
Revised Text
Examples
"Find me an everyday image
"Find me an everyday image that..."
"A photo that
that taken during the evening."
"A photo that is taken during the evening."
"Find me an everyday image
"Find everyday image with-.."
"A photo with
with over 10 people."
"A photo with over 10 people."
'Find me an everyday image showing some object or surface made of brick."
"Find me an everyday image showing-...
photo showing...
"A photo showing some object or surface made of brick."
Table 9: Recall@1 and Recall@5 of CLIP-ViT-L/14 evaluation with original text or with CLIP-style text in percentage points COCO-FACET benchmark.
Avg.
Recall @
CLIP VIT-L/14
91.5
55.2
54.0
3.5
3.5
4.5
4.2
6.8
33.7
w/ CLIP-style text
91.5
53.5
54.0
8.4
5.1
4.5
2.6
4.8
33.8
Recall @5
CLIP-ViT-L/14
98.4
80.8
72.7
13.5
11.4
10.1
14.3
18.5
47.0
w/ CLIP-style text
98.4
79.1
72.7
13.3
14.4
9.9
16.4
19.1
47.6
Table 10: GPT-written prompts for COCO-FACET.
Categories
Prompts
Animals
<| <limage_1>> Represent the given image with the following question: Which animals be seen in this image?
Scenes
< limage_1l>> Represent the given image with the following question: What type of location is depicted in this image?
Objects
<limage_1|> Represent the given image with the following question: Which objects are present in this image?
Count of People
< <limage_1|> Represent the given image with the following question: How many people are present in this image?
Materials
<limage_1|> Represent the given image with the following question: What material are the objects in this image made of?
Times
<limage_1|> Represent the given image with the following question: What time of day is depicted this image?
Weathers
<limage_1l> Represent the given image with the following question: What is the weather like in this image?
Gestures
<limage_11> Represent the given image with the following question: What gesture are the people making in this image?
27

---

Table 11: Human Human-written written prompts for COCO-FACET.
Categories
Prompts
Animals
<limage_1| __ Represent the given image with the following question: What animals are in this image?
Scenes
< <lmage__l>> Represent the given image with the following question: What scene is in the image?
Objects
<| <limage_1|> Represent the given image with the following question: What objects are the image? <limage_1l> Represent the given image with the following question: How many people are in the image?
Count of People
Materials
< <limage_1l> Represent the given image with the following question: What are the objects made of in the image?
Times
<| <limage_1|> Represent the given image with the following question: When the image taken? < <limage_1l> Represent the given image with the following question: What is the weather in the image?
Weathers
Gestures
< <limage_1|> Represent the given image with the following question: What is the person doing in the image?
Table 12: Recall@ I and R of text-to-image retrieval in percentage points on our COCO- FACET benchmark with prompt, GPT-written prompts, and human- written prompts lead to a similar performance gain.
human-writter written prompts. The human-
X
Avg.
Recall @
VLM2Vec-Phi-3.544
95.5
69.8
74.4
14.4
6.3
5.5
4.3
9.8
44.5
w/ GPT prompt
90.7
81.4
75.5
72.7
25.8
18.4
14.4
15.7
53.4
w/ human prompt
93.5
80.8
75.7
68.1
24.8
43.6
14.3
22.9
56.3
Recall @
VLMMVVc--hii-35-VV
99.3
90.7
90.7
36.6
18.2
12.9
19.2
27.1
58.9
GPT prompt
98.7
95.9
92.0
92.1
48.8
82.4
36.5
39.3
75.5
w/ human prompt
99.6
95.9
91.6
91.2
43.8
64.2
36.9
50.2
74.6
B.3 Promptable Image Embeddings
We list the obtained GPT-written prompts for eight categorics our COCO-FACET benchmark in Table 10 We also test human-written prompts listed in Table [I The results shown in Table T2 where we find that human-written prompts can lead to similar improvement.
B.4 Pre-Processing Embeddings
We use the following template for GPT-4o's prompt selection:
(Prompts} Given the instruction (text), choose the most relevant prompt for verifying the results. Please answer in one letter.
The "Prompts" part lists all the prompts in Table 10 in the format of "A. Represent the given image with the following question: What type of location is depicted in this image?".
28

---

Table 13: Recall and R of text-to-image retrieval using various base retrievers with promptable image embeddings compared with original image embeddings.
X
Avg.
Recall@* @
VLM2Vec-Phi-3.---
95.5
69.8
74.4
14.4
6.3
5.5
4.3
9.8
44.5
w/ GPT prompt
90.7
81.4
75.5
72.7
25.8
18.4
14.4
15.7
53.4
E5-V
92.7
70.4
71.2
31.4
10.5
7.5
5.2
20.1
46.0
w/ GPT prompt
96.3
77.3
77.0
60.3
18.9
36.2
7.1
24.7
54.9
MM-Embed
92.7
67.4
68.1
13.3
7.4
5.1
3.9
19.5
42.8
w/ GPT prompt
64.4
68.6
68.0
15.9
11.8
8.0
3.8
20.8
41.5
MMReI-MLLM-S2
97.2
72.1
76.0
29.8
10.0
8.4
3.6
24.1
48.8
w/ GPT prompt
91.7
78.5
82.2
78.6
27.2
21.1
8.7
23.3
57.6
LLaVE-2B
96.3
70.9
73.1
19.4
8.8
3.0
4.5
19.2
45.6
w/ GPT prompt
91.9
72.1
79.6
82.0
28.7
22.4
6.8
18.5
56.1
Recall @55
VLM2Vec-Phi--.5-VV
99.3
90.7
90.7
36.6
18.2
12.9
19.2
27.1
58.9
w/ GPT prompt
98.7
95.9
92.0
92.1
48.8
82.4
36.5
39.3
75.5
E5-V
98.3
91.9
89.0
60.1
25.2
18.6
16.1
35.5
61.8
w/ GPT prompt
99.2
95.4
92.7
79.2
48.2
57.0
30.1
45.2
73.1
MM-Embed
98.7
87.8
84.9
29.3
20.3
15.0
15.3
45.3
58.4
w/ GPT prompt
83.3
87.8
85.1
45.2
33.4
22.6
19.2
44.3
60.6
MMRet-MLLM-S2
99.9
92.4
91.6
45.2
28.3
19.7
17.6
49.7
64.5
w/ GPT prompt
98.2
95,4
96.0
94.4
52.4
75.7
33.7
45.2
77.7
LLaVE-2B
99.4
93.6
88.2
41.3
21.5
10.9
16.3
37.0
59.5
w/ GPT prompt
98.7
90.1
94.5
94.4
52.5
53.2
30.8
40.6
74.3
Table 14: Recall@1 and Recall@5 of FLAIR and CLIP-ViT-B/16 in percentage points on COCO- FACET. FLAIR does not show significant improvement compared with CLIP.
Avg.
Recall @
V/IB-//
94.0
44.2
57.4
1.1
1.6
3.8
3.3
9.6
35.0
FLAIR
82.2
52.9
54.5
1.2
3.0
6.3
4.6
6.3
33.0
Recall @
CLIP-VIT-B/16
99.1
75.6
77.6
14.7
10.7
17.4
12.4
20.9
49.8
FLAIR
97.1
81.4
80.2
9.7
10.4
11.8
17.9
19.5
50.3
C More results
C.1 Detailed Results of Various Base Retrievers with Prompt
We show that this strategy generalizes different base retrievers. See Table
C.2 Comparison with Other Promptable Embeddings
CLOC [Chen et al., 2024a and FLAIR (Xiao et al. 2024 are two non-MLL retrievers with similar ideas on promptable image embeddings, but they have constraints on the image focus given prompts: CLOC limits its image focus to be rectangular region (bounding box)-the prompt is either bounding box or text, and the text- cext-prompte image embedding only trained with a single bounding box prediction. This limited image focus is similar to the zooming cropping approach, which can struggle with non-region- based attributes like Scenes and Times. FLAIR relaxes this constraint by training the model with patch-level image- text alignment, but is still restricted to the single patch alignment, while non-region- based attributes might require a joint match with several patches (e.g.. :Count of People).
We evaluate FLAIR on COCO-FACET since CLOCC not open-sourced FLAIR more flexible with the image focus. We use the same set of GPT prompts VLM2Veo-PPh-- We also

---

Table 15: Recall @ and Recall @5 of accelerated text-to-image retrieval with approximated prompt- able image embeddings percentage points on our COCO-FACET benchmark, along with the standard error. The results of approximation are averaged over five independent runs.
X
Avg.
Rocall
VL.M2 Phi-3.5.V
95.5
69.8
74,4
14.4
6.3
5.5
4.3
44,5
w/ lincar appeox.
72.1 # 2.2
67.2
0.5
57.1
# 0.4 47
5 = 4.0
24.3
1.1 35.7 # 2.7
9.0 +
0.7
14.2 +
2.1
42.5 0.5
GPT prompt
90.7
81.4
72.7
18.4
53.4
Recall
VLM2Vo-Phi--3.-V
99.3
90.7
90.7
36.6
18.2
1299
1922
27.1
58.9
appeox.
81.6 2.1
4
83.2 4
0.5
73.
43.9
1.2
0.3
w/ GPT prompt
98.7
95.9
92.0
92.1
48.8
82.4
36.5
39.3
75.5
attach the results for CLIP-ViT-B/IG on the same scale with FLAIR for fair comparison (with the weights at htttts://hugte http://huggiggfcccc co/ /pppaa/ ciip-iit---iiiiiiiiiiiieiiiiiilm vit- base patcheeptthh66 under MIT License). In Table 14 the improvement brought by FLAIR on top of CLIP is not significant nor consistent on COCO-FACET. This shows the advantage of our pipeline that utilizes MLLM for promptable embeddings with a flexible image focus.
C.3 Detailed Accelerated Retrieval Results
We list the mean and error bar over five independent runs of the linear approximation in Table 15 The variability mainly comes from the random selection of samples used for deriving the matrix W. The eгrоr bar is calculated as the standard error (sample standard deviation divided by the square root of number of runs).
We evaluate the actual inference cost of our pipeline on an A6000 GPU. To simulate larger- scale retrieval settings, we duplicate the COCO-FACET image pool multiple times, the original dataset does not contain enough images. We use linear approximation with = 10 and test both GPU and CPU settings, since A6000 GPU only supports up 1M vectors in FAISS. As shown in Table [6 and Table 17 the promptable embedding step (KF + F) is the main cost driver w when N is small, but retrieval cost becomes more noticeable when N is large (10M and 20M in Table 17p As for the memory cost of the linear approximation, the peak GPU memory usage is 26.5 GB when we derive the promptable embeddings with batch size of 50. This fits comfortably on modern 40GB A100 or 48GB RTX 6000. In use cases with tighter resource constraints, the method can be adapted by reducing batch size with larger latency.
Table 16: Average inference time using linearly approximated promptable image embeddings on VI.M2Ve--Ph--3.5-- on Times with varied image pool size. Retrieval optimized by faiss.
A6000 GPU.
Image Pool Size
1K
1M
Avg. Per-Query Time (s)
7.33
7.53
Table 17: Average inference time using linearly approximated promptable image embeddings on LLLM2Vcc-Pi-3.5-. on TTimes with varied image pool size. Retrieval is optimized by faiss. Index on CPUs.
Image Pool Size
1K
IM
10M
20M
Avg. Per-Query Time (s)
7.25
7.61
11,60
36.39
C.4 Visualization Examples
Figure 4 visualizes which image regions the models attend to when matching a query by computing gradients of the image-text similarity score with respect to input pixels. We use VLLM2Vec-Phi-3.5. as an example, which processes images through a multi-crop strategy: one global crop capturing the entire image context, and multiple local crops arranged in a 2x2 spatial grid that capture fine-grained regional details. Fach crop is resized to 336x336 pixels and processed independently by the vision encoder. By backpropagating through the similarity score, obtain gradient magnitudes for each
30

---

VLM2Vec w/o Prompt
(b) VLM2Vec Prompt
VLM2Vec w/ Linear Approximation
Figure 4: Visualization of which image regions the models attend to for image-text matching. The query text is "Find an image that contains any car."
crop, which indicate how sensitive the similarity is to changes in different image regions. The final visualization combines both global and local attention through a weighted average (0.5 for the global map and 0.5 for the local map). producing comprehensive heatmap that highlights which image regions contribute
to the model's decision.
We observe the change in image focus by comparing the heatmaps in Figure In (a), the image focus is diverse and the image-text similarity negative. In (b), the prompt shifts the image focus to the front wheel of the car, increasing the similarity to 0.3. (c), the linear approximation (K = 100) is not effective using the GPT prompt, but it still reduces the image focus in the irrelevant region (background plants), which leads to a positive similarity.
C.5 Detailed Text-Based Retrieval Results
find that text-based retrieval suffers from hallucination and linguistic ambiguity. Some failure cases from Animals regarding birds are shown in Figure 5 as examples. We find that elements like sky or feathers could trigger the model to answer with bird's existence. In addition, this approach could not deal with polysemy like chicken shown in the fifth example. We also notice that the existence of bird patterns (in the third example) complicates this process, as the model could not prioritize real birds over the bird pattern the container.
D Test Cases in COCO-FACET
We collect some challenging test cases in COCO- for better understanding the drawbacks of retrievers. Examples are shown in Figure 6 There are several challenges: (1) The required attributes are not the main subject of the images, especially with the test cases from Animals and Objects. (2) The semantic understanding of attributes, like "O people" and "made of." (3) The visual grounding of attributes, like "sports arena" and "jumping" Current retrievers may not ground these attributes precisely. (4) Simple visual reasoning. In the Count of People category, the model is required to perform approximation of the number of people in the image. Notably, to reduce the difficulty, require that the negative samples differ from the positive sample by at least 3, but the model performance still low. In the example "a sunny day." the ground truth does not feature sunny sky but shows clear shadows the ground. (5) Robustness deceptive attributes. In the example of "during the day time," there are lights outside in the CLIP-retrieved image, but human can judge that this image taken at night.
E Limitations
E.1 COCO-FACET Benchmark
First, COCO-FACET is built on existing annotations of COCO images. Although we have conducted manual checking detailed Appendix A there might still contain inaccurate and ambiguous anno- tations. For example, some small animals or objects might not be annotated by bounding boxes in
31

---

USER:
<image>
Which animals can be seen in this image? Please answer in detail. ASSISTANT: in this image, there birds visible.
USER: <image>
Which animals can be seen in this image? Please answer in detail. ASSISTANT: the image, there while the other bird is flying in the sky. the presence of these hirds adds a sense of liveliness to the scene, as they interact with the architecture and the surroundings.
hirds visible. bird is perched light fixture,
USER: <image>
Which animals seen in this image? Please answer in detail. ASSISTANT: the image, there are two hirds visible. one bird is perched branch, while the other bird is sitting vase. the birds are part of the decorative elements in the scene, adding touch of nature the setting.
USER: <image>
Which animals can seen in this image? Please answer detail, ASSISTANT: in the image, there are two birds visible. they are flying over the water, adding of movement and life the scene.
USER: <image>
Which animals can be seen in this image? Please answer in detail. ASSISTANT: this image, there is chicken visible the plate.
USER: <image>
Which animals be seen in this image? Please unswer in detail. ASSISTANT: this image, dog can seen.
Figure 5: Failed top-1 retrieval results of the text-based retrieval. The query is "Find me an image that contains any bird." " in all cases.
MSCOCO, which could affect the evaluation of the Animal and Object attributes in COCO-FACET. Second, some existing universal multimodal embedders like mmF5 [Chen et al. 2025 are not. evaluated on our benchmark due to the limited computation resource.
32

---

Animals:
Objects: Find me image that contains any bottle.
Find an image that contains any sheep.
Ground Truth
CLIP
VLM2Vec w/o Prompt
Ground Truth
CLIP
VLM2Vec w/o Prompt
Scenes:
Count of People:
Find me an everyday image that shows the scene of sports arena.
Find me an everyday image with people in the picture.
Ground Truth
CLIP
VLM2Vec w/o Prompt
Ground Truth
CLIP
VLM2Vec w/o Prompt
Materials:
Times:
Find me an everyday image showing some object or surface made of wood.
Find me an everyday image that is taken during the daytime.
Ground Truth
CLIP
VLM2Vec w/o Prompt
Ground Truth
CLIP
VLM2Vec w/o Prompt
Weathers:
Gestures: Find me an everyday image that contains someone jumping.
Find me an everyday image that is taken in a sunny day.
Ground Truth
CLIP
VLM2Vec w/o Prompt
Ground Truth
CLIP
VLM2Vec w/o Prompt
Challenging test cases in COCO-FACET In each test case, the first image is the ground truth, the second is the top-1 image retrieved by CLIP, and the third is the top-1 image retrieved by VLM2Vec without promptable embeddings. VLM2 with GPT prompts can solve these test cases.
Figure 6:
E.2 Promptable Image Embeddings
First, our pipeline in Section 4.1 relies on the usage of GPT-4o's API. While other large language models, especially the open-sourced ones, can be good alternatives, have not tested them in scenario yet.
Second, the promptable image embeddings do not fully resolve the imbalance on different attributes. as we observe that the Recall@1 and Recall @ accuracies for Material, Weather, and Gesture are lower than other attributes.
33

---

Table 18: Recall @ I and 5 of selecting pre-processed promptable image embeddings of four disjoint prompts (the last row) and using image embeddings with without prompt (baseline) in percentage points on COCO-FACET.
Recall
VLM2Vec-Phi-33.
95.5
69.8
6.3
9.8
w/ GPT prompt
90.7
81.4
25.8
15.7
w/ Disjoint Prompt Set
97.0
71.5
8.1
11.4
@ecall@5
VLM2Vee--Phi335-XV
99.3
90.7
18.2
27.1
w/ GPT prompt
98.7
95.9
48.8
39.3
w/ Disjoint Prompt Set
99.3
90.1
19.5
25.4
Table 19: Recall and R of Linear Approximation on 100 samples from another random category (the last row), default Linear Approximation (the second row), and without prompts (the first row) percentage points on COCO-FACET. Results the last two rows are averaged over five independent runs.
Avg.
Recall
VLM2Vec-PPh-3.5-VV
95.5
69.8
74.4
14.4
6.3
5.5
4.3
9.8
44.5
w/ linear approx.
72.1
67.2
57.1
47.5
24.3
35.7
9.0
14.2
42.5
disjoint samples
56.9
62.4
54.8
37.7
20.4
23.2
5,4
8.4
37.4
Recall @5
VLM2Vec-Phi-3.5-V
99.3
90.7
90.7
36.6
18.2
12.9
19.2
27.1
58.9
w/ linear approx.
84.6
91.9
83.2
73.7
43.9
71.5
28.4
38.1
67.0
on disjoint samples
75.8
89.5
83.8
63.7
41.6
60.9
22.2
27.0
61.4
Third, the success of accelerated promptable image embeddings is not entirely zero-shot but relies prior knowledge of the query. For the pre-processing approach, although we show that it is applicable some unseen attributes, does limit support for entirely novel and highly specific attributes without prior knowledge. We simulate this setting by randomly splitting the categories on COCO-FACED into two parts, limiting the prompt set to be four random prompts from the original eight and testing on the other four categories. This leads to performance drop compared to the vanilla approach with ground-truue prompts (Table 18 For the linear approximation approach, W is derived from images of the same category as the query. We test the case when the categories of available samples are disjoint from the inference categories by randomly choosing another category and then uniformly sampling K = 100 samples from its pool. In Table 19 we observed that using such samples does not render the method completely ineffective, and still outperforms the baseline (without prompts, the first row) at Recall @5 However, there is a decline compared with using images from the same category. Hence, when there is no image from the target category, the linear approximation does not have a guaranteed performance.
F Broader Impacts
Improving attribute-fouused text-to-iman iomee retrieval can benefit applications that rely on fine ane-grained visual understanding. such as e-commerce. Our method enhances the precision of such retrieval tasks while maintaining efficiency, potentially enabling more responsive and accurate systems.
At the same time, fine-grained retrieval poses risks, including potential misuse in surveillance or amplification biases. addition, since our approach builds on pretrained multimodal models like Phi-3.5-V, V, it may inherit existing biases and vulnerability to adversarial attacks of such models.
To support responsible use, we encourage transparency around deployment contexts and recommend auditing tools to monitor for unintended outcomes. We release our benchmark and code to facilitate further research on both the benefits and limitations of attribute-foused retrieval.
34


Reviews:
[{"review_id": "qs91EkSelV", "reviewer": "Reviewer_tFqQ", "summary": "This paper introduces COCO-FACET, a benchmark for evaluating text-to-image retrievers on attribute-focused queries, revealing limitations in current approaches. The authors demonstrate that both CLIP-like and MLLM-based retrievers struggle with fine-grained visual attributes, particularly with non-dominant elements. As a solution, they propose using promptable image embeddings through MLLM-based universal embedders and present two acceleration strategies to make this approach more practical.", "questions": "The underlying principle of the Linear Approximation acceleration method requires clarification. In line 299, the authors propose using Wa for retrieval, which appears to rely solely on image embeddings without a prompt. Then what is the fundamental distinction between W and standard CLIP encoders? More specifically, while W is learned to approximate the nonlinear transformation U(a,p) for a specific prompt p, it's counterintuitive that this linear operation can generalize to highlight specific attributes across the entire image pool for different queries without the prompt being present during retrieval. Further explanation of how this approximation preserves attribute-specific information would strengthen the paper.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\n1. Well-constructed benchmark: The COCO-FACET dataset is methodically structured with 9,112 queries across 8 attribute types, providing a valuable resource for the community to evaluate attribute-focused retrieval capabilities.\n\n2. Comprehensive evaluation: The paper presents thorough benchmarking across 12 state-of-the-art retrieval models with varying architectures and scales, offering a clear picture of current limitations in the field.\n\n3. Effective acceleration strategies: The proposed acceleration methods demonstrate meaningful performance improvements.\n\n## Weaknesses\n\n1. Limited technical novelty: The core approach of using MLLMs to obtain multimodal representations has been explored in previous work. The paper's main contribution appears to be the benchmark itself rather than a novel technical solution to the identified problem.\n\n2. Efficiency concerns remain: While acceleration strategies are proposed, the fundamental challenge of computational cost in promptable image embedding approaches persists. The vanilla approach requires N×M forward passes, which remains impractical for large-scale retrieval systems.\n\n3. Conflicting motivations: There is a conflict between the stated goal of improving retrieval for uncommon attributes and the design of the pre-processing embeddings acceleration method, which requires predefined attribute types. This approach only works for a finite set of predetermined attributes, contradicting the motivation to handle diverse, unexpected attribute queries.", "was_revised": true, "final_justification": "After reviewing the authors' rebuttal to other reviewers, I find that their detailed explanation regarding computational overhead addresses most of my concerns. In addition, I would recommend that the authors include more intuitive explanations or visualizations in the appendix to facilitate easier understanding of the proposed Linear Approximation acceleration method for readers. I finally decided to increase my rating to 4.", "rebuttal": "Thank you for taking the time to review our paper, and we appreciate your comments and suggestions! We address your concerns as follows:\n\n1. **Your concern on our technical novelty, as using MLLMs to obtain multimodal representations has been explored in previous work.**\n    \n    We agree that multimodal representations have been explored previously, but within a limited scope. As stated in lines 95-100 in the Related Work section, previous practices either use multimodal embeddings for encoding given queries that have both image and text (e.g., composed image retrieval like CIRR [1]), or limit the prompts to be domain-specific (e.g., \"News, Miscellaneous, and Fashion\" in MM-Embed [2]) and hard-coded for a benchmark. \n    \n    We are the first to systematically study this method for attribute-focused embedding of target images by using various GPT-written prompts generated through a pipeline, and we reveal its effectiveness, even for some uncommon, challenging attributes. Moreover, the prompts are applied in a zero-shot way, compared with the improvement shown through domain-specific finetuning in MM-Embed [2]. We further proposed two acceleration approaches, showing a wider scope of its application.\n    \n2. **Your concern on the efficiency of our promptable approach.**\n    \n    While the vanilla approach requires high computational cost (N×M forward passes), the two acceleration strategies are much more efficient. The pre-processing strategy has the same total per-query cost as the no-prompt approach. For the linear approximation strategy, we only recompute promptable embeddings for K randomly selected images, resulting in a per-query embedding cost of $KF + F$ (lines 305–306), which does not scale with the index size $N$. The cosine similarity search over all $N$ embeddings in the pool still costs $ND$ for exact search (e.g., FAISS IndexFlatIP). Thus, the total per-query cost is $KF + F + ND$, compared to $F + ND$ of the baselines. This $ND$ search term dominates when $N$ is large, which is common in real-world usage.\n    \n3. **Your concern about the conflict between our motivations and the design of the pre-processing embeddings acceleration method.**\n\n    We agree that the pre-processing acceleration method assume a finite attribute vocabulary, which does limit support for entirely novel and highly specific attributes (e.g., \"Find me an image with its top-right part in yellow\"). This motivated us to design the fallback strategy (on-the-fly linear approximation) for this case.\n\n    Still, the goal of improving retrieval for uncommon attributes holds: we show that MLLM-based retrievers can encode rare attributes more effectively when prompted, and our benchmark enables measuring this. The pre-computing acceleration strategy is a practical compromise to improve scalability for pre-specified, commonly-used attributes, while preserving fallback support for rare or novel ones. Additionally, in lines 286-288 and Table 5, we observe that using the exact ground-truth prompt is often not necessary, as \"Count of People\" prompt works well on many testcases in \"Gesture\", showing applicability to unseen but similar attributes.\n\n4. **Your question about the underlying principle of the Linear Approximation acceleration method.**\n\n    First, we did not argue that this linear operation could recover the full function of the vanilla approach. In lines 308-311, we mentioned that it has limited expressiveness and cannot capture the complex cross-modal interaction happening in the original MLLM-based embedder. \n    \n    Second, from the modeling perspective, the linear approximation approach has the same structure with the standard CLIP encoder. Nonetheless, our results show that it can be attribute-focused, analogous to a CLIP model finetuned on attribute-specific data. While our approach only uses K random images from the pool without their labels or actual model training, finetuning CLIP usually requires a large number of images along with captions in the target domain. \n    \n    We agree that some visualization like heatmaps showing image focus would help the reader understand the approach better. We will add such visualization in the Appendix.\n\nIf you have further questions or concerns, we would be happy to address them as well!\n\n[1] Liu et al., Image Retrieval on Real-life Images with Pre-trained Vision-and-Language Models, ICCV 2021.\n\n[2] Lin et al., MM-Embed: Universal Multimodal Retrieval with Multimodal LLMs, ICLR 2025."}, {"review_id": "3U0KV40Kb2", "reviewer": "Reviewer_mpkJ", "summary": "The paper presents a new benchmark called COCO-Facet which focuses on fine-grained attributes for text-image retrieval tasks. The retrieval task aims at difficult categories such as gestures, times, material, and counts, among others, revealing that current VLM embedding models struggle with identifying these attributes reliably. Finally, the paper proposes to use MLLM embedding models with promptable embeddings to improve the retrieval performance significantly and offers insights about accelerating the inference speed of these large models.", "questions": "1. In Sec. 5.1, how does the performance change if you do not assume knowledge about the categories, i.e., the prompt set is disjoint from the GT categories.\n2. How do you choose the (a,b) pairs for Sec. 5.2? What happens if the categories for which K samples are available is disjoint from the inference categories, i.e., novel tasks/categories at test time?\n3. From the cited related work, it seems that FLAIR [2] is the closest to a promptable embedding model that is not based on an MLLM. Since it is concurrent work, I do not view it negatively that it is missing from the comparison, but it would certainly be interesting to see how it compares in Tab. 1 and/or Tab. 2. However, it does not seem to be available in a comparable model size and is a smaller model.\n\n[2] Xiao et al., FLAIR: VLM with Fine-grained Language-informed Image Representations, CVPR 2025", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "### Strengths\n- The proposed COCO-Facet dataset is well motivated, tackles an important problem of fine-grained attribute retrieval, and facilitates further development of stronger embedding models.\n- The quantitative evaluation is extensive, covering a wide range of VLMs, both native embedding models and MLLM adapted embedding models.\n- Employing promptable models is a reasonable step towards solving this task. The paper also addresses the inference speed issue by providing two possible solutions with their own trade-offs: pre-processing embeddings and linear approximation.\n\n### Weaknesses\n- **Missing comparison to existing benchmarks**\n  - The paper does not justify the introduction of this new dataset with respect to existing ones. For instance, MMVP [1] contains similar categories such as counts, color/appearance, structural characteristics, etc. A discussion around the new aspects this dataset introduces compared to existing benchmarks is important for judging its value.\n  - Comparing the proposed method improvements on related datasets could further reveal if the behavior of the improvements is similar or whether one dataset is more challenging than the other.\n- **Methods should be evaluated zero-shot**\n  - The proposed method make a lot of assumptions about prior knowledge about the inference task which is often not the case in real-world scenarios. For instance, the pre-processed embeddings contain the GT best-case prompt as part of the prompt set simplifying the problem significantly. Moreover, it seems the category of the retrieval query needs to be known at test time for the promptable embedding solutions.\n  - A more realistic setting would either test the pre-processed embeddings from COCO-Facet on another dataset, or the other way around, possibly even creating a uninformed prompt set independently.\n- **Technical contributions are limited**\n  - Since the main contribution is the dataset, I see this as a minor weakness. Nonetheless, promptable embedding are a key feature of the MLLM embedding models, although Sec. 4 makes it sounds like something innovative.\n  - Sec. 5.2 is an interesting approach, but lacks in experimental details and clarity. It seems the approach requires a base dataset of K samples per category but it is not explained how this set is chosen and how one would know the category of the inference sample. Related to the previous point, in realistic applications, training and test set might not come from the same distribution.\n\n[1] Tong et al., Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs, CVPR 2024", "was_revised": true, "final_justification": "The rebuttal revealed some valuable insights about the importance of prior knowledge for the proposed prompting strategy.\nIn the \"disjoint\" setting the performance gains are not nearly as strong and sometimes even weaker than without a prompt.\nAs a results, the proposed method is not as strong as the paper might make it appear to be.\n\nI believe it is important to discuss these findings as part of a limitation of the proposed method and direction for future work to close the gap between the zero-shot setting and with prior knowledge.\n\nIf this important findings is not transparently communicated and disclosed in the paper, I don't think it should be published, so I maintain my initially positive rating under the expectation that the authors incorporate this findings into the paper.", "rebuttal": "Thank you for taking the time to review our paper, and we appreciate your feedback and suggestions! We address your concerns as follows:\n\n1. **Your concern on the comparison of COCO-Facet to existing benchmarks.**\n\n    First, compared with commonly used text-to-image retrieval benchmarks (e.g., COCO and Flickr30K), COCO-Facet contains queries on uncommon or non-dominant attributes, which is our motivation for constructing it (described in the beginning of Section 3, lines 113-117). \n    \n    Second, compared with MMVP [1], COCO-Facet differs in motivation, task definition, and scale. COCO-Facet evaluates how well current models handle attribute-focused **text-to-image retrieval** queries, with 100 candidate images for each (random guess would get 1\\%). MMVP focuses on fine-grained **image-text matching**, where there are only two images and two queries per testcase for evaluating \"whether the VLM can distinguish two visually similar images\" (random guess would get 25\\%). Due to this difference, the model performance on them cannot be translated into each other directly. Moreover, COCO-Facet is on a scale 30x larger (9,112 queries to 270 questions in MMVP-VLM) which is thus less sensitive to noise or random fluctuations.\n    \n    Nonetheless, we show that the improvement of the promptable approach generalizes to the image-text matching task in MMVP benchmark. We use the given questions of MMVP in the prompt for the promptable approach. The average performance of the promptable approach is 44.4, which is higher than the best result reported in their paper (39.3 by DFN ViT-H-14). However, due to the different nature of MMVP and COCO-Facet, we cannot draw the conclusion about their relative difficulty based on the relative improvement.\n    \n    *Table 1: Results of CLIP-ViT-L/14-336px (for comparison), VLM2Vec-Phi-3.5-V with and without prompt on MMVP-VLM.*\n\n    |  | Orientation & Direction |Presence of Specific Features |State & Condition |Quantity & Count| Positional & Relational Context | Color & Appearance |Structural & Physical Characteristics |Texts | Viewpoint & Perspective| MMVP Avg. |\n    | --- |:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|\n    | CLIP-ViT-L/14-336px | 0.0 | 20.0 | 40.0 | 20.0 | 6.7 | 20.0 | 33.3 | 6.7 | **33.3** | 20.0|\n    | VLM2Vec-Phi-3.5-V |6.7 |13.3|26.7|33.3|6.7|**60.0**| 20.0|26.7 |6.7|22.2  | \n    | w/ MMVP Question as Prompt  | **26.7** | **46.7** |**60.0**|**73.3** |**26.7** |**60.0** |**40.0** |**40.0**| 26.7|**44.4** | \n\n2. **Your concern on assumptions about prior knowledge about the inference task.**\n\n    We agree that pre-defined attributes do limit support for entirely novel and highly specific attributes, but still they show applicability to unseen but similar attributes. In lines 286-288 and Table 5, we observe that choosing the exact ground-truth prompt is often not necessary, as \"Count of People\" prompt works well on many test cases in \"Gesture\". \n\n    We test the experimental design you proposed: we reevaluate the pre-processing strategy on COCO-Facet, but randomly split the categories into two parts, limiting the prompt set to be four random prompts from the original eight and testing on the other four categories. This setting leads to a performance drop compared to the vanilla approach with ground-truth prompts, but its Recall@1 exceeds the baseline, and the Recall@5 is on par with the baseline. Thus, for cases where the prompt is highly novel and only known at test time, we recommend using the linear approximation as a fallback option.\n    \n    *Table 2: Recall@1 of selecting pre-processed promptable image embeddings of four prompts and using image embeddings with or without prompt (baseline) on VLM2Vec-Phi-3.5-V in percentage points on COCO-Facet.*\n    | |Animals | Scenes | Materials | Gestures | \n    | ---- |:-----:|:-----:|:-----:|:-----:|\n    | No Prompt | 95.5 | 69.8 | 6.3  | 9.8  | \n    | w/ GT Prompt | 90.7 | **81.4** | **25.8**  | **15.7**  | \n    | Disjoint Prompt Set | **97.0** |  71.5 |  8.1  | 11.4  | \n\n    *Table 3: Recall@5 of selecting pre-processed promptable image embeddings of four prompts and using image embeddings with or without prompt (baseline) on VLM2Vec-Phi-3.5-V in percentage points on COCO-Facet.*\n    | |Animals | Scenes  | Materials| Gestures | \n    | ---- |:---:|:----:|:---:|:---:|\n    | No Prompt | **99.3**| 90.7   | 18.2   | 27.1  |\n    | w/ GT Prompt | 98.7 | **95.9** | **48.8**  | **39.3**  | \n    | Disjoint Prompt Set  | **99.3** |90.1 | **19.5** | 25.4| \n    \n    \n    \n3. **Your concern on the technical contributions of the promptable embedding approach, and details about the Linear Approximation.**\n\n    Although MLLM-based embedders were designed and used by many previous papers, with some trying hand-written, domain-specific (news and fashion) prompts and reporting improvements, we are the first to systematically study this approach for attribute-focused embedding of target images using a pipeline of GPT-generated prompts, demonstrating its effectiveness even for some uncommon and challenging attributes. \n    \n    For the Linear Approximation approach, we sampled the set of K images uniformly from the pool of candidates of all queries in the category (much larger than the candidate pool of one query). We test the case when the categories of available samples are disjoint from the inference categories by randomly choosing another category and then uniformly sampling K=100 samples from its pool. In Table 2 and 3, we observed that using samples disjoint from the inference category does not render the method completely ineffective, and still outperforms the baseline at Recall@5. \n    \n    *Table 4: Recall@1 of Linear Approximation on K=100 samples from another random category (disjoint), default Linear Approximation, and without prompts on VLM2Vec-Phi-3.5-V in percentage points on COCO-Facet. Results in the last two rows are averaged over five independent runs.*\n    | |Animals | Scenes | Objects | Count of People | Materials | Times | Weathers | Gestures | Avg.|\n    | ---- |:----:|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|\n    | No Prompt | **95.5** | **69.8**  |  **74.4** | 14.4 | 6.3  | 5.5 | 4.3 | 9.8  | **44.5**|\n    | Default |  72.1 | 67.2  | 57.1  |**47.5**  | **24.3**  | **35.7** | **9.0** | **14.2**  | 42.5|\n    | Disjoint | 56.9 | 62.4   | 54.8 |37.7 |  20.4 |23.2 | 5.4 | 8.4 | 37.4 |\n\n    *Table 5: Recall@5 of Linear Approximation on K=100 samples from another random category (disjoint), default Linear Approximation, and without prompts on VLM2Vec-Phi-3.5-V in percentage points on COCO-Facet. Results in the last two rows are averaged over five independent runs.*\n    | |Animals | Scenes | Objects | Count of People | Materials | Times | Weathers | Gestures | Avg.|\n    | ---- |:---:|:----:|:----:|:----:|:----:|:---:|:---:|:---:|:---:|\n    | No Prompt | **99.3** | 90.7  | **90.7**  | 36.6 | 18.2  |12.9  |19.2  | 27.1  |58.9 |\n    | Default | 84.6 | **91.9**  | 83.2  |**73.7**  | **43.9**  | **71.5** | **28.4** | **38.1**  |**67.0** |\n    | Disjoint  | 75.8 | 89.5 | 83.8  |63.7  |  41.6 | 60.9 |22.2 | 27.0 | 61.4 |\n    \n4. **Your question on the performance change if no prior knowledge about the categories is given, i.e., the prompt set is disjoint from the GT categories.**\n\n    Please refer to the response and results above (in 2).\n    \n5. **Your question on the K samples' choice and whether the approach works for novel tasks at test time in Section 5.2.**\n\n    Please refer to the response and results above (in 3).\n\n6. **Your question on FLAIR's performance on COCO-Facet.**\n\n    We attach the results for FLAIR [2] as follows. We also evaluated CLIP-ViT-B/16 in the same scale as this smaller model. We found that FLAIR offers little improvement, suggesting it struggles to focus on the required attribute. This is probably due to its single patch-level alignment, while a concept might align better with several patches jointly.\n    \n    *Table 6: Recall@1 of FLAIR and CLIP-ViT-B/16 (baseline) in percentage points on COCO-Facet.*\n    | |Animals | Scenes | Objects | Count of People | Materials | Times | Weathers | Gestures | Avg.|\n    | ---- |:----:|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|\n    | CLIP-ViT-B/16 | 94.0 |  44.2 |  57.4  | 1.1  |  1.6  | 3.8 | 3.3 |  9.6 | 35.0|\n    | FLAIR | 82.2 | 52.9 | 54.5  |  1.2  |  3.0  | 6.3  | 4.6 | 6.3 | 33.0|\n\n    *Table 7: Recall@5 of FLAIR and CLIP-ViT-B/16 (baseline) in percentage points on COCO-Facet.*\n    | |Animals | Scenes | Objects | Count of People | Materials | Times | Weathers | Gestures | Avg.|\n    | ---- |:----:|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|\n    | CLIP-ViT-B/16 | 99.1| 75.6  |  77.6  | 14.7 |  10.7  | 17.4 |  12.4  | 20.9 |49.8 |\n    | FLAIR | 97.1 | 81.4 |  80.2  |  9.7  |  10.4  |11.8  | 17.9  | 19.5 |50.3 |\n\nIf you have further questions or concerns, we would be happy to address them!\n\n[1] Tong et al., Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs, CVPR 2024.\n\n[2] Xiao et al., FLAIR: VLM with Fine-grained Language-informed Image Representations, CVPR 2025."}, {"review_id": "a8GUtstoy4", "reviewer": "Reviewer_Lakj", "summary": "This paper focuses on the attribute-focused T2I retrieval task, where queries emphasize specific attributes of the target image. Motivated by this setup, the authors introduce the COCO-FACET benchmark, built upon the COCO dataset, to evaluate performance in this setting. They also propose a promptable embedding method with MLLM model. I believe the motivation is strong and dataset can be a contribution, but there are some concerns on assumptions that used in method", "questions": "I wrote them above", "limitations": "I wrote them above", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n1. The targeted problem (attribute-focused) T2I task seems important and reasonable.\n2. The suggested dataset seems reasonable and constructed carefully to cope with intended goal. \n3. The paper is easy to follow and the related works are clearly explained.\n4. The baselines are fairly well compared with the recent ones. \n\n\n**Weaknesses**\n1. **Scalability and Assumptions of the Method**\nA key concern lies in the computational assumptions underlying the proposed method. As the authors themselves note, the promptable embeddings need to be recomputed for the entire gallery set for each query—if this is indeed how the original implementation works, it would be infeasible for large-scale gallery sets common in real-world applications. Even with the proposed linear approximation, the computational cost remains significantly higher than traditional T2I retrieval methods—potentially up to 100× more expensive for K=100.\nFurthermore, both acceleration strategies rely on assumptions that may not hold in practice, such as prior knowledge of the target category or the ability to pre-filter gallery images by category. These assumptions undermine the method’s applicability in realistic settings. The paper would benefit from an alternative approach that avoids recomputing gallery representations for each query. I think the current direction for the method would not be feasible in even a moderate size index set (gallery) setting.\n\n2. **Dataset Visualization and figure**\nMore concrete visualizations of the dataset would improve clarity and help readers better understand the proposed benchmark. Additionally, Figure 1 is potentially misleading, as the query text appears to be embedded within the gallery set, which may confuse readers about the retrieval setup.", "was_revised": true, "final_justification": "I carefully read all the responses and reviews. My main concern was the computational overhead that this method possibly has, but the rebuttal has largely resolved my concern on this, so I will raise my point.", "rebuttal": "Thank you for taking the time to review our paper, and we appreciate your feedback and suggestions! First of all, we would like to clarify a possible misunderstanding in your review regarding our approach: \n\n- *\"Even with the proposed linear approximation, the computational cost remains significantly higher than traditional T2I retrieval methods—potentially up to 100× more expensive for K=100 ... The paper would benefit from an alternative approach that avoids recomputing gallery representations for each query. I think the current direction for the method would not be feasible in even a moderate size index set (gallery) setting.\"* \n\n   The linear approximation is not 100× more expensive than prior methods for K=100 in terms of total per-query cost. For the approximation, we only need to recalculate the promptable embeddings of K images randomly selected from the pool, so the per-query embedding cost is $KF + F$ (lines 305-306), which **does not grow with the index set size $N$**. Besides the embedding cost, we need to perform cosine similarity search over the entire index set, which costs $N \\times D$ for $D$-dim embeddings in the exact search setting (FAISS's IndexFlatIP). Hence, the per-query embedding cost is not always the bottleneck in the total per-query cost. Mathematically, the new total per-query cost is $KF+F+ND$ versus prior $F+ND$, which is not 100x more expensive, especially when the index set size $N$ is very large (a common case in a realistic setting).\n    \nThen we address your concerns and respond to your suggestions as follows.\n\n1. **Your concern about the assumptions of the two acceleration strategies that may not hold in practice.**\n\n    For the pre-processing approach, we do not assume that the target category is known exactly, since we include GPT-4o for prompt selection at test time with the template attached in Appendix B.4. On the other hand, we agree that pre-defined attributes might not work for entirely novel and highly specific attributes disjoint from the prompt set, but still they show applicability to unseen but similar attributes. In lines 286-288 and Table 5, we observe that using the exact ground-truth prompt is often not necessary, as \"Count of People\" prompt works well on many test cases in \"Gesture\".\n    \n    As a fallback strategy for novel attributes, the on-the-fly linear approximation approach samples the set of K images uniformly from the candidate pool across all queries in the category (much larger than the candidate pool of one query). But it is not necessary to sample the images from the same category (or \"pre-filter gallery images by category\" as you mentioned) to achieve better performance on COCO-Facet. To illustrate this point, we show the results of sampling K images from a randomly chosen category other than the target one. In Table 1 and 2, we found that this does not lead to complete failure, and outperforms the baseline at Recall@5. \n    \n    *Table 1: Recall@1 of Linear Approximation on K=100 samples from another random category (disjoint), default Linear Approximation, and without prompts on VLM2Vec-Phi-3.5-V in percentage points on COCO-Facet. Results in the last two rows are averaged over five independent runs.*\n    | |Animals | Scenes | Objects | Count of People | Materials | Times | Weathers | Gestures | Avg.|\n    | ---- |:----:|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|\n    | No Prompt | **95.5** | **69.8**  |  **74.4** | 14.4 | 6.3  | 5.5 | 4.3 | 9.8  | **44.5**|\n    | Default |  72.1 | 67.2  | 57.1  |**47.5**  | **24.3**  | **35.7** | **9.0** | **14.2**  | 42.5|\n    | Disjoint | 56.9 | 62.4   | 54.8 |37.7 |  20.4 |23.2 | 5.4 | 8.4 | 37.4 |\n\n    *Table 2: Recall@5 of Linear Approximation on K=100 samples from another random category (disjoint), default Linear Approximation, and without prompts on VLM2Vec-Phi-3.5-V in percentage points on COCO-Facet. Results in the last two rows are averaged over five independent runs.*\n    | |Animals | Scenes | Objects | Count of People | Materials | Times | Weathers | Gestures | Avg.|\n    | ---- |:---:|:----:|:----:|:----:|:----:|:---:|:---:|:---:|:---:|\n    | No Prompt | **99.3** | 90.7  | **90.7**  | 36.6 | 18.2  |12.9  |19.2  | 27.1  |58.9 |\n    | Default | 84.6 | **91.9**  | 83.2  |**73.7**  | **43.9**  | **71.5** | **28.4** | **38.1**  |**67.0** |\n    | Disjoint  | 75.8 | 89.5 | 83.8  |63.7  |  41.6 | 60.9 |22.2 | 27.0 | 61.4 |\n\n2. **Your suggestion on the clarity of Figure 1.**\n\n    Thank you for your suggestion! For visualization of the benchmark, we included some examples from the benchmark and failure analysis in Figure 4 of Appendix D. For Figure 1, we will replace the bottom-left text \"Image\" with multiple images representing the gallery set to help readers better understand our approach.\n\nIf you have further questions or concerns, we would be happy to address them!"}, {"review_id": "2GHiapt2VU", "reviewer": "Reviewer_roin", "summary": "This paper introduces promptable image embeddings to enhance attribute-focused text-to-image (T2I) retrieval. The authors propose a new benchmark, COCO-FACET, and show that existing CLIP-like and multimodal retrieval models perform suboptimally on fine-grained attribute queries. They demonstrate that conditioning image embeddings on attribute-specific prompts substantially improves retrieval accuracy. Additionally, two acceleration strategies are presented to facilitate practical deployment.", "questions": "Please refer to the section outlining the weaknesses of the paper. I look forward to the authors' responses to these questions.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Pros:\n1. The authors clearly identify the limitations of global image embeddings and propose a well-motivated approach, which is highly relevant for T2I retrieval tasks.\n2. The introduction of the COCO-FACET benchmark is a valuable contribution, as it encourages further research into the retrieval of fine-grained visual attributes.\n\nCons:\n1. The proposed method is somewhat similar to previous works, such as CLOC and FLAIR, which also generate task-specific embeddings. The authors should more explicitly clarify the distinctions and advantages of their approach over existing methods.\n2. The method utilizes GPT-4o to generate prompts that guide image embeddings. However, since GPT-4o is itself a state-of-the-art multimodal large language model, there is a concern that GPT-4o alone might already solve the retrieval task independently, without the need for additional models like VLM2Vec. The paper does not evaluate this possibility or clarify whether GPT-4o serves solely as a prompt generator or is effectively part of the retrieval pipeline.\n3. The inclusion of acceleration strategies is a positive aspect. However, the paper lacks a detailed comparison of the actual inference time between the proposed approach and prior methods. Providing such a comparison would enhance the practical significance of the work. \n4. While the authors state that their method can handle attributes such as weather and time, supported by experimental results, the paper does not clearly explain how promptable embeddings capture these global or abstract concepts. A more detailed explanation or visualization of how these attributes are encoded, as well as an analysis of failure cases, would strengthen this point.", "was_revised": true, "final_justification": "The response addresses most of my concerns. Hence, I have raised my score to 4.", "rebuttal": "Thank you for taking the time to review our paper, and we appreciate your feedback and suggestions! First of all, we would like to clarify a possible misunderstanding in your review:\n\n- *\"However, since GPT-4o is itself a state-of-the-art multimodal large language model, there is a concern that GPT-4o alone might already solve the retrieval task independently, without the need for additional models like VLM2Vec.\"* \n\n    GPT-4o is used only as the prompt generator (and prompt selector in the pre-processing approach) without access to any image in our pipeline. It can be replaced by pure-language models with instruction-following ability. Hence, there is no such possibility that it is the actual solver of the retrieval task, instead of the promptable retriever. \n    \nThen we address your concerns as follows:\n\n1. **Your concern on the distinctions and advantages of our approach over existing methods (CLOC and FLAIR).**\n\n    Both CLOC [1] and FLAIR [2] have constraints on the image focus given prompts. CLOC proposed the idea of promptable image embeddings, but limited its image focus to be a rectangular region (bounding box). As introduced in their Section 3.2, their prompt is either a bounding box or text, and the text-prompted image embedding is only trained with a single bounding box prediction. This limited image focus is similar to the zooming or cropping approach, which would not work well on non-region-based attributes like Time and Scene, as we mentioned in lines 108-111. FLAIR relaxed this constraint by patch-level image-text alignment during training but is restricted to single-patch alignment, while non-region-based attributes might require a joint match with several patches (e.g., Count of People). \n    \n    To illustrate the distinction, we evaluated FLAIR on our COCO-Facet benchmark (since CLOC is not open-sourced and FLAIR is more flexible with the image focus due to its objective). As the promptable retrievers in the paper are not on the same scale, we evaluated the CLIP-ViT-B/16 model on the same scale with FLAIR for fair comparison. We attached the results in Table 1 and Table 2. The improvement brought by FLAIR on top of CLIP is not significant or consistent on the COCO-Facet benchmark, and overall it still has poor performance.\n    \n    *Table 1: Recall@1 of FLAIR and CLIP-ViT-B/16 (baseline) in percentage points on COCO-Facet.*\n    | |Animals | Scenes | Objects | Count of People | Materials | Times | Weathers | Gestures | Avg.|\n    | ---- |:----:|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|\n    | CLIP-ViT-B/16 | 94.0 |  44.2 |  57.4  | 1.1  |  1.6  | 3.8 | 3.3 |  9.6 | 35.0|\n    | FLAIR | 82.2 | 52.9 | 54.5  |  1.2  |  3.0  | 6.3  | 4.6 | 6.3 | 33.0|\n\n    *Table 2: Recall@5 of FLAIR and CLIP-ViT-B/16 (baseline) in percentage points on COCO-Facet.*\n    | |Animals | Scenes | Objects | Count of People | Materials | Times | Weathers | Gestures | Avg.|\n    | ---- |:----:|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|\n    | CLIP-ViT-B/16 | 99.1| 75.6  |  77.6  | 14.7 |  10.7  | 17.4 |  12.4  | 20.9 |49.8 |\n    | FLAIR | 97.1 | 81.4 |  80.2  |  9.7  |  10.4  |11.8  | 17.9  | 19.5 |50.3 |\n    \n2. **Your concern on the possibility that GPT-4o might already solve the task.**\n\n    As we clarified above, the GPT-4o only functions as the prompt generator (and prompt selector in the pre-processing approach) without looking at any image, and we attach the prompts it generated in Table 10 for your reference. Therefore, the improvement is brought by the promptable image embeddings derived by the retrievers instead of the multimodal LLM as the prompt generator.\n    \n3. **Your concern on the actual inference time between the proposed approach and prior methods.**\n\n    We thank the reviewer for the suggestion of reporting the actual inference time among the methods. We averaged the inference time over queries from the Times category on a single A6000 GPU. Note that COCO-Facet has a small image pool size (N=100) per query for fast evaluation, and thus K=100 resulted in the same processing time. So we added the setting of K=10 to reflect the difference:\n    \n    *Table 3: Average inference time of VLM2Vec-Phi-3.5-V without prompt, with prompt, and with linear approximation (K=10).*\n    |  | Avg. Per-Query Time| \n    | ---- |:---:|\n    | w/o Prompt |0.04s | \n    | w/ Prompt | 17.2s | \n    | w/ Linear Approximation (K=10) | 6.8s |\n    \n    We can see that the Linear Approximation allows acceleration when $K<N$. We will include the table in the Appendix.\n    \n4. **Your question on the explanation or visualization of how these attributes are encoded, as well as an analysis of failure cases.**\n\n    Since the MLLM-based universal embedders are previously shown to be capable of encoding question/instruction+image for composed image retrieval (e.g., CIRR [3], the query text being \"Show three bottles of soft drink\", the query image being one bottle as an example, and the target image being three bottles in the same style), they integrate the textual and visual information in a similar way that the original MLLM did in QA tasks, instead of doing direct addition. Hence, when we use attribute-focused questions as text inputs, we expect that the output embedding would be attribute-focused image embeddings.\n    \n    We agree that a visualization would help the reader understand the promptable approach better. We have attached some case analysis of COCO-Facet in Figure 4 of Appendix D. Following your suggestion, we will include some failure case analysis in the Appendix to better demonstrate the scope of our approach.\n    \nIf you have further questions or concerns, we would be happy to address them!\n\n[1] Chen et al., Contrastive Localized Language-Image Pre-Training, arXiv 2024.\n\n[2] Xiao et al., FLAIR: VLM with Fine-grained Language-informed Image Representations, CVPR 2025.\n\n[3] Liu et al., Image Retrieval on Real-life Images with Pre-trained Vision-and-Language Models, ICCV 2021."}]

Author rebuttal:

Meta-review: The paper was reviewed by four experts and received lukewarm reviews, albeit leaning towards recommending acceptance. Several of the identified weaknesses were addressed in the authors' response to the satisfaction of the reviewers. These clarifications and the many other suggestions from the reviewers should be considered as the authors prepare a final revision of the paper. However, on the balance the Area Chair believes that the paper (with the promised revisions) meets the bar for acceptance.
Decision: Accept (poster)