VIRTUE: Visual-Interactive Text-Image Universal Embedder
Authors: Wei-Yao Wang
Kazuya Tateishi
Qiyu Wu
Shusuke Takahashi
Yuki Mitsufuji
Venue: ICLR 2026 Poster (2026)
Keywords: Visual-interactive embedding model
VLM-based representation learning
interactive image-to-text retrieval benchmark
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: Multimodal representation learning models have demonstrated successful operation across complex tasks, and the integration of vision-language models (VLMs) has further enabled embedding models with instruction-following capabilities. However, existing embedding models lack visual-interactive capabilities to specify regions of interests from users (e.g., point, bounding box, mask), which have been explored in generative models to broaden their human-interactive applicability. Equipping embedding models with visual interactions not only would unlock new applications with localized grounding of user intent, which remains unexplored, but also enable the models to learn entity-level information within images to complement their global representations for conventional embedding tasks. In this paper, we propose a novel **V**isual-**I**nte**R**active **T**ext-Image **U**niversal **E**mbedder (**VIRTUE**) that extends the capabilities of the segmentation model and the vision-language model to the realm of representation learning. In VIRTUE, the segmentation model can process visual prompts that pinpoint specific regions within an image, thereby enabling the embedder to handle complex and ambiguous scenarios more precisely. To evaluate the visual-interaction ability of VIRTUE, we introduce a large-scale **S**egmentation-and-Scene **Ca**ption **R**etrieval (**SCaR**) benchmark comprising 1M samples that aims to retrieve the text caption by jointly considering the entity with a specific object and image scene. VIRTUE consistently achieves a state-of-the-art performance with significant improvements across 36 universal MMEB (**3.1\%–8.5\%**) and five visual-interactive SCaR (**15.2\%–20.3\%**) tasks. The code, models, and benchmarks are available at https://github.com/sony/virtue.

Published as a conference paper at ICLR 2026
VIRTUE: VISUAL-INTERACTIVE TEXT-IMAGE UNI- VERSAL EMBEDDER
Wei-Yao Wang** Kazuya Tateishi Qiyu W Shusuke Takahashi Yuki Mitsufuji Sony Group Corporation, * Sony AI [first name. last name ) wesossyy.oommm com com
ht
ABSTRACT
Multimodal representation learning models have demonstrated successful opera- tion across complex tasks, and the integration of vision-language models (VLMs) has further enabled embedding models with instruction-followii capabilities. However, existing embedding models lack visual-interactive capabilities to spec- ify regions of interest from users (e.g.. point, bounding box, mask), which have been explored in generative models to broaden their human-interactivee applicabil- ity. Equipping embedding models with visual interactions not only would un- lock new applications with localized grounding of user intent, which remains unexplored, but also enable the models to learn entity-level information within images complement their global representations for conventional embedding tasks. In this paper, we propose a novel Intuual-eteR Text- Image Universal Embedder (VIRTUE) that extends the capabilities of the segmentation model and the vision-language model to the realm of representation learning. In VIRTUE, the segmentation model can process visual prompts that pinpoint specific regions within an image. thereby enabling the embedder to handle complex and am- biguous scenarios more precisely. To evaluate the visual-interaction ability of VIRTUE, we introduce a large-scale Segmentation-and-S5 (SCaR) benchmark comprising 1M samples that aims retrieve the text cap- tion by jointly considering the entity with a specific object and image scene. VIRTUE consistently achieves a state-of-the-art performance with significant im- provements across 36 universal MMEB (3. (%5%% and five vistal-interactive SCaR https://gitubb.oo//oon//iii
Caption Retrieval
tasks. The code, models, and benchmarks are available at
INTRODUCTION
Embedding models have recently transitioned from two-tower architectures (e.g., CLIP (Radford al. )021). BLIP (Li al. (2022)), SigLIP (Zhai et al. 2023)) which have been used for embedding- based evaluation (Girdaar& et al. 2023) and cross-modal similarity matching (Hao et al.) langugle model (VLM)-based frameworks (e.g.. GME (Zhang et al. 2025), LamRA (Liu et al., 2025bl) owing to VLMss ability to ingest arbitrary combinations of textual and visual inputs into single embedding space. Thanks their inherent instruction- following capabilities, adopting VLMs as embedding models generalizes effectively across wide range of zero-shot multimodal reasoning applications, including interactive information retrieval Jiang al. 2025b) and augmeeted generation (Liu et al. 2025a)
2023 Han al. 2024),
Although VLM-based embedding models support interactive use, they rely on text as the primary ouman-machine interaction modality. In contrast, visual prompts, which have recently attracted attention in generative applications (You et al.] 2024; Yuan et al. 2024; Lian et al. 2025). serve important but overlooked interaction channel. Visual prompting can not only enhance the downstream generation performance (Li et al. (0226) but also provide precise spatial localization for fine- graingrineed understanding (Liu et al. 2022c).. This is particularly advantageous for embedding- based tasks, as it allows models to respond to visual inputs from the user beyond traditional global matching by capturing entity-level cues, thereby improving retrieval precision and alignment while complementing global representations.
Project Lead.

---

Published as a conference paper at ICLR 2026
Interactive 121
Interactive 127
cup with lemonade-colored liquid on a table with desserts
cup filled with water table with desserts
with lemonade-colored liquid
filled with water in the living room
in the kitchen
Figure I: iisual-interactive paradigms for image-to-image (12I) with masks assuming candidate images contain only dogs or only cats across different scenes, and image-to mn-ge-to-text (I2T) with bounding boxes. False retrievals occur when retrieved content does not match the query's 's scene context.
Considering visual-interactive image-to-image (12I) and image-to-text (I2T) retrieval scenarios, as shown in Fig. [ where user aims retrieve different entities within the same image but under shared global context, current embedding models rely solely on holistic image representations and fail to leverage explicit visual interactive signals (e.g., bounding boxes, points, and masks provided by users). As result, they cannot isolate and retrieve the targeted entity while maintaining aware- ness of the broader scene (e.g., "grass" for the 12I scenario and "on a table with desserts" for the 12T retrieval). One possible strategy is to convert visual prompts into textual descriptions to guide retrieval; however, embedding models are not trained with spatially grounded supervision, which limits their ability to generalize such interactive tasks. Another intuitive approach is to crop the region of interest (Subramanian et al. 2022). which can improve fine- eeeigraieed understanding but sacrifices global contextual cues for compositional reasoning (e.g., understanding an object within the full scene as presented in Appendix [E.4.]]. This limitation gives rise to a central challenge: How can visual interaction capabilities be incorporated into embedding models, and how can we systematically evaluate their compositional reasoning on targeted image regions?
In this paper. we propose VIRTUE, visual-interactive text-image universal embedder that com- bines an off-the-shelf segmentation model (SAM2 (Ravi et al.) (025)) with a pretrained VLM to jointly encode entity- and global-level information from images and the textual descriptions. For insual-nterrcttive scenarios, VIRTUE processes user-provided visual prompts by the prompt encoder within the segmentation model; for con-interactive scenarios, the prompt encoder fed uniformly sampled points to produce a feature map composed of multiple entity-level information. The VLM then ingests arbitrary combinations of image and text embeddings, where each image embedding comprises both an entity-level embedding (from the segmentation model) and global image em- bedding (from the VLM's vision encoder), and produces a single unified embedding for contrastive learning. In this manner, VIRTUE enables training on visual-interactin and www-vvuull-ntteacctee data and supports antity- retrieval while preserving global scene context.
Since existing benchmark evaluates visual-interactive embedding capabilities, we introduce SCaR, a large-scale Segmentation-and-SSee Caption Retrieval benchmark for visual-interactive image-to-text retrieval. In SCaR, image together with region of interest serves the task is to retrieve the caption that describes the specified object in its global scene context. We constructed SCaR from five publicly available datasets: ReiCOCO+ (Yu et al. 2016). RefCOCOg
query, and
Mao et al. (2016). VisualGenome (Krishna et al., 2017). COCO-Stulf (Caesar et al. 2018). and
ADE20k (Zhou et al., (017). The annotations include images, bounding boxes, and captions that describe entities, relations, and the global scene context. To increase difficulties in reasoning, neg- ative distractors are generated by replacing one of three elements of the ground-truth caption via prompting GPT-4V (OpenAl 2023) instead of random sampling; for datasets that lack human cap- tions (c.g., ADE20k), we generated ground-truth captions via carefully designed prompts to GPT-4V. To this end, SCaR comprises vast collection of 1M samples that are divided into training and val- idation sets. A distinguishing characteristic of the proposed SCaR dataset is its ability to evaluate not only visual-interactive reasoning but also compositional scenarios, requiring models to perform fine-grained, context-aware cross-modal reasoning that goes beyond global image matching.
In summary, contributions are three-fold:
Method Novelty: We propose VIRTUE, a visual- intuallitteacttve text-image universal embedder con- sisting of VL.M and a segmentation model to enable the visual interaction modality for human
2

---

Published as a conference paper at ICLR 2026
interactions. The segmentation model allows users to optionally provide different types of visual prompts via its prompt encoder and reinforces VIRTUE to capture entity-level representations in addition to global context. We also conduct systematic analysis of visual prompt integration to provide guidelines for enabling visual-interactive capabilities in embedding models.
Benchmark Novelty: As there is no public visual-interactive embedding benchmark, intro- duce SCaR, composed of IM samples for visual-interactive image-to-text retrieval, to evaluate VIRTUE's capabilities SCaR enables evaluation of advanced reasoning and compositional tasks in multimodal, Experiment Novelty: VIRTUE outperforms state-of-the-art embedding models on 36 MMEB tasks with significant gains from 3.1 1% to 8.5% and achieves improvements of 15.2% to 20.3% on five SCaR tasks, showing that equipping embedding models with visual-interactive cappiiaaaiteeccaaiiiiiiiiie benefits both visual-interactive and non-visual-inee interrattee scenarios.
embedding scenarios that remain unexplored.
2 RELATED WORKS
Multimodal Representation Learning. Early progress in text-image representation learning was driven by two-tower contrastive models that learn a joint embedding space by aligning an image encoder with text encoder (e.g., CLIP (Radford et al. 2021). BLIP (Li et al. (022a), SigLIP (Zhai et al. 2023), OpenCIIP (Cherti et al., 2023)). These models provide effective global image- text matching and have served as foundation models for building vision-language models (VLMs) that tackle zero-shot downstream tasks (Tong et al. 2024) Wang et al., (0256) Deitke et al.) 2025). Subsequent work has advanced embedding performance along various dimensions. For instance, UnilR (Wei et al. 2024) finetunes CLIP/BIIP with the late fusion text and image embeddings, and UNIME (Gu al.] 2025) distills text knowledge from an LLM followed by two-stage nega- tives contrastive learning. Magiclens (Zhang et al. 2024) incorporates open-ended instructions into dual-encoder architectures with training on large-scale instruction datasets. More recently, VLMs that accept arbitrary mixtures of visual and textual inputs and are trained with instruction-style ob- jectives have emerged as flexible and unified embedding providers that better perform on multi- modal reasoning and compositional queries, c.g., ES-V (Jiang et al. 2024]. VLM2 (Jiang et al.) 20256), GME (Zhang et al. 2025). and LamRA (Liu al. 20256). Despite these advancements, existing embedding models only support textual instructions and lack native support for direct vi- sual prompts: they are typically trained only on holistic image-text alignment (Oh et al., 2024). In contrast, VIRTUE integrates segmentation model with a pretrained VLM to fuse segmentation- derived, prompp-conditioned entity representations with global representations, producing unified embeddings that are both entity- and context-preserving newly constructed SCaR.
evaluated on both MMEB and our
Interactive Embedding Benchmarks. Since embedding models have shifted from uni-modal matching benchmarks (e.g., BEIR (Thakur et al.] 2021), MTEB (Muennighotf et al., 2023 ) to
instruction-based cross-modal global matching, recent studies have introduced Instruction-based
multimodal benchmarks to probe the reasoning abilities of embedding models. M-BEIR (Wei et al.]
2024) is a multimodal retrieval benchmark encompassing eight tasks from diverse domains using text instructions, while MMEB (Jiang et al. 20255) extends the evaluation to 36 multimodal datasets covering classification, VQA, retrieval, and grounding tasks to assess cnstuuction-oolloiinn across multifaceted perspectives. While we utilize MMEB to evaluate our proposed method for universal embedding abilities, these benchmarks focus on text-based instructions and do not evaluate visual- interactive scenarios in which visual prompts are provided as inputs. To fill this gap, we introduce SCaR, large-scale interactive image-to-text retrieval benchmark where each query consists of an image well as target bounding box, and the task is to retrieve captions that describe the specified entity within its global scene context. SCaR composed of five visual-grone gruua-ggounding and referring- expression datasets that test caption retrieval for region- region-in-contee in-context, with negative distractors gen- crated by GPT-4V to stress-test entity-in snnitt--i--ooneee in-context discrimination beyond the simple random negatives used in MMEB
3 SCAR: SIGMENTATION--ND-SCHI CAPTION RETRIEVAL BENCHMARK
3.1 SCAR OVERVIEW
Current publicly available benchmarks primarily evaluate text instruction mostruction-oollovingg capabilities for embedding models. Although MMEB contains oft-of-dommin visual grounding tasks for RefCOCO (Kazemzadeh et al. 2014), it simplifies them by cropping the specified region the target, thereby
3

---

Published as a conference paper at ICLR 2026
Input Sumple
Does the caprion Include 38 lements snljettboreaatooossseeeo
No
Generate the misting elements for the caption
Generatt segathee condidates by replacing elements from caption
Human Inspection
Ground truth counter?
salad
LiM-laaed Edttring H-WordNet constraines)
Scene swap
salad fork wooden deck
salad fork under the
fork belanced the edga the the Sable
Relation
salad fork tucked beside
tabto
swap
butter knite tab
Caption: saled
glass lable
table
Object swap
The data collection pipeline to build SCaR. We adopt GPT-4V to generate missing elements for the ground-truth caption as well as negative candidates. Collected samples (left) are filtered via AaM-then-human inspection (right) to ensure quality. Each SCaR sample contains an image with a bounding box, one ground-trume caption, and nine distractors.
Figure 2:
neglecting the broader scene context within an image. Consequently, existing embedding models struggle with inputs that include visual regions of interest in visual-interactive retrieval tasks. To address this limitation, we introduce @SCRR, a sewpueettttinaand-ccet caption retrieval benchmark that challenges models with reasoning and compositionality for text caption retrieval based on given image and a specified region. SCaR comprises images, segmentations with bounding boxes, and text captions from five public datasets: RefCOCO+ (Yu et al., 2 RefCOCOg (Mao et al., 2016), VisualGenome (Krishna et al. 2017). COCO-Stuff (Caesar et al. 2018), and ADE20k (Zhou al. 2017). Distinet from existing benchmarks, SCaR provides multiple negative candidates per sample, which are generated by GPT-4V (OpenAl (023) through element-swapping in the ground- truth caption with false replacements, forming a large-scale benchmark of 1M samples covering diverse applicability.
Task Definition. The main difference between conventional image-to-text retrieval (e.g., MSCOcO.i2 i2t in MMEB) and visual-interactivy image-to-text retrieval is the additional region-of-intent inpu Formally, given an input image I and bounding box P [Eimnn; Umin Twidth, Mheight with ten candidates C= [c].... ; c1o). the goal is to find the most relevant text caption lyt (rgmax(sim((bb)) P), (CC)),, where á the embedding model and sim denotes cosine similarity. Since SCaR requires models to reason about both the specified object and the broader scene, candidate captions are intentionally challenging and demand joint reasoning over fine-grann fine-grained bounding-box details and global image context.
3.2 COLLECTION PIPELINE
Fig. iillustrates the collection pipeline to construct SCaR, where all datasets are first converted to the COCO format to enable unified processing. To balance evaluation efficiency and coverage, we randomly sample up to five objects per image. For each sample, the image size, original caption, object category, and bounding box are provided in the prompt template, as shown in Fig. [ The prompt instructs GPT-4V to return a JSON object containing the ground-truth caption and nine negatives. Since some datasets lack complete descriptions in terms of <ob <object> <rela <scene> (e.g., ADE20k only provides object names, and some RefCOCO+ samples do not contain global scene context), the prompt asks GPT-4V to verify whether the caption contains all three elements, and to supplement any missing elements with careful and image-grounded descriptions.
Negative candidates are then generated via element-swapping from the gold caption. We define three swap strategies, each producing three candidates. 1) Global scene swap: Replace the scene
Both retrieval scenarios include the text instruction (the MMEB instruction is shown in Tab. 10 of Jiang et al. (2025b)). SCaR uses the following instruction: "Find the caption that best describes the segmented object, considering both local details and global context in the given image.". For simplicity, omit these instructions here.
While some datasets provide segmentation masks, we observed that GPT-4V struggles to interpret them re- liably. In contrast, bounding hoxes are easier for GPT-4V to parse and align with prior prompt-hased collection strategies (Wang al. 2025c Jiang et al. 2025a): therefore, we opt for bounding boxes for SCaR.
4

---

Published as a conference paper at ICLR 2026
Vicaa Prorupia
Segmentation Medel
HL
Zicioh s
@
Prompt Encoder
Dirmait
coine and
Mask Decoder
languaage Connectior
Enteading
Image Enooder
Large Language Model
Viiso languste Connectot
Vison Enceder
Nea
odject,
Learning
pana deving buth local detals contonts
Two zag one ast aiting un grany hold
Figure 3: Overview of VIRTUE. The framework trained with contrastive loss consists of a segmen- tation model, a segmentation-langul combinations of visual and textual inputs with optional visual prompt. If no prompt is provided, the model samples N points uniformly from the image to extract entity-level information.
connector (orange), and a VLM (blue). It supports arbitrary
phrase with a clearly distinct environment (e.g., from "table" to "pienic blanket"). 2) Relation swap: Modify the relation of the specified object by borrowing from nearby objects or introducing halluci- nated interactions (e.g., from "on the table" to "tucked beside the glass" or "under the napkin"). 3) Object swap: Substitute the target object with another object, either plausible but incorrect within the image or fabricated (e.g., from "salad fork" to "wine glass" or "butter knife"). In early trials, we observed that GPT-4V often generated ambiguous variants (e.g., "traffic light" vs. "stoplight", "zoo" vs. "safari park"), which could reduce the quality and reliability of the benchmark. To mit- igate this, we explicitly add constraints to the instruction requiring swapped objects and scenes to belong clearly distinct categories. We also encourage GPT-4V to produce diverse and creative negatives, ensuring coverage across different objects, relations, and scenes, while reducing the risk of overfitting to narrow patterns.
LLM--the-HHumal Inspection. Despite care- fully designed prompts for guiding GPT-4V, some deviations from established rules re- sult in the generation of subpar ground-truth and candidate generations, including generated ground-truth captions that lack scene context and negatives that involve synonymy or am- biguous objects and scenes. Therefore, we de- sign multi-stage filtering pipeline combining heuristic rules, L.Le- verification, and human inspection to improve dataset reliability and re- move unethical samples. After the collection, GPT-4V is guided to verify whether each ground-trute caption contains all three elements <object> <relation ion> <scene> and to output these el- ements in structured JSON format for both ground-truth and negative captions, shown in Fig. 6 Then, WordNet (Miller 1992) is applied to detect if any negative elements are synonyms of the corresponding gold elements. The sample is immediately discarded if any verification step fails. For the evaluation set, two independent human inspectors review all remaining samples with a focus on ambiguity, and remove any that are flagged by either inspector. For the training set, assess quality indirectly by training models on it and evaluating them on the SCaR evaluation set (Sec. (.3). This meticulous filtering regimen ensures the integrity and trustworthiness of the SCaR dataset, which comprises 957k training and 47k evaluation samples (see Tab. D.. More statistics and examples are provided in Appendix
Table 1: Dataset statistics of SCaR
Train
Evaluation
#lmages
#Annotations
#Images
#Amnotations
RcfCOCOg
21.730
40,674
1,300
1,539
ReICOCO+
16.847
1,500
2.764
COCO-Stutf
118,287
426,379
4,999
17,903
VisualGenome
86,414
357,583
5,000
15.571
ADE20K
20,210
94,271
2,000
9,368
Total
309,278
957,714
14.799
47.145
4 VIRTUE
As illustrated in Fig. 3 since our goal is to create a ceneral-purposs lisiun-language embedding model with the visual interaction capability, we propose VIRTUE, which consists of VLM that processes visual and textual inputs as vision and language embeddings, respectively, and a seg- mentation model along with segmentation-language connector that converts visual prompts (i.e.,
5

---

Published as a conference paper at ICLR 2026
bounding boxes, masks, points, or sampled points when not explicitly provided) into the segmenta- tion embeddings. Afterwards, the large language model (LLM) consumes the sequence of segmen- tation, vision, and language embeddings to generate the final input embedding using the final hidden state of the last token for contrastive learning.
4.1 CONVERTING VISUAL AND TEXTUAL INPUTS INTO EMBEDDINGS
Unlike the dual-tower structure of CLIP-based methods, VLMs incorporate vision encoder, vision-language connector, and an LLM, enabling inputs to be flexibly specified as unimodal (image or text) or bimodal (image-text pairs) within a shared representation space. The visual input streamline is processed by the vision encoder followed by the vision-language connector, both of which are components of the VLM, to produce |v| global context vision embeddings
H, E Blol-d = (h, h. Similarly, the textual streamline obtains |t] textual embeddings H, E Riddd (h.,...... ... ) through the text embedding layers of the LLM. To adapt the model to diverse downstream tasks, the query input further includes task definitions (e.g., "represent the given image with the following question") between the image and text, where the task-specific instructions
are drawn from MMEB (Jiang al., 20256) well as our proposed SCaR.
4.2 ENABLING VISUAL PROMPTS TO VLM EMBEDDERS
While existing VLM-based embedding models establish a general mechanism to accommodate di- verse modality combinations, they often struggle incorporate visual interaction prompts and may fail to capture fine-grained entity-level information. A common workaround is to crop bounding boxes, but cropping yields coarse rectangular regions and ineffective results (see Appendix [E.) that may include background, ignore inter-entity relations (c.g., "next to"), or cut across multiple enti- ties. To address these limitations, we propose the visual prompt streamline, which supports bound- ing boxes, clicks (points), and masks as inputs. The visual prompt is then processed with the given image by the segmentation model to produce a segmentation map, which is subsequently passed through our proposed segmentation-
connector to generate segmentation embeddings.
Segmentation Model. We adopt the pretrained SAM-2 Ravi at.. 2025) as the segmentation model within VIRTUE. A key advantage of using SAM-2 is that treating segmentation feature instead of relying on cropping provides a structured prior that aligns with human perception of discrete objects, producing features that more faithfully capture the semantics of the referenced entity. Specifically, SAM-2 includes a prompt encoder, an image encoder, and a mask decoder"] The image encoder processes the visual input produce unconditioned feature embeddings, whereas the prompt encoder accepts visual prompts to define the extent of the object in an image. To cover con- ventional www.-vu-uiintaaacttt
entity-level
scenarios (e.g., MMEB), set the visual prompt N uniformly sampled points when it is not explicitly provided to leverage SAM-2's inherent capability for au- tomatic segmentation. This serves as a surrogate for user interactions and provides fine-graing ane-grained entity-level cues in addition to the global context captured by the visual and textual streamlines. The mask decoder f takes the outputs of the prompt encoder Ep and the image encoder E, to produce a 64 x 64 segmentation feature map F. = (Ep(),, occlusion MLP heads, and upsampling process in SAM-2's mask decoder are discarded to utilize the segmentation feature map conditioned on visual prompts, since the segmentation feature map already encodes entity-level information from both the prompt and vision encoders; there is no need to merge multiple segmentation masks into a joint embedding space.
E, ET) 6.4-64.d The mask prediction, IoU,
Connector. While we utilize the segmentation feature map F, to avoid the extra overhead of converting reconstructed masks into embeddings, directly flattening F. results in sequence length of 4096, which cannot be feasibly processed when aligning to the LLM dimen- sion d due to GPU memory limitations. Therefore, we employ a 2D convolution layer Cone2D to compress the feature map from 4096 tokens to |S]. The resulting representations are then projected via two MLP layers: first into d, and then into the L.M''s hidden dimension d for joint learning as:
Segmentation-Langee
H. = M ML.P(Cono2((F.)) T
3 Visual prompts can he positivecnegattve for SAM-2, but we focus on positive prompts in this paper. We also omit the memory hank and memory attention designed for videos, as we focus on text and images.

---

Published as a conference paper at ICLR 2026
4.3 TRAINING SCHEMA
The segmentation Ha, vision H., and text H, embeddings are concatenated in the order of segmentation-vision-- and subsequently fed into the LLM generate the query embedding 2y or target embedding 2, from the final hidden state of the last token for contrastive learning. In this manner, the query embedding is encouraged to move closer to semantically similar targets while being pushed away from dissimilar ones, incorporating not only global matching signals but also entity-level information provided by the segmentation embeddings.
Contrastive Learning. Since VLMs are not originally tailored for representation learning, we adopt contrastive learning with the InfoNCE loss (van den Oord al. 2018) on query embeddings Zy and target embeddings each of which contains any combinations of Hs, He, and H,. Formally, InfoNCE is applied over in-batch negatives:
exp (sim(zg, 21)/T)
L = log
exp (sim(z,, (z. z/T/T) + Σ exp (sim(zg: 22)/T)
where D denotes the set of hard negatives and is temperature parameter. Following (Jiang al., 2025b), GradCache (Gao et al. 2021) is employed to enable larger batch sizes, thereby improving the generalizability of the learned embeddings by leveraging greater number of in-batch negatives.
EXPERIMENTS
5.1 EXPERIMENTAL SETUP
Implementation Details. We use Dwen2-VL-2B and Qwen2-VL-7B (Wang et al. 2024 as back- bone VLMs for VIRTUE-2B and VIRTUE-7B, both with sam2.1 1aw.1__hherr_bbbaa saueeppusei the segmenta- tion model. To adapt the pretrained VLMs for embedding tasks, we apply LoRA (Hu et al. 2022) to the LLM within VIRTUE following (Jiang al., (025b), while training the segmentation-languee connector from scratch. The segmentation embeddings are prepended to the inputs only when im- ages are provided as inputs. The segmentation model, vision encoder, and vision-language connec- tor are kept frozen to preserve the pretrained knowledge. VIRTUE is trained with 20 in-distribution MMEB-train datasets with batch size of 1024 and LoRA rank of Detailed settings are provided in Appendix C
with ablation and parameter studies in Appendices E. and]
We evaluate VIRTUE across 20 in-distribution test sets and dfitt--diit test sets from MMEB (Jiang et al. 2025b) to assess its universal instruction-followin embedding ca- pabilities across diverse classification. VQA, retrieval, and visual grounding scenarios. To examine the essual-nteractivvn capability, VIRTUE is evaluated on proposed SCaR benchmark compris- ing five datasets with both out-of-domain and in-domain performance. Consistent with the MMEB benchmark, report precision @ 1 across all experiments.
Benchmarks.
Baselines. We strive to provide a large number of comparisons against recent CLIP-basee and VLM-based families. As CLIP-based methods, we compare against CLIP (Radford et al.) 2021), BLIP2 (Li et al., (203), SigLIP (Zhai et al. (023), OpenCLIP (Cherti et al.] 2023). UnilR (Wei al. 2024), and Magiclens (Zhang al. 2024). As VLM-based methods, we compare against E (Jiagg et al., 2024). GME (Zhang et al. 2025). MMRet (Zhou et al. 2025), LamRA (Liu al. 2025b), VLM2 (Jiang 2025b), and UniME (Gu et al.) (025). For fair comparisons, results on MMEB are reported from their corresponding papers, and all models on SCaR, except for +SCaR-train, are used off-the-shelf with frozen weights. GME, LamRA, and VLM2Vec adopt
2B and 7B of Qwen2-VL. VL (Wang et al. 2024), E5-V uses VL-VA-N (Liu et al. 2024b), and MMRet and UniME use LLaVA-1.6-1 7-7 (Liu et al. 2024a).
5.2 COMPARISON ON MMEB FOR UNIVERSAL EMBEDDING TASKS
Tab. summarizes the overall performance of all methods, both with and without finetuning on MMEB-train Our VIRTUE family consistently outperforms all baselines across the four core meta- task scenarios, as well in both in-distribution (IND) and out-o--distiibution (OOD) settings. Quan-
https: com/

---

Published as a conference paper at ICLR 2026
Table 2: Results on MMEB. The scores are averaged per meta-task. The improvements are calcu- lated between VIRTUE and the best-performing 2B and 7B baselines. We highlight the best and second best number each column. Detailed scores are listed in Appendix E.7
Per Meta- Task Score
Average Score
Classification
VQA
Retrieval
Grounding
IND
0OD
Overall
Model
#Datasets -
10
10
12
4
20
16
36
w/o Finetuning on MMEB-
CLIP [Rdffor
51.8
37.1
38.7
BI.IP2 (f
AT2S
27.0
4.2
33.9
47.0
25.3
25.1
25.2
SigLIP 14:00.14:31 Zhai
2023
8.4
31.6
59,5
32.3
38.0
OpenCLIPuua (Cheiti
47.8
10.9
52.3
53.3
39.3
40.2
39.7
UnilRcuPe or (wereell 20/4
44.3
16.2
61.8
65.3
47.1
41.7
44.7
Magielenspu.is. (Zhmig
2024
38.8
8.3
35.4
26.0
31.0
23.7
27.8
GMEE2B (Zhane cl al 2025
29.9
66.9
55.5
49.2
55.2
51.9
"E3.V3BB Olang
21.3
3.9
T1.3
19.0
11.9
11.3
13.5
GMB-7B Zhang m 2025
$7.7
34.7
71.2
59.3
53.6
58.8
56.0
LamRA
[La al
26.5
70.0
62.7
53.0
55.4
w/ Finetuning MMEB-Train
CLIP
Radford et al 2021
55.2
19.7
$3.2
62.2
47.6
42.8
45.4
OpenCIIIPyy 11
Cherti
21.9
55.4
64.1
50.5
VL.M2Ne--2B 2B Tiang ct M125h
58.7
49.3
65.0
72.9
64.9
53.3
59.7
MMRct-7B
Zh30 d 2025
57.4
69.9
83.6
68.0
59.1
VeM2Vee-
7B Diang 2025b
62.7
56.9
69.4
82.2
21.4
58.1
65.5
UniME et 2025
60.6
$2.9
67.9
85.1
68.4
57.9
66.6
VIRTUE Finetuning on MMEB- Train
VIKTUE-2B
64.1
55.7
68.4
78.7
69.7
58.8
64.88
VIRTUE 7B
65.6
60.4
71.8
87.3
74.4
61.4
68.6
Improvements (2B)
+5.4
+6.4
+1.5
+5.8
+4.8
+3.6
+5.1
Improvements (7B)
+2.9
+3.0
+0.6
+2.2
+3.0
+2.6
+2.0
titatively, VIRTUE-2B achieves an average improvement of 5.1 points over CLIP--baed and other 2B models (from 59.7 64.8), while VIRTUE-7B surpasses existing 7B models with 2.0-point gain (from 66.6 to 68.6). While VLM-based models are superior to CLIP-based ones, VIRTUE significantly outperforms GME and LamRA with the same Qwen2-V 2L backbone, Ret and E5-V with different backbones across all meta-tasks, demonstrating the effectiveness and universality of VIRTUE. In addition, the comparisons between VIRTUE and VLM2 highlight the importance of incorporating entity-level information, since both models adopt the same training schema and data. The of uniformly sampled points for non-visual-interattiv tasks signifies that segmentation embeddings contribute positively to universal embedding performance by enriching global context with tinegrrained object-level details. Compared to UniME-7B, VIRTUE-7B sur- passes all meta-tasks scenarios, underscoring the generalized embedder ability of VIRTUE, which jointly integrates global context and segmentation-deriver context with in-batch negatives.
well MM-
5.3 COMPARISON ON SCAR FOR VISUAL-INTERACTI TASKS
To examine the visual-interactive capabilities, VIRTUE is compared with CLIP well the crop- based (ReCLIP (Subramanian et al., 2022)) and seuaal-hintingbbseee (explicitly add red circles to images (Shtedritski 2003). denoted +Red Circle) variants. VLM-based embedding models are compared with and without finetuning on MMEB-train due to their effectiveness and task- following abilities. Since none of the baselines naturally accept bounding boxes visual inputs, they are textualized as Referring object bbox: {bbox), including after queries, while VIRTUE takes not only the textualized input but also visual prompts. In addition, we further conduct experiments by naively cropping the specified object using the corresponding bounding box as the visual input for models (+Cropping); in this case, no textualized bounding box is required. To push the limits of visual-interacto capabilities and for fair comparisons, further finetune VIRTUE, VLM2 MMRet, and UniME on the MMEB-tran checkpoints for Ik steps using a batch size of 1024 and a learning rate of 2e-6, following the aforementioned formats, which denoted +SCaR-treet -aain..
Tab. reports the results under both oft-of- and in-domain (i.e., ASCaR-train scenarios. Without finetuning MMEB-trait CLIP outperforms VLM-based models. In contrast, simply adding visual hints, which was observed effectively in (Shtedritski et al. 2023), performs slightly worse than CLIP, likely due to the more challenging nature of SCaR, where captions emphasize rea-
8

---

Published as a conference paper at ICLR 2026
Table 3: Results proposed SCaR benchmark. All models incorporate bounding boxes in the
textual prompt. +Cropping: Use only the cropped region of the image based on the given bounding box imput. +SCaR-train: Further finetune Ik steps with the SCaR training
Model Datasets RefCOCO+
RefCOCOg
VisunlGenome
COCO-Stuff
ADE20K
Overall
w/o Finetuning on MMEB-Train
CLIP
18.1
22.4
23.0
15.5
18.9
19.6
ReCLIP
17.6
23.0
21.3
12.6
15.9
18.1
Red Circle
17.6
21.6
22.3
15.1
18.4
19.0
GME-2B
8.3
7.5
5.8
6.2
6.6
+Cropping
6.1
8.3
5.8
6.2
7.5
6.8
GMB-7B
7.4
+Crropping
6.0
9.8
4.4
7.0
5.3
6.5
LamRA-7B
7.9
3.1
8.9
+Cropping
8.2
7.9
3.1
8.9
5.3
6.7
w/ Finctuning on MMEB- Train
VLM2Vee-2B
29,5
22.3
24.6
24.1
+Cropping
22.3
25.8
17.1
19.5
22.5
21.4
+SCaR-train
55.5
34.1
40.8
43.2
VIRTUE-2B (Ours)
28.8
42.4
24.4
29.9
27.5
30.4
+Crogping
24.4
36.3
14.2
25.8
23.4
24.8
+raaR-train
64.2
65.3
41.4
54.2
56.0
56.2
à (2B)
+4.3
+12.9
+2.1
+10.5
+2.9
+6.3
(2B, +SCaR-train)
+7.3
+13.4
+12.8
VLM2Vec-7B
23.2
29.1
14.7
25.0
22.4
22.9
+Cropping
17.6
25.7
15.2
13.8
21.3
18.7
+SCaR train
40.1
39.5
36.3
31.4
25.0
34.5
MMRes-7B
27.1
21.8
15.2
26.0
22.6
22.5
+Cropping
26.0
20.2
14.9
19.1
19.0
+SCaR-train
45.8
43.6
39.7
27.5
31.2
37.6
UniME-7B
31.4
3288
19.0
25.3
23.0
26.3
+Cropping
25.8
24.0
15.2
24.6
22.3
22.4
trCaR-train
VIRTUFF-B (Ours)
31.0
35.3
19.6
27.1
23.8
27.8
+Cropping
27.1
31.5
17.4
21.8
22.8
+SCaR-train
63.2
66.1
48.2
52.2
54,9
56,9
(7B)
+1.6
+2.5
+0.6
+1.8
+0.8
+1.5
& (7B, +SCaR train)
+5,3
+6.8
+3.7
+10.3
+11.3
+7.5
soning and compositionality. VLM-based models without finetuning from MMEB-train (i.e., GME, LamRA) are more prone to underperform compared to models with finetuning from MMEB- (i.e.. VLM2Vec, MMRet, UniME), which be attributed to the visual grounding datasets within MMEB-train that crop the ground-truth region as the target. Meanwhile, our proposed VIRTUE improves by 6.3 and 1.5 points average for the 2B and 7B models, respectively, which implies that conditioning embeddings on visual prompts facilitates the learning of fine- grainer representa- tions for regions of interest. Although directly cropping the specified object enables fine-grained understanding, the detrimental effects of +Cropping across all models and ReCLIP verify that sac- rifices global scene information for compositional reasoning. The results of --SCRR-tran show that reasoning
further finetuning with our collected SCaR training set is able to boost risual-interace even for models that not originally support visual prompts. Nonetheless, VIRTUE, equipped with the segmentation streamline, leads to a larger performance gain of 9.5 points for the 2B model and 7.5 points for the model, which demonstrates the effectiveness SCaR-tran and incorporat- ing visual prompts for not only fine-grained information but also user-enabled visual interactions. We include additional reevaluations on MMEB and qualitative results on SCaR with SCaR-train in respectively. More application paradigms are presented in Appendix E.4
Appendices E.3 and E.5
5.4 IMPACTS OF VARIOUS VISUAL PROMPTS VIRTUE
To investigate the impacts of various visual prompts on VIRTUE, we analyze the robustness of VIRTUE-7B and VIRTUE--B-SCAR-IEE
under noisy and misaligned bounding boxes from the SCaR evaluation set. We further perform SCaR evaluations by replacing ground-truth bounding boxes with centroid points or random boxes, examining whether VIRTUE can effectively handle different types of interactive visual prompts for embedding generation. Finally, on MMEB, we study the effect of removing uniformly sampled points, either during inference only (w/o inference) or during both training and inference (w/o all), to understand how reliance on uniform point prompts influences performance.

---

Published as a conference paper at ICLR 2026
Performance
Performance
Performance
-00
WATUE-28
MRTUE-78
VIRTUE778
MITTEE 78 ScaN Train
VIRTUE 78 Trall
60
VITTU 78 Trai
01
20
20
80
146
10
Jitter Fraction (%)
Partial Masking Ratio (%)
Off-by-K Pixels
Figure 4:
Analysis of noise and misalignment in visual prompts on SCaR: jittered boxes (left),
partial masks (middle), and off-by-k pixels (right).
Table 4:
Analysis on SCaR using points and randomly generated bounding boxes.
Table 5: Effect of removing uniformly sampled points during inference (w/o inference) or both training and inference (w/o all) on MMEB. Results using uniform points are shown in bold.
VIRTUE-7B
VIRIUE-7B-SCaR-tminn
bbux
random
poins
bbox
random
points
SCaR
27.8
9.2
26.6
56.9
13.0
45.0
VIRTUE-7B
VIRTUE-7I-SCuRE
unifonn
Who inferenee
w/o
unifeem
wlo inference
Woo all
MMFR
68.6
24.1
65.5
66.8
41.9
43.4
Robustness to Noisy or Misaligned Visual Prompts. Fig. 4 Arreeents results with different fractions of jittered bounding boxes, varying mask ratios for partial boxes, and off-by-k pixels. Both VIRTUE- 7B and VIRTUE-BB-SCAR- demonstrating robustness of the segmentation streamline. On the other hand, using only 20% partial masks results for VIRTUE-T EE-SCaR-traa remains superior to baselines like LL.M2Vcc-7B-SCC train, suggesting that our model remains robust even when visual prompts provide limited coverage.
remain stable across different amounts of jitter and off-by-k offsets,
Effect of Using Points vs. (Random) Bounding Boxes, To understand if VIRTUE takes advantage of segmentation tokens, hypothesize that if the model effectively leverages segmentation tokens, randomly sampling visual prompts should degrade performance; if does not, performance should remain unchanged or improve (Makansi al. 2022 Wang al. 2025a). Tab. 4 shows the SCaR performance when bounding boxes (bbox), randomly sampled boxes (random), or centroid points (points) are adopted as visual prompts. Using points achieves performance comparable to bounding boxes for VIRTUE-7B, while VIRTUE-7B-SCA S--B--CaR-taaat exhibits a larger drop. indicating that fine- tuning on SCaR encourages a stronger reliance on bounding box prompts. The severe degradation with random boxes for both models further confirms that VIRTUE meaningfully uses the provided visual prompts when generating embeddings.
Impact of Removing Uniformly Sampled Points. Since VIRTUE incorporates uniformly sam- pled points as visual prompts for non-visuall-mteraativee scenarios, Tab. reportt results obtained by removing these points either only during inference (w/o inference) during both training and inference (w/o all). The performance declines in both settings indicate that uniform points help the segmentation streamline capture entity- structure, which complements global context and leads to more fine-grained embeddings.
CONCLUSION
In this paper, we proposed VIRTUE, novel visual-interactive text-image universal embedder for both instruction-followin and visual-interactive embedding scenarios. Distinct from existing mod- els that support only text as the interaction modality, VIRTUE equipped with pre-trained SAM2 as the segmentation model is able process visual prompts as inputs, enabling the model to jointly capture global context as well as entity-level information. We constructed SCaR, large-scale regmentation-and-scenn image-to-text retrieval benchmark, to assess VIRTUE's visual-interaction reasoning abilities, which consists of 1M samples with challenging negative candidates gener- ated by replacing elements of the ground-truth captions. Experiments on both MMEB and SCaR demonstrate that VIRTUE is consistently superior to state-of-the-art approaches by 3. 1%%..% and 15.2%-20.334 respectively. We believe VIRTUE serves as a generic framework for conventional instruction instructiion-ollloinng and iisuallinteractive embedding tasks, and that SCaR opens up new human- Al interaction opportunities for embedding models. We present further discussion of limitations and broader impacts in Appendices A and B
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENTS
We gratefully acknowledge the support of all those who contributed to this research. We thank Naofumi Akimoto and Toshimitsu Uesaka for their valuable feedback and insightful suggestions that helped refine this work.
ETHICS STATEMENTS
The SCaR dataset is constructed from publicly available datasets (RefCOCO+, RefCOCOg, Visu- alGenome, COCO-Stuff, and ADE20K), which mitigates potential privacy concerns and harmful content. The dataset is primarily generated using GPT-4V with subsequent human inspection to remove ethically problematic content. While GPT-4V may inherit biases from its training data, our prompt design explicitly encourages diverse and creative captions, and a comprehensive filtering pipeline is applied to examine the generated captions. This design reduces human annotation bias and avoids repetitive lexical overlap, as illustrated in Fig. 7 Moreover, although VIRTUE builds upon pre-trained VLM, is further fine-tuned into an embedding model, which significantly min- imizes the risk of harmful or copyrighted content generation while preserving the semantic richness needed for downstream tasks.
LLM Usage. In addition to GPT-4V used for building the SCaR benchmark, we use LLM solely for polishing the manuscript.
REPRODUCIBILITY STATEMENTS
The codebase, VIRTUE models, and SCaR benchmark samples are provided at https:/giitubb.oo//onn//t
to advance the community with the realm of interactive repre- sentation learning. and the experimental details as well as model configurations are summarized in Sec. [.] and Appendix C Evaluations the MMEB benchmark follows their official use, while SCaR is detailed in Sec. and Appendix D
REFERENCES
Holger Caesar, Jasper R. R. Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes in context. In CVPR, pp. 1209-1218. Computer Vision Foundation IEEE Computer Society, 2018.
Youg-You Chen, Zhengfeng Lai, Haotian Zhang, Xinze Wang, Marcin Eichner, Keen You, Meng Cao, Bowen Zhang, Yinfei Yang, and Zhe Gan. Contrastive localized language-image pre- training. In ICML. OpenReview.net, 2025.
Ho Kei Cheng, Masato Ishii, Akio Hayakawa, Takashi Shibuya, Alexander G. Schwing, and Yuki Mitsufuji. Mmaudio: Taming multimodal joint training for high-quality video-to-audie synthesis. In CVPR, pp. 28901-28911. Computer Vision Foundation IEEÉ, 2025.
Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gor- don, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. CVPR, pp. 2818-2829. IEEE, 2023.
Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Moham- madreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, Jiasen Lu. Taira Anderson. Erin Bransom, Kiana Ehsani, Huong Ngo. Yen-Sung Chen, Ajay Patel, Mark Yatskar, Chris Callison- Burch, Andrew Head, Rose Hendrix, Favyen Bastani, Eli VanderBilt, Nathan Lambert, Yvonne Chou, Arnavi Chheda, Jenna Sparks, Sam Skjonsberg, Michael Schmitz, Aaron Sarnat, Byron Bischoff, Pete Walsh, Chris Newell, Piper Wolters, Tanmay Gupta, Kuo-Hao Zeng. Jon Bor- chardt, Dirk Groeneveld, Crystal Nam. Sophie Lebrecht, Caitlin Wittif, Carissa Schoenick, Oscar Michel, Ranjay Krishna, Luca Weihs, Noah A. Smith, Hannaneh Hajishirzi, Ross B. Girshick, Ali Farhadi, and Aniruddha Kembhavi. Molmo and pixmo: Open weights and open data for state- of of-the-art Dision-languse models. CVPR, pp. 91-104. Computer Vision Foundation / IEEE, 2025.
11

---

Published as a conference paper at ICLR 2026
Robert M. French. Catastrophic forgetting in connectionist networks.
Trends in Cog-
nitive Sciences, 3(4):128-133 1999. ISSN 1364-6613. doi: htts://dii.or/// 1016//1364
URL https:// httpps://www... www.:..ceeneddiec...cmm wtt.tttccocsccieeieee
art.ccle/p11/1134661139999
Luyu Gao, Yunyi Zhang, Jiawei Han, and Jamie Callan. Scaling deep contrastive learning batch size under memory limited setup. In REC-ANNL@@ALL-JCNLLL pp. 316-321. Association for Computational Linguistics, 2021.
Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind one embedding space to bind them all. In CVPR, pp. 15180- 15190. IEEE, 2023.
Tiancheng Gu, Kaicheng Yang. Ziyong Feng. Xingjun Wang, Yanzhao Zhang, Dingkun Long, Yingda Chen, Weidong Cai, and Jiankang Deng. Breaking the modality barrier: Universal em- bedding learning with multimodal Ilms. CoRR, abs/2504. 17432,, 2025.
Haochen Han, Qinghua Zheng, Guang Dai, Minnan Luo, and Jingdong Wang. Learning to rematch mismatched pairs for robust cross-modal retrieval. In CVPR, pp.
IEEE, 2024.
Xiaoshuai Hao. Wanqian Zhang, Dayan Wu, Fei Zhu, and Bo Li. Dual alignment unsupervised domain adaptation for video-text retrivval. In CVPR, pp. 18962-18972. IEEE, 2023.
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. In EMNLP (I), pp. 7514-7528. Asso- ciation for Computational Linguistics, 2021.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In ICLR. OpenRe- view.net, 2022.
Chaoya Jiang, Yongrui Heng, Wei Ye, Han Yang, Haiyang Xu, Ming Yan, Ji Zhang, Fei Huang, and Shikun Zhang. Vlm-r3; Region recognition, reasoning, and refinement for enhanced multimodal chain-of-thought. CoRR, abs/2505.16192. 2025a.
Ting Jiang, Minghui Song, Zihan Zhang, Haizhen Huang, Weiwei Deng, Feng Sun, Qi Zhang, Deqing Wang, and Fuzhen Zhuang. E5-V: universal embeddings with multimodal large language models. CoRR, 2is/2407.11580, 2024.
Ziyan Jiang, Rui Meng, Xinyi Yang, Semih Yavuz, Yingbo Zhou, and Wenhu Chen. VIm2vec: Training laagun-language models for massive multimodal embedding tasks. In ICLR. OpenRe- view.net, 2025b.
Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara L. Berg. Referitgame: Referring to objects in photographs of natural scenes. In EMNLP, pp. 787-798. ACL, 2014.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Li Fei-Fei. Visual genome: Connecting language and vision using crowdsourced dense image annotations. Int. J. Comput. Vis., 123(1):32-73, 2017.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven C. H. Hoi. BLIP: bootstrapping language-image pre-training for unified vision-language understanding and generation. In ICML, volume 162 of Proceedings of Machine Learning Research, pp. 18888-12900 PMLR, 2022a.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. BLIP-2:
bootstrapping language- image pre-training with frozen image encoders and large language models. In ICML, volume 202 of Proceedings of Machine Learning Research, pp. 19730-19742. PMLR, 2023.
Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang. Chunyuan Li, Yiwu Zhong, Lijuan Wang. Lu Yuan, Lei Zhang, Jenq-Neng Hwang. Kai-Wei Chang, and Jianfeng Gao. Grounded language-image pre-training. In CVPR, pp. 10955-10965. IEFE, 2022b.
12

---

Published as a conference paper at ICLR 2026
Zhizhong Li and Derek Hoiem. Learning without forgetting. In ECCV (4), volume 9908 of Lecture Notes in Computer Science, pp. 614-629 Springer, 2016.
Long Lian, Yifan Ding, Yunhao Ge, Sifei Liu, Hanzi Mao, Boyi Li, Marco Pavone, Ming- Yu Liu, Trevor Darrell, Adam Yala, and Yin Cui. Describe anything: Detailed localized image and video captioning. CoRR, abs/2504.160777 2025.
Haotian Liu, Chunyuan Li. Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In CVPR, pp. 26286-26296 IEEE, 2024a.
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang. Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, oer, and world knowledge, January 2024b. URL https::////// // 30-
llavavvl... gi thub. camub..o//ba ht/p:///bol/////////////
Pei Liu, Xin Liu, Ruoyu Yao, Junming Liu, Siyuan Meng, Ding Wang, and Jun Ma. HM-RAG: hier- archical multi-agent multimodal retrieval augmented generation. CoRR, abs/2504. 12330, 2025a.
Shilong Liu, Zhaoyang Zeng. Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, and Lei Zhang. Grounding DINO: marrying DINO with grounded pre-training for open-set object detection. In ECCV (47), volume 15105 of Lecture Notes in Computer Science, pp. 38-55. Springer, 2024c.
Yikun Liu, Yajie Zhang, Jiayin Cai, Xiaolong Jiang, Yao Hu, Jiangchao Yao, Yanfeng Wang, and Weidi Xie. Lamra: Large multimodal model as your advanced retrieval assistant. In CVPR, pp. 4015-4025, Computer Vision Foundation IEEE, 2025b.
Osama Makansi, Julius von Kügelgen, Francesco Locatello, Peter Vincent Gehler, Dominik Janzing, Thomas Brox, and Bernhard Schölkopf. You mostly walk alone: Analyzing feature attribution in trajectory prediction. In ICLR. OpenReview.net, 2022.
Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L. Yuille, and Kevin Mur- phy. Generation and comprehension of unambiguous object descriptions. In CVPR, pp. 11-20. IEEE Computer Society, 2016.
George A. Miller. WORDNET: lexical database for english. In HLT. Morgan Kaufmann, 1992.
Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. MTEB: massive text embed- ding benchmark. In EACL, pp. 2006 2029. Association for Computational Linguistics, 2023.
Youngtaek Oh, Jae-Won Cho, Dong-Jin Kim, In So Kweon, and Junmo Kim. Preserving multi- modal capabilities of pre-trained vlms for improving vision-linguistic compositionality. In EMNLP, pp. 19060-19076. Association for Computational Linguistics, 2024.
OpenAl. Gpt-ov(ision) system card. 2023. URL httess///pp..emmattcccchoo.oeo CorpusID:263218033
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML, volume 139 of Proceedings of Machine Learning Research, pp. 8748-8763. PMLR, 2021.
Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloé Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Va- sudev Alwala, Nicolas Carion, Chao-Yuan Wu, Ross B. Girshick, Piotr Dollár, and Christoph Feichtenhofer. SAM 2: Segment anything in images and videos. In ICLR. OpenReview.net, 2025.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, pp. 10674-10685. IEEE, 2022.
Aleksandar Shtedritski. Christian Rupprecht, and Andrea Vedaldi. What does CLIP know about a red circle? visual prompt engineering for vims. In ICCV, pp. 11953-1 1963. IEFE, 2023.
13

---

Published as a conference paper at ICLR 2026
Sanjay Subramanian, William Merrill, Trevor Darrell, Matt Gardner, Sameer Singh, and Anna Rohrbach. Reclip: A strong zero-shot baseline for referring expression comprehension. In ACL Association for Computational Linguistics, 2022.
(1). pp.
Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. BEIR: A heterogeneous benchmark for zero-shot evaluation of information retrieval models. In NeurlPS Datasets and Benchmarks, 2021.
Peter Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Adithya Iyer. Sai Charitha Akula, Shusheng Yang, Jihan Yang, Manoj Middepogu, Ziteng Wang, Xichen Pan, Rob Fergus, Yann LeCun, and Saining Xie. Cambrian-1: A fully open, vision-centric exploration of multimodal Ilms. In NeurlPS, 2024.
Aäron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. CoRR, abs/1 007.03748, 2018.
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. CoRR, abs/2409.12191 2024.
Wei- Yao Wang, Wen-Chih Peng, and Wei Wang. Shuttleshap: A turn-based feature attribution approach for analyzing forecasting models badminton. In PAKDD (6), volume 15875 of Lecture Notes in Computer Science, pp. 444-456. Springer, 2025a.
Wei-Yao Wang, Zhao Wang, Helen Suzuki, and Yoshiyuki Kobayashi. Seeing is understand- ing: Unlocking causal attention into modality-mutual attention for multimodal 1lms. CoRR, abs/2503.02597. 2025b.
Xudong Wang, Shaolun Zhang, Shufan Li, Kehan Li, Konstantinos Kallidromitis, Yusuke Kato, Kazuki Kozuka, and Trevor Darrell. Seglim: Multi-round reasoning segmentation with large language models. In ICLR. OpenReview.net, 2025c.
Cong Wei, Yang Chen, Haonan Chen, Hexiang Hu, Ge Zhang, Jie Fu, Alan Ritter, and Wenhu Chen. Uniir: Training and benchmarking universal multimodal information retrievers. In ECCV (87), volume 15145 of Lecture Notes in Computer Science, pp. 387-404. Springer, 2024.
Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du Bowen Zhang, Zirui Wang, Liangliang Cao, Shih-Fu Chang, and Yinfei Yang. Ferret: Refer and ground anything anywhere at any granularity. In ICLR. OpenReview.net, 2024.
Licheng Yu, Patrick Poirson, Shan Yang, Alexander C. Berg, and Tamara L. Berg. Modeling context in referring expressions. In ECCV (2), volume 9906 of Lecture Notes in Computer Science, pp. 69-85. Springer, 2016.
Yugian Yuan, Wentong Li, Jian Liu, Dongqi Tang, Xinjie Luo, Chi Qin, Lei Zhang, and Jianke Zhu. Osprey: Pixel understanding with visual instruction tuning. In CVPR, pp. 28202-28211. IEEE, 2024.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training In ICCV, pp. 11941-11952. IEEE, 2023.
Kai Zhang, Yi Luan, Hexiang Hu, Kenton Lee, Siyuan Qiao, Wenhu Chen, Yu Su, and Ming-War Chang. Magiclens: Self sulfvsupe image retrieval with open-ended instructions. In ICML. OpenReview.net, 2024.
Xin Zhang, Yanzhao Zhang, Wen Xie, Mingxin Li. Ziqi Dai, Dingkun Long, Pengjun Xie, Meishan Zhang, Wenjie Li, and Min Zhang. Bridging modalities: Improving universal multimodal retrieval by multimodal large language models. In CVPR, pp. 9274-9285. Computer Vision Foundation / IEEE, 2025.
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torraba. Scene parsing through ADE20K dataset. In CVPR, pp. 5122-5130. IEEE Computer Society, 2017.
14

---

Published as a conference paper at ICLR 2026
Junjie Zhou, Yongping Xiong, Zheng Liu, Ze Liu, Shitao Xiao, Yueze Wang, Bo Zhao, Chen Jason Zhang, and Defu Lian. Megapairs: Massive data synthesis for universal multimodal retrieval. In ACL (I), pp. 19076-19095, Association for Computational Linguistics, 2025.
15

---

Published as a conference paper at ICLR 2026
In the Appendix, Sec. states the ethical consideration and limitations of this paper, Sec.
B
discusses broader impacts for this work, Sec. C summarizes the implementation configurations, and
Sec. D presents additional details for proposed SCaR, including the prompt templates (Secs.
[.2 and D.3 and additional statistics (Sec. D.4). We also include extensive experiments in Sec.
including an ablation study (Sec. [..T). hyperparameter study (Sec. [..2), reevaluation on M
with SCaR-train (Sec. EE.) application paradigms on enabling VIRTUE with visual interactions
(Sec. E.4), and qualitative results on SCaR (Secs.
E.5 and E.7
A LIMITATIONS AND FUTURE DIRECTIONS
While our approach represents significant step toward reinforcing visual-interactive capabilities in embedding models, the main limitation lies in training VIRTUE only with MMEB and SCaR, primarily due to computational constraints. Prior work (e.g., (Zhou et al.) (025)) has shown that incorporating more diverse datasets can further improve universality, suggesting that expanding our training sources would yield additional performance gains. Additionally, our evaluation emphasizes interactive image-to-tent to-text retrieval (via SCaR) as primary validation task. While this provides strong only cover interactive songe-to-imman retrieval as
evidence for the effectiveness of our approach, case studies in Sec. [..4I]] largely due to copyright and ethical concerns associated with constructing suitable benchmarks. Nevertheless, this remains an important and complementary dimension for assessing
embedding models. These limitations reflect our overarching objective: pave the way for embedding models that are not only text-interactive but also visually interactive, while maintaining a strong universal embedding performance. In future work, we aim to extend our framework with more advanced training strategies and diverse as well as multifaceted datasets, including safe and ethically curated setups for interactive image-to-image retrieval. to fully unlock the potential of visual- interriitere embedding models.
B BROADER IMPACTS
As VIRTUE introduces a new paradigm for visual- risual-interactive embedding models, it has the potential to serve as a multimodal encoder that naturally supports visual prompts for VLMMs in contrast to post-hoc finetuning approaches (e.g., (You et al., 2024: Lian et al.] (025)). It may also benefit the development of multimodal foundation models built on top of CLIP (e.g., video-text-to-audnn generation models (Cheng et al. 2025). text-to-image generation models (Rombach et al. 2022 given VIRTUE's superior performance over existing CLIP-based methods. Moreover, embedding- based evaluation methods such as CLIPScore (Hessel et al. (021) are widely used automatic and reference-fie free metrics for holistic image-text alignment. VIRTUE could advance this line of evaluation by serving as stronger embedding-bail standard: while CLIPScore represents subset functionality, VIRTUE further enables region-specific interactions in addition to capturing global context to compute text-image similarities.
C IMPLEMENTATION DETAILS
VIRTUE-2B and VIRTUE-7B are trained with the 7 of 0.02 and learning rate of 2e-5, and input resolution of 1344 x 1344. The lengths of vision and text embeddings default VLMs. Since |S] segmentation embeddings are prepended to the sequence, their attention masks and positional encodings are set sequentially, following the original VL. design. d, is 256 from SAM2.1's default configuration, and d is 1536 for 2B and 3584 for 7B from Qwen2-VL's default settings. For eval- uating MMEB, VIRTUE is trained with MMEB-train with 5k steps, and is further trained with Ik steps with SCaR-train for (+SCaR-train in Tab. 3 The VIRTUE 2B and 7B training was conducted on 8 H100 80GB, where MMEB -MMii took around 74 and 189 hours, respectively, and SCaR train took around 12 and 30 hours, respectively. The kernel size and stride of Conv2D, the number of sampled points N, and the number of segmentation embeddings S are empirically set to 4, 9, and 256, respectively. Detailed configurations are summarized in Tab.
16

---

Published as a conference paper at ICLR 2026
Table 6: Detailed configurations for VIRTUE 2B and 7B models.
Configurations
VIRTUE-2B
VIRTUE-7B
MMEB-Train
VLM
Qwen2- VL-2B
Qwen2 VL-7B
Sogmentation model
SAM2. ---hhhhr-bbas basra-baeeppe base-plus
Image resolution
1344x 1344
Begmentation-lanuuaee connector
Random initialized
LoRA rank
LoRA dropout
0.1
LoRA alpha
64
Temperature
0.02
Batch size
1024
Training steps
5000
Warmup steps
200
Learning rate
2e-5
d,
256
d
1536
3584
Sampled points N
Segmentation embeddings |
GPU
8xH100 80G
Precision
bf16
Optimizer
AdamW ( 0.9, 8g 0.999)
Training time
74 hours
189 hours
SCaR-Train
VLM
Checkpoint of VIRTUE-2B
Checkpoint of VIRUTE-7B
Segmentation model
I-M2.1-hicra- balueee--bb
Image resolution
1344 x344x1344 1344
commentation-langueen connector
Checkpoint of VIRTUE-2B
Checkpoint of VIRUTE-7B
LoRA rank
8
LoRA dropout
0.1
LoRA alpha
64
Temperature т
0.02
Batch size
1024
Training steps
1000
Warmup steps
100
Learning rate
2e-6
d,
256
1536
3584
Sampled points N # Segmentation embeddings [S]
9
256
GPU
8xH100 80G
Precision
bf16
Optimizer
Adam (8) = 0.9, 32 0.999)
Training time
12 hours
30 hours
D SCAR DETAILS
D.I DATASET DETAILS
We summarize the five datasets used for building SCaR below:
RefCOCO+ el. 2016) A large-scale dataset for referring expression comprehension and segmentation, containing around 45k expressions over 19k images from MS-COCO. Crucially, it prohibits location-based words (e.g., "left", "right"), forcing models rely on appearance and contextual reasoning-a more robust form of vision-language grounding that moves beyond simple spatial relationships. Its unique constraints make strong test of model's ability to truly understand an object's 's visual properties relative to its surroundings.
RefCOCOg (Mao et al. 2016): A referring expression dataset with around 50k expressions for around 30k images, collected with longer and more descriptive annotations. This rich
17

---

Published as a conference paper at ICLR 2026
Table 7: Inter-annotator agreement and the number (and proportion) of dropped samples from the original SCaR evaluation set. We report the average Cohen's 's Kappa between the two annotators.
Cohen's ' Kappa
Drop samples (both unnotators flagged) Drup samples (either annotator flagged)
0.89
4021 (7.7%)
1638 (3.1%)
linguistic detail makes a standard benchmark for evaluating fine-grained vision-language grounding, it requires models reason nuanced and complex natural language descriptions rather than just a few keywords.
VisualGenome (Krishna et al. 2017): A massive vision-language dataset that provides dense annotations for over 100k images, including objects, attributes, and region-level de- scriptions. Its most notable feature is the inclusion of scene graph annotations, which explicitly model relationships between objects. This structured data is invaluable for train- ing and evaluating models on compositional reasoning, enabling a deeper understanding of complex scenes beyond simple object detection.
COCO-Stuff (Caesar et al. 2018): An extension of the MS-COCO dataset with over 160k images annotated for "stuff" classes (e.g.. sky, grass) addition to the existing "thing" categories. This comprehensive annotation scheme makes it primary benchmark for mantic segmentation and tasks that require holistic scene understanding, as allows mod- els to learn the fine-gres ene-grained contextual relationships between foreground objects and their background environments.
ADE20K (Zhou al. 2017): A challenging scene parsing dataset with 20k images and 1,50 fine-grained semantic categories. The dataset's diversity, spanning a wide range of indoor and outdoor scenes, coupled with its high-quality pixel-level labels, makes it an essential standard for evaluating the performance of semantic segmentation and general scene understanding models.
D.2 PROMPT TEMPLATE FOR BUILDING SCAR
Fig. illustrates the detailed prompt template used for building the SCaR dataset (Sec. 3.2, where each sample varies in image size, caption, category, and bounding box (bbox) as provided in the original datasets. The ground-truth instruction guides GPT-4V to determine whether the three el- ements <objects <relations <scene> are satisfied; if any element is missing, GPT-4V required complete it. Subsequently, the negatives instruction directs GPT-4V to replace each element from the ground-truth caption. Notably, we explicitly include creativity and diversity in- structions to encourage GPT-4V generate more diverse and imaginative negatives, as visualized in the word clouds in Fig. 7 To ensure GPT-4V follows these instructions, provide example output at the end of the prompt.
D.3 PROMPT TEMPLATE FOR LLM-BASED FILTERING
Fig. shows the prompt for LLM--bssed filtering. Generally, adopt GPT-4V guide the data verification from multifaceted perspectives. If all conditions are passed, the output JSON object contains the status field with "passed" as well as the extracted objects, relations, and scenes for both ground-truth and negative captions, which are then passed on to WordNet to inspect again in terms of semantic differences.
D.4 ADDITIONAL. SCAR STATISTICS
We further analyze the composition of SCaR to better understand its benchmark characteristics. Tab. presents the inter-annotator agreement and the number of dropped samples in the SCaR evaluation set. The high agreement score indicates that the independent annotators were highly consistent in the dataset inspection process. After inspection, we removed approximately 1 11% of the samples: 7.7% were flagged by both annotators, and an additional 3.1 % were flagged by either annotator.
Fig. 7 summarizes detailed statistics including word clouds, annotation counts, and image distri- butions for both the training and evaluation splits. Attributed to the creative and diverse instruction
18

---

Published as a conference paper at ICLR 2026
Prompt Template for Collecting SCaR with GPT-4VV
You are un AI visual ussistant cupable of correctly unalyzing a single image. You receive specific object locations within the image, along with detailed coondinates.
co- ordinates are in the form of bounding boxes, represented as (x1. yl, x2, y2). These values correspond to the top left x, top left y. bottom right x. bottom right y. The height and width of the imuge you receive 427 and 640, respectively. The global caption, category names, and bounding box coordinates of objects are as follows:
caption: egg salad sandwich with an orange toothpick holding together.
category:
food
bbox: [135.57, 248.43, 157.89, 278.22)
Your job to output plain ISON only, with no Markdown or code fences, detining:
1. ground_truth
Full caption: If 'caption' already contains object, clear relation, and scene/context phrase (e:g, "Dog sleeping on the sofa") ) reuse verbutim.
"caption" contains an object and relation but no scene (e.g.. "Dog lying on the rug"). append concise scene descriptor observed in the image (e.g.. "Dog lying on the rug in the living room").
Relation-only:
Object-only: If 'caption' is just a bare object label (e.g.. "boat"). generate a concise jectt <relation> <scene>" description from the image:
1. Start with the exact object label.
Describe visible appearance or action (relation).
Add simple scene context based strictly on what you (scene).
Never rewrite or paraphrase a full or relation-only caption beyond adding the missing
scene; not invent new objects or actions.
negatives; array of exactly 9 caption objects, each with:
"text'; the caption string.
"type": one l'global..ontext', Buccground.relation "objoct.swap") J.
Use these rules for negatives:
-globul.context
Identify the scene phrase 'ground_rruth. Replace with different, clearly distinct
scene, not sibling near-synonym (e.g.. don't swap "200" with "safati park").
-Choose contexts that are plausible but semantically distant (c.g.. "in the kitchen" vs.
beach", not the playground" vs. "on the sports field")..
background relation
Keep the same main object and the scene phrase from 'ground "rround_rruhh.
CCaange its relation creative way way-feel free introduce novel interaction action, even if involves object or element not explicitly listed among the nearby objects. Ensure each relation is creative and diverse, and deliberately false when applied to the main object and its scene from given image.
cobject_swapp
obbject swap: swap out the object class a different one, keeping the same relation to the scene nearby object (e.g. "Chair with lamp between Avoid changes like "girl" to "woman", or "truffic light" to "stoplight". These are lexical variants, not distinct categories, Ensure the swapped object real-world valid and contextually plausible. Ensure swapped objects from clearly distinct category (not mere sibling classes or near synonyms).. Avoid replacing an object with another from the same fine-grained group (e.g.. don't swap "bowl" with "mug" or "plate"; "armchair" with "sofa")..
beds"). not replace
with synonyms or hyponymshhppmmmms
Vary the swapped classes avoid reusing "cat", "dog". or "chair" in every example.
Creativity & Diversity Constraint
Do only ensure uniqueness; actively push for variety within each negutive type so that your negatives are diverse and creative, forcing models to use richer context signals.
Make sure the 'negutives' contain exactly hard negutives for each type (' gllobal.coneex
"background_ relattoo'. sahject_swap') Ensure all negatives are challenging they should look plausible when cropping to the mask. but only resolvable with full-image context and mask identity (i.c.. a naive crop-only baseline would confuse with GT), Don't add comments in the JSON file.
Exumple (abbreviated) ourput:
"ground truth": "Traffic light above the crosswalk". "negutives": [
"text": "Thathic light the parking lot", "type": "global context",
"text": "Traffic light on the highway median". "type": "global context""
"lexi": "Trathic light next to the storefront", "type": "glohal__cnntet".
"text": "Traffic light by the bus stop". "type": "background relation".
"text": "Traffic light near the pedestrian crossing sign". "type": "background relation".
"text": "Trattic light beside the kiosk", "type": "hackground
"text": "Stop sign above the crosswalk" "type": "objecttswapp"
"text": "Street lamp above the crosswalk". "type": "object_swap".
"text": "Billboard above the crosswalk". "type": "wbjecttswapp"
Figure 5: The prompt template used for constructing our SCaR benchmark with GPT-4V where the text in red varies for each sample.
19

---

Published as a conference paper at ICLR 2026
Prompt Template for
Filtering
You are a highty-capable Al assistant designed for meticulous data verification. Your task is to analyze a single data sample from dataset generation pipeline. This sample consists of "ground-truth" caption and an array "negatives" captions, each with corresponding type. You will strictly adhere the following rules and provide structured JSON output. ## Input Dara:
You will receive single ISON object containing a ground-truth string and a negatives array. ## Verification Rules:
For each data sample. you must perform the following checks:
Ground Truth Caption Verification:
Structure Check: The ground-truth caption must verifiable having three-part
structure: <ab sobject> <rel aon <scene>:
<ob ject>;
The main noun object of the caption.
"relation>: A verb or prepositional phrase describing the object's 's action or its
position relative to the scene or other ohjects.
<scene>; broader context or location where the object and relation take place. Example: For "Traffic light above the crosswalk" the parts are:
<ab ject>; "Traffic light"
<relation>: "above"
<<scene>; "the crosswalk"
Output: If the structure is correct, extract these three elements. If not, mark the sample
as "failed".
2. Negative Captions Verification:
Count Check: There must be exactly negatives in the negatives array.
Tyype Check: The 9 negatives must be split exactly as 3 'global context', 3 "background relation', and 3 'object swap'. No other lypes are allowed.
Redundancy Check: Within each negative type, the captions must unique. No two 'global context' captions can he identical, no two "hackground relation" captions can be identical, and two 'object swap' captions be identical.
3. Cross-Sample Verification (Crucial for filtering):
Global Context Negative Check:
Each "global context' negative must have the exact same <object> and <relation> as the ground-truth caption.
'global context' negative he semantically distinet and synonym or near-synonym of the ground-truth <acene>. For example, "in the kitchen" is distinet from "on the beach". but "zoo" is not distinct from "safari park".
-The <scene>
Background Relation Negative Check:
Each 'background relation' negative must have the exact sume cobject> and <scene> as the ground-truth caption.
the "background relation' negative must semantically dis- tinct and not synonym of the ground-truth <relation>.
-The <relat
Object Swap Negative Check:
-Each 'object swap' negative must have the exact same <relat i ion> and <scene>
as the ground truth caption.
'object swap' negative must be different, distinct object class from the ground-truth <ob ject>. must not a synonym, hypunym, or hypemymm. example, "Traffic light" and "Stoplight" are invalid swaps. "Bowl" and "Mug" invalid swaps. "Cat" and "Dog" are invalid swaps. The new object must be contextually plausible. #!ll Outpul Format:
-The <object>> >
You must provide a single JSON object as your output. Do not include any Markdown, code fences, or additional commentary.
The ISON object should have the following keys:
"status": ; string, either "passed" all checks succeed, or "failed" any check fails. "reasons": An array of strings. If the status is "failed", this array should contain detailed list of every rule that was violated (e.g., "ground-truth caption lacks a scene", "Object swap negative is a synonym of the ground-truth object", "Number of global context negatives is not 3").
status is "passed", this urray should be empty.
"ground truth elements": An object containing the extracted components of ground- truth caption. This should be populated only if the ground-truth verification passes. "object": The extracted object string.
"relation": The extracted relation string. "scene": The extracted scene string.
"negative.clements" elements": An array of objects. Each object corresponds to a negative caption and should contain:
"text": The original negative caption text.
"type": The original negative caption typc.
"object": The extracted object string from the negative caption.
"relation": : The extracted relation string from the negative caption.
"scene":
The extracted seene string from the negative caption.
Figure 6: The prompt template used for verifying the collected samples via LLM-based filtering.
20

---

Published as a conference paper at ICLR 2026
(a) Word Cloud
(b) Annotation Composition
(c) Image Composition
Shu mar
Relcocog
ReCoodDg
Train
person
but
COCO-Souff
6.
64.9%
table
Jemes
22.3%
VisuaiGanoma
VisualGenom
(d) Cloud
Annotation Composition
Image Composition
AGEJ0X
Evaluation
build
man
CO
38.0%
per son
RarcoCOg
"Ph
23
Figure 7: Detailed statistics of the SCaR train and evaluation sets. (a), (d) Word clouds for the candi- dates. (b), (e): Dataset compositions in terms of numbers of samples. (c), (f): Dataset compositions in terms of numbers of images.
COCO Staff
VisualGenome
RelcOCOg
Heg: 8.
20.0
Meg: 7.8
Meg: 11.1
1000002
37.5
Lergin 1wonts
15.0
10.0
7.5
50
GT
14gg
Neg
Type
Type
Type
Refcoco
Neg:
Meg:
10
Lougth Longthewrdd
Type
Type
Figure Distributions of sentence lengths for ground-truit truth and negative captions across each dataset.
21

---

Published as a conference paper at ICLR 2026
Table 8: Examples of datasets in SCaR. The instruction across all datasets is Find the caption that best describes the segmented object, considering both local details and global context in the given image. Referring object bbox: (bbox). Each first candidate in red is the ground-truth caption.
Dataset
Query Image with bhox
Candidates
Motorsyclc forefront fully shown.
Motongylle in the garage fully shown.
Motoogyelle on racermcc fuily shown.
Motorcycle fildd fully
RefCOCO Yuu et al. 2016
Motorcycle being washed in forefront.
Motongylie with helmnet placed on the seat in forefront.
Motoreycle louded with packages forefront.
Bicycle foretions fully
Scocter forefrom fully shown.
Hurse in forefrvnt fully shown.
bench closest the palm
pedestal the beach.
hench glosest to the palm frce
concrete polestal in city station.
The hench closest to the palm tree and on concrete pelestall in shopping mall atrum.
The bench closest to the palm tree and concrete pedestal in hospinal waiting area.
RefCOCOg Mao
bench
hench covered colorful graffii and concrese pedestal podeeual heach.
pats
The bench holding a stack books and on a concrese pedestal at the beach.
The playgound slide closest the palm tree and on concrete poslestal the beach. pedestal the heach.
The trash closest
and bicycle closest palm tree
concrete podestal
Mouth of sculpoure by the waterfront. gallery.
Mouth sculpture in
sculpture
lush garden.
Mouth sculpture on a mountaintop.
Mouth of sculpture blowing smoke by the waterfront.
VisualGenome KKrihha 2017
Mouth sculpture illuminated spollights the waterfront.
sculpture eating apple by waterfront. Fin of sculpture by the waterfront.
Ear of sculpture by the waterfront.
Tail sculpture
waterfroas.
Cat sitting froet of computer screen.
Cat sitting on kitchen counteriop.
Cat sitting garden.
sitting
sill.
COC0-Stuff Chesar al. 2018
Cat curked up sloeping front of computer sereen.
Cal powing at coflee cup in front computer screen.
Cat playing with headphoones in front of computer scroen. sitting
computer screen.
Dog sitting in frunt of computer sereen.
Parroe sitting from of compater sereen.
standing the chairs
roofed lounge.
standing the chaits aubway platform.
Fan standing near the chairs in hospital waiting area.
Fan standing near the choirs in gymnasium.
ADE20K Zhou al. 2017
blowing
hanging
group ported plants glats-roofed lounge.
ceiling
in glass-roofa lounge.
Fan suriounded by scattered magazines on the flooe in glass-roafed lounge.
Sculptare standing near the chairs in glass-roofed lounge.
Lamp standing the choirs glass roofed lounge.
Plant standing near chairs glass-roofad lounge.
we observe that the word frequency distributions of ground- truth and negative captions are closely aligned across splits, suggesting that the benchmark is not dominated by small set of frequent words. Moreover, both image- and annotation-leve distri- butions reveal that COCO-Stufi and VisualGenome contribute the largest proportions of samples, reflecting their broad coverage of scenes and object relationships. Fig. 8 illustrates the sentence length distributions of ground-truth and negative captions across the five datasets in SCaR. Overall, the two distributions are closely aligned, indicating that the synthesized negative captions generated by GPT-4V preserve similar linguistic complexity to the ground-truth annotations rather than intro- ducing trivial artifacts. We also observe slightly heavier tails in the negative caption distributions, suggesting greater variability in sentence length. This property enhances the linguistic diversity within SCaR while maintaining a comparable level of difficulty, thereby preventing models from exploiting superficial cues such as caption length.
collection process described in Sec. 3.2
D.5 SCAR EXAMPLES
Tab. g presents SCaR examples for each dataset, where can be seen that the five datasets cover diverse domains. Additionally, the examples indicate that naively cropping with bounding boxes would sacrifice global scene context (c.g., "at the beach" in RefCOCOg, "by the waterfront" in VisualGenome, and "in glass-roofed lounge" in ADE20K).
22

---

Published as a conference paper at ICLR 2026
Table 9: Ablation study with VIRTUE-2B in terms of 1) VLM, 2) Task-specific instructions, 3) Image resolution, 4) Segmentation streamline, 5) Segmentation model choice: 6) # MLPs in the segmentation-languaae connector; 7) Length of segmentation embeddings: 8) Cross-modal alterna- tives; and 9) Finetuning SAM-2. The highlighted row denotes the configurations for VIRTUE-2B
and VIRTUE-7B.
1) Choice
MMEB
SCaR
2) Instruction
MMEB
SCaR
3) Resolution
MMEB
SCaR
Phi-3.5-V
61.7
26.5
59.2
24.3
672x672
60.1
27.6
VI.2B
6488
30.4
64.8
30.4
1344x
64.8
30.4
4) Altemative
MMEB
SCaR
5) Choice
MMEB
SCaR
61 MLP
MMEB
SCaR
Prompter
61.0
22.7
SAM2.1-S
60.1
21.6
60.4
Cropped
63.3
25.9
SAM2.I-B+
64.8
30.4
64.8
30.4
Segmeniation
64.8
30.4
S-M2.1-L
63.8
27.9
63.5
29.7
7) [S]
MMEB
SCaR
8) Alternatvee
MMEB
SCaR
Finctune
MMEB
SCaR
64
61.1
29.9
Cross atlention
63.1
30.0
64.8
30.4
Attention-paol
55.9
63.5
25.8
1024
60.2
35.1
Cony-MLP
64.8
30.4
E ADDITIONAL EXPERIMENTS
E. ABLATION STUDY
We conduct seven ablation variants to investigate the relative contributions of different components:
1) VLM backbones, 2) with and without an instruction in the query, 3) input image resolution, 4) segmentation streamline alternatives, 5) different sizes of SAM-2, 6) different numbers of MIPs connector (Eq. [ where one MLP projects from d, d, and three MLPs project from d, to 768, then to 1024, and finally to d: 7) segmentation embedding lengths [S]; 8) mrosl modeling alternatives; and 9) finetuning and freezing SAM-2. For 4) segmenta- tion streamline alternatives, we replace our segmentation model with Prompter from CLOC (Chen et a 2025), which encodes bounding boxes via randomly initialized Transformer, or directly crop images for the vision encoder (denoted as Cropped). Both variants concatenate embeddings from nine random crops, similar to the setting of nine sampled points in SAM2. for the segmentation- language connector for MMEB. For 8) cross-moda modeling alternatives. connect: Cross-attention, which uses the same architecture as the
in the segmentation-langeen
adopt two variants
for the saanecettttionllannaaa vasion-language connector in Qwen2-VL VI.. Attention-pooling: A natural approach using A: learnable queries to pool information from the region features (we used 64).
As shown in Tab. 9 although Phi-3.5-V has more parameters than Qwen2-VL-22 VL-2B, the latter yields better results on MMEB and SCaR. Removing text instructions and using lower resolutions lead significantly inferior performance. The degraded results from Prompter and cropping suggest that pre-trained SAM2.1 provides not only more precise segmentations than simple crops but also a stronger understanding from its pre-trained knowledge. Although SAM2.1-L performs on par with SAM2.1-B+ in terms of MMEB, its performance degrades on SCaR. In contrast, using SAM2. deleteriously impacts both benchmarks. Additionally, increasing MIP layers and segmentation embedding length |.S does not consistently yield gains across MMEB and SCaR. While increasing segmentation embeddings to 1024 delivers better results on SCaR, we stick to selecting 256 as our configuration balance computation and effectiveness. Using attention-based moss-modal methods does not yield better results, although cross-attention achieves performance comparable to our con- volutional MLPs. Similarly, finetuning SAM-2 performs worse than keeping it frozen- -especilly SCaR-sines since MMEB-trait does not contain user sperisie visual prompts (i.c., using uniform points). This mismatch biases the segmentation pipeline toward non-visual-interan scenarios.
23

---

Published as a conference paper at ICLR 2026
KNEB
Performance
Performance
Performance
NHES Scalt
WIMEE SCal
10-
532
1024
2348
226
LoRA Ranks
Batch Size
Sampling Points N
Figure 9: The impacts of varying LoRA ranks, batch sizes, visual-prompt design choice, and sam- pling points N on VIRTUE-2B.
Table 10: MMEB reevaluation results after finetuning on the SCaR-train dataset.
Per Meta-Task Score
Average Score
Classification
VQA
Retrieval
Grounding
IND
OOD
Overall
# Datasets
10
10
12
4
20
16
36
VLM2Vec-2B
58.7
49.3
65.0
72.9
64.9
53.3
59.7
+SCaR-train
35.8
7.5
14.7
25.8
19.4
19.2
19.8
VLM2Vec-7B
62.7
56.9
69.4
82.2
71.4
58.1
65.5
+SCaR-train
47.5
31.8
48.6
46.3
44.4
42.0
43.4
MMRet-7B
56.0
57.4
69.9
83.6
68.0
59.1
64.1
+SCaR-train
45.2
38.1
51.7
77.9
51.8
46.5
49.4
UniME-7B
60.6
52.9
67.9
85.1
68.4
57.9
66.6
+SCaR-train
55.0
46.4
64.3
84.5
62.7
54.3
59.0
VIRTUE-2B (Ours)
64.1
55.7
68.4
78.7
69.7
58.8
64.8
+SCaR-train
64.7
55.1
68.3
79.2
69.8
58.7
64.9
VIRTUE-78 (Ours)
65.6
60.4
71.8
87.3
74.4
61.4
68.6
+SCaR-train
64.4
58.1
67.0
86.4
72.2
60.0
66.8
E.2 HYPERPARAMETER STUDY
We further analyze the impacts of 1) LoRA rank (4, 8, 16, 32), 2) batch size (256, 512. 1024, 2048), and 3) sampling points used in non- -intsual-interattivee tasks (MMEB). Fig. 9 shows that varying LoRA ranks produces consistent outcomes, with rank showing slight edge. Furthermore, larger batch sizes improve generalizability for in-batch negatives, with a batch size of 1024 obtaining the best results. We also find that increasing the number of sampling points does not enhance model effectiveness for non-visual wiisual-nnteractive tasks, presumably because key entities within an image for general tasks can be captured with only a few points.
E.3 REEVALUATING MMEB WITH FURTHER FINETUNING ON SCAR-TRAIN
To evaluate the impact of incorporating SCaR-train on MMEB performance, we reevaluate VIeM2Ve--2B, VLM2 Vec-7B, MMRRt-7B, UniME-7B, VIRTUE-2B, and VIRTUE-7B with SCaR- train (+SCaR-train) (Sec. 5.3) and compare them against the original checkpoints trained solely on MMEB-train. As shown in Tab. T0, all baselines exhibit degraded performance, likely because the models overfit or shift focus toward the SCaR distribution for incremental learning (Li Hoiem 2016). Notably, VLM2 and MMRet suffer severe degradation, suggesting stronger susceptibility to catastrophic forgetting (French, 1999) when their training rely more heavily on additional data. In contrast, UniME, with its multi-stage contrastive learning design, demonstrates greater robust- ness against forgetting. Meanwhile, VIRTUE maintains a performance comparable to the original MMEB-trained checkpoints, with VIRTUE-2B even achieving a slight 0. -point gain. We attribute
$CL.C cannot be directly evaluated reproduced, its model, code, and parts of training data are not publicly available. We attempted contact the authors but did receive response.
24

---

Published as a conference paper at ICLR 2026
VIRTUE
Figure 10: In-the-wild iistal-interactive (top) and naive cropping (bottom) saage-to-image retrieval scenarios. VIRTUE-7B leverages visual prompts (bounding boxes in this paradigm) guide the retrieval of regions of interest while accounting for both entity-level details and global scene context.
this stability to the segmentation streamline, which enables the use of optional visual prompts and makes VIRTUE more adaptable universal embedder capable of handling both and cat--viull-intcracee tasks.
E.4 CASE STUDIES FOR VIRTUE'S VISUAL-INTERACTIVE CAPABILITIES
With visual interaction, VIRTUE enables new applications such as segment-level retrieval, where users select a region of interest to fetch semantically matching images, and entity-level hinting for on-the-fly fly correction, thereby extending the utility of embedding based systems far beyond tradi- tional global matching.
E.4.1 VISUAL-INTERACTIVE IN-THE- WILD 121 RETRIEVAL
While evaluating VIRTUE on SCaR confirms its ability to interact with visual interactions, we further conduct wild visual-interactive image-to-iman retrieval. Similar to the procedure constructed SCaR, prompt GPT-4o with "Generate some text captions that need to consist of "object, relation, scene" for searching images. Only contain entities within an image is sufficient.". Then, the generated prompt the Google Search AP get six ground-truth images (three depicting cars parked on the road and three depicting dogs on the sidewalk with nearby grass), and subsequently instruct GPT-4o to generate 20 negative captions conditioned on the selected text caption. Afterwards, those negative captions are used to search the corresponding images as negative candidates.
Fig. 10 illustrates the risual-interactive (top three rows) and naive cropping (bottom two rows) to-imaee paradigm enabled by VIRTUE-7B. The first row depicts conventional retrieval ap- plications, also supported by VIRTUE, where images are retrieved based on holistic query image information. What distinguishes VIRTUE is its ability incorporate user-specified visual prompts,
25

---

Published as a conference paper at ICLR 2026
VOA
What is in the motorcyclist's mouth?
This type of bus can be found what popular city?
Original prediction: microphone After providing bbox: cigurette
Original prediction: new york city After providing bhox: senttle
12T Retrieval
Original prediction: busted red fire hydrant X spewing water all over street creating rainbow. After providing bhox: There blue lights shining through palm trees.
Original prediction, pillow After providing bbox: pen
Figure 11:
On-the-fly fly correction with explicitly bounding box hinting of VQA and 12T retrieval paradigms. All of them are used with VIRTUE-2B.
as demonstrated in the second and third rows within the figure. For instance, selecting the parking car directs VIRTUE focus on the car while preserving the surrounding scene context (e.g., grass or trees along the road). Similarly, selecting the dog on sidewalk guides the model to retrieve dogs on sidewalks with nearby grass or trees. Attributed the segmentation streamline VIRTUE, both scenarios illustrate novel possibilities for interactive querying between humans and embedding mod- els. On the flip side, naive cropping (bottom two rows) discards scene context, leading the model to retrieve the selected entity in mismatched contexts from the given image (e.g., running the street or a dog running on the grass in the retrieved results, even though neither is moving in the query image).
E.4.2 ON-THE-FLY CORRECTION WITH VISUAL HINTING
To delve into the capability on-the-fly correction, we randomly sample MMEB cases that VIRTUE-2B initially misclassifies. We then manually provide the correct regions as visual prompts and re-run inference. As illustrated in Fig. I] VIRTUE successfully corrects predictions not only for VQA but also for retrieval tasks, relying solely on visual hints at inference time without addi- tional finetuning. This demonstrates a new mode of applicability for VLM-based embedding models, where users can guide the model interactively while avoiding the computational cost of conventional finetuning. Although direct cropping of hinted regions may also yield corrections, visual prompting VIRTUE eliminates the need for heuristic preprocessing and proves more robust in challenging cases. For example, in VQA tasks, cropping can fail when the region excludes crucial contextual in- formation (c.g., the motorcyclist), whereas visual prompts enable the model to integrate both global and fine-graine iine-grained cues.
E.5 QUALITATIVE RESULTS ON SCAR
Fig. 12 presents qualitative comparisons between baselines and our VIRTUE-7B model trained with SCaR-train. We observe that the baselines frequently misinterpret relational cues (c.g., "player with head down" to "player picking up bat") and even retrieve entirely incorrect objects (c.g., "tree" "cow and statue"). A likely reason is that their representations dominated by L.M-
26

---

Published as a conference paper at ICLR 2026
hous
Mallle Crass covaiig the yaad course
UnMME: Greus courtig yord thori of sone houss
VICTUE: Grass owaiing the yard in eross houss
Grass sevet's the padd a park
Playr with Hoat down oen
Grans swerli the on paf counts.
Paae with head ovdty peens
Guan with eccthone autunn lecuee of race
Paaee leng selaceuin back
Mingts
fower savatigg the yast ben of accee haues
Rurlese
with down the
cravel cavelty part
VLMoWee: Pester
&
V.Ni7 Hollors boott during shoet
MURket Farsin
wetitee Pinglo do
Ui MNT
Plicdo freidc twe
VIRTUE: Persnn
ViRTUE: Parple
a tightmms
Pinoo uarin 2 to Vn darioaan
Paople bance aiss unriing
Turan 2 C bunch muaasuu
Puroon rend ng banch
pickur banch sams
Prople sinepligg uncoor dhuck vitee
Bioydle ulting o banch catt
Avalnaae cogtigg lightr stnall
benchh duitng
Backppskk thin on banch atbiort
duriigg a mrhtce puinds
VLMoWWe: Dan has zank an ciunt for pander.
Vi MOMio Rook aroa mosit cann
Mulhic Cos has in in andian tample garies
UNMMM: Soau musk andiet timple gaider.
UniMt:
VITTUE: Theo troic encic dondo partee.
VIRTUE: Sooc amonged inddo opin salcii
Free has Stuse an anarent tample gardee
Boss antanged intoo
Tree in bepy city elbraction
Busk lengome
Thes lurm hougg in encunt lemole pardens
Buoi balarced ce auiiial her
puoct curggn loiing on donr lenngi ciice Coe In are anolent partir
Rood
make cuduate
Figure 12: Qualitative comparison of VLM2 MMRet-7B, UniME-7B, and VIRTUE-7B. Ground-truth captions are shown in green and incorrect ones red.
derived features (Tngg et al., 2024), which tend to overlook spatial information such as bounding boxes in the text query. In contrast, VIRTUE incorporates segmentation- based streamline that augments regions of interest with fine-grained visual representations, effectively guiding retrieval toward semantically and spatially accurate matches.
E.6 ANALYSIS ON LATENCY AND MEMORY
We report the average end-to-end inference time (ms) and GPU memory (GB) for four distinct operational modes of VIRTUE-7B on the SCaR benchmark (I2T) and provide baselines (VLM2Vec, MMRet, and UniMB) for 12I (retrieval and visual grounding) performance from MMEB, shown
Table 11: Analysis on inference time and memory for interactive 12T (SCaR) and I2I (retrieval and visual grounding of MMEB) of different query modes. For VIRTUE-7B, query modes with cropping and text-only remove the segmentation streamline.
Query Mode
Time (per sample)
Memory
Performance
12T (SCaR)
VIRTUE-78 7B
w/ visual prompts
422ms
18.8GB
27.8
w/o visual prompts (uniform points)
427ms
18.8GB
16.1
Cropping
139ms
17.7GB
22.8
Text-only
340ms
17.7GB
23.9
12I (Retrieval and Visual Grounding on MMEB)
VIRTUE-7B
w/ uniform points
402ms
18.8GB
70.7
w/o uniform points
357ms
17.7GB
46.5
VLM2Vec-7B
N/A
406ms
18.6GB
67.9
MMRet-7B
N/A
538ms
29.0GB
75.8
UniME-7B
N/A
419ms
28.9GB
66.4
27

---

Published as a conference paper at ICLR 2026
in Tab. For cropping and text-only modes, remove the visual prompt streamline. All 12T evaluations are reported at matched precision (bf16), and all results are obtained with a batch size of on single H100 GPU. While cropping and text-only modes offer better latency and memory, the performance becomes significant inferior. We further include the removal visual prompts for 121 scenarios, which is efficient but degrading significantly. Moreover, using uniform points is more efficient than the existing baselines.
E.7 DETAILED SCORES OF MMEB
Tab. presents detailed results of 6 VLM--ased models that are trained with MMEB for com- prehensive comparisons. The performances sourced from the corresponding papers. Due the space limits, the detailed scores of CLIP, BLIP2, SigLIP, OpenCLIP, UnilR, Magiclens, ES-V, GME, and LamRA can be referred from their official papers.
Table 12: Detailed results of the VLM-based baselines and our VIRTUE on MMEB. The 16 out-of- distribution datasets are highlighted in pink.
VLM2Ncc-2B
VIRTUE-2B
MMRct-7B
VLM2Vce-7B
UniME-7B
VIRTUE-7B
Classification (10 tasks)
ImageNet-1K
77.5
80.1
58.1
80.1
71.3
82.3
N24Ncws
73.7
79.7
79.5
81.1
HatefulMemes
58.3
67.5
69.7
64.6
74.1
VOC2007
74.3
80.7
90.4
85.4
SUN397
73.8
74.3
70.0
77.4
75.9
78.3
Place365
35.3
36.4
43,0
37.4
45.6
39.6
ImageNet-A
50.9
53.4
36.1
58.1
45.5
55.2
ImageNet-R
84.7
73.9
78.4
82.5
ObjectNet
37.1
40.1
36.4
44.3
Country-211
21.5
30.5
14.7
29.8
18.7
32.8
Average
58.7
64.1
56.0
62.7
60.6
65.6
VQA (10 tasks)
OK-VQA
48.5
56.7
73.3
56,8
68.3
59.6
A-OKVQA
39.5
50.5
56.7
47.3
58.7
52.2
DocVQA
82,5
87.9
78.5
89.7
67.6
91.7
InfographicsVQA
47.7
60.0
37.0
63.7
CharQQA
42.3
$6.9
61.0
Visual7W
51.2
$2.3
49.5
52.7
51.7
55.0
ScienceQA
30.7
40.9
45.2
38.5
40.5
46.7
VizWiz
38.6
44.3
51.7
39,9
42.7
43.3
GQA
48.3
48.7
59.0
55.1
63.6
54.7
TextVQA
71.6
65.2
76.4
Average
49.3
$6.9
$2.9
60.4
Retrieval (12 tasks)
VisDial
74.3
78.5
83.0
81.9
79.7
83.3
CIRR
46.8
56.0
61.4
51.1
$2.2
60.4
VisualNews 12i
73.1
75.1
74.2
80.5
74.8
80.8
VisualNews i2t
73.7
77.9
78.1
81.2
78.8
82.5
MSCOCO
73.4
73.9
78.6
77.2
74.9
78.0
MSCOCO.1.
68.5
73,9
76.3
NIGHTS
66.3
67.6
66.2
70.6
WebQA
85.9
88.7
90.2
88.3
89.8
91.2
FushionIQ
14.0
15.0
54,9
17.1
16.5
18.1
Wiki-SS-NQ
54.2
60.8
24.9
62,3
66.6
66.3
OVEN
68.3
69.2
87.5
66.5
55.7
67.2
EDIS
85.7
86.6
Average
65.0
69.4
67.9
71.8
Visual Grounding tasks)
MSCOCO
66.5
76.5
76.5
80.5
RefCOCO
80.9
86.0
89.8
89.3
89.3
94.2
RefCOCO ReICOCO-matching
75.7
83.1
90.6
90.6
90.6
92.0
68.3
75.0
77.0
84.1
84.1
82.6
Average
72.9
78.7
83.6
82.2
85.1
87.3
Final Score (36 tasks)
IND
64.9
69.7
68.0
71.4
68.4
74.4
OOD
53.3
58.8
59.1
58.
57.9
61.4
Average
59.7
64.8
64.1
65.5
66.6
68.6
28


Reviews:
[{"review_id": "yVxpPEUNsb", "reviewer": "Reviewer_Zhv1", "summary": "The paper proposes VIRTUE, an embedding framework that enhances a frozen vision-language model with a frozen segmentation model (SAM‑2) and a lightweight connector to inject region-level tokens for fine-grained visual interaction. To evaluate this, the authors introduce SCaR, a large benchmark for region-in-context caption retrieval with challenging negatives generated by GPT‑4V. VIRTUE achieves notable gains over baselines on both MMEB and SCaR, with ablations covering alternative encodings and model variants.", "questions": "Refer to weakness part.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The studied problem is sensible, articulating the gap between text‑only interactions in current embedders and region‑aware visual interactions.\n2. A new segmentation-and-scene caption retrieval benchmark is proposed. It also includes case studies for image-to-image retrieval and on-the-fly correction via visual hints, showing great practical value.\n3. The experimental results show the proposed framework achieves better results than baseline methods.", "weaknesses": "1) The core contribution is essentially feature concatenation from a frozen segmenter into a frozen VLM with a small connector and contrastive fine‑tuning. While effective, there is little architectural insight into how to fuse region tokens with global tokens beyond prepending and projecting; there is no principled modeling of cross‑token interactions (e.g., region‑aware attention, cross‑modal slotting, or routing). This makes the method feel incremental despite the strong results.\n2) Baselines that “do not accept boxes” are evaluated primarily via textualizing the bbox or naive cropping; yet there exist stronger region encoders and grounding‑aware alternatives (e.g., leveraging GroundingDINO features, ROIAlign/feature pooling over detected regions, or using grounding‑style LLMs like Ferret) that might close the gap. It is not surprising to me that the proposed method can achieve better performance.\n3) After SCaR‑finetuning, VIRTUE‑7B still drops on MMEB (68.6→66.8), while 2B is stable; baselines often degrade more. A deeper analysis is missing.\n4) The study varies SSS, SAM‑2 size, and a few hyperparameters, but does not (i) compare frozen vs lightly finetuned SAM‑2, (ii) probe attention patterns to confirm that the LLM actually leverages segmentation tokens for compositional cues, or (iii) test robustness to noisy or misaligned visual prompts (jittered boxes, partial masks, off‑by‑k clicks). \n5) Please report end‑to‑end latency and memory for typical query modes (I2T with and without prompts; I2I) versus cropping and text‑only baselines, at matched precision.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "06BpP3IMdQ", "reviewer": "Reviewer_5dth", "summary": "This paper introduces VIRTUE, a multimodal embedding model that extends image-text embedding models with visual prompts/instructions. Unlike existing embedding models that only accept text instructions, VIRTUE integrates a segmentation model to process visual prompts (bounding boxes, points, masks) alongside text and images. The authors also contribute SCaR, a new synthetic dataset for evaluating visual-interactive image-to-text retrieval.", "questions": "- why did you choose MMEB over alternative comparable benchmarks\n- I am unsure if this model actually supports interactive selection that does not align with the SAM2. Did you experiment with arbitrary interactive human selection?\n- Why are only some of the models fine-tuned on MMEB? I am unsure what the selection process was here? Especially interesting it why seemingly competitive models (GME-7B), weren’t fine-tuned.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- They address an existing gap that current models do not support visual prompts. I would, however, love to see this point argued more. E.g., by including examples of real-world uses.\n- Contribute a new dataset (SCAR) with challenging negatives\n- Their method proposed an elegant adaptation of VLMs for producing embeddings, with image instructions using bounding boxes, which I believe has real-world applications", "weaknesses": "- The proposed method, while increasing performance and addressing an existing limitation, does so by increasing model complexity, which might make the model harder to run, optimize, and adapt\n- The dataset (SCAR) only explores visual prompts, while I do think that most real-world use cases would use both visual and text prompts. E.g., in the \"find similar product\" feature, you could mark an item, which then retrieves using the bounding box and a prompt \"find products similar to the one highlighted\".\n- The heavy reliance on GPT-4V and lack of metrics, such as inter-annotator agreement, makes it hard to evaluate the overall quality of the dataset. It is also uncertain how much of the performance gain is learning specific quirks of GPT-4V and how much is generalizable\n- I would update figure 2 to include all elements in the filtering (e.g. wordnet)\n- Appendix E: Don't name your section \"extensive experiments\", just call it experiments\n- I would argue that the statement: “they only accept text as the human-machine interaction modality” (Anonymous, 2025, p. 1) is borderline incorrect. However, I agree that the model was at least not designed with that intention in mind.\n- see question on choice of fine-tuned models", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NsRrV6S1L5", "reviewer": "Reviewer_a385", "summary": "This paper proposes a visual-prompt MLLM-based embedding model and a corresponding new benchmark for measuring visual-prompt retrieval. The authors observe that existing MLLM-based embedding models can only accept textual instructions. Therefore, they proposed a new model VIRTUE that leverages similar visual prompts with SAM. Experiments are conducted on both the proposed benchmark and an existing benchmark.", "questions": "See above weakness", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The new problem setting is interesting and novel. I think visual prompts are complimentary to textual prompts to MLLM-based embedding models and can be practical in real-world applications.\n\n2. The paper contributes a new method and a new benchmark which may be beneficial for both the methodology and data perspectives. Meanwhile, the method itself seems reasonable.\n\n3. Experiments on both the proposed benchmark and existing benchmark seem to validate the effectiveness of the proposed method.", "weaknesses": "1. The scope of the new benchmark seems narrow. The main task of the new benchmark is caption retrieval. However, most existing benchmarks (e.g., MMEB and MBEIR[1]) for these models contain more tasks such as VQA retrieval, composed retrieval, or visual-document retrieval. The proposed benchmark seems to be more narrow than existing benchmarks that can undermine its universality.\n\n2. The technical novelty seems to be limited. The model seems like a combination of existing MLLM-based embedding models and SAM. The textual prompts and visual prompts are from existing works and the forwarding and training process are similar to existing MLLM-based embedding models.\n\n3. Experiments are only conducted on Qwen2-VL models. It is also recommended to validate the method on different herds of MLLMs. Meanwhile, other important baselines GME-QWen[2] which also uses QWen2-VL  are missing.\n\n4. While handling non-visual-interactive scenarios, the uniform visual prompts seem to be redundant. Since all inputs are concatenated into one sequence, I think directly discarding the visual prompts can be more efficient in this case.\n\n1. Wei C, Chen Y, Chen H, et al. Uniir: Training and benchmarking universal multimodal information retrievers[C]//European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024: 387-404.\n\n2. Zhang X, Zhang Y, Xie W, et al. GME: Improving Universal Multimodal Retrieval by Multimodal LLMs[J]. arXiv preprint arXiv:2412.16855, 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Fp6W9j7oSG", "reviewer": "Reviewer_jcPk", "summary": "This paper presents a new visual-interactive text-image universal embedding model designed to address the limitations of existing embedding models in handling visual cues such as bounding boxes, masks, and clicks. It combines the segmentation model (SAM-2) with a pre-trained vision-language model (VLM), enabling users to visually specify regions of interest within images, thereby enhancing the model's understanding of entity-level information in images.To evaluate the visual interaction capabilities of VIRTUE, the authors also constructed a large-scale benchmark named SCaR.", "questions": "1. VIRTUE has brought significant performance improvements. To clarify the respective contributions of the dataset and the methodology, I would like to konw performance variations of other methods in the two scenarios of using SCaR and not using SCaR, as well as the outcomes of the proposed method when SCaR is not employed.\n2. The generation of negative samples in the SCaR dataset relies on GPT-4V. Are there any biases or issues related to semantic repetition?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. New Task: This paper introduces a novel visual-text embedding paradigm that, for the first time, incorporates visual cues such as bounding boxes, points, and masks into a universal embedding model. This approach overcomes the limitations of traditional models that exclusively support textual prompts and expands the possibilities for human-computer interaction.\n2. Valuable Benchmarks: SCaR is the first large-scale benchmark designed for visual-interactive image-text retrieval, filling a gap in this field with significant diversity and challenges.\n3. Clear writing and comprehensive experiments.", "weaknesses": "The primary contribution of this paper lies in the introduction of a new dataset. The proposed method does not present any technical innovations; rather, it merely combines SAM and VLM. Consequently, the technical contributions of this paper are quite limited.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)