ABC: Achieving Better Control of Visual Embeddings using VLLMs
Authors: Benjamin Schneider
Florian Kerschbaum
Wenhu Chen
Venue: Accepted by TMLR ()

Abstract: Visual embedding models excel at zero-shot tasks like visual retrieval and classification.
However, these models cannot be used for tasks that contain ambiguity or require user in-
struction. These tasks necessitate an embedding model which outputs can use a natural
language instruction to control the representation of a visual embedding. Existing CLIP-
based approaches embed images and text independently, and fuse the result. We find that
this results in weak interactions between modalities, and poor user control over the repre-
sentation. We introduce ABC, an open-source multimodal embedding model that uses a
vision-language model backbone to deeply integrate image features with natural language
instructions. ABC achieves best-for-size performance on MSCOCO image-to-text retrieval
and is the top performing model on classification and VQA tasks in the Massive Multimodal
Embedding Benchmark. With a strongly unified vision-language representation, ABC can
use natural language to solve subtle and potentially ambiguous visual retrieval problems. To
evaluate this capability, we design CtrlBench, a benchmark that requires interleaving tex-
tual instructions with image content for correct retrieval. ABC advances the state of visual
embeddings, outputting high-quality visual representations with natural language control.
Our model and datasets are available at our project page: https://tiger-ai-lab.github.io/ABC/

Published in Transactions on Machine Learning Research (08/2025)
ABC: Achieving Better Control of Visual Embeddings using VLLMs
Benjamin Schneider, University of Waterloo
BesjoorinSchmeeeeeeeaattrooo...
Florian Kerschbaum, University of Waterloo
furiia..errchabmmeuuuueeeeooo
Wenhu Chen, University of Waterloo
wenhuchhnn@uwaalerrro
Reviewed on OpenReview: o/perrveeppnreveee. net/ formfo id= RezANmBpay
Abstract
Visual embedding models excel at zero-shot tasks like visual retrieval and classification. However, these models cannot be used for tasks that contain ambiguity or require user in- struction. These tasks necossitate an embedding model which outputs can use a natural language instruction to control the representation of visual embedding. Existing CLIP- based approaches embed images and text independently, and fuse the result. We find that this results in weak interactions between modalities, and poor control over the repre- sentation. We introduce ABC, an open-source multimodal embedding model that uses vision-language model backbone to deeply integrate image features with natural language instructions. ABC achieves best-for-size performance on MSCOCO image-to-text retrieval and the top performing model on classification and VQA tasks the Massive Multimodal Embedding Benchmark. With a strongly unified vision-language representation, ABC can natural language solve subtle and potentially ambignous visual retrieval problems. To evaluate this capability, we design Ctr CtrlBench, benchmark that requires interleaving tex- tual instructions with image content for correct retrieval. ABC advances the state of visual embeddings, outputcing high-quality visual representations with natural language control.
Our model and datascts are available at our project page:
saiiiillbb.gihuub... 1o/ABC/
1 Introduction
Visual embeddings have become a foundational representation computer vision. Image embedding models have become the state of the art for many zero-shot tasks, including visual retrieval (Chen et al.] 2024) and image classification (Yu et al.] 2022). Since the release of CLIP [Radford et. al.] 2021 its dual encoder architec- ture has remained the state of the art for producing high-quality
Natural Language Instruction
Apples
What is the bottom sign?
Peats
Candidate Pool
The a signage"
visual embeddings (Yu et al.) 2022 Sun et al.] 2023 Chen et al. 2024). However, CLIP only supports separately embedding images or text (Radford et al. (021). Therefore, complex visual embedding tasks which require additional specification are impossible. For ex- ample, CLIP model cannot distinguish which is the best answer in Figure ! All candidates are plausible unless the user provides ad- ditional instruction. However, additional conditioning on natural language question disambiguates the query, making it apparent that the bottom candidate the best answer (Stelmakh et al. 2022). Conditioning on natural language gives the user greater control over their visual query. This especially critical for retrieval augmented generation systems, which must unambiguously retrieve a small number of highly relevant. facts from large knowledge bases, support accurate text generation (Amira al. 2025
dark grey sign that saye Pears"
A gn that says plums
Figure A task from CtrlBench. Conditioning on the natural language instruction is required to unambigu- ously retrieve a candidate.

---

Published in Transactions on Machine Learning Research (08/2025)
Negative Mining
Pretraining
VisIO Encoder
BB DM
LLM
MLP
Enéazicing
Contranntallccc
so0n000 OOR
Happo dog.
Coomoo pct
LoRA
Beiggolico
peints
negatives
Sample e negatives for batch
Instruction Fine-tuning
Instruction Design
Can didinss
LLM
Vince Lncodar
What an coece?
Ah
02:Whor geting oy A2: oor sread
Toxt Eir bodile
QD What int the Twitoegrounn tarimrogs tocing
fouurrur di Imoon tocing the seoest boto leech
Controstive vel Kosk.
Figure 2: An overview of ABC's training regime. First, we mine negatives to augment each batch in our protraining dataset with almost plausible text candidates. Then we prctrain our model using contrastive loss on independently embedded lext-image pairs. Once our model has learned to output high-quality uncontrolled visual embeddings, we introduce instructions. We create multiple instructions and candidate captions for single image, with each one focusing different aspects of the image. Lastly, we fine-tune using multiple captions for same image, the positive caption for each query corresponds to the natural language instruction. Therefore, ABC uses the instruction to distinguish the correct candidate for the query.
We find that existing approaches suffer from two problems: (1) Reliance on generic or human-annotated instructions (Jiang et al. 2024 Zhang et. al. 2025). Unlike the abundant noisy image-text data used to train CLIP models Schuhmann et al. (2022) Changpinyo et al. (2021); Sharma et al. (2018) datasets containing multiple modalities and diverse instruction phrases significantly more scarce and challenging to curate at scale. This is problematic for embedding models, which require massive datasets for the best representations (Cherti et al., 2023a). Often vague instructions are used repetitively during training, resulting in overfitting of the instruction (Wei et al. 2023 Gudibande et al.) 2023 (2) Weak interaction between modalities. Previous works fuse embeddings outputted by CLIP models Zhang et al. (2024); Wei et al.) (2023). This approach prevents deeper interaction between modalities, resulting in superficial use of the instructions Jiang et al. (2024)
To this end, we introduce ABC, model that uses natural language instructions to control visual embeddings. ABC's Vision Large language Model (VLLM) backbone allows it to integrate natural language instructions when crafting visual embeddings. We find that training our model has two fundamental challenges: (1) Extracting useful contrastive embeddings from a pretrained generative VLLM. (2) Designing an instruction fine-tuning method that lets users modify multimodal embeddings using natural language instructions. To train ABC, we adopt multi-stage training process. In the initial pretraining stage, we use contrastive training with carefully selected negatives to develop a model that generates embeddings, similar to CLIP. In the instruction fine-tuning stage, we train lightweight adapter to project queries consisting of images and natural language instructions into the embedding space of the pretrained model. Our synthetic instructions to correspond to different aspects of the same image, therefore, ABC learns to use the instruction for fine- grained control. Our training results in a model that produces powerful and controllable visual embeddings.
ABC achicves impressive zero-shot performance retrieval, classification, and visual question answering (VQA) lasks. In MSCOCO (Lin et al. (2015) image-to-text retrieval, our model outperforms all CLIP models containing at most 8 billion parameters. Furthermore, shot classification and VQA splits of MMEB (Jiang et al.] 2024), multimodal embedding benchmark spanning 19 tasks. Lastly, we design CtrlBench to measure our model's ability to use natural language instructions to control retrieval. Using CtrlBench, we show that. ABC can accomplish visual retrieval tasks that are fundamentally ambiguous without utilizing natural language instructions.
model outperforms all other models the zero-
Our contribution is threcfold. (1) ABC: an open-source multimodal embedding model that uses natural lan- guage instructions to control visual embeddings. We demonstrate that ABC produces powerful embeddings
2

---

Published in Transactions on Machine Learning Research (08/2025)
by benchmarking on zero-shot tasks and robustly utilizes natural language instructions to control embed- dings. We study how key design and VLLM backbone- impact embedding quality. (2) We provide decoupled methodology for adapting VLLMs into dense embodding models. Previous work integrates instructions during the large-scale con- trastive training run. We show that. these stages can be decoupled, resulting a lightweight and adaptable fine-tuning stage that requires only 100 training steps using synthetic instructions. (3) Lastly, we introduce CtrlBench, a novel benchmark for measuring instruetion-controlledd retrieval. CtrlBench requires the model to interleave modalities to retrieve the correct response.
vision encoder resolution, contrastive loss temperature,
2 Background
Visual embeddings. Radford et al. (0021) proposed CLIP, which projects images and text to a shared dense embedding space. Due to its contrastive training, the similarity of vectors is computed as simple dot product (Hadsell et al. 2006). CLIP embeddings are used for tasks such as zero-shot image classification and image-to-text retrieval, by computing the similarity of an image query q with set of candidates c E C. An embedding model f maps queries and candidates vectors in R". Given a query q and a set of candidates C C IR" containing the correct answer ct,, the retrieved candidate is:
e arg max f(g) ((c) CEC'
The model is succeeds if c* = ct. Alternatively, when measuring metries such as recall@K, K of the top scoring candidates are returned. Subsequent. works have identified several key factors for training training high-quality CLIP models: DataComp (Gadre et al., 2023] demonstrated the importance of data choice during to create high-quality emboddings. (Cherti et al.) (023b) showed that largo-scale protraining and batch size are crucial for producing the best performing models. Lastly, Aligu (Jia et al. 2021 found that optimizing the setting of the temperature T, used to scale the contrastive loss function, was crucial.
Multimodal embeddings for Visual Question Answering. Visual embeddings, especially those of diverse scenes containing many objects, offer limited control over the embedding (Wei et al. 2023 0. However, a multimodal embedding q = (4; 4t), jointly conditioned on image q and text instruction qe allows for flexible user natural language control over embeddings. This allows the user to create embeddings corresponding to specific objects or concepts in an image (Jiang et al., 2024). For a VQA task, the model embeds (q, q:) jointly into a dense vector. The model uses cosine similarity to retrieve the maximally similar text candidate c' E C. As in the unconditioned setting, the model is successful if c* the correct answer. We use the VQA split of MMEB (Jiang et al.] 2024), which contains 10 lasks that span texl-conditioned classification and retrieval. We also introduce CtrlBench, a benchmark designed to measure conditondt visual retrieval when image-only retricval is ambiguous.
A Ambiguity in visual retrieval. Ambiguity is a well studied problem in information retrieval and uatural language processing (Keyvan & Huang 2022). Often, users provide queries that have multiple distinct, nevertheless plausible answers. Several rectifying approaches have been proposed, including using follow-up questions to disambiguate queries (Kim et al. 2023 Aliannejadi et al. 2019) and query refinement (Stelmakh) ambiguous if its corresponding candidate pool C contains multiple semantically distinct plausible answers. Additional conditioning, such as on a natural language instruction qt, disambiguates qi if the joint query q = (qs. has exactly one plausible answer in C. (Stelmakh et al. 2022).
et al. 2022). Following MMn et al. (2020), we define an image query
Extracting Embeddings from LLMs. Generative large language models are a source of pretrained bases training dense embeddings models. NV-Embed (Lee et al.) 2024] used contrastive training
LLM backbone to achieve the best performance the MMEB text embedding benchmark (Muennighofi
ct al. 2023). Several architectural modifications have been proposed for adapting LLMs to produce dense
embeddings Wang et al. [ [BehnanGhader et al. (2024); Lee et al. (2024). LLM2VEC LLM2VEC/BelnnnnChie
et al. 2024) changed the attention mask to be bidirectional to allow information flow between all tokens. Furthermore, several methods for extracting dense embedding from LLMs hiddens have been proposod.
These include picking the last token (Jiang et al. 2024), mean token pooling (BehnamGhader al. 2024
additional attention layer (Lee et al. 2024].
3

---

Published in Transactions on Machine Learning Research (08/2025)
3 ABC
Figure 2 is an overview of our training regime and model architecture. Our training regime consists of 2 distinct slages; pretraining and instruction fine-tuning, The pretraining stage trains on image-caption pairs to adapt features used for generative modeling into features for dense embeddings. As our pretraining does not require instructions, it can be casily scaled using any large image-captioning data source (Changpinyo et al. 2021 Schuhmann et al. 2022). In the second stage, we train an adapler to allow ABC to embed images conditioned on text instruction, enabling natural language control over visual representations. Using contrastive learning with synthetic instructions, the ABC learns to represent different visual aspects of the same image based solely textual commands.
3.1 Model Design
During pretraining images and text are embedded independently, and aligned using contrastive loss, much like CLIP (Radford al. 2021). Text and images are converted hiddens (0) Rexd using the embodding layer and vision encoder, respectively. The LLM module shared between both modalities. To adapt the LLM to output dense embeddings, we make several architectural changes. Following BehnamGhader et al. (2024), enable bidirectional attention, allowing all tokens attend to all other tokens. To create our dense emboddings, we average over the sequence dimension in the last hidden layer hD (equation 1 and project the result a E Rd using residually connected MP layer (equation
Pretraining Architecture.
2
1
Σ h
(1)
MLP(()= = x + Ag Bx)
(2)
Where A and B are parameter matrices and is the element-wise SELU function (Klambauer et al. 2011 7
To train our backbone, we apply LoRA (Hu et al.) 2021] adapters on both the vision encoder and LLM modules. We optimize the contrastive loss lemperature hyperparameter 7 during pretraining.
Instruction Fine-tuming Architecture. After pretraining, ABC can project queries consisting of images or text into a shared embodding space. However, control the representation of visual embeddings, we must project queries consisting of images and text. instructions into the embedding space. To accomplish this, train low-rank LoRA adapter on the LLM component. The LoRA hooks of the instruction adaptor are attached to the MLP and attention layers in each decoder block of the LLM backbone, we freeze all other weights during instruction fine-tuning. The adapter contrastively aligns image and instruction queries with text candidates embedded using the pretrained model. For queries containing both an image and text instruction, we adopt the instruction template from our VLLM backbone (Wang et al. 2024b):
<vision_stare <image simage_tokens> tokens> <vision_ent end> <im_start> Instruction: (text) <im_endd end>
(3)
We replace <image_tokens> with tokens output by vision encoder. prision_start: and <vision_ens are special vocabulary tokens used to denote the start and end of the sequence image tokens. Similarly, <im_start> and <im_enda are used to denote the start and end text tokens. If no natural language instruction used, the last token <vision_end> We do not propagroo through text candidato embeddings, only the queries containing image and instruction. This ensures that our image-text queries share the same features text and images embedded using the pretrained model. We can easily alternate between embedding with or without instructions by disabling the instruction fine-tuned LoRA adapter. We freeze the temperature during fine-tuning.
3.2 Pretraining Data
4

---

Published in Transactions on Machine Learning Research (08/2025)
To create pretraining dataset we employ negative mining on Conceptual Captions (Sharma et al.) 2018). We derive the mined negatives for our dataset as follows: (1) We do a small pretraining run using only in-batch negatives. (2) We use the resulting model to calculate similarity scores between all images and captions in our pretraining dataset. This approach avoids circular dependence on a third-party embedding model to train our embedding model. There- fore, it is easily extensible modalities where existing embedding model is not publicly available. (3) To prevent our negatives from being too similar to our positive samples, we set a similarity thresh- old e E [0, 1]. We only sample negatives that have a similarity score of at most e times the similarity score of the correct candidate. We randomly choose our mined hard negatives from the 100 candidate captions below the threshold. This results in the lext candidates shown in Figure g The mined text negatives are clearly relevant, but the correct caption is still the best answer.
hapey in hat hokding o gloae wine and peeking out
red satdd
Hipatrr oar in het and with pleasee
cot wcaring somo hot lingging
Lewoou
Figure 3: The positive caption (green) is the best caption for the image. The mined negatives (red) are relevant but not the best choice.
In our pretraining run, each batch consists of N image queries (without instructions) and M text candidates. We include M M-N N mined text negatives in each batch. Therefore, each image query has M corresponding mined negatives. Our pretraining loss function is given by Equation 44).
NN'
N
exp(J(ai) (((((()))()))/1))
preetrain =
Σ log
(4)
N Σラ (exp( (exp(((x)) + + exp( ((a))-((k))
For given query qi n. is the kn mined hard negative candidate and ct is its positive candidate, is the temperature hyperparameter used to scale the loss. n is a mined negative for æ, when i = j, otherwise it acts as a regular in-batch negalive. The embeddings are unit normalized before loss is computed. We scale the loss by No the number of image queries in the batch.
3.3 Instruction Fine-tuning Data
To create our instruction fine-tuning dataset we use Visual Genome (Krihhna et al. 2016 a dataset com- prised of images with captioned bounding boxes. When choosing which bounding boxes to use for each image, we filter by the size of the bounding box (width x height). We exclude the 5 largest bounding boxes, as we find they often do not represent specific objects or aspects the scene. For each image, we randomly choose 4 bounding boxes and their respective captions. We then prompt GPT-4o (OpenAT 2024] to create instructions corresponding to each bounding box caption. We sample multiple captions from each image so that they can be used negatives for each other during instruction fine-tuning. This ensures that each image query has multiple realistic captions in its batch. Therefore, utilizing the instruction is required to choose the correct tezt candidate unambiguously. We provide the prompt and generation settings used to create our instruction fine-tuning datasel in Appendix A We instruction fine-tune using exclusively in-batch negatives. However, we group all queries that contain the same image into the same batch. This ensures that queries always have multiple relevant text candidates.
4 Experiments
We evaluate two aspects of the model. As controlling a poor-quality embodding is useless, we first assess the quality of embeddings output by ABC. We evaluate our model on a variety of zero-shot retrieval and classification tasks (Section 4.3). We how variable resolution capabilities (Section 4.4), the settings for 7 when contrastively training VLIM (Section and VLLM backbone choice (Section 4.6) effect embedding quality. Secondly, we measure ABC's ability use natural language to control visual embeddings on several visual question answering tasks (Section 4.7 Motivated by shortcomings in VQA benchmarks, we construct
5

---

Published in Transactions on Machine Learning Research (08/2025)
CtrlBench. A benchmark designed to require the model to jointly use the image and instruction to retrieve the most appropriate caption.
4.1 Training Settings
We initialize ABC using Qwen2-VT-7B (Wang et al. 2024b). We train our negative mining model using only in-batch negatives with batch size of 256 for 1000 steps. We set 0.95 and sample 7 mined negatives for each image. Mined negatives are sampled across the pretraining dataset. We pretrain using batches of 512 image queries and 4096 text candidates sharded across 8 NVIDIA A A100-SXM4-80GG GPUs (Qu et al.) 2021) for 4000 steps. We use a LoRA adapter with a rank of 64 and a fixed alpha of 128. We limit the number of tokens output by the vision encoder to 512 during training. For our optimizer, we use AdamW (Loshchilov & Hutter 2019) with a learning rate of x 10-5, betas of 0.9 and 0.999 and a weight decay of 10 3, We warm- up for 3% of training steps and initialize the temperature as 7 10-2 In our instruction fine-tuning stage, use a lower rank LoRA adapter. We set the rank and alpha to 16 and 32, respectively. Our instruction fine-tuning stage can be short, as our VLLM backbone is already instruction fine-tuned. Therefore, we only instruction fine-tune for 100 steps. Each batch contains 128 unique images, with each image appearing four times, pairod with a different instruction and a corresponding positive text candidate.
Modifications to save VRAM. Due to our use of ToRA Hu al. (2021), ; the VRAM requirements for gradients and optimizer state is relatively low. Consequentially, the model activations used by autograd for the backward pass account for most of VRAM used during training. To address this, make aggressive use of activation checkpointing, recomputing the activations of each decoder block during the backward pass. Furthermore, we modify the VLLM backbone to skip the logits and cross-entropy loss computation. With the large vocabulary size of modern LLMs, the output layer has become the most memory-intensive layer (Wijmans et al., 2024). As we only use the hidden state for calculating embeddings, the logits tensor is unnecessary. We find that this simple change saves up to GB of memory per device, allowing us to use to significantly larger batch size.
4.2 Inference Settings
When evaluating ABC we adjust the vision encoder to output a number of tokens corresponding to the resolution of the image. We rescale images to the nearest resolution that fits our vision encoder, which has a 14 x 14 pixel patch size [Wang al., 2024b). For very high resolution images, we cap the number of visual tokens used to encode image to 1024, down-scaling the image accordingly. When evaluating models that supports variable resolution, we vary their token budgets similarly (Jiang et al.] 2024 Zhang et al. 2025). However, when we attempt to increase token budgets in CLIP models using the same method, we find that it degrades performance (Appendix G We find that these models perform best when evaluated using their training resolution. Therefore, we do not scale the token budget when evaluating CLIP models. This includes instructionfinttune CLIP models such as Uniir and MagicLens (Wei et al., 2023
Zhang et al. 2024).
MSCOCO (5K test set)
Flickr30K (1K test set)
Tmage + Text
Text + Image
Image + Text
Text + Image
Model
R
R@5
R@10
R@1
R
R@10
R@1
Ra5
R@10
R@1
R@5
R@10
CLIP Radford et al.
2021
58.4
81.5
88.1
37.8
62.4
72.2
88.0
98.7
99.4
68.7
90.6
95.2
ALIGN
Jia et al. 2021
58.6 6
83.0
89.7
45.6
69.8
78.6
88.6
98.7
99.7
75.7
93.8
96.8
FLAVA
Singh et 2022
42.7
76.8
38.4
67.5
67.7
94.0
65.2
89.4
FILIP * Yao
2221
61.3
$4.3
90.4
45.9
70.6
79.3
89.8
99.2
99.8
75.0
93.4
96.3
CoCa
Yu et
2022
66.3
86.2
91.8
51.2
74.2
82.0
92.5
99.5
99.9
80.4
95.7
97.7
OpenCLiP5C Cherti al.
2023a
67.3
86.9
92.6
51.4
74.9
83.0
92.9
99.3
99.8
79.5
95.0
97.1
111--22-CII-JI (Sun et al.
2023
68.8
87.8
92.8
51.1
75.0
82.7
93.9
99.4
99.8
78.8
94.2
96.8
ABC (ours)
69.2
87.9
93.2
47.6
72.1
80.6
90.7
99.0
99.5
74.6
92.6
95.45
Table 1: Comparison of retricval performance on MSCOCO Lin et al. (2015) and Flickr30K Plummer et al.
(2066) datasets (Karpathy split). Best performance is bold, second best is underlined. indicates closed- weight model.

---

Published in Transactions on Machine Learning Research (08/2025)
CLIP OpenCLIP
SigLIP
UnilR
MagicLens
BLIP2
MMRet
ABC (ours)
Classification (9 tasks)
ImageNet-1K Ruseakovsly
et al.
2015
55.8
03.5
45.4
58.3
48.0
10.3
49.1
71.2
HettffuMemmss Klela ot al A21
51.1
51.7
47.2
56.4
49.0
49.6
51.0
52.1
VOC2007 Everingham ol
50.7
52.4
64.3
66.2
51.6
52.1
74.6
81.4
SUN397 Xiao et al (2000
13.4
08.8
39.6
63.2
57.0
34.5
60.1
71.8
Place365 Lopez- Ciffentes et
2020
28.5
37.8
20.0
36.5
31.5
21.5
35.3
40.7
ImageNet- A Dillonga el nl.
2020
25.5
14.2
42.6
9.8
8.0
3.2
31.6
49.4
ImageNett-R Hendrycks
0.72
83.0
75.0
66.2
70.9
39.7
ObjectNet Barhu
2019
40.3
32.2
Country-211
Radlord
2021
16.8
14.2
11.3
6.2
Classification
13.7
48.8
43.2
44.5
39.3
26.0
47.4
60.0
Table 2: Zero-shot classification results on MMEB Jiang et al., 2024).
We compare with pretrained CLIP
models (Radford et al. 2021 Cherti et al.) 2023a instruction finetuned models derived from CLIP (Wei
ct al. 2023 Zhang al. 2024) and other L.L.M backbone approaches
(Li et al. 2023 Zhou et al. 2024).
4.3 Zero-shot Retrieval and Classification
We first evaluate the quality of ABC embeddings, as controlling low-quality visual embeddings useless. To test embedding quality, use zero-shot tasks: retrieval from pool of candidates and image classification. For retrieval, we use MSCOCO (Lin et al. 2015) and Flickr30K (Phummer et al., 2016). For image classification, we use MMEB Jiang et al. (2024), which provides splits of many standard image classification tasks, such as ImageNet-1K (Russakovsky et al. 2015), VOC-2007 (Everingharn et al.), and ObjectNet (Barbu el al. 2019). We note that the classification labels in MMEB are short, often only one or two words. This problematic for mmage-captioning models that have been trained on full sentences Yu let al. 2022). To alleviate this issuc, we use Radford in sentence template. We use "A photo of {label}. as our template for all classification evaluations.
(2021)'s technique of embodding classification labels
In Table ! our model demonstrates strong image-to-text retrieval capabilities, achieving competitive perfor- mance to models that have been contrastively trained with hundreds of GPUs and massive batch sizes (Sun et al. 2023 Cherti et al. 2023a). We achieve the best performance in MSCOCO image-to-text retrieval. Comparatively, our text-to-image retrieval is weaker, as we under-perform both EVA-02-CLIP-FE+ and Open Averaged across all classification tasks (Table 2], we achieve 11.2% better accuracy than the next-best best model. On ImageNet- (Russakovsky et al. 2015). ABC has 7.7% better zero-shot accuracy than OpenCLIP. ABC's performance established benchmarks such MSCOCO and ImageNet-1K indi- cate that the pretrained model's embeddings useful. In Section 4.7 evaluate the fine-tuned adapter's ability to embed images with text instructions into the pretrained model's embedding space.
1.0
ImageNet
ImageNel-A ObjectNet
0.8
400
Accuracy
0.6
300
0.4
ImageNet
100
ImageNet-A
ObjectNet
n
24
Pixell Image
20
2
24
0.0
2
2
ObjectNet (Barbu al. 2019) contains significantly higer resolution than ImageNet. (Russakovsky et. al.) 2015), which is downscaled led. (Right) Accuracy benchmarks when increasing the pixel count (tokens) used by vision encoder when creating embeddings.
Figure (Left) Pixel distributions of benchmarks.

---

Published in Transactions on Machine Learning Research (08/2025)
4.4 Image Resolution and Embedding Quality
Our VLLM backbone, Qwen2-VI-7B (Wang et al. 2024b), supports image imputs with variable resolution. This allows the user to effectively trade-off image resolution with inference speed by adjusting the number of tokens output by the vision encoder. We cxamine how this trade-off influences embedding quality for image classification. We find that performance on certain lasks, like ObjectNNet (Barbu et al. 2019) and ImageNet- A (Djolonga et al., 2020), strongly correlate with the resolution used in the VLLM vision encoder (Figure 4 On average, ObjectNet images have 13 times more pixels than those from ImageNet- 1K. When the number of tokens produced by the vision encoder is scaled up, we see a large improvement on ObjectNet, with accuracy increasing by 23.4%. However, lower resolution benchmarks like ImageNet-1KK do not benefit nearly as much from scaling resolution. Notably, smartphone camera takes photos with significantly higher resolution than images in ImageNet-1K 1K or ObjectNet, by default (Apple Inc. 2024). This motivates the need for benchmarks containing larger resolution images. Existing low-resolution benchmarks may underestimate the capabilities of models that can natively utilize high-rosolution images. Furthermore, we find that correctly classifying the "natural adversarial examples" of ImageNet-A (Djolonga et al.) 2020) is largely a function of resolution. Our accuracy on ImageNet-A A increases from only 12% to 47.5% simply by scaling the resolution used by the vision encoder during evaluation.
4.5 Mined Negatives and Temperature
Uncazle Irilsdzation 0.013
Protr sining Randumized Pretr sinin Negatiees
1.0
Cornec Inttiaation (2)
0.05
Gradent kom
Temperaturs
0.00
0.6
0.02
200
0.2
with Mined Neganhess
250
6000
1000
-0.02
2300
7500 12507 15050
2500
7500
10000
12500
15000
Shep
Step
Step
Figure 5: Visualization of temperature (7) and its effects on training dynamics. (Left) The effect of tem- perature initializations gradient norm while training negative mining model. (Middle) Temperature behavior during pretraining, without and without mined negatives. (Right) Validation accuracy during pretraining, with and without mined negatives.
The setting of the temperature hyperparameter (7) is known to be crucial for contrastive training (Jia et al., 2021). However, is often treated as a fixed hyperparameter (Jiang al. 2024 Wei et al.) 2023). A large fixed setting of results in a significantly worse model (Wang & Liu 2021]. whereas a too small setting can result in training instability (Figure 5). Therefore, we find that optimizing throughout the pretraining process is crucial. In shorter runs like training our negative mining model, we find that shows that both the initialization and optimization of r is crucial. For our negative mining model, we find T = 0.07 to be a temperature initialization that is both stable and performant.
Mined negatives and temperature. Table 3 we ablate over the number of mined hard negatives used during pretraining. Each pretraining run consists of 500 steps with a batch size of 256. Introducing 6 mined hard negatives improves top-1 image to text retrieval by 9.5% and top-1 text to image retrieval by 9.1%. Furthermore, we find that the mined negatives essential for training stability. Figure 5 shows pretraining run with a batch size of 128 images queries and 1024 text candidates. We compare two runs: one
# Negatives
Image + Text Text + Image
0
50.4
35.7
3
58.4
44.0
6
59.9
44.8
Table 3: Top-1 retrieval performance on MSCOCO with varying number of mined hard negatives.
8

---

Published in Transactions on Machine Learning Research (08/2025)
using randomized negatives sampled uniformly randomly from dataset and the other our mined hard negatives. We find that if randomized negalives are used instead of our mined negatives, the optimizer tends to push r very close to O. This results in numerical instability, loss spikes, and eventually catastrophic failure. When our mined negatives are used in our pretraining, the temperature stabilizes at a very reasonable value of around 0.03.
4.6 VLLM Backbone
We explore how the choice of VLLM backbone has effect the quality of our multimodal embedding. We ablate over 3 popular choices of VLLM backbone, all at approximately the 8 billion parameter scale: Owen2-VL-7B VL-7B (Wang et al., 2024b), our chosen backbone. InternVL-2-SB from the internVL family of VLMs (Chen et al. 2024). Lastly, its LLM component. We pretrain each model for 1000 stops with a query batch size 128 and candidate
(Liu et al. 2024a) with Mistral-7B (Jiang et al., 2023) as
batch size of 1024.
Model
Accuracy
MMLU
MMBenchen
LLaVA-NeXT
62.6
35.3
68.7
InternVL-2-SB
65.9
51.8
Qwen2-VI-7B
70.2
54.1
83.0
Table 4: Ablation over VLLM backbone choice.
Table 4 shows the validation accuracy of ABC with different backbones. We find that our backbone choice, two standard
Qwen2-VL-7B. produces the best results. We also note each backbone's performance generative VLLM benchmarks: MMMU Yue et al. (2024) and MMBench Liu et al. (2024b). We find that performance after contrastive training strongly correlates with the performance of the backbone on generative tasks. This indicates that training better VLLMs naturally results in better backbones for our model.
4.7 Controlling Embeddings using Natural Language
5 ABC's performance on the VQA split of MMEB. In just 100 training steps using instructions, model surpasses MMRet (Zhou al.) 2024) VQA, which trained with millions of instructions. Interestingly, CLIP derivatives fine-tuned with instructions such MagicLens (Zhang et al. 2024 and UniIR Wei et al. 2023) do not perform significantly better than their respective baselines, despite MMEB providing instructions for each task. This evidences that encoder-only embedding models struggle to effec- tively utilize natural language instructions (Jiang et al. 2024).
Problems with VQA. A VQA benchmark should require the use of both modalities together, and using only the image the text should be insufficient to accomplish the task. We note that ensuring this property requires inspecting not only the queries but the candidate pool well. Consider Figure 0 example from
CLIP
OpenCLIP
SigLIP
UnilR
MagicLens
BLIP2
MMRet
ABC (ours)
VQA (10 tasks)
OK-VQA Marino et al.
2019
7.5
11.5
2.4
25.4
8.7
28.0
A-OKVQA Schwenk
2022
3.8
1.5
8.8
2.9
3.2
DacVQA Mathew
12.6
Infographics VQA Matlew
202
4.6
1.6
2.7
4.6
2.0
ChartQA Masry ct 2029
1.4
1.5
0.5
11.7
VisualWW Zhu el al. 2016
4.0
2.6
1.2
14.5
2.5
1.3
9.0
25.6
ScienceQA La et al. 2122
9.1
10.2
7.9
12.8
5,2
6,8
23.3
26.3
VizWiz Gurari et al 2013
8.2
6.6
2.3
24.3
1.7
4.0
25.9
29.4
GQA Hudsou & Manning
2019
41.3
52.5
57.5
48.8
43.5
9.7
41.3
60.1
Text V Qiiigg et al. 2019
7.0
10.9
1.0
15.1
4.6
3.3
18.9
35.4
All VQ4
9.1
10.9
6.4
16.2
8.3
4.2
18.4
31.0
Table 5: Zero-shot VQA results MMEB
JJiang et al.
2024
Best is bold, second best is underlined.

---

Published in Transactions on Machine Learning Research (08/2025)
the Visual7W task in MMEB. When the candidate pool is examined, it is clear that there is only one answer that is plausible, given the image. Therefore, the instruction is not required to accomplish the VQA task. This a common pitfall when adapting open-ended generative VQA benchmarks into fixed candidate pool embedding tasks.
CtrlBench. We construct CtrlBench, an instruction-coottrlleed retrieval benchmark. CtrlBench has similar format to the Flicker30K (Plummer et al. 2016) test split. However, instead of each image having multiple valid captions, instruction from which the model can infer the most relevant text candidate. Therefore, CtrlBench tests both retrieval and instruc- tion following capabilities. To construct CtrlBench; we sample a 1000 images from ADE20K (Zhou al. 2017). To create 5000 in-
Query
instead provide an
Where this scene?
Instraction
Text Condidates
structions and text candidate pairs, we generate instructions for
Boachside
each image, each corresponding to distinct aspect of the image. To accomplish this, we use Gemini 2.5 Flash (Team et al. 2025). We provide the exact prompt that we use in Appendix B As cach image has associated captions in the candidate pool, model that cannot utilize instructions can (in expectation) achieve at most 20% R@l on the benchmark. Therefore, CtrlBench requires the embed- ding model to use the provided instructions disambiguate visual queries. We deduplicate captions from the dataset to prevent ambi- guity in text candidates, resulting in CtrlBench being slightly smaller than 1000 images.
A couch
ninjo
Elephants
Figure 6: An example of a. VQA task from the Visual7W (Zhu et al. 2016)
split of MMEB (Jiang al. 2024).
Model
Rt
Ro9a5
R@11
UnilR Wei
2023
0.9
2.59
4.30
MagieLens Zhang
2024
17.0
37.33
48.47
VLM2Vec Jiang
2021
2.95
8.19
12.36
GME-Qwent VL-21 Zhang et al.
2025
20.92
46.33
57.27
GME- SIEE--wen2VL--- VL-BB Zhang el a.
2125
24.75
54.48
65.27
ABC
(pretzained)
11.71
32.18 18
12.69
ABC (finetume)
50.94
78.43
87.47
Table 6: Rocall@K on CtrlBench.
Table e shows the performance of multimodal embedding models CtrlBench. We compare with 2 in- struction fine-tuned CLIP derivatives: UnilR Wei et al. (2023) and MagicLens Zhang et al. (204)) as well as VLM2Vec Jiang et al. (2024) and GME (Zhang et al. 2025), two concurrent works that adapt VLLMs into multimodal embedding models. We find that neither the CLIP-based architectures have above 20% R@I on CtrlBench, the performance that indicates that the model is non-trivially utilizing instructions. We find that VLLM architectures are better at Instruction-controlle retrieval. ABC and VLM2Vec have R@1 above 20%, proving that they can non-trivially utilize instructions.
5 Discussion and Related Work
5.1 Distribution Overlap in Training and Evaluation
Throughout our experiments, we found that several training data sources overlap with commonly used benchnarks. Several benchnark are constructed from the same underlying data source as training data or directly derived from training data (Barbu et al. 2019 Russakovsky et al. 2015 Plummer et al, 2016 Lin et al., 2015). This makes evaluating "zcro-shot" capabilitics difficult. To the author's knowledge, our pretraining data (Sharma al. 20018) and instruction-finttunnn data Krishna al. (2016) does not overlap with any benchmarks evaluate against, and all evaluations (Tables
and 6 out-of-distributtion.
10

---

Published in Transactions on Machine Learning Research (08/2025)
5.2 Decoupling Pretraining and Instruction Fine-tuning
We find that adapting an instruction fine-tuned VLLM to instructions is relatively cheap. Our instruction fine-tuning completes in less than an hour couple A100 GPUs. This allowed us to quickly iterate instruction fine-tuning stage, without pretraining from scratch. Comparatively, training a VLLM to create high-quality embeddings is much more resource intensive. Our pretraining run barely fits into the 640 GB of VRAM provided by a single A100 node, and takes several days to complete. Furthermore, our work indicates that these models could benefit substantially from further scaling (Appendix Dj.
5.3 Alternative Approaches
MagicLens (Zhang al. 2024) and UniTR (Wei et al. 2023 are multimodal embedding models that fuse CLIP embeddingg. By combining multiple CLIP models, they take image-text pairs and align them with image-text pairs using contrastive loss and instruction finetuning. Due to late fusion of modalities, these architect ures struggle to use instructions to modify representations (Jiang et al. 2024 Recently, VLM2Vec Jiang et al. (2024) and GME (Zhang et al., 2025) have also adopted contrastive training on VLLM backbones to produce multimodal embeddings. However, these approaches make extensive use of annotttnd instruction datasets, with hundreds of thousands of labeled examples used during train- ing. This limits the scalability and adaptability of their method.
5.4 Important Factors during Pretraining
Prior work has largely focused on what architectural adaptations the make the VLLM convert into embedding model. These include the attention mask, how the embedding is pooled and adapter architec- ture [BchnamChader et al., 2024 Loe et al. 2024 Jiang et al., 2024). Throughout that most of these choices are often interchangeable or only produce marginal improvements (Appendix [ However, we find that many of the crucial factors when training CLIP models are also important when adapting VLMs. In particular, well-chosen data, batch size and number of samples seen during training are all important factors (Appendix D), just like with CLIP models [aare et al. 2023 Cherti al. 2023a
experiments, we find
6 Conclusion
We introduce ABC, a multimodal embodding model that leverages a VLLM backbone to control image rep- resentations via natural language instructions. It achieves strong zero-shot results on variety of multimodal (asks, spanning retrieval, classification, and VQA. ABC consists of a multi-stage training process, which íso- lates the computationally expensive contrastive pretraining from wight instruction finetuning phase, We explore what factors are the most crucial when adapting VLMs to output multimodal embeddings. In particular, we find that vision encoder resolution, contrastive loss lemperature and VLLM backbone choice are all important factors. Lastly, we design CtrlBench to measure our model's ability to use instructions to accomplish subtle natural Sanguage-guides retrieval tasks.
7 Broader Impact Statement
The ability to exhibit greater control over visual representations, especially those used information retrieval systems, has potential for both positive and negative impacts. On the positive side, it allows users greater control over visual information retrieval. This capability is particularly useful for that can struggle to directly use and interpret images. However, introducing natural language control over visual representations has potentially negative impacts. We highlight two potential harms: (1) Accidental introduction of bias due to natural language conditioning. Large language models exhibit. similar biases to humans (Guo et al., 2024). LLM representations used in information retrieval may adopt these biases. Furthermore, allowing users natural language control over visual representations may result in users acci- dentally introducing their own biases into visual retrieval. (2) Malicious control over information retrieval. Introducing natural language control visual representations creates entry-point to filter modify
users
11

---

http///ppp222222222//2 Published in Transactions on Machine Learning Research (08/2025)
what information is retrieved by downstream users. This is alarming in the internet age, where control over information access is powerful tool for authoritarian control.
References
Mohammad Aliannejadi, Hamed Zamani, Fabio Crestani, and W. Bruce Croft. Asking clarifying questions in open-domain information-scckinng conversations. In Proceedings of the 42nd International ACM SIGIR Conference Reseorch and Development in Information Retrieval, SIGIR '19, pp. 475 484. ACM, July
2019. doi: 10.1145/330 /////3331141331111 188.33313665
URL http://áx: bettp:/dxx.do..ooo 10. 1145/3331184 3331166
Chen Amiraz, Florin Cuconasu, Simone Filice, and Zohar Karnin. The distracting effect: Understanding irrelevant passages in rag, 2025. URL
06914
Apple Inc. iphone 16- tech spocs. https://support.apppe wtt.lee..ooeeeeeeeeeeeeee http:////appppppppppppoooooooo/a 2024. Accessed: 2025-01- 22.
Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz. Objectnet: A large-scale bias-controlled dataset for pushing the lim- its of object recognition models. In H. Wallach, Н. Larochelle, A. Beygelzimer,
d' Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Cur- Associates, Inc., 2019. URL httpss://procedd www.iinnnnnnnuus..m wwwwwps.cc/ppappe.ppppp httpp//cccccppppppeeeeepppeeeeep filles/paper2211//11ee pdf
Parishad BehnamGhader, Vaibhav Adlakha, Marius Mosbach, Dzmitry Bahdanau, Nicolas Chapados, and
Siva Reddy. Llm2vec: Large language models are secretly powerful text encoders, 2024. URL https:
(rrzxi..rrg/ab//2000 05961
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Concept 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. CoRR, abs abs/2102.08981, 2021.
URL https:
//arxiv.org/ www..orgg/bb////oo/e abs/
Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistie tasks, 2024. URL htttp:://axiv..t /bs//
2312.14283
Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. In 2023 IEFE/CVF Conference on Computer Vision and Pattern Recognation (CVPR). IEEE, June 2023a. doi: 10. 111109//V115229.22225
URL http://dx http::/dxx.oii.. wtt.oooog/0..10//////0
CVPR52729. 2023.0022222222 002766
Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive banguage-image lcarning. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2818 2829. IEEE, June 2023b. doi: 1tg/100.109//CIPR22229 00276
URL http:/ doi.
Josip Djolonga, Jessica Yung, Michael Tschannen, Rob Romijnders, Lucas Beyer, Alexander Kolesnikov, Joan Puigcorver, Matthias Minderer, Alexander D' D'mourr Dan Moldovan, Sylvain Gelly, Neil Houlsby, Xiaohua Zhai, and Mario Lucie. On robustness and transferability of convolutional neural networks. CoRR, hos/2207.085555 2020. URL. ttetps://arxiv org/ abs// waab///000.00e0
M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. PASCAL Visual Object Classes Challenge 2007 (VOC2007) Results. http:/ htpp::/wwww htww///wwppppo wwwcall httgg///calllggs//////////oo////
The
network.org/calllnnges. VOC/voc2007/ VOC/voc2007/wwttsshpp wwwkssopp/iddx...i URL "http://hose" http://tsstttrrt...oee/oo robots. ox ac uk/ pascal/VOC/voc2207
12

---

Published in Transactions on Machine Learning Research (08/2025)
Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, Fyal Orgad, Rahim Entezari, Giannis Daras, Sarah Pratt, Vivek Ramanujan, Yonatan Bitton, Kalyani Marathe, Stephen Mussmann, Richard Vencu, Mchdi Cherti, Ranjay Krishna, Pang Wei Koh, Olga Saukh, Alexander Ratner, Shuran Song, Hannanch Hajishirzi, Ali Farhadi, Romain Beaumont, Sewoong Oh, Alex Dimakis, Jenia Jitsev, Yair Carmon, Vaishaal Shankar, and Ludwig Schmidt. Datacomp: In search of the next generation of multimodal datasets, 2023. URL https://arxiv, arggabs//304/ 14108
Luyu Gao, Yunyi Zhang, Jiawei Han, and Jamie Callan. Scaling deep contrastive learning batch size under memory limited setup, 2021. URL
06983
Arnav Gudibande, Eric Wallace, Charlie Snell, Xinyang Geng, Hao Liu, Pieter Abbeel, Sergey Levine, and Dawn Song. The false promise of imitating proprietary lims, 2023. URL https://arxiv.org// abwpg//bb/2305.
15717
Yufei Guo, Muzhe Guo, Juntao Su, Zhou Yang, Menggiu Zhu, Honglei Li, Mengyang Qiu, and Shuo Shuo Liu. Bias in large language models: Origin, evaluation, and mitigation, 2024. URL https: btttps//aaxiv..or
abs/2411. 10915
Danna Gurari, Qing Li, Abigale J. Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P. Bigham. Vizwiz grand challenge: Answering visual questions from blind people, 2018. URL https:
//arxiv. www. httg/abs/1802..00 00tp/1888200018
R. Hadsell, S. Chopra, and Y. LeCun. Dimensionality reduction by learning an invariant mapping. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recogrition (CVPR'06) volume 2,
pp. 1735 1742, 2006. doi: (.1100//CVP.2000..000
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justín Gilmer. The many faces of robustness: A critical analysis of out-of-distribution generalization, 2021. URL https://arxii:
org/ htgg/abb/0006..o 16241
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021. URL https://arxiv. org/
abs/2106.090 096855
Drew A. Hudson and Christopher D. Manning. Gqa: A new dataset for real-world visual reasoning and wttpp/wwwww.../bb///200.....
compositional question answering, 2019. URL
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Ducrig. Scaling up visual and vision-language representation learning with noisy text supervision. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 4904-4916. PMLR, 18 24 Jul 2021. URL htttp::/ mlr pttpss/v39/jiaeee..eee
html
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023. URL httts:///axiii org//ogg////o.ooo aba/abaa///// /ttes/23100000
Ziyan Jiang, Rui Meng, Xinyi Yang, Semih Yavuz, Yingbo Zhou, and Wenhu Chen. V1m2vec: Training lasion-language models for massive multimodal embedding tasks, 2024. URL https:/ http///axvvvv.// .rgg org/abs/ abs/
2410. 05160
Kimiya Keyvan and Jimmy Xiangji Huang. How approach ambiguous queries conversational scarch: A survey of techniques, approaches, tools, and challenges. ACM Comput. Surv., 55(6), December 2022. ISSN 0360-0300. doi: 10.1145/3534966 URL http:tp::://oo...rr//...111/////
13

---

Published in Transactions on Machine Learning Research (08/2025)
Douwe Kiela, Hamed Firooz, Aravind Mohan, Vedanuj Goswami, Amanpreet Singh, Casey A. Fitzpatrick, Peter Bull, Greg Lipstein, Tony Nelli, Ron Zhu, Niklas Muennighoff, Riza Velioglu, Jewgeni Rose, Phillip Lippe, Nithin Holla, Shantanu Chandra, Santhosh Rajamanickam, Georgios Antoniou, Ekaterina Shutova, Helen Yannakoudakis, Vlad Sandulescu, Umut Ozertem, Patrick Pantel, Lucia Specia, and Devi Parikh. The hateful memes challenge: Competition report. In Hugo Jair Escalante and Katja Hofmann (eds.); Proceedings of the NeurIPS 2020 Competition and Demonstration Track, volume 133 of Proceedings of Machine Learning Research, pp. 344 360. PMLR, 06 12 Dec 2021. URL httpp::/pproceeet
press/v133/ httss//33//iiela22 http//iieaallaa... html
Gangwoo Kim, Sungdong Kim, Byeongguk Jeon, Joonsuk Park, and Jaewoo Kang. Tree of clarifications: Answering ambiguous questions with retrieval-angmented large language modcls, 2023. URL https:
//arxiv. www.:or///aba///1/// 14696
Günter Klambauer, Thomas Unterthiner, Andreas Mayr, and Sepp Hochreiter. Self-normalizing neu- ral networks. In 1. Guyon, U. Von Luxburg, S. Bengio, Н. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Newral Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL
neurips. ctwpspercppprrrrpe piles///ppeee/////i///
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanic Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Fei-Fei Li. Visual genome:
Connecting language and vision using crowdsourced dense image annotations, 2016. URL https://
Dexiv.org/eee wrgg/abs/6022.7333333
Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Calanzaro, and Wei Ping. Nv-embed: Improved techniques for training Ilms as generalist embedding models, 2024. URL 17428
https:///rrt:: http://xxvvvorggbbbaaseesa
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping languag---nng pre-training with frozen image encoders and large language models, 2023. URL http::::/aaxii..or///attt wtw.org///bbb/reee 12597
Tsung- Yi Lin, Michael Maire, Serge Belongic, Lubomir Bourdev, Ross Girshick, James Hays, Pictro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dollár. Microsoft coco: Common objects in context, 2015.
URL
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, 2024a. URL https:// http:://xxxvvvoorrabbbb /t///bb//233 2aab..///3....m 03744
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. Mmbench: Is your multi-modal model an all-around player?, 2024b. URL htttss://arxi:i http:///axxvv.or//aaa// 06088
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. URL
Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In The 36th Conference on Neural Information Processing Systems (NeurIPS), 2022.
Alejandro
Marcos Escudero- Viñolo, Jesús Bescós, and Álvaro Martia-Martín. Semantic- aware scene recognition. Pattern Recognition, 102:107256, 2020. ISSN 0031-3208. doi: https:// 2oo.org/110.0116// pttppp......0010.....2. 10722. $00313203320306113
URL http:p//www...cccaaaaaae....ec/p
Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge, 2019. URL http:::tp//// pestps://arxive wttp:///.wr//aa//...0000000
14

---

Published in Transactions on Machine Learning Research (08/2025)
Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque Chartqa: A benchmark for question answering about charts with visual and logical reasoning, 2022. URL //apxiv////xivvv/// wwwpexivv...rreeeo abs/ 2203. 10244
Minesh Mathew, Viraj Bagal, Ruben Pérez Tito, Dimosthenis Karatzas, Ernest Valveny, and C. V Jawahar. Infographicvqa, 2021a. URL
12756
Minesh Mathew, Dimosthenis Karatzas, and C. V. Jawahar. Docvqa: A dataset for vqa document images, 2021b. URL httts://aaxiv..or wtgg/ab//0007.0000
Sewon Min, Julian Michael, Hannaneh Hajishirzi, and Luke Zettlemoyer. Ambigqa: Answering ambiguous open-domain questions, 2020. URL https://arxiy http://aaxxxv...gabbbaa wwtp://wr....bb///..00.40//
Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. Mteb: Massive text embedding bench- mark, 2023. URL http://awxii...gg/bbb2/220....
OpenAL. 4o system card, 2024. URL https://arxii: wwwp.xwwwwwrg/b///1100000 21276
Bryan A. Plummer, Liwei Wang, Chris M. Cervantes, Juan C. Caicedo, Julia Hockenmaier, and Svetlana Lazebník. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models, 2016. URL https://arxiv.. www.axiw..oggbb///eaaaee 04870
Yingqi Qu, Yuchen Ding, Jing Liu, Kai Liu, Ruiyang Ren, Wayne Xin Zhao, Daxiang Dong, Hua Wu, and Haifeng Wang. RocketQA: An optimized training approach to dense passage retrieval for open-domain question answering. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, pp.
Online, June 2021. Association for Computational Linguistics. doi: 10.18653/v vv /2021 maacc-mman...a........m.m 466. URL https://aallantholog..ooom org/ wtt.://00mm...mmmma.a....
naacl-main ma61-./6inn6///////
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision, 2021. URL attpps// /ttps:///xxivvve/ wwtp://ww..bbbbbb....oooo
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. Imagenet. large scale visual recognition challenge, 2015. URL
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kun- durthy, Katherine Crowson, Ludwig Schmidt,, Robert Kaczmarczyk, and Jenia Jitsev. Laion-5b: An open largo-scale dataset for training next generation image-text models, 2022. URL 0240.
Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kemneth Marino, and Roosbeh Motlaghi. A- okvqa: A benchmark for visual question answering using world knowledge, 2022. URL / org/ ab///2bb/222.... 01718
Piyush Sharma, Nan Ding, Scbastian Goodman, and Radu Soricut. Conceptual captions: cleanod, hyper- nymed, image alt-text dataset for aulomatic image captioning. In Annual Meeting of the Association for Computational Linguistics, 2018. URL httppp://ap...emmt..cccooo..... org/ http://ooossiii11111.
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read, 2019. URL https://arxiv wwtph//wwwwr//////....... 08920,
Amanprect Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojeiech Galuba, Marcus Rohrbach, and Douwe Kiela. Flava: A foundational language and vision alignment model, 2022. URL
https://arxin httppa//aaxiv.or//abb/////// org/abs//////ee.eeooooeee/////ge /ttps///12..0.0 04482
15

---

Published in Transactions on Machine Learning Research (08/2025)
Ivan Stelmakh, Yi Luan, Bhuwan Dhingra, and Ming-Wa Wei Chang. ASQA: Factoid questions meet long-form answers. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 8273 8288, Abu Dhabi, United Arab Emirates, www......mpmmaa......m.m
December 2022. Association for Computational Linguistics. doi: 20.18653/v1/ 20066//v1/222..o//
URL
httpss//acllathholo0 org. org. 2022. mmmlp-main.5660
Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip at scale, 2023. URL
/2303. 15389
Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean- -ean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Jo- han Schalkwyk, Andrew M. Dai, Anja Hauth, Katie Millican, David Silver, Melvin Johnson, loannis Antonoglou, Julian Schrittwieser, Amelia Glaese, Jilin Chen, Emily Pitler, Timothy Lillicrap, Angeliki Lasaridou, Orhan Firat, James Molloy, Michael Isard, Paul R. Barham, Tom Hennigan, Benjamin Lee, Fabio Viola, Malcolm Reynolds, Yuanzhong Xu, Ryan Doherty, Eli Collins, Clemens Meyer, Eliza Ruther- ford, Erica Moreira, Kareem Ayoub, Megha Goel, Jack Krawczyk, Cosmo Du, Ed Chi, Heng- Tze Cheng, Eric Ni, Purvi Shah, Patrick Kane, Betty Chan, Manaal Faruqui, Aliaksei Severyn, Hanzhao Lin, YaGuang LI, Yong Cheng, Abe Ittycheriah, Mahdis Mahdich, Mia Chen, Pei Sun, Dustin Tran, Sumit. Bagri, Balaji Jakshminarayaaaan, Jeremiah Liu, Andras Orban, Fabian Güra, Hao Zhou, Xinying Song, Aurelien Boffy, Harish Ganapathy, Steven Zheng, HyunJeong Choe, Ágoston Weisz, Tao Zhu, Yifeng Lu, Siddharth Gopal, Jarrod Kahn, Maciej Kula, Jeff Pitman, Rushin Shah, Emanuel Taropa, Majd Al Merey, Martin Bacuml, Zhifeng Chen, Laurent El Shafey, Yujing Zhang, Olcan Scrcinoglu, George Tucker, Enrique Pi- queras, Maxim Krikun, lain Barr, Nikolay Savinov, Ivo Danihelka, Becca Roelofs, Anaïs White, Anders Andreassen, Tamara von Glehn, Lakshman Yagati, Mehran Kazemi, Lucas Gonzalez, Misha Khalman, Jakub Sygnowski, Alexandre Frechette, Charlotte Smith, Laura Culp, Lev Prolecv, Yi Luan, Xi Chen, James Lottes, Nathan Schucher, Federico Lebron, Alban Rrustemi, Natalie Clay, Phil Crone, Tomas Ko- cisky, Jeffrey Zhao, Bartek Perz, Dian Yu, Heidi Howard, Adam Bloniarz, Jack W. Rae, Han Lu, Laurent Sifre, Marcello Maggioni, Fred Alcober, Dan Garrette, Megan Barnes, Shantanu Thakoor, Jacob Austin, Gabriel Barth-Maron, William Wong, Rishabh Joshi, Rahma Chaabouni, Deeni Fatiha, Arun Ahuja, Gaurav Singh Tomar, Evan Senter, Martin Chadwick, Πla Kornakov, Nithya Ataluur,, Iñaki Iturrate, Ruibo Liu, Yunxuan Li, Sarah Cogan, Jeremy Chen, Chao Jia, Chenjie Gu, Qiao Zhang, Jordan Grim- stad, Ale Jakse Hartman, Xavier Garcia, Thanumalayan Sankaranarayana Pillai, Jacob Devlin, Michael Laskin, Diego de Las Casas, Dasha Valter, Connie Tao, Lorenzo Blanco, Adrià Puigdomènech Badia, David Reitter, Mianna Chen, Jenny Brennan, Clara Rivera, Sergey Brin, Shariq Iqbal, Gabriela Surita, Jane Labanowski, Abhi Rao, Stephanic Winkler, Emilio Parisotto, Yiming Gu, Kate Olszewska, Ravi Addanki, Antoine Miech, Annie Louis, Denis Teplyashin, Geoff Brown, Elliot Call, Jan Balaguer, Jackie Xiang, Pidong Wang, Zoe Ashwood, Anton Briukhov, Albert Webson, Sanjay Ganapathy, Smit Sanghavi, Ajay Kannan, Ming-Wei Chang, Axel Stjerngren, Josip Djolonga, Yuting Sun, Ankur Bapna, Matthew Aitchison, Pedram Pejman, Henryk Michalewski, Tianhe Yu, Cindy Wang, Juliette Love, Junwhan Alm, Dawn Bloxwich, Kehang Han, Peter Humphreys, Thibault Sellam, James Bradbury, Varun Godbole, Sina Samangooei, Bogdan Damoc, Alex Kaskasoli, Sébastien M. R. Arnold, Vijay Vasudevan, Shubham Agrawal, Jason Riesa, Dmitry Lepikhin, Richard Tanburn, Srivatsan Srinivasan, Hyeontack Lim, Sarah Hodkinson, Pranav Shyam, Johan Ferret, Steven Hand, Ankush Garg, Tom Le Paine, Jian Li, Yujia Li, Minh Giang, Alexander Neitz, Zaheer Abbas, Sarah York, Machel Reid, Elizabeth Cole, Aakanksha Chowdhery, Dipanjan Das, Dominika Rogoziíska, Vitaliy Nikolaev, Pablo Sprechmann, Zachary Nado, Lukas Zilka, Flavien Prost, Luheng He, Marianne Monteiro, Gaurav Mishra, Chris Welty, Josh Newlan, Dawei Jia, Miltiadis Allamanis, Clara Huiyi Hu, Raoul de Liedekerke, Justin Gilmer, Carl Saroufim, Shruti Rijhwani, Shaobo Hou, Disha Shrivastava, Anirudh Baddepudi, Alex Goldin, Adnan Ozturel, Al- bin Cassirer, Yunhan Xu, Daniel Sohn, Devendra Sachan, Reinald Kim Amplayo, Craig Swanson, Dessie Petrova, Shashi Narayan, Arthur Guez, Siddhartha Brahma, Jessica Landon, Miteyan Patel, Ruizhe Zhao, Kevin Villela, Luyu Wang, Wenhao Jia, Matthew Rahtz, Mai Giméncz, Legg Yeung, James Keeling, Petko Georgiev, Diana Mincu, Boxi Wu, Salem Haykal, Rachel Saputro, Kiran Vodrahalli, James Qin, Zeynep Cankara, Abhanshu Sharma, Nick Fernando, Will Hawkins, Behnam Neyshabur, Solomon Kim, Adrian Hutter, Priyanka Agrawal, Alex Castro-Ros, George van den Driessche, Tao Wang, Fan Yang, Shuo yiin Chang, Paul Komarek, Ross Mcllroy, Mario Lučié, Guodong Zhang, Wael Farhan, Michael Sharman, Paul Natsev, Paul Michel, Yamini Bansal, Siyuan Qiao, Kris Cao, Siamak Shakeri, Christina
16

---

Published in Transactions on Machine Learning Research (08/2025)
Butterfield, Justin Chung, Paul Kishan Rubenstein, Shivani Agrawal, Arthur Mensch, Kedar Soparkar, Karel Tene, Timothy Chung, Aedan Pope, Loren Maggiore, Jackie Kay, Priya Jhakra, Shibo Wang, Joshua Maynez, Mary Phuong, Taylor Tobin, Andrea Tacchetti, Maja Trebacz, Kevin Robinson, Yash Katariya, Scbastian Riedel, Paige Bailey, Kefan Xiao, Nimesh Ghelani, Lora Aroyo, Ambrose Slone, Neil Houlsby, Xuehan Xiong, Zhen Yang, Elena Gribovskaya, Jonas Adler, Mateo Wirth, Lisa Lee, Music Li, Thais Kagohara, Jay Pavagadhi, Sophie Bridgers, Anna Bortsova, Sanjay Ghemawat, Zafarali Ahmed, Tianqi Liu, Richard Powell, Vijay Bolina, Mariko linuma, Polina Zablotskaia, James Besley, Do-Woon Chung, Timothy Dozat, Ramona Comanescu, Xiance Si, Jeremy Greer, Guolong Su, Martin Polacek, Raphaël Lopez Kaufman, Simon Tokumine, Hexiang Hu, Elena Buchatskaya, Yingjie Miao, Mohamed Elhawaty, Aditya Siddhant, Nenad Tomasev, Jinwei Xing, Christina Greer, Helen Miller, Shereen Ashraf, Aurko Roy, Zizhao Zhang, Ada Ma, Angelos Filos, Milos Besta, Rory Blevins, Ted Klimenko, Chih- Kuan Yeh, Soravit Changpinyo, Jiaqi Mu, Oscar Chang, Mantas Pajarskas, Carrie Muir, Vered Cohen, Charline Le Lan, Krishna Haridasan, Amit Marathe, Steven Hansen, Sholto Douglas, Rajkumar Samuel, Minggiu Wang, Sophia. Austin, Chang Lan, Jiepu Jiang, Justin Chiu, Jaime Alonso Lorenzo, Lars Lowe Sjösund, Sébastien Cevey, Zach Gleicher, Thi Avrahami, Anudhyan Boral, Hansa Srinivasan, Vittorio Selo, Rhys May, Konstantinos Aisopos, Léonard Hussenot, Livio Baldini Soares, Kate Baumli, Michael B. Chang, Adrià Recasens, Ben Caine, Alexander Pritzel, Filip Pavetic, Fabio Pardo, Anita Gergely, Justin Frye, Vinay Ramasesh, Dan Horgan, Kartikeya Badola, Nora Kassner, Subhrajit Roy, Ethan Dyer, Víc- tor Campos Campos, Alex Tomala, Yunhao Tang, Dalia El Badawy, Elspeth White, Basil Mustafa, Oran Lang, Abhishek Jindal, Sharad Vikram, Zhitao Gong, Sergi Caelles Ross Hemsley, Gregory Thornton, Fangxiaoyu Feng, Wojeiech Stokowiec, Ce Zheng, Phoebe Thacker, Çağlar Ünlü, Zhishuai Zhang, Mo- hammad Salch, James Svensson, Max Bileschi, Piyush Patil, Ankesh Anand, Roman Ring, Katerina Tsihlas, Arpi Vezer, Marco Selvi, Toby Shevlane, Mikel Rodriguez, Tom Kwiatkowski, Samira Daruki, Keran Rong, Allan Dafoe, Nicholas FitzGerald, Keren Gu-Lemberg, Mina Khan, Lisa Anne Hendricks, Marie Pellat, Vladimir Feinberg, James Cobon-Kerr, Tara Sainath, Maribeth Rauh, Sayed Hadi Hashemi, Richard Ives, Yana Hasson, Eric Noland, Yuan Cao, Nathan Byrd, Le Hou, Qingze Wang, Thibault Sottiaux, Michela Paganini, Jean-Baptiste Lespiau, Alexandre Moufarek, Samer Hassan, Kaushik Shiv- akumar, Joost van Amersfoort, Amol Mandhane, Pratik Joshi, Anirudh Goyal, Matthew Tung, Andrew Brock, Hannah Sheahan, Vedant Misra, Cheng Li, Nemanja Rakićević, Mostafa Dehghani, Fangyu Liu, Sid Mittal, Junhyuk Oh, Seb Noury, Eren Sezener, Fantine Huot, Matthew Lamm, Nicola De Cao, Charlie Chen, Sidharth Mudgal, Romina Stella, Kevin Brooks, Gautam Vasudevan, Chenxi Liu, Mainak Chain, Nivedita Melinkeri, Aaron Cohen, Venus Wang, Kristie Seymore, Sergey Zubkov, Rahul Goel, Summer Yue, Sai Krishnakumaran, Brian Albert, Nate Hurlcy, Motoki Sano, Anhad Mohanancy, Jonah Joughin, Egor Filonov, Tomasz Kepa, Yomna Eldawy, Jiawern Lim, Rahul Rishi, Shirin Badiezadegan, Taylor Bos, Jerry Chang, Sanil Jain, Sri Gayatri Sundara Padmanabhan, Subha Puttagunta, Kalpesh Krishna, Leslic Baker, Norbert Kalb, Vamsi Bedapudi, Adam Kurzrok, Shuntong Lei, Anthony Yu, Oren Litwin, Xiang Zhou, Zhichun Wu, Sam Sobell, Andrea Siciliano, Alan Papir, Robby Neale, Jonas Bragagnolo, Tej Toor, Tina Chen, Valentin Anklin, Feiran Wang, Richie Feng, Milad Gholami, Kevin Ling, Lijuan Liu, Jules Walter, Hamid Moghaddam, Arun Kishore, Jakub Adamck, Tyler Mercado, Jonathan Mallinson, Sid- dhinita Wandekar, Stephen Cagle, Eran Ofek, Guillermo Garrido, Clemens Lombriser, Maksim Mukha, Botu Sun, Hafeezul Rahman Mohammad, Josip Matak, Yadi Qian, Vikas Peswani, Pawel Janus, Quan Yuan, Leif Schelin, Oana David, Ankur Garg, Yifan Hc, Olcksii Duzhyi, Anton Älgmyr, Timothée Lot- taz, Qi Li, Vikas Yadav, Luyao Xu, Alex Chinien, Rakesh Shivanna, Aleksandr Chuklin, Josie Li, Carrie Spadine, Travis Wolfe, Kareem Mohamed, Subhabrata Das, Zihang Dai, Kyle He, Daniel von Dincklage, Shyam Upadhyay, Akanksha Maurya, Luyan Chi, Sebastian Krause, Khalid Salama, Pam G Rabinovitch, Pavan Kumar Reddy M, Aarush Selvan, Mikhail Dektiarev, Golnaz Ghiasi, Erdem Guven, Himanshu Gupta, Boyi Liu, Deepak Sharma, Idan Heimlich Shtacher, Shachi Paul, Oscar Akerlund, François- Xavier Aubet, Terry Huang, Chen Zhu, Eric Zhu, Elico Teixeira, Matthew Fritze, Francesco Bertolini, Liana- Eleonora Marinescu, Martin Bolle, Dominik Paulus, Khyatti Gupta, Tejasi Latkar, Max Chang, Jason Sanders, Roopa Wilson, Xuewei Wu, Yi-Xuan Tan, Lam Nguyen Thiet, Tulsee Doshi, Sid Lall, Swaroop Mishra, Wanming Chen, Thang Luong, Seth Benjamin, Jasmine Lee, Fwa Andrejczuk, Dominik Ra- biej, Vipul Ranjan, Krzysztof Styre, Pengcheng Yin, Jon Simon, Malcolm Rose Harriott, Mudit Bansal, Alexei Robsky, Geoff Bacon, David Greene, Daniil Mirylenka, Chen Zhou, Obaid Sarvana, Abhimanyu Goyal, Samuel Andermatt, Patrick Siegler, Ben Horn, Assaf Israel, Francesco Pongetti, Chih- Wei "Louis"
17

---

Published in Transactions on Machine Learning Research (08/2025)
Chen, Marco Selvatici, Pedro Silva, Kathie Wang, Jackson Tolins, Kelvin Guu, Roey Yogev, Xiaochen Cai, Alessandro Agostini, Maulik Shah, Hung Nguyen, Noah Ô Donnaile, Sébastien Pereira, Linda Friso, Adam Stambler, Adam Kurzrok, Chenkai Kuang, Yan Romanikhin, Mark Geller, ZJ Yan, Kane Jang, Cheng-Chun Lec, Wojeicch Fica, Eric Malmi, Qijun Tan, Dan Banica, Daniel Balle, Ryan Pham, Yan- ping Huang, Diana Avram, Hongzhi Shi, Jasjot Singh, Chris Hidey, Niharika Abuja, Pranab Saxena, Dan Dooley, Srividya Pranavi Potharaju, Eileen O'Neill, Anand Gokulchandran, Ryan Foley, Kai Zhao, Mike Dusenberry, Yuan Liu, Pulkit Mehta, Ragha Kotikalapudi, Chalence Safranek-Shrader, Andrew Goodman, Joshua Kessinger, Eran Globen, Prateek Kolhar, Chris Gorgolewski, Ali Ibrahim, Yang Song, Ali Eichen- baum, Thomas Brovelli, Sahitya Potluri, Preethi Lahoti, Cip Bactu, Ali Ghorbani, Charles Chen, Andy Crawford, Shalini Pal, Mukund Sridhar, Petru Gurita, Asier Mujika, Igor Petrovski, Pierre-Louis Cedoz, Chenmei Li, Shiyuan Chen, Niccolò Dal Santo, Siddharth Goyal, Jitesh Punjabi, Karthik Kappaganthu, Chester Kwak, Pallavi LV, Sarmishta Velury, Himadri Choudhury, Jamie Hall, Premal Shah, Ricardo Figucira, Matt Thomas, Minjic Lu, Ting Zhou, Chintu Kumar, Thomas Jurdi, Sharat Chikkerur, Yenai Ma, Adams Yu, Soo Kwak, Victor Ähdel, Sujeevan Rajayogam, Travis Choma, Fei Liu, Aditya Barua, Colin Ji, Ji Ho Park, Vincent Hellendoorn, Alex Bailey, Taylan Bilal, Huanjie Zhou, Mehrdad Khatir, Charles Sutton, Wojeiech Rzadkowski, Fiona Macintosh, Roopali Vij, Konstantin Shagin, Paul Medina, Chen Liang, Jinjing Zhou, Pararth Shah, Yingying Bi, Attila Dankovics, Shipra Banga, Sabine Lehmann, Marissa Bredesen, Zifan Lin, John Fric Hoffmann, Jonathan Lai, Raynald Chung, Kai Yang, Nihal Balani, Arthur Braäinskas, Andrei Sozanschi, Matthew Hayes, Héctor Fernández Alcalde, Peter Makarov, Will Chen, Antonio Stella, Liselotte Suijders, Michael Mandl, Ante Kärrman, Pawel Nowak, Xinyi Wu, Alex Dyck, Krishnan Vaidyanathan, Raghavender R. Jessica Mallet, Mitch Rudominer, Eric Johnston, Sushil Mittal, Akhil Udathu, Janara Christensen, Vishal Verma, Zach Irving, Andreas Santucci, Gamaleldin Elsayed, Elnaz Davoodi, Marin Georgiev, lan Temney Nan Hua, Geoffrey Cideron, Edouard Leurent, Mahmoud Alnahlawi, Ionut Georgescu, Nan Wei, Ivy Zheng, Dylan Scandinaro, Heinrich Jiang, Jasper Snoek, Mukund Sundararajan, Xuezhi Wang, Zack Ontiveros, Itay Karo, Jeremy Cole, Vinu Rajashekbar, Lara Tumch, Eyal BeviD,vid, Rishub Jain, Jonathan Uesato, Romina Datta, Oskar Bunyan, Shimu Wu, John Zhang, Piotr Stanczyk, Ye Zhang, David Steiner, Subhajit Naskar, Michael Azzam, Matthew John- son, Adam Paszke, Chung-Cheee -heng Chiu, Jaume Sanchez Elias, Afroz Mohiuddin, Faizan Muhammad, Jin Miao, Andrew Lee, Nino Vieillard, Jane Park, Jingeng Zhang, Jeff Stanway, Drew Garmon, Abhijit Kar- markar, Zhe Dong, Jong Lee, Aviral Kumar, Luowei Zhou, Jonathan Evens, William Isaac, Geoffrey Irving, Edward Loper, Michael Fink, Isha Arkatkar, Nanxin Chen, Tzhak Shafran, Ivan Petrychenko, Zhe Chen, Johnson Jia, Anselm Levskaya, Zhenkai Zhu, Peter Grabowski, Yu Mao, Alberto Magni, Kaisheng Yao, Javier Snaider, Norman Casagrande, Evan Palmer, Paul Suganthan, Alfonso Castaño, Irene Giannoumis, Wooyeol Kim, Mikolaj Rybiński, Ashwin Sreevatsa, Jennifer Prendki, David Soergel, Adrian Goedeck- emeyer, Willi Gierke, Mohsen Jafari, Meenu Gaba, Jeremy Wiesner, Diana Gage Wright, Yawen Wei, Harsha Vashisht, Yana Kulizhskaya, Jay Hoover, Maigo Le, Lu Li, Chimezie Iwuanyanwu, Lu Liu, Kevin Ramirez, Andrey Khorlin, Albert Cui, Tian LIN, Marcus Wu, Ricardo Aguilar, Keith Pallo, Abhishek Chakladar, Ginger Perng, Flena Allica Abellan, Mingyang Zhang, Ishita Dasgupta, Nate Kushman, Ivo Penchev, Alena Repina, Xihui Wu, Tom van der Weide, Priya Ponnapalli, Caroline Kaplan, Jiri Simsa, Shuangleng Li, Olivier Dousse, Fan Yang, Jeff Piper, Nathan Ie, Rama Pasumarthi, Nathan Lintz, Anitha Vijayakumar, Daniel Andor, Pedro Valenzuela, Minnie Lui, Cosmin Paduraru, Daiyi Peng, Katherine Lee, Shuyuan Zhang, Somer Greene, Due Dung Nguyen, Paula Kurylowicz, Cassidy Hardin, Lucas Dixon, Lili Janzer, Kiam Choo, Ziqiang Feng, Biao Zhang, Achintya Singhal, Dayou Du, Dan McKinnon, Natasha Antropova, Tolga Bolukbasi, Orgad Keller, David Reid, Daniel Finchelstein, Maria Abi Raad, Remi Crocker, Peter Hawkins, Robert Dadashi, Colin Gaffney, Ken Franko, Anna Bulanova, Rémi Leblond, Shirley Chung, Harry Askham, Luis C. Cobo, Kelvin Xu, Felix Fischer, Jun Xu, Christina Sorokin, Chris Alberti, Chu-Cheng Lin, Colin Evans, Alek Dimitriev, Hannah Forbes, Dylan Banarse, Zora Tung, Mark Omernick, Colton Bishop, Rachel Sterneck, Rohan Jain, Jiawei Xia, Ehsan Amid, Francesco Piccinno, Xingyu Wang, Praseem Banzal, Daniel J. Mankowitz, Alex Polozov, Victoria Krakovna, Sasha Brown, MohammadHosseen Bateni, Dennis Duan, Vlad Firoiu, Meghana Thotakuri, Tom Natan, Matthieu Geist, Ser tan Girgin, Hui Li, Jiayu Ye, Ofir Roval, Reiko Tojo, Michael Kwong, James Lee-Thorp, Christopher Yew, Danila Sinopalnikov, Sabela Ramos, John Mellor, Abhishek Sharma, Kathy Wu, David Miller, Nico- las Sonnerat, Denis Vnukov, Rory Greig, Jennifer Beattie, Emily Caveness, Libin Bai, Julian Eisenschlos, Alex Korchemniy, Tomy Tsai, Mimi Jasarevic, Weize Kong, Phuong Dao, Zeyu Zheng, Frederick Liu,
18

---

Published in Transactions on Machine Learning Research (08/2025)
Fan Yang, Rui Zhu, Tian Huey Teh, Jason Sanmiya, Evgeny Gladchenko, Neje Trdin, Daniel Toyama, Evan Rosen, Sasan Tavakkol, Linting Xue, Chen Elkind, Oliver Woodman, John Carpenter, George Papa- makarios, Rupert Kemp, Sushant Kafle, Tanya Grumina, Rishika Sinha, Alice Talbert, Diane Wu, Denese Owusu-Afriyic, Cosmo Du, Chloc Thornoon, Jordi Pont- Tost,, Pradyumna Narayana, Jing Li, Saaber Fatehi, John Wieting, Omar Ajmeri, Benigno Uria, Yeongil Ko, Laura Knight, Amélie Héliou, Ning Niu, Shane Gu, Chenxi Pang, Yeqing Li, Nir Levine, Ariel Stolovich, Rebeca Cantamaria-Ferannte Goenka, Wenny Yustalim, Robin Strudel, Ali Elqursh, Charlie Deck, Hyo Lee, Zonglin Li, Kyle Levin, Raphael Hoffmann,, Dan Holtmann- Rice,, Olivier Bachem, Sho Arora, Christy Koh, Soheil Hassas Yeganeh, Siim Pöder, Mukarram Tariq, Yanhua Sun, Lucian Tonita, Mojtaba Seyedhosseini, Pouya Tafti, Zhiyu Liu, Anmol Gulati, Jasmine Liu, Xinyu Ye, Bart Chrzaszez, Lily Wang, Nikhil Sethi, Tianrun Li, Ben Brown, Shreya Singh, Wei Fan, Aaron Parisi, Joe Stanton, Vinod Koverkathu, Christopher A. Choquette-Choo Yunjie Li, TJ Lu, Abe Ittycheriah, Prakash Shroff, Mani Varadarajan, Sanaz Bahargam, Rob Willoughby, David Gaddy, Guillaume Desjardins, Marco Cornero, Brona Robenck, Bhavishya Mittal, Ben Albrecht, Ashish Shenoy, Fedor Moiseev, Henrik Jacobsson, Alireza Ghaffarkhah, Morgane Rívière, Alanna Walton, Clément Crepy, Alicia Parrish, Zongwei Zhou, Clement Farabet, Carey Radebaugh, Praveen Srinivasan, Claudia van der Salm, Andreas Fidjeland, Salvatore Scellato, Eri Latorre-Chimoke Hanna Klimezak- Plucííska, David Bridson, Dario de Cesare, Tom Hudson, Piermaria Mendolicchio, Lexi Walker, Alex Morris, Matthew Mauger, Alexey Guseynov, Alison Reid, Seth Odoom, Lucia Loher, Victor Cotruta, Madhavi Yenugula, Dominik Grewe, Anastasia Petrushkina, Tom Duerig, Antonio Sanchez, Steve Yad- lowsky, Amy Shen, Amir Globerson, Lynette Webb, Sahil Dua, Dong Li, Surya Bhupatiraju, Dan Hurt, Haroon Qureshi, Ananth Agarwal, Tomer Shani, Matan Eyal, Anuj Khare, Shreyas Rammohan Belle, Lei Wang, Chetan Tekur, Mihir Sanjay Kale, Jinliang Wei, Ruoxin Sang, Brennan Saeta, Tyler Liechty, Yi Sun, Yao Zhao, Stephan Lee, Pandu Nayak, Doug Fritz, Manish Reddy Vuyyuru, John Aslanides, Nidhi Vyas, Martin Wicke, Xiao Ma, Evgenii Eltyshev, Nina Martin, Hardie Cate, James Manyika, Key- van Amiri, Yelin Kim, Xi Xiong, Kai Kang, Florian Luisier, Nilesh Tripuraneni, David Madras, Mandy Guo, Austin Waters, Oliver Wang, Joshua Ainslic, Jason Baldridge, Han Zhang, Garima Pruthi, Jakob Bauer, Feng Yang, Riham Mansour, Jason Gelman, Yang Xu, George Polovets, Ji Liu, Honglong Cai, War- ren Chen, XiangHai Sheng, Emily Xue, Sherjil Ozair, Christof Angermueller, Xiaowei Li, Anoop Sinha, Weiren Wang, Julia Wiesinger, Emmanouil Koukoumidis, Yuan Tian, Anand Tyer, Madhu Gurumurthy, Mark Goldenson, Parashar Shah, MK Blake, Hongkun Yu, Anthony Urbanowicz, Jennimaria Palomaki, Chrisantha Fernando, Ken Durden, Harsh Mehta, Nikola Momchev, Flahe Rahimtoroghi, Maria Georgaki, Amit Raul, Sebastian Ruder, Morgan Redshaw, Jinhyuk Lee, Denny Zhou, Komal Jalan, Dinghua Li, Blake Hechtman, Parker Schuh, Milad Nasr, Kieran Milan, Vladimir Mikulik, Juliana Franco, Tim Green, Nam Nguyen, Joe Kelley, Aroma Mahendru, Andrea Hu, Joshua Howland, Ben Vargas, Jeffrey Hui, Kshitij Bansal, Vikram Rao, Rakesh Ghiya, Emma Wang, Ke Ye, Jean Michel Sarr, Melanie Moranski Preston, Madeleine Elish, Steve Li, Aakash Kaku, Jigar Gupta, Ice Pasupat, Da-Cheng Juan, Milan Someswar, Tejvi M., Xinyum Chen, Aida Amini, Alex Fabrikant, Eric Chu, Xuanyi Dong, Amruta Muthal, Senaka Buthpitiya, Sarthak Jauhari, Nan Hua, Urvashi Khandelwal, Ayal Hitron, Jie Ren, Larissa Rinaldi, Shahar Drath, Avigail Dabush, Nan-Jiang Jiang, Harshal Godhia, Uli Sachs, Anthony Chen, Yicheng Fan, Hagai Taitelbaum, Hila Noga, Zhuyun Dai, James Wang, Chen Liang, Jemny Hamer, Chun-Sung Ferng, Chenel Elkind, Aviel Atias, Paulina Lee, Vit Listík, Mathias Carlen, Jan van de Kerkhof, Marcin Pikus, Krunoslav Zaher, Paul Müller, Sasha Zykova, Richard Stefance, Vitaly Gatsko, Christoph Hirnschall, Ashwin Scthi, Xingyu Federico Xu, Chetan Ahuja, Beth Tsai, Anca Stefanoiu, Bo Feng, Keshav Dhandhania, Manish Katyal, Akshay Gupta, Atharva Parulekar, Divya Pitta, Jing Zhao, Vivaan Bhatía, Yashodha Bhavnani, Omar Alhadlaq, Xiaolin Li, Peter Danenberg, Dennis Tu, Alex Pine, Vera Filippova, Abhipso Ghosh, Ben Limonchik, Bhargava Urala, Chaitanya Krishna Lanka, Derik Clive, Yi Sun, Edward Li, Hao Wu, Kevin Hongtongsak, Tanna Li, Kalind Thakkar, Kuanysh Omarov, Kushal Majmundar, Michael Alverson, Michael Kucharski, Mohak Patel, Mudit Jain, Maksim Zabelin, Paolo Pelagatti, Rohan Kohli, Saurabh Kumar, Joseph Kim, Swetha Sankar, Vineet Shah, Lakshmi Ramachandruni, Xiangkai Zeng, Ben Bariach, Laura Weidinger, Tu Vu, Alek Andreev, Antoine He, Kevin Hui, Sheleem Kashem, Amar Subramanya, Sissie Hsiao, Demis Hassabis, Koray Kavukcuoglu, Adam Sadovsky, Quoc Le, Trevor Strohman, Yonghui WU, Slav Petrov, Jeffrey Dean, and Oriol Vinyals. Gemini: A family of highly capable multimodal models, 2025. URL https: // //axxivvvaa www.:org//ao. htgg/abs/23115 2312. 11805
Sonam
19

---

Published in Transactions on Machine Learning Research (08/2025)
Feng Wang and Huaping Liu. Understanding the behaviour of contrastive loss, 2021. URL https://arxxy org/ 0aabs/2012.0999
Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. Improving text embeddings with large language models, 2024a. URL https://arxii: org//og////.. http/abb/2400...00 00368
Peng Wang, Shuai Bai, Sinan Tan, Shijic Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing languae model's perception of the world at any resolution, 2024b. URL https:
org/ 12191
Cong Wei, Yang Chen, Haonan Chen, Hexiang Hu, Ge Zhang, Jie Fu, Alan Ritter, and Wenhu Chen. Uniir: Training and benchmarking universal multimodal information retrievers, 2023. URL htt./httss:/// httpps://arxiv.org has//231..1113
Erik Wijmans, Brody Huval, Alexander Hertzberg, Vladlen Koltun, and Philipp Krähenbiihl. Cut your losses in largo-vocabulary language models, 2024. htRps::tt:://arrt wwwpe/iii.rrr./oooooeoo
Jianxiong Xiao, James Hays, Krista A. Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large- scale scene recognition from abbey to zoo. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recogrition, pp. 3485 3492, 2010. doi:
Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. Filip: Fine-grained interactive language-inage pre-training, 2021. URL https://arxia: http:///xxxvvv.or//aaa/ aba/
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models, 2022. URL https://arxil: org/ 2205.
01917
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzlu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mmmu: massive multi-discipline multimodal understanding and reasoning benchmark for expert agi, 2024. URL https:
//arxiv. wrw. httggabb//311.... 16502
Kai Zhang, Yi Luan, Hexiang Hu, Kenton Lee, Siyuan Qiao, Wenhu Chen, Yu Su, and Ming Wei Chang, Magiclens: Self-supervieed image retricval with open-ended instructions. In The Forty-first International Conference on Machine Learning (ICML), pp. to appear, 2024.
Xin Zhang, Yanzhao Zhang, Wen Xie, Mingxin Li, Ziqi Dai, Dingkun Tong, Pengjun Xie, Meishan Zhang, Wenjie Li, and Min Zhang. Gme: Improving universal multimodal retrieval by multimodal Ilms, 2025.
URL https:://aaa org/ wrg.absg///eeeoeeeeooooeeeeeee/e httg//bb/2411...5. 2t../2222.........
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela. Barriuso, and Antonio Torralba. Scene pars- ing through ade20k dataset. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
Junjie Zhou, Zheng Liu, Ze Liu, Shitao Xiao, Yueze Wang, Bo Zhao, Chen Jason Zhang, Defu Lian, and Yongping Xiong. Megapairs: Massive data synthesis for universal multimodal retrieval, 2024. URL https://arxiv.org/ hwtp://////abb/aaaaaaeaaeae 2412.14475
Yuke Zhu, Oliver Groth, Michael Bernstein, and Li Fei-Fei. Visual7w: Grounded question answering in images, 2016. URL https::/p///e ://rxxiv.////i////// wrg/abs///ee 03416
20

---

Published in Transactions on Machine Learning Research (08/2025)
A Instruction Fine-tuning Prompt Design
We use the following prompt template when using GPT-4o to generate instructions:
"<Image> Given this image, provide a user prompt where the following caption would be a reasonable answer: [Caption). Only return the prompt."
Where [Caption} is the caption for the bounding box from Visual Genome. We use temperature of 0 (deterministic) when generating instructions. We choose this template to collect diverse set of instructions that user would plausibly ask.
B CtrlBench Prompt Design
We use the following template in Gemini 2.5 Flash for each of the 1000 images:
"<Image> Your task is to identify and describe 5 distinct objects or elements in this image. For each object provide short sentence description of Please specific in your description. Then, for each description, provide uaer prompt where that description caption would be reasonable answer. Output your answer as <DESCRIPTION 1>, "prompt_1"; <PROMPT 1>, "desc_2": <DESCRIPTION 2>, "prompt_ __" <PROMPT 2>, idesc_3", <DESCRIPTIOn 3>, "prompt_ 3 <PROMPT 3?, "desc_4" =": <DESCRIPTION 4>, "prompt_4" <PROMPT 4>, "desc_5": <DESCRIPTION 5>, "prompt_5": <PROMPT 5>}
the following template: ("desc_1";
Save on API costs, we generate the candidates and queries for each a single prompt.
C Architecture Ablation
LLM2Vec [BehnamGhader et al. 2024) introduced several architecture modifications for adapting LLMs into text embedding models. In the multimodal setting, we find that using casual vs. bidirectional attention is marginal for improving model accuracy (Figure
1.0
1.0
Casual
Bidirectional
0.8
Validation Accuracy
0.6
Validation
0.4
LoRA
0.2
LoRA (r=32)
LoRA (r=64)
0.0
0.0
600
800
1000
200
Step
Step
Figure 7: (Left.) Causal Bidirectional attention. (Right) LoRA adapter rank ablation.
On average, bidirectional attention slightly outperforms casual attention. Ablating over adapter rank, we find that different rank adapters tend to converge to the same accuracy, with higher rank adapters performing
a fraction of a percentage better. Overall, better data and scaling (Appendix D present a much more promising direction for improving embedding quality.
Scaling Training
We find that batch size and number of samples seen during training are both important factors. By increasing our batch size by 4x, from 128 queries and 1024 candidates our full pretraining run size (512 queries and 4096 candidates), validation accuracy 800 sample validation batch using in-batch negatives
21

---

Published in Transactions on Machine Learning Research (08/2025)
21.0
Conrastive LOSS
20.5
20.0
19.5
1000
2000
4000
Step
Figure 8: More samples seen vs. loss.
increases from 92.5% to 95.0%. We control for total samples seen by training the smaller batch size model for 4x more steps. Interestingly, this effect is even more pronounced on OOD validation data. For example, on a similar batch of MSCOCO data, accuracy increases from 73.8% to 84.0%. Therefore, techniques for scaling batch size under limited VRAM, such as GradCache Gao et al. (021)), are a promising direction to improve our pretrained model. Furthermore, we find that more steps (more samples seen) is also straightforward way to increase the pretrained model's performance. As shown by Figure g doubling step count steadily decreases loss during our pretraining run.
E Comparison to VLM2Vec
With the MMEB benchmark, Jiang et al. (024)) also include mutltimodal embedding model: VeM2Vec.
We exclude VLM2Vec from Tables 2 and 5 as it has been trained on MSCOCO and several tasks from
the MMEB benchmark. We provide the comparison of the two models here while specifying which tasks are out of distribution (OOD) for VLM2Voe. All tasks are OOD for our model.
VLM2Vec
ABC (ours)
ImageNet- 1K
Russakovsky et al.
2015)
65.6
71.2
HatefulMemeet Kicla al. (2021)
67.1
52.1
VOC2007
Everingham et al.
88.6
81.4
SUN397 X Xiao et al. (2010)
72.7
71.8
Placc365
Lopez-Cifuena Cifuentes et al. 2020
42.6
40.7
ImageNet-AA Djolonga al. 2020
19.3
49.4
ImageNet-R Hendrycks et al. 2021
70.2
86.8
Object.Net Barbu et al. (2019
29.5
67.7
Country-21 Radford et al.
2021
13.0
18.5
Average
52.1
60.0
Average 00D
34.9
52.6
Table 7: Classification results for VLM2Vec and ABC (ours). Gray background indicates that VLM2Vec has been trained on this task.
Table compares the performance of our model to VLM2Vec Jiang et al. (024).. Gray rows indicate tasks that VLM2Voc is trained on. On average, VLM2Vec performs better on tasks that are its distribution, while our model outperforms on tasks where both models are OOD, Interestingly, our model performs better Teage even though is not trained it.
22

---

Published in Transactions on Machine Learning Research (08/2025)
F Results on VLLMs without Contrastive Training
Can dense embedding be extracted from VLLM's without additional contrastive training? We experiment
with two common methods for extracting dense embeddinggs from last-layer hiddens: (1) Using the last
token in the sequence (Jiang et al. 2024 (2) Averaging over the sequence dimension BehnamGhader
et al. 2024).
Pooling Method
Direction
Top-1
Top-5
Top-10 10
Mean Pooling
12t
6.30%
16.18%
23.70%
t2i
3.41%
9.25%
13.39%
Last Token
0.06%
0.26%
0.36%
t2i
0.08%
0.30%
0.73%
Table 8: Comparison of pooling methods across retrieval directions and on MSCOCO.
On MSCOCO retrieval find that mean pooling significantly oulperforms using the last. token hidden (Table 8). However, neither method performs very strong. Both methods are significantly worse than zero-shot CLIP model on MSCOCO (Table 1
G Inference Settings for CLIP
Resolution
Image
Text
Text Image
224x224
64.94
48.02
336x336
47.42
448x448
61.98
45.97
Table 9: ChenClip-ViTT-'ee accuracy (top-1) on MSCOCO at different image resolutions. Unlike ABC, accuracy degrades when scaling token budgets beyond SeinnCli--ViTg----e native 224 x 224 resolution.
We find that CLIP models perform best at their pretraining resolution. As shown in Table accuracy consistently degrades when scaling the token counts of CLIP models beyond their native resolution. It would be unfair to evaluate models at a setting that is both more compute intensive and less performant. Therefore, we evaluate CLIP models at their native resolution and loken count across all experiments.
23

---

Published in Transactions on Machine Learning Research (08/2025)
H
CtrlBench Examples
R
TENERA
Orgivaa de Nento Care"r
Q: What is happening in the top left corner of the image?
Q: What part of the animal is prominently dis- played in the image? A: the neck of polar bear
A: Tennis ball in the air
Q: What player about to hit with racket?
Q: What colors can you see on the bear the im- age?
A: round tennis ball
black, brown and white bear neck
Q: What color are the shorts the tennis player is wearing?
Q: What stands out to you about the bear's ap- pearance in this image?
A: black and yellow shorts
A: the bear has such tiny ears for such huge ani- mal
Q: What action taking place on the tennis court image?
A: mau serving tennis ball.
Q: What the doing with its eyes pic-
ture?
Q: What's prominently dividing the ground area in
the bear has eyes closed
the image?
A: White line on court
Q: What's the polar bear swimming in?
A: body of water
24


Reviews:
[{"review_id": "xNwFWdgZeY", "reviewer": "Reviewer_sjVm", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- Clever engineering in the method\n- Impressive improvements over baselines \n\nWeaknesses:\n- My biggest problem is why do we need specialized multimodal embedding models, if a large autoregressive VLM (trained to generate text, which the authors use as a backbone) can achieve this purpose? Why not use the embeddings from this model itself, without any further training/tuning? What is wrong with those standalone embeddings? These models already already change and control the image embeddings according to the instruction (due to prefix-LM where the image tokens and instructions tokens attend to each other). So natural language guided visual embeddings are already achieved by VLMs. This is the main question which is not addressed. There is also no comparison in the tables, with using those standalone embeddings from those VLMs directly. Why do we even need a multimodal embedding model to solve tasks such as VQA for example, if a text generative VLM can do this? \n- The improvements are not surprising, because of the use of a powerful new 7B VLM as an additional embedding model. This increases the memory requirements largely. Additionally, there is a huge amount of computation involved for this improvement in performance. I do not find that as a clever way of achieving a goal, when a huge new model is plugged in for that purpose. Moreover, unlike CLIP, there should be two runs on the same model for every sample. One with the pretreated model from stage 1 to get the embeddings for the answer candidates, and the other with the fine-tuned model to get the image+instruction embedding. \n- The approach in spirit is similar to VLM2Vec. Therefore, the novelty is limited. No comparisons with VLM2Vec in Tables 1,2,4. \n- No ablation studies on the hard negative mining in the pretreating stage. The improvements could be largely coming from the strong hard negative mining. \n- Are the results in the Tables using an image size of 224x224? Otherwise, they are not a fair comparison since the authors use Qwen-VL which can accept different image size. Most baselines use an image size of 224x224, while the authors use a larger image size?\n- The related work section does not mention how the approach is advantageous over current methods, which is the most important part of the related work. It simply states other works without describing how the proposed approach differs. \n\nMinor\n- In section 3, “uses self-supervised training on image-caption pairs” - image captioning is not a supervised task", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a method to achieve multimodal embeddings (visual embeddings made of image + text instructions, where visual embeddings can be guided and controlled by natural language instructions). Unlike CLIP which cannot handle tasks that require user instruction or disambiguation (e.g. distinguishing between two plausible image captions), this paper proposes to solve this issue by plugging in a Large VLM (Qwen2-VL-7B) into the embedding process. Unlike CLIP, VLMs are unified image-text model with early fusion that integrate image and text instructions to build multimodal embeddings. The authors modify the attention to bi-directional attention and remove the generative part of the VLM and finetune it. Specifically, there are two stages: 1) pretraining with constative learning on image-text pairs with hard negative mining of the text pairs (captions that achieve score just lower than the threshold in a preminlaty stage of training), 2) fine-tuning on instruction data gathered from Visual Genome and using GPT-4 to reformulate these to instruction-retrieval format, using also contrastive training with in-batch negatives (negatives of the same image but of different regions, as well as from other samples in the batch). Both pretraining and fine-tuning use separate LoRA adaptors.  Experiments are conducted to demonstrate the method’s raw embedding quality and its ability to handle instruction-guided queries (zero-shot image/text retrieval, zero-shot image classification, visual question answering (VQA), and a new proposed instruction-controlled retrieval benchmark).", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I am worried mostly about the practicability of the method, as 1) i do not see its advantage over autoregressive VLMs (which the authors use as a backbone and finetune) and using its embeddings directly, and 2) its increased computation time and memory. There are other weaknesses as mentioned.", "broader_impact_concerns": "No issues", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "qnudIdw5Aq", "reviewer": "Reviewer_xc7U", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "A few comments below that might require revisions or clarifications:\n\n1.\tThe proposed ABC as shown in Fig 2 involves a LLM in the training, but is it not operating as a generative large multimodal model (LMM) at testing? It is confusing how the model functions at inference time on different downstream tasks (classification, retrieval and VQA).\n2.\tI am not very convinced by the experimental design on the zero-shot retrieval and classification tasks in sec 4.2. The proposed method focuses on the inverleaved multimodal embedding to enable textual controllability on the visual embedding, but the experiments seem not reflecting this property. \n3.\tOn the evaluation of VQA task in Sec 4.6, authors mention about retrieval. However, generative LMMs when evaluating on VQA task often do not involve retrieval but simply respond in an open-ended fashion or as multi-choice QA. Some clarification on the differences should be discussed and comparison against recent generative LMMs should be included, perhaps using tool suites, e.g.  https://github.com/EvolvingLMMs-Lab/lmms-eval", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents ABC model that uses natural language instructions to control visual embeddings by adapting\nVLLMs into dense embedding models in a lightweight fashion with synthetic instructions. They also introduce\nCtrlBench, a novel benchmark for measuring instruction-controlled retrieval on VQA tasks to highlight the interleave modalities for retrieving the correct response.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Clarification of the proposed method ABC against existing generative and non-generative LMMs, and how ABC functions on different downstream tasks in Introduction and Experimental section.\n2. Include or revise the experimental design on the zero-shot retrieval and classification tasks to better reflect the claimed property on inverleaved multimodal embedding. For instance, classification/retrieval conditioned on text?\n3. Comparison with sota generative LMMs on VQA task.", "broader_impact_concerns": "N.A.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "4G6GtjDtlg", "reviewer": "Reviewer_JnhS", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "In the assessment of the manuscript, this reviewer has emphasised soundness, motivation, and interesting contributions that can positively impact the research community on a broader scale. By this, we mean that a method can stand out as promising without showing gains in every conceivable downstream task. Additionally, this reviewer has evaluated the work on the clarity, precision, and reproducibility. Based on these criteria, we highlight strengths and weaknesses in the submission.\n\n### Strengths\n\n1. The authors directly address issues in current benchmarks by releasing `CtrlBench`, which looks to alleviate issues where answers can be constructed without even a cursory glance at the source image. Pointing out these types of irregularities in how models are benchmarked and results are reported is always a worthwhile contribution. \n\n2. The paper has several technical details, providing optimisation and contributions in ML-engineering. The authors should be applauded for their commitment to sharing their insights in VLM modelling; which is unfortunately becoming rarer in the field. \n    - The proposed two-stage recipe via large-scale contrastive pre-training, followed by 100-step LoRA adapter is a handy setup. \n    - The paper is generous in spelling out several optimiser hyper-parameters, temperature schedules and memory management. \n\n3. The idea of conditioning with (as this reviewer understand the approach) inference-time switchable adapters comes across as an interesting approach, short of being entirely novel (related ideas have been explored in other generative modelling approaches). Nevertheless, it is an architectural design component that can be applied more generally as a conditioning step.\n\n\n### Weaknesses\n\n#### Major Concern: Motivating Ambiguity\n\nThe paper comes across as opaque about is purported problem statement. Except for a vague mention of *ambiguity* of image-text-text triplets in the introduction, the paper practically focuses on VQA, zero-shot retrieval and classification in a general setting. This would be fine if the work did not base the motivated approach contingently on what they state as *ambiguity*. Figure 1 is designed to highlight this ambiguity inherent in data, but seemingly fails to demonstrate what type of ambiguity is present in the figure. Why are these ambiguous? The fact that a *scene can provide multiple unrelated factoids* does not necessitate *ambiguity*. Aside from this brief mention, this stated issue on ambiguity is not elaborated on further in the text, and the authors move on to motivate that their method purportedly \"solves\" this ambiguity problem by modifying image embeddings with adapters. The next mention of ambiguity is presented alongside an example (Figure 6) which clearly shows how a question may be answered independently of the image. This shows a case where $p(a \\mid q, i) = p(a \\mid q)$, such that the correct answer can be deduced via the query alone. However, *this is wholly unrelated with embedding images conditioned on the query*, which the authors propose as a method to resolve ambiguity. **In short:** *Without formalising what the authors mean by ambiguity, the reader cannot be expected meaningfully ascertain that the motivation and method has any impact on the problem at hand. Increments in results may easily come from other factors, such as negative mining and multi-stage training pipelines*.\n\n#### Major Concern: Assumptions\n\nA second major concern strikes at the epistemological assumptions in the current work. ABC allows for stronger conditioning on visual prompts to tackle ambiguity (whatever the authors decide this may be). While this reviewer would agree that specification in queries can lead to better responses, this can go both ways. If a query has a well-defined response, adapting the visual representation to modify how the image is interpreted could conversely lead to *negative bias and subjectiveness that may be unwanted*, just as it may help to disambiguate. Taking a more classical view, ambiguity can be better resolved by being able to attach uncertainty to responses, and detect nonsensical or unclear queries in multimodal settings. These skills are woefully underrepresented in language models already, and it is not clear that adding more \"control\" of how the model  interprets an image is actually useful, outside of the field of alignment. **Respectfully**, *this reviewer maintain doubts about the central assumptions motivating the work*.\n\n#### Specific weaknesses\n\n1. The resolution story comes across as muddled. The authors present a gain on ObjectNet when they raise the token budget, then concede that the baseline datasets are down-scaled and “may underestimate the capabilities” of high-res models. If this reviewer is reading this correctly, all ObjectNet numbers are reported with up-scaled inputs, but the baselines stick to 224² crops. This seems like an apples-to-oranges comparison without control for confounding effects. Re-running OpenCLIP-G with similar token budgets would likely close much of the gap.\n\n2. The outlined retrieval in section 4.2 is asymmetric by construction. Image $\\rightarrow$ text negatives are mined, text $\\rightarrow$ image are not, leading to the very gap the authors report, but in the work this is treated as an “expected” outcome rather than a design flaw to fix.\n\n3. While the authors admit Visual Genome is both the fine-tuning and benchmark source, this is handwaved by a claim that Gemini paraphrasing fixes overlap. Paraphrasing captions does not eliminate visual memorisation. This reviewer worries that the presented methodology might lead to data leakage.\n\n4. The paper contains exposition that is non-formal and unclear, making reproducibility difficult;\n    - \"We create synthetic instructions that correspond to different aspects of the same image and use these as negatives.\" It is not clear what is actually done by the authors here. Is this related to Section 3.3 and `CrtlBench`? \n    - Where exactly do the LoRA hooks live? An “instruction adapter” is added after the frozen dual-encoder has finished contrastive pre-training, but the authors never state which blocks receive the rank-4 (or 8? 16?) matrices, whether they act on the vision trunk, the projection head, or the joint fusion module, and whether scaling $\\alpha$ is fixed or learned. \n    - Authors provide an illustration or a prompt string `<Image> Instruction: …` but do not spell out the actual special-token IDs, the segment-type embeddings, or the positional conventions. Does ABC prepend the image CLS before every instruction, or is the vision vector written into a reserved unused token? While these are reporting issues shared with other works in the field (particularly for VLMs), these details matter for the reader.\n    - The contrastive pre-training stage is unclear. What are the augmentations applied to the vision side, and what is the exact objective for mined versus random negatives? \n\n5. Aside from the purported benefits in the approach, there are ethical ramifications in allowing users interact with a model that can control image embedding in relation to the query. These are not addressed at all in the current work.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Achieve Better Control (ABC) is a visual language model (VLM) where the authors propose a method to let users steer image embeddings via contextual language prompts. As opposed to CLIP-style embeddings which measures alignment between language and image in a static context, ABC uses low-rank adapters to control image embeddings based on queries. The high-level goal seems to be better adaptation in VLMs, and in practical terms the authors focus on VQA and retrieval tasks. \n\nThe authors astutely point out issues with existing VQA benchmarks, such as cases where queries can be answered more or less independently from visual data, i.e. $p(a \\mid q, i) = p(a \\mid q)$, where $(q,a)$ is a query-answer pair and $i$ is an image embedding. In response, the authors craft a new benchmark that address these issues directly. The authors motivate conditional \"control\" via ambiguous VQA triplets, that need disambiguation to be meaningfully parsed, but does not explicitly formalise what is meant by ambiguity, nor how this is solved by the proposed method.\n\nThe authors conclude by stating that their approach provides strong zero-shot results in downstream tasks, and emphasise their contributions in evaluating the effect of resolution, temperature scaling, and backbone; all considered important for these tasks. However, some details regarding the effect of resolution is left muddled, as we mention in W2.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "- This reviewer has issues on the effect of controlling image embeddings to improve VLM tasks. Can the authors successfully demonstrate that the results are indeed a factor of the conditioning, and not confounding architectural modifications and training regimes? Demonstrating this is crucial, as the method is motivated by control as a driver, necessitating a controlled experiment.\n- Can the authors elaborate and clearly show what type of ambiguity is resolved by conditioning image embeddings on queries? Specifically, better examples are required to delineate this as a clear problem statement. Figure 6 is clearly motivated but does not address ambiguity; rather it demonstrates examples where there is *less* rather than *more* ambiguity in the VQA triplet. A formalisation of what ambiguity means should be included.\n- Can the authors provide concrete evidence that ABC eliminates confounders, and control for token budgets in an apples-to-apples comparison? As it stands, there are seemingly many uncontrolled factors that prevents the reader from delineating the contributions of the method from others. Simply increasing token budgets for the proposed method without symmetrically controlling for this in the baselines is not very convincing.\n- Similarly, with regard to the the issues mentioned in W2, can the authors address the asymmetry and show that the reported results are not an effect of the experimental design, but rather a property that comes from their proposed method?\n- Can the authors show that visual memorisation has not meaningfully contributed to biasing `CtrlBench`? As it stands, the benchmark could have data leakage, and the authors need to convincingly show that this is not an issue for adoption by the community.\n- The work is missing a broader impact statement on ethical concerns, as mentioned in the dedicated section. This will need to be addressed.\n- As more minor adjustments, details on some aspects on the overall method should be tightened; see W4 for suggestions.", "broader_impact_concerns": "The line of research proposed by the authors can arguably permit more aggressive control of information. Since ABC lets the querier push an instruction through a LoRA that perturbs the image representation before any downstream scoring, whoever controls the prompt also controls what “counts” as similarity. *This clearly warrants broader impact statement on the work.*\n\nThe capability for controlling image embeddings can of course go both ways. Taking a positive example; a radiologist can focus retrieval on *tubular opacities* rather than overall lung texture to more succinctly steer queries, contributing to better healthcare for patients. However, the very same approach can be used to suppress disfavoured political views or human rights violations. Suppose an authoritarian agent attaches the instruction *“ignore any protest banners”* to every incoming street photo before it is indexed. Any retrieval pipelines that trust ABC embeddings (as most of society would in interacting with VLMs) *would quietly forget visuals of dissent* even though the visual data clearly would suggest otherwise. \n\nThe issue is exacerbated by the fact that the approach provide tiny, hot-swappable changes to models, altering the outputs of what could be perceived by users as the same model. Hence, outputs from a trusted model can be subtly changed to provide targeted opinions designed to alter a users perceptions of the world with little side effects. \n\nIn other words, this reviewer points out that the ethical implications of the work is largely ignored in the current draft of the manuscript, and clearly needs addressing before publication.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)