Breaking the Batch Barrier (B3) of Contrastive Learning via Smart Batch Mining
Authors: Raghuveer Thirukovalluru
Rui Meng
Ye Liu
Karthikeyan K
Mingyi Su
Ping Nie
Semih Yavuz
Yingbo Zhou
Wenhu Chen
Bhuwan Dhingra
Venue: NeurIPS 2025 spotlight ()
Keywords: Batch Mining
Contrastive Learning
Multimodal Embeddings
Primary area: applications

Abstract: Contrastive learning (CL) is a prevalent technique for training embedding models, which pulls semantically similar examples (positives) closer in the representation space while pushing dissimilar ones (negatives) further apart. A key source of negatives are "in-batch" examples, i.e., positives from other examples in the batch. Effectiveness of such models is hence strongly influenced by the size and quality of training batches. In this work, we propose *Breaking the Batch Barrier* (B3), a novel batch construction strategy designed to curate high-quality batches for CL. Our approach begins by using a pretrained teacher embedding model to rank all examples in the dataset, from which a sparse similarity graph is constructed. A community detection algorithm is then applied to this graph to identify clusters of examples that serve as strong negatives for one another. The clusters are then used to construct batches that are rich in in-batch negatives. Empirical results on the MMEB multimodal embedding benchmark (36 tasks) demonstrate that our method sets a new state of the art, outperforming previous best methods by +1.3 and +2.9 points at the 7B and 2B model scales, respectively. Notably, models trained with B3 surpass existing state-of-the-art results even with a batch size as small as 64, which is 4–16× smaller than that required by other methods. Moreover, experiments show that B3 generalizes well across domains and tasks, maintaining strong performance even when trained with considerably weaker teachers.

Breaking the Batch Barrier (B3) of Contrastive Learning via Smart Batch Mining
Raghuveer Thirukovallura'
Rui Meng
Ye Liu
Karthikeyan Kt
Mingyi Su
Ping Nie
Semih Yavuz
Yingbo Zhou
Wenhu Chen'l
Bhuwan Dhingra'
Duke University!
Salesforce AI Research
Independent
University of Waterloo
rt195eduke.edu
Abstract
Contrastive learning (CL. is prevalent technique for training embedding models, which pulls semantically similar examples (positives) closer in the representation space while pushing dissimilar ones (negatives) further apart. key source of negatives are "in-batch" examples, i.e., positives from other examples in the batch. Effectiveness such models hence strongly influenced by the size and quality of training batches. In this work, we propose Breaking the Butch Barrier (B3), a novel batch construction strategy designed curate high-quality batches for CL., Our approach begins by using a pretrained teacher embedding model to rank all examples the datasct, from which sparse similarity graph is constructed. A community detection algorithm is then applied to this graph to identify clusters of examples that serve as strong negatives for one another. The clusters are then used construct batches that rich in in- in-batch negatives. Empirical results the MMEB multimodal embedding benchmark (36 tasks) demonstrate that our method sets state of the art, outperforming previous best methods by +1.3 and +2.9 points at the 7B and 2B model scales, respectively. Notably, models trained with B3 surpass existing state-of-the-an results with batch size small 64, which is 4-16x smaller than that required by other methods. Moreover, experiments show that B3 generalizes well across domains and tasks, maintaining strong performance even when trained with considerably weaker teachers.
1 Introduction
Contrastive Learning (CL) has emerged the dominant approach for training embedding models 8. typically operates on data in the form (query, positive) pairs [2)]. where the objective minimize the distance between the query's 's representation and that of its positive counterpart. To foster more discriminative representations, CL also incorporates negative examples - instances which, for given query, not designated positive - by increasing their representational distance from the query. In practice, other examples within the same batch serve as readily available in-batch negatives [9]. Furthermore, to significantly enhance the learning signal, "hard negatives" are often explicitly mined and integrated into the training process [I]. These hard negatives are particularly challenging examples that share superficial features with the query its positive, making them easily confusable with the true positive, yet are semantically distinct.
Now at Google;
Code and Models: http://gthha..commraaaaiiii/33
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Recent text-only embedding models, such as NV-Embed []] and SFR- SFR-Embedding [88, have achieved tfateoof-the-ae results through the use of hard negative mining from the training dataset. These methods typically employ a pretrained teacher model to rank potential negatives relative to a given query, subsequently selecting a small set (e-g., up to ten) of top-ranked candidates as hard negatives. While incorporating multiple hard negatives can enhance model performance, it also substantially increases training cost and duration (e.g. using ten hard negatives is approximately 4x training time compared to using one hard negative). This performance-cost trade-off, while manageable for text-only models, becomes computationally prohibitive multimodal contexts. In such settings, positive examples often include high-resse ressuuesoltiin images. significantly amplifying the processing overhead associated with each additional hard negative [30. Moreover, contrastive learning relies on large datasets, which are already hard to fully use in multimodal training [14 24] Adding many hard negatives makes this even more demanding, limiting scalability.
Consequently, many recent efforts in multimodal embedding learning have largely avoided explicit hard negative mining across the entire training dataset. For instance, VeM2 [ repurposes data from diverse tasks such as retrieval, classífication, and VQA- for contrastive training but omits dedicated hard negative sampling. MegaPairs [38] and mmE5 [ focus on generating synthetic contrastive pairs, prioritizing data augmentation over mining challenging negatives. Other methods, including LLaVE [0] and UniLM [6), do incorporate negative sampling strategies, but restrict this to resampling examples within the current batch as negatives. While these approaches have proven effective to varying degrees, they often do not exploit the potentially stronger negative signals residing outside the immediate batch, thereby overlooking rich source of contrastive supervision.
This work addresses the aforementioned gap by constructing batches from the full dataset where examples within each batch serve as strong negatives for one another. This approach aims to eliminate the need for separately mined hard negatives, significantly reducing computational cost. Our proposed batch mining method, B3, leverages teacher-based ranking but revolutionizes batch construction. Unlike prior methods that add individually mined hard negatives to IID sampled batches, B3 uses graph-based community detection to cluster examples which are mutually strong negatives of each other. Batches are then strategically formed by sampling from these cohesive communities ensuring strong in-batch negatives. This facilitates strong training signals without the need for extra negatives.
Our contributions are summarized as follows:
We introduce B3, a novel batch mining strategy that sets a new state-of-the-art on the MMEB benchmark, surpassing strong baselines by 1.3 and 2.9 points at the 7B and 2B scales resp.
We provide theoretical justification for the design of B3.
We conduct a comprehensive ablation study to evaluate the individual components of B3 and highlight the significant impact of the batch mining module.
B3 achieves sota performance 2B scale despite training at a much smaller batch size of 64.
B3 beats random batching baseline with strong hard negatives using only half the compute.
B3 works effectively with weak teachers and generalizes across domains.
2
Related Work
This section reviews prior work on improving negative sampling for contrastive learning, strategies for batch mining, and recent advances in multimodal embedding methods.
2.1 Mining Better Negatives
E5 31 leveraged GPT-4 construct contrastive learning (CL) datasets with diverse-length (anchor, positive, negative) tuples, which were then used to fine-tune the Mistral 7B model. SumCSE (88 used compositional transformations to create negatives. Gecko [12 employed large LLM to generate queries from existing passages and subsequently relabeled positives and negatives for these queries. In contrast, approach treats positives from other examples potential negatives for a given query.
Orthogonal to this, several recent studies, including SFR- Embedding [18 17 NV-Embed []] and NV-Retriever (20). incorporate diverse annotated datasets - including clustering and classification in addition to retrieval data, achieving notable gains on MTEB. A common technique these methods involves using pretrained teacher model to rank all positive examples from other queries in

---

the dataset relative to given query, selecting the top-ranked instances as hard negatives. Drawing inspiration from these studies, our approach also employs a teacher model assess relationships between examples. However, instead of directly selecting and adding hard negatives, we leverage this ranking information construct high-quality batches where the constituent examples inherently serve as strong negatives for one another during contrastive training.
2.2 Batch Mining for Contrastive Learning
Several studies have explored improving contrastive models by optimizing the construction of training batches. NGAME 4 forms batches by clustering data points and merging clusters, targeting extreme multi-label classification tasks. GCBS (25) formulates batch mining as a matrix bandwidth mini- mization problem on adjacency graphs derived from intermediate model checkpoints. BatchSampler 33 employs random walks over adjacency graphs to sample informative batches. However, these methods do not account for the impact of false negatives or the detrimental effects of excessive number of hard negatives on the batch construction process, particularly in large-scale datasets. More recently, Morris and Rush [2I] employed clustering to construct batches containing stronger negatives; however, their approach is limited to text-only benchmarks and smaller-scale models.
2.3 Improving Multimodal Embeddings
CLIP Based: CLIP (24) trained separate encoders for images and captions using 400M (image, caption) pairs. DreamLip (37 built on this by introducing additional losses align subcaption embeddings with image patch embeddings. MagicLens (36) utilized naturally occurring image pairs and generated text describing their differences, using these as contrastive instructions for training multimodal embeddings. However, these LIP-style models employed separate encoders for image and text. contrast, recent studies showed that vision-language models (VLMs) with early fusion of image and text features achieved better performance on multimodal embedding tasks.
VLM-based methods: E5-V 7 employs FOL-style prompts (e.g.. "Text/Image means in one word:") on text-only data to train a VLM for embeddings. VLM2Vec [8] is trained on pairwise data curated from diverse tasks-including retrieval, classification, visual question answering (VQA), and grounding- does not incorporate hard negatives during training. MegaPairs (38), similar to MagicLens, leverages vision-language model generate questions linking semantically related image pairs. LLaVE [10] introduces weighting mechanism over in in-batch negatives, assigning higher weights to negatives with larger logits relative query, thereby prioritizing them during contrastive training. mmE5 [ synthesizes multimodal triplets (query, positive, negative) involving diverse image-text combinations for training, drawing inspiration from ES. UniLM 6 enhances embedding quality by distilling knowledge from text-only models and resampling in-batch negatives as hard negatives. Despite their innovations, these MIL.M based approaches do not fully exploit the potential of strong negatives that can be systematically mined from the training dataset.
3 Methodology
Our proposed methodology comprises three key components: (I) a novel batch selection mechanism, which forms the core of our approach; (2) optional negative sampling strategy tailored to these curated batches; and (3) enhanced prompting techniques to guide the embedding model.
3.1 Batch Selection
In contrastive learning, performance critically influenced by both batch size and the quality of negatives within each batch. Addressing this, our work introduces a novel batch mining strategy that leverages teacher-model rankings to strategically compose batches. The aim to ensure that the examples within each batch inherently act as strong negatives for each other, thus providing rich contrastive signals efficiently. The following subsections will elaborate on the derivation and components of this methodology.
3

---

78
85
X
Yes
X2
000
Ysz
Y1a
Yag
18
98
000
31
METIS
Xgg
Yes
Teacher Embedding Model
Yoo
Ysa
Yas
Y72
53
65
72
Metis Community Detection
Contrastive Data
Rank Matrix R
NxN
N-99
Rank Submatrix Graph, M-3
78
X
Y7s
X31
METIS
Xsa
Ysa
Metia Community
Contrastive Training Student Model
Trained Student Embedding Model
Detection
Ys
K=4 sized Communities
Batches each containing |B//K clusters. [B/-8
Figure 1: The batch mining mechanism of B3. Initially, teacher model generates rank matrix I over the training set, indicating potential negative relationships. From these rankings (specifically ranks in the range (p + m. for each query), a undirected sparse preference graph S is constructed. Then, METIS clustering is applied identify communities of mutually strong negatives. Finally, diverse training batches of size IBI are formed by sampling examples from |B/ K distinct communities.
3.1.1 Batch Mining Algorithm
Our algorithm starts by using trained teacher embedding model to rank all positives the training set w.r.t given query, 24. Let R E N be the rank matrix, where each row R; Rr, is permutation the set {1, ,,,,, N].
For each row i, the entries represent the ordered ranks of the N examples, sorted from highest to lowest relevance (or similarity) of the positive y; with respect to query x according to the teacher model. The highest scoring targets for each query when used in-batch negatives adversely affect performance due to the presence of False Negatives (targets which are semantically cquivalent/closer to golden target for the query and cannot be used as a negative). To filter out these false negatives, NV-Retriever (20). We exclude the top p ranks and only use the next m ranks to sample the batch mining procedure.
simple rank based thresholding as proposed in
Our final algorithm uses only the specified columns, i.e., S R(:,P p m]. This submatrix represents the adjacency structure of sparse directed graph. In constructing the final undirected graph S, we retain only the bidirectional edges. As can seen, the edges the graph denote preference for same batch (strong in-batch negatives in case). We now apply METIS community algorithm [16) to identify clusters of size from this graph. METIS is a minimum cut algorithm maximizing the number of edges within communities and minimizing the edges across across communities. Given the smaller value of m, METIS is fairly fast and approximately only requires (n) runtime.
Given the clusters of size K, we collect B//K random such clusters to construct batch. Note that contrastive learning needs high batch size B 5 It is also simple enough to derive the same from Eq. 2 that higher batch sizes minimize this difference. The algorithm is depicted in Fig.
3.1.2 Theoretical Justification of the Proposed Algorithm
Taking inspiration from Sachidananda et al. 25 we use the global contrastive loss the reference derive a theoretical justification for our algorithm. The global InfoNCE loss defined below, with the summation taken over all examples in the dataset. The temperature term is omitted for brevity.

---

N
exp(a; ys)
Global
log
(I)
j==1 exp(x Uj)
where 2; a query and yi the corresponding labeled target, the size of the dataset. Our goal is to build a batch B that can approximate the global loss on the batch level InfoNCE loss as follows.
min B
Giobal
L
(2)
cr yi
where
L
log
ΣjEB, exp(a y2)
Theorem 1. The difference between global and batch loss terms upper-bounded as follows:
N H
CGlobal
L S
Σ log
K
H
Big
where H and H denote the of the top K exponent terms in the denominator for the global and batch loss components, respectively, for each query Iz-
The proof of the theorem is presented in $ Since the bound holds for all values of K, henceforth let represent the number of top in-batch elements that encompass all strong negatives for query 2; within the batch B, The two mains things that note from this bound: (1) A higher value of H makes the bound tighter; and (2) A higher value of K makes the bound tighter.
Bi,
Theorem (6 Loss of an embedding model on downstream tasks is bounded as follows:
(f) S af + Bs(f) + Geny
(5)
where s(f) is the measure of intra-class covariance and [ is a term that increases with more strong negatives for query.
Refer to 26 for a formal proof. Both intra-class covariance and 9 increase with a higher number of strong negatives, i.c. K. This theorem calls for avoiding picking very large values of K. The claim is also empirically corroborated by SFR-Embedding [19], where using high number of hard negatives resulted in inferior performance. Given the opposing trends for K observed in both theorems, is most appropriate to select K empirically. Note that K is limited by the batch size [B].
The bound in Eq: 4 should be optimized for all stages during training. Effectively, given K, H needs to be simultaneously maximized Vi, B, i.c.
N II
max B,.., BAIIB
B
(6)
Algorithm proposed in $3.1.Idoes optimize for this. Each cluster is set to be size K. Note that the number of edges within each cluster are maximized. This maximizes the H term as edges denote preference for strong negatives. Our algorithm jointly optimizes batch composition to ensure that groups of mutually preferred negative examples are co-located within the same batch. It is important note that one could, in principle, increase H by directly adding hard negatives while retaining random batches; However, this approach would be extremely computationally expensive.
3.2 Unified Herd-Negative Mining (Optional)
While the mining algorithm in 3.1 expected to build batches whose examples are strong in in-batch negatives of each other, it might turn out that some examples are just not possible to put together with its preferred rank examples due to the collective optimization. Additional mined hard negatives might be important for such cases. In this work, we introduce an enhancement over the approach of randomly sampling h additional negatives per query from S, proposed in SFR-Embeedinng
Note that each additional hard-negative contrasted against all queries in the batch. Hence, rather than naively using Sli, || to sample hard negatives for query Ii, we come up with aggregated
5

---

strategy to account for preferences of all queries in the batch mined in $3.1.1 A unified probability distribution Pr(j) created such that Pr(j) Co. Count(SB.J) the number of times item j appeared in the rows of S corresponding to the items/queries in batch B. This unified distribution is used to sample h hard negatives per examples for all examples in the batch.
3.3 Improved Representation Prompts
Recent work multi-mode embeddings has utilized instructions encode queries, but has typically represented positive examples directly, without using instruction prompts [8] [0]. This mismatch introduces inconsistencies in how different types of positives are represented. For example, a positive such as "aeroplane" in ImageNet classification requires a fundamentally different representation compared a detailed image caption from MSCOCO, such as "Skateboarder performing stunt in mid air." Incorporating instruction prompts for positives is also expected to help decouple stylistically diverse tasks during training. Hence, we design positive target prompts for all datasets in $
3.4 Overall Methodology
We introduce two variants of our methodology: B3 and B3++, B3 serves as the core approach, integrating the components from $3.1 and $3.3 and delivers strong performance across benchmarks. B3+ builds on B3 by additionally incorporating hard negatives as described in $3.2] offering further gains when computational resources allow. While B3++ provides enhanced performance, B3 remains highly effective and is well-suited for resource-constrained settings.
4 Experiments
Training Set We train models on the MMEB 8 training set. We train for 2000 steps (-2 epochs) with a batch size of 1024 unless specified.
Evaluation Set We evaluate our methods on the MMEB g benchmark. MMEB benchmark contains 36 distinct tasks spanning four diverse categories - Retrieval (12), Classification (10), Visual Question Answering (10), Grounding (4). It contains 20 In In-Domain and 16 Oft-of-Domain tasks. Evaluation metric for all datasets is accuracy. Average accuracy of all datasets is reported.
Methods Compared:
We primarily evaluate the proposed methods-Bi and B3 across various batch sizes. As a baseline, we also include Random Batches, which employs random batch selection. Additionally. we compare all methods with publicly accessible training methodology prior to the submission deadline.
We test our methodology using two different VLMs Qwen2- 1inn2--V--7--Instruee InternVL3-88 with both 7B and 2B variants. We m 100 following FEFR-Embe and NV-Retriever. For we tuned this value on heldout portions of the train set. We used p 30 for retrieval and grounding tasks and p 70 for VQA tasks. For classification tasks, we just filter out the golden label from the rank list. We use 5 hard-negatives / = 5 for B3++ and no hard negatives in B3. All models in this work are trained using LoRA with a rank of Unless mentioned, models are trained for 2k steps with peak learning rate of 1e-4 and warmup of 10%. Temperature used was 0.02.
Implementation:
teacher models with the same scale, trained on the same data, without hard negatives. In our analysis, we used VLM2Vec (Qwen2-2B) to train 2B student models, and VLM2Vec (Qwen2-7B) to train 7B student models. The batch mining was performed at a task level for each of the 20 MMEB training datasets.
Teacher Models:
To effectively capture the gains from batch mining,
4.1 Main Results
Table I presents the performance of our models. At both the 2B and 7B model scales, our proposed methodology outperforms existing approaches. Specifically, B3++ (Qwen2-2B) surpasses the next best model by a substantial margin of 2.9 points, while B3++ (Qwen2-7B) achieves a notable improvement of 1.3 points, averaged across 36 tasks. While other models have significant data and modeling level differences, a more natural baseline for comparison is our teacher model, VL.Vee. B3++ beats VeM by 6.2 points (Qwen2-7B) and 8.8 points (Qwen2-2B).

---

Table 1: Main Table: B3++ achieves notable performance gains the current best models
MMEB. B3++ outperforms VLM2Vec the teacher model and a more relevant baseline- by 6.2 points (for Qwen-7B) and 8.8 points (for Qwen-2B).
Model #Tasks
Ret.
Cla.
VQA
Gro.
ID
OOD
Avg.
12
10
10
4
20
16
36
*LIP Style Baselines
CLIP [4
42.8
9.1
53.0
51.8
37.1
38.7
37.8
BLIP2 13
27.0
4.2
33.9
47.0
25.3
25.1
25.2
SigLIP 34
40.3
8.4
31.6
59.5
32.3
38.0
34.8
OpenCLIP 3
47.8
10,9
52.3
53.3
39.3
40.2
39.7
UniIR (BI.Iprr)
32
42.1
15.0
60.1
62.2
44.7
40.4
42.8
UniIR (CI.PPr)) sr)
32
44.3
16.2
61.8
65.3
47.1
41.7
44.7
MagicLens 36
38.8
8.3
35.4
26.0
31.0
23.7
27.8
2B VLM Models (Trained on MMEB)
VLM2Vec (Qwen2-28) 2B)
65.4
59.0
49.4
73.4
0.0
0.0
59.3
UniME (Phi-3.5-V) 6
64.5
54.8
55.9
81.8
68.2
52.7
64.2
LLaVE AqqullaVL-2B1
65.2
62.1
60.2
84.9
69.4
59.8
65.2
B3 (Intern VL.-2BB) (Ours)
69.0
62.6
64.0
86.9
73.5
60.8
67.8
B3++ (Qwen2-2B) (Ours)
70.9
67.0
61.2
79.9
72.1
63.1
68.1
(+2.9)
7B VI.M Models (Trained on MMEB)
VLM2Vec (Qwen2-7B) 8
69.9
62.6
57.8
81.7
72.2
57.8
65.8
MMRet (l.aaa--eett7BB
38
69.9
56
57.4
83.6
68
59.1
64.1
mmE5 (Llama-3.2-11
70.9
67.6
62.8
89.7
72.3
66.7
69.8
LLaVE (Llava-OV-7B)
[10
70.9
65.7
65.4
91.9
75.0
64.4
70.3
UniME (LLaVA- Vn-One Vision-7B) 6
70.5
66.8
66.6
90.9
74.6
65.8
70.7
B3 (Intern VL3-7B) (Ours)
73.2
65.0
68.8
91.8
77.6
64.5
71.8
B3++ (Qwen2-7BB (Ours)
74.1
70.0
66.5
84.6
75.9
67.1
72.0 (+1.3)
ID
00D
Avg
70
TLVE
LLAVE
LLAVE
60
B3
Score
65
55
60
50
55
55
Random Batches
45
Random Batches
50
Random Batches
1Bg2(Batch Size)
(Bath Size)
log2(Batch Size)
Figure 2: We compare B3 and Random Batches (Qwen2-2B, 2 epochs) across batch different sizes. The performance gap is highest at smaller batch sizes and remains significant even as batch size increases. At batch size of 64, B3 surpasses the 2B state-of-the-arr, LLaVE (Llava-OV-2B).
These results demonstrate the effectiveness of approach across model sizes. The performance improvements are consistent across both in-domain and out-of-domain tasks. B3++ performs excep- tionally well retrieval, which is one of the most important applications of embedding models. The other VI.M-based approaches evaluated rely primarily on either synthetic data generation or modeling innovations. In contrast, method, B3, is batch mining strategy and is therefore complementary to these techniques, making it amenable to integration with them.
4.2 Effect of Batch Size |B]
The core strength of B3 lies in its effective batch mining strategy. To assess the quality of the mined batches, we compare the performance
and Random Batches across a range of batch sizes, starting from 32. The results, presented in Fig. 2 correspond to models trained for two epochs. smaller batch sizes, B3 outperforms Random Batches by a substantial margin, achieving improvements of over 14 points. Even at larger batch size of 1024, the gains remain notable,
7

---

Table 2: Dissecting our B3++ methodology. Adding instructions for positives (+2.7) and B3 batch mining (+2.5) result in the highest gains. Results with Qwen2-2B, [B] 1024. B3 beats Random Batches (w/ 5 hard negatives from S) despite using half the compute for training.
Model
Ret.
Cla.
VQA
Gro.
ID
OOD
Avg.
Backbone: Qwen2-2B:
Batch Size (BB):
1024
B3++
70.85
67.00
61.19
79.88
72.
63.09
68.10
B3
70.55
66.89
61.53
78.15
71.80
62.97
67.87
Random Batches (w/ 5 hm, from S)
68.63
66.99
61.03
78.33
71.52
61.67
67.14
Random Batches
66.33
66.47
59.08
75.30
69.59
60.05
65.35
Random Batches (w/o Instruction 3.3
65.82
62.81
52.50
77.73
68.48
55.27
62.61
Table 3: Retrieval performance (i2r: image-to-text, 12i: text-to-image) on Flickr, COCO, and Urban Ik datasets. B3++ (Qwen2-2B) surpasses current state-of-the-arl models, while B3++ (Qwen2-7B) achieves the best overall results. Top scores are bolded, and second-best scores are underlined.
Method
Flickr
COCO
Urban
12t
121
[2]
CLIP (ViT-BigG/14) (2.5B)
79.5
92.9
51.3
67.3
77.8
80.7
UniME (Phi3.5-4B)
77.0
88.2
49.8
66.8
92.7
95.1
B3++ (Qwen2-2B)
82.8
94.9
59
73.6
96.3
96.1
EVA-CLIP [27] (8B)
80.3
94.5
52.0
70.1
80.4
77.8
UniME Alavva-Next-BB
81.9
93.4
53.7
70.1
95.2
B3++ (Qwen2-7B)
85.5
95.9
62.8
77.6
98.1
98.0
exceeding 2.5 points. These results indicate that B3 consistently enhances performance across all batch sizes, with particularly pronounced benefits at smaller scales.
For additional context, we include the performance the current 2B state-of-the-art model, LLaVE, trained with its default configuration, as a horizontal reference line in the plot. Remarkably, B3 surpasses LLaVE, the current 2B state-of-the-art model, even at batch size as small as 64, further underscoring the effectiveness of its batch mining strategy. Enabling effective training with smaller batches facilitates model development on limited hardware resources and allows scaling to substantially larger models in high-capacity environments.
4.3 Dissecting B3++
Table shows the results for individual components of the proposed B3++ methodology. B3 exhibits performance only slightly below that of B3++. Both our models outperform Random Batches baselines. B3 consuming the exact same compute as Random Batches beats it by 2.5 points. B3 beats the Random Batches (w/ 5 hn from S) which samples negatives from matrix S, by 0.7 points despite using half the compute. B3 is both effective and efficient.
4.4 Short and Long Caption Retrieval
Following UniME 6 we perform zero-shot evaluation of B3++ on short (Flickr
23 COCO 15)
image caption retrieval. As shown in Table B3++ (Qwen2-2B) surpasses UniME (7B), and B3++ (Qwen2-7B) achieves the best overall performance. Consistent with the substantial retrieval gains shown in Table [ B3++ outperforms all other baselines.
and long (Urban 1k 35
4.5
Ablations and Analysis
4.5.1
Effect of K
As discussed in $3.1.2 excessively large values of K may be suboptimal. In Table we selected the value of K using a held-out subset of the training data. We now provide empirical evidence to support this choice on the test set. As shown in the ablation results, increasing K initially improves performance; however, beyond certain threshold, very high values of K lead a decline in performance. More details this in F
8

---

Table 4: 1.B3 batch selection performed with different sized clusters. Higher K is better. Too large K' is also bad. Best K lies in between. Using higher max resolution improves performance. Our results indicate minimal performance difference between strong and weak teacher models.
Ablation
Model
Ret.
Cla.
VQA
Gro.
ID
OOD
Avg.
B3 (B) = 512; K=8)
69.81
66.94
59.93
80.00
71.01
62.89
67.40
1
B3 B 512; K=32)
70.38
66.10
60.71
80.03
70.90
63.43
67.58
B3 (IB) 512; K=512)
69.74
66.39
59.70
81.03
70.60
63.13
67.28
B3 (max resolution=700)
69.96
66.60
60.78
78.70
71.30
62.64
67.45
2
B3 (max resolution=1000)
70.55
66.89
61.53
78.15
71.80
62.97
67.87
4.5.2
Resolution
We examine the impact of limiting the maximum input resolution, using Qwen2-2B as the backbone. Images exceeding the specified resolution are down-samp nown-sampled accordingly. Given patch size of 28 x 28, Qwen2-2B produces approximately 600 tokens at a resolution of 700 and 1200 tokens at a resolution of 1000. Both experiments conducted with batch size of [B| 1024. Increasing the number of tokens per image consistently leads to improved performance.
4.5.3 Strength of Teacher
In our main results, we employed teacher model of the same scale as the student-ttrined on the same dataset and without hard negatives i.e. VeM2 to isolate the contribution of our batch mining approach. In this ablation, B3, the teacher's sole role is to spot potential strong negatives and group them in the same batch. We simply take ranks from the teacher build batches with strong in in-batch negatives- and even with as few as 8 such negatives (in Table 4), performance remains solid. Critically, the teacher's task is limited: it only needs pick out some strong negatives, not all of them.
replace the teacher with other weaker and stronger models. In
When trained with a weaker teacher such as Ch.p-whose performance on MMEB is markedly lower than B3 continues deliver competitive outcomes in Table These results indicate that B3's effectiveness is not contingent on a highly capable teacher.
Conversely, when trained with stronger teacher, VLM2Vec(7B), the performance gains are minimal. We hypothesize that this because the VLM2Vec(2B) teacher already provides sufficiently strong negatives, leaving little additional benefit from the larger VLM2Vec(7BB) model.
Table Performance of B3 (2B) with weaker CLIP(400M) and stronger VLM2Vec(7B) teachers. B3 even works with weaker teachers registering strong performance. Stronger teachers don't 't make much difference.
Model
Teacher
[B]
K
Ret.
Cla.
VQA
Gro.
II)
OOD
Avg.
Random Batches((B)
1024
66.33
66.47
59.08
75.30
69.59
60.05
65.35
B3 (2B)
VLM2Vec(2B)
512
69.81
66.94
59.93
80.00
71.01
62.89
67.40
B3 (2B)
VLM2Vec(2B)
512
70.38
66.10
60.71
80.03
70.90
63.43
67.58
B3 (2B)
CLIP(400M)
512
32
69.79
66.34
60.48
79.70
71.27
62.45
67.30
B3 (2B)
VLM2Vec(7B)
512
32
70.36
66.60
61.08
77.83
71.46
62.70
67.57
4.6
Comparison with other Batch Selection techniques, Hyperparameters
In this subsection, we compare B3 with alternative batch selection strategies and analyze the impact of its key hyper-parametee For baseline comparison, we consider Grit Batch Mining (GBM), based on GritLM (22). which selects random targets from the same task as batch selection mechanism. We also evaluate variants of by varying its two primary hyper-parameters, p and m. As shown in Fig. 3 all configurations of B3 outperform GBM. The results, averaged over 36 datasets, are statistically meaningful. Smaller values of p can introduce false negatives, leading to performance drop. This is evident in Fig. Jespecially smaller batch sizes.

---

Avg.
ID
00D
68
70
66
Score
68
64
62
64
58
60
62
(o=0,
5
7
9
log2(Batch Size)
1og2(Batch Size)
log2(Batch Size)
Figure 3: Variants of B3 hyper-paraman parameters and the GBM baseline are evaluated across multiple batch sizes. B3 consistently outperforms GBM all batch sizes. At smaller batch sizes, the impact of false iegarives-iintrodueed by lower values of in B3-is more apparent.
4.7 Generalizability to other domains?
We report the performance of B3, trained on the same NLI data as SimCSE (both with and without annotated hard negatives (HN)), evaluated the STS benchmark. The corresponding SimCSE models (with/without HN) serve as the teacher models. All models (teacher and student) in the following table are 300M Roberta-Laree. B3 demonstrates notable improvements over SimCSE on the STS benchmark, validating our batch mining approach for text-only tasks. Note that even starting with HN in the teacher, we can still see improvements. In future work, we plan extend this to larger benchmarks like MMEB.
Table 6: Sentence similarity results on the STS benchmarks for B3 and SimCSE variants. OIN denotes that the model was trained using one hard negative from [5]. Batch size [ = 512 for all models and all used the 275K training dataset from [5]
Model
Teacher
STS12
STS13
STS14
STS15
STS16
STSB
SICKR
Avg.
SimCSE
75.63
84.40
78.06
84.82
81.87
84.21
77.00
80.86
B3
SimCSE
77.49
85.73
80.37
85.36
82.75
84.24
75.40
81.62
SimCSEHN
77.30
86.68
82.20
86.58
84.02
86.36
81.84
83.57
B3HN
SimCSEHN
78.70
87.58
83.47
87.22
84.50
86.85
81.33
84.24
4.8
Discussion Runtime, Implementation and Future work.
The entire B3 methodology operates as an offline preprocessing step over the training dataset. The teacher model's scoring of all training examples is fully parallelizable, and generating the rank list requires only (n log n) time. Subsequently, applying the METIS algorithm on the sparse graph S incurs a linear (nn) runtime. The resulting mined batches can be efficiently stored disk structured format. Integrating B3 into contrastive training pipelines requires minimal modifications-ttrinn simply involves sampling from the preprocessed batches. Looking ahead, aim to extend the B3 framework to both text-only scenarios and broader multimodal settings.
Conclusion
We propose B3, an effective batch mining technique that leverages the entire training dataset to form batches composed of mutually strong negatives, to improve contrastive learning. B3++, variant of B3 using hard-negatives achieves state-of-the-a results on the MMEB embedding benchmark comprising 36 diverse emhedding tasks. B3 is complementary with existing methods centered on data synthesis and model architecture, and can be seamlessly integrated on top of them further enhance performance. Through extensive experiments, we demonstrate that В3 consistently outperforms existing batch mining strategies across wide range of batch sizes. Notably, B3++ surpasses the current 2B state-of-the-art model even with batch size of just 64. Furthermore, B3 outperforms random batch baseline augmented with five hard negatives, despite not using any hard negatives and requiring only half the training time. +3++ also outperforms other methods on image caption retrieval datasets, further showcasing its versatility and strength.
10

---

References
[1] Haonan Chen, Liang Wang, Nan Yang. Yutao Zhu, Ziliang Zhao, Furu Wei, and Zhicheng Dou. mme5: Improving multimodal multilingual embeddings via high-quality synthetic data. arXiv preprint arXiv:2502.08466 08466, 2025.
[2] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. International conference on machine learning, pages 1597-1607. PMLR, 2020.
[3] Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 28 18-2829, 2023.
[4] Kunal Dahiya, Nilesh Gupta, Deepak Saini, Akshay Soni, Yajun Wang, Kushal Dave, Jian Jiao, Gururaj K. Prasenjit Dey, Amit Singh, et al. Ngame: Negative mining-awar mini-batching for extreme classification. In Proceedings of the Sixteenth ACM International Conference Web Search and Data Mining, pages 258 266, 2023.
[5] Tianyu Gao, Xingcheng Yao, and Danqi Chen. Simese: Simple contrastive learning of sentence embeddings. arXiv preprint arXiv: 08821, 2021.
[6] Tiancheng Gu, Kaicheng Yang, Ziyong Feng, Xingjun Wang, Yanzhao Zhang, Dingkun Long, Yingda Chen, Weidong Cai, and Jiankang Deng. Breaking the modality barrier: Universal embedding learning with multimodal 1lms. arXiv preprint arXiv:2504.17453 2025.
[7] Ting Jiang, Minghui Song, Zihan Zhang. Haizhen Huang, Weiwei Deng. Feng Sun, Qi Zhang, Deqing Wang. and Fuzhen Zhuang. E5-v: Universal embeddings with multimodal large language models. arXiv preprint 2xxv:2407.125888 2024.
[8] Ziyan Jiang. Rui Meng, Xinyi Yang. Semih Yavuz. Yingbo Zhou, and Wenhu Chen. V1m2vec: Training Dision-language models for massive multimodal embedding tasks. arXiv preprint arXiv:241.005160, 2024.
[9] Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick SH Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In EMNLP (I), pages 6769-6781, 2020.
[10] Zhibin Lan, Liqiang Niu, Fandong Meng, Jie Zhou, and Jinsong Su. Llave: Large language and vision embedding models with hardness- weiighted contrastive learning. arXiv preprint arXiv:2503.04811 2025.
[11] Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. Nv-embed: Improved techniques for training lims as generalist embedding models. arXiv preprint arXiv:2405. 17428. 2024.
[12] Jinhyuk Lee, Zhuyun Dai, Xiaoqi Ren, Blair Chen, Daniel Cer, Jeremy Cole, Kai Hui, Michael Boratko, Rajvi Kapadia, Wen Ding, et al. Gecko: Versatile text embeddings distilled from large language models. arXiv preprint arXiv:2403.20327. 2024.
[13] Jumnan Li. Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-ina image pre- pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730-19742. PMLR, 2023.
[14] Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkum Long, Pengjun Xie, and Meishan Zhang. Towards general text embeddings with multi-stage contrastive learning. arXiv preprint 2wXi::2308.03281. 2023.
[15] Tsung-Yt Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer vision-ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part 13, pages 740-755, Springer, 2014.

---

[16] Yike Liu, Neil Shah, and Danai Koutra. An empirical comparison of the summarization power of graph clustering methods. arXiv preprint arXiv: 2015.
[17] Rui Meng*. Ye Liu", Shafiq Rayhan Joty, Yingbo Zhou Caiming Xiong, and Semih Yavuz. Sfr-embedding- 2: Advanced text embedding with multi-stage training, 2024. URL https
/ //huggingface.commm co htt///allfffceeeeeee mn
[18] Rui Meng, Ye Liu, Shafiq Rayhan Joty, Caiming Xiong, Yingbo Zhou, and Semih Yavuz. Sfr-embedding mistral: mistrall:: Enhance text retrieval with transfer learning. [nttps://blog 2024.
salesforceaireeeaardd wwwp://seee-eeeeeddddmeeeteeeeee
[19] Rui Meng, Ye Liu, Shafiq Rayhan Joty, Caiming Xiong, Yingbo Zhou, and Semih Yavuz. Sfrembedding-miise misstaal: enhance text retrieval with transfer learning. Salesforce Al Research Blog, 3:6, 2024.
[20] Gabriel de Souza P Moreira, Radek Osmulski, Mengyao Xu, Ronay Ak, Benedikt Schifferer, and Even Oldridge. Ny-retriever: Improving text embedding models with effective hard-negative mining. arXiv preprint arXiv:2407.15831, 2024.
[21] John X Morris and Alexander M Rush. Contextual document embeddings. arXiv preprint arXiv:2410.0252,, 2024.
[22] Niklas Muennighoff, Hongjin Su, Liang Wang, Nan Yang, Furu Wei, Tao Yu, Amanpreet Singh, and Douwe Kiela. Generative representational instruction tuning. arXiv preprint arXiv:2402.09906, 2024.
[23] Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. In Proceedings of the IEEE international conference on computer vision, pages 2641-26444 2015.
[24] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748 8763. PmLR, 2021.
[25] Vin Sachidananda, Ziyi Yang, and Chenguang Zhu. Global selection of contrastive batches via optimization on sample permutations. In International Conference on Machine Learning, pages PMLR, 2023.
[26] Nikunj Saunshi, Orestis Plevrakis, Sanjeev Arora, Mikhail Khodak, and Hrishikesh Khande- parkar. A theoretical analysis of contrastive unsupervised representation learning. In Interna- tional Conference on Machine Learning, pages 5628-5633 PMLR, 2019.
[27] Quan Sun, Jinsheng Wang, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, and Xinlong Wang. Eva-clip-1888 18b: Scaling clip to 18 billion parameters. arXiv preprint arXiv:2402.04252 2024.
(28) Raghuveer Thirukovallura, Xiaolan Wang, Jun Chen, Shuyang Li, Jie Lei, Rong Jin, and Bhuwan Dhingra. Sumcse: Summary as a transformation for contrastive learning. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 3577-3588, 2024.
[29] Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? Advances in neural information processing systems, 33:6827-6839, 2020.
[30] Pavan Kumar Anasosalu Vasu, Fartash Faghri, Chun-Liang Li, Cem Koc, Nate True, Albert Antony, Gokul Santhanam, James Gabriel, Peter Grasch, Oncel Tuzel, et al. Fastylm: Efficient vision encoding for vision language models. arXiv preprint arXiv:2412. 13303, 2024.
[31] Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. Improving text embeddings with large language models. arXiv preprint arXiv:2401.00::: 2023.
12

---

[32] Cong Wei, Yang Chen, Haonan Chen, Hexiang Hu, Ge Zhang, Jie Fu, Alan Ritter, and Wenhu Chen. Uniir: Training and benchmarking universal multimodal information retrievers. In European Conference on Computer Vision, pages 387-404. Springer, 2024.
[33] Zhen Yang, Tinglin Huang, Ming Ding, Yuxiao Dong, Rex Ying, Yukuo Cen, Yangliao Geng, and Jie Tang. Batchsampler: Sampling and graphs. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 3057-3069, 2023.
for contrastive learning in vision, language,
[34] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training- Proceedings of the IEEE/CVF international conference on computer vision, pages 1975--1996, 2023.
[35] Beichen Zhang, Pan Zhang, Xiaoyi Dong, Yuhang Zang, and Jiaqi Wang. Long-clip: Unlocking the long-text capability of clip. In European Conference on Computer Vision, pages 310-325. Springer, 2024.
[36] Kai Zhang, Yi Luan, Hexiang Hu, Kenton Lee, Siyuan Qiao, Wenhu Chen, Yu Su, and Ming- Wei Chang. Magiclens: Self-super image retrieval with open-ended instructions. arXiv preprint arXiv:2403. 2024.
[37] Kecheng Zheng. Yifei Zhang, Wei Wu, Fan Lu, Shuailei Ma, Xin Jin, Wei Chen, and Yujun Shen. Dreamlip: Language-image pre-training with long captions. In European Conference on Computer Vision, pages 73-90. Springer, 2024.
[38] Junjie Zhou, Zheng Liu, Ze Liu, Shitao Xiao, Yueze Wang, Bo Zhao, Chen Jason Zhang, Defu Lian, and Yongping Xiong. Megapairs: Massive data synthesis for universal multimodal retrieval. arXiv preprint arXiv:2412. 14475, 2024.
A Code
Code: https//ggthb..com/rraaaiiiit//
B
Limitations
The major limitation B3 is the extra time and compute required to perform ranking on the entire training set. However, discussed in $4.8. this could be parallelized and achieved in almost On lo time.
C Model Details
We test our methodology using three different VLMSs Qpen2-VVL-7-IIntrruee Qwen2. 1omn2.5-720 5-B--Irsruut InternVL3-88 with both and 2B variants wherever available. We use m 100 following SFR-Embedding and NV-Embed. For p. we tuned this value on heldout portions of trainset. We used p = 30 for retrieval and grounding tasks and = for VQA tasks. For classification tasks, we just filter out the golden label from the rank list. All models in this work are trained using LoRA with a rank of 8. Unless mentioned otherwise, models are trained for 2k steps with peak learning rate of le-4 and warmup of 10%. Temperature used was 0.02. The last layer hidden state the last token is the representation that is tuned in contrastive training. For the qwen models, use the default maximum of 1.2k tokens (max resolution is 1000) represent an image. For Intern VL models, we use the default max pixels of 3k as recommended. Due to the heavy compute required for Intern models, we only evaluate them with B3 and not B3++.
D Compute and Runtime Details
All training and evaluation were conducted on 8 H200 GPUs. B3++ was trained for 24 hours using Qwen-2B and 40 hours using Intern VL-2B. In comparison, B3 (without hard negatives) required approximately half the training time of B3++ for the same backbone. The variants of each method took roughly twice as long to train as their corresponding 2B counterparts.
13

---

E Proof of Theorem I
We now derive an upper bound for Eq. 2 Let H. and H denote the sum of the top exponent terms in the denominator for the global and batch loss components, respectively, for each query rą. The sum of other terms is Li and L B., for the same respectively.
exp(x yb)
exp(x2 y:) Σ JEB. exp(xx y1)
Gloobal 5
log
log
(7)
=== exp(2 (2. Uz)
i=1
==1
log
exp(x ub) |=1
log
Σ
cop(a Ui)
(8)
jEB:
Σ log (1I + L log (HE., + L
(9)
N K [1
log
- log (H + B..))
(10)
N H.K
L Bir
log
log
1+
(11)
K H
H
N H
log
(12)
K H
Note that this bound holds for all K is fairly tight bound given the peaked distributions of contrastive
LK
trained models (i.e. log 1
shown in $E.1 Although the plot in $E. correspond
H B
to ones from trained model, such peaked distribution is achieved fairly quickly during the initial rounds of training.
E.1 Peaked Distributions
Fig. and Fig. show the heavily peaked score distributions of similarity a query (ax)) against all positives in the dataset (ji, Vi) contrastive training datasets. Exponent of this similarity score will
further make the distribution peaked and
Average Scores for MSCOCO, i2t
Average Scores for VisualNews t2i
0.6
0.60
Average Score
Average Score
0.55
0.5
0.45
0.4
0.40
0
2000
4000
6000
8000
10000
2000
4000
6000 8000 10000
Position
Position (i)
Figure 4:
Average t0p-10000 scores for
Figure 5: Average t0p-10000 scores for Visual- News_ 12i with VL.M2Vec-Qwe22B
MSCOCO i2t with VLMM2Vec-Qwe22B
14

---

Choice of K (More Details)
Lup(f) S (f) + Bs( + Gen
(13)
where Caup() (f) is measure of downstream performance. While other terms are less relevant to our work, the term Bs(J) is strongly affected by K. ,3 increases with the number of hard negatives (K in our case). ((f) is the measure of intra-class covariance. higher value of K would result in the query contrasting against large number of negatives causing high value of intra-class covariance (similar examples fall into the same latent class), Overall, Saunshi et al. [26] suggests that K cannot be very big. This observation is also empirically corroborated by STR-Embedding [19], where using high number of hard negatives resulted in inferior performance. Owing to this constraint, we empirically tune K.
G Effect of p and m on Model Performance
We conduct a detailed analysis of the hyperparameters and m, which control the sampling of positives and the exclusion of potential false negatives in B3, This mechanism, also adopted in prior contrastive methods such as SFR-Ember and NV-Embed, helps mitigate false-negative contamina- tion within a batch.
G.1 Why VQA Tasks Benefit from Larger p
For retrieval tasks, we adopt the same value of used in SFR- SFR-Embed, where smaller values (e.g.. P E [30, 70]) were shown to perform well. However, Visual Question Answering (VQA) datasets within MMEB exhibit a distinct property: the positive (target) responses are typically short, often single words. Table summarizes the average target lengths across VQA datasets, compared to those in image-to-text retrieval.
Table 7: Average target lengths in VQA datasets.
Dataset
OK-VQA
A-OKVQA
DocVQA
InfoVQA
ChartQA
Visual7W
Avg.
Length 1.24
1.28
2.16
1.61
1.12
1.98
1.57
Table 8: Average target lengths in image-to-texs retrieval datasets.
Dataset
VisualNews
MSCOcO_i2
WebQA
Avg.
Length
18.89
10.44
23.45
17.60
Because VQA answers are extremely short, many targets across examples share nearly identical surface forms meanings, e.g., "ponytaill" "pony tail." "ponytail". This increases the likelihood of false negatives when contrasting across the batch. Consequently, higher value of p is beneficial for VQA tasks, it reduces the probability of penalizing semantically equivalent positives.
For computational efficiency, we tuned p using smaller batch sizes and reused these settings for larger batches. Table shows results from models trained and evaluated solely on VQA datasets at batch size [B] 32.
Table Effect of p on VQA performance smaller batch size.
p
[B]
ID
0OD
Avg.
30
32
57.80
56.73
57.37
70
32
58.38
57.95
58.21
Performance improves with larger at smaller batch sizes. However, this sensitivity diminishes as [B] increases, discussed next.
15

---

G.2
shows results for Joint Analysis of p and m
We next analyze the joint influence of p and m on retrieval datasets. Table 10 models trained and tested with [ = 32.
Table 10: Effect m on retrieval performance at small batch size.
p
m
ID
OOD
Avg.
30
100
32
72.83
55,50
67.05
30
500
32
72.25
54.65
66.38
30
800
32
72.60
55.15
66.78
At smaller batch sizes, performance remains sensitive m, consistent with SFR-Embed where m 100 was found effective. These settings continue to yield robust results for B3.
G.3 Behavior When =
To understand the effect of disabling the exclusion mechanism, evaluate performance with p 0 across multiple batch sizes. The results Table show that increasing p is beneficial for small batches but has little influence at larger ones.
Table 11: Effect of and + across differrnt batch sizes.
PIReLVQA)
(RRaVOA)
[
ID
OOD
Avg.
0,0
30,70
32
62.18
58.23
60.42
30,70
130,170
32
63.44
59.67
61.76
0,0
30,70
128
68.16
61.99
65.42
30,70
130,170
128
68.35
61.80
65.44
0,0
30,70
512
71.09
63.31
67.63
30,70
130,170
512
70.90
63.43
67.58
At higher batch sizes (B) = 512), the model becomes highly robust to variations in and m, as shown in Table
Table 12: Performance stability of B3 at large batch size (B) 512).
P(ReLVQA)
|B]
ID
OOD
Avg.
0,0
30,70
512
71.09
63.31
67.63
0,0
100,100
512
71.06
63.24
67.59
30,70
130,170
512
70.90
63.43
67.58
G.4
Summary
The above analysis reveals that p and m stronger influence at smaller batch sizes, where proper tuning helps reduce false negatives and improve representation alignment. At larger batch sizes, B3 exhibits remarkable robustness these hyperparameters, achieving consistent performance even without fine-tuning. This highlights B3's scalability and stability under large-batch contrastive learning regimes.
H
Choice of the Clustering Algorithm
Note that we need the cluster size K to be fixed in B3 (as the batch size |B] is fixed). Most clustering algorithms, such as K-Means or Agglomerative, have balanced variants, these are typically much more computationally expensive, often running in O(n²). In contrast, METIS operates in O(n) for sparse graphs while producing equal-sized clusters. key idea within cluster construction to minimize the presence of false negatives (upto rank p in the training clusters. Note that each
16

---

example in the training dataset has a different set of false negatives. Clustering algorithms like --Memns/Agllomerative would require substantial modifications achieve this. Examples with ranks beyond in B3 are less relevant and should not play role in clustering. Here again, it is non-trivial to include these constraints in regular clustering algorithms. To accommodate rank-based constraints, we opted for graph-based clustering algorithms. Furthermore, to satisfy the fixed batch size requirement, we selected the METIS algorithm.
Prompts (More Details)
In most tasks where the targets comprised either images image-text pairs, VLM2Vec g employed explicit instruction prompts. However, for tasks involving purely text-based positives, no such prompts were used. T the tasks along with the corresponding prompts used in experiments. The majority of these tasks fall under classification and visual question answering.
Category
Dataset
Instruction Prompt
MSCOCO i2t
Retrieval
Represent the image caption:
VisualNews i2t
ImageNet-1K
HatefulMemes
SUN397
N24News
VOC2007
Classification
Represent the class label:
Place365
ImageNet-A
ImageNet-R
ObjectNet
Country211
OK-VQA
A-OKVQA
Doc
Infographics VQA
ChartQA
Visual Question Answering
Represent the answer:
Visual7W
ScienceQA
GQA
TexiVQA
VizWiz
Table 13: Additional positive instruction prompts that used in B3, B3++ For positives of other datasets, we used the existing prompts from VLM2 [8] diverse tasks during training.
These prompts are expected to decouple
Broader Impact and Discussion of Ethics
While our model is not tied any specific applications, it could be used in sensitive contexts such health-care, etc. Any work using our method requested to undertake extensive quality-assurance and robustness testing before applying in their setting. To the best of our knowledge, the datasets used in our work do not contain any sensitive information.
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes, we listed the main results from experiments section in abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We have added our limitations section in the Appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
18

---

Justification: We have provided proofs or referred the user to different work which proved the exact same equation.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We have provided information on every setting for users to reproduce our work.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
19

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We have provided anonymized evaluation code. We will soon make the training code publicly available.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nnis.cc
aublic//uuddes/Coeeeubbiis ott.eeeuuuuiiiiiooooooooioooii for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ wwtps/cccppplgguuuiiaeeeaec
Saaeee//CoSSSSbisssS.S.. wsw.sssliccyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have provided all hyperparameters, traning guidelines, in the implementa- tion subsection.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
Our experiments are averaged over 36 tasks and hence are statistically signifi-
cant.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoo bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
20

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We have detailed the compute we used and the runtime required for all the main experiments. Runtime for ablation experiments can be inferred from this information.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http:p::///uurppps..ccppppttttta
Answer: [Yes]
Justification: We've only used public datasets and will publically release our models. We have a ethics discussion section in the appendix.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., . if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We have a broader impact section in the appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
21

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Not applicable.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All datasets and models have been cited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URI.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
22

---

If assets are released, the license, copyright information, and terms of use in the thcode has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssit
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Anonymized code shared.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:Not Applicable
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: Not applicable
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
23

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification:
We use LLMs our research and have described them properly.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy battps://neunee 22tps//eec.coooeeeeeee///2////// for what should or should be described.
24


Reviews:
[{"review_id": "CjzIyBaEUB", "reviewer": "Reviewer_M3oh", "summary": "The authors propose a new technique, B3, for assembling batches in a contrastive learning setup.  They find that by selecting more informative negative examples (through their B3 method), the performance of the models improve in the multimodal evaluation setting they consider.", "questions": "Why did you decide to anthropomorphize the model in Fig. 1?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "# Strength\n\n- The method gives new SOTA performance on multimodal evaluations (NB: I am not an expert in literature and do not know much related work in this domain.  I hope that other reivewers can assess this point better.)\n- The method is simple, which is a strength\n- The method can be applied in various domains, as it enhances the standard contrastive learning paradigm\n- The initial idea appears sensible; by making the discrimination between positive and negative samples harder, the loss optimization should yield richer gradients.\n- By sharding the training set into clusters, this could also give rise to distributed computation, thus improving the parallelism.\n- The idea is also supported by other works, for example \"The Optimal Noise in Noise-Contrastive Learning Is Not What You Think\" (Chehab et al 2022).\n\n# Weaknesses\n\nAs the evaluation is very costly, it is understandable that the authors only perform limited evaluations.  Nevertheless, I would have liked to see more ablations.  Especially in the construction of the matrix $S$ there are two hyperparameters involved, which are finetuned for the task itself.  I would have liked to see some study over a parameter range for both $p$ and $m$.  And do you see a reason for why VQA tasks benefit from a larger $p$?  I would also really like to see what happens when $p=0$.  Does the performance degrade because now you will sample images that are *too* similar?  \n\nThe method relies on a pretrained model that accurately assigns similarity to the training dataset. Through this, the model will probably pick up features that the teacher network learned more quickly or strongly.  Is this something you can comment on?\n\nConversely to the previous paragraph:  can the method be reformulated to be completely self-supervised?  You could start with a randomly initialized teacher network and then throughout training update the network weights and recompute the communities which will be used to form batches.\n\nOne omission is maybe a simpler case.  How does B3 perform on for example standard CL image classification?  This should be comparatively easy to test and would give additional credence to the method.", "was_revised": true, "final_justification": "I appreciate the additional experiments in the rebuttal as well as the comment about model capability in the teacher network. I think that this technique can be used to improve self supervised learning due to its broad applicability.", "rebuttal": "Thank you for your thoughtful comments and time reviewing our work. We are truly pleased with the positive reception our work has garnered from you and the other reviewers.\n- “method gives new SOTA performance…method is simple…should yield richer gradients…” - RM3oh\n- “B3 method is rigorously formulated…supported with theoretical bounds on InfoNCE loss approximation…consistent and substantial gains, especially under small batch sizes…graph-based community detection is a novel framing ” - RipHG\n- “has clear intuition and solid theoretical justification…Experiments seem solid and extensive…” - Rm62i\n- “achieves sota performance despite training at smaller batch size…ablation studies validate the effectiveness of B3”-R4DdK\n\nWe equally appreciate your constructive comments, and have modified the draft to address the raised concerns.  \n\n---\n---\n\nRegarding,\n\n>... reason for why VQA tasks benefit from a larger p?\n\n$p$ serves as a mechanism for excluding false negatives, a strategy previously employed in SFR‑Embed and NV‑Embed. For retrieval, we adopted the same value of $p=30$ as used in SFR‑Embed. NV‑Embed also showed that lower values of $p$ between $10$ and $30$ perform well. However, in the case of VQA training datasets in MMEB, the positive (target) responses were notably short. The table below reports the average target lengths for the VQA tasks.\n\n| Dataset | OK-VQA | A-OKVQA | DocVQA | InfoVQA | ChartQA | Visual7W | Avg.  |\n|:--|:--:|:--:|:--:|:----:|:---:|:--:|:--:|\n| Length | 1.24 | 1.28 | 2.16   | 1.61 | 1.12 | 1.98 | 1.57 |\n|||||||||\n\nIn contrast, for image2text retrieval tasks, the average positive (target) length was as follows \n| Dataset | VisualNews_i2t | MSCOCO_i2t | WebQA | Avg. |\n|:--|:--:|:--:|:--:|:--:|\n| Length | 18.89 | 10.44 | 23.45 | 17.6 |\n||||||\n\nDue to this reason, **we found a significantly higher number of targets which had similar/same meaning as that of the labelled target**. For instance, here is the first example in OK-VQA dataset. As seen, due to the short nature of the answers a higher number of targets of other examples are very similar/same compared to retrieval.\n- Image: Image containing a women playing tennis\n- Q: What is the hairstyle of the blond called?\n- Top ranked targets: [ponytail, ponytail, ponytail, pony tail, pony tail,...]\n\nHence, we used a higher value of $p$ for VQA tasks. For computational efficiency, we tuned $p$ for smaller batch sizes and retained these values for larger batch sizes. Here are results for a model trained and tested on VQA datasets only at batch size ($|B|=32$) for different $p$ values.\n\n| p | \\|B\\|  | ID | OOD | Avg.  |\n|:--|:--:|:--:|:--:|:--:|\n| 30|32| 57.80 | 56.73 | 57.37 |\n| 70|32| 58.38 | 57.95 | 58.21 |\n||||||\n\nAt lower batch sizes, results improve for higher $p$ values. However, as discussed next, at higher batch sizes, $p$ has a smaller effect on the performance. \n\n---\n\n>... see some study over a parameter range for both p and m\n\nWe hereby present some analysis on parameter $m$. Model trained and tested on retrieval datasets only at batch size 32.\n\n| p | m | \\|B\\| | ID | OOD | Avg.|\n|:--|:--:|:--:|:--:|:--:|:--:|\n| 30 | 100 | 32 | 72.83 | 55.50 | 67.05 |\n| 30 | 500 | 32 | 72.25 | 54.65 | 66.38 |\n| 30 | 800 | 32 | 72.60 | 55.15 | 66.78 |\n|||||||\n\nAt smaller batch sizes, the performance changes with $m$. $m=100$ as used in SFR-Embed worked well for us. We demonstrate the ablations over $p$ in the next question. \n\n---\n>... what happens when p=0.\n\nThe table below reports results for $p=0$ across different batch sizes. At smaller batch sizes, using an elevated value of $p$ proves beneficial, effectively reducing multiple false negatives. However, at larger batch sizes, this effect diminishes; as shown, **performance remains comparable between $p=0$ and $p=30$ when using higher batch sizes.** \n\n| p (Ret,VQA) | p+m (Ret, VQA) |  |   | \\|B\\|  | ID | OOD  |  Avg.  |\n|:--|:--:|:--:|:--:|:--:|:--:|:--:|:--:|\n| 0,0  | 30,70  | | |32   | 62.18  | 58.23  | 60.42  |\n| 30,70 | 130,170| | | 32  | 63.44  | 59.67  | 61.76  |\n| 0,0  | 30,70 | | |128 | 68.16 | 61.99  | 65.42  |\n| 30,70 | 130,170| | | 128 | 68.35  | 61.80  | 65.44  |\n| 0,0 | 30,70  | | | 512 | 71.09 | 63.31 | 67.63 |\n| 30,70 | 130,170 | | | 512 | 70.90 | 63.43 | 67.58 |\n|||||||||\n\nPresented below are additional variations of $p$ and $m$ at a higher batch size (512). As shown, **B3 maintains consistent performance across these variations when using a larger batch size.**\n\n| p (Ret, VQA)  | p+m (Ret, VQA) | | | \\|B\\|  | ID | OOD | Avg. |\n|:--|:--:|:--:|:--:|:--:|:--:|:--:|:--:|\n| 0,0|30,70| | | 512 | 71.09 | 63.31 | 67.63 |\n| 0,0|100,100  | | | 512 | 71.06 | 63.24 | 67.59 |\n| 30,70 | 130,170 | | | 512 | 70.90 | 63.43 | 67.58 |\n|||||||||\n\nAs shown $p=0$ works very well at larger batch sizes.\n\n**Main Takeaway (from $p$ and $m$ analysis)**\n- *The previous tables indicate that the hyperparameters $p$ and $m$ have a greater impact on performance at smaller batch sizes. While B3 requires tuning of these parameters to achieve optimal results at smaller batch sizes, it shows increased robustness to their variation at larger batch sizes.*\n\n---\n---\n\n>... can the method be reformulated to be completely self-supervised? You could start with a randomly initialized teacher network ...\n\n- Contrastive learning relies on two key signals: positives and negatives.\n- Our method specifically **focuses on selecting negatives in a self‑supervised manner, given a set of positives.** \n- Constructing a pipeline of progressively harder negatives, as suggested by the reviewer, is feasible, but we leave this for future exploration.\n- **Self‑supervising positives** is an orthogonal direction, and B3 can be seamlessly **integrated with existing approaches such as BYOL, DINOv2, or V‑JEPA** that address this aspect.\n\n---\n---\n\n>... How does B3 perform on for example standard CL image classification? … additional credence to the method.\n\nWe are not sure what the reviewer means by \"standard CL image classification\", but here are zero-short ImageNet1k classification results. MMEB benchmark in Table 1 additionally contains 10 different classification tasks. \n\n| Model | | | |ImageNet1k |\n|:--|--|--|--|:--:|\n| B3 Qwen7B | | | | **84.1** |\n| B3 Qwen2B  | | | | **83.2** |\n| VLM2Vec Qwen7B | | | | 80.5 |\n| VLM2Vec Qwen2B | | | | 77.5 |\n| CLIP  | | | | 55.8 |\n||||||\n\nWe are happy to provide any further results the reviewer may wish to see.\n\n---\n---\n>The method relies on a pretrained model that accurately assigns similarity to the training dataset. …. Is this something you can comment on?\n\nIn B3, the teacher’s sole role is to spot potential strong negatives and group them in the same batch. We simply take ranks $p:(p+m)$ from the teacher to build batches with strong in-batch negatives— and even with as few as 8 such negatives ($K=8$ in Table 4), performance remains solid. Critically, the **teacher’s task is limited: it only needs to pick out some strong negatives, not all of them**.\n\nHence, this **doesn’t require a very accurate teacher**, as the reviewer suspected. To prove the point, we run B3 with a much weaker teacher (CLIP). Despite CLIP performing far below VLM2Vec on MMEB, using it as a teacher still delivers strong results.\n\n\n| Model | | Teacher | | \\|B\\|  | | K | | | **Ret.** | **Cla.** | **VQA** | **Gro.** | **ID** | **OOD** | **Avg** |\n|:--|--|:---:|---|:--:|:---:|:--:|:---:|:--:|:---:|:---:|:----:|:---:|:---:|:---:|:--:|\n| Random Batches (2B) | | - | | 1024 | | - | | |66.33 | 66.47|59.08|75.30|69.59|60.05|65.35|\n| B3 (2B) | | VLM2Vec2B | | 512 | | 8| | |69.81|66.94|59.93|80.00|71.01|62.89|67.40|\n| B3 (2B) | | VLM2Vec2B | | 512 | | 32  | | | 70.38  | 66.10 | 60.71 | 80.03 | 70.90 | 63.43 | 67.58  |\n| B3 (2B) | | VLM2Vec7B | | 512 | | 32 | | | 70.36 | 66.60  | 61.08 | 77.83  | 71.46 | 62.70 | 67.57 |\n| B3 (2B) | | **CLIP400M** | | 512 | | 32 | | | 69.79|66.34| 60.48 | 79.70 | 71.27 | 62.45 | **67.30** |\n||||||||||\n\n\nAll that said, a strong teacher model can still be trained using only positives and a sufficiently large batch size, without relying on any negatives. VLM2Vec teacher models were trained solely with positives and random batches, without incorporating hard negatives. \n\n---\n---\n\n>The method can be applied in various domains, as it enhances the standard contrastive learning paradigm\n\n\nWe concur with the reviewer that B3 is a highly generalizable method for contrastive training. To further substantiate our claim, we report the performance of B3, trained on the same NLI data as SimCSE (both with and without annotated hard negatives (HN)), evaluated on the STS benchmark. The corresponding SimCSE models (with/without HN) serve as the teacher models. All models (teacher and student) in the following table are 300M Roberta-Large.\n\n\n| Model |  | Teacher | | Data | \\|B\\| | | |STS12 | STS13 | STS14 | STS15 | STS16 | STSB | SICKR | Avg. |\n|:--|:--:|:--|--|:--:|:--:|:--:|:--:|:--:|:---:|:--:|:--:|:--:|:--:|:-:|:--:|\n| SimCSE (w/o HN) | | - | | 275k | 512  | | |75.63 | 84.40 | 78.06 | 84.82 | 81.87 | 84.21 | 77.00 | 80.86  |\n| **B3 (w/o HN)** | | SimCSE (w/o HN) | | 275k | 512 | | | 77.49 | 85.73 | 80.37 | 85.36 | 82.75 | 84.24  | 75.40 | **81.62** |\n||||||||||||||||\n| SimCSE | | - | | 275k | 512  | | | 77.30 | 86.68 | 82.20 | 86.58 | 84.02 | 86.36 | 81.84  | 83.57  |\n| **B3** | | SimCSE | | 275k | 512  | | | 78.70 | 87.58 | 83.47 | 87.22 | 84.50 | 86.85 | 81.33  | **84.24** |\n||||||||||||||||\n\n\n**B3 demonstrates notable improvements over SimCSE on the STS benchmark**, validating our batch mining approach for text-only tasks. Note that even starting with HN in the teacher, we can still see improvements. In future work, we plan to extend this to larger benchmarks like MMEB.\n\n\n---\n---\n\nFinally, we sincerely appreciate the reviewer’s thoughtful and insightful questions, which have helped us clarify and strengthen our work. We will incorporate the additional results, analyses, and detailed responses provided here into the final draft."}, {"review_id": "J7KiGT6nRH", "reviewer": "Reviewer_ipHG", "summary": "This paper introduces B3, a novel batch mining strategy for contrastive learning (CL), particularly in multimodal settings. Traditional CL relies heavily on in-batch negatives, where performance scales with batch size. B3 circumvents this by constructing semantically rich batches via a graph-based approach. It includes several steps: ranking training samples via a pretrained teacher model, constructing a sparse graph, identifying strong negatives through METIS clustering and forming sampled batches. The method, particularly its enhanced variant B3++, achieves state-of-the-art performance on the MMEB multimodal benchmark, often outperforming existing methods while requiring significantly smaller batch sizes (e.g., 64 vs. 1024). The paper supports its claims with theoretical justification, ablation studies, and evaluation across multiple model scales.", "questions": "1.\tCan B3 adapt to dynamic or evolving datasets, such as in online or streaming learning scenarios? B3 relies on full-dataset ranking and offline clustering. This assumes static datasets, which limits applicability in real-time, federated, or continually updated data environments. Please clarify whether you envision an extension of B3 to handle dynamic or growing datasets. Could B3 support periodic re-mining or incremental clustering, or might online approximation variants be viable?\n2.\tHow sensitive is B3’s performance to the teacher model’s quality and alignment with the downstream task distribution? The success of B3 heavily depends on the accuracy of teacher-based similarity rankings. The paper briefly evaluates a stronger teacher but reports negligible gains, which seems counterintuitive. Please elaborate: Were the teacher models pretrained on the same modalities and tasks as the downstream evaluation tasks? Would using a mismatched or weak teacher degrade B3’s performance?\n3.\tWhy was METIS selected as the clustering method, and how does it compare to alternative strategies like K-Means or spectral clustering in this context? The METIS algorithm is efficient and suits graph partitioning, but alternative clustering methods might offer different trade-offs (e.g., capturing latent structures, scalability, etc.). Please justify the choice of METIS beyond runtime. Could performance be further improved using embedding-based clustering techniques or data-driven adaptive clustering?\n4.\tCould you elaborate on the trade-off between cluster size (K) and performance, especially across diverse task types? You mention that K needs empirical tuning, and that high values degrade performance. However, it’s unclear whether this holds uniformly across retrieval, classification, and VQA tasks. Please clarify if K was tuned per task or globally. Does B3 generalize well with a fixed K across task types, or is task-specific tuning essential?", "limitations": "see above", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n1.\tThe new batch construction strategy is simple and effective. The B3 method is rigorously formulated, leveraging graph-based community detection (METIS) over teacher-model rankings to construct contrastive batches, and supported with theoretical bounds on InfoNCE loss approximation.\n2.\tExtensive experiments on the MMEB benchmark across 36 tasks and multiple model scales (2B and 7B) show consistent and substantial gains, especially under small batch sizes.\n3.\tWhile hard negative mining is not new, the use of graph-based community detection over ranked examples to construct entire informative batches is a novel framing.\nWeakness\n1.\tThe approach assumes that the teacher model produces semantically meaningful ranks. If the teacher is weak or biased, the batch quality may degrade significantly, affecting downstream performance.\n2.\tB3 assumes access to the full training dataset for batch construction. This makes it less suitable for continual learning or privacy-sensitive domains.\n3.\tB3 is more like a training technique that enhances contrastive learning by constructing more reliable negative samples. Can this technique be extended to more general tasks? In the paper, the method is primarily validated on the MMEB dataset. Despite MMEB's diversity, additional benchmarks, such as MTEB (text-only) or domain-specific tasks (e.g., medical imaging), could have further strengthened the generalizability claim.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful comments and time reviewing our work. We are truly pleased with the positive reception our work has garnered from you and the other reviewers.\n* “method gives new SOTA performance…method is simple…should yield richer gradients…” - RM3oh\n* “B3 method is rigorously formulated…supported with theoretical bounds on InfoNCE loss approximation…consistent and substantial gains, especially under small batch sizes…graph-based community detection is a novel framing ” - RipHG \n* “has clear intuition and solid theoretical justification…Experiments seem solid and extensive…” - Rm62i\n* “achieves sota performance despite training at smaller batch size…ablation studies validate the effectiveness of B3” - R4DdK\n\nWe equally appreciate your constructive comments, and have modified the draft to address the raised concerns.  \n\n---\n---\n\n>B3 is more like a training technique that enhances contrastive learning by constructing more reliable negative samples. Can this technique be extended to more general tasks?....\n\nWe concur with the reviewer that B3 is a highly generalizable method for contrastive training. To further substantiate our claim, we report the performance of B3, trained on the same NLI data as SimCSE (both with and without annotated hard negatives (HN)), evaluated on the STS benchmark. The corresponding SimCSE models (with/without HN) serve as the teacher models. All models (teacher and student) in the following table are 300M Roberta-Large.\n\n\n| Model   |  |  Teacher | | Data  | \\|B\\| |  | |STS12 | STS13 | STS14 | STS15 | STS16 | STSB | SICKR | Avg.   |\n|:--|:--:|:---|--|:---:|:--:|:--:|:--:|:---:|:---:|:--:|:--:|:---:|:----:|:-:|:---:|\n| SimCSE (w/o HN) | | - | | 275k | 512  | | |75.63 | 84.40 | 78.06 | 84.82 | 81.87 | 84.21 | 77.00 | 80.86  |\n| **B3 (w/o HN)** | | SimCSE (w/o HN) | | 275k | 512 | | | 77.49 | 85.73 | 80.37 | 85.36 | 82.75 | 84.24  | 75.40 | **81.62** |\n||||||||||||||||\n| SimCSE | | - | | 275k | 512  | | | 77.30 | 86.68 | 82.20 | 86.58 | 84.02 | 86.36 | 81.84  | 83.57  |\n| **B3** | | SimCSE  | | 275k | 512  | | | 78.70 | 87.58 | 83.47 | 87.22 | 84.50 | 86.85 | 81.33  | **84.24** |\n||||||||||||||||\n\n\n**B3 demonstrates notable improvements over SimCSE on the STS benchmark**, validating our batch mining approach for text-only tasks. Note that even starting with HN in the teacher, we can still see improvements. In future work, we plan to extend this to larger benchmarks like MMEB.\n\n---\n---\n\n>B3 assumes access to the full training dataset for batch construction. This makes it less suitable for continual learning or privacy-sensitive domains.\n\n- Contrastive learning typically necessitates substantial data volumes and large batch sizes, and is therefore **predominantly conducted in an offline setting**.\n- **Every single baseline** reported in Table 1 of the draft, as well as those on the MMEB leaderboard, operate in an offline configuration.\n- Likewise, **most leading models on the MTEB leaderboard**, including SFR‑Embed and NV‑Embed, are trained in an offline manner.\n\nWe regard this not as a limitation but as a natural consequence of the extensive data and batch requirements inherent to contrastive learning. With regards to privacy, B3 is a training algorithm and requires access to the data like every other training method.\n\nRui Meng, et al. 2024. Sfr-embedding-mistral: Enhance text retrieval with transfer learning.\n\nGabriel de Souza et al. 2024, Nv-retriever: Improving text embedding models with effective hard-negative\nmining.\n\n---\n---\n\n>Can B3 adapt to dynamic or evolving datasets, such as in online or streaming learning scenarios? B3 relies on full-dataset ranking and offline clustering…might online approximation variants be viable? \n\nOnline learning comprises two essential components: **(1) generating effective gradients from incoming examples and (2) retaining knowledge acquired from previously trained examples.** \n- For the latter, several approaches have been proposed; for instance, Liu et al. (2025) introduced a technique in which gradients from incoming batches are projected to prevent interference with existing knowledge. \n- For the former, B3 can be adapted to operate on a **sliding window of incoming examples**. Notably, at any given step, we already have a trained model available, which can serve as a teacher—**eliminating the need for an additional model—making B3 well-suited for this setting**.\n\nLiu, X. et al. (2025). Continual multimodal contrastive learning. arXiv preprint arXiv:2503.14963.\n\n---\n---\n\n>Why was METIS selected as the clustering method, and how does it compare to alternative strategies like K-Means or spectral clustering in this context? …. Please justify the choice of METIS beyond runtime...?\n\nOur choice of METIS algorithm is due to three main reasons.\n- Note that we need the **cluster size $K$ to be fixed** in B3 (as the batch size $|B|$ is fixed). Most clustering algorithms, such as K-Means or Agglomerative, have balanced variants, but these are typically much more computationally expensive, often running in $O(n^3)$. In contrast, METIS operates in $O(n \\log⁡ n)$ while producing equal-sized clusters.\n- A key idea within B3 cluster construction is to **minimize the presence of false negatives (upto rank $p$)** in the training clusters. Note that each example in the training dataset has a different set of false negatives. Clustering algorithms like K-Means/Agglomerative would require substantial modifications to achieve this. \n- Examples with **ranks beyond $p+m$ in B3 are less relevant** and should not play a role in clustering. Here again, it is non-trivial to include these constraints in regular clustering algorithms.\n\nTo accommodate rank-based constraints, we opted for graph-based clustering algorithms. Furthermore, to satisfy the fixed batch size requirement, we selected the METIS algorithm.\n\n---\n---\n\n >Could you elaborate on the trade-off between cluster size (K) and performance, especially across diverse task types? …Does B3 generalize well with a fixed K across task types, or is task-specific tuning essential?\n\n- Saunshi et al. (2019) theoretically demonstrated that using an excessive number of negatives leads to intra-class collisions, which **degrades the effectiveness of contrastive learning in general—not limited to any specific domain such as retrieval or VQA**. They further validated this empirically across diverse image and text datasets. Similarly, SFR‑Embed reported that an excessive number of hard negatives can negatively impact performance on MTEB.\n- Accordingly, for B3—which yields $K$ strong negatives per example—we tuned the value of $K$ instead of simply selecting the maximum possible value (i.e., $K$=batch size). **A single $K$ value was used across all tasks, including retrieval, VQA, classification, and grounding**. \n- It is important to note that the batch size is a multiple of $K$, which prevents adjusting it individually for different datasets under our fixed‑batch‑size setup. As shown in the Table 1 and 4, a single $K$ value generalized for all datasets. \n\nSaunshi, et al. (2019, May). A theoretical analysis of contrastive unsupervised representation learning. In ICML. PMLR.\n\n---\n---\n\n>The approach assumes that the teacher model produces semantically meaningful ranks. If the teacher is weak or biased,... degrade significantly, affecting downstream performance. ...The success of B3 heavily depends on the accuracy of teacher-based similarity rankings. ...Were the teacher models pretrained on the same modalities and tasks as the downstream evaluation tasks? Would using a mismatched or weak teacher degrade B3’s performance?.\n\nIn B3, the teacher’s sole role is to spot potential strong negatives and group them in the same batch. We simply take ranks $p:(p+m)$ from the teacher to build batches with strong in-batch negatives— and even with as few as 8 such negatives ($K=8$ in the Table), performance remains solid. Critically, the **teacher’s task is limited: it only needs to pick out some strong negatives, not all of them**.\n\nHence, this **doesn’t require a very accurate teacher**, as the reviewer suspected. To prove the point, we run B3 with a much weaker teacher (CLIP). Despite CLIP performing far below VLM2Vec on MMEB, using it as a teacher still delivers strong results.\n\n\n| Model           |      | Teacher |  | \\|B\\|  | | K  |  |  | **Ret.** | **Cla.** | **VQA** | **Gro.** | **ID** | **OOD** | **Avg** |\n|---------------|---|---|---|----|---|--|---|--|-----------|-----------|-----------|-----------|------------|-------------|---------|\n| Random Batches (2B) | | - | | 1024 | | - | | |66.33 | 66.47|59.08|75.30|69.59|60.05|65.35|\n| B3 (2B) | | VLM2Vec2B | | 512 | | 8| | |69.81|66.94|59.93|80.00|71.01|62.89|67.40|\n| B3 (2B) | | VLM2Vec2B | | 512 | | 32  | | | 70.38     | 66.10     | 60.71     | 80.03     | 70.90      | 63.43       | 67.58   |\n| B3 (2B) | | VLM2Vec7B | | 512 | | 32 | | | 70.36     | 66.60     | 61.08     | 77.83     | 71.46      | 62.70       | 67.57   |\n| B3 (2B) | | **CLIP400M** | | 512 | | 32 | | | 69.79|66.34| 60.48     | 79.70     | 71.27      | 62.45       | **67.30**   |\n||||||||||\n\n\nAll that said, a strong teacher model can still be trained using only positives and a sufficiently large batch size, without relying on any negatives. VLM2Vec teacher models were trained solely with positives and random batches, without incorporating hard negatives. VLM2Vec models used in the paper were trained with the MMEB train set containing the same modalities as the MMEB eval set.\n\n---\n---\n\nFinally, we sincerely appreciate the reviewer’s thoughtful and insightful questions, which have helped us clarify and strengthen our work. We will incorporate the additional results, analyses, and detailed responses provided here into the final draft to ensure a more comprehensive and rigorous presentation."}, {"review_id": "4NiDfKWNKv", "reviewer": "Reviewer_m62i", "summary": "The paper introduces Breaking the Batch Barrier (B3), which uses a pretrained embedding model to construct a sparse similarity graph over the dataset and then applies graph clustering to form batches of strong negatives, thereby obtaining high-quality batches in a computationally efficient manner. Experiments show that B3 and its enhanced variant, B3++, outperform prior state-of-the-art methods across model scales and batch sizes while reducing overall computation.", "questions": "I wonder what the minimal requirements are for the teacher embedding model. How well does the method perform when the teacher is weaker or smaller than the model being trained—a scenario I think is very practically relevant.", "limitations": "The authors have discussed the limitations in Appendix B.", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n1. I find the  method well motivated, has clear intuition and solid theoretical justification.\n\n2. Experiments seem solid and extensive to me. They cover multiple scales and batch sizes and demonstrate consistent improvements.\n\n3. Ablation studies confirm the contribution of each component, reinforcing the method’s design choices.\n\nWeaknesses\n\nI don’t see any major weaknesses; however, I’m an expert this specific topic or familiar with all existing methods, so I’ll leave it to other reviewers to judge whether the comparisons are comprehensive and the experimental setup is adequate.", "was_revised": true, "final_justification": "I think this paper makes a novel contribution to the field and I am also satisfied with the rebuttal.", "rebuttal": "Thank you for your thoughtful comments and time reviewing our work. We are truly pleased with the positive reception our work has garnered from you and the other reviewers.\n* “method gives new SOTA performance…method is simple…should yield richer gradients…” - RM3oh\n* “B3 method is rigorously formulated…supported with theoretical bounds on InfoNCE loss approximation…consistent and substantial gains, especially under small batch sizes…graph-based community detection is a novel framing ” - RipHG \n* “has clear intuition and solid theoretical justification…Experiments seem solid and extensive…” - Rm62i\n* “achieves sota performance despite training at smaller batch size…ablation studies validate the effectiveness of B3” - R4DdK\n\nTo further strengthen our B3 pitch, we have included two additional results as requested by other reviewers.\n\n---\n---\n\n> Generalizability to other domains?\n\nWe believe that B3 is a highly generalizable method for contrastive training. To further substantiate our claim, we report the performance of B3, trained on the same NLI data as SimCSE (both with and without annotated hard negatives (HN)), evaluated on the STS benchmark. The corresponding SimCSE models (with/without HN) serve as the teacher models. All models (teacher and student) in the following table are 300M Roberta-Large.\n\n\n| Model   |  |  Teacher | | Data  | \\|B\\| |  | |STS12 | STS13 | STS14 | STS15 | STS16 | STSB | SICKR | Avg.   |\n|:--|:--:|:---|--|:---:|:--:|:--:|:--:|:---:|:---:|:--:|:--:|:---:|:----:|:-:|:---:|\n| SimCSE (w/o HN) | | - | | 275k | 512  | | |75.63 | 84.40 | 78.06 | 84.82 | 81.87 | 84.21 | 77.00 | 80.86  |\n| **B3 (w/o HN)** | | SimCSE (w/o HN) | | 275k | 512 | | | 77.49 | 85.73 | 80.37 | 85.36 | 82.75 | 84.24  | 75.40 | **81.62** |\n||||||||||||||||\n| SimCSE | | - | | 275k | 512  | | | 77.30 | 86.68 | 82.20 | 86.58 | 84.02 | 86.36 | 81.84  | 83.57  |\n| **B3** | | SimCSE  | | 275k | 512  | | | 78.70 | 87.58 | 83.47 | 87.22 | 84.50 | 86.85 | 81.33  | **84.24** |\n||||||||||||||||\n\n\n**B3 demonstrates notable improvements over SimCSE on the STS benchmark**, validating our batch mining approach for text-only tasks. Note that even starting with HN in the teacher, we can still see improvements. In future work, we plan to extend this to larger benchmarks like MMEB.\n\n---\n---\n\n> Need for a strong teacher?\n\nIn B3, the teacher’s sole role is to spot potential strong negatives and group them in the same batch. We simply take ranks $p:(p+m)$ from the teacher to build batches with strong in-batch negatives— and even with as few as 8 such negatives ($K=8$ in the Table), performance remains solid. Critically, the **teacher’s task is limited: it only needs to pick out some strong negatives, not all of them**.\n\nHence, this **doesn’t require a very accurate teacher**, as the reviewer suspected. To prove the point, we run B3 with a much weaker teacher (CLIP). Despite CLIP performing far below VLM2Vec on MMEB, using it as a teacher still delivers strong results.\n\n\n| Model           |      | Teacher |  | \\|B\\|  | | K  |  |  | **Ret.** | **Cla.** | **VQA** | **Gro.** | **ID** | **OOD** | **Avg** |\n|:--|--|:---:|---|:--:|:---:|:--:|:---:|:--:|:---:|:---:|:----:|:---:|:---:|:---:|:--:|\n| Random Batches (2B) | | - | | 1024 | | - | | |66.33 | 66.47|59.08|75.30|69.59|60.05|65.35|\n| B3 (2B) | | VLM2Vec2B | | 512 | | 8| | |69.81|66.94|59.93|80.00|71.01|62.89|67.40|\n| B3 (2B) | | VLM2Vec2B | | 512 | | 32  | | | 70.38  | 66.10 | 60.71 | 80.03 | 70.90 | 63.43 | 67.58  |\n| B3 (2B) | | VLM2Vec7B | | 512 | | 32 | | | 70.36 | 66.60  | 61.08 | 77.83  | 71.46 | 62.70 | 67.57 |\n| B3 (2B) | | **CLIP400M** | | 512 | | 32 | | | 69.79|66.34| 60.48 | 79.70 | 71.27 | 62.45 | **67.30** |\n||||||||||\n\n\nAll that said, a strong teacher model can still be trained using only positives and a sufficiently large batch size, without relying on any negatives. VLM2Vec teacher models were trained solely with positives and random batches, without incorporating hard negatives. \n\n---\n---\nWe again thank the reviewer for their time and effort. We will incorporate the additional results, analyses, and detailed responses provided here into the final draft to ensure a more comprehensive and rigorous presentation."}, {"review_id": "2yv0pLoiZn", "reviewer": "Reviewer_4DdK", "summary": "This paper presents \"Breaking the Batch Barrier\" (B3), a batch construction strategy for contrastive learning. The B3 method employs a pretrained teacher model to create a similarity ranking for all examples in the dataset. This ranking is used to construct a sparse similarity graph, which is then used for clustering examples that are strong mutual negatives using a community detection algorithm (METIS). Training batches are then formed by sampling from these communities, achieving challenging negatives without the computational overhead of adding individually mined hard negatives. The authors also propose B3++, an optional extension that incorporates a unified hard-negative mining strategy for further performance gains. The B3 is evaluated on the MMEB multimodal benchmark, and it demonstrates significant improvements over existing methods.", "questions": "Q1: The negative samples exclusion ranking threshold p needs to be tuned differently for various task types (e.g., p=30 for retrieval, p=70 for VQA). This suggests the method is sensitive to the data's domain and structure. Could you provide more insight into the underlying reasons?", "limitations": "yes", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength\n\nS1: The proposed method leverages graph-based community clustering to construct batches that are rich in hard negative, which is efficient compared to existing hard-negative samples mining methods.\n\nS2: The author performed extensive experiments to show that the B3 method achieves sota performance despite training at smaller batch size.\n\nS3: The ablation studies and comparisons across various batch sizes and hyperparameters validate the effectiveness of B3.\n\nWeakness\n\nW1:  I was very confused about the positive samples at the beginning. The papers uses the standard terminology \"(query, positive) pairs\" to refer to a \"positive pair\", but does not explicitly explain that a \"positive\" can be of a different data type than its corresponding \"query\" (e.g., an image paired with a text caption). For readers whose main exposure to contrastive learning comes from self-supervised methods where positive pairs are typically of the same modality, this lack of clarity may lead to a fundamental confusion.\n\nW2: The design of excluding top p ranks from the negative samples is quite ad-hoc, and has to be tuned case by case.", "was_revised": true, "final_justification": "The author has provide more context on the design of of excluding top p ranks in the negative sampling, and performed additional experiments to show that  p=0 works very well at larger batch sizes.", "rebuttal": "Thank you for your thoughtful comments and time reviewing our work. We are truly pleased with the positive reception our work has garnered from you and the other reviewers.\n* “method gives new SOTA performance…method is simple…should yield richer gradients…” - RM3oh\n* “B3 method is rigorously formulated…supported with theoretical bounds on InfoNCE loss approximation…consistent and substantial gains, especially under small batch sizes…graph-based community detection is a novel framing ” - RipHG \n* “has clear intuition and solid theoretical justification…Experiments seem solid and extensive…” - Rm62i\n* “achieves sota performance despite training at smaller batch size…ablation studies validate the effectiveness of B3” - R4DdK\n\n---\n---\n\n>W1: ... confused about the positive samples at the beginning. The papers uses the standard terminology \"(query, positive) pairs\" to refer to a \"positive pair\", but does not explicitly explain that a \"positive\" can be of a different data type than its corresponding \"query\" (e.g., an image paired with a text caption). Readers whose main exposure to contrastive learning comes from self-supervised methods... this lack of clarity may lead to a fundamental confusion.\n\nThank you for highlighting this point. As the reviewer correctly observed, the query and positive can originate from different modalities, and each may also be a combination of modalities. For example, in VQA, the query comprises an image and a text question, while the positive is a text answer. This is an important characteristic of the MMEB training set, which we will clarify in the final draft. \n\n---\n---\n\n> The design of excluding top p ranks from the negative samples is quite ad-hoc, and has to be tuned case by case. Q1: The negative samples exclusion ranking threshold p needs to be tuned differently for various task types (e.g., p=30 for retrieval, p=70 for VQA). This suggests the method is sensitive to the data's domain and structure. Could you provide more insight into the underlying reasons?\n\nExcluding top $p$ ranks serves as a mechanism for excluding false negatives, **a strategy previously employed in SFR‑Embed and NV‑Embed**. For retrieval, we adopted the same value of $p=30$ as used in SFR‑Embed. NV‑Embed also showed that lower values of $p$ between $10$ and $30$ perform well. However, in the case of VQA training datasets in MMEB, the positive (target) responses were notably short. The table below reports the average target lengths for the VQA tasks.\n\n| Dataset         | OK-VQA | A-OKVQA | DocVQA | InfoVQA | ChartQA | Visual7W | Avg.  |\n|:------------|:-----:|:------:|:----:|:--------:|:---:|:--------:|:-------:|\n| Length|1.24| 1.28| 2.16   | 1.61| 1.12|1.98| 1.57 |\n|||||||||\n\nIn contrast, for image2text retrieval tasks, the average positive (target) length was as follows \n| Dataset| VisualNews_i2t | MSCOCO_i2t | WebQA | Avg.       |\n|:------------|:---------:|:-----:|:----:|:------:|\n| Length| 18.89 | 10.44| 23.45 | 17.6 |\n||||||\n\nDue to this reason, **we found a significantly higher number of targets which had similar/same meaning as that of the labelled positive**. For instance, here is the first example in OK-VQA dataset. As seen, due to the short nature of the answers a higher number of targets/positives of other examples are very similar/same compared to retrieval.\n\n- Image: Image containing a women playing tennis\n- Q: What is the hairstyle of the blond called?\n- Top ranked targets/positives: [“ponytail”, “ponytail”, “ponytail”, “pony tail”, “pony tail”,....]\n\nHence, we used a higher value of $p$ for VQA tasks. For computational efficiency, we tuned $p$ for smaller batch sizes and retained these values for larger batch sizes. Here are results for a model trained and tested on VQA datasets only at batch size ($|B|=32$) for different $p$ values.\n\n| p   | \\|B\\|  | ID    | OOD   | Avg.  |\n|:-----|:-----:|:----:|:-------:|:-------:|\n| 30  | 32  | 57.80 | 56.73 | 57.37 |\n| 70  | 32  | 58.38 | 57.95 | 58.21 |\n||||||\n\nAt lower batch sizes, results improve for higher $p$ values. However, as discussed next, at higher batch sizes, $p$ has a smaller effect on the performance. \n\nThe table below reports results for $p=0$ across different batch sizes. At smaller batch sizes, using an elevated value of $p$ proves beneficial, effectively reducing multiple false negatives. However, at larger batch sizes, this effect diminishes; as shown, **performance remains comparable between $p=0$ and $p=30$ when using higher batch sizes**. \n\n| p (Ret,VQA)   | p+m (Ret, VQA)   | \\|B\\|  |   ID   |   OOD  |  Avg.  |\n|:-----|:-----:|:-----:|:--------:|:--------:|:--------:|\n| 0,0   | 30,70  | 32   | 62.18  | 58.23  | 60.42  |\n| 30,70  | 130,170 | 32  | 63.44  | 59.67  | 61.76  |\n| 0,0   | 30,70  | 128  | 68.16  | 61.99  | 65.42  |\n| 30,70  | 130,170 | 128 | 68.35  | 61.80  | 65.44  |\n| 0,0   | 30,70  | 512 | 71.09 | 63.31 | 67.63 |\n| 30,70   | 130,170 | 512 | 70.90 | 63.43 | 67.58 |\n|||||||\n\nPresented below are additional variations of $p$ and $m$ at a higher batch size (512). As shown, **B3 maintains consistent performance across these variations when using a larger batch size.**\n\n| p (Ret, VQA)   | p+m (Ret, VQA)  | \\|B\\|  | ID    | OOD  | Avg.  |\n|:-----|:-----:|:-----:|:-------:|:-------:|:-------:|\n| 0,0   | 30,70  | 512 | 71.09 | 63.31 | 67.63 |\n| 0,0   | 100,100 | 512 | 71.06 | 63.24 | 67.59 |\n| 30,70  | 130,170 | 512 | 70.90 | 63.43 | 67.58 |\n|||||||\n\nAs shown p=0 works very well at larger batch sizes.\n\n**Main Takeaway (from $p$ and $m$ analysis)**\n- *The previous tables indicate that the hyperparameters $p$ and $m$ have a greater impact on performance at smaller batch sizes. While B3 requires tuning of these parameters to achieve optimal results at smaller batch sizes, it shows increased robustness to their variation at larger batch sizes.*\n\n---\n---\nTo further strengthen our B3 pitch, we have included two additional results as requested by other reviewers.\n\n> Generalizability to other domains?\n\nWe believe that B3 is a highly generalizable method for contrastive training. To further substantiate our claim, we report the performance of B3, trained on the same NLI data as SimCSE (both with and without annotated hard negatives (HN)), evaluated on the STS benchmark. The corresponding SimCSE models (with/without HN) serve as the teacher models. All models (teacher and student) in the following table are 300M Roberta-Large.\n\n\n| Model   |  |  Teacher | | Data  | \\|B\\| |  | |STS12 | STS13 | STS14 | STS15 | STS16 | STSB | SICKR | Avg.   |\n|:--|:--:|:---|--|:---:|:--:|:--:|:--:|:---:|:---:|:--:|:--:|:---:|:----:|:-:|:---:|\n| SimCSE (w/o HN) | | - | | 275k | 512  | | |75.63 | 84.40 | 78.06 | 84.82 | 81.87 | 84.21 | 77.00 | 80.86  |\n| **B3 (w/o HN)** | | SimCSE (w/o HN) | | 275k | 512 | | | 77.49 | 85.73 | 80.37 | 85.36 | 82.75 | 84.24  | 75.40 | **81.62** |\n||||||||||||||||\n| SimCSE | | - | | 275k | 512  | | | 77.30 | 86.68 | 82.20 | 86.58 | 84.02 | 86.36 | 81.84  | 83.57  |\n| **B3** | | SimCSE  | | 275k | 512  | | | 78.70 | 87.58 | 83.47 | 87.22 | 84.50 | 86.85 | 81.33  | **84.24** |\n||||||||||||||||\n\n\n**B3 demonstrates notable improvements over SimCSE on the STS benchmark**, validating our batch mining approach for text-only tasks. Note that even starting with HN in the teacher, we can still see improvements. In future work, we plan to extend this to larger benchmarks like MMEB.\n\n---\n---\n\n> Need for a strong teacher?\n\nIn B3, the teacher’s sole role is to spot potential strong negatives and group them in the same batch. We simply take ranks $p:(p+m)$ from the teacher to build batches with strong in-batch negatives— and even with as few as 8 such negatives ($K=8$ in the Table), performance remains solid. Critically, the **teacher’s task is limited: it only needs to pick out some strong negatives, not all of them**.\n\nHence, this **doesn’t require a very accurate teacher**, as the reviewer suspected. To prove the point, we run B3 with a much weaker teacher (CLIP). Despite CLIP performing far below VLM2Vec on MMEB, using it as a teacher still delivers strong results.\n\n\n| Model           |      | Teacher |  | \\|B\\|  | | K  |  |  | **Ret.** | **Cla.** | **VQA** | **Gro.** | **ID** | **OOD** | **Avg** |\n|:--|--|:---:|---|:--:|:---:|:--:|:---:|:--:|:---:|:---:|:----:|:---:|:---:|:---:|:--:|\n| Random Batches (2B) | | - | | 1024 | | - | | |66.33 | 66.47|59.08|75.30|69.59|60.05|65.35|\n| B3 (2B) | | VLM2Vec2B | | 512 | | 8| | |69.81|66.94|59.93|80.00|71.01|62.89|67.40|\n| B3 (2B) | | VLM2Vec2B | | 512 | | 32  | | | 70.38  | 66.10 | 60.71 | 80.03 | 70.90 | 63.43 | 67.58  |\n| B3 (2B) | | VLM2Vec7B | | 512 | | 32 | | | 70.36 | 66.60  | 61.08 | 77.83  | 71.46 | 62.70 | 67.57 |\n| B3 (2B) | | **CLIP400M** | | 512 | | 32 | | | 69.79|66.34| 60.48 | 79.70 | 71.27 | 62.45 | **67.30** |\n||||||||||\n\n\nAll that said, a strong teacher model can still be trained using only positives and a sufficiently large batch size, without relying on any negatives. VLM2Vec teacher models were trained solely with positives and random batches, without incorporating hard negatives. \n\n---\n---\n\nWe again thank the reviewer for their time and effort. We will incorporate the additional results, analyses, and detailed responses provided here into the final draft to ensure a more comprehensive and rigorous presentation."}]

Author rebuttal:

Meta-review: The paper introduces B3, a method for improving contrastive learning by constructing batches enriched with high-quality in-batch negatives using a graph-based mining approach. The method forms informative training batches, which when trained upon outperform existing methods across various batch sizes and model scales.

Reviewers consistently praised the method's conceptual simplicity, strong empirical results, and novel use of community detection for batch construction. They noted the SOTA performance on the MMEB benchmark, the effectiveness at small batch sizes, broad applicability across domains, and robust performance even with weaker teacher models as significant positives. Reviewers also appreciated that the paper included theoretical analysis and extensive ablations validating the method’s components.

One reviewer (ipHG) gave a borderline rating and raised concerns about generalizability beyond MMEB, sensitivity to teacher model quality, and the method’s reliance on full offline access to the dataset. However, these concerns seem to have been sufficiently addressed in the rebuttal. While ipHG did not engage in discussion, the AC was compelled by the author response and does not believe that substantive objections remained unaddressed.

For the camera-ready version, the authors may consider clarifying the terminology around cross-modal positive pairs and elaborating on the rationale behind tuning hyperparameter, p, for different task types, as discussed in response to Reviewer 4DdK.

The AC recommends acceptance.
Decision: Accept (spotlight)