BiXSE: Improving Dense Retrieval via Probabilistic Graded Relevance Distillation
Authors: Christos Tsirigotis
Vaibhav Adlakha
Joao Monteiro
Aaron Courville
Perouz Taslakian
Venue: COLM 2025 (2025)
Keywords: sentence embeddings
dense retrieval
retrieval
ranking
synthetic data
TL;DR: We propose an effective training method on synthetic graded relevance scores for improving dense encoders

Abstract: Neural sentence embedding models for dense retrieval typically rely on binary relevance labels, treating query-document pairs as either relevant or irrelevant. However, real-world relevance often exists on a continuum, and recent advances in large language models (LLMs) have made it feasible to scale the generation of fine-grained graded relevance labels. In this work, we propose \textbf{BiXSE}, a simple and effective pointwise training method that optimizes binary cross-entropy (BCE) over LLM-generated graded relevance scores. BiXSE interprets these scores as probabilistic targets, enabling granular supervision from a single labeled query-document pair per query. Unlike pairwise or listwise losses that require multiple annotated comparisons per query, BiXSE achieves strong performance with reduced annotation and compute costs by leveraging in-batch negatives. Extensive experiments across sentence embedding (MMTEB) and retrieval benchmarks (BEIR, TREC-DL) show that BiXSE consistently outperforms softmax-based contrastive learning (InfoNCE), and matches or exceeds strong pairwise ranking baselines when trained on LLM-supervised data. BiXSE offers a robust, scalable alternative for training dense retrieval models as graded relevance supervision becomes increasingly accessible.

Published as a conference paper at COLM 2025
Improving Dense Retrieval via Probabilistic Graded Relevance Distillation
BiXSE:
Christos Tsirigotis*
Vaibhav Adlakha
João Monteiro Apple MLR
Université de Montéal
McGill University
Mila Quebec Al Institute
Mila Quebec Af Institute
Aaron C. Courville
Perouz Taslakian
Université de Montéal
ServiceNow Research
Mila Quebec AI Institute
IVADO, Canada CIFAR AT Chair
Abstract
Neural sentence embedding models for dense retrieval typically rely on binary relevance labels, treating query -very-ddcummnt pairs either relevant or irrelevant. However, real-world relevance often exists on a continuum, and recent advances in large language models (LLMs) have made it fea- sible to scale the generation of fine-grained graded relevance labels. In this work, we propose BiXSE, a simple and effective pointwise training method that optimizes binary cross-entropy (BCE) over LLM-generated graded relevance scores. BiXSE interprets these scores as probabilistic tar- gets, enabling granular supervision from single labeled query-document pair per query. Unlike pairwise or listwise losses that require multiple annotated comparisons per query, BiXSE achieves strong performance with reduced annotation and compute costs by leveraging in-batch negatives. Extensive experiments across sentence embedding (MMTEB) and retrieval benchmarks (BEIR, TREC-DL) show that BiXSE consistently outperforms softmax-based contrastive learning (InfoNCE), and matches or exceeds strong pairwise ranking baselines when trained on LLM -up-suupeeiseed data. BiXSE offers a robust, scalable alternative for training dense retrieval mod- els as graded relevance supervision becomes increasingly accessible.
1 Introduction
Training sentence embedding models for dense retrieval, in DPR (Karpukhin et al., 2020) or Sentence-BERT (Reimers & Gurevych, 2019), typically relies on contrastive learning (Oord et al., 2018, InfoNCE) with binary relevance labels where query-document pairs are marked as either relevant (positive) or irrelevant (negative). Models are trained to produce similar embeddings for query-positive pairs and dissimilar embeddings for query-negative pairs. While effective, this binary approach can be limiting, real-world relevance often exists on a continuum rather than as a strict yes/no. For instance, a document might be partially relevant to query, yet traditional contrastive training would treat it equally irrelevant as a completely unrelated document. This issue is particularly pronounced when using mined "hard negatives", documents typically selected among the top-ranking retrieval candidates according to an initial retriever model and labeled as irrelevant (Karpukhin et al., 2020; Wang et al., 2022; Lee et al., 2024b; de Souza P. Moreira et al., 2024). While hard negatives are crucial for effective learning (Lee et al., 2024b), they are uniformly labeled as having zero relevance, even if some may actually be partially relevant, introducing false negatives into the training data (Ni et al., 2021; Qu et al., 2021). Thus, binary simplification provides incomplete supervision with moderately relevant passages receiving no credit
*Correspondance to: Christos Tsirigotis <tsirigoc@mi quebec >, Perouz Taslakian <perouz,
>. Code release at: ht wtt.gttb..cmm/tsrrif/ttitt...

---

Published as a conference paper at COLM 2025
Base Retriever
Ranking Instruction
On scale from 1 to 4, how wall does this document enswer query?
Quarya
<<Mined Documents
LLM Ranker
Gemini
<<Ronking instruction
<<Graded Relevance
What is the capital of France?
for call LLM
e-Retrieval Instructions
0.66
e=Dacument 1>0
<cQueryaa
Binary Extroy
Document 2>>
e<Documents
0.91
Document Na
4 d (az + (3
Figure 1:
BIXSE amortizes the expensive inference process of an effective graded LLM ranker into the training of a dense retrieval model via a simple binary cross-entropy loss.
and produces noisy datasets containing false negatives, potentially over-penalizing the model during training.
These limitations can be addressed by using graded relevance scores that quantify the degree of relevance between text pairs. This can be achieved for instance via ordinal scales which capture varying relevance levels (e.g. integers 0-3 or 1-5). Historically, graded relevance evaluation has been central in the information retrieval community for assessing system quality (Järvelin & Kekäläinen, 2000; 2002; Sakai, 2021). For example, in the Deep Learning track of the Text REtrieval Conference (TREC-DL) 2023 (Craswell et al., 2024), systems were evaluated based on how closely their scores align with human judgments across graded relevance levels: 0 ("Irelevvntt", 1 ('Relevant topic, but does not contain the answer'), 2 (HHihly relevant, partial or unclear answer"), and 3 ('Perfeclly relevant, exact answer'). ). On the other hand, manually annotating text pairs with graded relevance is labor-intensive, restricting dataset scalability beyond evaluation benchmark sizes.
Early efforts to provide with scalable graded relevance signals involved training cross- encodingg rankers on annotated data, followed by using their judgments as pseudo-labels to be distilled into bi-encoders (Hofstätter al., 2021; Santhanam et al., 2022; Chen et al., 2024; Huang & Chen, 2024). More recently, prompting LLMs like GPT-4 to serve zero-shot rankers has yielded surprisingly strong results, sometimes surpassing supervised retriev- ers (Sun et al., 2023). These LLM-based rankers can produce nuanced relevance judgments at scale, unconstrained by binary decisions. For instance, an LLM can be prompted with: "On scale from 1 to 5, how well does this passage answer the query?" or offered options like 'Not relevant', , 'Partially relevant', and 'Highly relevant'. Zhuang et al. (2024) demonstrated that permitting GPT-4 PaLM to select among fine-grained relevance labels (instead of binary yes/ no decisions) significantly enhances ranking accuracy by more effectively handling borderline cases and reducing labeling noise.
Prior to the emergence of LLM-based relevance scoring, graded supervision was primarily leveraged via training with listwise pairwise objectives (Burges et al., 2005; Wang al., 2018; Qu et al., 2021; Reddi et al., 2021; Huang & Chen, 2024). However, such methods often rely on labeling multiple hard negative documents per query, significantly increasing annotation cost when using powerful LLMs as teachers, which hinders scalability. The op- portunity of LLM-generated large-scale graded relevance data, however, calls for revisiting training objectives to align with the increased costs of quality graded relevance data. In this work, we propose Binary Cross-Entropy Sentence Embeddings (BIXSE), a simple pointwise training method that directly optimizes binary cross-entropy (BCE) loss on graded relevance scores. BIXSE interprets graded relevance scores as probabilities within the range [0, 1] to represent relevance continuity from completely irrelevant (0) to absolutely relevant (1). Unlike pairwise or listwise objectives, which rely on multiple supervised com- parisons per query, BIXSE scales efficiently by enabling competitive performance by just
2

---

Published as a conference paper at COLM 2025
65.0
70.0
Average nDCG@10
DDCG@10
60.0
65.0
60.0
55.0
55.0
0.05
Model Size (Billions Parameters)
Label Flipping Probability
BIXSE
32B ranker (0-3 rel.)
32B ranker (0-1 rel.)
Figure 2: Left: BIXSE vs standard InfoNCE training of QWEN2.5 dense encoders. Our method outperforms the standard softmax-based contrastive recipe (InfoNCE) across model sizes when training the LightBlue dataset, which contains multilingual query-document pairs graded by zero-shot -WWEN2.5-3BBIINSI- 100k random samples of TREC-DL. qrels collected from 2019 to 2023. Right: BIXSE displays improved robustness to noise compared to InfoNCE models. We train MODERNBERT models on the open portion of E5 dataset and control the chance of flipping the binary label between the positive and the hard negative document. We report the average nDCG@10 score on subsampled version of BEIR.
ranker. We measure nDCG@10 on
using a single labeled query-document pair per query, while capturing structure implicitly via in-batch negatives.
We validate BIXSE through extensive experiments across retrieval and sentence embedding benchmarks, demonstrating consistent gains over standard InfoNCE objectives. Further- more, BIXSE is, to the best of our knowledge, the first pointwise training method to consistently match or outperform strong pairwise ranking baseline losses when training on I.L.-labeled graded relevance datasets. BIXSE scales across architectures and languages, approaches the performance of much larger zero-shot LLLM-based rankers, and exhibits improved robustness to label noise compared to InfoNCE. Notably, it benefits from learning across a wider spectrum of graded relevance and achieves peak performance even without aggressive data filtering, making it a strong and efficient alternative for training dense models on LLM-generated supervision. As graded relevance becomes increasingly easy to generate, we argue that BIXSE offers a practical, robust, and scalable training paradigm for the next generation of dense retrieval systems.
2 Related Work
Label Noise in Retrieval Datasets and Mining Hard Examples. Retrieval datasets generally consist of a query and a corresponding document with a relevance score. In the simplest case, the dataset only consists of positive documents (relevance score of 1.0). However, the dataset curation can introduce label noise in the dataset (Qu et al., 2021; Wang et al., 2022). For example, to select positive passage using question-answering dataset, Karpukhin et al. (2020) declare the highest-ranked passage from BM25 that contains the answer string as the positive passage. This process generates false positives it is possible to match passages that are not relevant but include the answer string. Label noise also exists in form of false negatives. For example, Qu al. (2021) report that within the top-retrieved passages for MSMARCO (Bajaj et al., 2018) dataset, 70% of them are actually positives while not being explicitly marked as positive. Reasonably, label noise is further amplified when retrieved passages are used hard negatives during training (Xiong et al., 2021; de Souza P. Moreira et al., 2024). We thus argue that utilizing training objectives that are more robust to noise can lead to downstream improvements in text encoders.
Training Dual Encoders with Synthetic Data. Many recent approaches have turned to synthetic data to get high-quality and diverse training samples at scale (Zhang et al., 2023a;
3

---

Published as a conference paper at COLM 2025
Wang et al., 2023; Dai et al., 2023; Muennighoff et al., 2024). E5 (Wang et al., 2023) proposed a two-step generation, where LLM first brainstorms potential downstream retrieval tasks, and then generates samples for each of those tasks. Promptagator (Dai et al., 2023) prompts an LLM to generate synthetic queries for existing passages and trains dense retrievers on these generated pairs. More recent approaches like Gecko (Lee et al., 2024b) and Gemini Embeddings (Lee et al., 2025) use LLMs for both sample generation and filtering, yielding high quality training dataset. The strong results of Gemini on MMTEB demonstrate the effectivess of LLM based dataset generation and filtering. Despite this progress, synthetic data generation has predominantly treated relevance as a binary signal, as documents are either relevant or not. The use of graded relevance scores, such as those common in TREC and traditional IR evaluation, remains largely unexplored as a training signal. Our work investigates this underutilized axis of synthetic supervision in training dense retrieval models.
Distillation Methods for Dense Retrieval and Ranking Models. In addition to data filtering, another key way to improve dual encoders is via distillation from stronger re- rankers. The idea is to train the dense retriever to imitate the outputs of a more powerful but slower model (the teacher). Typically, the teacher is a cross-encoder that can deeply inspect each query-document pair, producing superior relevance judgments.
Several loss formulations have been proposed to train with relevance judgements from the teacher model. Cheng et al. (2023) introduce "soft" InfoNCE, where regular one-hot label in InfoNCE is replaced by the soft labels from the teacher. We showcase in Section 5 that BIXSE outperforms this loss formulation. MarginMSE (Hofstätter et al., 2021) minimizes the mean squared error between teacher and student score margins for positive-negative pairs. A limitation of this approach is its inability to profit from in-batch negatives, as performance tends to decay with increased number of negative documents per positive ones. BIXSE fixes this by incorporating a special logit bias term targetted at counteracting this effect.
Pairwise training objectives have also been widely used for dense retrieval. RankNet (Burges et al., 2005), and its recent adaptations such as PairDistill (Huang & Chen, 2024), supervise the model by comparing pairs of documents for the same query. Instead of assigning an absolute relevance score to each document independently, these methods teach the model to prefer one document over another based on their relative graded relevance. In LambdaLoss (Wang et al., 2018), individual pairwise losses are further weighted according to their estimated impact on evaluation metrics such as nDCG. While effective, these pairwise methods tend to achieve their best performance when each query is compared against multiple other labeled documents. This increases annotation costs when labels come from expensive cross-encoders or LLM rankers. Because BiXSE applies binary cross-entropy loss at the pointwise level, it scales more naturally to large datasets and varied supervision sources. It enables training with fewer graded annotations per query, making it well-suited to scenarios where labeling costs are a concern. Our experiments show that BiXSE is competitive against strong pairwise baselines, while offering a lower annotation cost per query.
RocketQA(Qu et al., 2021) and its successor RocketQAv2 (Ren et al., 2021) employ an iterative listwise training procedure: a cross-encoder teacher is used to label a large set of positives and hard negatives, and the dual encoder is jointly trained on those via minimizing a KL on the document batch likelihoods. Similarly, RankDistill (Reddi et al., 2021) encourages the student to reproduce the teacher's 's top-k rankings, by using the teachers' scores to construct targets for the document batch likelihoods. However, these listwise or beacher-in-the-lopg approaches tend to be computationally expensive and less scalable running a cross-encoder over many candidate passages for every query (often repeatedly in training) incurs a large cost. Moreover, while listwise objectives seem desirable as they align with ranking metrics like nDCG, they assume access to complete and consistent slates of query-document pairs an assumption that could be violated in practice due to noisy, sparse, or inconsistent supervision from LLMs or heuristics. In contrast, BiXSE's pointwise formulation offers scalable and robust alternative, enabling effective learning from graded signals without requiring coherent global rankings.
4

---

Published as a conference paper at COLM 2025
3 Preliminaries
We present the notation needed to expose our training procedure by contrasting it to the typical multi-class contrastive learning of dense encoders, or InfoNCE (Oord et al., 2018) as it is commonly referred to in the context of self-supervised learning literature. Let be the text encoder we would like to train, which, in general, is a neural network that takes text as input and outputs presented with batches of B text tuples B {(9; di" dd; f Let qi be a query associated with positive document di and potentially a (hard) negative document d We define x (boldface math font) to be the L'-normalized embedding pooled from the text encoder f, that is x = f(x) [x(
fixed-dimensions vector. During training, the encoder is
To train with the InfoNCE objective, we first separately compute the normalized embeddings for each query qi = flai)
[[( and similarly for each positive document d7, and each hard negative document d,- Now, consider a scoring function s: (g.y) + IR between a pair (q, d) of text according to the dense encoder f. Typically, this is taken as a scaled dot-product between the normalized embeddings, s(q, d) == q d, for scalar hyperparameter & > 0 that acts as an inverse temperature. Then, the training loss for f can be expressed in terms of the function s(x, y) as
1 B
exp(s(q..d))) ie iB EjcB exp(s(gnd exp(s(qu.d7)
LInfONCE
Σ
log
(1)
This loss function corresponds to the negative log-likelihood of classifying each query qi into one of 2B possible classes, where each class represents a document from B. The target class for query 9: is always determined by the positive d, for every i E [B]. The alternative classes are defined by the hard negative document di and in-batch negatives', meaning that positives d and hard negatives d, associated by the dataset with other queries q;-} A i are also considered as negative documents with respect to 9: Karpukhin et al. (2020) show that this assumption is essential for achieving strong performance on downstream tasks, particularly in the absence of hard negative documents.
InfoNCE contrastive learning has been the de-facto paradigm for training dense retrievers and generalist sentence embedders. The training loss formulation in Equation (1) relies on binary relevance, i.e. a boolean assignment (relevant/ posittive or non-relevant/neggtive for every query-document pair.
BiXSE: Binary Cross-Entropy Sentence Embeddings
The search for a more fine-grained learning signal starts within the information retrieval community (Järvelin & Kekäläinen, 2000; 2002; Sakai, 2021) and its efforts to define a query- document relevance that is more closely aligned with human judgement. Such reliance on human annotations has hindered the scalability of graded relevance labeled datasets. However, recent LLMs have demonstrated strong zero-shot ranking capabilities (Sun et al., 2023; Zhuang et al., 2024), surpassing dense retrieval models based on human judgement, without requiring additional fine-tuning on ranking data. Although automated, querying LLMs for generating a quality dataset remains an expensive procedure constrained by financial considerations. Therefore, it is crucial to develop scalable solutions that still provide competitive advantages when training with graded relevance data. As discussed in Section 2, listwise approaches (Qu et al., 2021; Ren et al., 2021; Reddi et al., 2021) require labeling multiple documents per query by the LLM, which constraints the scalability of data along another qualitative axis such the number of queries contained in a dataset given a fixed LLM token consumption budget. At the same time, many of those query-document
or cross-batch if they belong to mini batches sampled on different computational devices of a data parallel training trial. In the paper, we use in-hatch and cross- mass- interchangeably to mean that all other unlabeled query-document combinations are used negatives.
5

---

Published as a conference paper at COLM 2025
pairs are filtered out in state-of-the-art pipelines (Lee et al., 2024b; 2025) based on LLM- generated graded relevance, leading to token waste. By exploring a pointwise solution, our goal is to develop scalable, token-efficient and effective method for leveraging graded relevance information.
Our work focuses improving the effectiveness of models trained on dataset with graded relevance scores. In contrast to the setting in Section 3, batches from this dataset may now consist of query associated with a set of K > 0 documents, and a score that indicates the
relevance of each document. More formally, for given query 9: let a be the k document
associated with 9:, for each ., in the batch. We let 2; [0,1] denote the continuous relevance score between query 9; and document d(a), with being most relevant
and 1 absolutely relevant. Then, a batch B contains a set of tuples (q,, { {d( 2z ) 1 ,)-
(k)
This setup generalizes the one described in Section 3 where relevance is a binary concept:
previously, document is either relevant to query (denoted as d,7) or is not (d with notion of degree of relevance. Our proposed setup, in which relevance degree is represented as the score 2;, a generalization of the binary setup with 2; set for positive documents df and to 0 for hard negative documents d7-
Given that we are studying a pointwise solution, we restrict our discussion to the setup where each query di is relevant single document (K 1). This does not limit generality, as triplets if needed.
our pointwise loss can be applied to additional gueery-oocument-eeevaaee Accordingly, we thus redefine batch as := (4; 4, 3,1, as set of query-document- relevance triplets during training.
One way we can practically acquire graded relevance labels at scale is by asking an LLM to output an ordinal judgement about a query-document pair given relevance definition in the instruction (Zhuang et al., 2024), as visualized in Figure 1. Having access to the logits of the model furthermore enables us to convert finite number of discrete ordinal judgements to continuous ones by using the constrained predicted probability among the discrete score options to average the possible ordinal outcomes. More formally, given relevance definition that contains set of possible discrete scores S 5= 10,1, .N, can convert these scores to continuous ones = := Lees PLLM 9, d) (Lightblue, 2025). Then, z can be converted to [0, 1] via a (rank-preserving) increasing function, such as an affine transformation -.
As before, batches of tuples in B are presented the encoder during training. Similarly as before, each query and document are separately processed via encoder f to produce ('-normalized embeddings 9; and d; respectively. This time we opt for a different choice of scoring function, which is defined as s(q.d) aq d + for some logit scale and bias parameters a > and B. We enhance the loss formula from Equation (1) by incorporating crucial logit bias term to improve performance. We make use of in-batch negatives by supposing labels Zi, = 0 for + j. otherwise we use the graded relevance score from the
batch Z 2,- Let (xx) (1 + exp(-x))- be the sigmoid/ logistic function. Then, the encoder f is trained to minimize the binary cross-entropy (BCE) loss
LIXSE = B
Σ
Σ
Ziz (oggos(()).)))) + (1 z1,)) logor - s(gi.d)).
(2)
iE(B
The behavior of this loss function depends on the value of Zij. When žr, > 0.5, we want the sigmoid to output a value closer to 1, implying that the normalized embeddings qi and d, need be more aligned.
The role of logit bias B. The use of in-batch negatives introduces strong label imbalance: for each query qi, the model sees one (likely) positive document d, and skewing the label distribution toward zero. This imbalance becomes more pronounced with larger batch sizes, making it easier for the encoder ff to minimize the loss by predicting low relevance scores across the board. We interpret the logit bias ß in Equation (2) as mechanism to correct for this skew by modeling the marginal label distribution. Since / is not conditioned on query-document pairs, it cannot learn relevance itself, but it can only
negatives,
6

---

Published as a conference paper at COLM 2025
Models finetuned on English-only data
Categories
BEIR
MTEB(eng, v2) Retr.
TREC-DL 19-23*
MTEB(eng, v2) All
MODERNBERT-BASE
INFONCE 41.32
39.17
42.32
51.79
BIXSE
42.29 (+2.3)
41.11 (+5.0%)
47.67 (+12.6%
55.66 (+7.5%)
META-LLAMA-
3.
INFONCE
49.31
48.76
58.39
62.87
BIXSE
49.51 (+0.4))
50.60 (+3.8%
59.85 (+2.5%)
63.13 (+0.4%)
Models finetuned multilingual data
Categories
BEIR
MTEB(multi, v1) Retr.
TREC-DL 19-23
MTEB(multi, v1) All
QWEN2.5-02 12--0.B--INNSTUE
INFONCE 38.25
50.24
55,29
48.98
BIXSE 73.91 (+6.0))
51.90(43.37
55.71 (+0.8%)
49.68 (+1.4%)
QWEN2.5-1.5B 1...5-1.5B-ISTTLLC
INFONCE 43.66
55.00
57.93
52.64
BIXSE 48.08 (+10.1)
58.45 (+6.3%)
59.95 (+3.5%)
53.59 (+1.8%
OWEN2.5-3B-INSTRU
INFONCE 48.83
59.22
62.40
55.74
BIXSE 50.55 (+3.5%)
61.67 (+4.1%)
64.70 (+3.7%)
57.02 (+2.3%)
Table 1: Average aggregated NDCG@10 performance various retrieval benchmarks of models trained with the same data resources. BIXSE consistently outperforms models trained with the standard softmax-based contrastive learning loss. *We benchmark against corpus formed by collecting all annotated documents in the available qrels from 2019 until 2023.
model the marginal distribution of labels. To encourage this separation of roles, we optimize with significantly higher learning rate than f, ensuring that the encoder f must rely the actual query-document content to minimize the loss. We study the effect of this design choice in Figure 5 and Appendix 6.
Experiments
Base Models. For our experiments we finetune a variety of base models seeking to validate the benefits of our approach for different pretraining methods and model sizes. For this reason, we experiment with MODERNBERT (Warner et al., 2024), bidirectional masked language model (MLM), as well as with META-LLAMA-3.2 (Meta AI, 2024) and QWEN2.5 (Qwen Team, 2024) models exemplifying decoder-as-encoode approaches. As use the representation extracted token. On the other hand, we
sentence embedding for the MODERNBERT MODERRBERTacchhitettuu by the standard pooler on top of the convert decoder LLM architecture following the L.M2Vec recipe (BehnamGhader et al., 2024). In particular, we disable the autoregressive masking, thus enabling bidirectional connections among tokens, and we pool token representations into fixed dimensional sentence embedding by averaging the last layer representations over the query tokens, or the document tokens accordingly. We skip the first MLM finetuning step of LLM2 instead we directly finetune the adapted models with constrastive objectives. Finally, we enable our models to be prompted with task-specific instructions (Su et al., 2023) when we encode queries, allowing for extra flexibility in the representation space.
Datasets. Our experiments span across multilingual and English-exclusive data. We also experiment with data labeled with graded relevance scores, binary and mixed cases, hoping to demonstrate the effectiveness of BiXSE in -odoaaaalopptiibleeee manner. For binary relevance data, we use the public portion of the E5 dataset (Wang et al., 2022) reconstructed by Springer et al. (2024). In Table the Appendix, we describe the composition of training sets and their corresponding tasks. In Section 6 of the Appendix, we provide the list of instructions we used to augment queries. We use the same task instructions as previous works Wang et al. (2023); Springer et al. (2024); BehnamChader et al. (2024). As for graded
7

---

Published as a conference paper at COLM 2025
BEIR (short)
MTEB (multi, v1) Retrieval
TREC-DL 19-23
70
70/26
68.96
08:83
52
51:13
51,27
56
55.29
55,71
67.40
nDCG@10
65
48
53.62
42,50
57.14
42
50.63
Margin MSE
InfoNCE
Soft InfoNCE
BIXSE BIXSE (tinary)
BIXSE (no logit bias)
Margin MSE
InfoNCE
Soft InfoNCE
BIKSE (binary)
BIXSE (no logit blas)
BIXSE
Margin MSE
InfoNCE
Soft InfoNCE
BIXSE (binary)
BIXSE ino legit bias)
BIXSE
Figure 3: Ablating loss function variants by training OWEN2.5-0.5B-B INSTRUCT models; binary cross-entropy loss performs the best when both graded relevance scores and logit bias in the scoring function are used.
relevance data, we base our experiments on a public dataset (Lightblue, 2025) that we will call LightBlue (or LB in abbreviation). LightBlue a collection of 35 high quality question- answering datasets covering more than 95 languages, whose authors curate for training distilled cross-encoders. For query-document pairs in the collection, the authors ask WWW.225:22--NNSSRUCTTTTTTT TATTB-NNTTUTT-TT--IINNN
model zero-shot rank their relevance with discrete number scaling from to 5. The provide with model logits over the token responses, which we then convert to a continuous label in [0, 1], as we discussed in Section 4. More details about LightBlue dataset are discussed in Appendix 6.
Benchmarks. We evaluate our approach on several benchmarks spanning heterogeneous retrieval, graded relevance retrieval, and universal text embedders. BEIR (Thakur et al., 2021) is a zero-shot heterogeneous retrieval evaluation framework encompassing diverse tasks and domains. It consists of 18 different datasets from nine different tasks - Fact checking, citation prediction, duplicate question retrieval, argument retrieval, news retrieval, question answering, tweet retrieval, bio-medical IR, and entity retrieval.
To evaluate our method beyond retrieval, we also consider MMTEB (Enevoldsen et al., 2025) (multilingual version of MTEB Muennighoff et al. (2023)) which covers multiple embedding- based tasks across various languages. MMTEB introduced several benchmarks - MTEB (eng, v2) is the faster zero-shot version of widely popular MTEB benchmark (Muennighoff et al., 2023) which contains 41 English embedding tasks across seven task categories retrieval, reranking, clustering, pair classification, classification, sentence similarity, and summarization. MTEB tmulti, v1) on the other hand consists of 131 tasks from several task categories and languages. show results on both retrieval and the complete set of MMTEB benchmarks.
Last, we perform evaluations against graded relevance scores reflecting human judgement about query-document relations using publically available data from TREC-DL (Craswell et al., 2024) competitions. We assemble the qrels from all competitions from 2019 till 2023 into collection of 422612 graded relevance scores over 1988 queries on the MSMARCO (Bajaj et al., 2018) passage corpus.
Main Results. We train models English data consisting of a mixture of binary rele- vance E5 and graded relevance LB datasets, and multilingual models purely on graded relevance scored data from LB. Table 1 reports NDCG@10 performance of models trained with BIXSE versus InfoNCE across English and multilingual settings, covering both retrieval and sentence embedding tasks. BIXSE consistently outperforms InfoNCE across all base models and benchmarks, with the most substantial gains observed on TREC-DL 2019-2023 highlighting BIXSE's strength in modeling nuanced relevance signals; for instance, BIXSE achieves +12.6% improvement on MODERNBERT-DAA and +3.5% on W...N2.5-1.5B-INTTEE. and MTEB (Retrieval), BIXSE maintains consistent improvements - e.g. +10.1% on BEIR
+3.7% on On binary relevance benchmarks such as BEIR
Pnttps://huggingfaac http://awinggaaaac.////////////e co/Qwen/Qwen2. 5-38------------ http://www.......mm-mmmennnrrrrr Instruct-GPTQ-IInt
8

---

Published as a conference paper at COLM 2025
BEIR (short)
TREC-DL 2019-2023
Average nDCG@10
71.0
55.5
70.0
ADCG@10
55.0
69.0
68.0
54.0
67.0
53.5
0.1
0.5
0.7
0.1
0.5
0.7 0,9
Data Filter Cutoff Relevance
Data Filter Cutoff Relevance
BIXSE
InfoNCE
Figure Performance of QWEN2.5-00 2.5.550-00----SRR-E models trained using BIXSE or InfoNCE models under different graded relevance cutoff values for positives. InfoNCE improves as discard data pairs of lower graded relevance, while BIXSE follows reverse U-curve trend. Overall the best BIXSE model outperforms the best InfoNCE one, showing that BIXSE effectively enables learning from graded relevance labels, and consequently less token consumption waste during dataset creation.
with QWEN2.5-1.5P -NSSSUUTT Gains also extend to the average of all tasks in MTEB, general sentence embedding suite, where BIXSE yields up to +7.5% improvement, indicat- ing that training with graded supervision enhances embedding quality beyond retrieval. Altogether, across six model backbones and four benchmark families, BIXSE delivers re- liable, architecture-agnostic improvements in both binary and graded settings, validating its core motivation of leveraging LLM-derived graded relevance labels through a simple yet effective binary cross cross-entropy objective. In Appendix 6, we provide with analytical reports of model performance per individual tasks contained in BEIR MTEB, whereas in Table 9 of Appendix 6 we provide with analytical model performance results on the top-100 document reranking tasks of TREC-DL for each year separately.
Comparison to other training objectives. We further compare BIXSE against alternative training objectives such as Soft InfoNCE (Cheng et al, 2023) and Margin MSE (Hofstätter et al., 2021), as well as ablated variants of BIXSE. While Soft InfoNCE offers smoother alter- native one-hot probability targets and Margin MSE explicitly models pairwise differences, both underperform BIXSE across all benchmarks, highlighting the advantage of directly optimizing for graded relevance via BCE. Notably, ablations that remove the logit bias or discretize the targets into binary values consistently degrade performance, confirming the importance BIXSE's full formulation. Finally, we have conducted a comprehensive comparison between BiXSE and other strong pairwise ranking objective baselines, which we detail in Appendix 6. Across both Lightßlue and BGE-M3 (Chen et al., 2024) training datasets, BIXSE consistently matches outperforms alternatives like Pairwise BCE (Burges et al., 2005; Huang & Chen, 2024) and Lambdal (Wang et al., 2018), while requiring fewer labeled negatives and enabling more efficient in-batch training. These findings vali- date the scalability and supervision efficiency of BIXSE, when compared against pairwise alternatives.
Effective distillation from reranker. To contextualize BIXSE's performance, in the left plot of Figure we compare our dense encoders to two 32B LLM-based rerankers. We instruct Wiieen.5-32B-Instruu to score query-document pairs from TREC-DL (2019-2023), using prompts aligned with the original human annotation guidelines. In the 0-3 relevance setting, the model uses four-point relevance scale; in the 0-1 setting, it operates under a simplified binary prompt distinguishing "Irrelevant" from "Relevant, Exact Answer." " Despite the model size gap, our 3B BIXSE encoder achieves an nDCG@10 of 62.06 within 3.0 points of the 32B ranker with full 0-3 prompting (65.06), and just 0.8 points behind the 32B ranker trained with binary prompts (62.86). This result demonstrates that BIXSE can effectively distill LLM supervision into fast, efficient dense retrieval models without sacrificing ranking quality.

---

Published as a conference paper at COLM 2025
Robustness to noise. A key motivation for BIXSE is its robustness to labeling noise, which is prevalent in large-scale retrieval datasets due to imperfect negative mining, synthetic supervision, or human annotation inconsistencies. To test this, we simulate controlled label noise by flipping the binary labels between the positive and hard negative documents in the E5 dataset according to prescribed probability, prior to training. We observe in the right plot of Figure 2 that, as the noise level increases, BIXSE exhibits a markedly more gradual degradation in nDCG@10 compared to InfoNCE. While both methods see performance drops, the curve for BIXSE remains more concave and stable, indicating greater tolerance to noise. We invite the reader Appendix 6, where we discuss our intuition behind these results.
Learn best by filtering less. We investigate how BIXSE and InfoNCE respond to different thresholds for filtering training data by graded relevance. In large-scale datasets built from LLM-generated scores, a common preprocessing step is discard query-document pairs with low relevance to improve training quality. We vary the minimum relevance cutoff required to retain a pair, and treat the remaining data differently: for InfoNCE, retained pairs are binarized and used positives; for BIXSE, we preserve their graded scores and train using the full BCE formulation. To control for dataset size, we fix the number of training examples across all cutoff values by subsampling each filtered set to match the smallest resulting dataset (corresponding to the highest cutoff of 0.9). The results, shown in Figure 4, reveal that InfoNCE performance improves monotonically with stricter filtering, suggesting that it benefits from high-confidence positives and sharper separation from negatives. In contrast, BIXSE follows reverse U-shaped trend, achieving peak performance moderate cutoff (e.g., 0.7), indicating that it can effectively learn from broader range of relevance signals. Crucially, the best BIXSE model outperforms the best InfoNCE model, suggesting that BIXSE not only enables better generalization from nuanced supervision, but also reduces the need for aggressive data guuingg-alowwer more effective use of available training pairs with less token waste.
Conclusion
We present BIXSE, simple and effective training objective that enables dense retrieval models to learn directly from graded relevance labels. By replacing contrastive formulations with a binary cross-entropy loss over continuous targets, BIXSE captures fine- fine-grained supervision more faithfully, yielding consistent performance gains across diverse retrieval and embedding benchmarks. Our results show that BIXSE not only outperforms standard objectives like InfoNCE and Margin MSE, but also narrows the gap to large zero-shot LLM rankers, improves robustness label noise, and learns better without aggressive filtering. At the same time, it performs competitively with state-of-the-a while requiring fewer annotated documents per query. As LLMs make graded relevance supervision increasingly accessible, BIXSE provides practical and scalable solution for distilling this supervision into fast and generalizable dense encoders.
pairwise losses,
Acknowledgments
This work was done while Christos Tsirigotis and Vaibhav Adlakha were interning at ServiceNow Research, Montréal. For João Monteiro, work was done prior to joining Apple MLR. Christos acknowledger funding from the Mitacs Accelerate program, as well as from Aaron Courville's CCAI chair and CRC chair.
References
Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, Mir Rosenberg, Xia Song, Alina Stoica, Saurabh Tiwary, and Tong Wang. Ms marco: A human generated machine reading comprehension dataset. arXiv preprint ArX 1.09668, 2018.
10

---

Published as a conference paper at COLM 2025
Parishad BehnamGhader, Vaibhav Adlakha, Marius Mosbach, Dzmitry Bahdanau, Nicolas Chapados, and Siva Reddy. Llm2vec: Large language models are secretly powerful text encoders. Xio preprint ar Xix:2404.05900 2024.
Chris Burges, Tal Shaked, Erin Renshaw, Ari Lazier, Matt Deeds, Nicole Hamilton, and Greg Hullender. Learning to rank using gradient descent. In Proceedings of the 22nd International Conference on Machine Learning, 2005.
Jianly Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. Bge m3-embedding: Multi-lingual, fultiiftuctionaaittyy -ulti-granularity text embeddings through self-knowledge distillation, 2024.
Qinyuan Cheng, Xiaogui Yang, Tianxiang Sun, Linyang Li, and Xipeng Qiu. Improving con- trastive learning of sentence embeddings from AI feedback. In Findings of the Association for Computational Linguistics: ACL 2023, 2023.
Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Hossein A. Rahmani, Daniel Campos, Jimmy Lin, Ellen M. Voorhees, and Ian Soboroff. Overview of the trec 2023 deep learning track. In Text REtrieval Conference (TREC). NIST, TREC, 2024. URL https: h/ww://wwwwwww wwwwwooffttocooommmmmoomommoco.e com/ of- trec- 2023 -ep3--ee---errming-tre
Zhuyun Dai, Vincent Y Zhao, Ji Ma, Yi Luan, Jianmo Ni, Jing Lu, Anton Bakalov, Kelvin Guu, Keith Hall, and Ming-Wei Chang. Promptagator: Few-shot dense retrieval from 8 examples. In The Eleventh International Conference Learning Representations, 2023.
DataCanary, hilfialkaff, Jiang Lili, Risdal Meg, Dandekar Nikhil, and tomtung. Quora question pairs, 2017. URL httpps://aag/// com/ wttpoo////cccciiiooeeeeemmooeeie /
Gabriel de Souza P. Moreira, Radek Osmulski, Mengyao Xu, Ronay Ak, Benedikt Schifferer, and Even Oldridge. Nv-retriever: Improving text embedding models with effective hard-negative mining. arXiv preprint ar arXio:2407.1158 2024.
Kenneth Enevoldsen, Isaac Chung, Imene Kerboua, Márton Kardos, Ashwin Mathur, David Jay Gala, Wissam Siblini, Dominik Krzeminski, Genta Indra Winata, Saba Sturua, Saiteja Utpala, Mathieu Ciancone, Marion Schaeffer, Diganta Misra, Shreeya Dhakal, Jonathan Rystrøm, Roman Solomatin, Ömer Veysel Çagatan, Akash Kundu, Martin Bern- storff, Shitao Xiao, Akshita Sukhlecha, Bhavish Pahwa, Rafat Poświata, Kranthi Kiran GV, Shawon Ashraf, Daniel Auras, Björn Plüster, Jan Philipp Harries, Loic Magne, Isabelle Mohr, Dawei Zhu, Hippolyte Gisserot-Boukhlef, Tom Aarsen, Jan Kostkan, Konrad Woj- tasik, Taemin Lee, Marek Suppa, Crystina Zhang, Roberta Rocca, Mohammed Hamdy, Andrianos Michail, John Yang, Manuel Faysse, Aleksei Vatolin, Nandan Thakur, Manan Dey, Dipam Vasani, Pranjal A Chitale, Simone Tedeschi, Nguyen Tai, Artem Snegirev, Mariya Hendriksen, Michael Günther, Mengzhou Xia, Weijia Shi, Xing Han Lù, Jor- dan Clive, Gayatri K, Maksimova Anna, Silvan Wehrli, Maria Tikhonova, Henil Shalin Panchal, Aleksandr Abramov, Malte Ostendorff, Zheng Liu, Simon Clematide, Lester James Validad Miranda, Alena Fenogenova, Guangyu Song, Ruqiya Bin Safi, Wen-Ding Li, Alessia Borghini, Federico Cassano, Lasse Hansen, Sara Hooker, Chenghao Xiao, Vaibhav Adlakha, Orion Weller, Siva Reddy, and Niklas Muennighoff. MMTEB: Massive multilingual text embedding benchmark.' In The Thirteenth International Conference on Learning Representations, 2025.
Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli EL15: Long form question answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. SimCSE: Simple contrastive learning of sentence embeddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 2021.
Wei He, Kai Liu, Jing Liu, Yajuan Lyu, Shiqi Zhao, Xinyan Xiao, Yuan Liu, Yizhong Wang, Hua Wu, Qiaoqiao She, Xuan Liu, Tian Wu, and Haifeng Wang. DuReader: a Chinese machine reading comprehension dataset from real-world applications. In Proceedings of the Workshop on Machine Reading for Question Answering, 2018
11

---

Published as a conference paper at COLM 2025
R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In International Conference on Learning Representations, 2019.
Sebastian Hofstätter, Sophia Althammer, Michael Schröder, Mete Sertkan, and Allan Han- bury. Improving efficient neural ranking models with cross-architecture knowledge distillation, 2021.
Chao-Wei Huang and Yun-Nung Chen. PairDistill: Pairwise relevance distillation for dense retrieval. In Proceedings of the 2024 Conference Empirical Methods in Natural Language Processing, 2024.
Kalervo Järvelin and Jaana Kekäläinen. Ir evaluation methods for retrieving highly relevant documents. Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, 2000.
Kalervo Järvelin and Jaana Kekäläinen. Cumulated gain-based evaluation of ir techniques. ACM Trans. Inf. Syst., 20(4), 2002.
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. TriviaQA:
A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2017.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question an- swering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference Learning Representations, 2015. 2015.
Tom Kwiatkowski, Jennimaria Palomaki, Michael Redfield, Olivia andk Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. Natural questions: A benchmark for question answering research. Transactions of the Association for Compulational Linguistics, 7, 2019.
Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. Nv-embed: Improved techniques for training Ilms generalist embedding models. ariiv preprint ar Xix:::2405.11 174228 2024a.
Jinhyuk Lee, Zhuyun Dai, Xiaoqi Ren, Blair Chen, Daniel Cer, Jeremy R. Cole, Kai Hui, Michael Boratko, Rajvi Kapadia, Wen Ding, Yi Luan, Sai Meher Karthik Duddu, Gus- tavo Hernandez Abrego, Weiqiang Shi, Nithi Gupta, Aditya Kusupati, Prateek Jain, Siddhartha Reddy Jonnalagadda, Ming- Chang, and Iftekhar Naim. Gecko: Versatile text embeddings distilled from large language models, 2024b.
Jinhyuk Lee, Feiyang Chen, Sahil Dua, Daniel Cer, Madhuri Shanbhogue, Iftekhar Naim, Gustavo Hernnndez Ábrego, Zhe Li, Kaifeng Chen, Henrique Schechter Vera, Xiaoqi Ren, Shanfeng Zhang, Daniel Salz, Michael Boratko, Jay Han, Blair Chen, Shuo Huang, Vikram Rao, Paul Suganthan, Feng Han, Andreas Doumanoglou, Nithi Gupta, Fedor Moiseev, Cathy Yip, Aashi Jain, Simon Baumgartner, Shahrokh Shahi, Frank Palma Gomez, Sandeep Mariserla, Min Choi, Parashar Shah, Sonam Goenka, Ke Chen, Ye Xia, Koert Chen, Sai Meher Karthik Duddu, Yichang Chen, Trevor Walker, Wenlei Zhou, Rakesh Ghiya, Zach Gleicher, Karan Gill, Zhe Dong, Mojtaba Seyedhosseini, Yunhsuan Sung, Raphael Hoffmann, and Tom Duerig. Gemini embedding: Generalizable embeddings from gemini, 2025.
Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. Towards general text embeddings with multi-stage contrastive learning. ar Xiv preprint ar arXin:2308.0325 2023.
12

---

Published as a conference paper at COLM 2025
Lightblue. Lightblue reranker distillation dataset. https://huggingfaat ctt.taattceoottatsssttssssssssse 2025. Accessed: 2025-03-27.
Sadhika Malladi, Kaifeng Lyu, Abhishek Panigrahi, and Sanjeev Arora. On the SDEs and scaling rules for adaptive gradient algorithms. In Advances in Neural Information Processing Systems, 2022.
Meta AL. LLaMA-32-1B-Instruee 2024. URL https://huggingge co/meta-1lama Llama-3. Instruct. LLaMA 3.2 series, 1B-parameter instruction-tuned model.
Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. MTEB: Massive text embedding benchmark. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, 2023.
Niklas Muennighoff, Hongjin Su, Liang Wang, Nan Yang, Furu Wei, Tao Yu, Amanpreet Singh, and Douwe Kiela. Generative representational instruction tuning. arxiv preprint wwXiv:2402.09906 2024.
Ansong Ni, Matt Gardner, and Pradeep Dasigi. Mitigating false-negative contexts in multi- document question answering with retrieval marginalization. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 2021.
Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Hernandez Abrego, Ji Ma, Vincent Zhao, Yi Luan, Keith Hall, Ming-Wei Chang, and Yinfei Yang. Large dual encoders are generalizable retrievers. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, 2022.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. ariiv preprint orxi::1807.000 00748,,
007.03488, 2018.
Yingqi Qu, Yuchen Ding, Jing Liu, Kai Liu, Ruiyang Ren, Wayne Xin Zhao, Daxiang Dong, Hua Wu, and Haifeng Wang- RocketQA: An optimized training approach to dense passage retrieval for open-domain question answering. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2021.
Qwen Team. Swve225-0.5B-Instruee 2024. URL h co//wen/Qwen2..5- 1B-Instruct. Qwen2.5 series, 0.58-parameter instruction-tuned model.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning, 2021.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, 2016.
Sashank Reddi, Rama Kumar Pasumarthi, Aditya Menon, Ankit Singh Rawat, Felix Yu, Seungyeon Kim, Andreas Veit, and Sanjiv Kumar. Rankdistil: Knowledge distillation for ranking. In Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, 2021.
Nils Reimers and Iryna Gurevych. Sentence-BERI: Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 2019.
Ruiyang Ren, Yingqi Qu, Jing Liu, Wayne Xin Zhao, QiaoQiao She, Hua Wu, Haifeng Wang, and Ri-Rong Wen. RocketQAv2: A joint training method for dense passage retrieval and passage re-ranking. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 2021.
13

---

Published as a conference paper at COLM 2025
Stephen Robertson and Hugo Zaragoza. The probabilistic relevance framework: Bm25 and beyond. Found. Trends Inf. Retr., 2009.
releabl
Tetsuya Sakai. Graded Relevance, chapter Graded Relevance, pp. 1-20. Springer Singapore, 2021. ISBN 978-981-15-5554-1. doi: 10.1007 11110779778991---555---4
Keshav Santhanam, Omar Khattab, Jon Saad-Falcon, Christopher Potts, and Matei Zaharia. ColBERTv2: Effective and efficient retrieval via lightweight late interaction. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2022.
Jacob Mitchell Springer, Suhas Kotha, Daniel Fried, Graham Neubig, and Aditi Raghunathan. Repetition improves language model embeddings. ar Xiv preprint ar X12:24402.15449, 2024.
Hongjin Su, Weijia Shi, Jungo Kasai, Yizhong Wang, Yushi Hu, Mari Ostendorf, Wen-tau Yih, Noah A. Smith, Luke Zettlemoyer, and Tao Yu. One embedder, any task: Instruction- finetuned text embeddings. In Findings of the Association for Computational Linguistics: ACL 2023, 2023.
Weiwei Sun, Lingyong Yan, Xinyu Ma, Shuaiqiang Wang, Pengjie Ren, Zhumin Chen, Dawei Yin, and Zhaochun Ren. ChatGPT good at search? investigating large language models as re-ranking agents. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023.
Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. BEIR: A heterogeneous benchmark for zero-shot evaluation of information retrieval models. In Thirty-fifth Conference on Neural Information Processing Systems Datasels and Benchmarks Track (Round 2), 2021.
James Thorne, Andreas Vlachos, Christos Christodooulpoulls, and Arpit Mittal. FEVER: large-scale dataset for fact extraction and VERification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume 1 (Long Papers), 2018
Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. Text embeddings by superi-sue contrastive pre-training arviv preprint XrXiv:2112.03533, 2022.
Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. Improving text embeddings with large language models. ar Xiv preprint Kiix:2401.00366 2023.
Xuanhui Wang, Cheng Li, Nadav Golbandi, Mike Bendersky, and Mare Najork. The lambdaloss framework for ranking metric optimization. In Proceedings of The 27th ACM International Conference on Information and Knowledge Management (CIKM '18), 2018.
Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, Oskar Hallström, Said Taghadouini, Alexis Gallagher, Raja Biswas, Faisal Ladhak, Tom Aarsen, Nathan Cooper, Griffin Adams, Jeremy Howard, and lacopo Poli. Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference, 2024.
Xiaohui Xie, Qian Dong, Bingning Wang, Feiyang Lv, Ting Yao, Weinan Gan, Zhijing Wu, Xiangsheng Li, Haitao Li, Yiqun Liu, and Jin Ma. T2ranking: A large-scale chinese benchmark for passage ranking. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2023.
Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul N. Bennett, Junaid Ahmed, and Arool Overwijk. Approximate nearest neighbor negative contrastive learning for dense text retrieval. In International Conference on Learning Representations, 2021.
14

---

Published as a conference paper at COLM 2025
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhut- dinov, and Christopher D. Manning. HotpotQA: dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023.
Junlei Zhang, Zhenzhong Lan, and Junxian He. Contrastive learning of learning from scratch. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023a.
sentence embeddings
Xinyu Zhang, Xueguang Ma, Peng Shi, and Jimmy Lin. Mr. TyDi: A multi-lingual bench- mark for dense retrieval. In Proceedings of the 1st Workshop on Multilingual Representation Learning, 2021.
Xinyu Zhang, Nandan Thakur, Odunayo Ogundepo, Ehsan Kamalloo, David Alfonso- Hermelo, Xiaoguang Li, Qun Liu, Mehdi Rezagholizadeh, and Jimmy Lin. MIRACL: A Multilingual Retrieval Dataset Covering 18 Diverse Languages. Transactions of the Association for Computational Linguistics, 2023b.
Honglei Zhuang, Zhen Qin, Kai Hui, Junru Wu, Le Yan, Xuanhui Wang, and Michael Bendersky. Beyond yes and no: Improving zero-shot LLM rankers via scoring fine- grained relevance labels. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2:
Short Papers), 2024.
15

---

Published as a conference paper at COLM 2025
Appendix
Extended Related Work
Neural Dense Retrieval. Classical retrieval methods like TF-IDF and BM25 (Robertson & Zaragoza, 2009) represent texts as sparse, high-dimensional vectors based on lexical overlap, effectively capturing keyword matches but lacking deeper semantic understanding. Neural dense retrieval methods address this by embedding texts into dense semantic vector spaces using pre-trained language models (Karpukhin et al., 2020; Xiong et al., 2021).
Dense retrieval methods broadly fall into two categories based on their trade-offs between computational efficiency and retrieval accuracy: bi-encoders (dual encoders) and cross- encoders. Bi-encoders encode queries and documents independently, enabling efficient approximate nearest neighbor (ANN) searches suitable for large-scale retrieval (Ni et al., 2022; Li et al., 2023; Wang al., 2022; BehnamGhader al., 2024; Lee al., 2024a), but they fail to capture fine-grained contextual understanding. Cross-encoders jointly encode query- document pairs, providing more accurate relevance estimates but higher computational costs, limiting their scalability. The simplest example of cross-encoder is feeding query- document pairs into large language models (LLMs) to explicitly assess relevance (Sun al., 2023).
With advancements in sentence representation techniques, the research community has expanded its focus from narrow retrieval tasks to general-purpose text embeddings. BEIR (Thakur et al., 2021) a zero-shot evaluation framework encompassing diverse retrieval tasks and domains whereas MMTEB (Enevoldsen et al., 2025) covers multiple embedding- -baee tasks like retrieval, classification, and clustering across various languages. To encourage generalization across tasks, it is now common to prepend each input with natural language task description, guiding the encoder to produce task-specific representa- tions (Su et al., 2023). In this paper, adopt similar formulation by conditioning on prompt that describes the relevance task.
Representation Learning with Binary Cross- Entsopy. To our knowledge, binary cross- entropy loss in self-supervised representation learning was first explored by Hjelm et al. (2019). Their work introduces Deep InfoMax (DIM), framework for representation learning that maximizes mutual information between different views of the same image. In its Jensen- Shannon mutual information formulation (Hjelm al., 2019, See Equation 4, Section 3.1), the DIM loss functions as binary cross-entropy loss and the logits are computed from pairs of views: pairs derived from the same image are labeled as positive, while pairs from different images are labeled as negative.
More recently, Zhai et al. (2023) introduced SigLIP, framework that pre-trains vision- language encoders by aligning image embeddings with their corresponding caption embed- dings. Compared to its softmax-based anchor classification alternative (Radford et al., 2021, CLIP), pretraining with binary cross-entropy loss in SigLIP has demonstrated better scaling of downstream task performance as the number of in-batch negatives increases. In our work, adapt the SigLIP objective to fine-tune large language models (LLMs) (BehnamGhader et al., 2024) for general-purpon text embedding tasks.
16

---

Published as a conference paper at COLM 2025
Lightblue Reranker Distillation Dataset
Lightblue (2025) constructed this dataset through a four-step process aimed at creating di- verse, high-quality resource for evaluating query-text relevance. First, they collected queries and associated text passages from 35 publicly available datasets spanning over 95 languages. For datasets lacking hard negatives, they mined them using the DAAI/BGEEMM embedding model to ensure challenging contrastive setup. Then, we used the QWEN (WWNN/5-E3222 INSTRUCT-GPTQQ CET-CCT----TTPP model to rate the relatedness of each query-text pair on 5-point scale, producing token-level probabilities for scores "1" through "5".
Training and Evaluation Details
Table 2: Composition of the public portion of the E5 training dataset (Wang et al., 2023), reconstructed by Springer et al. (2024). In the task categories below, Natural Language Inference is denoted by NLI and Question-Answering as QA. Details on the instructions used for each dataset can be found at Section 6 of the Appendix.
Dataset Name
Task Category
Meaning anchor/ /ssoociated
AINNI (Gao 2021)
NLI
premise/hypotheeis
DuReader (He et al., 2018)
Passage Retrieval
juery/passage
ELI5 (Fan et 2019)
Popular Responses
forum question/ /user response
FEVER (Thurne et al., 2018)
Fact Checking
claim/ document
HotpotQA (Yang et. 2018)
Passage Retrieval Multi-Hop QA
/parr//
Miracl (Zhang al,, 20236)
Passage Retrieval
emery/passage
MrTydi (Zhang 2021)
Passage Retrieval
query/ passaaa
MSMARCO (Bajaj al. 2018)
Passage Retrieval
query/ dlocumeett passage
Natural Questions (Kwiatkowski et al., 2019)
Passage Retrieval
query/ Wikipedia article
Quora Duplicates (DataCanary al., 2017)
Duplicates Classification
forum question forum question
SQuAD (Rajpurkar et al., 2016)
Passage Retrieval
query/ /eii///iiipiia article
T2Ranking al., 2023)
Passage Retrieval
query/ web passage
TriviaQA (Joshi et al., 2017)
Passage Retrieval
Qnery/Wikipediin article
Table 3: Instructions used for datasets contained in the public portion of the E5 dataset and the Lightblue reranker distillation mixture (LB).
Dataset
Instruction(s)
E5/NLI
Given premise, retrieve hypothesis that is entailed by the premise
Retrieve semantically similar text
E5/DuReader
Given Chinese search query, retrieve web passages answer question
ES/ELI5
Provided a user question. retrieve the highest voted answers Reddit EL15 forum Given a claim, retrieve documents that support or refute the claim Given multi-hop question, retrieve documents that help answer the question
E5/IEVER
AA/HolpotQAA
E5/MIRACL
Given a question, retrieve Wikipedia passages that answer the question Given question.
E5 35/MrTyDi
Wikipedia passages that answer the question Given web search query, retrieve relevant passages that answer the query Given weh search query, retrieve relevant documents Given question, retrieve Wikipedia passages that answer the question Given a question, retrieve questions that are semantically equivalent to the given question Find questions that have the Retrieve Wikipedia passages that answer the question Given Chinese search query, retrieve
E5/ S//SMMRRC RCO Passage
E5/ Document
answer query
ES/NQ
E5/ 25/QuaaaDpllicaes
meaning the input question
E5/SQuAD
55 15/T2Ranking
passages that the question
E5 //TriviaAA
Retrieve Wikipedia passages that answer the question
LB/cpgqa
Given question about clinical practice guidelines, retrieve relevant passages that answer the question
LB/logqa
Given question about a system's 's log, retrieve the most relevant log
LB/ B/lsat
Retrieve the passage which is most relevant tu the given LSAT question most relevant the given question Given a biomedical research question, retrieve relevant passages that answer it
L.B narrativegaa
Retrieve the story
LB/pubmedoo
LB/ LB/apsss
Given a sports question, retrieve relevant passages that answer the question
Training Details. We employ the Adam optimizer (Kingma & Ba, 2015) with B1 = 0.9, 62 = 0.98. We apply no weight decay. For the learning rate scheduler, we adopt a linear warmup over 5% of the total training steps, followed by linear decay till end of training. We train all models for a constant of 4 epochs. The logit scale a is tuned and we find the value 20 to work well, which corresponds to temperature of 0.05. In all experiments,
17

---

Published as a conference paper at COLM 2025
we make use of in-batch negatives, except for Margin MSE where we observed that the performance drops. We implement mask-conditioned sampling for batching, in which all the samples in batch belong to the same task. Doing so improves the quality of in-batch negatives as all the associated text belong to the same domain. The learning rate is initially set to base batch size of 16 and then scaled according to the square root of the ratio between the total batch size across devices and the base batch size (Malladi et al., 2022), expressed
as / / lotal.batcc.size We apply gradient checkpointing. For our experiments, we use batch size 256 for all models, except some ablation where we use 128 for GPU economy. Context length is set to 8192 for all models, except 3B models where it is set at 4096 for memory economy. For similar reasons, we use LORA parameter efficient finetuning with 32 ranks for those models.
Model Selection. For model selection, we derive a validation score by using a subset of tasks from BEIR, which we call BEIR (short) throughout the paper. In particular, we take the DBPedia, HotpotQA, FiQA2018, FEVER, QuoraRetrieval 256 queries and up to 131,072 passages from each of them. We do this because evaluating many retrieval tasks is time-consuming due to their large corpus size, often in the millions of documents. We periodically evaluate the performance of the sentence embedders that we train on the validation set, by computing the average NDCG@10 score over our validation tasks, and we select the best model that occurs during a training trial. Similarly, we use the same validation score to perform hyperparameter search and analysis.
NQ tasks, and we subsample
LB, E5, BEIR and MMTEB instructions. As we have mentioned at Section 5, for some trials we use the public portion of the E5 dataset, reproduced by Springer et al. (2024), for training our sentence embedders. In Table 2, we describe the composition of training sets and their corresponding tasks. In addition to comprising public permissible datasets, this training data mix has minimal overlap with MTEB tasks, enabling us to evaluate the robustness of our models domains beyond the training set. Training encoders can leverage an extra conditioning signal to improve generalization across tasks (Su et al., 2023). This conditioning signals comes in the form of a task-specific instruction, which can be combined with the anchor sentence, and potentially with its associated sentences. In Section 6, we provide the list of instructions we used to augment the anchor texts, and also their associated text in the case of symmetric tasks. We use the same task instructions as previous works (Wang et al., 2023; Springer et al., 2024; BehnamGhader et al., 2024).
Similarly for the Lightblue (LB) distillation dataset (Lightblue, 2025), each row of text pairs originates from multilingual QA dataset. The dataset and its composition is described in detail in Section 6. For some of the datasets contained, we define a task instruction that reflects the domain of the query contained in a row of the training dataset. If specific task instruction has not been defined for dataset, we fallback to default one: "Retrieve the most relevant passages to the given query"" In Section 6, we provide the list of instructions we used for some of the datasets in the Lightblue data mix.
During inference, we allow a user to provide an instruction of their liking. For benchmarks, like BEIR or MMTEB, use the predefined set of task instruction provided by the software package for each of the tasks contained. Please refer to the MMTEB benchmark paper for more details (Enevoldsen et al., 2025). For TREC-DL evaluations, we use the instruction corresponding to MSMARCO: "Given a web search query, retrieve relevant passages that answer the query".
rankers. In Figure 2, we evaluate rankers in a 100kwqrel subset of TREC-DL 19-23 benchmark we have collected. For ranking with graded relevance (0-3), we apply the following system prompt: "Your task is to judge how well the passage answers the query on a scale from 0 to 3. 0- Irrelevant, 1 Relevant lopic, but does not contain the answer, 2 - Highll relevant, partial or unclear answer, and 3 Perfectly relevant, exact answer. Answer with 0, 1, or 3.". For the binary relevance evaluation (0-1), we apply the following instruction: "Your task is to judge how well the passage answers the query. 0- Irrelevant, Perfectly relevant, exact answer. Answer with O, or 1.".
System prompts for zero-shot QWEN2.5-32B-INSTER against zero-shot DWEN2.5-32B-INSTRRU
18

---

Published as a conference paper at COLM 2025
73.2
Average nDCG@10
42.0
@
73.0
Average nDCG
41.0
72.8
72.5
40.0
72.2
39.0
72.0
64
128
256
512
5e-05
0.0001
0.0005
0.001
0.005
0.01
0.05
0.1
0.5
Total Batch Size
Logit Bias Learning Rate
BiXSE
InfoNCE
Figure 5: Batch size and logit bias
Extended Study
BIXSE InfoNCE and weakly-supervised dataset noise. We hypothesize that BCE achieves better robustness to ceakly-supervised dataset noise, because the effect noise more diluted in the gradients of the BCE objective. Suppose the binary label case with hard negatives. At each training step the model encounters a batch of size B, the BCE-loss then is the average of losses from 2B binary classification predictions, while softmax-based loss is the average of losses from B multi-class classification predictions. Assuming that the in-batch negatives are indeed true negatives: If one pair of texts in the batch is mislabeled, then for the softmax-loss this means that the predictions is optimized towards false target, whereas for the BCE-loss g predictions is optimized towards false target.
BIXSE vs InfoNCE and in-batch negatives. We evaluate QwEN2.5-0.5B-INSTE models by monitoring the average nDCG@10 BEIR how performance changes the total training batch size increases. For both InfoNCE and BIXSE, we use in-batch negatives, so larger batches provide more negative examples per query. As shown in Figure 5 (left), InfoNCE benefits noticeably from increasing batch size, consistent with its reliance on strong negative contrast. BIXSE, by contrast, achieves strong performance even at smaller batch sizes and shows only modest gains with larger ones. This suggests that BIXSE is less dependent on large quantities of in-batch negatives.
We can get intuition about the batch size resilience of BCE losses by analyzing loss gradients. In softmax-based based contrastive learning, the denominator, in which the negative pairs appear, can be expressed as log-sum-exp. This means that each of the (Nx, yn) pairs' gradient contribution is weighted by each pair's probability to have Yn classified from X,.
a
M
exp (5(Xx, yn))
log dyn
Σ exp (s(A). ym)) ΣM exp (s(*;, ym)) dyn s(xx,
#==1
Combining this with large logit scales a (10, 100), that are used in order achieve good downstream performance empirically, we can see that the negative pair with the largest logit s (m²) contributes almost all of the gradient, while the rest of the negative pairs with smaller logits sX ym) S s (x Vmr) have significantly smaller gradient contributions. In contrast, BCE loss weights all negative pairs independently from one another.
a 5xxx yxx
log o (-5((x
(s(xx,)n))
dyn
Nyn
Combining this with large logit scales E (10,100) once more, we can see that only the negative pairs that are perceived as positive by the model are going to be used and they will
19

---

Published as a conference paper at COLM 2025
Benchmark
BIXSE (OURS)
PAIRWISE BCE
LAMBDA NDCG V1
LAMBDA NDCG V2
BEIR (short)
73.91
72.17
72.43
73.78
BEIR (full 15-task)
45.05
43.30
41.40
44.68
MTEB (Multilingul v1)
55.46
53.50
53,46
55.62
TREC (qrels 19-23)
57.89
55.74
55.25
36,39
TREC 2021 (top-100 docs)
66.55
65.02
65.04
64.84
TREC 2022 (top-100 docs)
36.85
37.58
37.68
37.25
TREC 2023 (top-100 docs)
38.04
37.29
38.72
38.23
Table 9: NDCG@10 for LightBlue-trained models (in-batch only, batch size 256) across standard retrieval benchmarks. Bold indicates the best score per benchmark.
have approximately equal weight ( 1) among themselves. We argue that this makes the BCE formulation use in-batch negatives more efficiently than softmax-based formulations, as it utillzess simultaneously all erroneous predictions about negative pairs instead of the most erroneous one each training step.
Effect of logit bias learning rate. We further investigate the role of the logit bias & by varying its learning rate while keeping the rest of the model fixed. As shown in Figure (right) by the average nDCG@10 on BEIR (short), performance improves when the logit bias trained with higher learning rates. For reference, we use learning rate of R: 0.0001 for S.EEN..5-0.BBINS.
base models at batch size 256. The results support our design choice: a fast-updating & can quickly adapt to and absorb marginal label imbalance, particu- larly that induced by in-batch negatives, allowing the encoder to focus modeling actual query-document relevance. Performance drops when B is under-optimized, indicating that insufficient correction for label skew reduces the effectiveness of in-batch negatives.
Analytical Report against InfoNCE
Comparative Evaluation Against Pairwise Ranking Objectives
To assess BIXSE's competitiveness relative to traditional pairwise losses, we conducted extensive experiments comparing it to three strong baselines: Pairwise BCE (as used in RankNet (Burges et al., 2005) and PairDistill (Huang & Chen, 2024)), LambdaLoss (Wang et al., 2018) with two NDCG weighting variants, and Margin (Hofstätter al., 2021). These experiments were carried out on two diverse datasets, LightBlue (multilingual, in- batch only) and BGE-M3 (Chen et al., 2024) training datasets (English only subset, with hard negatives), using the QWEN2.5-0.55 5..55B---BB-- 1 model architecture. For fair comparison, we run hyperparameter search for all methods to tune for learning rates and logit scales, as well as hyperparameters specific to each training loss.
Multilingual with LLM-graded relevance annotations As shown in Table and Table 9, BIXSE clearly outperforms InfoNCE and delivers stronger NDCG@10 scores than pairwise alternatives across all evaluation suites, including BEIR (short), BEIR (full), MTEB (Multilingual v1), and TREC-DL 2021-2023 (where use the top-100 doc- ument lists provided by the official competitions). LightBlue contains only one graded document per query,
LightBlue Results:
make use of in-batch negatives for all training methods. We train for epoch with batch size 256.
BGE-M3 Results: English data with mined and scored hard-negatives To investigate training efficiency under fixed compute and memory budgets, we evaluate model perfor- mance across configurations that varied the number of hard negatives and batch size. As we see in Tables 11 and 13, BIXSE benefits greatly from scaling batch size and leveraging in-batch negatives, while pairwise losses require multiple explicitly annotated negatives per query.
These results highlight the core advantage of BIXSE: performance improves with batch size without requiring additional annotated negatives. In contrast, pairwise methods depend more heavily hard negatives and saturate early.
20

---

Published as a conference paper at COLM 2025
MTEB Task
BIXSE (BCE)
PAIRWISE BCE
LAMBDA NDCG
LAMBDA NDCG v2
StackOverflowQA
83.60
85.45
88.06
TwitterFlerneRettivvll
50,70
47.04
46.59
52.36
AILAStatutes
24.14
21 42
21.50
27 27.42
ArguAna
52.51
52.10
54.18
HagridRetrieval
98.90
98.75
98.50
98.75
EggallenchCCorprratee obbying
93.39
92.16
92.09
92.78
SMMBPasskeRRetriee
84.75
79.25
84.00
84.50
SCIDOCS
16.12
17.71
16.61
17.42
SpartQA
6.05
8.15
3.92
11.77
TempReasonLl
1.92
1.57
1.64
1.40
TRECCOVID
72.12
72.30
58.82
70,88
WinoGrande
42.77
43.02
45.39
40.51
Belebele Retrieval
57.76
56.01
56.65
57
MLQARetrieval
70,68
66.84
68.35
69.64
23,73
21.29
23.38
25.29
86.34
79.68
85,42
84.48
CovidRetrieval
80.14
76.95
80. 18
79.90
44.91
41.80
Table 10: Analytic breakdown of NDCG@10 on multilingual MTEB tasks for LightBlue- trained models. Each column corresponds to different training objective: BIXSÉ (BCE), PAIRWISE BCE, LAMBDALOSS (NDCG v1), and LAMBDALOSS (NDCG v2).
# Hard Negatives/ Batch Size
BIXSE (BCE)
PAIRWISE BCE
LAMBDALOSS (NDCG V2)
15 / 16
43.63
41.68
49.84
7 7 32
44.95
44.04
50.55
3 / 64
46.29
45.82
51.82
1/ 128
45.92
51.06
50.84
0 256
48.88
45.66
45.49
Table 11: Summary of model performance (average NDCG@10) the 15-task BEIR bench- mark for models trained on the BGE dataset.
Efficiency and Scalability Considerations BIXSE offers compelling annotation and train- ing cost benefits:
Annotation Cost: BiXSE achieves competitive performance using a single (graded)
relevance label per query. Pairwise methods require at least four to eight super- vised comparisons in order to perform optimally, increasing the cost when using expensive teacher models (e.g., LLMs cross-encoders).
Training Cost: BIXSE's pointwise BCE loss scales quadratically with batch size (O(B2)), while pairwise losses scale cubically (O(B²) when using all document pairs. More importantly, the memory required to store pairwise score tensors grows rapidly and becomes impractical for large batches. For instance, storing score tensors for a batch of 16,384 using LambdaLoss exceeds 134 GB, while BIXSE only requires 134 MB.
We acknowledge the strength of LambdaLoss nDCG-v2 described by Wang et al. (2018). performs close or better to BIXSE on several benchmarks, particularly when provided with ample labeled negatives. However, it lacks the same degree of annotation and memory efficiency. Importantly, BIXSE achieves similar or better downstream performance without compromising scale.
21

---

Published as a conference paper at COLM 2025
BEIR Task
BIXSE (BCE)
PAIRWISE BCE
LAMBDALOSS (NDCG v2)
TREC-COVID
53.4
50.2
57.1
BioASQ
45.2
42.9
46.8
NFCorpus
38.1
36.5
40.2
NQ
49.8
48.7
52.6
HotpotQA
62.4
60.1
64.0
FiQA-2018
34.9
35.5
38.4
ArguAna
30.6
28.9
34.1
Touché-2020
28.3
26.8
31.7
Quora
89.3
91.2
90.8
DBPedia
42.6
41.3
45.1
SCIDOCS
18.4
17.7
20.5
FEVER
72.9
70.1
75.4
Climate-FEVER
24.7
23.9
27.3
SciFact
59.3
57.2
60.9
CQADupStack 32.1
31.6
34.0
Table 12: Analytic breakdown of model performance (NDCG@10) across individual tasks in
BEIR. Results shown for the best-performing configuration per loss type among BGE-trained models.
# Hard Negatives Batch Size
BIXSE (BCE)
PAIRWISE BCE
LAMBDALOSS (NDCG V2)
15 / 16
44.09
40.89
48.42
7 32
45.45
45.62
49.92
3
46.60
44.05
51.91
1 // 128
46.50
49.38
49.49
0 256
50.62
46.46
47.73
Table 13: Summary of model performance (average NDCG@10 score) across MTEB v2 English retrieval tasks) for BGE-trained models. Results are grouped by number of hard negatives and batch size.
MTEB Task
BIXSE (BCE)
PAIRWISE BCE
LAMBDALOSs (NDCG v2)
ArguAna
65.02
68.29
70.98
COADDpstacKGaminnReeivvelllll
54.32
54.47
54.09
CQADupstackUmixxetries
39.99
42.25
41.34
ClimateFEVER
26.88
23.86
27.06
FEVER
75.90
78.87
71.98
FiQA2018
46.40
39.65
46.40
HotpotQA
64.42
61.92
64.42
SCIDOCS
22.24
19.86
22.24
TRECCOVID
73,29
74.73
76.04
Touche202URetrieval..3
54.96
54.99
54.99
Table 14: Analytic breakdown of NDCG@10 for individual retrieval tasks in MTEB (English
v2) for BGE-trained models. Each column reports the best-performing configuration for that loss.
22


Reviews:
[{"review_id": "vlRCdsclDv", "reviewer": "Reviewer_t3xV", "summary": "The authors identify the issues of requiring hard binary decisions for many tasks during LLM learning.\nThey introduce a method of training encoders using graded relevance scores from humans. This approach aligns training more closely with human preferences compared to the hard binary decisions typically used.\nAuthors validate their results on a huge data set of TREC relevance judgements.\nThe proposed methods significantly out performed the baselines.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The collection of graded relevance scores from humans collected from the TREC-DL is a probably the largest possible collection available.\nAuthor results clearly demonstrate the claims strengths. \n\nThe paper is well-written and easy to follow. The motivation, methodology, and results are clearly presented.", "reasons_to_reject": "The authors did not discuss any of the timing results of their experiments, such as training time or inference time, which are crucial for understanding the computational efficiency of their approach.\nPresumably, there is a significant computational penalty.\nMore quantitative analysis of computational overhead (e.g., latency, memory usage, FLOPs, GPU utilization) would be helpful, especially for large-scale retrieval scenarios.\n\nThe authors should discuss trade-offs involved in computational encoding.\nInclude more detailed analysis of computational costs, especially for large-scale deployments with and without batches.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EqNwbfRVws", "reviewer": "Reviewer_Dk2A", "summary": "This paper introduces BiXSE, a new training objective for dense retrieval and sentence embedding models that leverages graded relevance labels distilled from large language models. Rather than relying on softmax-based contrastive losses like InfoNCE, BiXSE uses binary cross-entropy to predict relevance probabilities between query-document pairs. This approach supports both binary and graded supervision and is shown to be more robust to label noise, scalable across model sizes and languages, and more data-efficient by avoiding over-filtering. Empirical results across TREC-DL, BEIR, and MTEB benchmarks show consistent performance gains over InfoNCE and other recent training objectives.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "BiXSE offers a practical and theoretically grounded solution for learning from LLM-generated graded supervision, addressing limitations of binary labeling in retrieval. It supports a backward-compatible and scalable training paradigm with minimal changes to model architecture or training infrastructure. The BCE formulation is shown to be robust to label noise, less dependent on large batch sizes or heavy data filtering, and more effective at distilling signals from high-resource LLM rankers. Extensive experiments across benchmarks and model backbones show clear and consistent improvements over InfoNCE and alternative methods, validating the core hypothesis that graded relevance supervision can be leveraged more effectively via a pointwise BCE loss.", "reasons_to_reject": "Despite strong empirical results, the method still relies on graded relevance annotations, which require access to powerful LLM rankers or curated datasets. It remains unclear how well BiXSE generalizes when LLM-generated labels are inconsistent or noisy across domains. The method’s reliance on a logit bias term and tuning its learning rate introduces another layer of optimization complexity. While the approach is positioned as a general-purpose alternative to contrastive learning, experiments are focused only on IR and sentence embedding tasks; the extent to which it transfers to tasks like multi-hop QA or dialog retrieval is unknown. Finally, some theoretical intuitions about robustness and BCE gradient behavior are described but not formally analyzed.", "questions_to_authors": "- Can BiXSE be extended to support ranking-based losses (e.g., listwise or pairwise) that also leverage graded labels while retaining robustness?\n- Have you tested how LLM scoring variability (e.g., using GPT-4 vs. PaLM) affects training stability or generalization under BiXSE?\n- Would it be possible to use BiXSE with only ordinal labels (e.g., 0–3) without normalizing to [0, 1] probabilities? Would that be more stable?\n- Could you elaborate more on how task-conditioned batching interacts with graded relevance—are there edge cases where this batching hurts?\n- Given the observed robustness of BiXSE to label noise, have you considered its potential as a student objective in semi-supervised settings with pseudo-labels?\n- In settings where graded labels are not available, does BiXSE degrade gracefully to binary data? How would you recommend adapting it to low-resource scenarios?", "extra_scores": {}}, {"review_id": "fuYnlQtX9t", "reviewer": "Reviewer_CF35", "summary": "The paper propose optimizing a two-tower retrieval model with binary cross-entropy loss using graded relevance scores. The paper argues that the point-wise approach is more token efficient under the LLM as a scorer framework.\n\n***Quality and clarity:\nThe paper is written clearly and extensive experiments are done to confirm the usefulness of their method.\n\n***Originality and significance:\nThe use of cross-entropy loss for ranking problem has been explored extensively, as early as the work of RankNet [1].\n\nThe use of graded relevance score has also been explored (RabkDistil or [2])\n\nI wonder if it is fair to argue that the proposed approach is more efficient than previous distillation work. For example, the RankDistil (Reddi et al 2021) work include a pairwise approach where the label is assumed to be given to each pair (q_i, d_i). It seems that RankDistil would be a reasonable baseline. Another baseline is the pairwise derivation of the probability in cross-entropy loss, as implemented in [1].\n\nCitations:\n[1] Learning to Rank using Gradient Descent (Burges et al. 2015)\n[2] PairDistill: Pairwise Relevance Distillation for Dense Retrieval (Huang et al. 2024)", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "It is interesting to explore different ranking loss for distilled from graded relevance scores.", "reasons_to_reject": "The originality and how the author argues for it. As listed above, continuous relevance score as distillation target has been explored and the argument that pointwise loss is more data efficient is not necessary true. Pairwise approach can depend on the diff between a pair of (q_i, d_i) and (q_j, d_j) with only requires label for (q, d) pairs where the index are matching.", "questions_to_authors": "Typo: line 173 (Section 4) \"?\" before the reference to the citation.", "extra_scores": {}}, {"review_id": "A5ZdWTS3ma", "reviewer": "Reviewer_odwD", "summary": "This paper asserts that embedding models used for retrieval are typically trained with binary relevance labels and argues that dense retrieval models should instead be trained using graded relevance labels. The paper proposes an adaptation of a binary cross-entropy loss for use with graded relevance labels, which are converted to scores before being plugged into the loss. The graded relevance labels are generated using Qwen2.5 32B. In evaluations comparing this proposed loss to InfoNCE, the proposed loss performs better, though other relevant comparison points are not considered. In the discussion period, authors reported new results including important baselines like LambdaLoss and stated they would include these new results in the manuscript.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "It is interesting to see an investigation into how LLM-provided graded relevance labels can be used to improve performance. Results provided in the discussion period substantially strengthen the submitted manuscript's argument that these are an effective way to train a model, which is a nice finding that may remove the need for the common practice of cross-encoder score distillation.", "reasons_to_reject": "As mentioned in this work, it is common to train embedding models for retrieval using distillation from a cross-encoder teacher. It is not really the case the training on binary relevance labels is the typical scenario, as the introduction implies. For example, prominent methods like ColBERTv2 were trained using distillation.\n\nMore importantly, it is not the case that the literature lacks approaches for training on graded relevance labels. This was a key focus of learning-to-rank methods for many years, and there are even mature libraries implementing them (e.g., see https://github.com/tensorflow/ranking and https://rax.readthedocs.io/en/stable/api.html). Reasonable and clear baselines like LambdaLoss or a margin loss (between different relevance grades) are not considered in this work (with the exception of MarginMSE, which is a margin loss adapted to use scores from the proposed approach rather than graded relevance labels). While the work attempts to argue that only pointwise losses should be considered, a pairwise loss is not less efficient than using a pointwise loss with both negative and positive documents, as is done in this work. A pairwise loss like LambdaLoss can even be used to optimize a ranking metric directly (see: The LambdaLoss Framework for Ranking Metric Optimization, Wang et al). A loss like this is a clear baseline, as is cross-encoder distillation.\n\nIn the discussion period, the authors provided new results addressing these concerns and stated they would update their paper accordingly. This review score was significantly increased in response.\n\nFurthermore, the BEIR results reported in Table 1 do not appear strong compared to even smaller models like SPLADEv3, but it is not clear exactly what BEIR datasets are being used here. It would be nice to see separate metrics for TREC-DL 19-23 in order to make comparisons with prior work.", "questions_to_authors": "What BEIR subset are you using? Is this the standard 13 benchmark subset (e.g. used in SPLADEv3) or another subset?", "extra_scores": {}}]

Author rebuttal:

Meta-review: **Summary**

This paper addresses a key limitation in training dense retrieval and embedding models which over rely on binary relevance labels. The authors propose BiXSE to effectively incorporate graded relevance scores, potentially derived from human annotations or LLMs. This approach allows the model to better capture fine-grained preference signals and demonstrates improved robustness to label noise, data efficiency, and scalability across model sizes and languages.

**Clarity**

The paper is clearly written and well-organized. The authors support their claims with thorough experimental validation across a range of benchmarks, including TREC-DL, BEIR, and MTEB. The inclusion of new baselines during the discussion period (e.g., LambdaLoss) further strengthens the paper.

**Novelty**

While the idea of using cross-entropy loss for ranking is not novel (e.g., RankNet), and graded supervision has been explored in previous work (e.g., RankDistil) mentioned by some reviewers, the paper presents a practical and well-integrated formulation that adapts these ideas to the context of modern dense retrieval models and LLM-generated supervision. The proposed method offers a distinct contribution in its simplicity, generality, and empirical gains, even if it builds on familiar components.

**Pros**
- A practical training objective that supports both binary and graded supervision.
- Demonstrated improvements across multiple datasets and benchmarks.
- Clear exposition and strong empirical validation.
- Addresses an important pain point in retrieval model training (binary vs. graded relevance).

**Cons**
- The novelty is moderate; key ideas have some precedent in prior work (e.g., RankNet, RankDistil).
- Some initially missing baselines (e.g., pairwise methods) were only added later in the rebuttal.
- The claim of greater efficiency over prior distillation approaches could be more thoroughly analyzed.

**Recommendation**

All reviewers rated the paper between 6 and 7 (6, 7, 7, 7) and reached a consensus that the paper is solid and worthy of acceptance. 
Despite concerns regarding novelty, the practical utility, empirical strength, and methodological clarity of the proposed approach justify inclusion in the conference.
Decision: Accept