Towards Trustworthy Reranking: A Simple yet Effective Abstention Mechanism
Authors: Hippolyte Gisserot-Boukhlef
Manuel Faysse
Emmanuel Malherbe
CELINE HUDELOT
Pierre Colombo
Venue: Accepted by TMLR ()

Abstract: Neural Information Retrieval (NIR) has significantly improved upon heuristic-based Information Retrieval (IR) systems. Yet, failures remain frequent, the models used often being unable to retrieve documents relevant to the user's query. We address this challenge by proposing a lightweight abstention mechanism tailored for real-world constraints, with particular emphasis placed on the reranking phase. We introduce a protocol for evaluating abstention strategies in black-box scenarios (typically encountered when relying on API services), demonstrating their efficacy, and propose a simple yet effective data-driven mechanism. We provide open-source code for experiment replication and abstention implementation, fostering wider adoption and application in diverse contexts.

Published in Transactions on Machine Learning Research (09/2024)
Towards Trustworthy Reranking: A Simple yet Effective Abstention Mechanism
Hippolyte Gisserot-Boukhlet
Artefact Research Center
MICS, CentruleSupétee, Universilé Paris-Suclay
Manuel Faysse
Illin Technology
MICS, CentroleSupéles, Université Paris-Soclay
Emmanuel Malherbe
Artefact Reseorch Center
Céline Hudelot MICS, CentraleSupéles, Université Paris-Saclay
Pierre Colombo
pierre ei
Equall.ai ai MICS, CentroleSupékee, Université Paris-Soclay
Reviewed on OpenReview; https: www.revee.........ooeoe net/ per/forum? id- ENKUMWfRIJ
Abstract
Neural Information Retrieval (NIR) has significantly improved upon heuristic-based Infor- mation Retrieval (IR) systems. Yet, failures remain frequent, the models used often being unable to retrieve documents relevant to the user's query. We address this challenge by proposing a lightweight abstention mechanism tailored for real-world constraints, with par- licular emphasis placed on the reranking phase. We introduce protocol for evaluating abstention strategies in black-box scenarios (typically encountered when relying on AP1 services), demonstrating their efficacy, and propose simple yet effective data-driven mech- anism. We provide open-source code for experiment replication and abstention implemen- tation, fostering wider adoption and application in diverse contexts.
1 Introduction
In recent years, NIR has emerged as promising approach to addressing the challenges of IR on various tasks (Guo et al. 2016 Mitra & Craswell 2017 Zhao al. 2022). Central to the NIR paradigm are the pivotal stages of retricval and reranking (Robertson et al. 1995 Ni et al. 2021 Neelakantan et al. 2022 which collectively play a fundamental role in shaping the performance and outcomes of IR systems (Thakur
le al. )021). While the objective of the retrieval stage is to efficiently fetch candidate documents from a vast corpus based on a user's query (frequentist. (Ramos et al. 2003 Robertson et al. 2009) or bi-encoder-based approaches (Karpukhin
al. 2020)), ) reranking aims at reordering these retricved documents in slower albeit more effective way, using more sophisticated techniques (bi-encoder- or cross-encoderr-bseel (Nogueira)
& Cho 2019
Khattab & Zaharia 2020))
However, despite advancements in NIR techniques, retrieval and reranking processos frequently fail, due for instance to the quality of the models being used, to ambiguous user queries or insufficient relevant. documents
The common approach have quick retriever that scales well and more computationally intensive reranker that reorders the top retrieved contexts.

---

Published in Transactions on Machine Learning Research (09/2024)
Observation
Estimation
Thresholding
User query
Confidence assessment using score-based function
ranking outputted
only confidence above threshold
Confidence
Score
Ranking
12
candidate documents
Black-box
dg ....
relevance
(2)
Abstention
assessment
(Optional) Calibration
Reference dataset (scores ground truths)
Sp
Relevance scores
Ground truths
z²
s(xt)
m(x) y')
O
s(22)
m m
Uyeg
x/
s(z")
m(2"
Figure Procodure diagram for black-box confidence estimation and abstention decision in reranking setting. In the reference-free scenario, confidence function is a simple heuristic (c.g., maximum). In the data-driven scenario, light. non-trivial function of the relevance scores (e-g., learned linear combination).
in the database. These can have unsuitable consequences, particularly in contexts like Netrieval-Augmentee Generation (RAG) (Lewis al. 2020), where retrieved information is directly used in downstream tasks,
thus undermining the overall system reliability (Yoran et al.) 2023 Wang al. 2023: Back et al. 2023
Recognizing the critical need to address these challenges, the implementation of abstention mechanisms emerges as a significant promise within the IR landscape. This technique offers a pragmatic approach by abstaining from delivering results when the model shows signs of uncertainty, thus ensuring users are not misled by potentially erroneous information. Although the subject of abstention in machine learning has long been of interest (Chow 1957 El- Yaniv & Wiener 2010 Geifman & El- Yaniv) 2017 2019), most research has focused on the classification setting, leaving lack of significant initiatives
IR. A few related works
propose computationally intensive learning strategies (Cheng et al. 2010; Mao et al. 2023 that are difficult to implement in the case of NIR, where models can have up to billions of parameters.
Scope. In this paper, we explore lightweight abstention mechanisms that adhere to realistic industrial constraints, namely (i) black-box access-only to query-documents relevance scores, which is typically the case when relying on APT services, (ii) marginal computational and latency overhead, (iii) configurable abstention rates depending on the application. In line with industrial applications of IR systems for RAG search engines, our method focuses on assessing the reranking quality of the top retrieved documents.
Contributions. Our key contributions are:
1. We dcvise a detailed protocol for evaluating abstention strategies in black-box reranking sctting and demonstrate their relevance.
2. We introduce simple yet effective reference-bere based abstention mechanism, outperforming reference- free heuristics at zero incurred cost. We perform ablations and analyses on this method and are confident about its potential in practical settings.
3. We release a code package and artifacts" to enable full replication of our experiments and the imple- mentation of plug-and-playy abstention mechanisms for any use case.
httte///thhaammmnnnrereerhhhhhen under
under MIT license.
2

---

Published in Transactions on Machine Learning Research (09/2024)
2 Problem Statement & Related Work
2.1 Notations
General notations. Let V denote the vocabulary and V the set of all possible textual inputs (Kleene closure of V). Let Q C V be the set of queries and D C V the document database. In the reranking setting, a query q E Q associated with & E candidate documents (d1, .dk) E DD coming from the preceding retrieval phase, some of them being relevant to the query and the others not. When available, we denole by y {0, 13 the ground truth vector, such that y/j
dj is relevant to q and yj otherwise
Dataset. When specified, we have access to a labeled reference dataset 8 composed of N N instances, cach of them comprising of a query and k candidate documents, (g. di, ;d,) E Q x D and ground truth y E (0, 13* Formally,
= (=' (
Relevance scoring. A crucial step in the reranking task is the calculation of query-document relevance scores. We denote by fo Q D + R the encoder-based relevance function that maps query-document pair to its corresponding relevance score, with O E θ standing for the encoder parameters. We also define 90 Q D + R², the function taking a query and & candidate documents as input and returning the corresponding vector of relevance scores. Formallly given I E 2 DK
go (x) = (Jo (g. d ... so (q, d+)).
Additionally, we define the sort function s: R + R that takes vector of scores as input and returns its sorted version in increasing order. For a given z E TR&,, s(z) is such that 81 S ... S 8% (z) We denote by Sq : Q x D* + IR* the function that takes a query and k candidate documents and directly returns the sorted relevance scores, i.c.,
80 9
Document ranking. Once the scores are computed, we have a notion of the relevance level of each document with respect to the query. The idea is then to rank the documents by relevance score. For a given vector of scores z E R², let r(z) E 65 denote the corresponding vector of positions the ascending sort, s(z), where 6 is the symmetric group of k elements comprising of the k! permutations of {1,-.. .k]. In other terms, rz(z) j if 2, is ranked jth in the ascending sort of z. We then define the full reranking function Q x D + 6R, taking a query and k candidate documents input and returning the corresponding ranking by relevance score. More formally,
T8 = go
(4)
Instance-wise metric. Finally, when ground truth vector y is available, it is possible to evaluate the quality of the ranking generated by те on given query-documents tuple x E Q DF. Therefore, we denote by m : & x 13 + R the evaluation function that takes a vector of scores so(x) and ground truth vector as input and returns the corresponding metric value. Without any lack of generality, we assume that m increases with the ranking quality ("higher is better").
2.2 Abstention in Reranking
In this work, our goal is to design an abstention mechanism built directly on top of reranker re- We therefore aim to find a confidence function : Q x D + R and a threshold 7 E R such that ranking re(x) is outputted for given query-documents tuple x E Q x D* if and only c(x) > r. Formally, we want to come up with
function pe (2 x D*) E G U {1} such that for given x and T,
(ro(z), if (f)
po(x, T) [1,
(5)
otherwise
"Note that y not one-hot vector as several documents can be relevant to single query. Ssy(M) (x) denotes the jth element of s(x).
3

---

Published in Transactions on Machine Learning Research (09/2024)
where 1 denotes the abstention decision.
Requirements. In compliance with the set of constraints defined in the introduction (Section must be usable in a fully black-box setting and not incur significant extra computational costs, either during training or at inference time. Formally, we set the two following requirements:
1. We focus the case in which the encoder parameters 0 are not learnable.
2. Furthermore, must only take relevance scores as input, i.e,, {u go u: :R 11*++1 +R]
2.3 Related Work
Abstention, also known selective prediction, has been around for long time, originally for classification purposes (Chow 1957 El-Yaniv & Wiener 2010), and has often been associated with topics such as confi- dence estimation, out-of-domain (OOD) detection (Schölkopf et al. 1999 Liang et al.) 2017 Darrin et al, 2022 2024 and prediction error detection (Hendrycks & Gimpel 2016). These fields collectively encom- pass three main approaches: learning- based methods, which require a dedicated training process; white- box methods, which rely on full access to the model's internal features; and black-box methods, which focus solely on the model's output.
Among learning-basee methods, the most classic initiatives include algorithms such as SVMs, nearest neigh- bors, boosting (Hellman 1970 Fumera & Roli
2002 Wiener & E1- Yaniv, 2015 Cortes et al. 2016 or
Bayesian methods (Blundell et al. 2015) such as Markov Chain Monte Carlo [Geyer 1992) and Variational
Inference (Hinton & Van Camp
1993 Graves 2011), that include uncertainty estimation as a whole part of the training process. Similarly, aleatoric uncertainty estimation requires the use of a log-likelihood loss function (Loquercio et al. (020), thus greatly modifying the training process. Some approaches also include specific regularizers in the loss function (Xin al.
2021
Zeng al. 2021), while others incorporate ab-
stention a class during training (Rajpurkar et al. 2018 most of the time with certain cost associated
(Batllttt & Wegkamp 2008 El-Yaniv & Wiener 2010 Cortes et al.
2016 Geifman & El- Yaniv, 2019). It
comes obvious that none of these methods align with the no-training requirement set in Section 2.2
Most white-box methods, on the other hand, do not require specific training. When considering selective prediction in neural networks, a straightforward and effective technique to set a threshold over a confidence score derived from pre-trained network, which was already optimized to predict all points (Cordella et al. 1995 De Stefano et al.) 2000 Wiener & El- 2012). Monte Carlo dropout is another popular approach involving randomly deactivating neurons of model at inference time to estimate its epistemic uncertainty
(Houlsby et al.; 2001 Gal et al., 2017 Smith & Gal 2018). Others rely on statistical analysis of the
model's features (hidden layers) [ee et al. 2018 Ren et al., 2021 Podolskiy et al., 2021 Haroush et al.
2021 Gomes et al., 2022) while ensemble-based methods (Gal & Ghahramani] 2016 Lakshninarayanan
2019 exlend this idea and estimate confidence based on statistics of the ensemble model's features. The latter approaches might significantly increase inference time, which is not in line with our set of constraints (Section 2.2p. Moreover, white-box access to the model cannot always be guaranteed in practical situations (c.g., API services).
et al. 2017 Geilman & Yaniv
Black-box approaches assume only access to the output layer of the classifier, i.e., the soft probabilities or logits. Some works estimate uncertainty by using the probability of the mode (Hendrycks & Gimpel, et al. 2023 Hein et al. 2019 Hsu et al. 2020 Wang et al.] (023) while others utilize the entire logit, distribution (Liu et al. 2020), the most widely known and straightforward confidence calibration technique remaining temperature scaling (Platt et al., 1999). However, is unclear how to apply these frameworks the ranking scenario, where relevance scores are neither soft probabilities nor logits but unscaled scalars.
2016
Selective prediction and confidence estimation in NLP have seen targeted applications but have room for ex- pansion across broader scenarios. Dong el al. 2018) describe a model developed to estimate confidence specif- ically for semantic parsing, while Kamath et al. (2020) delve into selective prediction for distribution (OOD) question answering, allowing the model to abstain from answering questions doemed too difficult or outside the training distribution. More recently, Xin et al. (2021) have applied selective prediction through loss regularization in various classification settings. Additionally, datasets such as SQuAD2.0 (Rajpurkar
4

---

Published in Transactions on Machine Learning Research (09/2024)
et al. 2018) illustrate the challenge of unanswerable questions. In the IR field specifically, strategies like those proposed by Cheng et al. (2010) and Mao et al. (2023) allow model to abstain at certain cost during training by assessing the confidence level on partial ranks. Here again, such methods do not fit the black-box constraint mentioned Section 2.2
3 Confidence Assessment for Document Reranking
In this paper, as stated Section 2.2 the confidence scorer we build only takes into account the list of relevance scores. Our intuition that, by analyzing the distribution of the ranker's scores, it should be possible to estimate its confidence
given instance. In the next two subsections, we describe two methods for calculating the confidence score: the first one is reference-free, meaning it does not require any external data, while the second one relies access to reference data for calibration purposes.
3.1 Reference- Scenario
First, we present the reference-free approach, which can be divided into three stages (Figure
1. Observation. We first observe the relevance scores ()) (Equation for given test instance x (q,d),.... da) E Q x D
2. Estimation. We then assess confidence using simple heuristic function Whase, based solely on these relevance scores (e.g,, maximum).
3. Thresholding. Finally, we compare the confidence score computed in step to a threshold т. If Whase (z) > T, we make a prediction using ranker ræ, otherwise, we abstain (Equation 5).
In this work, we evaluate a bunch of reference- free confidence functions, relying on simple statistics computed on the relevance scores. We focus more particularly on three functions inspired from MSP (Hendrycks & Gimpel| ) (i) the maximum relevance score Umax, (ii) the standard deviation Usdd and (iii) the difference between the first and second highest relevance scores U1_2 (Narayanan et al. 2012).
3.2 Data-Driven Scenario
In this scenario, rather than arbitrarily constructing a relevance--corr--bsee confidence scorer, consider that we have access to a reference set S = {(a (uu,,"))= } (Fquation 1 and are therefore able to evaluate ranker ro (Equation 4 on each of the labeled instances. We thus define dataset Sg:
Se ((s(()) (go(z).)) (æ,y) S).
(6)
As a reminder, (x(x) is the ascending sort query-document relevance scores (Equation 3) and m (go(x).)) is the evaluation of the ranking induced by g)( given ground truth y and metric function m.
An intuitive approach is fit simple supervised model predict ranking performance (2), given vector of sorted relevance scores (x)). In this section, develop regression- approach (Figure)
0. Calibration. We use reference set to derive So (Equation 6) and then fit a regressor lreg RR* - R on Sg. Next, we derive Useg = hneg os, the function that takes a vector of unsorted scores as input and returns the predicted ranking quality.
1. Observation. As in Section 3.1 observe the relevance scores z (1) for given test instance
2. Estimation. We then assess confidence using the useg function. Intuitively, the greater Urog(z), the more confident we are that re correctly ranks the documents with respect to the query.
3. Thresholding. As in Section 3.1 an abstention decision is finally made by comparing ureg(z) to T. Section 5.3 we propose an approach to choose т.
In this work, we use sewaarrregeesion-baeeee confidence function Win Fisher 1922] Formally, for a given unsorted vector of relevance scores z,
Win (z) = (1)1(z) (25)(2),
5

---

Published in Transactions on Machine Learning Research (09/2024)
where Su.... 3g R are the coefficients fitted on Sg. with an ly regularization parameter À 0.1 (Hoerl &
Kennard 1970
More data-driven confidence functions are explored in Appendix D
4
Experimental Setup
In this section, we propose a novel experimental setup for assessing performance of abstention mechanisms in black-box reranking setting. We present both our benchmark and evaluation metrics.
4.1 Models and Datasets
An extensive benchmark built. evaluate the abstention mechanisms presented Section We collect six open-source reranking datasets (Choest et al., 2021 in three different languages, English, French and Chinese: stectoverrLortpppusssiieeeeeaeen (Zhang et al.
2015), denoted STACKOVER-
FLOW in the experiments, astubbutuudrggeetiis---rrratatte scidocs-reranking (Cohan et al. 2020). denoted ScIDocs, fteb-ff-rerankin
Lee et al.
2015 denoted AsKUBUNTU,
awlaplloffss22 (Lefebvre-
Brossard et al. 2023). denoled ALLOPROF, CMedQAvl-reranking (Zhang el al. 2017). denoled CMEDQAvI,
and Mmarco-reranking [Bonifacio et al. 2021 denoted MMARCO.
We also collect 22 open-source models for evaluation on each of the six datasets. These include models of various sizes, bi-encoders and cross-encoders, monolingual and mul- tilingual: ember-vi (paper to be released soon), 11m-embedder Zhang et al. 2023 bge-base-en-vi..5 Dge-reraaner-base//eee 06-smal1-v2/1argo-v (Wang et al. 2022), Cemmarc-MMnniMee LE--M----------- msmarco-distie patprroooissiilbee-tttt-e
(Xiao et al. 2023), multilinguas mutiiiinggal----ma11aaaaa
ms-marc--ThyEERT--222-
Me-marro-MiniLN---655
RRemerr & Gurevych 2021), stsb-TinyBERT-
L-4, atabddistilrobetta-bess
mutti-ga-iisiiberr--ooo
multi gali--ga
all-MinilN-16-v2, all-distilroberta-dd, all-mpnet-base-v2, quora-distilrobett--tee and qul--iistilrobbrra-baes [Reimers & Gurevych 2019
In addition, cach dataset is preprocessed in such way that there only 10 candidate documents per query and maxinum of five positive documents, in order to create scenario close lo real-life reranking use cases, Moreover, to fit the black-bor setting, query-documents relevance scores are calculated upstream of the evaluation and are not used thercafter. Further details are given in appondix
4.2 Instance-Wise Metrics
In this work, we rely on three of the metrics most commonly used in the IR sctting (c.g., MTEB leaderboard (Muennighoff el al. 2022 ): Average Precision (AP), Normalized Discounted Cumulative Gain (NDCG) and
Reciprocal Rank (RR): AP (Zhu 2004) computes a proxy of the area under the precision-recall curve; NDCG (Järvelin & Kekäläinen 2002) measures the relevance of the top-ranked items by discounting those further down the list using logarithmic factor; and RR (Voorhees et al. 1999 computes the inverse rank of the first relevant element in the predicted ranking. For all three metrics, higher values indicate bet performance. When relevant, we rely on their averaged versions across multiple instances: mAP, mNDCG and mRR.
4.3 Assessing Abstention Performance
The compromise between abstention rate and performance in predictive modeling represents a delicate balance that requires careful considcration (EI Yaniv & Wiener 2010). First, we want to make sure that an increasing abstention rate implies increasing performance, otherwise the mechanism employed is ineffective or even deleterious. Secondly, we aim abstain the best possible way for every abstention rate, i.c., we want to maximize the growth of the performance-abstenti curve.
Inspired by the risk-coverage curve EEl-annyy & Wiener 2010; Geifman & El- Elniv 2017 we evaluate a abstention strategies by reporting the normalized Area Under the performancc-alstentiion Curve, where the
eScikit-lcarn implementation (Pedregosa ct al. 2011
We take the liberty evaluating models lanuaages in which they have been trained, as we posit is possible detect relevant patterns the relevance scores in any case. We discuss this assertion in Section 5.2

---

Published in Transactions on Machine Learning Research (09/2024)
x-axis corresponds to the abstention rate of the strategy and the y-axis to the achieved average performance regarding metric function Our evaluation protocol is as follows, assuming access to test set, 8":
Multi-chresholding. First of all, we evaluate abstention mechanism Pe (Equation 5 on test set S' for a list of abstention thresholds Tı <.. < Tn R. For given 7 E IR, the performance of mechanism pe on S' and its corresponding abstention rate are respectively denoted by
Prom (pe) |S
Σ
(roo()),))
(2.y)ES;
(7)
Ry (po) 11 S
S², being the set of predicted instances, i.e., = rates, (Rr, (po)) and associated performances, (Pr,. (Po));=1 and then compute the area under the performancce-asseettion curve AUCm (po) for metric function m.
= y) E S' po (z.T) #L}. We collect abstention
2. Random baseline. Second, we evaluate D, the mechanism that performs random (i.e.; inesfective) abstention. Intuitively, random abstention shows flat performance as the abstention rate increases, equal to P-xc, (po) (Equation 7
3. Oracle evaluation. Then, evaluate the oracle pt on the same task. Intuitively, the oracle has access to the test labels and can therefore select the best instances for given abstention rate. AUCm (p*) thus upper-bounds AUCm (pe).
4. Normalization. We finally compute normalized AUC. Formally,
AUCm (po) AUCm (P) nAUC (po) AUCm. (p*) AUC (Ø)
In particular, nAU'CC means that abstention mechanism pe reaches oracle performance. In contrast, nAUC < 0 indicates that Pa has a deleterious effect, with declining average ranking performance while abstention increases.
In this study, order to guarantce consistent comparisons betwoen abstention mechanisms, we randomly set aside 20% of the initial dataset as lest set, treating the remaining 80% the reference set. Results, unless specified otherwise, averaged across five random seeds.
5 Results
5.1
Abstention Performance
We report nAUC (Equation ( for all methods described in Section averaged model-wise (Table D and illustrate the dynamics of the perfornance-abssentionn curves for the ember-vi model on the STACKOvER- FLOW dataset in Figure 2
Abstention works. All evaluated abstention- methods improve downstream evaluation metrics (nAUCs greater than 0), showcasing the relevance of abstention approaches a practical setting.
Reference-based abstention works better. The value of reference-based abstention approaches is demonstrated in Table [ in which we notice that the Amear-regressiie regreesioon-baedd method Ula performs aver- age largely better than all reference-free baselines, edging out the best bascline strategy (ugrd) by almost 10 points on the mAP metric.
Having shown the superiority reference- based abstention methods across the entire benchmark, we conduct (unless otherwise specified) the following experiments on the ember-v1 model using mAP for evaluation (main metric in the MTEB leaderboard (Muennighoff et al.) 2022)), ), and compare Uja Usdd respectively the best reference-based and reference-fres methods mAP mPP-wise.

---

Published in Transactions on Machine Learning Research (09/2024)
Table Abstention performance. nAUCs % averaged model-wise, for each method, dataset and metric.
Method Metric
Dataset
ScIDoes
mAP
49.2
58.7
4.7
65.4
mNDCG
54.9
62.9
10.2
66.6
mRR
80.2
80.3
41.3
80.5
ASKUTUNTU
mAP
24.1
16.2
7.3
22.0
mNDCG
28.1
15.7
8.6
21.5
mRR
32.7
16.6
14.0
26.2
STACKOVERFLOW
mAP
20.7
24.6
35.7
42.6
mNCG
21.0
25.0
35.6
42.6
mRR
21.4
24.8
36.1
42.8
ALLOPROF
mAP
16.6
17.0
19.3
29.7
mNDCG
16.8
16.9
18.8
29.3
mRR
16.9
19.1
CMEDQAvI
mAP
24.4
23.3
22.3
30.6
mNDG
22.3
mRR
26.5
24.2
23.4
31.5
MMARCO
mAP
30.1
31.0
39.4
34.0
mNDCG
30.5
30.9
38.8
31.1
mRR
30.1
31.0
39.6
31.3
Average
mAP
27.5
28.5
21.5
37.4
mNDCG
29.4
29.1
22.4
38.0
mRR
34.6
32.3
28.9
40.7
0.85
0.95
0.96
mNDCG
0.90
Umax
0.90
0.85
0.85
0.60
Uno
0.75
0.0
0.2
0.4
0.6
0.8
0.0
0.2 0.4
0.6
0.8
0.0
0.2
0.4
0.6
0.8
Abstantion Rate
Abstention Rate
Abstention Rate
Figure 2: Performancc-absteetiinn curves for the ember-vi model on the STACKOVERFLOW dataset. All methods show increasing curves, indicating effective abstention. The reference-based method, Wins stands out above the others, demonstrating superior abstention performance. Note that u" and û respectively denote the oracle and the random baseline, associated with the p* and و mechanisms (Sections 2.2 and 4.3
5.2 Abstention Effectiveness vs. Raw Model Performance
We investigate whether correlation exists between abstention effectiveness and model raw performance (i.e., without abstention) on the reranking task. For each model-dataset pair, we compute both no-abstention mAP and abstention nAUC, using Win as a confidence function (Figure Table 2)
better ranker implies better abstention. Figure and Table 2 showcase clear positive correlation between model raw performance and abstention effectiveness, albeit with disparities from one dataset another. Better reranking methods naturally produce more calibrated score distributions, which in turn leads to improved abstention performance. This highlights the value of incorporating abstention mechanisms into high-performing retrieval systems and reinforces that their purpose is not to turn weak rankers into strong ones, but to enhance the quality of models that already perform reasonably well. For thoroughness, we evaluated a range of open-source models with varying performance levels, though in practice, users typically select models they already know to be sufficiently effective.
8 random (Ineffective) abstention method cannot differentiate hetween good and bad instances. Thus, in expectation, random abstention the test. dataset results constant, performance whatever the abstention rate.
8

---

Published in Transactions on Machine Learning Research (09/2024)
SciDocs
Table 2: Pearson's correlations between no-abstention mAP and abstention nAUC, for each dataset. General correlation is computed over the whole scatter plot, i.e, across all datasct tades pairs.
AskUbuntu
nAuc
StackOverflow
Alloprof
CMedQAv1
Mmarco
Dataset
Correlation
0.2
0.4
066
08
no-abstention mAP
SciDoos
0.85
ASKUBUNTU
Figure 3: Abstention performance (nAUC in %) vs. no-abstention performance (mAP), for all models and datasets, using Whn as confidence function. Each data point represents model-dataset pair.
STACKOVEAFIION
0.90
ALLOPROF
0.56
CMaDQAvI
0.91
MMAROо
General
0.73
MAEs for target achieved abstention rates and performance levels (in %) (model: ember-v1, dataset: STACKOVERFLOW, metric: mAP).
Table 3:
Table 4: nAUCs averaged test-sct-wise (model: ember-v1, metric: mAP).
Method
Mmaa
Target A bbstention
10%
50%
90%
ALLOPROF
30.1
32.7
17.1
31.4
Rate
Ustd
1.10
1.80
1.08
CMEUQAVI
ASKUBUNTU
27.9
28.5
32.1
31.1
19.8
17.2
34.2
34.2
Wiin
1.07
1.79
1.09
MMARCO
27.5
32.2
18.9
34.9
mAP
1.24
1.52
3.09
ScnDoos
STACKOOEELLLO
20.3
30.7
11.9
20.6
Uja
1.22
1.33
1.51
Average
17.9
5.3 Threshold Calibration
In practical industrial scttings, a major challenge with abstention methods lies in choosing the right threshold to ensure a desired abstention rate or performance level on new data. To evaluate threshold calibration quality, we choose three target abstention rates (10% 50%, and 90%) and rely on the reference se to infer the corresponding threshold and level of performance (here, the mAP). Then, the Mean Absolute Error (MAE) between target and achieved values test instances is reported (Table 3), averaging across 1000 random seeds on the STACKOVERRFLLCOO dataset. 10
"in is better calibrated than Ustd The lower part of Table shows that the MAE between the target and achieved mAP increases with the abstention rate, which logical because a higher abstention rate reduces the number of evaluated instances and therefore increases volatility. However, we observe that the MAF increases much less significantly for the win method than for the uard method, with the latter having MAE twice high at target abstention rate of 90%. This provides strong evidence that Wja is more reliable high abstention rates.
5.4 Domain Adaptation Study
In a practical setting, it is rare to have reference set that perfectly matches the distribution of samples seen at test time. To measure the robustness of our method to data drifts, we fit them on a given dataset and evaluate on all others.
Reference-based approaches are globally not robust to domain changes. From Table 4 we see on average that the dference-based method underperforms reference-free baselines when fitted on the "wrong" reference set. However, certain reference datasets seem enable methods to generalize better overall, even
reference essential in order determine which threshold value corresponds to given abstention or level performance. For example, the mAP corresponding target abstention rate 50% in the reference is and the values achieved test different seeds 0.5 and the MAE be equal 1/2 ((0.6 0.5] |0.6 0.7l) 0.1.
"In this setting,

---

Published in Transactions on Machine Learning Research (09/2024)
Table 5: Minimum reference set size for Win to outperform Ustd (model: ember-v1, metric: mAP).
SolDocs
AskUbuntu
StackOverTow
64
NAUG
00
Dataset
Break- Even
-0.25
1000
2003
1930 Minaroo
Alloprof
CMedQAvi
ScIDocs
12
04
USSNTUU
9
NAUC
STACKOVERFLOW
19
641
ALLOPROF
58
CMEDQAAI
50
1003
250
75
Reference Size
Reference Size
Reference Size
MMARCO
80
Figure 4: nAUC vs. reference set size for Uti and Ustd all datasets (model: ember-v1, metric: mAP).
Average
38
Figure 5: Comparison between relevance (ge) and con- fidence (umn) assessment times on one Apple MI CPU (model: all-MiniLM-16-v,, dataset: STACKOVERFLON)
Avg. Runtime (me)
out of distribution, which mainly an artifact of the number of positive documents per instance (further insights in Appendix Cj.
5.5 Reference Set Minimal Size
Calibrated reference-based mothods outperform reference-free baselines (Section 5.1 but require having ac- cess an in-domain reference set (Section 5.4). To assess the efforts required for calibrating these techniques extract best abstention performance, we study the impact of reference set size on method performance.
Small reference sets suffice. From Figure 4 and Table 5 note that Min requires only fairly small amount of reference data (less than 40 samples average) to outperform Ustd, which demonstrates the effectiveness of using reference set even when data is scarce. This provides strong evidence on the applica- bility of our method to practical industrial contexts, the labeling of around 50 samples appearing perfectly tractable in such settings.
5.6 Computational Overhead Analysis
Crucial to the adoption of our abstention method is the minor latency overhead incurred at inference time. In our setup, we consider documents are pro-embedded by the bi-encoders, and stored in a vector database. We average timed results over 100 runs on a single instance prediction, running the calculations on one Apple M1 CPU (Figure 5). To upper-bound the abstention method overhead, we baseline the smallest model with the fastest embedding time available (211-MiniL--16-v2))
Our data-based abstention method free of any significant time overhead. Win based confidence estimation accounts for only 1.2% of relevance scores calculation time (Figure 5 making it almost cost-free option II
6 Conclusion
We developed a fightweight abstention mechanism lailored to real-world constraints, agnostic to the ranking method and shown be effective across various settings and use cases. Our method, which only requires black-box access to relevance scores and small reference set for calibration, is essentially free-lunch to plug to any retrieval system, in order to gain better control over the whole IR pipeline.
II we used cross-cncoder, documents could not be pre-embedded they need to be appended the input qucry. In that scenario, the computation time required ge would therefore increase, while that required for Mjin would remain unchanged, thus reducing the relative running time.
10

---

Published in Transactions on Machine Learning Research (09/2024)
Limitations and Future Work
Our study investigates abstention mechanisms specifically within the reranking stage of an IR pipeline. While this provides framework for estimating the confidence of the final prediction, the absence of datasets with explicit document-wise labels has limited our ability to evaluate these methods the primary retrieval stage. These datasets typically lack labels indicating relevant and irrelevant documents, as is the case with classical retrieval benchmarks like BEIR (Thakur et al. 2021). However, in theory, our methods should be applicable this earlier stage of the pipeline, making this a potential area for future exploration.
Moreover, experiments have mainly focused reranking the top-10 retrieved documents, aligning with common industrial use cases, A natural extension would be to explore how abstention mechanisms perform when applied to larger sets of documents, such as the top-50 or top- 100 results. Since our inear--eeression- based abstention method is designed to scale efficiently, it could potentially benefit from the richer decision- making context provided by more document. scores, although this impact has yet to be quantified.
A broader consideration, however, lies in the potential of generative models, such as those used in RAG, handle abstention their own (Zhang et al. 2024). These models might inherently possess the capability to estimale confidence without needing separale abstention mechanism, compelling prospect for further investigation.
Another promising avenuc to test abstention mochanisms on relevance-scoring mothods that offer more de- lailed information, such as token-level relevance scores for each query-docunent pair. Methods like ColBERT (Khattab & Zaharia 2020) or the ColPALI (Faysse et al. 2024) model family provide richer signals, which could lead improved abstention performance. It would be interesting see if the additional granularity in relevance scores results in better decision-making, particularly in scenarios where token-level understanding could refine the abstention process.
Ethical Statement
The proposed abstention mechanism enables IR systems to refrain from making predictions when signs of unrcliability are detected. This approach enhances the accuracy and effectiveness of retrieval systems, but also contributes to the construction of more trustworthy Al technology. by reducing the risks of providing potentially inaccurate or irrelevant results to users. Furthermore, abstention enables downstream systems to avoid using computational resources on unreliable or irrelevant predictions, optimizing resource utilization and therefore minimizing energy consumption. In the context of RAG systems, where largo-scale computations are prevalent, introducing abstention mechanisms is positive stride towards trustworthy and sustainable AI development, reducing overall computational demands while potentially improving system performance.
Acknowledgements
Training compute obtained the Jean Zay supercomputer operated by GENCI IDRIS through com- pute grant 2023-AD011014000111 AD010111777 as well as on Adastra through project c1615122, cad15031, cad14770.
AD010614770
11

---

Published in Transactions on Machine Learning Research (09/2024)
References
Jinheon Baek, Soyeong Jeong, Minki Kang, Jong C Park, and Sung Ju Hwang. Knowledge-nugmented language model verification. arXiv preprint Xiv:22230 12836, 2023.
Peter Bartlett and Marten Wegkamp. Classification with a reject option using a hinge loss. Journal of Machine Learning Research, 9(8), 2008.
Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural network. In Francis Bach and David Blei (eds.). Proceedings of the 32nd International Conference Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 1613 1622, Lille, France, 07-09 Jul 2015. PMLR. URL https://proceeddigg mlr. press//37/blundee www.dd///ll1......mmmm html
Luiz Henrique Bonifacio, Vitor Jeronymo, Hugo Queiroz Abonizio, Israel Campiotti, Marzieh Fadaee, Roberto Lotufo, and Rodrigo Nogueira. mmarco: multilingual version of ms marco passage ranking dataset, 2021.
Weiwei Cheng, Michaël Rademaker, Bernard De Baets, and Eyke Hüllermeier. Predicting partial orders: ranking with abstention. In Machine Learning and Knowledge Discovery in Databases: European Confer- ence, ECML PKDD 2010, Barcelona, Spain, September 20-24, 2010, Proceedings, Part T 21, pp. 215-230. Springer, 2010.
Chi-Keung Chow. An optimum character recognition system using decision functions. IRE Transactions on Electronic Computers, (4):247-254) 1957.
Arman Cohan, Sergey Feldman, Iz Beltagy, Doug Downey, and Daniel S. Weld. Specter: Document-level representation learning using citation-informed transformers. In ACL, 2020.
Luigi Pietro Cordella, Claudio De Stefano, Francesco Tortorella, and Mario Vento. A method for improving classification reliability of multilayer perceptrons. IEEE Transactions on Neurol Networks, 6(5):1140 1147, 1995.
Corinna Cortes, Giulia DeSalvo, and Mehryar Mohri. Boosting with abstention. Advances in Neural Infor- mation Processing Systems, 29, 2016.
David Roxbee Cox and F. Joyce Snell. Analysis of binary data, volume 32. CRC press, 1989.
Maxime Darrin, Pablo Piantanida, and Pierre Colombo. Rainproof: An umbrella to shield text generators from out-of-disttibation data. arvvv preprint XXX112212200111,, 2022.
Maxime Darrin, Guillaune Staerman, Eduardo Dadalto Câmara Gomes, Jackie CK Cheung, Pablo Pi- antanida, and Pierre Colombo. Unsupervised layer-wise score aggregation for textual ood detection. Proceedings of the AAAT Conference on Artificial Intelligence, volume 38, pp. 17880-17888, 2024.
Claudio De Stefano, Carlo Sansone, and Mario Vento. To reject not to reject: that is the question- answer in case of neural classifiers. IEEE Transactions Systems, Man, and Cybernetics, Part C (Applications and Reviews), 30(1):84-94, 2000.
Li Dong, Chris Quirk, and Mirella Lapata. Confidence modeling for neural semantic parsing. arXiv preprint arxiv:1805. 1006:1100.00464, 2018.
Ran Fl-Yaniv and Yair Wiener. On the foundations of noise-free selective classification. Journal of Machine Learning Research, 11(53))1605 1641, 2010. URL http://iir.rrrr//pppee//////aaaa html
Manuel Faysse Hugues Sibille, Tony Wu, Gautier Viaud, Céline Hudelot, and Pierre Colombo. Colpali: Efficient document retricval with vision language models. arXiv preprint arXiv:2407.014400 2024.
Ronald A Fisher. The goodness of fit of regression formulae, and the distribution of regression coefficients.
Journal of the Royal Statisticnl Sociely, 85(4):597-615 612, 1922.
12

---

Published in Transactions on Machine Learning Research (09/2024)
Giorgio Fumera and Fabio Roli. Support vector machines with embedded reject option. In Pattern Recog- nition with Support Veclor Machines: First International Workshop, SVM 2002 Ningara Falls, Canada, August 10, 2002 Proceedings, pp. 68 82. Springer, 2002.
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pp. 1050-1059. PMLR, 2016.
Yarin Gal, Riashat Islam, and Zoubin Ghahramani. Deep bayesian active learning with image data. In International conference on machine learning, pp. 1183-1192. PMLR, 2017.
Yonatan Geifman and Ran El-Yaniv. Selective classification for deep neural networks. Tn I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Newral Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL htttps://proceedinn neurips. peurips.cc/paper httpppsccccpppppffeeeeeepeeepe
.df
Yonatan Geifman and Ran El-Yaniv Selective A deep neural network with integrated reject option. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 2151-2159. PMLR, 09-15 Jun 2019. URL httpp:://pprcceednnnneeee/////ee mlr www.p..res/////////ee http://////geeeeann/// get.g/niffaaannaae html.
Charles J Geyer. Practical markov chain monte carlo. Statistical science, pp. 473 483, 1992.
Fduardo Dadalto Camara Gomes, Florence Alberge, Pierre Duhamel, and Pablo Piantanida. Igeood: An information geometry approach to out-of-distribution detection. arXiv preprint arXiv:2203.07798 2022.
Alex Graves. Practical variational inference for neural networks Advances in neural information processing systems, 24, 2011.
Jiafeng Guo, Yixing Fan, Qingyao Ai, and W Bruce Croft. A deep relevance matching model for ad-hoc retrieval. In Proceedings of the 25th ACM international on conference on information and knowledge management, pp. 55-64, 2016.
Matan Haroush, Tzviel Frostig, Ruth Heller, and Danicl Soudry. A statistical framework for efficient out of distribution detection in deep neural networks. arXie preprint arXiu:2102.122: 129677 2021.
Matthias Hcin, Maksym Andriushchenko, and Julian Bitterwolf. Why relu networks yield high-confidence predictions far away from the training data and how to mitigate the problem. In Proceedings of the IEFE/CVE Conference on Computer Vision and Pattern Recognition, pp. 41-50, 2019.
Martin E Hellman. The nearest neighbor classification rule with reject option. IEEE Transactions Systems Science. and Cybernetics, 6(3):179-18)) 185, 1970.
Dan Hendryeks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples 2016.
in neural networks. arXiv preprint
Geoffrey E Hinton and Drew Van Camp. Keeping the neural networks simple by minimizing the description length of the weights. In Proceedings of the sixth annual conference on Computational learning theory, pp. 5 13, 1993.
Tin Kam Ho. Random decision forests. In Proceedings of 3rd international conference on document analysis and recognition, volume 1, pp. 278-282, IEEE, 1995.
Arthur E Hoerl and Robert W Kennard. Ridge regression: Biased estimation for nonorthogonal problems. Technometrics, 12(1):55-67 67, 1970.
Neil Houlsby, Ferene Huszár, Zoubin Ghahramani, and Máté Lengyel. Bayesian active learning for classifi- cation and preference learning. ar preprint urXi::1112.5745, 2011.
Yen-Chang Hsu, Yilin Shen, Hongxia Jin, and Zsolt Kira. Generalized odin: Detecting out-of-distributtion image without learning from diitr---istribttion data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10951-10960, 2020.
13

---

Published in Transactions on Machine Learning Research (09/2024)
Kalervo Järvelin and Jaana Kekäläinen. Cumulated gain-based evaluation of ir techniques. ACM Transac- tions on Information Systems (TOIS), 20(4):422-446, 2002.
Amita Kamath, Robin Jia, and Perey Liang. Selective question answering under domain shift. arviv preprinl arXiv:2006.09// 2020.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. ar preprint arXiv:2004.04966, 2020.
Omar Khattab and Matei Zaharia. Colbert: Efficient and effective passage search via contextualized late interaction over bert. In Proceedings of the 43rd International ACM SIGIR conference research and development in Information Retrieval, pp. 49-48, 2020.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncer- lainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017.
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out- of-iistrinttion samples and adversarial attacks. Advances in neurol information processing systems, 31, 2018.
Antoine Broosare-Brosaad,, Stephane Gazaille, and Michel Desmarais. Alloprof: a new french question-answer education dataset and its use in an information retrieval case study, arx preprint arXiv:2302.07738 2023.
Tao Lei, Hrishikesh Joshi, Regina Barzilay, Tommi Jaakkola, Katerina Tymoshenko, Alessandro Mos- chitti, and Lluis Marquez. Semi-supervised question retrieval with gated convolutions. orXiv preprint arXi::111.005766 2015.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-augmmened generation for knowledge- intensive nlp tasks. Advances in Newral Information Processing Systems, 93:9459-9474, 2020.
Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, Joe Davison, Mario $aško, Gunjan Chhablani, Bhavitvya Malik, Simon Brandeis, Teven Le Scao, Victor Sanh, Canwen Xu, Nicolas Patry, An- gelina MiMillan-Majon Philipp Schmid, Sylvain Gugger, Clément Delangue, Théo Matussière, Lysandre Debut, Stas Bekman, Pierric Cistac, Thibault Gochringer, Victor Mustar, François Lagunas, Alexander Rush, and Thomas Wolf. Datasets: A community library for natural language processing, In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp- 175 184, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. URL pettss//accaante http:///aaaatooo........ org/ 2021, emnlp-demo 21
Shiyu Liang, Yixuan Li, and Rayadurgam Srikant. Enhancing the reliability of out-of-distribution image detection in neural networks. arXiv preprint arXio: Xi.:::1060000000 706602690, 2017.
Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Eneegy-based out-of-distribution detection. Advances in neural information processing systems, 33:21464-21-4 21477,,,, 2020.
Antonio Loquercio, Mattia Segu, and Davide Scaramuzza. A general framework for uncertainty estimation in deep learning. IEEE Robolics and Aulomation Lellers, 5(2):3153 3160, 2020.
Prasanta Chandra Mahalanobis. On the generalized distance in statistics. Sankhyā: The Indian Journal of Slatistics, Series A (2008-), 80:S1 S7, 2018.
Anqi Mao, Mehryar Mohri, and Yutao Zhong, Ranking with abstention. arXiv preprint orXiv:2307.00035 2023.
Bhaskar Mitra and Nick Craswell, Neural models for information retrieval. arXiv preprint 1IW::1005000009 2017.
14

---

Published in Transactions on Machine Learning Research (09/2024)
Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. Mteb: Massive text embedding bench- mark. XrXiv preprint arXin:2210.0733 2022.
Arvind Narayanan, Hristo Paskov, Neil Zhenqiang Gong, John Bethencourt, Emil Stefanov, Eui Chul Richard Shin, and Dawn Song. On the feasibility of internet-scale author identification. In 2012 IEFE Symposium on Security and Privacy, pp. 300 314. IEEE, 2012.
Arvind Noelakantan, Tao Xu, Raul Puri, Alec Radford, Jesse Michael Han, Jerry Tworck, Qiming Yuan, Nikolas Tezak, Jong Wook Kim, Chris Hallacy, al. Text and code embeddings by contrastive pre-training. arXiv preprint ar 0rXt:22011.0000, 2022.
Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Hernández Abrego, Ji Ma, Vincent Y Zhao, Yi Luan, Keith B Hall, Ming- Wei Chang, al. Large dual encoders are generalizable retrievers. arXiv preprint arXiv:21 12.07999, 2021.
Rodrigo Nogueira and Kyunghyun Cho. Passage re-ranking with bert. arXiv preprint arXiv:1901.04055, 2019.
Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathicu Blondel, Peter Prettenhofor, Ron Weiss, Vincont Dubourg, et al. Scikit-lcarn: Machine learning in python. the Journal of machine Learning research, 12:2825 2830, 2011.
John Platt et al. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in large margin classifiers, 10(3):61 -4, 1999.
Alexander Podolskiy, Dmitry Lipin, Andrey Bout, Ekaterina Artemova, and Irina Piontkovskaya. Revis- iting mahalanobis distance for transformer-based out-of-domain detection. In Proceedings of the AAAT Conference on Artificial Intelligence, volume 35, pp. 13675 13682, 2021.
Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't know: Unanswerable questions for squad. arXiv preprint arXiv: 03822, 2018.
Juan Ramos et al. Using tf-idf to determine word relevance in document queries. In Proceedings of the first instructional conference on machine learning, volume 242, pp. 29 48. Citeseer, 2003.
Nils Reimors and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11 2019. URL http:://axiiii org/ http//////199/... 10084
Nils Reimers and Iryna Gurevych. The curse of dense low-dimensional information retrieval for large index sizes, In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pp. 605 611, Online, 2021. Association for Computational Linguisties. URL https://arxiv.. http:////axxvv.o/aa/s/so// wrg. abgg/22b//012. 14210
Jie Ren, Stanislav Fort, Jeremiah Liu, Abhijit Guha Roy, Shreyas Padby, and Balaji Lalshminarayanan. A simple fix to mahalanobis distance for improving near-ood detection. arxiv preprint arXiw:2106.000 09022, 2021.
Stephen Robertson, Hugo Zaragoza, et al. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trendoo in Information Retrieval, 3(4)3333 389, 2009.
Stephen E Robertson, Steve Walker, Susan Jones, Micheline M Okapi al (rec-3. Nisl Special Publication Sp, 109:109, 1995.
Mike Gatford, et al.
David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning representations by back- propagating errors. nature, 323(6888):533 536, 1986.
Bernhard Schölkopf, Robert C Williamson, Alex Smola, John Shawe- Shawe-Taylor, and John Platt. Support vector method for novelty detection. Advances in neural information processing systems, 12, 1999.
15

---

Published in Transactions on Machine Learning Research (09/2024)
Lewis Smith and Yarin Gal. Understanding measures of uncertainty for adversarial example detection. arXiv preprinl Xrv:
1 1803. 08538, 2018.
Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. Beir: A heteroge- nous benchmark for zero-shot evaluation of information retrieval models. arXiv preprint 2Xi::204400603 2021.
Ellen M Voorhees et al. The trec-8 question answering track report. In Tree, volume 99, pp. 77 82, 1999.
Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. Text embeddings by weakly-supervised contrastive pre-training. arXiv preprint 1xXiv2212.035322 2022.
Zhiruo Wang, Jun Araki, Zhengbao Jiang, Md Rizwan Parvez, and Graham Neubig. Learning to filter context for retrieval-augmennter generation. aiviv preprinl xrXim:231.00837, 2023.
Yair Wiener and Ran Ylniv.... Pointwise tracking the optimal regression function. Advances in Neural Information Processing Systems, 25, 2012.
Yair Wiener and Ran El-Yaniv. Agnostic pointwise-compeliivve selective classification. Journal of Artificial Intelligence Research,
2015.
Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff. C-pack: Packaged resources to advance general chinese embedding, 2023.
Ji Xin, Raphael Tang, Yaoliang Yu, and Jimmy Lin. The art of abstention: Selective prediction and error regularization for natural language processing. In Proceedings of the 59th Annual Mecting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 1040-1051, 2021.
Ori Yoran, Tomer Wolfson, Ori Ram, and Jonathan Berant. Making retrieval-augmenned language models robust to irrelevant context. arXiv preprint arXiv:2310. 01558, 2023.
Zhiyuan Zeng, Keqing He, Yuanmeng Yan, Zijun Liu, Yanan Wu, Hong Xu, Huixing Jiang, and Weiran Xu. Modeling discriminative representations for out-of-domain detection with supervised contrastive learning. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.). Proceedings of the 59th Annual Meeting of the Association for Computational Lánguisties and the 11th International Joint Conference Natural Language Processing (Volume 2: Short Papers), pp. 870-878, Online, August 2021. Association for Compulational Linguistics. doi: 10.18653/v1 /////2112222222/2 acl-short.110, URL http:////cllaaatooloogg/// org/
2021 achorshort........ 110
Paitian Zhang, Shitao Xiao, Zheng Liu, Zhicheng Dou, and Jian-Yun Nie. Retrieve anything to augment large language models, 2023.
Sheng Zhang, Xin Zhang, Hui Wang, Jiajun Cheng, Pei Li, and Zhaoyun Ding. Chinese medical question answer matching using end-to-end character-level multi-scale cnns. Applied Sciences, 7(8):767, 2017.
Tianjun Zhang, Shishir G Patil, Naman Jain, Sheng Shen, Matei Zaharia, lon Stoica, and Joseph E Gonzalez. Raft: Adapting language model to domain specific rag. arXiv preprint Xrx::24403. 10131, 2024.
Yun Zhang, David Lo, Xin Xia, and Jian-Ling Sun. Multi-factor duplicate question detection in stack overflow. Journal of Compuler Science and Technology, 30:981-90 997, 2015.
Wayne Xin Zhao Jing Liu, Ruiyang Ren, and Ji-Rong Wen. Dense text retrieval based on pretrained language models: A survey, arXiv preprint orXiv:2211.148% 2022.
Mu Zhu. Recall, precision and average precision. Department of Statistics and Actuarial Science, Universily of Waterloo, Waterloo, 2(30):6, 2004.
16

---

Published in Transactions on Machine Learning Research (09/2024)
Table 6: Details on raw datasets.
Table 7: Dataset details after preprocessing.
Beforcaor
Langenag Fux
Fass
Sax
Pass
X
June F
AsgUauSTU
Exglab
173
23
-11
Sui
Exgles
2012
23
31
2533
1.2
2067
3
Auzomi
Bhoohh
3216
37
1065
1d
Maruuco
-00
-303
1301
1000
XDHABCO
1140
18
3
3.1
Table 8: Model details.
Reference
Languages
Type Similarity
[0]
anber-v1
TRA
English
335
Lin-enbedder
Zhang al
2023
English
bi
109
bge-baae-en-v1.5
Xiao el al 2023
English
bl 008
109
bge-raranker-bana
Xino et al.
2123
English, Chinese cross n.a.
278
bge-reranker-large
Xiao et
2023
English, Chinese cross n.a.
500
e5-enall-v2
Wang
2022
English
33
e5-large-v2
Wang el
2022
English
bi
335
multillingual-o--mmal
Wang cl al.
21222
94
bi
118
maltllingual-e--laagg
Wang cl a.
2022
cos
560
ramarco-Mtntil-----co---0
Remmers
Gutrevyrh
2021
English
nanarco-distillertttoor-
Gureeych
2021
English
86
warparoo-TinyeENT-----
Reimers & Gubeevych
102
English
cross
Retmers & Guzevych
2021
English
cro8s
11S.
23
stab Tiab-TiayEERT-L-0
Reimers Guzevych
2019
English
cross
11.0.
14
stab-disttllrbberta-buss
Reimers Guzevych
2019
English
cross
1L...
malti-gga-diattibartt--
Remmers Gorevych
2019
English
86
multi---e-/iiiiil-----------
Remmers & Giteevych
2019
English
bi
23
all a11-WiniDM-18-12
Retmers & Guzevych
2019
English
bi
008
23
al1 a22-dietilroberta-11
Reimers Gurevych 2019
English
008
82
all al1-mppettbase--22
Reimers Gurevych 2019
English
008
100
noor-diattlroobeteebbaee
Reimers Gurevych 2019
English
crces
hall-distilrobertaabbes
Reimers & Guzevycl
2019
English
cross
82
A Additional Information on Models and Datasets
In this section, we give details the datasets used in our work, before the preprocessing phase (Table 6 and after preprocessing (Table 7). As in the main text, N denotes the number of instances and k represents the number of candidate documents. In the same logic, kmins kmax and respectively denote the minimum, maximum and average number of candidate documents in the datasct. In addition, Pmin+ Pmax, P denote the minimum, maximum and average number of positive documents respectively. Additionally, we provide information on models' raw performance (without abstention) across the whole benchmark in Table
In the data preproccssing phase, we limit the number of candidate documents to 10 and the mumber of positive docunents per instance to five, for the sake of realism and simplicity. Intuitively, these 10 documents are to be interpreted as the output of a retriever. In practice, for each instance taken from a raw dataset, we randomly sample the positive documents (maximum of five) and complete with a random sampling of the negative documents. When a raw instance contains fewer than 10 documents, it automatically discarded.
Additionally, we provide some details about the models used in our study (Table S]: their reference paper, the language(s) in which they were trained, their type (bi-encoder or cross-encoder), the similarity function used (when applicable) and finally their number of parameters denoted |0] (in millions).
B Additional Results on Abstention Performance
In this section, we propose a comprehensive evaluation our abstention strategies, for each model on the benchmark (Table 10). It clear that our referencc-based strategics consistently outperform all basclines, whatever the size, nature (bi- or cross-encoder) or training language of the model concerned. These results support the observations made in Section 5.1 of the main text.
17

---

Published in Transactions on Machine Learning Research (09/2024)
Table Models' raw performances (mAP, mNDCG, mRR without abstention) on the whole benchmark.
Thatsort- Mistric
SeiThoo
AscUHETT uNDCG
STANRCDORTLL GUDDO
Atzcenne
CSERndAA uNDDG
Average
a182
12.32
aNDDO
0.06
0..55
0.50
0.56
0.55
0.84
9.24
3.81
0.75
0.55
0.66
0.36
0.51
0.64
0.56
0:43
0.07
0.64
0.65
0.75
0.00
110 uabeddee
14.4
11.07
11.00
0.71
01.53
0.882
0.20
3.25
0.22
14.22
11.54
14.35
11.11
11.6
11.00
0.47
02.90
0.49
1154
15.74
ILU6
0.80
0.68
0.73
0.86
3.27
0.29
0.23
0.33
0.31
0.53
0166
0.53
0:45
0.58
0.45
0.85
0.75
0.16
1880
11.06
3.08
3.20
3.20
01:56
11.00
0106
0.57
vorlazgee12 2
0.56
0858
0.50
0.71
0.58
0.81
3.29
3.77
3.53
0.56
0.53
0.57
0.50
0.67
0.50
3.20
0.61
0.90
0.56
0.76
0.59
0.50
18.6
0.56
8.00
0.52
0.80
3.88
3.70
0.83
0.50
0.70
1651
0.58
0.50
0.50
3.83
0.80
0.83
0.77
0.54
0.50
wutestingsal- al-large
0094
11.9
0.71
0183
0.81
3.20
0.77
3.20
1137
1130
1130
0181
0890
16:6
0.5
0.57
0.44
0:45
0.57
1157
1156
0363
0.57
3723
11.05
1.16
01.00
1.56
0.50
0.57
1150
663
we-nanes--Mi Mini -8-1--
0.50
0.55
0.67
0.00
0.70
0.73
3.05
3.55
0.6
0.59
D.64
0.53
0.02
0.55
0.45
3.43
0.50
0.43
0.60
0.71
0.6
0657
11.4
1156
0.77
0.73
3.35
348
3.35
16335
1445
1L35
0.50
1156
0..2
0.21
0.83
0.21
0L55
0166
0.5
0.57
01.0
372
283
0.44
0.57
0.00
0.46
all Miw. 10M---5-1
11.0
1150
0388
1135
0.36
088
0.45
0.50
0.72
All- di-eeetssheetteee
11.6
1156
1100
0.75
085
0.85
9.01
071
1850
1010
1.00
800
8.5
0.57
0.44
871
6.7
0.06
0.56
0.50
0.00
0.55
0.83
9.00
8288
301
0.50
0.58
0.53
061
016
8.66
0.50
0.42
0.64
6.6
0.5
Berra-di hann
11.22
113
0181
0.79
1L.7
1136
0330
0256
021
0.41
0.50
6654
0.56
0.20
Coefficient Value
0.4
1.5
0.10
02
1.0
0.06
0.00
-0.05
0.2
0.10
-04
-1.0
0.0
2.5
5.0
7.5
0.0
2.5
5.0
7.5
0.0
2.5
5.0
7.5
2.5
2.0
Coefficient Value
1.5
1.0
1.0
0.5
0.0
0.0
0.0
-05
-10
-10
7.5
2.5
Document
Document
Document
Figure 6: Coefficients of the Wiin confidence function (model: ember-v1, metric: mAP). Recall that docu- ments are sorted increasing order of relevance scores. Therefore, document 10 corresponds to the document with highest relevance score while document is the least relevant to the query.
C Additional Results on Domain Adaptation
In this section, we provide additional results concerning domain adaptation. In particular, we propose to take closer look at the coefficients of our confidence function (Figure 6) and to relate this information to abstention performance on the whole benchmark (Table
A simple obscrvation that can be made from Figure is that the value taken by the cocfficients depends very strongly on the number of positive documents per instance (see Table 7 Indeed, when we look the results for the ScIDocs dataset (five positive documents per instance average), see that Ujan has positive weights on the first five documents and negative weights the last five. A similar observation can be made for the other five datasets.
These observations are corroborated by Table [I] which shows that domain adaptation works well when the average number of positive documents per instance is approximately the same (c.g., from STaCKOvERFLOW to ALLOPROF). On the contrary, we can see that when calibration performed on ScIDoos, performance transfers much less well, since the rest of the datasets have a significantly lower number of positive documents
18

---

Published in Transactions on Machine Learning Research (09/2024)
Table 10: Methods nAUCs (in %) on the whole benchmark.
Dateset
SciDoes
ASSUUUNTU
ALLOPSO0
DESSADCO
Method
Meat
Seat
Post
NL.3
Most
11.3
Model
|
Metrie
RAP
50.7
33.2
10.2
51.7
48.0
50..
41.2
37.3
WNDDGG
16.5
16.1
35-3
13.9
16.9
31.8
86.3
30.3
17.5
51.0
18.9
30.4
90.9
27.
79.7
59.5
31.3
41.4
33.5
35.8
milll
63.1
53.3
31.5
13.5
40-2
13.3
22:3
23.0
30.0
31.0
17.0
19.85
195-9
27
58.6
23.7
15.6
42.3
13.5
35.5
412
02.3
1.3
15.3
11.7
2315
1.7
21.01
50.01
12.3
47.3
12.
15.3
23.7
22.3
08.0
22.5
1318
21.6
16.3
152
25.3
03.5
23.7
WNDDC
52.0
16.4
3.4
03.9
30.2
19.4
2.1
39.7
26.5
32.86
$7.00
52.7
30.0
23.8
22.5
31.4
21:11
26.9
20.65
38.0
15.3
23.4
90.6
25.3
15.1
10.4
16.3
+0.4
42/0
19.9
15.6
20.86
20.5
$7.95
15.3
15.7
22.3
23:41
29..5
20.1
21.4
23.85
37.0
19.5
22-4
-01.5
25.7
1611
15.3
11.1
03:11
31:3
13.3
03.
20.0
10.3
10.1
17.1
257
22.3
13.1
05.5
26.5
27.7
15.3
-3.5
SINDCG
18.5
53.1
06.5
11.2
10.2
39.28
24.8
52.8
41.11
13.3
24.3
28.8
15.33
28.8
56.2
25.2
16.5
40.5
23.0
99.2
45.2
58.7
40.4
03.3
41-7
0.1
32.00
26.8
52.5
41.4
15.6
28.0
94.9
16.2
15.6
25.6
98.9
43.3
aNDOG
23.4
-6.7
31.7
28.4
44.3
05.1
61.4
aBR
67.0
36.4
87.5
13:0
22.0
30.8
96.9
$7.5
46.0
07.5
53.3
64.6
54.2
15.6
12:3
05.5
23:27
-1.7
1277
17.1
26.11
21.4
3556
25.5
11.88
0131
10.11
15.06
7130
75.3
75.1
8522
14.2
WNDCG
70.5
54.0
15.0
06.3
25.3
23.5
-1.5
15.5
17.0
94.1
24.9
33.6
35.9
32.9
43.5
08.8
T4.
50.6
01.3
75.5
76.3
72.3
85.3
with
50.4
53.4
19.3
76.9
15.3
23.8
0.9
6.1
17.3
54.9
24.6
33.9
36.0
33.5
41.0
08.8
51.0
68.6
64.6
78.5
77.5
75.1
85.1
40.1
$3.5
1.1
09.2
23..
15A4
10.30
23:31
22.6
21.4
38.5
41.5
12.3
17.3
21.5
23.4
12.9
15.5
35.5
45.7
41.3
19.22
45.5
16.1
$6.9
7.3
70.7
25.3
14.3
18.7
2012
22.8
28.0
$8.5
47.6
12.7
17:3
20.0
23.0
50.6
52.9
20.5
37.4
45.1
45.7
41.1
40.3
min
16.6
16.5
41.3
67.3
37.0
10:4
30.5
20.-
20.7
38.6
13.5
17.4
19.4
16.7
30.8
37.4
85-1
15:5
11.3
18.55
mAP
55.7
1775
12.3
2014
21.
10.5
15.3
2015
03.11
5012
WNDCG
08.1
22.9
16.5
53.6
58.1
36.1
15.4
22.6
59.7
53.6
54.3
61.0
41.3
22:3
10239
032
161
15.3
15.5
053
510
WNDCG
10.9
08.5
13.93
533-7
51.6
93.3
19.5
57.5
70.3
63.5
41.6
85.0
52.7
56.4
41.5
64.3
87.6
23.7
0.5
32.00
26.6
51.4
46.5
$5.6
9.1
22.1
10.7
22.2
00.8
68.6
54.0
75.5
62.9
43.3
65,0
63.0
1011
1.8
17:3
11235
-15.3
18:8
2111
27.5
1522
10.3
35-1
31.3
11.3
16.1
0220
71.3
01.33
WNDCG
15.6
36.9
7.4
70.0
21.3
10:7
-16.3
16.7
22.3
28.11
13.0
50.3
17.4
27.3
37-11
-2.77
19.8
51.1
05.3
7h.3
51.3
89.8
60.55
58.0
56.7
40.7
36.2
34.5
13.2
-13.5
17.0
58.7
57.8
18.5
50.5
15.1
27.1
27.1
42.7
70.2
10.0
16.7
70.0
76.5
07.1
87.3
03.3
REnarCa MistEN 08-c2a-55
mAP
51.5
39.9
06.3
19:3
1230
10.11
19:3
22.1
22.
38.0
1988
15.1
20135
11:01
16.9
18.2
21:38
25.8
41.3
10.35
WNDCG
50.0
55.1
2.4
03.4
24.3
10.1
10.33
22.4
22.1
22.7
38.0
41.0
10.8
9.8
14.1
19.33
10.8
16.3
21/8
25.3
45.2
15:.
aNDOG
56.6
20.2
21.1
24.7
56.5
63.9
mill
72.7
76.1
41.0
79.5
15.4
13.7
11.1
26.9
21.2
26.6
37.2
13.4
7.5
19:9
53.4
15.
15.6
27.5
00.7
250.
27.3
10.6
1511
58.
30.0
12.5
41.7
12.1
7.3
14:0
53.07
6.4
-0.1
-2.5
53
25.4
4571
2111
20.3
WNDCG
76.5
53.0
35.4
02.9
29.4
23.0
.0.3
22.4
94.5
20.5
31.0
41.5
12.4
7.5
13.5
53.8
-8.0
-3.5
5.0
25.8
43.3
27.4
29.
wil
16.8
60.6
26.1
+3.4
33-3
26-4
2.4
21.41
24.6
2012
32.4
41.3
13.2
6.11
14.1
21.6
55.6
-1.5
-2.9
1.5
27:3
46.2
21.8
11.44
MAP
16.3
16:1
21-3
2011
18.5
21.3
15.5
11:3
21.7
25-3
15:3
31.4
16.
6:1
15.8
-1.
3133
15.1
WNDCG
16.6
03.0
75.1
06.5
05.5
97.3
1531
19.2
21.0
38.0
21.7
75.0
19.4
31.1
13.4
16.3
-1.9
33.4
12:3
67.4
56.9
72.4
56.4
23.5
02.0
20.0
19.4
21.9
$8.0
44.6
25.0
25.1
20.2
$3.9
14.5
-7.8
02.4
19.7
aRR
78.9
20.7
21.4
17.6
-1.8
13:00
20.1
17.1
102
1031
21.5
15.7
1311
11.
2174
SNDCG
56.0
$1.8
13.4
05.3
20.1
1.35
2012
21.8
28.5
27.9
-4.9
80.5
4.5
18.9
14.1
-1.8
16.5
5.5
11.2
-633
122.7
with
78.6
50.4
30.1
60.7
13.9
13.9
14.9
22.1
50.5
16.6
36.9
-5.7
9.1
4.1
18.0
15.6
-1.6
10.5
6.7
8.8
-3.3
-0.1
21
WOTE PnTEt-qu-din di Marr--A--
map
12.3
005
-1-1
15.1
33:3
1817
31131
130
38.6
1978
117
11:33
3570
151
15:3
701
7552
10:55
'53'5"
WNDCG
58.6
66.2
4.3
05.5
33.0
25.4
22.6
22.5
10.6
24.7
38.6
46.0
20.5
24.7
20.7
91.4
12.7
17.5
10.5
25.5
25.8
00.7
31.4
21.8
mit
52.5
35.9
+1.9
45.3
35.01
27.1
30.01
2011
26.5
38.5
35.6
25.1
30.4
31.0
18.5
18:0
21.5
37.6
36.6
-3-3
00.6
maP
-2.-
08.3
15:
0.77
11:01
157
19.7
25.1
12.0
31:4
1931
30:35
2015
5114
14.7
35.0
15:3
2138
21.1
-310
70.6
70.5
33.7
14.6
16.1
5012
11.5
35.3
31.9
15.3
21.1
23.3
SAF
23.0
$5.0
25.0
25.2
21.0
WNDCG
10.7
-3.7
71.9
19.4
13.5
18.3
18.3
75.7
35.5
32.3
12.3
59.6
11.5
11.3
92.6
aRR
50.8
56.5
24.5
13.3
85.6
21.0
13.9
32.5
20.8
25.6
36.5
43.6
37.9
-7.4
25.4
01.
26.8
26.1
27.9
36.0
25.8
13.3
23.8
47.8
m
45.5
15.5
-3.5
15.3
256
0.33
2012
10.1
10.6
25.4
11.7
41.3
25.1
17.6
17.2
20.3
5.0
12:4
16.6
20.0
15.7
1531
21.33
23.35
WNDCG
16.5
50.7
-0.3
72:0
31.0
86:3
26.5
16:3
16.7
25.01
11.5
41.3
15.9
17:2
10.5
25:11
1.9
12.4
15.4
18.6
10.8
17:33
24.33
21.11
BRR
50.7
56.3
36.5
93.1
43:3
19:0
22.9
56.1
17.1
94.7
15.1
47.4
15.8
15.9
12.0
51.1
7.9
18.0
15.5
15.
19.6
15.1
25.7
25.0
MAP
11.0
14.0
-11.4
-4.1
31.5
15.5
11.3
2013
50.0
36.1
16.1
20-3
22.2
25:4
01.4
10
19.5
10.6
27.5
7.5
15.8
12.3
4.1
SNDDG
$5.0
50.7
-3.5
25.7
ab.
17.9
21.5
2117
26.8
43.4
25.6
21.3
25.3
20.0
16.7
21.4
18.5
26.7
19.3
19.7
91.6
aNDOG
64.5
60.9
81.2
21.2
17.9
-1.6
-1.4
5.4
15.8
6.7
mill
76.6
75.5
32.8
31.8
31.1
32..5
17.6
17.3
30.0
-1.2
-3.3
-1.0
5.5
6.9
194
26.3
detE-tteetttsertseeee Saze
BAP
19.5
18.7
11.2
8.5
1811
13.8
12.
11.3
122
0.5
-1.
332
-25
178
8.1
122
1.5
WNDCG
10.0
35.0
13.2
25.5
10.3
0.8
6.6
187
13.6
19.5
5.1
16.5
2.1
0.8
-1.0
-2.
-.8
500
-1.
-86
15.1
7.9
1.7
2.6
SIBR
28.6
21.2
15.5
24.3
8.3
0.8
2.6
15.1
18.8
12.0
5.0
18.0
1.5
0.8
-02
-19
1.5
02
-57
15.8
800
22
28
per instance. We therefore stress again the importance of using a reference set drawn from the same distribution as the test instances to avoid this kind of pitfall, since we remind that the number of positive documents per instance is unknown to the user.
D Additional Eiference-Basell Confidence Functions
In this section, we introduce other functions and approaches to tackle confidence assessment in the reference- basod setting.
D.1 Additional Regression- Based Confidence Functions
Uef and Umlp- def is based on a random forest (Ho) 1995) fitted using 100 independent estimators and squared error impurity criterion. 100 was chosen among several values (10, 50, 100, 200, 500, 1000) as the best compromise between efficiency and effectiveness regarding nAUC. Unlp is based on a Multi-Layer Perceptron (MLP) (Rumelhart et al.
We propose two additional regression- based confidence functions (Section 3.2 b
19

---

Published in Transactions on Machine Learning Research (09/2024)
Table 11: nAUCs (in %) for all reference-test relevant pairs (model: ember-vi, metric: mAP).
Method
Mmax
Vetd
Reference Set
Test Set
ScIDocs
ABKUTUNTU
23.8
21.7
8.1
24.6
STACKOVERFLON
19.6
28.6
47.6
-10.0
ALLOPROP
15.9
18.8
21.5
8.6
CRDQAVI
27.0
26.9
21.3
17.6
MMARCO
28.8
21.3
13.0
17.3
ABKUTU
ScIDoss
51.2
64.9
-4.3
62.8
STACKÖVERFLOOW
19.6
28.6
47.6
26.5
ALLOPROP
15.9
18.8
21.5
21.8
CMEDQAvI
27.0
26.9
21.3
30.1
MMARCO
28.8
21.3
13.0
29.9
STACKOVERFLOW
ScIDocs
51.2
64.9
15.5
UBKUBU
23.8
21.7
8.1
15.1
ALLOPROP
15.9
18.8
21.5
25.7
CMEDQAVI
27.0
26.9
21.3
28.8
MMARCO
28.8
21.3
13.0
18.0
ALLOPROF
ScIDocs
51.2
64.9
34.1
ASKUBUNTU
8.1
20.9
STACKÖVERFLOW
19.6
28.6
47.6
48.1
CMEDQAvI
27.0
26.9
21.3
32.7
MMARCO
28.8
21.3
13.0
21.4
CMEDQAVI
ScIDoes
51.2
64.9
-4.3
55.9
ASKUBUNTU
23.8
21.7
8.1
23.0
STACKOVERFLON
28.6
47.6
43.8
ALLOPROP
15.9
18.8
21.5
27.5
MMARCO
28.8
21.3
13.0
20.7
MMARCO
SciDos
51.2
64.9
-4.3
44.9
ASKUBUNTU
23.8
21.7
8.1
23.2
STACKÖVERFLOW
19.6
47.6
44.7
ALLOPROF
21.5
28.1
CMEDQAVI
27.0
26.9
21.3
33.8
1986) with one hidden layer size 128 (retained among several values: 32, 64, 128, 256), ReLU activation, and mean squared error loss function. 0.05 was chosen among multiple learning rate values (0.001, 0.005, 0.01, 0.05, 0.1), batch size equal to that of the reference was selected (one single iteration per epoch), and 500 training iterations were performed, it showed the best afficiency-effectiivness trade-off in terms of downstream nAUC.
Results using these two functions are reported in Table 12 logically outperform the reference-free baselines, they are still less effective than simple linear regression. This is not entirely surprising, given the limited data and constrained feature space at stake in this setup.
We observe that although these two new methods
D.2 Other Approaches
D.2.1 Classification-Basse Based Approach
Confidence assessment be classification problem in which the good instances constitute the positive class, the bad instances constitute the negative class and the rest belongs to neutral class. We therefore need to slightly modify reference set So (Fquation G to make it suitable for the classification setting. We thus build
Sglff - (z. 1 (g>m+y ()vm==)) (z, v) So).
(9)
where m > m E IR are the thresholds above and below which an instance is considered good and bad respectively. Then, we fit a probabilistic classifier IR* + [0, 1]3 on Sglr taking vector of sorted scores s(2) R input and returning the estimated probabilities of s(x) to belong to classes -1 (bad instances).
20

---

Published in Transactions on Machine Learning Research (09/2024)
0 (average instances) and 1 (good instances). We finally define confidence function welr such that for given unsorted vector of relevance scores Mer((z) = #2 (s(z)) To (s(z)) (probability that stems from good instance minus the probability that z stems from a bad instance). The higher uar(z), the more confident we are that z E Rh stems from a good instance.
In this work, we use a logistic regressor (Cox Snell 1989) with l regularization parameter 0.1, Wlog and consider by default that the top and bottom 25% instances with respect to metric function m represent the good and bad instances respectively. We challenge this parameter in Appendix D.3.2
D.2.2 Bestance-Based Approach
Confidence estimation can also be seen as a distance problem. While Ujin and Mog are fully supervised, it is possible to imagine a distance-based confidence function Udist that evaluates how close an instance to the set of good instances and how far it is from the set of bad instances.
For this purpose, based on thresholds m and m defined in Section D.2.1 we build z and Z the sets composed of good and bad instances respectively. Formally,
2 = {z (z, v) E So. y> mm) Z = 2 (z. y) E So y < m
Then, we construct a confidence scorer Udist based on the distributions observed Zg and ZØ We define Udist = 8 (s()), 2/) 8 (s(.), 2 as the difference between the distance the of bad instances and the distance to the of good instances 12 Intuitively, the higher udiat (z), the more confident we are that z stems from good instance.
In this experiment, we rely the Mahalanobis distance (Mahalanobis 2021). We compute the Mahalanobis distance between set Z C 1 and Z E RR as
2018 Lee et al. 2018 Ren et al.
8mah (z, 2) = (z uz) Dz (z -11z)
where jaz E &R is the average vector of reference set Z and Σz E B is the inverse covariance matrix.
D.3
Experimental Results
D.3.1
Abstention Performance
shows the normalized AUCs averaged model-wise for each dataset of the benchmark. We globally see that Ujin remains the best reference- based method, although Wlog and Umah both show satisfactory results, outperforming the reference-[ree baselines.
Table 12
D.3.2 Impact of Instance Qualification
In this section, analyze the impact of the instance qualification threshold on the performance of Moo and Umah confidence functions. More precisely, the idea of this experiment is to vary the qualification threshold 13 and to test its impact on nAUC over the whole benchmark. For the sake of readability, we present the results for the ember-v1 model and for the mAP only (sce Figure 7
The first observation we can make is that the confidence functions Mlog and Umahh are sensitive to variations the qualification threshold. For all the datasets presented, is noticed that both methods performs better for thresholds between 30% and 40%. Intuitively, too low threshold would lead to considering too few instances as good bad, making it difficult to provide proper characterization and therefore resulting in
12 As for the other reference-busod functions, Maiss takes vector of unsorted inside the function.
input, sorting being encompassed
13 As reminder, a threshold of 10% means that good instances represent the top-10% with regard to the metric of interest, while bad instances represent the 1aatom-10%.
21

---

Published in Transactions on Machine Learning Research (09/2024)
Table 12: nAUCs (in %) averaged over the whole set of available models, for every method including Wlog and Umah-
Method Metric
Mmna
Mard
M1-2
Wila
Mmupp
Mat
Miog
Mmah
Dataset
ScIDoes
mAP
49.2
58.7
4.7
65.4
65.6
67.3
68.8
65.8
mNDCG
54.9
62.9
10.2
66.6
65.3
68.6
70.2
66.8
80.2
80.3
41.3
80.5
76.8
70.7
78.3
ABKUTUNTU
mAP
24.1
16.2
7.3
22.0
19.4
14.1
19.9
10.1
mNDCG
28.1
15.7
8.6
24.5
21.2
14.0
22.0
10.5
mRR
32.7
16.6
14.0
26.2
22.8
14.8
24.1
11.3
STACKOVBRFLOW
MAP
20.7
24.6
35.7
42.6
39.5
37.1
42.0
37.5
NDCC
21.0
25.0
35.6
42.6
36.0
37.1
42.1
37.4
24.8
42.8
39.6
37.3
42.2
37.7
ALLOPROF
mAP
16.6
17.0
19.3
29.7
25.6
24.3
29.3
25.1
mNDCG
16.8
16.9
18.8
29.3
23.7
24.1
28.9
24.9
mRR
16.9
16.7
19.1
28.9
24.9
23.5
28.6
24.3
CMEDQAVI
mAP
24.4
23.3
22.3
30.6
23.3
24.8
29.0
25.7
mNDCG
25.0
23.5
22.3
30.7
22.2
25.4
27.0
24.8
26.3
24.2
23.4
31.5
24.7
25.5
30,2
25.5
MMARCO
mAP
30.1
31.0
39.4
34.0
32.0
29.7
25.1
22.7
MNDCG
30.5
30.9
38.8
34.1
30.3
28.7
25.1
23.3
mRR
30.1
31.0
39.6
34.3
31.8
29.4
23.7
22.8
Average
mAP
27.5
28.5
21.5
37.4
34.2
32.9
35.7
31.1
mNDCG
29.4
38.0
33.0
35.9
31.3
mRR
34.6
32.3
28.9
40.7
36.8
33.5
37.9
31.2
SciDocs
AskUbuntu
StackOverflow
0.78
0.4
0.70
0.77
0.75
0.75
0.74
0.400
0425
0.1
Alloprof
CMedQAv1
Mmarco
0.300
0.8
0.275
0.40
0.4
0.226
0.200
0.30
0.0
UlAn
0.176
Ulog
Umah
-0.4
0.4
Qualification Threshold
Qualification Threshold
Qualification Threshoid
Figure nAUC vs. instance quantile threshold for all datascts (model: ember-v1, metric: mAP).
less relevant confidence estimation. Conversely, a higher threshold creates less specific classes but with more instances, which seems to be better suited to our use case.
Additionally, while umah performs less well than uin overall, Wlog competes fairly well when its qualification threshold is optimized. But the slight transformation required for the Sa dataset (Equation 9 makes it little less competitive in terms of computation time.
22


Reviews:
[{"review_id": "7bMzDuWqxi", "reviewer": "Reviewer_7qdD", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "* The biggest strength of this paper is that it is practically applicable and can be quite useful to study for various real-world applications - especially with different text/multi-modal embeddings model providers that are being built.\n* The proposed method (both reference-free and data-drive) are quite simple, intuitive and easy to follow and implement.\n* The benchmark that was built for this paper can also be quite useful to study the overall abstention property, both with black-box and white-box embedding/retrieval algorithms.\n\n\n* The main weakness of this paper is that the proposed method does not have enough novelty. There is novelty and practicality in the topic that this paper explores and also how the benchmark was created - but the core method feels like somewhat of a trivial baseline.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, authors address the problem of defining a strategy for abstention within the context of information retrieval. Additionally, to increase the practical applicability of their setup, they propose to work in a setting where the retrieval model might be a black-box/API, which does not allow the option to fine-tune or retrain the model. As no prior work exists in this particular setup, authors contribute in two aspects - i) proposing an algorithm to solve the problem and ii) define a benchmark which can be used to evaluate the proposed method.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "As mentioned above, the paper overall is nicely written, has practical applications and can be useful. However, the core contribution - the algorithm does not have enough novelty. At this point, I do not have a lot of suggestions to provide that does not involve changing the core algorithm or adding some non-trivial technical contributions - which would then fundamentally change the paper.", "broader_impact_concerns": "N/A.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "x59ItUWX6A", "reviewer": "Reviewer_vRAv", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThe paper is well written and easy to understand. \n\nThe motivation is clearly defined and the methods are quite simple and easy to implement.\n\nThe experimental setup is carefully designed and experimental results are convincing.\n\nWeakness:\n\nThe proposed method is really simple and I'm wondering whether more sophisticated functions such as MLP neural nets will give better results than the linear regression function. Some experiments should be conducted.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a simple but effective abstention method for reranking for neural information retrieval system with realistic industrial application constraints. Two methods to compute the confidence scores are proposed, one is for reference free situation which uses various ad hoc functions , the other is for data driven situation which uses linear regression, classification or distance functions. The experiments are carefully designed and extensive conducted to show the effectiveness of the proposed approach.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Some experiments using MLP or other neural nets instead of  the linear regression function should be conducted.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "yNLxQfNfC1", "reviewer": "Reviewer_i7z9", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strength\nAlthough the abstention mechanism (also referred to as selective prediction) has been widely studied, this work is among the first to introduce such a mechanism to information retrieval.\n- The proposed methods, both reference-free and data-driven, do not induce much extra computation.\n## Weakness\nConsidering that the reference-based (also referred to as data-drive) method utilizes a separate model, the zero incurred cost claim is a bit overstated.\n- The literature on *selective prediction* is absent. \n- The finding that abstention performance is correlated with ranker performance is not surprising as better rankers can compute better-calibrated relevance scores. This actually induces another concern: the performance of the proposed method highly depends on the ranker performance, making it hard to select the threshold $\\tau$ in practical settings. For instance, if the ranker is good, a low abstention rate could obtain good performance, while with a bad ranker, even a high abstention rate could underperform. \n## Question\n- How is reranking in this work different from retrieval?\n- What metric $m$ is used? Do the findings hold for other metrics?\n- Why is the $\\tilde{u}$ curve in Figure 2 constant?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work proposes an abstention mechanism for neural information retrieval (NIR) and the corresponding evaluation metric. Two kinds of uncertainty functions are proposed to add the abstention mechanism, namely *reference-free* and *data-driven*. According to the empirical evaluation in Section 5, the proposed methods, especially $U_{\\rm lin}$, can achieve good performance-abstention curves.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "# Requested Changes\n- As Figure 1 is an illustration of the proposed method, consider moving it to earlier pages.", "broader_impact_concerns": "The limitations are discussed.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper describes and studies empirically methods for abstention in reranking in the context of information retrieval. Specifically, the authors consider two settings: reference free and data driven. In the reference free scenario, the authors do not assume access to any labels on the relevance of documents to a query, and the proposed methods are heuristics based on the predicted relevance scores. In the  data driven scenario, a linear mapping (or other parametric mappings) can be learned to convert the predicted relevance scores to a confidence score for abstention.

The methods discussed in the paper are simple, but the authors perform a thorough empirical study, and also propose a new benchmark. The authors also addressed the substantial comments of the reviewers in the manuscript, and the only remaining criticism of the paper is limited novelty, which is not required for acceptance to TMLR.
Decision: Accept (poster)