Learning to Reject with a Fixed Predictor: Application to Decontextualization
Authors: Christopher Mohri
Daniel Andor
Eunsol Choi
Michael Collins
Anqi Mao
Yutao Zhong
Venue: ICLR 2024 poster (2024)
Keywords: Rejection
abstention
loss function
consistency
learning theory
decontextualization
natural language processing
Primary area: general machine learning (i.e., none of the above)
TL;DR: We introduce a new loss function for learning a rejector with a fixed predictor, and focus on the decontextualization task.

Abstract: We study the problem of classification with a reject option for a fixed predictor, crucial to natural language processing. We introduce a new problem formulation for this scenario, and an algorithm minimizing a new surrogate loss function. We provide a complete theoretical analysis of the surrogate loss function with a strong $H$-consistency guarantee. For evaluation, we choose the \textit{decontextualization} task, and provide a manually-labelled dataset of $2\mathord,000$ examples. Our algorithm significantly outperforms the baselines considered, with a $\sim 25$% improvement in coverage when halving the error rate, which is only $\sim 3$% away from the theoretical limit.

Published conference paper at ICLR 2024
LEARNING TO REJECT WITH A FIXED PREDICTOR: APPLICATION TO DECONTEXTUALIZATION
Christopher Mohri', Daniel Andor Eunsol Choi², Michael Collins", Angi Mao', Yutao Zhong* 'Stanford University, Google, 'The University of Texas at Austin, *Courant Institute xmohri @sranford. edu, (danielandien micoll lins) @google. com
Sunsol@@utexas edu, faqmao, rutao)@cims nyu. edu
ABSTRACT
We study the problem of classification with reject option for fixed predictor, crucial to natural language processing. We introduce a new problem formulation for this scenario, and algorithm minimizing new surrogate loss function. We provide a complete theoretical analysis of the surrogate loss function with a strong H--consisencyy guarantee. For evaluation, we choose the decontextualization task, and provide a manually-labellle dataset of 2,000 examples. Our algorithm signif- icantly outperforms the baselines considered, with 25% improvement in cov- erage when halving the error rate, which is only ~ <3% away from the theoretical limit.
INTRODUCTION
Large language models, often trained with billions of parameters, have achieved impressive perfor- mance in recent years (Raffel et al., 2020) and are used in a wide variety of natural language gen- eration tasks. However, their output is sometimes undesirable, with hallucinated content (Maynez et al., 2020; Filippova, 2020), and much work remains to fully understand their properties. In many applications, such healthcare, question-answering systems, customer service, incorrect predic- tions are particularly costly and must be avoided. This motivates the design of algorithms for large language models and other natural language processing (NLP) tasks that achieve high precision on large fraction of the input set, while abstaining on the rest. How can we devise such accurate models that allow a reject option?
A common technique adopted in the past is that of confidence-base models, where rejection is defined by some threshold on the predictor's scores and admits fixed cost (Hendrickx al., 2021). Chow (1957; 1970) was the first to provide analysis of the trade-off between error and rejection rate, as well as the associated Bayes optimal solution. The rejection rule was later studied based on the receiver operating characteristic (ROC) curve (Tortorella, 2000; Santos-Pereira and Pires, 2005; Pietraszek, 2007; Landgrebe et al., 2006). Some subsequent work has focused on minimizing surrogate loss functions for the cost-hased objective, with various theoretical guarantees (Bartlett and Wegkamp, 2008; Grandvalet et al., 2008; Yuan and Wegkamp, 2010). In NLP, it has been reported that scores from popular large language models such as T5, BART, and GPT-2 are poorly calibrated (Jiang et al., 2020; Kumar and Sarawagi. 2019; Lewis et al., 2019; Raffel et al., 2020), similar to modern neural networks (Guo et al., 2017), but Xin et al. (2021) proposed simple regularization trick during training to improve these scores. The method used by several other authors can also be viewed as an instance of confidence-bel based models (Kamath et al., 2020; Zhang al., 2021: Garg and Moschitti, 2021; Dong et al., 2018; Varshney et al., 2022; Li et al., 2024; Chen et al., 2024). Here, the idea consists of first learning a scoring function defined over pairs (æ, y). and next applying the confidence- basedd technique to the scoring function.
However, as shown by Cortes et al. (2016)[see Figure 2]. straightforward confidence-bace methods are in general suboptimal. When the predictor learned is not the Bayes optimal solution, in general, more complex rejection rule is needed to achieve better performance. The authors suggested instead seeking suitable rejector out of family of rejection functions that may be richer than that of confidence- based threshold ones. They gave theory and algorithms for learning the predictor and rejector simultaneously by minimizing rejection loss, whose definition takes into account the cost

---

Published a conference paper at ICLR 2024
of rejection, c. As shown by Ni al. (2019), extending this predictor-rejector framework to the multi-class setting is challenging and remains an open problem, which has been solved positively by Mao et al. (2024a;c) and extended to multi-expert deferral (Mao et al., 2023a; 2024b). Instead, Charoenphakdee et al. (2021) decomposed multi-class abstention into multiple one-versus-all binary classification problems from a cost-sensitive point of view. Recently, such cost-sensitive approach has been shown empirically to be inferior to the latest approach for abstention (Cao et al., 2022).
We aim to design accurate models with a rejection option for NLP generation tasks such as decon- textualization (Choi et al., 2021). Decontextualization involves editing sentence within a passage such that it can stand alone. It is critical in this task to make confident predictions, or abstain; if are to edit author's words, must be sure do so correctly. One solution would consist of adopting the rejection loss function of Cortes et al. (2016; 2023) and of learning, simultaneously. a predictor f and a rejector that minimize the rejection loss. However, for NLP tasks such decon- textualization, this faces a key obstacle: the full set of accurate outputs for a given input sentence may be large and is typically not at the learner' disposal. For example, in decontextualization, some standard transformations such as passivation applied to an accurate output can immediately lead to a large number of other accurate outputs. To minimize the rejection loss, however, the learner must be able to evaluate the correctness of any potential predictor on any example in the training sample. But, apart from the single label in the training sample, other potential accurate labels are not pro- vided and thus the correctness of a potential predictor cannot be checked. How can we then learn an accurate snpection-based model for such settings?
One way to proceed is instead to first learn predictor f. using the standard techniques adopted for large language models, for example by minimizing the cross-entrope loss in next-token prediction. Next, given that predictor, it is not hard to manually assign a binary label to the output of f on some relatively small set of held-out examples indicating their correctness. The problem of interest then consists of finding the best rejector function r minimizing the rejection loss with fixed. Here, the rejector function takes the input æ together with the output fx) as its own input. We refer to this procedure learning to reject with a fixed predictor f.
The resulting binary rejection loss function for r is hard to optimize directly and, instead, we need to resort to a surrogate loss. Cortes et al. (2016) gave a consistent surrogate loss for their full rejection loss. which involved learning both f and r. However, since f is fixed in our context, we cannot benefit from the corresponding consistency guarantees. Instead, we first propose a parametric family of surrogate losses inspired by their work for learning r alone. Next, we prove that these surrogate losses benefit from strong consistency results, when a suitable constraint holds for the parameters.
Our results make use of the recent work of Awasthi et al. (2022a), which gave general tools for deriving K-consistency bounds. These are bounds relating directly the excess error or estimation error of the surrogate loss to those of the original loss (here the rejection loss). Thus, they are stronger guarantees than asymptotic Bayes-consistengy results. Furthermore, they can be extended to other hypothesis sets H than that of the family of all measurable functions. We use those tools to prove the first H-consistency bound for our surrogate rejection loss, which provides a strong justification for its adoption for tackling the problem of interest.
The rest of this paper is organized as follows. In Section 2, we formulate our learning problem, which consists of learning rejector given fixed predictor. In Section 3, we introduce confidence- based models and contrast them with our two-step model. In Section 4, we derive our surrogate rejection loss. In Section 5, we prove a strong K-consistency bound for the proposed surrogate rejection loss. In Section 6, we give a description of the decontextualizztioon task and the annotation procedure. Section reports the experimental results of our surrogate rejection loss, and shows that it compares favorably with several baselines in a decontextualization task.
2 LEARNING PROBLEM
We consider the problem of sequencceto-ssequenee modeling with high confidence in natural lan- guage processing. The general objective is to design an algorithm that only returns an output when it is highly likely to be correct, while still guaranteeing high coverage.
Let X denote the input and y the output set of sequences and let D be a distribution over X x y. The problem can be formalized in terms of
predictor f:X. - y and rejector
2

---

Published conference paper at ICLR 2024
r:X y R. A non-positive sign for the output of the rejector is interpreted as rejection, and a positive one as acceptance. Formally, on an input e X,, we have
[f()), if if f()) >0 reject, if ((x) s O.
(frr)()
Given a hypothesis set of sequence-to-sequenee predictors and a family of rejectors R, for a cost c [0, 1]. the natural target loss for the abstention problem can be formulated as follows:
r,, [())= (=)))(=)))= +C +c))(s)())))
We wish to minimize the error (Ty(w)=w) on accepted examples (Tr(s,(==))o)) while imposing fixed cost on rejected examples. As the cost rejection increases, one can expect a higher cover- age, but lower precision. One difference with respect to the framework of Cortes et al. (2016; 2023) that, here, the rejector takes as argument the prediction as well. More generally, as already pointed out in that previous work, the rejection cost can be function of x and f(x), not just constant. The problem of selective classification (Gelbhart and EI- Yaniv, 2019), which is based on the choice of threshold function, be viewed special case of this framework.
Distribution model. Before describing any method for tackling this problem, we wish to discuss the distributional model adopted certain NLP tasks such decontextualization. In standard learning tasks, there may be multiple correct labels for the same input & and with an i...d. training sample, expect come across all of these labels with their conditional probabilities given st
In complex NLP tasks such as decontextualization however, there may be a relatively large number of correct yss for given z: the task is highly non-deterministic. As discussed, some standard transformations such as passivation applied to one can immediately lead to a much larger number of correct output sentences. Thus, it not realistic to demand from labelers to supply all possible correct sentences for a given input x. On the other hand, we do not wish to consider it to be an error if a model returns a desirable sentence that does not exactly match any of the labels provided by the labelers.
What should be the correct distributional model to adopt for such NL.P tasks? We can consider two models: deterministic model where only single label is accepted as correct for any input sentence; or a more general and more useful stochastic or non-deteriinistic model where, in addition to the single label y (or few labels) provided by labelers, any other label y returned by the model is viewed as correct provided that y' is sufficiently similar to y. based on some pre-defined similarity measure. It is important to note that this pre-defined similarity measure may be difficult to specify, and may therefore require expert human annotation.
Adopting the non-deterministic model, for (2. D and given f e F, amend our indicator function T [x)))y measuring incorrectness in (1). Instead, we measure {(=)) where Ar.y S y implements some similarity measure and describes a set of acceptable outputs y. Provided that we have a boolean random variable a {-1,+1 derived from (z, v) and f(z) indicating membership to Asyy the event where fx) is an acceptable output, we can simplify this indicator function to Ta=-1 Since ff is fixed, we remove it as an argument to r, and the distribution over (z, v) D leads to distribution (xc a) ~ DD induced by f. We will refer to the following as the induced rejection loss defined for any rejector r and pair (I, a):
((r,x, T---1 Tr(=)>o +cIT(2)00-
In the following, we will distinguish two methods for learning the rejection function r: the so-called confidence-buse based method where r is simply defined as a threshold based on the predictor f, and two-step learning method where first f learned and then r.
LEARNING METHODS
In this section, we describe in more detail the two learning methods previously mentioned.
3.1 CONFIDENCE-BASE BASED METHOD
The confidence-baseee method is perhaps the most commonly used one to define a rejection function.
Let f(x) argmax,, sxx, y) for some scoring function X x R, which could be p(ylo;w),
3

---

Published conference paper at ICLR 2024
where w represents model parameters. Then, a threshold value 0 is set based on some function of the scores s(.. y) assigned to each y e y for given x E X. If s; and 83 are the final scores assigned
to (Ii, D and (zyz Uj ~ DD respectively, we wish to have monotonicity: 8, 2 83 <> 11 (+x))o) S (ay)))y (Geifman and El-Yaniv, 2017).
The MaxProb method is simply defined in terms of the highest score (Hendrycks and Gimpel, 2016). In that case, the rejection function is function mapping from X to RR defined by r(x) sx,, f(x))- 8, for some choice of the threshold f E R. Another popular scoring function is based on Monte- Carlo dropout (Smith and Gal, 2018; Gal and Ghahramani, 2016), measuring statistics such as the mean or negative variance of the scores s (z. 3). Fitting the threshold to guarantee that it matches a target precision has been studied in (Geifman and EI- Yaniv, 2017). In our experiments, we follow simpler procedure, as this is not our focus; we are interested in the quality of the underlying scores.
3.2 TWO-STEP METHOD
In the two-step method, a predictor f is first learned by minimizing surrogate loss for I (a)) such as the cross-entropy loss. Next, the rejection function r is learned as a binary classifier.
Note that, to learn in the non-deterministic distributional model requires binary labels for pairs (x, (xr)) indicating if the output sequence (xx) is indeed a good label for ær, or formally, if [(x) Ag.y- As already discussed, that information cannot be directly derived from the label y of z in the training sample since the correct labels are typically not unique in NLP tasks. One way to derive that information is to manually label such pairs. This admits two disadvantages: the manually assigned labels are specific to the predictor f previously learned and thus cannot be reused for different predictors; and of course this requires manual labeling which is typically costly. However, if the cost is not too significant, one can label a moderately-sized dataset and then train a classifier on it.
One approach for training such classifier is use the standard cross-entropy loss function. More specifically, for pairs ((x, f(x)), a), where a represents the label or annotation, one can train with direct supervision using the binary cross-entropy loss. However, similar the MaxProb method, this also requires setting a threshold 0 for acceptance based on model scores. One can only hope that the classifier produces higher-quality scores, where quality is associated with monotonicity. Thus, both methods are based on straightforward threshold rejection. Additionally, to the best of our knowledge, minimizing the cross-entropy loss does not have any proven guarantee with respect to our main objective: minimizing the induced rejection loss. In the next section, we tackle both of these problems: we introduce a new loss function with a built-in threshold that directly minimizes the induced rejection loss.
In Appendix C, we briefly discuss a comparison to a cost-sensitive classification formulation. This solution is reported to be inferior to the sfatc-of-the-art empirically (Cao et al., 2022) and is not proven to benefit from an K-consistency bound, unlike the solution we present in the next sections.
4 SURROGATE REJECTION LOSS
Cortes et al. (2016; 2023) study the joint optimization problem in (1) where the predictor ʃ is binary classifier, and define a convex surrogate loss upper-bounding their rejection loss L. We use the same technique to upper-bound the induced rejection loss. Specifically, the following inequality holds, where a and / are positive parameters and I + o(-2) and I + vb-x) are convex functions upper-bounding lzzsp:
((r,210) Ta--1 Tr(o)= t +cIn( Tago Traa +c m<))0
I max (Teso T=(e) e Treess) s max max maa
Next, since the maximum is lower-bounded by the average, we can write:
a1), s max(1e-ee .o cllc))ss]= maxx{o==ae co cHar(s)ss)
< maxx (max(x(((((2)) - a))) cx(-))(x)))) ((((()) -a) +
4

---

Published conference paper at ICLR 2024
We use the exponential function for & and v. giving our surrogate rejection loss function:
((r, æ. a) 5 (lr())aal) + ce ce-9r(x),
While the induced rejection loss 2,) is provably NP-hard to optimize, our surrogate loss is convex and differentiable. A key insight is that models optimized with our loss function have a built-in threshold of O; they directly optimized for a specific precision. Thus, there is need to further specify some score threshold as in all methods previously described. In those methods, one can still target certain precision levels through the choice of threshold, but the underlying scores not necessarily favorable for that precision level.
While Cortes al. (2016; 2023) proved theoretical guarantees for a joint minimization of their loss function in their binary setting, these do not naturally apply to our problem (the predictor usually does not have zero error and is not the Bayes predictor). In the next section, we prove strong theoretical guarantees for minimizing our surrogate loss function.
K-CONSISTENCY BOUND
In this section we prove H-coosistency bounds, concept introduced by Awasthi et al. (2022a:b), of our surrogate rejection loss function with respect to the induced rejection loss. To the best of our knowledge, these non-asymptotic bounds are the strongest guarantees known regarding the mini- mization of surrogate loss functions (Awasthi et al., 2021a;b; 2023; 2024; Mao et al., 2023b:d:e;c;). We first introduce some basic concepts and adopt the notation of Awasthi et al. (2022a).
5.1 PRELIMINARIES
Let X denote the input space, y = (-1,+11 the binary label space, and 'D distribution over X x y. Let R denote a family of rejection functions mapping from X R. Then, the generalization error (e(r) and minimal generalization error Ri R for loss function E(r, xr y) are defined by
Re(r)
E [E(] 2) and R2 inf Re(r). (ar,y)-
rER
We will adopt the standard notation for the conditional distribution of Y = 1 given X = r: (x)= D(Y 1 | z). The generalization error can be expressed as R&(r) = Ex(e((r,))) where (r(r,x) is the conditional ê-risk defined by Cr(r, = n(x)((r, +1) (1
-1). The minimal conditional E-risk is denoted by C7,x (2) = infrex Ce (r. We also use the following shorthand for the gap (r,
()) ei and refer to it as calibration gap.
A key quantity that appears in their bounds is the (e, R) -iiiimmizizabilitt gap My which is the difference of the best-in class error and the expectation of the minimal conditional é-risk:
Me Rj
E X
This is inherent property of the hypothesis set R and distribution D) that cannot hope to estimate or minimize. As discussed later, the minimizability gap is zero when R is the family of all measurable functions Rall. For simplicity, we omit dependency on R in all notation when R Rall-
5.2 DEFINITION OF THE LOSSES AND THE DESIRED GUARANTEE
We consider the induced rejection loss function & = E defined for any rejection function or rejector r: R and (I,a) ε X x 11) by (2(r,
Tg [=(=)> +c We will consider a surrogate loss function & parameterized by a, 3 > 0 and defined for any rejection function or rejector r: - RR and (I, a) ε X x (-1,+1) by Pi (r, a) ( + ce -Br(z)) We will prove K-consistency bounds for the surrogate loss é with respect to the target loss l2, when r is in the family of all measurable functions Rall These are excess emro bounds of the form Reg (r) - R2, s
fRe, (r) -R,,, ) valid for all r for an increasing function f. To do so, we will use the following general theorem from (Awasthi al., 2022a).
Theorem 1. Assume that there exists a convex function V: R, + IR with (0) = 0 such that the fol- lowing holds for all R and x X: ('AEe, (r, AC,, (r, Then, the following inequality holds for any r e R:
V (r) Rig. M),x)) RR, (r) Ri + Mr
5

---

Published conference paper at ICLR 2024
0 Y,
Figure 1: Lower bound for AC The green denotes the values r(z) can take. Left: r(x) s 0; Right: 20) both eases, the infimum is attained at r(x) O.
As shown by Steinwart (2007, Lemma 2.5), since (1, and (2 x, can expressed in terms of r(z) and alone, both minimizability gaps MR, and Me,R,, vanish (Me, Run Me,.R.n 0). To make use of this theorem, in the next sections, we will derive the expression of the calibration gaps AC, AC2, and ACe,-
5.3 CALIBRATION GAPS
The following gives the expression of the calibration gaps. The proofs for Lemma 2 and Lemma 3 are deferred to Appendix A.1 and Appendix A.2 respectively.
Lemma 2. The Bayes solution r' for the rejection loss be expressed for all X by r" (2) (2) (1 - c). The calibration gap for the rejection loss is given for any r Rall and x E X by
ACt, (r, (n())) (1-c)] (re))
Lemma 3. Let (x) be defined by (x) (T))e ( + (1 n()))e and define 7 by 7 a+23
Then, the calibration gap for the surrogate loss is given for any E Rall and x X by
28c
(r, e In(x) + cE -r(x))
1-7 (x)
5.4 MAIN RESULT
In this section, we present our main result. A key challenge in finding function Yi relating the two calibration gaps is that AC,, depends on the value r(z) R, while AC2, only depends on the sign of (x)), via that of r" (x). The following provides key solution to this problem.
Proposition 4. Assume that there exists a convex function V.R I with (0) 0 such that the following holds for all e Rall and x X. V (In(x)) - (1 c) (a)))) AC,, (0, Let Ta be
defined by Io = e (1-c)e- c)e and assume that 23c I.. Then, for any r Ran:
(r) R22,) S Re, -r)- R
(3)
The proof is presented in Appendix A.3. The result shows that, instead, we only need find function V relating the gap AC,, to ACe, (0, x), which is no longer a quantity depending on r(x). To do so, we look to lower-bound AC,, over the infimum r(x). Since AC,, a (strictly) convex function of r=x), if can select the parameters and 3 to ensure r" ('xx)> > <> ro(x) 0, where To is the Bayes solution for the surrogate rejection loss, then this infimum occurs at r(x) O. This is illustrated in Figure 1. Proposition 4 states that this can be arranged if a and B are related by 2.c Te. In view of this proposition, will adopt the assumption 3 T. and analyze ACe, (0, Note that the equivalence proven in the proof holds if and only if this equality holds.
2. Then, the following Theorem 5. Let ax, 3 > be such that 23c Io where lo = =e (1 (1-c)e -)
inequality holds for any r e Rall:
(c+ Io)e (Re, (r) Rí,
R (r) R2, S
See Appendix A.3 for the proof. The theorem shows that if the excess surrogate loss (Re, (r) Ri reduced to e, then the excess rejection loss (Re, (r) RR;) is bounded by ((Va). This provides strong guarantee for the surrogate rejection loss function proposed when the condition 28c T

---

Published conference paper at ICLR 2024
Rejection
Surragate Rejection
temm
1055
less
2-
100
((2)
Figure 2: Surrogate rejection loss function of r(x:) when 0.05, a and B following 29c Ic. with both terms in the sum. Left: negatively-labelled points (-1). Right: positively- labelled points (+1).
holds. Similar results can be derived for other families functions R, such as that linear functions neural networks with one hidden-layer in (Awasthi al., 2022a) (see Appendix D). This gives a principled method for defining the relation between a and f3. The value of the other parameter, say o, be set arbitrarily via hyper-parameter search.
5.5 VISUALIZATION OF SURROGATE LOSS
In Figure 2, we plot our surrogate loss as function of r(x) on (-0.5, +0.5), and arbitrarily choose
e 0.05 and a = 2 with 3 following the relationship defined in 20c Ig- We include the plot for both negatively-annotae
points (a -1) and positively-annotated points (a = +1). The first term always increasing, and the second is always decreasing.
negatively-annotatee points, minimum attained at r(x) O, and for positively-annotated points, the minimum is attained at r(z) 0. The following is key insight from our DC-consistency bound: this property holds for any c, and any a and / satisfying 28c Je, as the signs of r' (x) and ro(r) match. This relationship thus ensures that the minimums of both plots are in the proper regions.
We observe the following property:
DECONTEXTUALFZATIE
TASK
We choose the NLP task of decontextualization as primary case study. This is challenging task because only modest amount annotated data is available and because each input typically admits large number of correct labels. We give description of the task and our annotation procedure.
6.1 DEFINITION
Decontextualization is the task editing a sentence within passage so that it can be interpretable out of context (Choi et al., 2021). Specifically, given sentence-context pair (s,c). sentence s' is valid decontextualization of if: (1) the sentence s' is interpretable in the empty context; and (2) the truth-conditional meaning of s' in the empty context is the same the truth-conditional meaning of in context We refer readers to (Choi et al., 2021) for full description.
6.2 ANNOTATION
For our experiments, we labeled 2,000 decontextualizations of a fixed MT5 XXL model (Xue al., 2020) ourselves, fine-tuned on the decontextualization task. The training data for the original decon- textualization task is sample from the English portion of the Wikipedia corpus (Choi al., 2021). The input is formed by concatenating the title and subtitle of the relevant page with '[HEAD]', and then appending the relevant paragraph after '[SEP]' see Figure 3. Our 2,000 annotated decontex- tualizations originally from random sample of English Wikipedia. Examples that the model labelled as 'impossible" or "unnecessary' were not considered. We observe that annotating for this task is difficult: some take several minutes to evaluate.
When labeling or evaluating the validity of a decomtextualization, we consider the correctness of the edits: the added information must be correct, and the deletions cannot change the meaning of

---

Published conference paper at ICLR 2024
Goldie Taylor [HEADD Career; Corporate [SEP] Taylor has worked for the Sara Lee Corporation director of global communications and public affairs. Goldie Taylor has served executive consultant to NBC News and CNN Worldwide.
Figure Decontextuullization labeled 'title".
the sentence. Sometimes, however, this is impossible to discern without using information from the title or subtitle. We thus labeled the outputs as 'yes', 'no' or 'title'. We present 'title' example in Figure 3. The decontextualization request is for the bolded sentence, and 'Goldie" is then inserted. However, 'Goldie' only appears in the title. While it is probable that 'Taylor' refers to 'Goldie Taylor', we must rely on the information from the title. It is however also possible that "Taylor' refers to family member of "Goldie Taylor' and that the paragraph entirely unrelated to the title.
In our case, since 'title" examples are likely to be factual (while unsupported by the context provided to the model), we evaluate experimentally by including them with 'yes'. In other setting such as novels, 'title' examples are less likely to be factual, paragraphs deep within chapter have little connection to their title.
7 EXPERIMENTAL EVALUATION
In this section, report results for the deseribed learning methods.
7.1 DATASET
We randomly split our 2.000 annotation examples into 1,500 train/500 validation examples and perform 4-fold cross-validation. 1,019 (50.95%) of the annotations are "yes', 761 (38.05%) are 'title', and the remaining 220 (11.00%) are 'no.' As already mentioned, we consider the 'title' examples as 'yes", so we have about 89% positive examples. The decontextualizatin rejection task constructed as follows: we concatenate the input and output the decontextualizztton model with the token '[OUT]' to form the input. The target consists of just one token, 'yes' or 'no.'
7.2 METHOSS
Maxprob: We the score that the fixed MT5 XXL predictor assigned to its own output sequence. The best threshold for some precision level is determined on the training data, and then evaluated on the vali- dation data for both coverage and precision. new threshold is determined for each split.
Coverage
Cross-entropy loss:
We further fine-tune a T5X
1.1 XXL decontextualization model (Roberts et al.,
2022), limited to one output token, on the decontex- tualization rejection task, and use the score the value of the logits yes The standard cross-entropy loss function is used, and threshold is similarly fit- ted on half of the validation data and evaluated on the other half. We perform hyper-paramett search over {le - 4, le - 3, le 2} for the learning rate, and (0, 0.05, ... 0.2) for the dropout rate.
0.4
Theoretical Limit
Maxprob
0.2
Crass-en ent
Rejection loss
0.94 0.96
1.00
Precision
Figure 4: Precision coverage decon- textualization. Standard deviations for both precision and coverage from the 4 cross- validation splits.
Surrogate loss: In our formulation, we have re- jector r: F. Thus, convert the output of TSX model to real number for r(x). we simply use r(x) = softmax (oottmax(oggis(z)))) (logits(z)) 0.5. We further fine-
tune the same model, but with our surrogate loss function: [I(())-al + ce Br(z), In Figure 4, each point corresponds
model trained with value of e {0.02, 0.03, 0.04. 0.05, 0.07, 0.1. 0.15). For the two most extreme points on the curve, we fit a threshold slightly different from on the models for 0.02 and 0.15. We set and do not perform dyper-paaaamtte search.
8

---

Published conference paper at ICLR 2024
Theoretical limit: The theoretical limit of coverage can be defined as b/p, where b is the fraction of positively labeled points and p is the desired precision level. The precision is obtained exactly, and standard deviations for coverage are a result of the slightly different class imbalances in the cross-validation splits.
7.3 DISCUSSION
The performance of the four methods is reported in Table I: Precision vs. Coverage for surrogate Figure 4. The details of the surrogate loss are shown rejection loss decontextuulization. in Table I and the details for various baselines are pre- sented in Appendix B.1 (see Table 2).
Precision
Coverage
0.15
0.915 + 0.003
0.949 0.011
We observe that our rejection loss clearly outperforms the baselines considered, and for the lower half of the precision levels, closely follows the theoretical limit. We provide, at c 0.07 for example. about a halving of the error rate (1 11100% to 1)) while maintaining a broad 90.6% coverage. The theoretical limit for 5% error is only slightly higher at 93.6% coverage, and the closest baseline, the cross-entropy loss, only provides ~64.4% coverage at ~5.1% error. Another
0.15
0.937 = 0.005
0.921 0.007
0.10
0.938 0.003
0.912 0.011
0.07
0.945 0.005
0.906 0.010
0.05
0.949
0.001
0.881 0.008
0.04
0.953
0.009
0.863 0.024
0.03
0.969
0.009
0.621 0.027
0.02
0.983
0.006
0.448 0. 0.10
0.02
0.996 = 0.007
0.247 0.159
important observation is the stability of this result. Not only does the surrogate loss perform much better on average, but the standard deviations are also significantly smaller.
7.4 IMAGE CLASSIFICATION
While our focus is on LLM predictors and multiple correct labels, we provide additional empirical eval- uation on two simpler image classification datasets: Fashion-MNIST (Xiao et al., 2017) and KMNIST (Clanuwat et al., 2018). These are standard 10-class classification tasks, both with 60, 000 training sam- ples and 10, 000 test samples. Half of the training data is used for training predictor, and the other half for training a rejector. We use a 5-layer fully-comnected neural network. While this is much easier task than decontextualizzion, and the model class far less complex, we still observe improvement when min- imizing our surrogate rejection loss (see Figure 5). Due to fewer computational constraints, we include Figure 5: Precision vs. coverage on KM- cost-sensitive bascline, which uses the cross-entropy NIST. Standard deviations for precision and loss but reweights positive class by c/ c1- for vary- coverage are from 4 training runs. ing The full details are deferred to Appendix B.2.
KMNNITT
0.8
0.7
Coverage
0.6
0.5
0.4
Cost-sensitive
Maxprob
Cross-em
Rejection loss
0.90
0.92
0.94
0.96
0.98
Precision
8 CONCLUSION
We presented a theoretically-justfiee approach to classification with a reject option for applications where the predictor remains fixed. Our main contributions include the following: (1) new for- mulation of the problem of learning rejector with fixed predictor (for cases where there may be many correct labels); (2) introduction of new surrogate loss function for our scenario, with the proof of strong K-consistency bound guarantee; (3) definition of the notion of correctness for de- contextualization, and use to provide dataset of 2,000 manually-labeled decontextualization:: (4) experimental results demonstrating 10-25% improvement over basclines in coverage at various precision levels on decontextualization..
We observe that our algorithm can be used in other settings where a binary label indicating correct- ness of a prediction is available. Annotation in NLP can course be expensive, and this limits the breadth of experimental evaluation. We encourage the use of our algorithm in difficult rejection tasks with a large output space and a large number of correct labels. In particular, our algorithm can be used for abstention with large language models, in variety of contexts such text generation.

---

Published conference paper at ICLR 2024
REFERENCES
Pranjal Awasthi, Natalie Frank, Anqi Mao, Mchryar Mohri, and Yutao Zhong. 2021a. Calibration and consistency of adversarial surrogate losses. In Advances in Neural Information Processing Systems, pages 9804 9815.
Pranjal Awasthi, Anqi Mao, Mehryar Mohri, and Yutao Zhong. 2021b. finer calibration analysis for adversarial robustness. arXiv preprint arx55:2005.01550.
Pranjal Awasthi, Anqi Mao, Mchryar Mohri, and Yutao Zhong. 2022a. H-consistency bounds for surrogate loss minimizers. In International Conference on Machine Learning, pages 1117-1174.
Pranjal Awasthi, Anqi Mao, Mehryar Mohri, and Yutao Zhong. 2022b. Multi-class H-consistency bounds. In Advances in neural information processing systems.
Pranjal Awasthi, Anqi Mao, Mehryar Mohri, and Yutao Zhong. 2023. Theoretically grounded loss functions and algorithms for adversarial robustness. In International Conference on Artificial Intelligence and Statistics, pages 10077-10094.
Pranjal Awasthi, Anqi Mao, Mehryar Mohri, and Yutao Zhong. 2024. DC-programming for neural network optimizations. Journal of Global Optimization, pages 1-17.
Peter L. Bartlett and Marten H. Wegkamp. 2008. Classification with a reject option using hinge loss. Journal of Machine Learning Research, 1823:11823
Yuzhou Cao, Tianchi Cai, Lei Feng, Lihong Gu, Jinjie Gu, Bo An, Gang Niu, and Masashi Sugiyama. 2022. Generalizing consistent multi-class classification with rejection to be com- patible with arbitrary losses. In Advances in neural information processing systems.
Nontawat Charoenphakdee, Zhenghang Cui, Yivan Zhang, and Masashi Sugiyama. 2021. Classifica- tion with rejection based on cost-sensitive classification. In International Conference on Machine Learning, pages 1507-1517.
Guanting Chen, Xiaocheng Li, Chunlin Sun, and Hanzhao Wang. 2024. Learning to make adherence-aware advice. In International Conference Learning Representations.
Eunsol Choi, Jennimaria Palomaki, Matthew Lamm, Tom Kwiatkowski, Dipanjan Das, and Michael Collins. 2021. Decontextualization: Making sentences stand-alone. CORR, abs/2102.05169
C. K. Chow. 1957. An optimum character recognition system using decision functions. IRE Trans- actions Electronic Computers, EC-6(4):247-254.
C. K. Chow. 1970. On optimum recognition error and reject tradeoff. IEEE Transactions on Infor- mation Theory, 16:41-46.
Tarin Clanuwat, Mikel Bober-Irizar, Asanobu Kitamoto, Alex Lamb, Kazuaki Yamamoto, and David Ha. 2018. Deep learning for classical japanese literature. arXiv preprint arXiv: 18/2.0118/
Corinna Cortes, Giulia DeSalvo, and Mehryar Mohri. 2016. Learning with rejection. In Interna- tional Conference Algorithmic Learning Theory, pages 67-82.
Corinna Cortes, Giulia DeSalvo, and Mehryar Mohri. 2023. Theory and algorithms for learning with rejection in binary classification. Annals of Mathematics and Artificial Intelligence, pages 1-39.
Li Dong, Chris Quirk, and Mirella Lapata. 2018. Confidence modeling for neural semantic parsing. arXiv preprint 1805 04604.
Charles Elkan, 2001. The foundations of cost-sensitive learning. In International joint conference artificial intelligence, pages 973-978.
Katja Filippova. 2020. Controlled hallucinations: Learning to generate faithfully from noisy data.
In Conference Empirical Methods in Natural Language Processing, pages 864-870.
10

---

Published conference paper at ICLR 2024
Yarin Gal and Zoubin Ghahramani. 2016. Dropout as bayesian approximation: Representing model uncertainty in deep learning. In International Conference on Machine Learning, pages 1050-1059.
Siddhant Garg and Alessandro Moschitti. 2021. Will this question be answered? question filter- ing via answer model distillation for efficient question answering. In Conference on Empirical Methods in Natural Language Processing, pages 7329-7346.
Yonatan Geifman and Ran El-Yaniv. 2017. Selective classification for deep neural networks. In Advances in neural information processing systems.
Roei Gelbhart and Ran El-Yanis 2019. The relationship between agnostic selective classification, active learning and the disagreement coefficient. The Journal of Machine Learning Research, 20(1):1136-1177
Yves Grandvalet, Alain Rakotomamonjy, Joseph Keshet, and Stéphane Canu. 2008. Support vector machines with a reject option. In Advances Neural Information Processing Systems.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. 2017. On calibration of modern neural networks. In International Conference on Machine Learning, pages 1321-1330.
Kilian Hendrickx, Lorenzo Perini, Dries Van der Plas, Wannes Meert, and Jesse Davis. 2021. Ma- chine learning with reject option: A survey. arXiv preprint arXivv2
Dan Hendrycks and Kevin Gimpel. 2016. A baseline for detecting misclassified and out-of- distribution examples in neural networks. arXiv preprint 16Xiv
Zhengbao Jiang, Frank F Xu, Jun Araki, and Graham Neubig. 2020. How can we know what language models know? Transactions of the Association for Computational Linguistics, 8:423- 438.
Amita Kamath, Robin Jia, and Percy Liang. 2020. Selective question answering under domain shift. CoRR, abs/2006.09462.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
Aviral Kumar and Sunita Sarawagi. 2019. Calibration of encoder decoder models for neural machine translation. arXiv preprint 113.:v19030000000
Thomas C.W. Landgrebe, David M.J. Tax, Pavel Paclík, and Robert P.W. Duin. 2006. The interaction between classification and reject performance for distance-based reject-option classifiers. Pattern Recognition Letters, 27(8)9908-91
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. Bart: Denoising sequence-to-sequeme pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:
Xiaocheng Li, Shang Liu, Chunlin Sun, and Hanzhao Wang. 2024. When no-rejection learning is optimal for regression with rejection. In International Conference Artificial Intelligence and Statistics.
Anqi Mao, Christopher Mohri, Mehryar Mohri, and Yutao Zhong. 2023a. Two-stage learning to defer with multiple experts. In Advances in Neural Information Processing Systems.
Anqi Mao, Mehryar Mohri, and Yutao Zhong, 2023b. Cross-entropy loss functions: Theoretical analysis and applications. In International conference on Machine learning.
Anqi Mao, Mehryar Mohri, and Yutao Zhong. 2023c. H-consistency bounds: Characterization and extensions. In Advances in Neural Information Processing Systems.

---

Published conference paper at ICLR 2024
Anqi Mao, Mehryar Mohri, and Yutao Zhong. 2023d. H-consistency bounds for pairwise misranking loss surrogates. In International conference Machine learning.
Anqi Mao. Mehryar Mohri, and Yutao Zhong. 2023e. Ranking with abstention. In JCML 2023 Workshop The Many Facets of Preference-Based Learning.
Anqi Mao, Mehryar Mohri, and Yutao Zhong. 2023f. Structured prediction with stronger consis- tency guarantees. In Advances in Neural Information Processing Systems.
Anqi Mao. Mehryar Mohri, and Yutao Zhong. 2024a. Predictor-rejector multi-class abstention: The- oretical analysis and algorithms. In International Conference on Algorithmic Learning Theory.
Anqi Mao, Mehryar Mohri, and Yutao Zhong. 2024b. Principled approaches for learning defer with multiple experts. In International Symposium on Artificial Intelligence and Mathematics.
Anqi Mao, Mehryar Mohri, and Yutao Zhong. 2024c. Theoretically grounded loss functions and algorithms for score-based multi-class abstention. In International Conference Artificial In- telligence and Statistics.
Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. On faithfulness and factuality in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 1906-1919.
Chenri Ni, Nontawat Charoenphakdee, Junya Honda, and Masashi Sugiyama. 2019. On the cali- bration of multiclass classification with rejection. In Advances Neural Information Processing Systems.
Tadeusz Pietraszek. 2007. On the use of ROC analysis for the optimization of abstaining classifiers. Machine Learning, 68:137-169.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. 2020. Exploring the limits of transfer learning with unified text- to-text transformer. The Journal of Machine Learning Research, 21(1):5485-555 55511.
Adam Roberts, Hyung Won Chung, Anselm Levskaya, Gaurav Mishra, James Bradbury, Daniel Andor, Sharan Narang, Brian Lester, Colin Gaffney, Afroz Mohiuddin, Curtis Hawthorne, Aitor Lewkowycz, Alex Salcianu, Marc van Zee, Jacob Austin, Sebastian Goodman, Livio Baldini Soares, Haitang Hu, Sasha Tsvyashchenko, Aakanksha Chowdhery, Jasmijn Bastings, Jannis Bu- lian, Xavier Garcia, Jianmo Ni, Andrew Chen, Kathleen Kenealy, Jonathan H. Clark, Stephan Lee, Dan Garrette, James Lee-Thorp. Colin Raffel, Noam Shazeer, Marvin Ritter, Maarten Bosma, Alexandre Passos, Jeremy Kaitin-Shepard Noah Fiedel, Mark Omernick, Bremnan Saeta, Ryan Sepassi, Alexander Spiridonov, Joshua Newlan, and Andrea Gesmundo. 2022. Scaling up models and data with t 5x and seqio. arXiv preprint arXiv:2203.17/88
Carla M Santos-Pereine and Ana M Pires. 2005. On optimal reject rules and roc curves. Pattern recognition letters, 26(7):943-99) 952.
Clayton Scott. 2012. Calibrated asymmetric surrogate losses. Electronic Journal Statistics, 6:958- 992.
Lewis Smith and Yarin Gal. 2018. Understanding measures of uncertainty for adversarial example detection. arXiv preprint arXiv:1803.08533
Ingo Steinwart. 2007. How to compare different loss functions and their risks. Constructive Approx- imation, 26(2):225-287.
Francesco Tortorella. 2000. An optimal reject rule for binary classifiers. Proc. Int. Workshop on Statisticul Pattern Recognition, pages 611 611-620.
Neeraj Varshney, Swaroop Mishra, and Chitta Baral. 2022. Towards improving selective prediction ability of nlp systems. In Workshop Representation Learning for NL..
12

---

Published conference paper at ICLR 2024
Han Xiao, Kashif Rasul, and Roland Vollgraf. 2017. Fashion-mnin a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:/708.077
Xin, Raphael Tang. Yaoliang Yu, and Jimmy Lin. 2021. The of abstention: Selective prediction and error regularization for natural language processing. In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the Ith International Joint Conference on Natural Language Processing (Volume I: Long Papers), pages 1040-1051.
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2020. mT3: A massively multilingual pre-trained text-to-text trans- former. arXiv preprint arXiv:2010.11954
Ming Yuan and Marten Wegkamp. 2010. Classification methods with reject option based on convex risk minimization. Journal of Machine Learning Research, 11:111-130.
Shujian Zhang, Chengyue Gong, and Eunsol Choi. 2021. Knowing more about questions can help: Improving calibration in question answering. arXiv preprint 106.:x:200600444:0
13

---

Published conference paper at ICLR 2024
A JC-CONSISTNNCY BOUND PROOF
A.1 CALIBRATION GAP FOR REJECTION LOSS
The following gives the expression of the calibration gap Aet,-
Lemma 2. The Bayes solution r' for the rejection loss can be expressed for all x E X by r (x) (x)) c1- The calibration gap for the rejection loss is given for any r Rall and X by
Ai (r,x) -(,,))=lvit) In(x) (1 (1-c)) (a)))
Proof. For any Rall and X. write
Ce2 (r, (1)))
+ [1 (x)))_)()))
(1 [r)= (c
+[1 [1- --==
< +[1 (
For the optimal eyy, would always pick the lower of or - n(x), which gives: (x)
(z)
min(c,1 1 - ())) The corresponding Bayes solution r" can be defined by r" r=(2)) n/x) - (1 c).
Thus, the calibration gap is given by
AC2, (r,x)) = =o]s + (x)) To)>0
- -min(c, 1 (=))
If (ax) correctly chooses the lower of the two, have r(x:)r (a)) 0 and then ACt, Other- wise,
ACC, (r,z)) (c- (1-)(x))- -c otherwise
if rf)
Thus, for all X, we have (r,, = (n(x) (1 (1-c) ( This completes the proof.
A.2 CALIBRATION GAP FOR SURROGATE LOSS
Here, we analyze the calibration gap for the surrogate loss.
Lemma 3. Let (x) be defined by In(x) m(<) + (1 (x) and define 7 by 7 a+23
Then, the calibration gap for the surrogate loss is given for any E Raul and x E X by
23c
cc
ce Br(x)
Iz(2))
Proof. By definition, the calibration function for l can expressed for all x E X by (r, = (r,
+ (1)]
((x) [c)((0)))))11) ce 9r(z)]
[[1 [ [+( ce- -B((a))
= =eir(0 1x)) Ar(z)
Since the exponential function is convex, AC,, (r, x) a convex function of r(x). Thus, for Ralls we obtain the minimum ro(x) by differentiating with respect to r(x) and setting to 0:
(r() 1,(x) 3ce (r(x) =
28c
< ro()) = log
(lg(1))
14

---

Published conference paper at ICLR 2024
Plugging in this expression in Ct2 gives the corresponding minimal calibration e,, (z): c2 (a)
[ 2 7=(x)²=> (+))
This completes the proof.
A.3 X-CONSSITENNYY BOUND
In this section, we prove our main result. The following will provide key tool to derive our result. Proposition 4. Assume that there exists a convex function V. IR, - R with (0) 0 such that the following holds for all r E Ranl and x X: (In(x)) - (1 (1-0) (1-c) (())rr)))))) AC, (0, Let Le be defined by Ic ce (1- c)e² 3 and assume that 23c Ic. Then, for any Ral:
V Re, (r) Ri,,) s Rei (r) Rig
(3)
Proof. We will show that the following holds: (=)s0 ACt, (r, (0, The result then follows by Theorem and Lemma 2. Since we have 2 Ics the following equivalence holds:
2Bc >1
<o( 03
a11(2)
< [x) < I
+ n > ci-le
cx
4 > > (1-c)e -1-o)e
<r'x)>
This implies (s)) Ce2 (r.z) (=) Ce, (r,z). Now, since ro(x) is the unique minimizer of the strictly convex function Ce, (r,x) of r (x), then, function of r(x). Ce, (r,z)) is decreasing from oo to (z) and increasing from there to +00. Thus, if (o()) > 0, the infimum of Ce, (r,z) over r(x) s 0 is reached for r(z) O. Similarly, if (2( < 0, the infimum of Ce, (r, over r(z) 2 0 is reached for r(x) = O. This shows that inf, (x) Ce, (r,z) Ce, (0, x), and completes the proof.
The proof of our main result makes use of the following identity, which is refinement of Bernoulli's s inequality. The result could be of independent interest in other contexts, we give a concise proof below.
Lemma 6 (Bemoulli-type incquality). The following identity holds for all x, E (0, 1),
(r-1)x²
(1+x)* 11+
4
Proof. Let fa(2) = (1+z)* [1 rx+ xm))aa We will show that f, f, s 0 for all x,r (0,
We have (x) = r11 x= - (r+ r(r-1)m and (0) 0. To that (1) 0, observe
2
2 $ 1. The left-hand side of the last inequality is a convex function of and equal to 1 when r= or Thus, the left-hand side is less than equal 1 for r E (0, 1). giving (1) s 0. Since (x) is a convex function of x, with (0) S and f. s 0,
r2 (r+ ((-1)) (r+)))))
s ~ 2- (r-1)
then (x) 0 for all (0, 1), which shows fr is decreasing. Then, since (0) 0, fx( s
for all x,, r E (0, 1).
The following our main result; relates the surrogate excess error that of the rejection loss.
Theorem S. Let or, 3 > 0 be such that 29c Tep where T, ce + (1 - -)e- 2. Then, the following inequality holds for any E Rall:
(c+ c
Reg (r) - Ri 5
(Re, (r) - R2,
ea e
15

---

Published conference paper at ICLR 2024
Proof. Using the expression of Ae,, given by Lemma 3, we can write
2,3c
AC (0,x) [,))
1-7 a
[,) + (Tc e) [n(xr)
Le
We can express this formula in terms of u(x) = n1()) (1 - c), using [,(x) = Ju(x)) + Tc. with
(a( = [ ]u(x)
AC,, (0,z)
=+ (I+ c) c)1 1+
Ju(x)
1-7
Le
Le ctle )u(x) (t)))
c+ I To
cle [J(()) 2
4c+ [e Ie
where we used Lemma 6. The function ((() defined by this expression verifies the condition of Proposition and therefore we have V (Re, (h) Rig s Re, (h) - -R7, An explicit upper- on Re, (h) R2 be written in terms of v 1- Re, (h) R / - (h) R7,, ). To derive the expression of 1r11, write (u), that is:
I
u(x))
e E 2 1²
4
cle
Ie
2
(c+
+ |u|
c
c
Thus, we have, for all r Rall. Re, (r) R, s
2
(otle)le (Re, (r) - Ri;
ct c
16

---

Published conference paper at ICLR 2024
B EXPERIMENTAL DETAILS
B.1 DECONTEXTUALIZATION
In this section, we report the detailed results for our experiments the decontextualizztion task. Table 2 presents the mean and standard deviation of the precision and coverage of various baselines over 4 cross- valssaaaii splits and Table 1 in Section 7.3 provides detailed results of the surrogate loss.
Table 2: Precision Coverage for various baselines on decontextuullization, with theoretical limit.
MAXPROB
CROSS-ENTROPY
THEORETICAL LIMIT
Target precision
precision
coverage
precision
coverage
precision
coverage
0.899 0.002
0.907 0.017
0.903 = 0.016
0.968 0.045
0.90
0.989 0.001
0. 92
0.924 0.001
0.672 0.052
0.930
0.771 0.146
0.92
0.967 0.001
0. 93
0.934 + 0.025
0.552
0.939
0.015
0.677 # 0.102
0.93
0.957 + 0.001
94
0.938 0.022
0.467 0.035
0.949 0.012
0.644 = 0.103
0.94
0.950 0.001
0. 0.95
0.942 0.023
0.405+0.03)
0.965
0.509
0.95
0.936 0.001
0. 96
0.959 0.022
0.321 + 0.041
0.364 + 0.096
0.96
0.927 + 0.001
0.97
0.972 1 0.018
0 0.012
0.980 0.008
0.330 1 0.086
0.97
0.917 1 0.001
0.98
0.972 0.018
0.19&40001
0.981
0.013
0.298 0.069
0.98
0.908 0.001
0.99
0.983 0.013
0.015 0.015
1
0.986 0.015
0.150 0.059
0.99
0.898 0.001
B.2 IMAGE CLASSIFICATION
In this section, provide details of our experiments on -NNIS,, fashion image dataset, and KMNIST, a cursive Japanese letter dataset. Both are perfectly balanced between their 10 classes. In both cases, we the training data. The remaining half is reserved for the rejector. Training the rejector is a bi- nary classification task: for pairs (z, y) occuring in the usual dataset, we construct another dataset (x))) -((=)=y)) where f is the predictor and (x) ) is the probability that f assigns to its pre- diction on x. In our experiments, we observe that it is important to append fz(x) as a feature to I. Note that constructing this binary classification dataset does not require manual annotation. For -ashion-MNIST, our predictor is trained to 85.3% accuracy on its test set, and for KMNIST, our pre- dictor is trained 79.1% 1% accuracy on its test set. While it is possible improve the performance of these predictors, this is not our focus. We are focused on rejection task given some fixed predictor.
5-layer fully-connected neural network train predictor with half of
Next, we detail the methods for rejection.
Maxprob. Similar to the decontextualization experiment, we fit thresholds on the scores assigned by the predictor. Since this method is deterministic (and the error bars here are over rejector training runs), there are no eтror bars to report.
Cross-entropy loss. We train another 5-layer neural network on the constructed binary classification dataset using the cross-entropy loss. Similar to the decontextualization experiment, thresholds are fitted on the scores of this neural netowrk.
Rejection loss. We train second 5-layer neural network on the constructed binary classifi- cation dataset using our proposed surrogate rejection loss. For Fashion-MNIST, e is varied in (0.05, 0.2, 0.3, 0.5). For KMNIST, is varied in (0.025, 0.05, 0.1, 0.15). Each point on the plot represents a model trained with a different value of c. We set a in the surrogate rejection loss function to 3.5.
Cost-sensitive loss. We train a third 5-layer neural network on the constructed binary classification dataset using the cross-entropy loss, but with the positive class reweighted by c1-1c). For Fashion- MNIST, c is varied in (0.05, 0.1, 0.2, 0.3, 0.5). For KMNIST, is varied in {0.03, 0.05, 0.1, Each point the plot represents model trained with different value of c.
For all methods, we use the Adam optimizer (Kingma and Ba, 2014), and tune the learning rate in [1e 4, le 7] and number of epochs in [20, 100].
17

---

Published conference paper at ICLR 2024
Fashion-MNIST
0.75
0.70
Coverage
0.65
0.60
Cost-sensitive
Maxprob
Cross-ent
Rejection loss
0.45
0.94
0,95
0.96
0.97
0.98
0.99
Precision
Figure 6: Precision vs. Coverage on Bushion-MNISTI Standard deviations for both precision and coverage are from 4 different training runs.
The precision coverage graph for KMNIST is reported in Figure 5 in Section 7.4 and the precision vs. coverage graph for Fashion-MNIST is reported in Figure 6. We do not plot the theoretical limit since method is it in this setting. In both cases, generally observe that the rejection loss lies above the baselines. It is likely that the predictor in this setting is much better calibrated than large language model, and thus Maxprob is a much stronger baseline with not much room for improvement as on decontextualization. We also note that it may also be possible to improve the performance of our method by tuning a.
C COMPARISON WITH COST-SENSITIVE CLASSIFICATION
It is worth pointing out that minimizing the induced rejection loss is equivalent to minimizing cost-sensitive classification loss (Elkan, 2001; Steinwart, 2007; Scott, 2012; Charoenphakdee al., 2021), since by using the decomposition Ia--1 L<=)>=- = (1 c) Iaa--1 4==)> +cla--1 (u)= and cllr(a)<u = clla-+1 <0( +clla--1 Ir(=)=u, the loss (2) can be rewritten as [(r, x, a) Il==1 [x(=)> +cIr
c1 1---1 4 +clla=-1 {x())0 +cla=+1 Lr(s) + +cl (x)s0
(1- Ia=-1 +cla=-1 L()= +cIas-1,
where the last step we use the fact that e clla-11 Ir(=)so + +cTa=-1 Tr = cla-- In light this expression, since the last term cla--1 does not depend on r, if x +s o(-x) is convex function upper-bounding Iesu, then, Eg defined follows for any r R and (x, e Xx (-1,+13, is natural surrogate loss for
(r x, a) (1-c) ((r(x)) clla=+1 d(-r(z)).
We will refer to ls as cost-sensitive surrogate losses for the induced rejection loss. However, this cost-sensitive approach suffers from several issues: (i) There is lack of any H--cossstenccy bound guarantees for cost-sensitive surrogate losses with respect to the induced rejection loss. Conversely, theoretical analysis can potentially extend sensitive classification. This would provide a theoretically justified algorithm for that context. Our novel contribution lies in introducing loss function for the induced rejection loss backed by strong H-consistency bounds; (ii) It has been shown in (Cao et al., 2022) that the cost-sensitive approach (Charoenphakdee et al., 2021) can not produce the otate-of-the-art performance the learning with rejection framework, which motivates us to propose a new theoretically guaranteed surrogate loss in our rejection scenario; (iii) As shown in (Charoenphakdee et al., 2021), the cost-sensitive ap- proach equivalently solves n one-versus-all binary classification problems, where is the number of classes. Therefore, when the size of the sub-sample containing some of the classes is relatively small, the one- one-versus-all binary classification problem may face challenges due insufficient data or increased risk of overfitting. This issue stands out for the decontextualizatioo task, where the samples corresponding to -1 are much fewer than those corresponding to +1: (iv) Our empirical results on the benchmark datasets show that the cost-sensitive approach is inferior to our proposed surrogate loss function, which substantiate the effectiveness of approach.
K-consistent surrogate loss function for cost-
18

---

Published conference paper at ICLR 2024
D H-CONSISTENCY BOUNDS BEYOND Hall AND PROOF
Here, will show that surrogate losses benefit from R-consistency bounds with the hypothesis set R extending beyond the family all measurable functions Rali- Without loss of generality, we consider X = {ir [ e (Rd |<> S {1.. Let p. q e [1, +00 be conjugate numbers such that will consider bounded hypothesis sets R. that is. there exists function F: + R. such that for all r R and x X. [r(x) s 7(x), and values in [-7() [(x)] can be reached. As shown by Awasthi al. (2022a), for the family of linear models Rin {2r w hidden-layer ReLU networks RNN {or + Σÿ-1 My (wy b,)+ ||u]|| s A, |wyll g1 s W, [by] s B), where (+)+ max(--0). we have F(z) = W + B and F(x) AW | AB respectively.
= 1. We
|wll, W, |6|6| s B} and one-
D.1 MAIN RESULT
In this section, we present our main result on R-consistency bounds with bounded hypothesis sets R (Theorem 7), including Rin and RNN considered in (Awasthi al., 2022a) special cases
(Corollary 8). The proofs are presented in Appendix D.4.
Theorem 7. Assume that R is bounded with function F: + R. Let 3 > 0 be such that 2 Ie. where I. ce v (1-c)e- a. Then, the following inequality holds for any e R:
(r) R2g, x
Me S (T (r) - Re x M),
(4)
s Actto 1- e (llelo)inf F(x) 2
(etto)le
c 4
where T(2)
4(c+Ic) of of ((otc)int enx P(o)
otherwise.
Corollary 8. Let R= Rin or RNN. Let a., 3 0 be such that 29c Te where T ce i (1-c)e-a.
Then, the following inequality holds for any e R:
(r, Rig. + Me T(RE, (r) R x Me,
(5)
(etTo)ee
cic 0 0s 1 o..
olfote) 26
1 -
e
where T(z)=
((c+T.)
otherwise
and B is replaced by
al (1)4))2
26.
9
AB for R RNN.
D.2 CALIBRATION GAP FOR REJECTION LOSS
We first extend Lemma 2 to any hypothesis set R that is regular for rejection. Definition 9. We say that a hypothesis set R is regular for rejection if for all x X, there exist T-, R such that r- (x) > and r. (x) s O.
It is clear that all bounded hypothesis sets including Rim and RNN are regular for rejection. The following gives the expression of the calibration gap ACi, for all hypothesis sets R that are regular for rejection. The proof is nearly identical to Lemma 2.
Lemma 10. Assume that is regular for rejection. The best-in- fest-in-class solution r" for the rejection loss can be expressed for all e X by r" (x) nfxx) (1 (1-c). The calibration gap for the rejection loss is given for any r E R and x X by
ACc, (1, In(3)) (1 c)) (r)s))
19

---

Published conference paper at ICLR 2024
Proof. For any r and x X, we can write
Ce2 (T,2)) (2))22(
[] - [=)]]]](((
())) [+1--1 (r))0 +CTr
+[1 [1- --=-1 (o tc
c (1)= + [
For the optimal R. since R is regular, would always pick the lower of e or 1 - 77x,, which gives: C (2) minn(, 1 (x)). The corresponding best-in-class solution r' can be defined by (=) = (=)) (1-c). Thus, the calibration gap is given by
AC2, (r, clli())<o +[1 L
-minfc, 1 (x)],,
rar) correctly chooses the lower the two, we have r(x))r((a) (x) > 0 and then AC,, Other- wise,
(e,,(1,z) cc- (c-(1nn(x) if (f) 0
(1- (1-n(x)) otherwise
Thus, for all E X, we have (r,, = (n(x)) (1 - c)) (a))) This completes the proof.
D.3 CALIBRATION GAP FOR SURROGATE LOSS
Next, we extend Lemma to bounded hypothesis sets R. The following gives the expression of the calibration gap for the surrogate loss. The proof directly extends that of Lemma 3. Lemma 11. Assume that R is bounded with function T:X + R. Let [T(x) n())e" + (1-n(xz))e,
ro(x)= log
28c
and == a+2 Then, the calibration gap for the surrogate loss is given
(T,())
for any r e R and x E X by
e eer(x) + +e ce- Dr(x) -B((x)
- 28c (28e)
( 1x(x)1
-F(x) s rr(()) (0(())((())))))) s7(x)
ACG (r,
eir(n) (2)) (z) ce (r(a) - fr(x)
e 1, (x) ce 97(z)
To(z)> > F(x)
- (,() (
ce ar(()))
sF(x)
ro(z) < -7(x).
Proof. By definition, the calibration function for can be expressed for all X by
Ce, (r,z)
+1) [1)
=n(x)
ce 9r(z))
n(x)] (1(())-11 + ce- (r())
= 1n (x) 3r())
Since the exponential function convex, AC,, (r,z) convex function of r(x). Thus, for r R, we obtain the minimum (=) by differentiating with respect to r(x) and setting to O:
a e rr() Ie(r(=),((T))))
IT(x) 3ce (r()) =
2
23c
$ r)))))) log
(z)(())
Note that for all x e X, (r(*)rre)) = [-T(x), If ro(x) is within this range,
plugging in (2) in C,, gives the corresponding minimal calibration gap ex (z): e, (x)
[ (ffe (x)
Otherwise, the corresponding minimal calibration gap is achieved at
20

---

Published conference paper at ICLR 2024
either r(t) -(x) or r(z() -x))). Plugging in these expressions give the corresponding minimal calibration gap ej2 (x:):
[ (2)0) (x)²) [
-r(x) <m(()) r(x) ro(x) T(x)
e, (o)
e I (x) ce -F(
()(() (x()) 0r()
ro(o) < <T())).
This completes the proof.
D.4 K-CONSISTENCY BOUND
In this section, we prove our main result. The following result extends Proposition 4 to any hypoth- esis set R that is regular for rejection and will provide key tool to derive result. The proof is nearly identical to Proposition 4.
Proposition 12. Assume that R is regular for rejection. Assume that there exists a corvex func- tion V.R. + with (0) O such that the following holds for all r and X: & (In(x) - (1 (1-c) c) (=)=)))(=))0) s AC (0, Let Io be defined by Le = cE + (1 - c)e & and
assume that 20c Ic. Then, for any r RR:
(r) Rig. + Me,xx) s Rτ, (r) R2 + M,
(6)
Proof. We will show that the following holds: inf (x))0 ACe, (r, (0, The result then follows by Theorem and Lemma 10. Since we have 28c Ics the following equivalence holds:
2Bc <o
> (I
[Txx) < Io
-- 17(1)) cl-c2
+ (1-cle -1-c)e-((-c)) &
-
r+'2))
This implies inf inf(( (2) C₁2 (r,z) :)==== (n) Ce2 x). Now, since ro(x) is unique minimizer of the strictly convex function C₁, (r,x) of (2:), then, as a function of r(=). Ce, (r, decreasing from -oo to ro( and increasing from there to +00. Thus, if fo(2)) > 0, the infimum of Ce (r,z) x) over r(x) S is reached for r(x) O. Similarly, if ro < 0, the infimum of Cr, (r,z) over r(z) 0 is reached for r(x) 0. This shows that (=)) Ce, (r,x) Ce, (0, x), and completes the proof.
The following is our main result; it relates the surrogate estimation error to that of the rejection loss.
Theorem 7. Assume that bounded with function 7: T:X R. Let a, 3 be such that 29c I where T, ce ā + (1 - c)e === 2. Then, the following inequality holds for any r e R:
Re, (r) - -R + Me, $ T( (RRe, (r) -Ri, x Me,
(4)
(ctL.)le
0
1 4ett. 1 e
a(Teto) in F(o)
2
2 -
where I(z)=
4(c+F.)
otherwise.
1-c
Note inf of ((a)
25
c 4 4
21

---

Published conference paper at ICLR 2024
Proof. Using the expression of Ae,, given by Lemma 11, we can write
[,(x) ( 2 (1))²² (((z))c-0) - (x) ce (T(()
-T(x) (n(z))
Aee, (0,z)
[n(x)+ (n(x)+c-e e {)(a) In (x) (r())
ro(0) < <-i(x).
15(x) [o(x)+o(((tee))
1e 11(20)
1,())) b-y
-f(x) s ro(()) sF(x)
1,(x+ +c- e8/(1) ec (x)) O6
ula7()
(o(x)>((z)
+x(=) c- (r(()) I (x) ce
2c
<()))~)(z))
Without loss of generality, consider r" (2) ()())) (1- 0. Then ro(x) 0. As with
the proof of Theorem 5, we can express AC (0, in terms u(x) = 772) - (1 1 c). using
(2) = Ju(x) + Ic. with Ju(x) = =[e e (r). Note that the condition ro( s F(z) can be expressed as
I of ST(x)) = u(x) S
Fc 2c
2,c
log
(z)((z))
e c 4
When 0 E (o( s F(r), we have
cTe Ju(x)
2
clc [(())
2
AC₁, (0,x)² 2 cct La
c - - 2
le
4 c Ie
Le
When ro(x)> > F()), we have
u(le1e)r(w)
1 l-e
1 cla
AC,, (0,x)) 2 4 c Ie
[e - [(]u))).
I
Therefore, the function (u) defined by
p(tote) in 26
Ie 1-e
1 ele
u(x)
c (ote) T(m)
0 u(x) S
Vuu)=
4 To
2 10-00
ol
1-c"
1 cla
[e e 8 Ju(x)
otherwise
4 c+T.
verifies the condition of Proposition 12 and therefore have Ti (Re, (h) -Ri,) Re, (h) Ry An explicit upper-bound on Re, (h) R can be written in terms of w- Re, (h) R7, (Re, (h) R2i)). To derive the expression of v-1 we write 05 2 1 cta [1 1-e in M2)) Tott
v(u), that is: when
c+ ct lo cTa
Tu(x)
(c+ lo)lo Lo)e 2.
c - g = |u|
==
To
e -
Otherwise,
a(feto) inl P(o)
clc
e
4(c+ Ic)
[e - (=]]u(() = u a(fate) inf
1ctl. 1e
lc
Thus, we have, for all r E R, Re, (r) Rig s sT(Re, - R2, where
(ctl,)le
05 Totte
(f
26
r(x)=
f(c+J.))
otherwise.
al 1
inf F(a)
2c
2 8
22

---

Published conference paper at ICLR 2024
Theorem implies the following corollary for R = Rin and RNN-
Corollary 8. Let R= = Riin or RNN. Let o;, 3 0 be such that 29c To where Te = ce (1-c)e-#.
=
Then, the following inequality holds for any RR:
Res (r)
Rez.
Me
T Reg (r)
R Me,
(5)
os 1 cf
ollote) 26
2
e
where 1'(z)
((+1,)) alIc+c)2 2c
and B is replaced by
otherwise
c
1 9
AB for R RNN-
Proof. Using the fact that inf nex F(x) inx (W|xl,,*
infnex(W[rl, + B= B) for R Rün and inf zetr F(x)
infzex (AW AB) AB for R RNN. by Theorem 7, we complete the proof.
23


Reviews:
[{"review_id": "0hp45EzgNz", "reviewer": "Reviewer_AHeX", "summary": "For the purpose of letting a model 'reject' or 'abstain' from classifying some samples, a technique of using a rejection loss is proposed; from which a surrogate loss is derived and used. This is in contrast to either using the 'confidence' output or training for an (n+1)-th class in the sense that the predictor can be fixed. Theoretical guarantees for this surrogate loss are provided, and this framework is evaluated on real-world decontextualization tasks, and also for image classification. The results show promise and seem to perform better than other methods studied.", "questions": "- page 5. it is said that \"underlying scores are not favourable for that precision level\". Why is that?\n- These are possibly standard deviation bars in figures 4 and 5. How were they generated? Is it from different folds of cross-validation?\n- Are Maxprob and cross-entropy trained on different models? Why is that?\n\n**Minor Typing / Formatting / Clarity issues**\n- page 4. last equation \"<=0\" should be in the subscript\n- please recheck format for citations: some citations use et al while other list all authors.\n\n**Comment**: I chose \"good\" in soundness, presentation and contribution, but \"3: reject\" in the overall rating. That's mainly because of the mathematical inconsistency, which I hope can be resolved.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Very promising real-world results\n- Theoretical guarantees motivated well and provided\n- Good theoretical comparison to other methods, and good motivation for the need of proposed method provided\n- NLP examples but possibly further extensions to other areas\n- Surrogate loss performance better than other models and also quite close to theoretical limits at times.", "weaknesses": "- Although GitHub repo links and other identifying information cannot be written in a paper under review, I did not see any indication of the intention to make the code public, nor is it provided in supplementary materials\n- Page 4. First equation/inequality.\n\t- $c$ is positive. indicator functions are either 0 or 1, so:\n$\\mathbb{I}\\_{a\\leq 0} \\mathbb{I}\\_{r(x)>0} + c \\mathbb{I}\\_{r(x)\\leq 0} \\geq max(\\mathbb{I}\\_{a\\leq 0}\\mathbb{I}\\_{-r(x)<0}, c \\mathbb{I}\\_{r(x)\\leq 0})$\n\n\tmax of two different terms that are positive should be less than or equal to the sum.\n\t - And second comparison should be equal. As the first term in the first max is saying: \"Both $a$ and $-r$ should be less than zero for the indicator product to be one\". And the the first term in the second max is saying \"the max of both $a$ and $-r$ should be less than zero for the indicator to be one\". Both of these statements imply each other and therefore the last relation should be of equality. \t \n\t - The bound becomes a lower bound, not an upper bound.\n\t - I did not check the last relation. That might still hold despite this, but need to know why.\n - There is no test set. Only train and validation, where cross-validation is used so training algorithm sees all data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ctsTSJzaZd", "reviewer": "Reviewer_KRjg", "summary": "This paper studies the problem of learning to reject with a fixed predictor, motivated by the case when the fixed predictor is a pretrained large language model (LLM). The goal is to learn a rejector function that allows for post-hoc filtering of LLM outputs: by rejecting lower-quality outputs, the combined predictor/rejector system can have higher precision at the cost of lower coverage.\nThis is especially critical in high-stakes sequence-to-sequence problems in domains such as health data.\n\nThe paper designs an H-consistent surrogate loss for learning to reject with a fixed predictor using the framework of Awasthi et al. (2022). This loss is then used for the decontextualization problem, which is to rephrase a sentence into a form that's understandable without the context around the original sentence.\n\nOn the decontextualization experiment, the proposed surrogate outperforms strong baselines from the learning-to-reject and model calibration literature.", "questions": "- Why do we care about $\\mathcal{H}$-consistency even though we only consider the space $\\mathcal{R}_{all}$ of all measurable functions? It would be helpful to further emphasize this earlier in the paper as a major difference / novelty / reason why more naive baselines don't work (see earlier comment under weaknesses).\n\n- small typo in display at bottom of pg4; $\\le 0$ should be in the subscript of $\\mathbb{I}$\n\n- small typo in display at bottom of pg4; not enough parentheses\n\n- Minor semantics/notation: $r(x,f(x))$ reads like a score for rejecting, so the notation reads like $r>0 \\implies$ reject, when it's actually being used in the opposite sense. This tripped me up several times. I know this notation is inherited from Cortes et al. but I think it's a bit confusing.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The surrogate loss derived in the paper doesn't require tuning of a threshold parameter, which is a drawback of confidence-based approaches.\n\n- Empirical evaluation that avoids a common pitfalls of other work in this area: because different loss functions are being compared, it's only fair to compare performance with the optimal learning rate, since changing the loss function changes the scale of gradient updates.\n\n- The paper's theoretically-derived relationship between the two hyperparameters $\\alpha$ and $\\beta$ works well empirically.", "weaknesses": "- The method is only applied to the decontextualization problem when it actually seems to have much more broad applicability. I can think of several LLM applications where the ability to increase precision by rejecting would be useful. For example, we could try to learn a rejector that cuts down on hallucinations in summarization or text simplification. More exploration of this technique beyond the decontextualization problem would make it more impactful.\n\n- > Additionally, to the best of our knowledge, minimizing the cross-entropy loss does not have any proven guarantee with respect to our main objective: minimizing the induced rejection loss.\n\n    - When $\\mathcal{R} = \\mathcal{R}\\_{all}$, can't we use a cost-sensitive surrogate loss plus usual (Bayes) consistency results? \n      I.e., I don't understand the following claim in Appendix C given that the paper's results only consider $\\mathcal{R} = \\mathcal{R}\\_{all}$:\n     >  (i) There is a lack of any H-consistency bound guarantees for cost-sensitive surrogate losses with respect to the induced rejection loss. \n\n        In general, I think the paper could use more presentation on why the naive approach of directly training the rejector to predict the labels $a$ using a standard cost-sensitive surrogate loss doesn't work, since the results only consider the $\\mathcal{R} = \\mathcal{R}\\_{all}$ case. That, or some results for linear or 1-layer-NN rejectors, as in Awasthi et al. (2022), would strengthen the theoretical part of the paper.\n\n- No ablation on $\\alpha$ even though it's important in the bound (Thm 5), and different experiments use different values (e.g., 4 in main text experiments, 3.5 in appendix vision experiments)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jFXPkB5Vir", "reviewer": "Reviewer_irSz", "summary": "In the submission, a novel loss function parameterised by the learnable rejectors is studied to train the LLM model to handle diverse outputs for one prompt. To compute the existing loss function for tackling this problem is NP-hard. At the same time, the authors proposed a trackable surrogate loss, which is differentiable and convex, and by optimising it, the generalisation error is minimised supported by the theorem proposed as the main result in the submission. In the experiments, by training the model with the proposed loss function, the models are improved in terms of precision vs. Coverage compared with the existing method.", "questions": "1. According to my understanding, applying the surrogate loss requires labelling the output from the given model. Then the model is further trained by the learned surrogate loss. Thus some extra information is introduced. Can the Cross entropy loss and Maxprob have the same information? \n2. What is the format of the ejector?\n3. WHy the F1 score is not applied for clear comparison?\n5. For the std comparison is not very clear whether rejection loss is much better than Maxprob in Figure 4 and in Figure 5.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The derivation is clear and sound. I am not an expert on the proofing behind so I only scan the appendix. However, by following each proposition and theorem, it is easy to follow the logic step by step and reach the point that by minimising the proposed loss function the target generalisation error is minimised. \n2. Nice plot for intuitively explaining the property of the surrogate loss with the changing of the r(x).", "weaknesses": "1. The experiment setting is not well explained and the settings are not comprehensive. As the authors mentioned,  the LLM predictors are their main focus but in the experiments, there is only one type of LLM. As T5X is a family of models, there are other choices and more architectures from other families will be more comprehensive for evaluating the proposed loss. \n2. The loss function is tested on image classification but on a tiny setting and this pure classification task does not really relate to the LLM setting. \n3. To train the rejector still required to label new information, it is hard to distinguish whether the improvement is from the additional information or the loss function.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)