Learning to Defer with an Uncertain Rejector via Conformal Prediction

Venue: NeurIPS_cc_2024_Workshop_BDU
Published: 2024-09-06 13:49:47

Abstract: We perform uncertainty quantification for the rejector sub-component of the learning-to-defer framework.  In particular, we use conformal prediction to allow the reject to output sets, instead of just the binary outcome of 'defer' or not. Our method builds on the existing L2D framework, which previously optimized a specific surrogate loss, namely the one-over-all loss. We evaluate our approach by reporting abstention and consensus prediction results on the CIFAR-10 dataset, demonstrating improvements over the traditional method.

Main paper: 
 introduction: 
 Learning-to-Defer (L2D) [8, 11] is a framework for human-AI collaboration that divides responsibility between machine and human decision makers. For every test instance, a 'rejector' function decides if the case should be passed to either a human or model (but not both). The rejector thus can be seen as a meta-classifier that determines how to assign responsibility based on which decision maker (human or machine) is more likely to make the correct prediction. While L2D systems offer the promise of improved safety and robustness-by having a human available for support-this promise critically depends on the rejector's performance. Being a machine learning model itself, the rejector is susceptible to the usual failure modes, such as the distribution shift between training and test time. Yet, unlike with traditional predictive models, there is an extra point of failure in that the human's predictions can also shift [14] .In this paper, we perform principled uncertainty quantification for the rejector sub-component of L2D systems. Specifically, we use the framework of conformal prediction to allow the rejector to output sets, instead of just a single binary outcome (defer or not). This allows the rejector to express its uncertainty about whether the human or machine should be assigned to make the decision. In turn, this allows for safer decision making, possibly by abstaining from the prediction all together or querying both the human and model for their predictions.We report experimental results on the CIFAR-10 dataset where the proposed uncertainty framework can help the existing L2D systems at the test time. We found our uncertainty-based deferral could assist L2D rejector in abstaining. 
background: 
 Setting, Data, and Model We focus on multiclass L2D (with one expert), though the ideas are presented can straightforwardly generalize to L2D-based regression [17] . Let X denote the feature space and Y the label space, a categorical encoding of K ∈ N ≥2 classes. Let x n ∈ X denote a feature vector, and y n ∈ Y denotes the associated class index. L2D assumes that we have access to human predictions, denoted m n ∈ Y for the associated feature vector x n . The training data then includes the features, the true label, and the human's prediction: D = {x n , y n , m n } N n=1 . The human is assumed to have some skill at the prediction task but is not an oracle. For example, the feature vector could be a medical image, m n is the expert's diagnosis from looking at the image, and y n is a true label that can only be obtained from a biopsy. L2D also assumes that the human has access to background knowledge that the classifier does not, such as years of medical training in the aforementioned example. The L2D framework requires two sub-models: a classifier and a rejector [6, 5] . We denote the classifier as h : X → Y and the rejector as r : X → {0, 1}. When r(x) = 0, the classifier makes the decision, and when r(x) = 1, the classifier abstains and defers the decision to the human.Learning Learning in L2D requires we fit both the rejector and classifier. We assume that whoever makes the prediction-model or human-incurs a loss of zero (correct) or one (incorrect). Using the rejector to toggle between the human and model, we have the overall classifier-rejector loss:EQUATIONwhere I denotes an indicator function that checks if the prediction and label are equal. Minimizing this loss results in the Bayes optimal classifier and rejector:EQUATIONwhere P(y|x) is the probability of the label under the data generating process, and P(m = y|x) is the probability that the expert is correct. The assumption that the expert has additional knowledge is what allows it to possibly outperform the Bayes optimal classifier.One-vs-All Surrogate Loss Several consistent surrogate losses have been proposed for Equation 1 [11, 15, 10, 9, 2, 3] . For our implementation, we focus on Verma and Nalisnick [15] 's one-vs-all (OvA) parameterization, as it provides a calibrated rejector that will later be useful for conformal prediction. This parameterization assumes the classifier and rejector are unified via an augmented label space: Y ⊥ = Y ∪ {⊥}, where ⊥ denotes the rejection option. Then letg k : X → R for k ∈ [1, K]where k denotes the class index, and let g K+1 : X → R denote the rejection (⊥) option.The g functions are analogous to the logits of a neural-network-based classifier. The OvA surrogate loss is given as:EQUATIONwhere ϕ : {±1} × R → R + is a binary surrogate loss. For instance, when ϕ is the logistic loss, we have ϕ[f (x)] = log(1 + exp{-f (x)}). At test time, the classifier is obtained by taking the maximum over g functions:ŷ = h(x) = arg max k∈[1,K] g k (x).The rejection function is given as:r(x) = I[g K+1 (x) ≥ max k g k (x)].Conformal Prediction Conformal prediction (CP) is a distribution-free approach to uncertainty quantification with finite-sample guarantees [13] . Given a test-time feature vector x N +1 , CP seeks to construct a prediction set C(x N +1 ; τ) ⊆ Y such that the true label y N +1 is included with probability 1 -α:P y N +1 ∈ C (x N +1 ; τ) ≥ 1 -α, for α ∈ [0, 1]. τ is a parameter that is set on held-out calibration data, as will be described below. This statement is a marginal guarantee, meaning that it will hold, on average, over test samples but will not necessarily hold for any particular sample. CP's aforementioned guarantee is built off the crucial assumption that the test data is drawn exchangeably with the held-out set. Given a trained model whose softmax outputs are denotedf (x) = [f 1 (x), . . . , f K (x)].Then let the CP score function be given by: s (x, y;f ) = 1 -f y (x),where f y (x) is the softmax score for the true label. CP calibration then proceeds by evaluating s (x, y; f ) on all points in the held-out set and setting τ to be the (1 -α) quantile (with a finitesample correction) of the empirical distribution of scores. For a test time point x N +1 , the prediction set is constructed as: C(x N +1 ) = {j|f j (x N +1 ) > 1 -τ }, which represents the softmax dimensions that outscore the threshold 1 -τ . 
uncertain deferral via conformal prediction: 
 We will now apply the CP framework to quantify the uncertainty in the rejector sub-component of an L2D system. Concretely, instead of just outputting 0 (model) or 1 (human), we want the CP-based rejector to output a set C r (x; τ), which is an element of the superset {{0}, {1}, {0, 1}}.C r (x; τ) = {0, 1} means that the rejector is unsure if the decision should be allocated to the human or model. Recalling the Bayes optimal decision rule for the rejector (Equation 2), it would be ideal if C r (x; τ) could satisfy the following guarantee:P I P m N +1 = y N +1 | x N +1 ≥ max y∈Y P y N +1 = y | x N +1 ∈ C r (x N +1 ; τ) ≥ 1 -αwhich means that, marginally, the probability that the output of the Bayes optimal rejector is in the set is at least 1 -α. Constructing an adaptive set via validation statistics, unfortunately, requires we have access to P(m = y|x) to compute a non-conformity score. Moreover, if we did have access to P(m = y|x) (or a close approximation), then we could exactly quantify the uncertainty in deferral by direct use of P(m = y|x) and have no need for conformal sets.We instead consider I m N +1 = y N +1 , an indicator function representing if the human will make the correct prediction. We wish to construct prediction sets such that this binary variable will have an analogous coverage guarantee:EQUATION)This statement is not equivalent to the one above since the expert could be correct (i.e. I m N +1 = y N +1 = 1) but P(y|x) still be a better predictive model (i.e. r * (x) = 0). In other words, it is considering the expert's performance in isolation of the classifier's. However, the semantics are retained since C r (x N +1 ; τ) = {0} means that the expert will likely be wrong and so using the classifier is either a good decision or not an inferior one (if the model would also be wrong). This relaxation also allows us to define a conformity statistic from which to compute a practical set, as we will next discuss below. Later we will go on to discuss how the uncertainty sets will be used to improve decision-making within the L2D framework.We can construct deferral sets that follow the guarantee in Equation 4 by treating the deferral decision as a binary classification problem. For the OvA parameterization, the probability of the expert being correct is directly parameterized: P OvA (m = y|x) = (1 + exp{-g K+1 (x)}) -1 . Moreover, this estimator is usually well-calibrated [15] . Thus we construct a non-conformity score similarly as above but using P OvA (m = y|x):EQUATIONThis is the same score as introduced above for multi-class classification, but modified for binary classification. To obtain the threshold τ , one would follow the usual procedure of computing these non-conformity scores on a validation set [1] , obtaining the (1 -α) empirical quantile, and then applying the threshold at test time as follows:C r (x; τ ) =    {0} if (1 -P OvA (m = y|x) ≥ 1 -τ ) and (P OvA (m = y|x) < 1 -τ ) {1} if (P OvA (m = y|x) ≥ 1 -τ ) and (1 -P OvA (m = y|x) < 1 -τ ) {0, 1} otherwise (6)Now that we have detailed how to construct CP deferral sets, we next address how to use them to improve decision making within the L2D framework. While there are alternative uses, below we detail two that we believe will be practical and useful in a wide variety of applications.Abstention The use that likely first comes to mind is prediction with the option to abstain [4] . In the traditional case, the classifier only makes a prediction if it is confident; otherwise, it abstains.Our CP deferral sets allow for a similar workflow, but instead of abstaining because the prediction is uncertain, the L2D system will abstain because it is uncertain about to whom to allocate responsibility, the machine or human. Specifically, if C r (x N +1 ; τ ) = {0, 1}, then the L2D system will abstain. Otherwise, it will use the decision maker indicated by the set.We next consider how to make a prediction even if C r (x N +1 ; τ ) = {0, 1}.If the rejector is uncertain to defer or not, we propose querying both the model and human for their predictions. If they agree, then that consensus prediction is output as the L2D system's final prediction. If they do not agree, then the system abstains from making any prediction. This workflow has the same appeal to safety as the abstention-only option, but it will likely have higher coverage since it will make predictions when the abstention-only workflow would not. 
experiments: 
 Dataset and Synthetic Experts CIFAR-10 comprises 60,000 32x32 color images, uniformly distributed across 10 classes [7] , partitioned into 50,000 training and 10,000 testign images. To simulate a human expert, CIFAR10-K dataset applied the following method over the CIFAR-10 dataset [11] : if the image label y i ≤ k, expert will predict perfectly, otherwise expert make an uniform random prediction over all classes.Original We built our models using the Ψ OvA surrogate loss function, [16] . In this experiment, we used k = 6 and applied a simple three-layer convolutional neural network (CNN) for CIFAR10-K to balance the expert predictions [12] . The CNN classifier itself achieved an accuracy of 70.37% before learning to defer.Conformal Following the conformal approaches proposed in Section 3, our conformal method would be utilized to quantify the uncertainty of the rejectors trained with the surrogate loss functions. In this approach, we would use the validation data set to calibrate the conformal prediction by computing the threshold τ . 1 shows that the conformal method outperforms the original rejectors in terms of abstention and consensus predictions when safety guarantees are set with α = 0.1 from five repeated runs.Additionally, high levels of abstention and consensus sample coverage suggest that the predictions effectively encompass the majority of the testing data. 
conclusion: 
 In this paper, we have introduced an uncertainty-based method designed to enhance existing L2D systems and help their rejectors incorporating the uncertainty. However, the conformal scoring function shall be carefully parameterized to best present the probability of expert making correct prediction. Future work could explore the extent to which this method helps maintain system safety and robustness across various failure mode such like distribution shifts. 
 
 Appendix:  



Reviews: review: title: title: Novel Beta Unit Hyper-Cube kernel improves Bayesian Optimization on bounded domains
main_review: main_review: **Summary**

The paper proposes a new non-stationary kernel, the Beta Unit Hyper-Cube (BUC) kernel, designed for Bayesian Optimization on bounded domains. It provides theoretical bounds for information gain and cumulative regret and demonstrates the BUC kernel’s superior performance compared to Matérn and RBF kernels in optimizing black-box functions and model compression tasks.

 - **Quality:**
Strong theoretical foundation with robust experiments, though slightly limited in task variety.

 - **Clarity:**
Well-structured but dense theoretical sections could be more intuitive for non-experts.

- **Originality:**
Novel introduction of the BUC kernel tailored for bounded domains, addressing a significant gap in the field.

- **Significance:**
Important for tasks involving bounded domains, but broader applications beyond model compression would increase impact.

**Strength**
- Novel BUC kernel for bounded domain optimization.
- Strong theoretical analysis with proven bounds.
- Outperforms traditional kernels in model compression tasks.

**Weakness**
- The theoretical analysis is to dense
- Limited to a narrow set of applications (e.g., model compression).
- Could benefit from broader task diversity to show general applicability.

**Questions**
- How does the BUC kernel perform in a broader range of real-world tasks beyond model compression, such as hyperparameter optimization or robotics?
- What are the practical guidelines for choosing the bandwidth parameter h for the BUC kernel, and how sensitive is performance to this parameter?
- Can the BUC kernel efficiently scale to very high-dimensional problems, especially when applied to large datasets
- What is the computational complexity of the BUC kernel compared to Matérn and RBF kernels, particularly in large-scale optimization tasks
title: title: Official Review of Submission122 by Reviewer Dt7d
main_review: main_review: Summary: This paper uses conformal prediction for the rejector sub-component of the Learning-to-Defer (L2D) framework, allowing the rejector to output a set of possible outcomes rather than just a binary decision (defer or not). This method improves the safety and robustness of L2D systems by allowing the rejector to express its uncertainty. 

Strengths:
1. The idea of combining CP with the L2D framework is unique and promising. Explicitly quantifying uncertainty allows the system to handle decision-making under uncertainty more robustly.
2. The problem setting is important and relevant to real-world scenarios, demonstrating how CP can be applied beyond simple uncertainty quantification to enhance decision-making in human-AI collaboration systems.
3. The paper is well-written and maintains a high level of technical soundness, presenting the concepts clearly and justifying the proposed approach.

Weaknesses:
1. While the idea is promising, the paper could benefit from a more in-depth discussion on different kinds of non-conformity scores, such as APS [1] and RAPS [2], as mentioned in the conclusion. 
2. Integrating CP into the L2D framework may introduce additional computational complexity, which is not fully addressed or evaluated, especially in scenarios requiring real-time decision-making.

[1] Romano, Yaniv, Matteo Sesia, and Emmanuel Candes. "Classification with valid and adaptive coverage." Advances in Neural Information Processing Systems 33 (2020): 3581-3591.

[2] Angelopoulos, Anastasios Nikolas, et al. "Uncertainty Sets for Image Classifiers using Conformal Prediction." International Conference on Learning Representations.