SEL-BALD: Deep Bayesian Active Learning with Selective Labels
Authors: Ruijiang Gao
Mingzhang Yin
Maytal Saar-Tsechansky
Venue: NeurIPS 2024 poster (2024)
Keywords: Bayesian Active Learning with Disagreement; Selective Labels;
Primary area: active_learning
TL;DR: We propose novel methods for active learning under the selective labels problem.

Abstract: Machine learning systems are widely used in many high-stakes contexts in which experimental designs for assigning treatments are infeasible. When evaluating decisions is costly, such as investigating fraud cases, or evaluating biopsy decisions, a sample-efficient strategy is needed. However, while existing active learning methods assume humans will always label the instances selected by the machine learning model, in many critical applications, humans may decline to label  instances selected by the machine learning model due to reasons such as regulation constraint, domain knowledge, or algorithmic aversion, thus not sample efficient. 
In this paper, we study the Active Learning with Instance Rejection (ALIR) problem, which considers the human discretion behavior for high-stakes decision making problems. We propose new active learning algorithms under deep bayesian active learning for selective labeling (SEL-BALD) to address the ALIR problem. Our algorithms consider how to acquire information for both the machine learning model and the human discretion model. We conduct experiments on both synthetic and real-world datasets to demonstrate the effectiveness of our proposed algorithms.

SEL-BALD: Deep Bayesian Active Learning with Selective Labels
Ruijiang Gao
Mingzhang Yin
Naveen Jindal School of Management University of Texas at Dallas Richardson, TX 75082
Warrington College of Business
University of Florida Gainesville, 32611 mingzhang. wiw.aagggiiinvaarnnttooooo ufl edu
ruijianng.ga@@ttaallas gwwoaaaaaallssa......a edu
Maytal Saar-Tiecchnsky
Information, Risk, and Operations Management
University of Texas at Austin
Austin, TX 78712
maytal www.us.eeeee....o edu
Abstract
Machine learning systems are widely used in many high-stakes contexts in which experimental designs for assigning treatments are infeasible. When evaluating decisions costly, such as investigating fraud cases, or evaluating biopsy decisions, sample-efficient strategy is needed. However, while existing active learning methods assume humans will always label the instances selected by the machine learning model, in many critical applications, humans may decline label instances selected by the machine learning model due to reasons such as regulation constraint, domain knowledge, or algorithmic aversion, thus not sample efficient. In this paper, we study the Active Learning with Instance Rejection (ALIR) problem, which considers the human discretion behavior for high-stakes decision making problems. We propose new active learning algorithms under deep bayesian active learning for selective labeling (SEL-BALD) address the ALIR problem. Our algorithms consider how to acquire information for both the machine learning model and the human discretion model. We conduct experiments both synthetic and datasets to demonstrate the effectiveness of our proposed algorithms.
Introduction
Machine learning is increasingly deployed in high-risk applications, including medical diagnosis, fraud detection, and criminal justice [Zeng et al. 2017 Cecchini et al. 2010 Savage 2020). Given enough labeled data and the same of feature input, a machine learning model may achieve better performance than human experts Abramolff et al. 2023]. Therefore, there have been growing efforts develop machine learning (ML) models to assist humans improving high-risk decisions.
While modern deep learning algorithmms' strong performance across applications makes them par- ticularly promising for supporting experts, they often Jata-hungry and require large amount of labeled data to perform well. Yet, in many impactful high-risk applications acquiring a large number of labeled data is highly costly costly human experts' time and effort. For instance, labeling insurance claims requires human experts' resources establish if insurance claim is fraudulent; similarly, determining medical diagnoses may require diagnostic procedures that may be time-consuming and possibly risky for patients. Active learning methods can be promising path address this problem, given they aim select especially
be prohibitive, such as when labeling necessitates
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

40 Labeled
A0: Not Labelled
A
D - (u
P
Accepe (4,))
XED
Rejezt NM
Dpou Pla - 1/x0 -0
Deel 1x)
Updared Model
(a) ALIR
(b) Toy Example
Figure 1: (a): Active Learning with Instance Rejection (ALIR). The requester wants to acquire additional labeled samples to improve the ML classifier f. Due to the selective labels problem, only accepted labeling requests are labeled and added to the dataset D. (b): The most informative samples to the current ML classifier may not be the best samples to label in ALIR. The instances the left of Dool are more informative to the current classifier yet humans will never label these instances.
informative instances for labeling to minimize the number of labeled instances necessary to achieve desirable performance [Cohn et al. 1996].
To do so, most active learning methods operate iteratively, where an initial machine learning model is first trained on small labeled data set and the model then informs the selection of particularly informative instances to be labeled. A new model is then retrained on the augmented labeled data and this process is repeated until desirable goal is achieved.
However, most active learning methods assume instances selected for labeling by the algorithm will always be labeled [Settles] 2009]. However, in many real-world high-risk contexts, such as acquiring diagnoses through costly invasive diagnostic procedures or obtaining the conclusions of insurance fraud investigations, experimental design is infeasible and it is not possible to automatically acquire any label the active learning algorithm deems informative. Rather, requests for label acquisition must be considered and approved by human expert. Indeed, some diagnostic biopsies may never be approved by the human doctor because they are inconsistent with medical practices (Dai and Singh 2021]; if the requests are declined, the biopsy outcomes will not be revealed to benefit the machine learning model. Similarly, an insurance fraud agent may decline request for an investigation because of the remote location or because of a prior belief that the case is unlikely to be fraudulent. Any given human discretion behavior over requests to acquire labels at a cost is unknown and can be influenced by various factors. This is known as the selective labely problem [Lakkaraju et al. 2017] Kleinberg et al., 2018 De-Arteaga et al. 2018 Weil 202T]. in which the outcomes are more likely to be available for subset of instances. When human expert must approve the decision to obtain label, seleetive labeling would result from the expert's 's discretion over the labeling process.
Independently of the labeling costs, human decision makers also need to invest time and effort in considering the algorithm's 's recommendation before making decision. For example, the insurance agent needs to evaluate each case in order to decide whether an investigation should be launched and doctors will go through patients' medical record and assess the risk prior to approving diagnostic procedure. Therefore, our goal is to maximize the machine learning model's predictive performance within a total budget considering both the examination and labeling cost.
Ultimately, while traditional active learning focuses on identifying instances that would be most informative for learning, when the human must approve each labeling request, the human may be inundated with requests to consider acquisitions, any portion of which may be declined and thus not benefit learning, while incurring a significant loss of the human expert's resources. This poses a new challenge for active learning in which it would be valuable to bring to bear the human discretion of any labeling request so as to identify labeling requests that can achieve the most improvement in model performance for given labeling and human discretion costs. We refer to this problem as Active Learning with Instance Rejection (ALIR), illustrated in Figure [aa] Crucially, in ALIR, the instances deemed most informative for the current MI. classifier may not be the best candidates for labeling, as humans may choose to reject these instances (as shown in Figure [(b)). This challenge
highlights the need for developing new active learning methods tailored to the ALIR framework.

---

In this paper, we make the following contributions:
We study the Active Learning with Instance Rejection (ALIR) problem that considers the human discretion behavior in high-stakes selective labeling contexts. Unlike conventional active learning, where every instance is labeled by humans, ALIR allows humans to exercise discretion, selectively choosing whether or not to label certain instances.
We propose different active learning algorithms for ALIR under the bayesian active learning with disagreement framework and discuss their trade-offs. Unlike traditional active learning, the benefit of each method depends on the underlying human discretion behavior. We show theoretically that ALIR is equivalent to traditional active learning when the human discretion behavior is homogeneous across instances and propose new methods for unknown, heterogeneous discretion behaviors.
We conduct comprehensive experiments on both synthetic and real-world datasets to demon- strate the effectiveness of our proposed algorithms.
2
Related Work
Active Learning:
To render costly data annotation more efficient, Active Learning has been studied
extensively in the machine learning community (Lewis and Gale 1994, Schohn and Cohn 2000
Nguyen and Smeulders 2004 Freytag al. 2014 Gal al., 2017 Sener and Savarese 2017
Gong et al.] 2019] Gao and S Tsechansky 2020). Traditionally, the key challenge addressed
by most methods is to label the most informative unlabeled instances from which to induce the classifier. Existing methods assume humans will always label the instances selected by the active learning algorithm. We consider the selective labels problem where in many high-stakes decision making problems, humans may not label the selected instances due to their past experience, domain knowledge, or algorithmic aversion Dai and Singh 202 De-Arteaga et al., 2018 Dietvorst et al.) 2015). Recent papers have studied active learning for treatment effect estimation from observational data (Jesson al. 202T] Deng al. 2011 [undinn al., 2019]. Unlike these papers, we study the selective labels setup where the outcome is selectively observed, which is different from the treatment effect estimation problem where we can observe an outcome under the selected treatment arm. The ALIR problem is also known as the Active Learning with Abstention Feedback problem and studied
by other papers (Fang et al., 2012 Yan al. 2013 Amin et al. 2021 Nguyen et al. 2022 Robinson et al. 2024]. Fang et al. [0122] is close to the proposed e- BALD method but we use the bayesian neural network for modeling human behaviors, Yan et al.] (2015) additionally models noisy human
feedbacks. AAmin et al.] [2021] considers different setting where a machine can determine whether
human or the AI should label the instance. A concurrent work [Robinson et al. 2024] motivates the problem from the e-commerce perspective where users may not click on the ads the platform sent and not observing the purchase label. Unlike their work, we motivate the problem from the human-Al
perspective and provide coommattion-thhrettie results for the proposed criteria with a focus on the deep baycsian active learning setting.
Selective Labels: Lakkaraju et al. [0177] proposes the "contraction" technique to evaluate a machine learning classifier's performance in the selective labels problem. However, the point identification of the model performance is generally impossible without strong assumptions and partial identification
studied in the recent literature [De-Arteaga et al., 2018 D Amour 2019 Guo al.) 2022] Yin et al., 2024). Unlike previous papers on studying the limitations in evaluation and optimizing predictive rules in the selective labels problem, we focus on how to acquire useful information to improve the machine learning model in the selective labels problem. Rambachan and Roth [2019) studies how historical numan-generatee data may bias the machine learning model in the selective labels problem and our paper studies how to work with the human decision makers to improve the machine learning model in selecting informative samples.
Human-Al AI Collaboration: There are works in the machine learning community designing different forms of human-Al collaborated decision-making systems such as learning to defer [Cortes et al.)
2016, Madras et al.) 2018 Gao et al. 2021 Gao and Yin| 2023 Gao et al.) 2023], or learning to
advise [Bastani et al. 2021 Grand-Clément and Pauphilet 2024, Bhatt et al., 2023 Cao et al. 2024
Unlike these work, we focus on how to collect information under human supervision. Our work
also motivated by the literature on overriding decisions and trust in algorithm adoptions [Sun)
et al. 2022
Wang et al. 2022
Dietvorst et al. [2015) finds that humans are likely to override the
3

---

algorithm's 's decisions when they observe machine learning algorithms make mistakes, Bansal et al. (02TT) studies how the quality of the explanation and the predictive quality of the machine learning model affect human decisions. There are also works that study different interventions to improve the trust of humans the machine learning model (Dietvorst et al. 2018 Lakkaraju and Bastani 2020 Zhang et al., 2020]. de Véricourt and Gurkan [023] uses an anal malyyical modeling framework and stylized human behavior model to show that humans may never know the true performance of Al in the selective labels problem. Our work is different from these papers as we focus on how to acquire useful information to improve the machine learning model in the selective labels problem.
Problem Statement and Background
3.1 Selective Labels Problem
We have access to a pool of unlabeled data D Pooa = {2x)==1. where each instance has a underlying
label V/ E {1,... K.. We denote the marginal distribution of X P(X) and the conditional distribution of Y given X as P(Y We assume that the unlabeled data is drawn i.i.d. from P(X). A human decision maker will make yes decision (a, = 1) or decision (a, 0) for the decision subject. The corresponding outcome ys can only be observed when a; =
We assume there exists human discretion function e(X) that maps the input X to probability of being labeled by the human decision maker. We denote the probability of X being labeled as P(A= 1 = e(X). This corresponds to how humans will examine and label the instances. Then for an instance Z₁. 0. = 1, it will be labeled, returning the label yı. and be added to the current labeled dataset D {x,: Vi } =1a==1- Otherwise, the human labeler will reject it and outcome will be available for the instance. No matter whether the instance is labeled or not, (I,, a;) will be added to the current human decision dataset L = {ar, az 1 which records all the human labeling information. A predictive model with likelihood P(y) x. f) parametrized by ~ P(e)D (D) can then be trained on the labeled dataset D.
3.2
Active Learning
The requester now wants to acquire some additional labeled samples to improve the ML classifier. Since human labelers need time and resources to decide whether to investigate each instance (e.g.. have a meeting to decide whether to operate on the organ, decide on whether to audit the firm) and label it (e.g., perform the biopsy, audit the firm), the requester wants to minimize the number of instances that need to be labeled. We denote the cost for examining the instance as ce and the cost for labeling the instance e₁. The total cost is the sum of the current examination cost and label cost. Therefore, with a total budget of B, the requester wants to select a subset of instances S C Pooo to be labeled by the human labelers while maximizing the performance of the ML classifier P(y x. 0).
Given model P(yl pr, f), pool dataset D puul and labeled dataset D, active learning algorithms use an acquisition function A(x, ff) to select where to label next by arg max A(z, 0). For example, A(x, 0) unif() corresponds to the random acquisition function, which unif() returns a uniformly random number between 0 and 1; and A(x, 8) Hy) D) = E. Ply = c/æ D) log c/r, D)) corresponds to the uncertainty sampling principle that picks the next sample with the largest predictive entropy. Our algorithm extends the following Bayesian Active Learning with Disagreement
(BALD) algorithm (Houlsby et al., 2011 Gal et al. 2017
Bayesian Active Learning with Disagreement (BALD): BALD (Houlsby et al. 2011 uses the epistemic uncertainty as the aquisition function. The epistemic uncertainty measures the uncertainty about model parameters instead of the inherent uncertainty about the data that cannot be reduced by acquiring more samples. The information gain is defined as the mutual information between the label and the model parameters which can be written as follows:
T(y, 0|., D)= = Eyle, D (0 H( x, D))
=-
Σ Ply cr, D) log( P(y c|ar, D)) + E
Σ Ply = dlr, (2,

---

Here the posterior D) can be approximated by the approximated posterior (( from the Bayesian neural network, and using MC sampling [Gal al. 2017] approximate Equation (T as
Σ
v.
log
pe log(d))
where Di is the probability of input taken class under the (-th MC sample from the approximated posterior (0). We denote this objective as I(yy 0|2,, D) I[y. 0| D).
BALD will then select the instance by arg maX [ly 0 n4 D).
4
Methods
4.1
Naively Apply Bayesian Active Learning with Disagreement (BALD)
If we ignore the selective labels problem and simply treat it as the traditional active learning setup where every instance will always be labeled, we can use the same computation procedure in the standard BALD algorithm (in the supervised learning setup) to select the next instances to label. Definition 4.1 (Naive-BALD).
Naive (y, 02 D) : xr D,a= 1)
Ply c|z, D, ) ) P(y 0/,00, 1))
- P(OD)( Σ P(y
1) log(
(2)
However, when humans are involved, we need to consider the probability of each instance being labeled. When the instance with high information gain is unlikely to be labeled, BALD may send many instances to the human labelers but only few of them will be labeled, thus wasting the time and resources of the human labelers.
4.2 Bayesian Active Learning with Disagreement with Selective Labels (SEL-BALD)
First, we show the information theoretical objective under ALIR. When considering the human discretion behavior, the information gain i(y, 0|., D) can be derived
Definition 4.2 (e-BALD).
i(y, 0|., D) =e(x))((( 0|2, 1)
Ply = cl., D. P(y c|r 0,0 0
E(x
Σ P(y = 0, log(
(3)
The first equality is proved in Theorem A.1 in Appendix A Intuitively, this objective indicates should focus on instances with higher probabilities to labeled by human decision makers. If e(x) is known exactly, we can directly select the instances by the acquisition function e(x)i(y, 0|z D). It's worth noting that when e(x) = 0 < c S 1, which means human decision makers will choose to label all instances with the same probability c, the Raive-BALD and e-BALD objective is equivalent and the problem reduces to the traditional active learning setting.
Since in our setting, e(x) is not known a priori, we can fit another predictive model é(x) P(a 1 J2x Φ) the human decision dataset L. to approximate this objective. However, in practice, é(x) may be inaccurate given small dataset. Therefore, we need to acquire samples that are 1) likely to be labeled by the human decision makers and are both informative to the machine learning model and the human discretion model update.
We consider the information gain about the human discretion response and the model parameters. Assume we use a Bayesian neural network (x) to approximate the human discretion model e(x), can define the information gain about the human discretion response and the model parameters as
o(, () x=
Σ
0e log 1
0
+
Σ c.t
ge log(o")
cEf0.1)
5

---

Algorithm 1 Bayesian Active Learning for Selective Labeling with Instance Rejection (SEL-BALD)
Input:
Initial labeled dataset D, L. pool dataset D Pouot MC-samples T, budget B. Output: Updated machine learning model f(x). while Budget is not used up do
Select the instance 2; that maximizes the acquisition function in Equation 5p 6 or
Human will decide whether label the instance Ti. if the instance 2; is labeled then
Add (Iz, 34) to D and (Ty, a4) L.
else
Add (Ii, az) to C. end if'
Update the human discretion model Cor+i (x) on L. Update the machine learning model for41 (z)) on D. end while
Here J is the probability of input x being labeled as e under the 1-th MC sample from the approxi- mated posterior (()). We denote this objective as İ(l, ø|æ, C) = I(I, Φ b L). Selecting samples that maximize this acquisition function will help to learn more effectively about the human discretion model. This joint information gain can be estimated as
Definition 4.3 (Joint-BALD).
((fa, v). (0, oHH,, e(x)i(y, D) ø|x; L)
The first term in Equation 5 is equivalent to Equation (3) and the second term is the information gain between the human discretion model and human responses. If we have the perfect knowledge about the human discretion model, then the second term equals O and Joint-BALD reduces e-BALD. On the other hand, we have high epistemic uncertainty about human responses, the second term will encourage exploration to learn the human discretion model better. Once the second term becomes small (we have a good understanding of the human discretion model), Joint-BALD will automatically reduce to e-BALD and start exploiting.
We prove Equation (5) in Theorem A.2
Motivated by the upper confidence bound (UCB) principle in online regret minimization Li et al. 2010). we can define the acquisition function as Definition 4.4 (Joint-BALD-UCB).
Auer 2002
f(l,,)), (0,00)r, Q B) D) + i(1,, 6lr,
(6)
Here Q is the empirical quantile function and (2)) is the upper confidence bound of the probabilities (drawn using the MC samples) of x; being labeled by the human decision makers of our current best estimation. The hyperparameter 3 balances the exploration-exploitation trade-off. With larger 3 + 1, the acquisition function will have more exploration samples with higher uncertainties. Similarly, we can use Thompson Sampling [Thompson, 1933 to control the degree of exploration, which leads to the following objective:
Definition 4.5 BOLD--BALD-TS)..
f( v), (0, o) x. x, D) = ((e 0lx, fi(1, $/0, C)
(x)( represents the empirical sample drawn from the MC samples. Each time we will random sample set model parameters from the posterior and select the instance with the largest information gain calculated using Equation (7). Compared to UCB, foint-BALD does not have to specify the hyperparameter. The complete algorithm for Bayesian active learning with selective labeling (SEL-BALD) is shown in Algorithm
5 Experiments
First, we want to emphasize that in ALIR, the benefit of each method is more nuanced than traditional active learning and highly depends on human decision behavior and the underlying data distribution.

---

As our analysis above suggests, Waive-BALD and e-BALD correspond to the information gain when e(z) c, < c S 1. When c 1. this corresponds to the traditional active learning problem where humans always label the instance, and the efficacy of Bdive-BALD and e-BALD have been demonstrated in existing papers [Gal et 2017]. We also conduct the experiment under homogeneous human decision behavior in Appendix C In this section, we aim to show the benefit of the proposed method when the human discretion behavior is unknown and heterogeneous.
We conduct experiments on both synthetic and real-world datasets to demonstrate the effectiveness of our proposed algorithm when human discretion behavior is unknown and heterogeneous. We assume there is a cost of 1 to examine every data instance and cost of 5 to label the instance. For the predictive model and human discretion model class, we use Bayesian neural network and use MCdropout [Gal et al., 2017] to approximate the posterior (we set the number of MC samples as 40 in all experiments). The model architecture is a 3-layer fully connected neural network with LeakyRel.U activation function. We use the Adam optimizer with a learning rate of 0.01. We run the experiments on a server with 3 Nvidia A100 graphics cards and AMD EPYC 7763 64-Core Processor. We compare our proposed methods against the Naive-BALD algorithm and the random acquisition function (RANDOM). We set 3 0.75 for -An--BALD-UBB in all experiments. For results, we report the accuracy and number of samples acquired of the machine learning model on a subset of the test set where the instance has a positive probability to be included in the dataset. We do not consider the performance of instances that have zero probability of showing up in the data since human decision makers will always reject such decision instances and a machine learning model can have arbitrary performance such subpopulation. The code is available at https:/ //gi thub. con/ruijjangg1//SLL
5.1 Synthetic Data
We use a comprehensive synthetic example to illustrate the benefit of the proposed algorithms. We generate a synthetic two-cricle dataset with 2 features. The response variable is binary and can be considered as fraud of an insurance claim. We generate 20000 data samples and downsample 90% of the data points with 2 2 0 where æ1 is the second dimension of the input feature. This is to make the toy example little more challenging for the RANDOM bascline and simulate a real-world scenario where there are lot of insurance claims in the candidate pool that won't 't be investigated where active learning is necessary to reduce the potential waste of human workers' time. The resulting dataset has around 3700 samples as the training set and around 1600 samples as the test set. The visualization of the synthetic data is shown in Figure [T(])
The qualitative results for each label acquisition method are shown in Figure 2 and the quantitative metries are reported in Table Since there are many high-risk samples that human decision makers do not label, most of the samples RANDOM selects are not labeled by the human decision maker. Similarly, Skive-BBLDD is highly influenced by the initial labels and human discretion behavior. If the most informative samples to the current machine learning classifier have a low probability to be labeled by humans, Naive-BALD will keep sending such instances to humans to examine and waste all the budget. The initial labels sampled are shown in Figure [1 Since there are not many samples when x1 < 0, the human discretion model cannot estimate the correct human decisions accurately. As a result, e-BALD relies on the noisy estimation of the human discretion model and greedily explores the lower right region of the feature space while it is important to acquire samples from the left region in order to learn the correct decision boundary. Compared to e-BALD, Joint-BAI.D has a better exploration and labeled a few instances on the left, however, it is also influenced by the noisy estimation of e(x) and focus more on the points the right side of the sample space. Bait-BBLD-UCB and Sant-BBAL--T- both encourage more exploration for e(x). As a result, both methods can learn the correct decision boundary in sample-efficient way.
Finally, we show the estimated human discretion model in Figure 3 based on its current noisy estimation of the human discretion model, it fails to learn the correct decision boundary and maintains a wrong probability estimation about the left lower region of the feature space. While Joint-BALD has a slightly better estimation, it still focuses more on the high- high-probability region and has large estimation error on the left lower region. Joint-BAID--UCH and Bitt-BBLD-UCB correctly recover the decision boundary with the help of sufficient exploration.
Since e-BALD acquires samples
7

---

(a) Synthetic Data
(b) Initial Labels
(c) RANDOM
(d) NaiDe-BALD
(e) e-BALD
(I) Joint-BALD
(g) BouttBBLD--CBB
(h)
Figure 2: Qualitative Results for Synthetic Data. Figure (I shows the underlying synthetic data distribution. Figure [Tbb] shows the initial labels randomly acquired. Figure 2 (c)-(h) show the labeled data the corresponding learned decision boundary after spending a budget of 450. Human decision makers always reject label when budget on examination since many candidate applications are in the high-risk region and have low probabilities to be labeled. In contrast, e- c-BALD selects the samples that most likely to be labeled according to a noisy estimation of the human discretion model, which overlooks the lower left region which is important for learning the correct decision boundary.
O. RANDOM and Naive-BALD BALD spend most of the
RANDOM
Naive-BALD
c-BALD
Joint-BALD
Joint-BALD-UCB
Joint-BAl Boint-BALD-TS
Accuracy 0.78
0.55
0.76
0.77
0,99
0.98
Examination (%)
0.91
0.95
0.39
0.52
0,53
0,50
Laheling (%)
0.09
0.05
0.61
0.48
0.47
0.50
Table I: Quantitative Statistics for Synthetic Data. We report the accuracy and the fraction of the budget spent for examination and labeling. RANDOM and Naive-BALD spend most of the budget on examination and few labels is acquired. e-BALD acquires the most labels but many of them are not informative due the noisy estimation of ezx). BoLL--BALD-UCB and BAL--S balance the examination and labeling budget automatically and achieve the best accuracy.
5.2 Case Study: Fraud Detection
For our case study, we use the Live-Me-some-Che (GMC) dataset [Credit Fusion 2011]. GMC dataset is real-world loan dataset with 10 features including age, monthly income, debt ratio, number of dependents, and binary response variable that indicates whether the loan application will default. We balance the dataset by random undersampling and randomly choose 400 samples as the test set. We use this dataset to build hypothetical insurance claim case. Assume each instance is an insurance claim where human agent can investigate whether it is fraudulent. The insurance company wants to build machine learning model predict fraud cases in the future. Since such investigation is costly, the company wants to use an active learning method and send some selected instances to agents for potential investigation and agents have the freedom to decline the investigation request since the company does not want to waste human resources and potential profit from acquiring these samples.
Human Behavior Model: For the human discretion model e(xx), we assume human decision makers will have the following acceptance probabilities:
0.3
High Debt and Low Income
<(x)=
0.9
High Debt and High Income
Low Debt.
8

---

(a) e-BALD
(b) Joint-BALD
(c) Joint-BALD-UCE
(d) Bhat-BBLL--SS
Figure 3: Estimated Human Discretion Behavior for Synthetic Data. We show the decision boundary
for the estimated human discretion behavior for each method after spending budget of 450. e-BALD underestimates the lower left region of the feature space due to the lack of exploration. Joint-BALD- UCB and BAB--B recover the correct decision boundary.
(a) GMC Acc.
(b) GMC Sample
(c) MNIST Acc.
(d) MNIST Sample
Figure 4: Results for GMC and MNIST. We report the accuracy and number of samples labeled for both datasets. RANDOM and Naive-BALD spend most of the budget on examination since they do not consider human discretion behavior. On both datasets, Joint-BALD-UCE and Joint- BAL--BAL-- demonstrate robust performance across different budgets.
detailed discussion about the initial human discretion model can be found in Appendix D This simulates the selective labels setup where the biased human decision makers are skeptical about insured customers with high debt and these customers are more likely to be investigated.
We report the accuracy and number of samples acquired for each method in Figure 4[a)and Figure 4bb respectively. The results averaged over 3 runs with query size of 10, 50 randomly examined instances initially, and a budget 450. Compared to other methods, RANDOM and Naive-BALJ) acquire the least samples given the same budget since both methods do not consider the human discretion behaviors, which leads to suboptimal model performance. e-bald has small benefit over all other methods when the total budget is small, which expected when the initial estimated human discretion model is useful since e-BALD greedily acquires samples based on the current best estimation. However, this effect soon disappears and e-bald has worse performance compared to Lant-BALD-UCB and
which have a better exploration of e(x) for larger budget.
5.3 MNIST
We also examine each active learning method on a nigh-dimensionee dataset MNIST [LeCun 1998]. MNIST is large handwritten digit dataset with 28x28 pixels and 10 classes. The test set size is 10000 and we use 10000 samples the training set as Ppont- We report accuracy and number of samples acquired for each active learning method on the test set. The results are averaged over 3 1000. For
runs with a query size of 20, 100 randomly examined instances initially, and a budget the human discretion behavior, we assume Pa 1|x) for all classes 0-6, and human decision makers have the probabilities of 0.3, 0.3, 0.9 to label the instances with classes 7, 8, 9 respectively. This also corresponds to the realistic setting where many insurance claims will not be investigated and human discretion behavior is different for different instances.
The results are shown in Figure 4(c) and Figure 4(d) baselines like RANDOM and Naive-BALD struggle to acquire samples humans will label, and waste most of the budget on examination. For high-dimensional data, the performance of e-BALD appears worse the beginning potentially because the propensity score harder estimate for high- dimensional data D Amour et al. 2021 On the other hand, Joint-BALD BALD-UCE and
respectively. Similarly to our previous findings,

---

are able to learn the human discretion model better through exploration, and lead to a larger number of samples labeled and better model performance.
5.4 Additional Experiments
While we motivate the Active Learning with Instance Rejection problem in the cold-start setting, the ALIR problem can still happen when human behavior changes, thus we need to acquire new labels to better model the underlying function and the human discretion model. We offer an illustrative example of ALIR under changing human behaviors in Appendix B In the example, we assume the company starts to use machine learning model to advise human decision makers, thus leading a change in the human discretion behavior. We further demonstrate that the ALIR problem still exist when human behaviors change and the proposed methods in the paper can also be applied in such settings.
Changing Human Behaviors:
More Real- World Datasets: We compare the proposed methods with baselines on Fashion MNIST Xiao al. 2017]. CIFAR-I0 (Krizhevsky 2009), Adult [Becker and Kohavi 1996) and Mushroom mus 1981 datasets in Appendix We find similar findings that the proposed methods improve
over Naive-BALD and RANDOM under the selective labels setting.
Other Human Behavior Model: In addition to the piecewise linear function used to simulate the human behaviors, compare with Logistic Regression based human behavior model in Appendix F We randomly sample a few positive and negative samples from the datasets to train a Logistic Regression model and use it simulate human responses. The results are similar to results presented in the main paper.
While we focus improving the BALD method in this paper, it is possible to extend the proposed method for these metrics by replacing the information gain term with another uncertainty term such as the entropy. We demonstrate that in Appendix G We implement Joint- Saant-Enrooy--CCC UCB that replaces the information gain with the entropy measure on the synthetic data. Joint- Entropy-UCUB can also improve the traditional entropy objective in this case.
Other Uncertainty Metric:
6 Conclusion and Future Work
In this paper, we study the Active Learning with Instance Rejection (ALIR) problem, which is an active learning problem that considers the human discretion behavior for high-stakes decision making problems. We propose new active learning algorithms e- e-BALD, Woint-BALD: Joint- BALD-UCB. and Joint-BALD-TS using deep Bayesian active learning for selective labeling (SEL-BALD) to address the ALIR problem and validate the effectiveness of our proposed algorithm on both synthetic and real-world datasets. With unknown and heterogeneous human behavior, Joint-BALL BALD-UCK and Joint-BALD-T BALD-TS often lead to robust model performance improvement across different budgets.
However, there are several limitations to work. First, we consider all humans single entity and assume they have the same discretion model which is an average of all humans. In practice, different human decision makers may have different discretion models. Second, it is also promising to use semi-supervised learning (Grandvalet and Bengio 2004) to train the predictive model while it brings additional challenges the ALIR problem. In addition, e(x) may not be accurately modeled which could affect the performance of the proposed algorithm. One potential issue is the misspecification of the model class used represent human behavior. For instance, if linear model like Logistic Regression is applied to fundamentally non-linear human behavior pattern, this could introduce estimation bias into the proposed AL method. To mitigate this, non- non-paraeetric models could be employed, as they are less prone to such misspecification bias. Last, it is also worth noting that in certain applications like recidivism prediction or organ transplantation, prioritizing instances to be labeled may have significant fairness implications, therefore our method may not apply. However, the ALIR problem still exist in many real-world problems such as tax audit or fraud detection. would be interesting to study the faimess concerns in ALIR. We believe these bring promising future research opportunities and leave these as future work.
10

---

References
Mushroom. UCT Machine Learning Repository, 1981. DOE: https://bl.or///0244444///599999
Michael D Abramoff, Noelle Whitestone, Jennifer L Patnaik, Emily Rich, Munir Ahmed, Lutful Husain, Mohammad Yeadul Hassan, Md Sajidul Huq Tanjil, Dena Weitzman, Tinglong Dai, et al. Autonomous artificial intelligence increases real-world specialist clinic productivity in cluster-randomized trial. NPJ digital medicine, 6(1):184, 2023.
Kareem Amin, Giulia DeSalvo, and Afshin Rostamizadeh. Learning with labeling induced abstentions. Advances in Neural Information Processing Systems, 34: 14412666--2588 2021.
Peter Auer. Using confidence bounds for exploitationn-explration trade-offss Journal of Machine Learning Research, 3NNov):397-422, 2002.
Gagan Bansal, Tongshuang Wu, Joyce Zhou, Raymond Fok, Besmira Nushi, Ece Kamar, Marco Tulio Ribeiro, and Daniel Weld. Does the whole exceed its parts? the effect of ai explanations on complementary team performance. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pages 1-16. 2021.
Hamsa Bastani, Osbert Bastani, and Wichinpong Park Sinchaisri. Improving human decision-making with machine learning. arXiv preprint 1xXi:21108.08444 5, 2021.
Barry Becker and Ronny Kohavi. Adult. http://bo..rr//112444222///////
UCI Machine Learning Repository, 1996. DOI:
Umang Bhatt, Valerie Chen, Katherine M Collins, Parameswaran Kamalaruban, Emma Kallina, Adrian Weller, and Ameet Talwalkar. Learning personalized decision support policies. arXiv preprint arXiv:2304.067001 2023.
Junyu Cao, Ruijiang Gao, and Esmaeil Keyvanshokooh. Hr-bandit: Human-ai collaborated linear recourse bandit. arXiv preprint arXiv:2410. 144440 2024.
Mark Cecchini, Haldun Aytug, Gary J Koehler, and Praveen Pathak. Detecting management fraud in public companies. Management Science, 56(7):1146-1166 2010.
David Cohn, Zoubin Ghahramani, and Michael Jordan. Active learning with statistical models. Journal of artificial intelligence research, 4:129-1444 145, 1996.
Corinna Cortes, Giulia DeSalvo, and Mehryar Mohri. Learning with rejection. In Algorithmic Learning Theory: 27th International Conference, ALT 2016, Bari, Italy, October 19-21, 2016 Proceedings 27, pages 67-82. Springer, 2016.
Will Cukierski Credit Fusion. Give me some credit, 2011.
URL
com/
Tinglong Dai and Shubhranshu Singh. Artificial intelligence on call: The physician's decision of whether to use ai clinical practice. Available at SSRN, 2021.
Maria De-Arteaga, Artur Dubrawski, and Alexandra Chouldechova. Learning under selective labels in the presence of expert consistency. arXiv preprint arXiv: arXiv:/807.0000 1807.00905 2018.
Francis de Véricourt and Huseyin Gurkan. Is your machine better than you? you may never know. Management Science, 2023.
Kun Deng, Joelle Pineau, and Susan Murphy. Active learning for personalizing treatment. In 2011 IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning (ADPRL), pages 32-39. IEEE, 2011.
Berkeley J Dietvorst, Joseph P Simmons, and Cade Massey, Algorithm aversion: people erroncously avoid algorithms after seeing them err. Journal of Experimental Psychology: General, 144(1):114, 2015.
Berkeley J Dietvorst, Joseph P Simmons, and Cade Massey. Overcoming algorithm aversion: People will use imperfect algorithms they can (even slightly) modify them. Management science, 64(3): 1175-1170, 2018.
11

---

Alexander D' Amour. On multi-caus cause approaches to causal inference with unobserved counfounding: Two cautionary failure cases and a promising alternative. In The 22nd international conference on artificial intelligence and statistics, pages 3478-3486. PMLR, 2019.
Alexander D' Amour, Peng Ding, Avi Feller, Lihua Lei, and Jasjeet Sekhon. Overlap in observational studies with figh--imensiooall covariates. Journal of Econometrics, 221(2):644 654, 2021.
Meng Fang, Xingquan Zhu, and Chengqi Zhang. Active learning from oracle with knowledge blind spot. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 26, pages 2421-2422, 2012.
Alexander Freytag, Erik Rodner, and Joachim Denzler. Selecting influential examples: Active learning with expected model output changes. In European Conference on Computer Vision, pages 562-577. Springer, 2014.
Yarin Gal, Riashat Islam, and Zoubin Ghahramani. Deep bayesian active learning with image data. In International conference on machine learning, pages 1183-1192. PMLR, 2017.
Ruijiang Gao and Maytal -sechassccan,k. Cost-aceuracy aware adaptive labeling for active learning. In AAAI, volume 34, pages 2569-2576, 2020.
Ruijiang Gao and Mingzhang Yin. Confounding-rooee policy improvement with human-ai teams. arXiv preprint arXiv:2310.08822 2023.
Ruijiang Gao, Maytal Tscc-nsech,nkky Maria De-Arteaga, Ligong Han, Min Kyung Lee, and Matthew Lease. Human- ai collaboration with bandit feedback. arXiv preprint trXiv:2105.00614 2021.
Ruijiang Gao, Maytal Saar-Tsechansky Maria De-Arteaga, Ligong Han, Wei Sun, Min Kyung Lee, and Matthew Lease. Learning complementary policies for human-ai teams. arXiv preprint arXiv:2302.02944, 2023.
Wenbo Gong, Sehastian Tsshiatschek,, Sehastian Nowozin, Richard E Turner, José Miguel Hernández- Lobato, and Cheng Zhang. Icebreaker: Element-wise efficient information acquisition with a bayesian deep latent gaussian model. Advances neural information processing systems, 32, 2019.
Julien Grand-Clément and Jean Pauphilet. The best decisions are not the best advice: Making adherence-ave aware recommendations. Management Science, 2024.
Yves Grandvalet and Yoshua Bengio. Semi-supervised learning by entropy minimization. Advances in neural information processing systems, 17, 2004.
Wenshuo Guo, Mingzhang Yin, Yixin Wang, and Michael Jordan. Partial identification with noisy covariates: A robust optimization approach. In Conference on causal learning and reasoning, pages 318-335. PMLR, 2022.
Neil Houlsby, Ferene Huszár, Zoubin Ghahramani, and Máté Lengyel. Bayesian active learning for classification and preference learning. arXiv preprint arXiv: 2011.
Andrew Jesson, Panagiotis Tigas, Joost van Amersfoort, Andreas Kirsch, Uri Shalit, and Yarin Gal. Causal-bald: Deep bayesian active learning of outcomes to infer treatment-effects from observational data. Advances in Neural Information Processing Systems, 34:30465-30478, 2021.
Andreas Kirsch, Joost Van Amersfoort, and Yarin Gal. Batchbald: Efficient and diverse batch acquisition for deep bayesian active learning. Advances in neural information processing systems, 32, 2019.
Jon Kleinberg, Himabindu Lakkaraju. Jure Leskovee. Jens Ludwig, and Sendhil Mullainathan. Human decisions and machine predictions. The quarterly journal of economics, 133(1)2337-293, 2018.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
12

---

Himabindu Lakkaraju and Osbert Bastani. how do i fool you?" manipulating user trust via misleading black box explanations. In Proceedings of the AAAI/ACM Conference on Al, Ethics, and Society, pages 79-85, 2020.
Himabindu Lakkaraju. Jon Kleinberg, Jure Leskovec, Jens Ludwig, and Sendhil Mullainathan. The selective labels problem: Evaluating algorithmic predictions in the presence of unobservables. In Proceedings of the 23rd ACM SIGKDD International Conference Knowledge Discovery and Data Mining, puges 275-284, 2017.
Yann LeCun. The mnist database of handwritten digits. http://yann. lecun. com/exdb/mmist// 1998.
David D Lewis and William A Gale. A sequential algorithm for training text classifiers. In SIGIR 94, pages 3-12. Springer, 1994.
Lihong Li, Wei Chu, John Langford, and Robert E Schapire. A contextual-bandit approach to personalized news article recommendation. In Proceedings of the 19th international conference on World wide web, pages 661-670, 2010.
David Madras, Toni Pitassi, and Richard Zemel. Predict responsibly: improving fairness and accuracy by learning to defer. NeurIPS, 31:6147-6157 2018.
Cuong V Nguyen, Lam Si Tung Ho, Huan Xu, Vu Dinh, and Binh T Nguyen. Bayesian active learning with abstention feedbacks. Neurocomputing, 471:242-250, 2022.
Hicu Nguyen and Amold Smeulders. Active learning using pre-clustering. In Proceedings of the twenty-first international conference on Machine learning, page 79. ACM, 2004.
Ashesh Rambachan and Jonathan Roth. Bias in, bias out? evaluating the folk wisdom. arXiv preprint arXiv:1909.08512 2019.
Thomas S Robinson, Niek Tax, Richard Mudd, and Ido Guy. Active learning with biased non non-response to label requests. Data Mining and Knowledge Discovery, pages 1-24, 2024.
Neil Savage. How ai is improving cancer diagnostics. Nature, 579(7800) 2020.
Greg Schohn and David Cohn. Less is more: Active learning with support vector machines. In ICML, volume 2, page 6. Citeseer, 2000.
Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. arXiv preprint arXiv: 1:Xiv:170880 700.00048, 2017.
Burr Settles. Active learning literature survey. 2009.
Jiankun Sun, Dennis Zhang, Haoyuan Hu, and Jan A Van Mieghem. Predicting human discre- tion to adjust algorithmic prescription: A large-scale field experiment in warchouse operations. Management Science, 68(2):846-863 2022.
liris Sundin, Peter Schulam, Eero Siivola, Aki Vehtari, Suchi Saria, and Samuel Kaski. Active learning for decision decision-making from imbalanced observational data. In International conference on machine learning, pages 6046-6055. PMLR, 2019.
William R Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika, 25(3-4):285-294, 1933.
Xinru Wang, Zhuoran Lu, and Ming Yin. Will you accept the ai recommendation? predicting human behavior ai-assisted decision making. In Proceedings of the ACM Web Conference 2022, WWW '22, page I 1697-1708, New York. NY, USA, 2022. Association for Computing Machinery. ISBN 978 0781450390965. doi: 10.1145/445447.3552244 /tt/ttss::///o. 1044// 3512240
1145/344544
Dennis Wei. Decision-making under selective labels: Optimal finite-domain policies and beyond. In International Conference on Machine Learning, pages 11035-11046. PMLR, 2021.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms, arXiv preprint 17877:170..07747, 2017.
13

---

Songbai Yan, Kamalika Chaudhuri. and Tara Javidi. Active learning from noisy and abstention feedback. 2015 53rd Annual Allerton Conference on Communication, Control, and Computing (Allerton), pages 1352-1357. IEFE, 2015.
Mingzhang Yin, Claudia Shi, Yixin Wang, and David M Blei. Conformal sensitivity analysis for individual treatment effects. Journal of the American Statistical Association, 119(545):122-133) 2024.
Jiaming Zeng. Berk Ustun, and Cynthia Rudin. Interpretable classification models for recidivism prediction. Journal of the Royal Statistical Society Series A: Statistics in Society, 180(3))689-722, 2017.
Yunfeng Zhang, Q Vera Liao, and Rachel KE Bellamy. Effect of confidence and explanation on accuracy and trust calibration in ai-assisted decision making, In Proceedings of the 2020 conference on fairness, accountabillty, and transparency, pages 295- 305, 2020.
A Proofs
Theorem A.1 (c-BALD). Assume when a = O, the observation is a fixed non-in -nniinformatie value yo (e.g., NA). If the human decision maker has probability e(x) to label the instance, then
T(y: 0|. D) ((x:) 0|., D, 1).
Proof. Let e(x) = p(a = 1/2)).
11y; 011, D)
D log ply10,z, D)
p(yl x D)
=e(x)By,, 1 log p(y10, x. D. =
5yo (v) Sy (3)
p(ylr,D,a= D.a= + +(1-c(x)) c(x)) 0.0. log
=e(x)]l(y; 0|2 1)
The second equality is because of the law of total expectation.
Theorem A.2 (Joint-BALD), Under the condition of Theorem A.1
I(fa, v3: (0, o) D) (,, 1/ L).
Proof. Recall that L = (xi, as i= and D = {xi, 1/1 la
I(a, y; 0, L,
I(a; Φ/x, L, D) + Isy;; dla, C,
=I((;; 0, Øla, L) + T(y; 0 g1a, D)
-E log pla H. L.)
+ Eajz E By log ply x, a, D)
p(a
plylar a. D)
=II(a; øla, L) + e B - log 0,y0, a
plyle, a 1,
+ (1 - F( log 8y0 (v)
8,, (v)
=I(a; 0/.r L) + E, D log ply 0, x, 1,
=I(a; δ|æ, L) + e(x) 0|. D,
The first equality is by the chain rule of mutual information: the second equality is because the observation of all in L and all the observation of y is in D; the third equality is because the distribution of and are independent of ó and 0, respectively.
D
14

---

B Extensions on AI Advising/Changing Human Behavior
While we motivate the Active Learning with Instance Rejection problem in the cold-start setting in the main paper, the Active Learning with Instance Rejection can still happen when human behavior changes, thus we need to acquire new labels to better model the underlying function and the human discretion model. Here we offer an illustrative toy example of ALIR under changing human behaviors. In this toy example, we assume the company starts to use a machine learning model to advise human decision makers, thus leading to human discretion behavior change.
We use synthetic example illustrate the AI advising setting We generate a synthetic two-moon dataset with 2 features. The response variable is binary and can be considered as whether an insurance fraudulent. We generate 20000 data samples and downsample 90% of the data points with zo I where is the first dimension of the input feature. This simulates real-world scenario where there are a lot of valid insurance claims in the candidate pool where active learning necessary to reduce the potential waste of human workers' time. The resulting dataset has around 3700 samples as the training set and around 1600 samples as the test set. The visualization of the synthetic data is shown in Figure ((a)
Initial Human Behavior Model: For the initial human discretion model eg. we assume human decision makers will only investigate claims with 3 < xo < 6 and follow the acceptance probabilities below:
0.6
3 S æ0 S 6 and æ > 3
hz)x)=
0.4
3 < 30 6 and æ1 S
88)
otherwise.
This simulates the selective labels setup where initially human decision makers will only investigate claims of a certain subpopulation, and the machine learning model fitted on this subpopulation may not generalize well to the whole population even with ample data. Here the human decision makers have a higher probability to investigate fraud cases.
Updated Human Behavior Model:
We the initial human behavior collect 200 samples with instance rejection, then we train a Bayesian neural network fo()) on the initial labeled dataset 'D and deploy it to assist human decision makers. After the machine learning model is deployed, we assume the human decision makers will have a new behavior model e(x) that follows the acceptance probabilities below:
0.6p + (1 ())((())
3 < xp S 6 and > 3
0.2p (1 f)f((z))
vn S and 5ι 3
h(x)
Op+ (1 - p))((=)
Zo S 3
otherwise.
Here p is an algorithmic-aversion hyperparameter that controls the level of trust in the machine learning model. When - 0, the human decision makers will fully trust the machine learning model. When P = 1, the human decision makers will ignore the machine learning model and follow the initial human behavior model. We set p 0.5 to simulate partial adoption of the machine learning model. However, for the high-risk region xo >> 6, the human decision makers will never investigate no matter what the machine learning model advises.
Since the updated human behavior model may have higher label probabilities for the cases at certain regions (c.g., Zo 3), we can use different active learning strategies to acquire samples to improve the machine learning model's generalizability. We test Naive-BALD, RANDOM, and Joint- BALD-UCB on this synthetic dataset for illustration. We set the query size as 20 and the number of acquisition steps as 10. For batch acquisition with query size K, we use the greedy strategy to select the K samples with the largest acquisition function value.
We can see that Joint-BAL.--UG selects the samples that are likely to be labeled by the human decision makers and are both informative to the machine learning model and the human discretion model update. In contrast, Naive-BALD selects the samples that are informative to the machine learning model but may
First, we visualize the labeled data after 3 and steps of active learning in Figure 5
15

---

(a) Synthetic Data
(b) Random Step 3
(c) Naive-BALD Step 3
(d) UCB
Step 3
(e) Random Step 9 (0) Naive-BALD Step
(g) UCB Step 9
Figure 5: Synthetie Data. Figure (aa) shows the underlying synthetic data distribution. Figure (b)-(d) and (e)-(g) shows the labeled data after 3 and 9 steps of active learning. Human decision makers always reject to label when 2g > 6. BALIR selects the samples that are likely to be labeled by the human decision makers and are both informative to the machine learning model and the human discretion model update. In contrast, BALD selects the samples that are informative to the machine learning model but may not be labeled by the human decision makers. RANDOM selects the samples randomly and since many candidate applications are in the high-risk region, also has low probability to select the samples that will be labeled by the human decision makers.
not be labeled by the human decision makers. this case, Bhive-BALD got stuck in the high-risk region zo > 6 where human decision makers will always reject the loan applications, thus leading to no improvement for the machine learning model. RANDOM selects the samples randomly and since in the high-risk region, also has low probability of selecting the
many candidate applications samples that will be labeled by the human decision makers. Joint-BALD- the correct decision boundary. This example aims to demonstrate that the ALIR problem still exist when human behavior change and the proposed methods in the paper can also be applied in such settings.
successfully recovers
C Homogeneous Human Behavior
In this section, we show how different active learning methods perform under homogenous human behaviors. As demonstrated in the main paper, when e(x) c, < Naive-BALD corresponds to the correct information gain and we expect all the proposed methods to perform similarly.
We the experimental setup in Section 5.3 with the MNIST database and change the human decision behavior as e(x) = 0.8. The model accuracy and number of samples labeled are reported in Figure 6(a) and Figure 6(b) respectively. Naive-BALD and e-BALD also have similar model performance compared to other SEL-BALD methods we proposed in the paper.
D Human Behavior Model in Case Study
The human discretion model used in the case study is
0.3 Debt > QDDebt 0.7) and Income < (Income, 0.7)
(xx)
0.9 Debt (Deebt 0.7) and Income (Income, 0.7)
0 Debt < (Deebt, 0.7).
where Q(:,a) ) is the empirical 100 x a quantile of the metric. This simulates the selective labels setup where human decision makers are skeptical about insured customers who have high debt.
16

---

0.02
0.79
Aumbers www.es Santoi
140
0.78
0.11
0.70
88
Thmu Flaele
(a) Accuracy
(b) Number of Sample
Figure 6: Results for MNIST with Homogeneous Human Behavior. We report accuracy and number of samples labeled. All the proposed methods have similar model performance with homogeneous human discretion behavior across different budgets.
E Experiments with More Datasets
We compare the proposed methods with baselines Fashion MNIST [Xiao al. 2017]. CIFAR-10 [Krihhvskyy] [009]. Adult [Becker and Kohavi 1996) and Mushroom [mus]] 1981 datasets in this section. For Fashion MNIST and CIFAR-IO, we use the same human behavior model the one used in MNIST. For Mushroom, we assume humans have the following probability to label the instances: 0 when 1 < 0; 0.3 when #1 > 0 and x4 O; 0.9 otherwise. For Adult, assume humans have the following probability to label the instances: O when zo Qz.5(zo); 0.3 when zy S Q0.5(zo) and I1 (2)) 0.9 otherwise.
The results for the accuracy and the number of samples labeled are shown in Figure 7 and Figure g respectively. We find similar qualitative conelusions that Joint-BALD variants often produce better results than Naive-BALD and RANDOM.
Sea Cods
Colt
Com
Colt
(a) Fashion MNIST
(b) CIFAR-10
(c) Adult
(d) Mushroom
Figure 7: Experimental Results (Aceuracy) for Additional Datasets. We report the accuracy for
each cost on different additional datasets. Joint-BALD variants often produce better results than Naive- BALD and RANDOM across different costs. Fashion-MNIST and CIFAR-10 use the same setting as MNIST in the main paper.
F Experiments with ML-based Human Behavior Model
In addition to the piecewise linear human behavior model we used, we compare with a Logistic Regression based human behavior model. We randomly sample 3 positive and negative samples from the datasets to train a Logistic Regression model and use it to simulate human responses. Humans have a higher probability to label the instances if the predicted probabilities from the Logistic Regression are larger and humans will reject to label when the predicted probabilities are low.
The results with Adult dataset is shown in Figure (a)) and Figure 9(b) and results with Mushroom dataset is shown in Figure TO(a) Figure T0(b) The proposed methods can significantly outperform baselines like Haive-BALD We also note RANDOM sometimes is strong baseline (c.g., in the
17

---

(a) Fashion MNIST
(b) CIFAR-10
(c) Adult
(d) Mushroom
Figure 8: Experimental Results (Number of Samples) for Additional Datasets. We report the number of samples labeled for each cost on different additional datasets. Naive-BALD and RANDOM label much less samples than the Joint-BALD variants and e-BALD often labels the most labels.
(a) Accuracy
(b) Num
(a) Accuracy
(b) Num
Figure 9: ML Human Behavior Model (Adult) Figure 10: ML Human Behavior (Mushroom)
Mushroom dataset) in the batch active learning setting documented in the previous work Kirsch
et al. 2019
G Experiments with Other Uncertainty Metric
While mainly focus improving the BALD objective in this paper, our methods can potentially be generalized by replacing the mutual information with other metrics in the objective. For example, replace the information gain with other uncertainty measures such entropy. A similar Joint-Entropy-UCH objective can be written as
Ent( v) (0, ( Ent 0|. D) + Ent(l, $/v; L)
As shown in Figure we implement Joint-Entron loint-Entropy-UCB with entropy measure on the synthetic data. Coint-Entropy-UCH UCB can also improve the traditional entropy objective in this case.
(a) Synthetic Data
(b) Initial Labels
Entropy
(d) Coint-Entropy-UUe
Figure 11: Extension other uncertainty-bant metric: Qualitative Results for Synthetic Data after spending a budget of 450. Our method can also help improve the performance of other uncertainty- based active learning metric like Entropy.
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We study the active learning with instance rejection problem and propose several methods address the problem. Unlike traditional active learning, the benefit of each method is more nuanced and depend on the underlying human behavior model. We show Joint-BALD- BALD-UCB and Joint-BALD- BALD-TS is more robust for unknown and heterogeneous human discretion models.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. It is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss limitations and future work in the last section.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
19

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification:
We provide proofs in Appendix A with full assumptions.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide the experimental setting and justification of our design choice. We will also release our code.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
20

---

reproducible, they 5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification:
Our code will be made public upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
public/guides
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
Codeeobmiiiieeee SanSSubbissss sorbbbisssiooi ssissnoicccyy for more details. 7/nips cc/ wtc/publicc/ /ttp://guudeeeeeeeuubiaiiiii
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide these details in the Experiments section.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We provide confidence intervals in our figures. For our synthetic data, it is from single run demonstrate the failure case of different methods, not applicable.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
21

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out of range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We discuss computing recourse in the experiments section.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: Our paper has no violation to the code of ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss potential societal impacts in the last section.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
22

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Our paper uses public dataset.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
23

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such the institution conducting the review.
24


Reviews:
[{"review_id": "kOqISweqXq", "reviewer": "Reviewer_WcLX", "summary": "This paper considers an active learning scenario where the human annotation is done under the restriction that the annotator is biased in deciding the labeling. The Bayesian framework makes the e-BALD and various versions based on a posterior sample of labeling probabilities. The authors consider the three cases in human labeling probability, including the non-labeling of specific cases.  Three versions of the mean, quantile (UCB), and samples reveal different aspects in experiments. The trade-off between exploration and exploitation in labeling probability is observed. Using the quantile can have a detailed balance between examination and labeling (examination means checking the attribute and non-labeling).", "questions": "Q1: Can you provide other metrics, such as the F1 score? In my opinion, the 1-score can be lowered by selective labeling.  \nQ2: What’s the meaning of Total costs in the legend of Fig. 4?", "limitations": "Sufficiently discussed in the last section.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The problem and task are novel. The BADL modified by the labeling probability is simple, clear, and well-defined. In experiments, the proposed algorithm performs better in the labeling rate for the candidates by the query.", "weaknesses": "There can be many issues. \nThe first one is the applicability of various metrics used in active learning.  Usually, entropy, margin, and variation ratio are traditional measures. Especially, the BALD combined with deep neural networks can show poor performance. It requires the study of applicability to other metrics or advanced active learning algorithms such as BADGE.  \nThe second one concerns the datasets used in experiments. Three datasets are not sufficient; more datasets, such as Fashion MNIT or ImageNet-tiny, can be considered, and their statistical significance can be studied more thoroughly. \nThe last issue is human behavior, the scenario is relatively simple addressed by the simple logistic regression. The behavior of human annotators can be diverse because of their complex properties. It requires human models, which can be complex. When the human annotator's behaviors are complex, what’s the performance? Also, models such as logistic regression can be mimicked to generate samples.       \n\nMinor: \nbayesian neural  -> Bayesian neural\ne-bald -> e-BALD", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your recognition of the novelty of the problem and the clarity of our approach.  \n\n**Other AL Metrics**: Thanks for the suggestion! While we mainly focus on the BALD objective in this paper, our methods can be generalized by replacing the mutual information with other metrics in the objective. For example, we can replace the information gain with other uncertainty measures such as entropy. \n\nAs shown in Figure 1 in the response PDF, we implement Joint-Entropy-UCB with entropy measure on the synthetic data. Joint-Entropy-UCB can also improve the traditional entropy objective in this case. The BADGE method measures uncertainty at the gradient space. One potential way to apply BADGE is to compute the gradients for the last layer of the predictive model $p(y|x,\\theta)$ and discretion model $p(a | x, \\phi)$, weighted combine the gradients with the weights $(e(x), 1)$, and then use K-means++, however, this adaptation may be more challenging compared to traditional AL metrics and we believe will be interesting future work. We will discuss the connection with BADGE in the revised version. \n\n**Datasets and Human Behavior**: Thanks for the suggestion. We added additional experimental results on Fashion MNIST, CIFAR-10, and two UCI datasets (Adult and Mushroom). The results are included in Figures 2 and 3 in the PDF response. We find similar qualitative conclusions that Joint-BALD variants often produce better results than Naive-BALD and RANDOM across different costs. \n\nFollowing your suggestion, we also implemented a Logistic Regression based human behavior model in Figure 5 in the PDF response with similar qualitative conclusions as in the main paper. \n\nPlease find the results for additional datasets, human behaviors, and more details in **Response to All**. \n\n**F1 Score:** In our experiments, we find F1 score often has a similar qualitative conclusion as the Accuracy. To illustrate that, we include a comparison between Accuracy and F1 score in Figure 4 in the PDF response for your reference. We will add the discussion about the F1 score in the paper. \n\n**Total Cost**: In the ALIR problem, we assume there is a cost to decide whether to label an instance (examine cost) and a cost to actually label the instance (label cost) (L130-131 in the main paper). The total cost is the sum of the current examination cost and label cost (the cost structure is defined in L212-213 in the paper). We will make this more clear in the revised manuscript.\n\nWe appreciate your insightful suggestions and have addressed it thoroughly in our revised manuscript to enhance the overall quality of the work. Please do let us know if you have any further concerns, or if this adequately addresses all the issues that you raised with the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NTTBXmDWcN", "reviewer": "Reviewer_fApQ", "summary": "The paper introduces the Active Learning with Instance Rejection (ALIR) problem, addressing the issue of selective labeling where human discretion impacts the labeling process. In particular, humans might not always provide a label for a point returned by active learning; this abstention needs to be modeled explicitly. The authors propose new algorithms under the SEL-BALD framework to improve model performance using a human discretion model. The paper demonstrates the effectiveness of these algorithms through experiments on both synthetic and real-world datasets.", "questions": "- Can you expand the techniques used to train the human discretion model?\n- How can potential biases introduced by assumptions on the human discretion model be addressed?\n- Is $c_e$ part of $c_l$?", "limitations": "See weaknesses above.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The paper's focus on Active Learning with Instance Rejection (ALIR) focusses on a real-world problem where human discretion plays a critical role in the labeling process. This assumption seems to be realistic and well-motivated.\n- The introduction of the SEL-BALD (Selective Bayesian Active Learning by Disagreement) framework effectively balances the dual objectives of selecting informative instances and accommodating human labeling behavior.\n- The empirical results show the benefits of considering human discretion in active learning. This underscores the practical advantages of the proposed method(s).\n- The paper is well-structured and clearly written.", "weaknesses": "- The integration of Bayesian neural networks for modeling both machine learning tasks and human judgment significantly increases computational complexity. These Bayesian models are computationally demanding to train.\n\n- While the paper includes experiments with both synthetic and real-world datasets, the scope of real-world applications examined is relatively limited. Critical domains such as medical diagnostics or financial fraud detection present unique challenges. A more comprehensive evaluation across diverse applications would better illustrate the versatility and reliability of the approach.\n\n- The paper's assumption that human discretion can be accurately modeled using predictive tools like Bayesian neural networks may be overly optimistic. Real-world human decision-making is influenced by numerous complex and often unpredictable factors. The efficacy of these models can vary significantly across different contexts. Accurate modeling of human discretion typically requires extensive, high-quality historical data on human decisions, which may not always be available. In cases where data is limited, noisy, or biased, the human discretion model's performance could be significantly compromised. Furthermore, human decision-making processes can change over time due to regulatory updates, new information, or shifts in organizational policies. The study does not address how the human discretion model would adapt to such changes, potentially leading to outdated or inaccurate predictions.\n\n- The legibility of figure legends is consistently poor due to their small size. This aspect requires significant improvement for better readability and comprehension.\n\n- Definitions 4.1 through 4.5 would benefit from more detailed explanations accompanying the equations, providing context and clarification for the mathematical formulations presented", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your positive comments on our problem, method, and writing. Nevertheless, we understand there are concerns impacting the evaluation.\n\nTo address these:\n\n- **Computational Complexity:** In this paper, we mainly focus on improving the BALD method, therefore the baseline models are also Bayesian models that share the same computational complexity. In addition, we use the MC dropout as a Bayesian Approximation [Gal & Ghahramani 2016] which is not computationally demanding compared to a deterministic model with the same architecture. For other traditional AL metrics that work for simpler models such as Entropy, it is also possible to extend the proposed method for these metrics by replacing the information gain term with the uncertainty term. We demonstrate that in Figure 1 in the response PDF, we implement Joint-Entropy-UCB with entropy measure on the synthetic data. Joint-Entropy-UCB can also improve the traditional entropy objective in this case. We will add the discussion in the revised manuscript. \n\n- **Modeling Human Behaviors is Challenging:** We agree that human discretion models may be hard to accurately predict, which is the main reason we proposed a UCB and TS variant to better explore the unknown human behaviors. In addition, we can use a non-parametric learner such as Gaussian Process (GP) or BNN (by universal approximation theorem) to learn the human discretion behavior that guarantees the true model can be learned with enough samples. \nFurthermore, recent studies and real-world applications have demonstrated strong predictive performance of human discretion behaviors [1,2] or human mental models [2,3], which indicates the human discretion model can be learned in practice, which we hope can address your concern. \n\n- **Changing Human Discretion Model:** We agree that human decision-making processes can change over time. We would like to kindly highlight that we discuss and evaluate this scenario in Appendix B where the true human behavior model changes at a given time step in the instance collection. Our method can still be applied in this setting and we find the Joint-BALD-UCB successfully selects the samples that are likely to be labeled and are informative to the predictive model and human discretion model, and it can better recover the correct decision boundary than baselines.\n\n- **Real-world application and Data sets:** Thanks for the suggestion. First, we kindly note that one of our experiments is motivated by a financial fraud investigation. Second, we added additional experimental results on Fashion MNIST, CIFAR-10, and two UCI datasets (Adult and Mushroom) in the rebuttal. The results are included in Figures 2 and 3 in the one-page PDF response. We find similar qualitative conclusions that Joint-BALD variants often produce better results than Naive-BALD and RANDOM across different costs. Though selective labeling is prevalent in industry use cases, it is challenging for us to find public real-world data with recorded human decision-maker judgments, so we choose to vary the human behavior model in the paper so that we can get a more detailed comparison across different methods.\n\nPlease find the results for additional datasets and human behaviors in **Response to All**. \n\n- **Figures and Writings:** In the revised paper, we ensure all the figure legends are clear with proper fonts (like the figures in our PDF response). We also add intuitive explanations to the objective terms in the Definitions. We add implementation details of the training discretion model in the Appendix. \n\n- **Assumptions Biases:** We impose minimum assumption on the human discretion model since $e(x)$ can be any complex function. In practice, we can use a non-parametric learner such as Gaussian Process (GP) or BNN (by universal approximation theorem) to learn the human discretion behavior that guarantees the true model can be learned with enough samples. \n\n- **$c_e$ and $c_l$:** In the ALIR problem, we assume there is a cost to decide whether to label an instance (examine cost $c_e$) and a cost to actually label the instance (label cost $c_l$) (L130-131 in the main paper). The total cost is the sum of the current examination cost and label cost. We will make this more clear in the revised manuscript. \n\nWe appreciate your insightful suggestions and have addressed it thoroughly in our revised manuscript to enhance the overall quality of the work. Please do let us know if you have any further concerns, or if this adequately addresses all the issues that you raised with the paper.\n\n[1] Sun, Jiankun, et al. \"Predicting human discretion to adjust algorithmic prescription: A large-scale field experiment in warehouse operations.\" Management Science 68.2 (2022): 846-865.\n\n[2] Wang, Xinru, Zhuoran Lu, and Ming Yin. \"Will you accept the ai recommendation? predicting human behavior in ai-assisted decision making.\" Proceedings of the ACM web conference 2022. \n\n[3] Bansal, Gagan, et al. \"Beyond accuracy: The role of mental models in human-AI team performance.\" Proceedings of the AAAI conference on human computation and crowdsourcing. Vol. 7. 2019.\n\n[4] Madras, David, Toni Pitassi, and Richard Zemel. \"Predict responsibly: improving fairness and accuracy by learning to defer.\" Advances in neural information processing systems 31 (2018).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QhoQAItzQS", "reviewer": "Reviewer_KKqr", "summary": "This paper presents a direct extension of BALD sampling criterion to address the need for active annotation rejection. The method assumes a rejection distribution over the candidate data samples in pool-based active learning and proposes an active sampling strategy that jointly considers the BALD informativeness and rejection cost.", "questions": "-Would it be possible to include a comparison of the model's rejection predictions against typical true rejection distributions in your experiments?- Should the first term in eq 4 be negative?\n-The constant c in section 4.2 seems to differ from the c defined in section 3.1. You might want to use a different notation.", "limitations": "NA", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.The problem studied in this paper can be considered an instance of cost-aware active learning. The challenge of having samples that are difficult or even impossible to label is a very practical issue in active learning. This paper provides a valuable contribution to the field of active learning by explicitly estimating the rejection function, which offers better guidance for active learning and is beneficial for improving data efficiency.\n\n2.The rejection component proposed in this paper is naturally integrated into Bayesian sampling. The final hybrid sampling criterion revolves closely around information gain and uncertainty reduction, which I find to be reasonable and effective. Approaching active sampling from a Bayesian perspective also makes it more interpretable.\n\n3.The paper is written in a concise and fluent manner, with high readability. The core method is introduced clearly and is easy to understand.", "weaknesses": "1.The core of this paper's design is the human discretion function e(x), but there are some unclear aspects regarding this function in my view. First, the approximation of e(x) seems to be achieved through an additional Bayesian network, but the detailed features of this Bayesian network are omitted in the paper. As the authors mention, given the small sample nature of active learning, the estimation of e(x) is likely to have an underfitting problem. However, the two solutions proposed by the authors in line 171 do not seem to address this issue adequately, and the authors should provide more explanation on this. Additionally, since e(x) is estimated by a Bayesian network, it should be involved in the computation in the form of some posterior distribution. However, in the examples given in the experimental results section, the true e(x) is presented as an unnormalized piecewise function. Therefore, using a posterior to approximate an e(x) that is not a valid density function raises questions about its effectiveness and limitations. I believe this is directly related to the deeper reason for label rejection, i.e., the analytical form of e(x). If the authors cannot theoretically prove that the proposed method generally works for all types of rejection reasons, they should at least compare the model's prediction of rejection (instead of the final overall sampling score) with several typical true rejection distributions in the experiments.\n2.As a cost-aware active learning method, I find the experimental section of this paper somewhat weak. The real-world dataset used does not record the true rejection behavior of human annotators, which means, to some extent, that the experiments on MNIST are still synthetic. The true reasons for human annotators rejecting to label might be much more complicated than the prototype e(x) used in this paper. Therefore, I believe the evaluation of this specific problem requires a dataset with actual recorded human annotator rejections.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are delighted to learn your acknowledgment of our Bayesian approach to a practical problem in active learning.  We address your points of concern in detail below.\n\n- **W1 (Estimating $e(x)$):**  (i) We now add more implementation details of the Bayesian network $e_\\phi(x)$ in the appendix.  In the simulation, we use the MC dropout as a Bayesian Approximation [Gal & Ghahramani 2016] (we set the number of MC samples as 40 in all experiments). The model architecture is a 3-layer fully connected neural network with LeakyReLU activation function..  (ii) underfitting problem: in L171, we propose to query data points that are informative to the human discretion model in the second point, which aims to address the underfitting issue. More specifically, in Def 4.3-4.5, we include an additional term that quantifies the information gain on the human discretion model to help learn $e(x)$ better. In addition, we propose the UCB and TS variants to guide our label acquisition using the underfitted $e(x)$. As shown in Figure 3, the underfitted $e(x)$ will indeed harm methods like $e$-BALD but our proposed method can get a better performance. (iii) We agree with the reviewer that the true $e(x)$ is unnormalized, i.e. $\\int e(x) dx \\neq 1$. The discretion model $e_\\phi(x) = p(a=1 | x, \\phi)$ is a function mapping from the domain $\\mathcal{X}$ of $x$ to any value in $[0,1]$, which also does not require normalization over $x$. The Bayesian computation yields the posterior $p(\\phi | \\mathcal{D})$ of the parameters $\\phi$ instead of $e(x)$. So our $e_\\phi(x)$ can be applied to approximate any function $\\mathcal{X} \\to [0,1]$.  \n\n- **Datasets and Human Behavior Models:** Thanks for the suggestion. We added additional experimental results on Fashion MNIST, CIFAR-10, and two UCI datasets (Adult and Mushroom). The results are included in Figures 2 and 3 in the one-page PDF response. We find similar qualitative conclusions that Joint-BALD variants often produce better results than Naive-BALD and RANDOM across different costs. Though selective labeling is prevalent in industry use cases, we hope you can understand that it is challenging for us to find public real-world data with recorded human decision-maker judgments on such applications, so we choose to vary the human behavior model in the paper so that we can get a more detailed comparison and insights between different methods. \n\nPlease find the results for additional datasets and human behaviors in **Response to All** and the attached PDF.\n\n- **True/Predicted Human Behavior:** We kindly note that Figure 3 in the main paper demonstrates the comparison of $e(x)$ and the estimated $\\hat{e}_\\phi(x)$ for the synthetic data. We will address the typos and notations in the revised version.\n\nWe appreciate your insightful suggestions and have addressed it thoroughly in our revised manuscript to enhance the overall quality of the work. Please do let us know if you have any further concerns, or if this adequately addresses all the issues that you raised with the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lCoCi9X5dt", "reviewer": "Reviewer_dpcV", "summary": "This paper explores the setting where a learner has a label budget that can be used to acquire labeled data from a user. However, differently from traditional active learning approaches, the paper assumes that the user may not want/be able to label some examples. That is, for specific (limited) examples, the user returns a label when queried, while for other examples it does not return anything. The key challenge of the paper is how to motify traditional query strategies to account for this new possibility of not getting any label from the user. The paper solves this challenge by modeling the user through a Bayesian network and including two additional terms in the AL score function: the first term scales the potential gain obtained by labeling an instance by the probability that the instance would get labeled when queried, while the second represents the gain obtained for modeling the user (i.e., where the user labels the data).", "questions": "No questions.", "limitations": "Limitations are clearly discussed in Section 6.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "### Main Strengths\n\n1. The problem setting is realistic, unexplored, and yet relevant for industry use cases: after all, some experts may not want, may not be allowed, or even may not be able to provide labels.\n\n2. The paper is clearly written: claims are well supported, the notation is well detailed, and the flow is easy as well as the structure.\n\n3. The proposed approach is reasonable and meaningful: the paper proposes multiple methods (such that one is the generalization of the other) and shows empirically how they perform. The Bayesian Active Learning with Disagreement approach is still quite used - despite being more than 10 years old. Thus, extending BALD to the new setting can have a positive impact on several industrial use cases.\n\n4. The paper is technically sound and highlights whenever a quantity is an estimate and when is the true value.", "weaknesses": "### Main Weaknesses:\n\n1. The experiments include MNIST, which is a little bit outdated and nowadays considered a simple dataset. I'd recommend using more elaborated multiclass datasets (e.g., SVHN, Waterbirds, ...). Using more common datasets would strengthen the work.\n\n2. Some literature study might be missing: learning to defer is an area where a model defers the prediction to a human whenever the human is more likely to provide a correct prediction, whereas the model is preferred in other cases. Thus, this research direction also investigates how to model a human. Also, learning to reject is a similar area, and [1] uses active learning assuming that the user may not provide some labels. Finally, there is a connection with PU Learning (Positive and Unlabeled Learning), where the propensity score named e(x) represents the probability of labeling an instance. Including a brief connection to these areas would improve the quality of the paper.\n\n[1]: https://www.ijcai.org/proceedings/2020/0403.pdf", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We're encouraged by the reviewer's positive comments on the novelty of the problem and our method contribution. Below, we address the concerns raised in your review.\n\n- **Datasets:** Thanks for the suggestion. We followed your recommendation and added additional experimental results on Fashion MNIST, CIFAR-10, and two UCI datasets (Adult and Mushroom). The results are included in Figures 2 and 3 in the one-page PDF response. We find similar qualitative conclusions that Joint-BALD variants often produce better results than Naive-BALD and RANDOM across different costs.  Please refer to **Response to All** and the attached PDF for details.\n\n - **References:** Thanks for the great references! Indeed, learning to reject and learning to defer (adaptive learning to reject) are closely related to our work. These works consider how to select AI or the human at testing time to decide on an instance while our paper focuses on how to design the active learning strategy with human discretion behaviors. [Perini et al. 2020] studies how to estimate the class prior with a given active learning strategy in the PU learning setup. PU learning also considers a partially observed label setting similar to ours. We will discuss the relationships with these literature in detail in the revised manuscript.\n\nWe appreciate your insightful suggestions and have addressed it thoroughly in our revised manuscript to enhance the overall quality of the work. Please do let us know if you have any further concerns, or if this adequately addresses all the issues that you raised with the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# Response to All:

We thank the reviewers for your positive and constructive feedback. Your comments signiﬁcantly help us improve the paper.

## Datasets

In addition to the synthetic data, MNIST, and the loan fraud detection data in the paper, we further evaluate the proposed and the baseline methods on Fashion MNIST, CIFAR-10, and two UCI datasets (Adult and Mushroom). The results are included in Figures 2 and 3 in the one-page PDF response. We find similar qualitative conclusions that Joint-BALD variants often produce better results than Naive-BALD and RANDOM across different costs. 
Please refer to the one-page PDF response for details. 

## Human Behavior

We agree with the reviewers that a human annotator’s behavior can be complex. We would like to note that the proposed Joint-BALD does not pose functional restrictions on the human discretion model $e_\phi(x)$, which can be an arbitrarily complex function. For example, it can be modeled by a nonparametric learner like Gaussian Process or Bayesian neural networks. 

Though selective labeling is prevalent in industry use cases, it is challenging for us to find public real-world data with recorded human decision-maker judgments, so we choose to vary the human behavior model in the paper so that we can get a more detailed comparison across different methods. In the paper, we examined homogeneous (Appendix C) and more complex heterogeneous human behaviors and found the advantages of Joint-BALD and its variants for unknown behavior heterogeneity. We also illustrate the advantage of Joint-BALD in adapting to changing human behavior (Appendix B). In addition, following reviewer WcLXs’ suggestion, we add an experiment using the Logistic Regression modeled human behavior model in Figure 5 in the one-page PDF response. More specifically, we randomly select 3 samples with the positive and negative class each and train a logistic regression to mimic the human decision behavior, then human labeling decisions are generated using the logistic regression. We find a similar conclusion that Joint-BALD variants often produce better results than Naive-BALD and RANDOM across different costs as demonstrated in the main paper.

Meta-review: After discussion with the reviewers and a robust rebuttal from the authors (including additional results based on feedback), I recommend accepting the submission (poster).

The main concerns were around limited (1) a limited number of datasets (the authors have added 3 more), (2) simplistic human discretion models (the authors have added an ml-based human model), (3) lack of experiments with real human data (I tend to agree this is beyond the scope of typical academic settings).

I expect the authors to include the new results in the final version of the paper and expand on what could not be fit into the rebuttal pdf (e.g., include ML-based human model results for all datasets, include full set of experiments with uncertainty based sampling AL methods). Also, move some of the results and discussion of the dynamic/changing rater setting to the main paper.
Decision: Accept (poster)