SEL-BALD: Deep Bayesian Active Learning for Selective Labeling with Instance Rejection

Authors: ruijiang  gao
mingzhang  yin
maytal  saar-tsechansky information
operations  risk
  management
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 23:26:55

Abstract: Machine learning systems are widely used in many high-stakes contexts in which experimental designs for assigning treatments are infeasible. When evaluating a decision instance is costly, such as investigating a fraud case, or evaluating a biopsy decision, a sample-efficient strategy is needed. However, while existing active learning methods assume humans will always label the instances selected by the machine learning model, in many critical applications, humans may decline to label  instances selected by the machine learning model due to reasons such as regulation constraint, domain knowledge, or algorithmic aversion, thus not sample efficient. 
In this paper, we propose the Active Learning with Instance Rejection (ALIR) problem, which is a new active learning problem that considers the human discretion behavior for high-stakes decision making problems. We propose new active learning algorithms under deep Bayesian active learning for selective labeling (SEL-BALD) to address the ALIR problem. Our algorithms consider how to acquire information for both the machine learning model and the human discretion model. We conduct experiments on both synthetic and real-world datasets to demonstrate the effectiveness of our proposed algorithms.

Main paper: 
 introduction: 
 Machine learning is increasingly deployed in high-risk applications, including medical diagnosis, fraud detection, and criminal justice [Zeng et al., 2017 , Cecchini et al., 2010 , Savage, 2020] . Given enough labeled data and the same set of feature input, a machine learning model may achieve better performance than human experts [Abramoff et al., 2023] . Therefore, there have been growing efforts to develop machine learning (ML) models to assist humans in improving high-risk decisions.While modern deep learning algorithms' strong performance across applications makes them particularly promising for supporting experts, they are often data-hungry and require a large amount of labeled data to perform well. Yet, in many impactful high-risk applications acquiring a large number of labeled data is highly costly or can be prohibitive, such as when labeling necessitates costly human experts' time and effort. For instance, labeling insurance claims requires human experts' resources to establish if an insurance claim is fraudulent; similarly, determining medical diagnoses may require diagnostic procedures that may be time-consuming and possibly risky for patients. Active learning methods can be a promising path to address this problem, given they aim to select especially informative instances for labeling to minimize the number of labeled instances necessary to achieve desirable performance [Cohn et al., 1996] .To do so, most active learning methods operate iteratively, where an initial machine learning model is first trained on a small labeled data set and the model then informs the selection of particularly informative instances to be labeled. A new model is then retrained on the augmented labeled data and this process is repeated until a desirable goal is achieved.However, active learning methods assume instances selected for labeling by the algorithm will always be labeled [Settles, 2009] . However, in many real-world high-risk contexts, such as acquiring diagnoses through costly invasive diagnostic procedures or obtaining the conclusions of insurance fraud investigations, experimental design is infeasible and it is not possible to automatically acquire any label the active learning algorithm deems informative. Rather, requests for label acquisition must be considered and approved by a human expert. Indeed, some diagnostic biopsies may never be approved by the human doctor because they are inconsistent with medical practices [Dai and Singh, 2021] ; if the requests are declined, the biopsy outcomes will not be revealed to benefit the machine learning model. Similarly, an insurance fraud agent may decline a request for an investigation because of the remote location or because of a prior belief that the case is unlikely to be fraudulent. Any given human discretion behavior over requests to acquire labels at a cost is unknown and can be influenced by various factors. This is known as the selective labels problem [Lakkaraju et al., 2017 , Kleinberg et al., 2018 , De-Arteaga et al., 2018 , Wei, 2021] , in which the outcomes are more likely to be available for a subset of instances. When a human expert must approve the decision to obtain a label, selective labeling would result from the expert's discretion over the labeling process.Independently of the labeling costs, human decision makers also need to invest time and effort in considering the algorithm's recommendation before making a decision. For example, the insurance agent needs to evaluate each case in order to decide whether an investigation should be launched and doctors will go through patients' medical record and assess the risk prior to approving a diagnostic procedure. Therefore, our goal is to maximize the machine learning model's predictive performance within a total budget considering both the examination and labeling cost.Ultimately, while traditional active learning focuses on identifying instances that would be most informative for learning, when the human must approve each labeling request, the human may be inundated with requests to consider acquisitions, any portion of which may be declined and thus not benefit learning, while incurring a significant loss of the human expert's resources. This poses a new challenge for active learning in which it would be valuable to bring to bear the human discretion of any labeling request so as to identify labeling requests that can achieve the most improvement in model performance for a given labeling and human discretion costs. We refer to this problem as Active Learning with Instance Rejection (ALIR), illustrated in Figure 1 (a). Crucially, in ALIR, the instances deemed most informative for the current ML classifier may not be the best candidates for labeling, as humans may choose to reject these instances (as shown in Figure 1(b) ). This challenge highlights the need for developing new active learning methods tailored to the ALIR framework.In this paper, we make the following contributions:• We propose a new Active Learning with Instance Rejection (ALIR) problem that considers the human discretion behavior in high-stakes selective labeling contexts. Unlike conventional active learning, where every instance is labeled by humans, ALIR allows humans to exercise discretion, selectively choosing whether or not to label certain instances.• We propose different active learning algorithms for ALIR and discuss their trade-offs. Unlike traditional active learning, the benefit of each method depends on the underlying human discretion behavior. We show theoretically that ALIR is equivalent to traditional active learning when the human discretion behavior is homogeneous across instances and propose new methods for unknown, heterogeneous discretion behaviors.• We conduct comprehensive experiments on both synthetic and real-world datasets to demonstrate the effectiveness of our proposed algorithms.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: We propose the active learning with instance rejection problem and propose several methods to address the problem. Unlike traditional active learning, the benefit of each method is more nuanced and depend on the underlying human behavior model. We show Joint-BALD-UCB and Joint-BALD-TS is more robust for unknown and heterogeneous human discretion models.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
related work: 
 Active Learning: To render costly data annotation more efficient, Active Learning has been studied extensively in the machine learning community [Lewis and Gale, 1994 , Schohn and Cohn, 2000 , Nguyen and Smeulders, 2004 , Freytag et al., 2014 , Gal et al., 2017 , Sener and Savarese, 2017 , Gong et al., 2019 , Gao and Saar-Tsechansky, 2020] . Traditionally, the key challenge addressed by most methods is to label the most informative unlabeled instances from which to induce the classifier. Existing methods assume humans will always label the instances selected by the active learning algorithm. We consider the selective labels problem where in many high-stakes decision making problems, humans may not label the selected instances due to their past experience, domain knowledge, or algorithmic aversion [Dai and Singh, 2021 , De-Arteaga et al., 2018 , Dietvorst et al., 2015] . Recent papers have studied active learning for treatment effect estimation from observational data [Jesson et al., 2021 , Deng et al., 2011 , Sundin et al., 2019] . Unlike these papers, we study the selective labels setup where the outcome is selectively observed, which is different from the treatment effect estimation problem where we can observe an outcome under the selected treatment arm.Selective Labels Lakkaraju et al. [2017] proposes the "contraction" technique to evaluate a machine learning classifier's performance in the selective labels problem. However, the point identification of the model performance is generally impossible without strong assumptions and partial identification is studied in the recent literature [De-Arteaga et al., 2018 , D'Amour, 2019 , Rambachan et al., 2022] . Unlike previous papers on studying the limitations in evaluation and optimizing predictive rules in the selective labels problem, we focus on how to acquire useful information to improve the machine learning model in the selective labels problem. Rambachan and Roth [2019] studies how historical human-generated data may bias the machine learning model in the selective labels problem and our paper studies how to work with the human decision makers to improve the machine learning model in selecting informative samples.There are works in the machine learning community designing different forms of human-AI collaborated decision-making systems such as learning to defer [Cortes et al., 2016 , Madras et al., 2018 , Gao et al., 2021 , Gao and Yin, 2023 , Gao et al., 2023] , or learning to advise [Bastani et al., 2021 , Grand-Clément and Pauphilet, 2024 , Bhatt et al., 2023 , Cao et al., 2024] . Unlike these work, we focus on how to collect information under human supervision. Our work is also motivated by the literature on overriding decisions and trust in algorithm adoptions [Sun et al., 2022 , Wang et al., 2022] . Dietvorst et al. [2015] finds that humans are likely to override the algorithm's decisions when they observe machine learning algorithms make mistakes. Bansal et al. [2021] studies how the quality of the explanation and the predictive quality of the machine learning model affect human decisions. There are also works that study different interventions to improve the trust of humans in the machine learning model [Dietvorst et al., 2018 , Lakkaraju and Bastani, 2020 , Zhang et al., 2020] . de Véricourt and Gurkan [2023] uses an analytical modeling framework and stylized human behavior model to show that humans may never know the true performance of AI in the selective labels problem. Our work is different from these papers as we focus on how to acquire useful information to improve the machine learning model in the selective labels problem.3 Problem Statement and BackgroundQuestion: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: We discuss limitations and future work in the last section.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
selective labels problem: 
 We have access to a pool of unlabeled data D pool = {x i } n i=1 , where each instance has a underlying label y i ∈ {1, • • • , K}. We denote the marginal distribution of X as P (X) and the conditional distribution of Y given X as P (Y |X). We assume that the unlabeled data is drawn i.i.d. from P (X). A human decision maker will make a yes decision (a i = 1) or no decision (a i = 0) for the decision subject. The corresponding outcome y i can only be observed when a i = 1.We assume there exists a human discretion function e(X) that maps the input X to a probability of being labeled by the human decision maker. We denote the probability of X being labeled as P (A = 1|X) = e(X). This corresponds to how humans will examine and label the instances. Then for an instance x i , if a i = 1, it will be labeled, returning the label y i , and be added to the current labeled dataset D = {x i , y i } i:ai=1 . Otherwise, the human labeler will reject it and no outcome will be available for the instance. No matter whether the instance is labeled or not, (x i , a i ) will be added to the current human decision dataset L = {x i , a i } n i=1 , which records all the human labeling information. A predictive model with likelihood P (y|x, θ) parametrized by θ ∼ P (Θ|D) can then be trained on the labeled dataset D.The requester now wants to acquire some additional labeled samples to improve the ML classifier. Since human labelers need time and resources to decide whether to investigate each instance (e.g., have a meeting to decide whether to operate on the organ, decide on whether to audit the firm) and label it (e.g., perform the biopsy, audit the firm), the requester wants to minimize the number of instances that need to be labeled. We denote the cost for examining the instance as c e and the cost for labeling the instance as c l . The total cost is the sum of the current examination cost and label cost. Therefore, with a total budget of B, the requester wants to select a subset of instances S ⊆ D pool to be labeled by the human labelers while maximizing the performance of the ML classifier P (y|x, θ).Given a model P (y|x, θ), pool dataset D pool , and labeled dataset D, active learning algorithms use an acquisition function A(x, θ) to select where to label next by arg max x∈Dpool A(x, θ). For example, A(x, θ) = unif() corresponds to the random acquisition function, which unif() returns a uniformly random number between 0 and 1; and A(x, θ) = H(y|x, D) = c P (y = c|x, D) log(P (y = c|x, D)) corresponds to the uncertainty sampling principle that picks the next sample with the largest predictive entropy. Our algorithm extends the following Bayesian Active Learning with Disagreement (BALD) algorithm [Houlsby et al., 2011 , Gal et al., 2017] .Bayesian Active Learning with Disagreement (BALD): BALD [Houlsby et al., 2011] uses the epistemic uncertainty as the aquisition function. The epistemic uncertainty measures the uncertainty about model parameters instead of the inherent uncertainty about the data that cannot be reduced by acquiring more samples. The information gain is defined as the mutual information between the label and the model parameters which can be written as follows:I(y, θ|x, D) = E y|x,D [H(θ|D) -H(θ|y, x, D)] = - c P (y = c|x, D) log(P (y = c|x, D)) + E θ∼P (θ|D) c P (y = c|x, θ) log(P (y = c|x, θ)) .(1)Here the posterior P (θ|D) can be approximated by the approximated posterior q(θ|D) from the Bayesian neural network, and using MC sampling [Gal et al., 2017] to approximate Equation (1) as≈ - c 1 T t pt c log 1 T t pt c + 1 T c,t pt c log(p t c ),where pt c is the probability of input x taken class c under the t-th MC sample from the approximated posterior q(θ). We denote this objective as Î(y, θ|x, D) ≈ I(y, θ|x, D).BALD will then select the instance by arg max xi∈Dpool Î(y, θ|x i , D). 
naively apply bayesian active learning with disagreement (bald): 
 If we ignore the selective labels problem and simply treat it as the traditional active learning setup where every instance will always be labeled, we can use the same computation procedure in the standard BALD algorithm (in the supervised learning setup) to select the next instances to label. (2)However, when humans are involved, we need to consider the probability of each instance being labeled. When the instance with high information gain is unlikely to be labeled, BALD may send many instances to the human labelers but only few of them will be labeled, thus wasting the time and resources of the human labelers.4.2 Bayesian Active Learning with Disagreement with Selective Labels (SEL-BALD)First, we show the information theoretical objective under ALIR. When considering the human discretion behavior, the information gain Î(y, θ|x, D) can be derived as Definition 4.2 (e-BALD).EQUATIONThe first equality is proved in Theorem A.1 in Appendix A. Intuitively, this objective indicates we should focus on instances with higher probabilities to be labeled by human decision makers. If e(x) is known exactly, we can directly select the instances by the acquisition function e(x) Î(y, θ|x, D).It's worth noting that when e(x) ≡ c, 0 < c ≤ 1, which means human decision makers will choose to label all instances with the same probability c, the Naive-BALD and e-BALD objective is equivalent and the problem reduces to the traditional active learning setting.Since in our setting, e(x) is not known a priori, we can fit another predictive model ê(x) = P (a = 1|x, ϕ) on the human decision dataset L to approximate this objective. However, in practice, ê(x) may be inaccurate given a small dataset. Therefore, we need to acquire samples that are 1) likely to be labeled by the human decision makers and 2) are both informative to the machine learning model and the human discretion model update.We consider the information gain about the human discretion response and the model parameters.Assume we try to use a Bayesian neural network e ϕ (x) to approximate the human discretion model e(x), we can define the information gain about the human discretion response and the model parameters asEQUATIONHere ĝt c is the probability of input x being labeled as c under the t-th MC sample from the approximated posterior q(ϕ). We denote this objective as Î(l, ϕ|x, L) ≈ I(l, ϕ|x, L). Selecting samples that maximize this acquisition function will help us to learn more effectively about the human discretion model. This joint information gain can be estimated as Definition 4.3 (Joint-BALD).EQUATIONAlgorithm 1 Bayesian Active Learning for Selective Labeling with Instance Rejection (SEL-BALD)Input: Initial labeled dataset D, L, pool dataset D pool , β, MC-samples T , budget B. Output: Updated machine learning model f (x).while Budget is not used up do Select the instance x i that maximizes the acquisition function in Equation ( 5), ( 6), or (7). Human will decide whether to label the instancex i . if the instance x i is labeled then Add (x i , y i ) to D and (x i , a i ) to L. else Add (x i , a i ) to L. end if Update the human discretion model e ϕ k+1 (x) on L.Update the machine learning model f θ k+1 (x) on D. end whileWe prove Equation ( 5) in Theorem A.2. The first term in Equation ( 5) is equivalent to Equation (3) and the second term is the information gain between the human discretion model and human responses. If we have the perfect knowledge about the human discretion model, then the second term equals 0 and Joint-BALD reduces to e-BALD. On the other hand, if we have a high epistemic uncertainty about human responses, the second term will encourage exploration to learn the human discretion model better. Once the second term becomes small (we have a good understanding of the human discretion model), Joint-BALD will automatically reduce to e-BALD and start exploiting.Motivated by the upper confidence bound (UCB) principle in online regret minimization [Auer, 2002 , Li et al., 2010] , we can define the acquisition function as Definition 4.4 (Joint-BALD-UCB).EQUATIONHere Q is the empirical quantile function and Q(ê ϕ (x i ), β) is the upper confidence bound of the probabilities (drawn using the MC samples) of x i being labeled by the human decision makers of our current best estimation. The hyperparameter β balances the exploration-exploitation trade-off. With a larger β → 1, the acquisition function will have more exploration of samples with higher uncertainties. Similarly, we can use Thompson Sampling [Thompson, 1933] to control the degree of exploration, which leads to the following objective: Definition 4.5 (Joint-BALD-TS).EQUATIONq(ê ϕ (x)) represents the empirical sample drawn from the MC samples. Each time we will random sample a set of model parameters from the posterior and select the instance with the largest information gain calculated using Equation ( 7). Compared to UCB, Joint-BALD-TS does not have to specify the hyperparameter. The complete algorithm for Bayesian active learning with selective labeling (SEL-BALD) is shown in Algorithm 1. 
experiments: 
 First, we want to emphasize that in ALIR, the benefit of each method is more nuanced than traditional active learning and highly depends on human decision behavior and the underlying data distribution. As our analysis above suggests, Naive-BALD and e-BALD correspond to the information gain when e(x) ≡ c, 0 < c ≤ 1. When c = 1, this corresponds to the traditional active learning problem where humans always label the instance, and the efficacy of Naive-BALD and e-BALD have been demonstrated in existing papers [Gal et al., 2017] . We also conduct the experiment under homogeneous human decision behavior in Appendix C. In this section, we aim to show the benefit of the proposed method when the human discretion behavior is unknown and heterogeneous.We conduct experiments on both synthetic and real-world datasets to demonstrate the effectiveness of our proposed algorithm when human discretion behavior is unknown and heterogeneous. We assume there is a cost of 1 to examine every data instance and a cost of 5 to label the instance.For the predictive model and human discretion model class, we use a Bayesian neural network and use MCdropout [Gal et al., 2017] to approximate the posterior (we set the number of MC samples as 40 in all experiments). The model architecture is a 3-layer fully connected neural network with LeakyReLU activation function. We use the Adam optimizer with a learning rate of 0.01. We run the experiments on a server with 3 Nvidia A100 graphics cards and AMD EPYC 7763 64-Core Processor. We compare our proposed methods against the Naive-BALD algorithm and the random acquisition function (RANDOM). We set β = 0.75 for Joint-BALD-UCB in all experiments. For results, we report the accuracy and number of samples acquired of the machine learning model on a subset of the test set where the instance has a positive probability to be included in the dataset. We do not consider the performance of instances that have zero probability of showing up in the data since human decision makers will always reject such decision instances and a machine learning model can have arbitrary performance on such subpopulation. The code is available at https://github.com/ruijiang81/SEL-BALD.We use a comprehensive synthetic example to illustrate the benefit of the proposed algorithms. We generate a synthetic two-cricle dataset with 2 features. The response variable is binary and can be considered as fraud of an insurance claim. We generate 20000 data samples and downsample 90% of the data points with x 1 ≥ 0 where x 1 is the second dimension of the input feature. This is to make the toy example a little more challenging for the RANDOM baseline and simulate a real-world scenario where there are a lot of insurance claims in the candidate pool that won't be investigated where active learning is necessary to reduce the potential waste of human workers' time. The resulting dataset has around 3700 samples as the training set and around 1600 samples as the test set. The visualization of the synthetic data is shown in Figure 11(a) .The qualitative results for each label acquisition method are shown in Figure 2 and the quantitative metrics are reported in Table 1 . Since there are many high-risk samples that human decision makers do not label, most of the samples RANDOM selects are not labeled by the human decision maker. Similarly, Naive-BALD is highly influenced by the initial labels and human discretion behavior. If the most informative samples to the current machine learning classifier have a low probability to be We report the accuracy and the fraction of the budget spent for examination and labeling. RANDOM and Naive-BALD spend most of the budget on examination and few labels is acquired. e-BALD acquires the most labels but many of them are not informative due to the noisy estimation of e(x). Joint-BALD-UCB and Joint-BALD-TS balance the examination and labeling budget automatically and achieve the best accuracy. Finally, we show the estimated human discretion model in Figure 3 . Since e-BALD acquires samples based on its current noisy estimation of the human discretion model, it fails to learn the correct decision boundary and maintains a wrong probability estimation about the left lower region of the feature space. While Joint-BALD has a slightly better estimation, it still focuses more on the high-probability region and has a large estimation error on the left lower region. Joint-BALD-UCB and Joint-BALD-UCB correctly recover the decision boundary with the help of sufficient exploration.For our case study, we use the Give-Me-some-Credit (GMC) dataset [Credit Fusion, 2011] . GMC dataset is a real-world loan dataset with 10 features including age, monthly income, debt ratio, number of dependents, and a binary response variable that indicates whether the loan application will default. We balance the dataset by random undersampling and randomly choose 400 samples as the test set.We use this dataset to build a hypothetical insurance claim case. Assume each instance is an insurance claim where a human agent can investigate whether it is fraudulent. The insurance company wants to build a machine learning model to predict fraud cases in the future. Since such investigation is costly, the company wants to use an active learning method and send some selected instances to agents for potential investigation and agents have the freedom to decline the investigation request since the company does not want to waste human resources and potential profit from acquiring these samples.Human Behavior Model: For the human discretion model e(x), we assume human decision makers will have the following acceptance probabilities: A detailed discussion about the initial human discretion model can be found in Appendix D. This simulates the selective labels setup where the biased human decision makers are skeptical about insured customers with high debt and these customers are more likely to be investigated.We report the accuracy and number of samples acquired for each method in Figure 4 (a) and Figure 4 (b) respectively. The results are averaged over 3 runs with a query size of 10, 50 randomly examined instances initially, and a budget of 450. Compared to other methods, RANDOM and Naive-BALD acquire the least samples given the same budget since both methods do not consider the human discretion behaviors, which leads to suboptimal model performance. e-bald has a small benefit over all other methods when the total budget is small, which is expected when the initial estimated human discretion model is useful since e-BALD greedily acquires samples based on the current best estimation. However, this effect soon disappears and e-bald has a worse performance compared to Joint-BALD-UCB and Joint-BALD-TS which have a better exploration of e(x) for a larger budget.We also examine each active learning method on a high-dimensional dataset MNIST [LeCun, 1998] .MNIST is a large handwritten digit dataset with 28x28 pixels and 10 classes. The test set size is 10000 and we use 10000 samples in the training set as D pool . We report the accuracy and number of samples acquired for each active learning method on the test set. The results are averaged over 3 runs with a query size of 20, 100 randomly examined instances initially, and a budget of 1000. For the human discretion behavior, we assume P (a = 1|x) = 0 for all classes 0-6, and human decision makers have the probabilities of 0.3, 0.3, 0.9 to label the instances with classes 7, 8, 9 respectively. This also corresponds to the realistic setting where many insurance claims will not be investigated and human discretion behavior is different for different instances.The results are shown in Figure 4 (c) and Figure 4 (d) respectively. Similarly to our previous findings, baselines like RANDOM and Naive-BALD struggle to acquire samples humans will label, and waste most of the budget on examination. For high-dimensional data, the performance of e-BALD appears worse in the beginning potentially because the propensity score is harder to estimate for highdimensional data [D'Amour et al., 2021] . On the other hand, Joint-BALD-UCB and Joint-BALD-TS are able to learn the human discretion model better through exploration, and lead to a larger number of samples labeled and better model performance.Changing Human Behaviors: While we motivate the Active Learning with Instance Rejection problem in the cold-start setting, the ALIR problem can still happen when human behavior changes, thus we need to acquire new labels to better model the underlying function and the human discretion model. We offer an illustrative example of ALIR under changing human behaviors in Appendix B.In the example, we assume the company starts to use a machine learning model to advise human decision makers, thus leading to a change in the human discretion behavior. We further demonstrate that the ALIR problem still exist when human behaviors change and the proposed methods in the paper can also be applied in such settings.More Real-World Datasets: We compare the proposed methods with baselines on Fashion MNIST [Xiao et al., 2017] , CIFAR-10 [ Krizhevsky, 2009] , Adult [Becker and Kohavi, 1996] and Mushroom [mus, 1981] datasets in Appendix E. We find similar findings that the proposed methods improve over Naive-BALD and RANDOM under the selective labels setting.Other Human Behavior Model: In addition to the piecewise linear function used to simulate the human behaviors, we compare with a Logistic Regression based human behavior model in Appendix F. We randomly sample a few positive and negative samples from the datasets to train a Logistic Regression model and use it to simulate human responses. The results are similar to our results presented in the main paper.Other Uncertainty Metric: While we focus on improving the BALD method in this paper, it is possible to extend the proposed method for these metrics by replacing the information gain term with another uncertainty term such as the entropy. We demonstrate that in Appendix G. We implement Joint-Entropy-UCB that replaces the information gain with the entropy measure on the synthetic data.Joint-Entropy-UCB can also improve the traditional entropy objective in this case.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: Our code will be made public upon acceptance. Guidelines:• The answer NA means that paper does not include experiments requiring code. • If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: We discuss potential societal impacts in the last section. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
conclusion and future work: 
 In this paper, we propose the Active Learning with Instance Rejection (ALIR) problem, which is a new active learning problem that considers the human discretion behavior for high-stakes decision making problems. We propose a new active learning algorithm e-BALD, Joint-BALD, Joint-BALD-UCB, and Joint-BALD-TS using deep Bayesian active learning for selective labeling (SEL-BALD) to address the ALIR problem and validate the effectiveness of our proposed algorithm on both synthetic and real-world datasets. With unknown and heterogeneous human behavior, Joint-BALD-UCB and Joint-BALD-TS often lead to robust model performance improvement across different budgets.However, there are several limitations to our work. First, we consider all humans as a single entity and assume they have the same discretion model which is an average of all humans. In practice, different human decision makers may have different discretion models. Second, it is also promising to use semi-supervised learning [Grandvalet and Bengio, 2004] to train the predictive model while it brings additional challenges to the ALIR problem. In addition, e(x) may not be accurately modeled which could affect the performance of the proposed algorithm. One potential issue is the misspecification of the model class used to represent human behavior. For instance, if a linear model like Logistic Regression is applied to a fundamentally non-linear human behavior pattern, this could introduce estimation bias into the proposed AL method. To mitigate this, non-parametric models could be employed, as they are less prone to such misspecification bias. Last, it is also worth noting that in certain applications like recidivism prediction or organ transplantation, prioritizing instances to be labeled may have significant fairness implications, therefore our method may not apply. However, the ALIR problem still exist in many real-world problems such as tax audit or fraud detection. It would be interesting to study the fairness concerns in ALIR. We believe these bring promising future research opportunities and leave these as future work. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: Our paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: Our paper uses public dataset.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: Theorem A.1 (e-BALD). Assume when a = 0, the observation is a fixed non-informative value y 0 (e.g., NA). If the human decision maker has a probability e(x) to label the instance, then I(y; θ|x, D) = e(x)I(y; θ|x, D, a = 1).Proof. Let e(x) = p(a = 1|x). The second equality is because of the law of total expectation. The first equality is by the chain rule of mutual information; the second equality is because the observation of all a is in L and all the observation of y is in D; the third equality is because the distribution of y and a are independent of ϕ and θ, respectively.While we motivate the Active Learning with Instance Rejection problem in the cold-start setting in the main paper, the Active Learning with Instance Rejection can still happen when human behavior changes, thus we need to acquire new labels to better model the underlying function and the human discretion model. Here we offer an illustrative toy example of ALIR under changing human behaviors. In this toy example, we assume the company starts to use a machine learning model to advise human decision makers, thus leading to human discretion behavior change. We use a synthetic example to illustrate the AI advising setting We generate a synthetic two-moon dataset with 2 features. The response variable is binary and can be considered as whether an insurance is fraudulent. We generate 20000 data samples and downsample 90% of the data points with x 0 ≤ 6 where x 0 is the first dimension of the input feature. This simulates a real-world scenario where there are a lot of valid insurance claims in the candidate pool where active learning is necessary to reduce the potential waste of human workers' time. The resulting dataset has around 3700 samples as the training set and around 1600 samples as the test set. The visualization of the synthetic data is shown in Figure 5(a) .Initial Human Behavior Model: For the initial human discretion model e 0 , we assume human decision makers will only investigate claims with 3 ≤ x 0 ≤ 6 and follow the acceptance probabilities below:  and x 1 > 3  0.4 3 ≤ x 0 ≤ 6 and x 1 ≤ 3  0 otherwise.h 0 (x) =    0.6 3 ≤ x 0 ≤ 6(8)This simulates the selective labels setup where initially human decision makers will only investigate claims of a certain subpopulation, and the machine learning model fitted on this subpopulation may not generalize well to the whole population even with ample data. Here the human decision makers have a higher probability to investigate fraud cases.Updated Human Behavior Model: We use the initial human behavior to collect 200 samples with instance rejection, then we train a Bayesian neural network f 0 (x) on the initial labeled dataset D and deploy it to assist human decision makers. After the machine learning model is deployed, we assume the human decision makers will have a new behavior model e(x) that follows the acceptance probabilities below:h(x) =        0.6ρ + (1 -ρ)f 0 (x) 3 ≤ x 0 ≤ 6 and x 1 > 3 0.2ρ + (1 -ρ)f 0 (x) 3 ≤ x 0 ≤ 6 and x 1 ≤ 3 0ρ + (1 -ρ)f 0 (x) x 0 ≤ 3 0 otherwise.Here ρ is an algorithmic-aversion hyperparameter that controls the level of trust in the machine learning model. When ρ = 0, the human decision makers will fully trust the machine learning model. When ρ = 1, the human decision makers will ignore the machine learning model and follow the initial human behavior model. We set ρ = 0.5 to simulate a partial adoption of the machine learning model. However, for the high-risk region x 0 > 6, the human decision makers will never investigate no matter what the machine learning model advises.Since the updated human behavior model may have higher label probabilities for the cases at certain regions (e.g., x 0 ≤ 3), we can use different active learning strategies to acquire new samples to improve the machine learning model's generalizability. We test Naive-BALD, RANDOM, and Joint-BALD-UCB on this synthetic dataset for illustration. We set the query size as 20 and the number of acquisition steps as 10. For batch acquisition with query size K, we use the greedy strategy to select the K samples with the largest acquisition function value.First, we visualize the labeled data after 3 and 9 steps of active learning in Figure 5 . We can see that Joint-BALD-UCB selects the samples that are likely to be labeled by the human decision makers and are both informative to the machine learning model and the human discretion model update. In contrast, Naive-BALD selects the samples that are informative to the machine learning model but may not be labeled by the human decision makers. In this case, Naive-BALD got stuck in the high-risk region x 0 > 6 where human decision makers will always reject the loan applications, thus leading to no improvement for the machine learning model. RANDOM selects the samples randomly and since many candidate applications are in the high-risk region, it also has a low probability of selecting the samples that will be labeled by the human decision makers. Joint-BALD-UCB successfully recovers the correct decision boundary. This example aims to demonstrate that the ALIR problem still exist when human behavior change and the proposed methods in the paper can also be applied in such settings.In this section, we show how different active learning methods perform under homogenous human behaviors. As demonstrated in the main paper, when e(x) ≡ c, 0 < c ≤ 1, Naive-BALD corresponds to the correct information gain and we expect all the proposed methods to perform similarly. e)-(g) shows the labeled data after 3 and 9 steps of active learning. Human decision makers always reject to label when x 0 > 6. BALIR selects the samples that are likely to be labeled by the human decision makers and are both informative to the machine learning model and the human discretion model update. In contrast, BALD selects the samples that are informative to the machine learning model but may not be labeled by the human decision makers. RANDOM selects the samples randomly and since many candidate applications are in the high-risk region, it also has a low probability to select the samples that will be labeled by the human decision makers. The human discretion model used in the case study ise(x) =      0.3 Debt > Q(Debt, 0.7) and Income ≤ Q(Income, 0.7) 0.9 Debt > Q(Debt, 0.7) and Income > Q(Income, 0.7) 0 Debt ≤ Q(Debt, 0.7).where Q(•, α) is the empirical 100 × α quantile of the metric. This simulates the selective labels setup where human decision makers are skeptical about insured customers who have high debt.We compare the proposed methods with baselines on Fashion MNIST [Xiao et al., 2017] , CIFAR-10 [ Krizhevsky, 2009] , Adult [Becker and Kohavi, 1996] and Mushroom [mus, 1981] datasets in this section. For Fashion MNIST and CIFAR-10, we use the same human behavior model as the one used in MNIST. For Mushroom, we assume humans have the following probability to label the instances: 0 when x 1 < 0; 0.3 when x 1 > 0 and x 4 > 0; 0.9 otherwise. For Adult, we assume humans have the following probability to label the instances: 0 when x 0 > Q 0.5 (x 0 ); 0.3 when x 0 ≤ Q 0.5 (x 0 ) and x 1 > Q 0.5 (x 1 ); 0.9 otherwise.The results for the accuracy and the number of samples labeled are shown in Figure 7 and Figure 8 , respectively. We find similar qualitative conclusions that Joint-BALD variants often produce better results than Naive-BALD and RANDOM. In addition to the piecewise linear human behavior model we used, we compare with a Logistic Regression based human behavior model. We randomly sample 3 positive and negative samples While we mainly focus on improving the BALD objective in this paper, our methods can potentially be generalized by replacing the mutual information with other metrics in the objective. For example, we can replace the information gain with other uncertainty measures such as entropy. A similar Joint-Entropy-UCB objective can be written as Ent({l, y}, {θ, ϕ}|x, L, D) = Q(ê ϕ (x), β)Ent(y, θ|x, D) + Ent(l, ϕ|x, L)As shown in Figure 11 , we implement Joint-Entropy-UCB with entropy measure on the synthetic data. Joint-Entropy-UCB can also improve the traditional entropy objective in this case. NeurIPS Paper Checklist 


Decision: After discussion with the reviewers and a robust rebuttal from the authors (including additional results based on feedback), I recommend accepting the submission (poster).

The main concerns were around limited (1) a limited number of datasets (the authors have added 3 more), (2) simplistic human discretion models (the authors have added an ml-based human model), (3) lack of experiments with real human data (I tend to agree this is beyond the scope of typical academic settings).

I expect the authors to include the new results in the final version of the paper and expand on what could not be fit into the rebuttal pdf (e.g., include ML-based human model results for all datasets, include full set of experiments with uncertainty based sampling AL methods). Also, move some of the results and discussion of the dynamic/changing rater setting to the main paper.

Reviews: review: paper_summary: paper_summary: This paper considers an active learning scenario where the human annotation is done under the restriction that the annotator is biased in deciding the labeling. The Bayesian framework makes the e-BALD and various versions based on a posterior sample of labeling probabilities. The authors consider the three cases in human labeling probability, including the non-labeling of specific cases.  Three versions of the mean, quantile (UCB), and samples reveal different aspects in experiments. The trade-off between exploration and exploitation in labeling probability is observed. Using the quantile can have a detailed balance between examination and labeling (examination means checking the attribute and non-labeling).
strength_weakness: strength_weakness: There can be many issues. 
The first one is the applicability of various metrics used in active learning.  Usually, entropy, margin, and variation ratio are traditional measures. Especially, the BALD combined with deep neural networks can show poor performance. It requires the study of applicability to other metrics or advanced active learning algorithms such as BADGE.  
The second one concerns the datasets used in experiments. Three datasets are not sufficient; more datasets, such as Fashion MNIT or ImageNet-tiny, can be considered, and their statistical significance can be studied more thoroughly. 
The last issue is human behavior, the scenario is relatively simple addressed by the simple logistic regression. The behavior of human annotators can be diverse because of their complex properties. It requires human models, which can be complex. When the human annotator's behaviors are complex, what’s the performance? Also, models such as logistic regression can be mimicked to generate samples.       

Minor: 
bayesian neural  -> Bayesian neural
e-bald -> e-BALD
questions: questions: Q1: Can you provide other metrics, such as the F1 score? In my opinion, the 1-score can be lowered by selective labeling.  
Q2: What’s the meaning of Total costs in the legend of Fig. 4?
limitations: limitations: Sufficiently discussed in the last section.
paper_summary: paper_summary: The paper introduces the Active Learning with Instance Rejection (ALIR) problem, addressing the issue of selective labeling where human discretion impacts the labeling process. In particular, humans might not always provide a label for a point returned by active learning; this abstention needs to be modeled explicitly. The authors propose new algorithms under the SEL-BALD framework to improve model performance using a human discretion model. The paper demonstrates the effectiveness of these algorithms through experiments on both synthetic and real-world datasets.
strength_weakness: strength_weakness: - The integration of Bayesian neural networks for modeling both machine learning tasks and human judgment significantly increases computational complexity. These Bayesian models are computationally demanding to train.

- While the paper includes experiments with both synthetic and real-world datasets, the scope of real-world applications examined is relatively limited. Critical domains such as medical diagnostics or financial fraud detection present unique challenges. A more comprehensive evaluation across diverse applications would better illustrate the versatility and reliability of the approach.

- The paper's assumption that human discretion can be accurately modeled using predictive tools like Bayesian neural networks may be overly optimistic. Real-world human decision-making is influenced by numerous complex and often unpredictable factors. The efficacy of these models can vary significantly across different contexts. Accurate modeling of human discretion typically requires extensive, high-quality historical data on human decisions, which may not always be available. In cases where data is limited, noisy, or biased, the human discretion model's performance could be significantly compromised. Furthermore, human decision-making processes can change over time due to regulatory updates, new information, or shifts in organizational policies. The study does not address how the human discretion model would adapt to such changes, potentially leading to outdated or inaccurate predictions.

- The legibility of figure legends is consistently poor due to their small size. This aspect requires significant improvement for better readability and comprehension.

- Definitions 4.1 through 4.5 would benefit from more detailed explanations accompanying the equations, providing context and clarification for the mathematical formulations presented
questions: questions: - Can you expand the techniques used to train the human discretion model?
- How can potential biases introduced by assumptions on the human discretion model be addressed?
- Is $c_e$ part of $c_l$?
limitations: limitations: See weaknesses above.
paper_summary: paper_summary: This paper presents a direct extension of BALD sampling criterion to address the need for active annotation rejection. The method assumes a rejection distribution over the candidate data samples in pool-based active learning and proposes an active sampling strategy that jointly considers the BALD informativeness and rejection cost.
strength_weakness: strength_weakness: 1.The core of this paper's design is the human discretion function e(x), but there are some unclear aspects regarding this function in my view. First, the approximation of e(x) seems to be achieved through an additional Bayesian network, but the detailed features of this Bayesian network are omitted in the paper. As the authors mention, given the small sample nature of active learning, the estimation of e(x) is likely to have an underfitting problem. However, the two solutions proposed by the authors in line 171 do not seem to address this issue adequately, and the authors should provide more explanation on this. Additionally, since e(x) is estimated by a Bayesian network, it should be involved in the computation in the form of some posterior distribution. However, in the examples given in the experimental results section, the true e(x) is presented as an unnormalized piecewise function. Therefore, using a posterior to approximate an e(x) that is not a valid density function raises questions about its effectiveness and limitations. I believe this is directly related to the deeper reason for label rejection, i.e., the analytical form of e(x). If the authors cannot theoretically prove that the proposed method generally works for all types of rejection reasons, they should at least compare the model's prediction of rejection (instead of the final overall sampling score) with several typical true rejection distributions in the experiments.
2.As a cost-aware active learning method, I find the experimental section of this paper somewhat weak. The real-world dataset used does not record the true rejection behavior of human annotators, which means, to some extent, that the experiments on MNIST are still synthetic. The true reasons for human annotators rejecting to label might be much more complicated than the prototype e(x) used in this paper. Therefore, I believe the evaluation of this specific problem requires a dataset with actual recorded human annotator rejections.
questions: questions: -Would it be possible to include a comparison of the model's rejection predictions against typical true rejection distributions in your experiments?- Should the first term in eq 4 be negative?
-The constant c in section 4.2 seems to differ from the c defined in section 3.1. You might want to use a different notation.
limitations: limitations: NA
paper_summary: paper_summary: This paper explores the setting where a learner has a label budget that can be used to acquire labeled data from a user. However, differently from traditional active learning approaches, the paper assumes that the user may not want/be able to label some examples. That is, for specific (limited) examples, the user returns a label when queried, while for other examples it does not return anything. The key challenge of the paper is how to motify traditional query strategies to account for this new possibility of not getting any label from the user. The paper solves this challenge by modeling the user through a Bayesian network and including two additional terms in the AL score function: the first term scales the potential gain obtained by labeling an instance by the probability that the instance would get labeled when queried, while the second represents the gain obtained for modeling the user (i.e., where the user labels the data).
strength_weakness: strength_weakness: ### Main Weaknesses:

1. The experiments include MNIST, which is a little bit outdated and nowadays considered a simple dataset. I'd recommend using more elaborated multiclass datasets (e.g., SVHN, Waterbirds, ...). Using more common datasets would strengthen the work.

2. Some literature study might be missing: learning to defer is an area where a model defers the prediction to a human whenever the human is more likely to provide a correct prediction, whereas the model is preferred in other cases. Thus, this research direction also investigates how to model a human. Also, learning to reject is a similar area, and [1] uses active learning assuming that the user may not provide some labels. Finally, there is a connection with PU Learning (Positive and Unlabeled Learning), where the propensity score named e(x) represents the probability of labeling an instance. Including a brief connection to these areas would improve the quality of the paper.

[1]: https://www.ijcai.org/proceedings/2020/0403.pdf
questions: questions: No questions.
limitations: limitations: Limitations are clearly discussed in Section 6.

Comments: title: Thank You
Thank You
question about $e(x)$
Rebuttal response
Reply
Reply to Rebuttal
Author Rebuttal Available
Thank you for the rebuttal
comment: Dear Reviewer WcLX, 

Could you let us know if our response addresses your question? Thank you. 

Best,

Authors of SEL-BALD
Thank you so much for being positive about our work. We will incorporate all your inputs into the final version. Thank you for your time and effort in helping with the review of our paper.
Thank you so much for being positive about our work. We will include your valuable feedback in the revised version.
Thank you so much. We are very grateful to you for reading our rebuttal, and also increasing your score accordingly. We will incorporate all your inputs into the final version and make sure to include more discussions about changing human behaviors in Appendix B in the revised main paper.
Thank you for your question. There are indeed scenarios where the function $e(x)$ may not be accurately learned, which could affect the performance of the proposed algorithm. One potential issue is the misspecification of the model class used to represent human behavior. For instance, if a linear model like Logistic Regression is applied to a fundamentally non-linear human behavior pattern, this could introduce estimation bias into the proposed AL method. To mitigate this, non-parametric models could be employed, as they are less prone to such misspecification bias.

Another challenge involves measurement bias in the human behavior model. For example, the features recorded in the data may not fully capture the features humans actually perceive, leading to further inaccuracies in modeling human behavior. While addressing this issue is beyond the scope of our current work, future research could explore bounding the measurement error using various distance metrics [1] such as TV distance $TV(p(x),p(\tilde{x}))\leq\delta$ to define possible ranges for $e(x)$ and optimize over the worst case to ensure robustness.

We will add the above discussion in the revised manuscript. 

[1] Guo, W., Yin, M., Wang, Y., & Jordan, M. (2022, June). Partial identification with noisy covariates: A robust optimization approach. In Conference on causal learning and reasoning (pp. 318-335). PMLR.
I thank the authors for their rebuttal. This has clarified many of my concerns. I still think that the robustness wrt a changing human discretion model deserves more attention in the main paper. Nevertheless, I have raised my score.
Thanks for your reply. My remaining interest is when the $e(x)$ can bother the performance of AL to ensure the robustness of the proposed algorithm. Is it impossible by the theoretical validation?
Dear Reviewer fApQ,

We hope this message finds you well. If you have any remaining questions or concerns, we would greatly appreciate your feedback. We believe we have thoroughly addressed the issues you raised and would be sincerely grateful if you could kindly reconsider your scores.

Thank you,

Authors of SEL-BALD
Dear Reviewer WcLX

We sincerely hope that our response has adequately addressed the major points raised in your review. We would be very grateful if you could kindly let us know if you have any remaining reservations or if all of your concerns have been resolved. We would also greatly appreciate it if you could reconsider your scores in light of our clarifications.

Thank you,

Authors of SEL-BALD
Dear authors,

Thanks for addressing my points. As stated in my review, I believe the paper is in a good shape and provides good contribution. Although the other reviewers highlighted some weakness, the pros of accepting the paper outnumber the cons, in my view. Thus, I recommend acceptance.
Thanks for the question. In our motivation, a human insurance agent needs to investigate a potential fraud case or a human doctor needs to perform a biopsy, which is why we describe it as human behavior. It may also be possible that a firm deploys another AI to determine the risk of such a labeling procedure. In this case, since the firm has query access, $e(x)$ is either known or can be estimated accurately with a large number of queries, then based on our theoretical analysis, the company can directly use $e$-BALD as the active learning strategy. The unknown human behavior brings additional challenges to this problem. We will add this discussion to the revised manuscript.
Thanks for your reply. The added results seem better during the rebuttal period. Also, the adaptation to other AL algorithms can be limited to the Bayesian framework, which is natural.  I have one more question about human behavior. The terminology of "Human behavior" is limited, in my opinion, since the rejection can occur by other machines. Can the case that some rejection or other actions can bother the proposed AL alg.'s performance exist?
Dear Reviewers,

Thank you for your reviews and your time.  The authors have provided feedback to your questions as well as additional empirical results.  Please take a careful look and provide any follow up questions and update your reviews as necessary.

Thank you,
Area Chair for Submission 12433
Thank you for the detailed rebuttal. The additional implementation details of the Bayesian network and the steps taken to address underfitting provide the necessary clarification regarding the estimation of e(x). The use of MC dropout and the model architecture explanation help in understanding the approach better. Including information gain and introducing UCB and TS variants further address my concerns. Regarding the experimental section, the addition of datasets like Fashion MNIST, CIFAR-10, and two UCI datasets strengthens the evaluation. While real-world datasets with recorded human rejections are challenging to obtain, your varied human behavior models offer valuable insights. Overall, while the paper's novelty and theoretical contributions are modest, the provided clarifications and additional experiments adequately address my concerns. I am inclined to maintain a positive rating as borderline accept.
Thanks!