ASPEST: Bridging the Gap Between Active Learning and Selective Prediction
Authors: Jiefeng Chen
Jinsung Yoon
Sayna Ebrahimi
Sercan O Arik
Somesh Jha
Tomas Pfister
Venue: Accepted by TMLR ()

Abstract: Selective prediction aims to learn a reliable model that abstains from making predictions when uncertain. These predictions can then be deferred to humans for further evaluation. As an everlasting challenge for machine learning, in many real-world scenarios, the distribution of test data is different from the training data. This results in more inaccurate predictions, and often increased dependence on humans, which can be difficult and expensive. Active learning aims to lower the overall labeling effort, and hence human dependence, by querying the most informative examples. Selective prediction and active learning have been approached from different angles, with the connection between them missing. In this work, we introduce a new learning paradigm, active selective prediction, which aims to query more informative samples from the shifted target domain while increasing accuracy and coverage. For this new paradigm, we propose a simple yet effective approach, ASPEST, that utilizes ensembles of model snapshots with self-training with their aggregated outputs as pseudo labels. Extensive experiments on numerous image, text and structured datasets, which suffer from domain shifts, demonstrate that ASPEST can significantly outperform prior work on selective prediction and active learning (e.g. on the MNIST$\to$SVHN benchmark with the labeling budget of 100, ASPEST improves the AUACC metric from 79.36% to 88.84%) and achieves more optimal utilization of humans in the loop.

Published in Transactions on Machine Learning Research (03/2024)
ASPEST: Bridging the Gap Between Active Learning and Selective Prediction
Jiefeng Chen" University of Wisconsin- Madisson
jiefeng jiefeng@cs.wisc.eeee eds
Jinsung Yoon Google
jinsungyonn@gggggge..com
Sayna Ebrahimi Google
saynae
Sercan Ö. Arık Google
soarik@ggoole.com
Somesh Jha University of Wisconsin--Madisn Google
jha Cha&csswiiee wisc. edu
Tomas Pfister
tpfister@@google.com com
Google
Reviewed on OpenReview:
https: // openreview. net/ forum? id= 3R3
Abstract
Selective prediction aims learn a reliable model that abstains from making predictions when uncertain. These predictions then be deferred to humans for further evaluation. As everlasting challenge for machine learning, in many real-world scenarios, the distribution of test data is different from the training data. This results in more inaccurate predic- tions, and often increased dependence on humans, which can be difficult and expensive. Active learning aims to lower the overall labeling effort, and hence human dependence, by querying the most informative examples. Selective prediction and active learning have been approached from different angles, with the connection between them missing. In this work, we introduce a new learning paradigm, active selective prediction, which aims query more informative samples from the shifted target domain while increasing accuracy and coverage. propose simple yet effective approach, ASPEST, that utilizes
For this new paradigm, ensembles of model snapshots with self-training with their aggregated outputs as pseudo la- bels. Extensive experiments on numerous image, text and structured datasets, which suffer from domain shifts, demonstrate that ASPEST can significantly outperform prior work on selective prediction and active learning (e.g. on the MNIST-SVHI benchmark with the labeling budget of 100, ASPEST improves the AUACC metric from 79.36% 88.84%) and
achieves more optimal utilization of humans in the loop.
1 Introduction
Deep Neural Networks (DNNs) have shown notable success in many applications that require complex
understanding of input data HHe et al. 2016a Devlin et al. 2018 Hamnu et al. 2014), including the
* Work doue during internship Google. Our code available
prediction

---

Published in Transactions on Machine Learning Research (03/2024)
ones that involve high-stakes decision making (Yngg) 2020 applications, it is typically required to allow them to abstain from their predictions that are likely to be wrong, and ask humans for assistance (a task known as selective prediction) (El- Yeniv et al. 2010 Geifman & El- Yaniv, 2017). Although selective prediction can render the predictions more reliable, it does so at the cost human interventions. For example, if model achieves 80% accuracy on the lest data, an ideal selective prediction algorithm should reject those 20% misclassified samples and send them to human for review.
For safe deployment of DNNs in high-stakes
Training time
Active learning
Acquisition function
Iuuman
Tabolii
Unlabelad tost data
Distribution P
Source validation data
Source tnrined Model
99% Accuracy
test data
training data
Update
Low confidence
craining data
Selective prodictor
Distribution Q
Unlaheled data
Accuracy
Model
1-th round
Selective prediction
round
After deployment
Prediction
to users
(a) Distribution shift problem
(b) Active selective prediction
Figure 1: Illustration of the distribution shift problem and the proposed active selective prediction solution. Under distribution shift, the model trained on the source training dataset will suffer a large performance degradation on the unlabeled test dataset. We propose to use active selective prediction to solve this problem, where active learning used to improve selective prediction under distribution shift. The selective predictor shown in Fig. (b) is built upon the sourco-trained model depicted in Fig. (a). In this setting, active Icarning selects small subset of data for labeling which are used to improve selective prediction the remaining unlabeled test data. This yields more reliable predictions and more optimized use of humans in the loop.
Distribution shift can significantly exacerbate the need for such human intervention. The success of DNNs often relies on the assumption that both training and test data are sampled independently and identically from the same distribution. In practice, this assumption may not hold and can degrade the performance on the test domain (Barbu et al. 2019 Koh et al.j 2021). For example, for satellite imaging applications, images taken in different years can vary drastically due to weather, light, and climate conditions (Koh el al., 2021). Existing selective prediction methods usually rely on model confidence to reject inputs (Geifiman & be poorly calibrated, especially
El-Yaniv 2017). However, it has been observed that model confidence with distribution shifts (Ovadia et al., 2019). The selective classifier might end up accepting many mis- classified test inputs, making the predictions unreliable. Thus, selective prediction might yield an accuracy below the desired target performance, or obtain a low coverage, necessitating significant human intervention.
To improve the performance selective prediction, one idea to rely on active learning and to have humans label a small subset of selected test data. The correct labels provided by humans can then be used improve the accuracy and coverage (see Sec. 3.2) of selective prediction on the remaining unlabeled test data, thus reducing the need for subsequent human labeling efforts. In separate forms, selective prediction |Geifman & El- Yaniv 2017 2019) and active learning (Settles 2009 have been studied extensively, however, to the best of our knowledge, this paper is first to propose performing active lcarning to improve seloctive prediction jointly, with the focus on the major real-world challenge of distribution shifts. Active domain adaplation (Su) let al.) 2020 Fu et al., 2021 Prabhu et al.) 2021) is one area close to this setting, however, it does not consider selective prediction. In selective prediction, not only does classificr need to be learned, but a selection scoring function also needs be constructed for rejecting misclassified inputs. Thus, going beyond conventional active learning methods that focus on selecting examples for labeling to improve the accuracy,
2

---

Published in Transactions on Machine Learning Research (03/2024)
Machine Learning Paradigu
Accuracy
Coverage
Acquisition Function
Selectivity
Adapt /
Adapt ş
Targtt Metric
If g(s) 2 7, outpot f(a);;
cxefenn Ta aod 2: 2 AUACC
Solective Prodiction
High
Low
None
otherwise, output
Active Loarning
Mediam
100%
a(B.f)
Always outpus ((x)
J
x
960
( 2 t, outpet f(r); otherwise; outpot 1.
cuefare las acelove or AUACC
Active Selectåve Prediction
High
High
/
Comparing different machine learning paradigms. With distribution shift and a small labeling budget, active learning can achieve 100% coverage (predictions on all test data), but it fails to achieve the target accuracy. Selective prediction can achieve the target accuracy, but the coverage is low, which results in significant human intervention. Active selective prediction achieves the target accuracy with much higher coverage, significantly reducing human labeling effort. B: selected batch for labeling. f: classifier and g: selection scoring function. acc: accuracy and cou: coverage.
Table 1:
we propose to also use those selected labeled examples to improve the selection scoring function. The optimal acquisition function (used select examples for labeling) for this new setting different compared to those in traditional active learning- e-g. if considence-based selection scoring function is employed, the selected labeled samples should have the goal of improving the estimation of that confidence score.
In this paper, we introduce a new machine learning paradigm: active selective prediction under distribution shift (see Fig. [,, which combines selective prediction and active learning to improve accuracy and cover- age, and hence use human labeling in a more efficient way. Table ! shows the differences among selective prodiction, active learning and active selective prediction. Active selective prediction highly important for most real-world deployment scenarios (e.g,, the batch prediction scenario where users give a batch of test inputs potentially with distribution shifts, and request predictions from a deployed pre-trained model). To the best of our knowledge, are the first to formulate and investigate this problem, along with the judi- ciously chosen evaluation metrics for it (Sec. B. We also introduce novel and simple yet effective method, ASPEST, for this active selective prediction problem (Sec. E. The key components of ASPEST, checkpoint ensembling and self-training, are designed to address the key challenge (i.e., the overconfidence issue) in the active selective prediction problem. On numerous worldwrrld datasets, we show that ASPEST consistently outperforms other baselines proposed for active learning and selective prediction (Sec.
2 Related Work
Selective prediction (also known as prediction with rejection/deferral options) constitutes a common de- ployment scenario for DNNs, especially in high-stakes decision making scenarios. In selective prediction, models abstain from yielding outputs if their confidence on the likelihood of correctness is not sufficiently high. Such abstinence usually incurs deferrals to humans and results in additional cost (Mozannar & Sontag, the 2020). Increasing the coverage the ratio of the samples for which the DNN outputs can be reliable
fundamental goal EEl- Yaniv et al., 2010 Fumerr & Roli 2002 [ellman| 1970 Geifman & EI- Yaniv 2019 Geifman & El-Yaniv (0177) considers selective prediction for DNNs with the 'Softmax Response' method, which applies carefully selected threshold on the maximal response of the softmax layer to construct the selective classifier. Takshminarayanan et al. (2017) shows that using deep ensembles can improve predictive uncertainty estimates and thus improve selective prediction. Rabanser et al. (2022) proposes a novel method, NNTD, for selective prediction that utilizes DNN training dynamics by using checkpoints during training. Our proposed method ASPEST also uses checkpoints to construct ensembles for selective prediction. Tn contrast to NNTD and other aforementioned methods, we combine selective prediction with active lcarning to improve its data efficiency while considering holistic perspective of having humans in the loop. This new active selective prediction setup warrants new methods for selective prediction along with active learning.
Active learning employs acquisition functions to select unlabcled examples for labeling, and uses these labeled examples to train models to utilize the human labeling budget more effectively while training DNNs (Settles, 2009 Dasgupta 2011h. Commonly-used active learning methods employ acquisition functions by considering uncertainty (Gal et al. 2017: Ducoffe & Precioso 2018 Beluch et al.] 2018) or diversity (Sener & Savarese 2017 Sinha et al. 2019). or their combination (Ash et al.) [019 Huang et al. 2010). One core challenge for active learning is the "cold start" problem: often the improvement obtained from active learning is less
3

---

Published in Transactions on Machine Learning Research (03/2024)
significant when the amount of labeled data is significantly smaller (Yuan et al. 2020 Hacohen et al. 2022) Moreover, active learning can be particularly challenging under distribution shift (Kirsch et al. 2021 Zhao
et al. 2021). Recently, active domain adaptation has been studied, where domain adaptation is com bindined with active learning (Su et al., 2020 Fu et al. 2021 Prabhu et al.) 2021). Different from traditional active learning, active domain adaptation typically adapts model pre-trained on the labeled source domain to the unlabeled target domain. In our work, we also try to adapt a source trained model to the unlabeled target test set using active learning, while focusing on building a selective classification model and reducing the human labeling effort. More related work are discussed in Appendix A
3 Active Selective Prediction
In this section, we first formulate the active selective prediction problem and then present the proposed evaluation metrics to quantify the efficacy of the methods.
3.1 Problem Setup
Let X' be the input space and y = 21, K} the label space The training data distribution is given as Px,y and the test data distribution is Qx.y (both are defined in the space X' x J). There might exist distributton shifts such as covariate shifts (i.e., Qx,y might be different from Px,y). Suppose for each imput x, an oracle (e-g.; the human annotator) can assign a ground-truth class label yz to it. Given a classifier f : X + y' trained on a source training dataset Dir Px, (~ means "sampled from"), and an unlabeled target test dataset Ux- {xI,..... .,), ~ Qx, our goal is to employ f to yield reliable predictions on Ux in human-in-the-loop scenario. Holistically, we consider the two approaches to involve humans via the prodictions they provide on the data: (i) selective prediction where uncertain predictions are deferred to humans to maintain a certain accuracy larget; and (ii) active learning where a subset of Ux unlabeled samples are selected for humans to improve the model with the extra labeled data to be used at the subsequent iterations. The concept of active selective prediction emerges from the potential synergistic optimization of these two approaches. By judiciously integrating active learning and selective prediction, we aim to minimize the demand for human labeling resources. Active learning in this context is tailored to select minimal yet impactful set of samples for labeling, which in turn informs the development of highly accurate selective predictor. This predictor effectively rejects misclassified samples, deferring them for human evaluation. As a result of this optimal integration, the number of samples requiring human intervention is significantly reduced, thereby enhancing the overall efficiency and resource utilization. Since these two approaches to involve humans have different objectives, their joint optimization to best use the human labeling resources is not straightforward.
As an extension of the classifier f (initialized by f), we propose to employ selective classifier fs including selection scoring function y X + R to yield reliable predictions on Ux. We define the predicted probability of the model f on the k-th class as f(x k). Then, the classifier is f(x) = arg maxkey f(x | k). g can be based on statistical operations on the outputs of f (e.g., g(x) = maxkey f(x k)). With f and g, the selective prediction model fa is defined as:
fx(x;)) = f(x)
if g(x)= 2 T, 1 if g() < T
(1)
where T is a threshold. If fa(x) -, then the DNN system would defer the prodictions to a human in the loop. To improve the overall accuracy to reach the target, such deferrals require manual labeling. To reduce the human labeling cost and improve the accuracy of the selective classifier, we consider labeling small subset of Ux and adapt the selective classifier fa on the labeled subset via active learning. The goal is to significantly improve the accuracy and coverage of the selective classifier fs and thus reduce the total human labeling elfort.
Suppose the labeling budget for active learning is M (i.e., M examples from Ux are selected to be labeled by humans). We assume that humans in the loop can provide correct labels. For active learning, we consider
In this paper, we focus the classification problem, although it can extended the regression problem.
4

---

Published in Transactions on Machine Learning Research (03/2024)
the transductive learning paradigm (Vapnik 1998), which assumes all training and test data observed beforehand and we can make use of the unlabeled test data for learning. Specifically, the active learning is performed on Ux to build the selective classifier fs, with performance evaluation of fs only on Ux. We adapt the source-trained classifier f to obtain fe instead of training fa from scratch to maintain feasibly-low computational cost.
Let's first consider the single-round setting. Suppose the acquisition function is a : X F IR where N F is the classifier space and G is the selection scoring function space. This acquisition function is different from the one used in traditional active learning (Gal al., 2017) since traditional active learning docsn't 't have the goal of improving g. In the beginning, f is initialized by f. We then select batch B for labeling by solving the following objective:
B = max
BCU (.,,
(2)
for which the labels are obtained get B" Then, we use B" to update f and g (e.g., via fine-tuning).
The above can be extended to multi-round selting. Suppose we have T rounds and the labeling budget for each round is = ]. In the beginning, fo is initialized by f. At the t-th round, first select batch B; for labeling by solving the following objective:
B arg
a(B, f1-1.001))
(3)
B 18
BCUX YO
for which the labels are obtained get B, Then we use B' to update fi and It-1 get fr and ge (e.g.. via fine-tuning the model on BT). With multiple-rounds setting, define B = U/_1 BY
3.2 Evaluation Metrics
To quantify the efficacy of the methods that optimize human-in-the-loop adaptation and decision making performance, appropriate metrics needed. The performance of the selective classifier fo (defined in Eq. (1) on Ux is evaluated by the accuracy and coverage metrics, which are defined as:
T) = Exxux T[[(x) yr A ( 2 t A $ B*)
(4)
Ex~Ux [(((x) 2 T B+
E ExNUx I[g(x) 2 A B"
coo(fs.
(5)
Ex~Ux 1[x & B+]
We can tune the threshold I to achieve certain coverage. There could be accuracy-coverage trado-off as increase coverage, the accuracy could be lower. We consider the following metrics that are agnostic to the threshold r: (1) maximum accuracy at a target coverage denoted as acc cov 2 tai (2) maximum coverage
at a target accuracy denoted as corjace 2 tai (3) Area Under Accuracy-Coverage Curve denoted as AUACC. The definitions of these metrics are given in Appendix
The target coverage le is tailored to each datasot, acknowledging the varying levels of difficulty inherent different datasets. Similar lep the target accuracy t, set according to the dataset's complexity. For more challenging datasets, where achieving high accuracy is difficult, lower target accuracy thresholds are set. Conversely, for datasets where models typically achieve higher accuracy, more stringent accuracy targets are established. By adapting the target accuracy and coverage thresholds to the datasct-specific challenges, we ensure that our evaluation is both rigorous and relevant to each dataset's context. Additionally, we employ the AUACC metric as our primary metric for comparative analysis. This choice is strategic, as it helps standardize our evaluations across varicty of datasots and addresses potential inconsistencies that might arise from solely relying on freehhldd-asse metrics.
3.3 Challenges
For active selective prediction, we want to utilize active learning to improve the coverage and accuracy of the selective classifier Ss, that consists of classifier and a selection scoring function g. In contrast. to
5

---

Published in Transactions on Machine Learning Research (03/2024)
classification with high confidence
Current decision boundary
Current decision boundary
Mis- classification
Rejection Region
Rejection
Regan
Mis-
classification
Selected samples for labeling in active learning
A Negative class
Unlabeled test samples
Mis-classification with high confidence
Adapted
Adapted
decision
decision
Positive class
boundary
boundary
(a) Sample selection based on uncertainty
(b) Sample selection based on diversity
Figure 2: Illustration of the challenges in active selective prediction using a linear model to maximize the margin (distance to the decision boundary) for binary classification. We consider a single-round active learning setup, without the inclusion of already labeled data. The current decision boundary is derived from labeled source training data. Since focus assessing performance with respect to the unlabeled test data, where our evaluation metrics are applied, we omit source training data from this illustration. The model confidence is considered to be proportional to the margin (when the margin is larger, the confidence is higher and vice versa). Fig. (a) shows if the samples close to the current decision boundary are selected for labeling (uncertainty-basel sample selection), then the adapted model suffers from the overconfidence issue (mis-classification with high confidence), which results acceptance of some mis-classified points. Fig. (b) shows if diverse samples are selected for labeling (e.g., using the k- Center-Greedy algorithm (Sener & Savarese 2017)), then the adapted model suffers from low accuracy. This leads to rejection of many points, necessitating significant human intervention.
conventional active learning, which only aims to improve the accuracy of J' active selective prediction also aims to improve so that it can accept those examples where f predicts correctly and reject those where f predicts incorrectly. Especially with distribution shift and a small labeling budget M, it can be challenging to train for high accuracy. Therefore, g is critical in achieving high coverage and accuracy of fa, for which we consider the confidence of f (i.e., the maximum softmax score of f) and train f such that its confidence can be used to distinguish correct and incorrect predictions. This might not be achieved easily since it has been observed that/ / can have overconfident predictions especially under distribution shift (Goodfellow et al. 2014 Hein et al., 2019). Besides, for active learning, typically we select samples for labeling based on uncertainty or diversity. However, in active selective prediction, sample selection based on uncortainty may lead to overconfideence and sample selection based on diversity may lead to low accuracy of f, as illustrated in Fig. 2 Our experiments in Appendix F22 show that these issues indeed exist the methods based on uncertainty sampling (e.g., SR+MMargin) achieve relatively high accuracy, but suffer from the overconfidence issue, while the methods based on diversity sampling (e.g. SR+CCG) don't have the overconfidence issue, but suller from low accuracy of f. Moreover, the hybrid methods based on uncertainly and diversity sampling (SR+CLUE and SR+BADGE) still suffer from the overconfidence issue. To tackle these, we propose a novel method, ASPEST, described next.
4 Proposed Method: ASPEST
We propose a novel method Active Selective Prediction using Ensembles and Self-training (ASPEST), which utilizes two key techniques, checkpoint ensembles and self-training, to solve the active selective prediction problem. The key constituents, checkpoint ensembles and self-training, are designed to tackle the funda- mental challenges in active selective prediction, with the idcas of selecting samples for labeling based on uncertainty to achieve high accuracy and using checkpoint ensembles and self-training to alleviate overconfi-
6

---

Published in Transactions on Machine Learning Research (03/2024)
dence. We empirically analyze why they can tackle the challenges in Section 5.3 In Appendix the complexity of the ASPEST algorithm.
D analyze
We first describe how the weights from the intermediate checkpoints during training are used to construct the checkpoint ensemble. Since we have all the test inputs, we don't need to save the checkpoints during training, but just record their outputs on the test set Ux- Specifically, we use a n K matrix P (recall that n = |Ux record the average of the softmax outputs of the checkpoint ensemble
and K is the number of classes) and use Nr to record the number of checkpoints in the current checkpoint ensemble. During training, we get a stream of checkpoints (assuming no two checkpoints arrive at the same time), and for each incoming chockpoint f, we update P and N₂ as:
1 PIA - No +1 (Po.k No + f(x; (b) for isis n and 1 S s K, Ne +1
(6)
Since it has been observed that an ensemble of DNNs (known "deep ensembles") usually produces confidence score that is better calibrated compared to a single DNN (Lakshminarayanan et al., 2017). we consider f to be in the form of deep ensembles and g to be the confidence of the ensemble. Specifically, we continue fine-tuning N models independently via Stochastic Gradient Descent (SGD) with different random soeds (c.g., the randomness can come from different random orders of training batches). At the beginming, we set each model H0 = J G = I,...,N), and set No = 0 and model og with the sourco-trained classifier f instead of random initialization, to minimize the computational cost. We fine-tune each model Sd Dir for n, steps via SGD using the following training objective: ming; (xx))ED (c where tc is the cross-entropy loss and @ is the model paramcters of fū For every C steps when training each fú, we update P and Ne using Eq 16 with the checkpoint model fú-
Onxk- Here, we initialize each
After constructing the initial checkpoint ensemble, we perform a T-round active learning process. In cach round of active learning, we first select samples for labeling based on the margin of the checkpoint ensemble, then fine-tune the models on the selected labeled test data, and finally perform self-training. We describe the procedure below:
Sample selection. In the t-th round, we select a batch B, with a size of m [ from Ux via:
B₁ arg
max SCUXXUU
Σ S(x,)
X.EB
where Bo = 0, S(x,) P.g max Pa, and أ = arg max Pi,.- We use an oracle to assign ground- truth labels to the examples in B, to get B,. Here, we select the test samples for labeling based on the margin of the checkpoint ensemble. The test samples with lower margin should be closer to the decision boundary and they are data points where the ensemble is uncertain about its predictions. Training on those data points can either make the predictions of the ensemble more accurate or make the ensemble have higher confidence on its correct predictions.
Fine-tuning. After the sample selection, reset Ne and P as Ne 0 and P OnxK, because want to remove those checkpoints in the previous rounds with worse performance from the checkpoint ensemble (experiments in Appendix F.9 show that after each round of active learning, the accuracy of the ensemble will significantly improve). We then fine-tune each model Pi-1 (j = 1, I.....N) independently via SGD with different randomness on the selected labeled test data to get f using the following training objective:
min Bxx,)ee B, ECE(x, y, 00) + A Bx (cE(x,U;00
(8)
where 0 is the model parameters of Hi and is a hyper-parameter. Note that here we use joint training Dir and Uf_, Br to avoid over-fitting to the small set of labeled test data and prevent the models from forgetting the source training knowledge (see the results in Appendix F.5 for the effect of using joint training
F.5
and the effect of A). For every ce epoch when fine-tuning each model P-1, we update P and N₂ using Eq. (6) with the checkpoint model K--1-

---

Published in Transactions on Machine Learning Research (03/2024)
Self-training After fine-tuning the models the selected labeled test data and with the checkpoint ensemble, we construct a pseudo-labeled set R via:
R= (Pa,,.. P.,x)) X; EX (1
max P., 1)]; kEY
where maXkey Pt, is the confidence of the chockpoint ensemble x; and 17 is threshold (refer to Section 5.2 for the ellect of v)) We do not add those lest data points with confidence equal to into the pseudo-labeled set because training on those data points cannot change the models much and may even hurt the performance (refer to Appendix F.8 for the justification such a design). We then perform self-training on the pscudo- labeled set R. For computational efficiency, we only apply self-training on a subset of R. We construct the subset, Raub by randomly sampling up to (p- n] data points from R, where E [0, 1]. We train each model fl (j = N) further on the pseudo-labeldd subset Rsub via SGD using the following training objective:
min OF
R(x))) (x:(x,y;0)) + ((())ED (cE(x,y;8)
(10)
where EKL is the KI.-Divergence loss which is defined as: (x,(xx);;)) = Ek= Vs log( s( We the KL-Divergence loss with soft pseudo-labels to alleviate the overconfidence issue since the predicted labels might be wrong and training the models on those misclassified pseudo-labeled data using the typical cross entropy loss will cause the overconfidence issue, which hurts selective prediction performance. For every ce epoch of self-training each model Ii we will update P and No using Eq (6) with the checkpoint model f². We add checkpoints during self-training into checkpoint ensemble to improve the sample selection in the next round of active learning.
After T rounds active learning, we use the checkpoint ensemble as the final selective classifier: the classifier fxx.) arg m P.,s and the selection scoring function gxx,) maxkey P.K
5
Experiments
This section presents experimental results, especially focusing the following questions: (Q1) Can use small labeling budget to significantly improve selective prediction performance under distribution shift? (Q2) Does the proposed ASPEST outperform baselines across different datasets with distribution shift?
(Q3) What is the effect of checkpoint ensembles and self-training in ASPEST?
5.1 Setup
Datasets. To demonstrate the practical applicability of active selective prediction, we conduct exper- iments not only on synthotic data but also on diverse real-world datasets, where distribution shifts are inherent and not artificially constructed. Specifically, we use the following datasets with distribution shift: (i) ANIST-SVHH
[LeCun] 1998 Netzer et al. 201 1), (ii) CIFAR-10-CCINTC-U (Krizhevsky et. al. 2009)
Darlow et al. 2018) ; (iii) FMoW (Koh al.] 2021), (iv) Amazon Review (Koh al. 2002),, (v) Domain-
Net (Peng et al. 2019 and (vi) Otto (Benjamin Bossan 2015). Details of the datasets described in
Appendix E.2
Model architectures and source training. On MNIST-SVHN, we use CNN (LeCun et al. 1989 On CIFAR-10-:CINIC-10, we use ResNet-20 (He et al.) 2016b). On the FMoW dataset, we use NensetNet- 121 (Huang et al. 20177b. On Amazon Review, we use the pre-trained RoBERTa (Liu et al., 2019) On the DomainNet datasct, we use ResNet-50 (He et al., (016a)). On the Otto datasct, we use multi- layer perceptron. On each dalaset, we train the models on the training set Dtr More details on model architectures and training on source data are presented in Appendix E.3
Active Icarning paper-parameters. We evaluate different methods with different labeling budget M values on each dataset. By default, we set the number of rounds T = 10 for all methods (Appendix F.6 presents the effect of T). During the active learning process, we fine-tune the model on the selected labeled test data. During fine-tuning, we don't apply any data augmentation the test data. We use the same fine- tuning syper-parameters for different methods to ensure a fair comparison. More details on the fine-tuning hyper-parameters can be found in Appendix E.4
8

---

Published in Transactions on Machine Learning Research (03/2024)
Baselines. We consider Softmax Response (SR) Geiffma & El- El-vvniv 2017 and Deep Ensembles (DE) [aakshinarrayaaan et al. 2017) with various active learning sampling methods the baselines. Uniform means combining SR with an acquisition function based on uniform sampling (similarly for DE and other acquisition functions). We consider sampling methods from both traditional active learning (c.g.. BADGE (Ash et al. 2019)) and aclive domain adaplation (e.g., CLUE (Prabhu et al. 2021 Appendix further describes the dctails of the baselines.
Haper-parameters of ASPEST. Table 2 comprehensively lists all the hyperparameters used in ASPEST, along with their respective default values. We set À = 1, n4 1000 and N = 5 (see Appendix F.7 for the effect of N), which are the same those for Deep Ensembles, for fair comparisons. For all datascts, we use C2 200, = 0.1, 0.9, the number of self-training epochs to be 20 and Ce = 5. It is important to note that the majority of these hyperparameters are general training parameters, and as such, they would not significantly benefit from per-dataset tuning. This significantly reduces the complexity and the neod for extensive validation datasets. Specifically, for hyperparameters Rss Cs, Ce. and p, we employ fixed values based empirical evidence, thus eliminating the need for their adjustment across different datasets. For other hyperparameters X, N, T, and 7, perform tuning based on the performance observed on validation dataset (i.c., DomainNet R-I).. Once tuned, these values are kept consistent across all other datasets. This approach mirrors real-world scenarios where practitioners often rely on a single, representative validation dataset to tune key hyperparameters before applying the model to various datasets.
Hyper-parameter
Notation
Default value
Trade-off parameter to balance the source and larget losses the training objectives The number of training steps in the initial fine-tuning
1000
Number models in the ensemble
Number of rounds in active learning
T
10
The step interval to update the checkpoint ensemble in the initial fine-tuming The epoch interval update the checkpoint ensemble The fraction to subsample the pseudo-labeled set
200
Ce
0.1
The threshold used to construct the pseudo-labeled set
0.9
Table 2: The hyperparameters used in the proposed method ASPEST.
5.2
Results
Method
cou" cou"lace 2 90% f
Accuracy of f
SR (M-0) 0)
0.08.10.0
24.68
Selective Prediction
DE (M-0)
0. 121200.11
26. 86.10.
Margin (M=1K)
N/A
82.26 +0.3
kCG (M=1K)
N/A
59.98
Active Learning
CLUE (M-1K)
N/A
81.6 1-.
BADGE (M IK)
N/A
12.114
Active Selective Prediction
ASPEST (M- IK)
94.9120.4
89.20:1033
Table 3: Results on MNIST-SVHN to describe the effect of combining selective prediction with active learning. The mean and std of each metric over three random runs are reported (mean=std). cov" is defined in Appendix F.4 All numbers are percentages. Bold numbers are superior results.
Impacts of combining selective prediction with active learning. We evaluate the accuracy of the source trained models the test set Ux of different datasets. The results in Appendix F.T show that the models trained on the souree training set Der suffer performance drop on the target test set Ux. and sometimes this drop can be large. For example, the model trained on MNIST has a source test accuracy of 99.40%. However, its accuracy on the target test set Ux from SVHN only 24.68%. we directly build a selective classifier on top of the source trained model, then to achieve a target accuracy of 90%, the coverage would be at most 27.42%. In Table we demonstrate that for a target accuracy of 90%, the coverage achieved by the selective prediction baselines SR and DE is very low (nearly 0%). It means that almost all
9

---

Published in Transactions on Machine Learning Research (03/2024)
test examples need human intervention labeling. This is large cost since the test set of SVHN contains over 26K images. The results in Table 3 also show that the active learning baselines Margin, kCG, CLUE and BADGE fail to achieve the target accuracy of 90% with a labeling budget of 1K. However, by combining selective prediction with active learning with the proposed method ASPEST, we only need to label 1K test examples achieve a target accuracy of 90% with a coverage of 94.91%. Thus, during active learning and selective prediction processes, only 5.09% test examples from SVHN need to be labeled by a human to achieve the target accuracy of 90%, resulting in a significant reduction of the overall human labeling cost.
Similar results are observed for other datasets (see Appendix F.4
Dataset
Domain Net R-C (easy)
Amazon Review
Otto
Metric
covlace 2 80% f
AUACC f
coolace 2 80% f AUACC 1
coolacc 80% f
AUACC
SR- Uniform
25.56 0.6
63.31 +0.4
13.71+11.3
72.71+1.5
63.58+0.7
84.46+0.2
SR- 16--onfidence
25.96 0.2
61.20+0066
11.28+8.9
72.89+0.7
69.63+1.7
85.91+003
SR- -Entroy
25.11-10
63.52-006
5.55+77.8
71.96+1.66
67.79++0.8
85.41+003
SR. + Margin
61.37=0.8
11,48+100.9
73.25++1.0
68.10
85.56++0.1
SR. kCG
0.3
58.88
20.02+11..
72.3443.2
64.84+0.77
85.08++0.2
3R+CLLE.
27.17=0.8
61.38-0.66
4.15+55.9
73.43+10.4
68.21+112
85.82-1:0.3
BADGE
27.78=0.8
61..0-10.5
22.58:10.4
73.80-10.6
67.23+110
85.41--0.3
DE + Uniform
30.82_0.8
67.60-10.4
34.35:1:44
76.20:10.3
70.74:0..5
86.7810.1
DE+ Entropy
29. 19.13-10 0.9
67.48+0.3
31.74+1.4
75.98++0.4
75.71++0.3
87.87+-0.1
DE+ Confidence
29.90-02
67.15-00.3
35.12++1.8
76.63+40.2
75.22-+0.2
87.81++0.1
DE+ Margin
31.82=1..
68.85=0044
33.12+1.33
76.184:0.2
75.19-10.8
87.89+10.2
DE+ AVD
32223-0.2
68. 08.73:-0.2
33.03:11.5
76.21440.4
75.91+10.2
87.89:10.0
DE+ CLUE
30. 30.
67.
33.92:13.0
76.27:10.6
69.66:00.5
86.67:10.1
BADGE
30. 10.16.1.3
08.46-10.3
32.23:13.7
76.10.10.7
73.23:1..2
87.55-10.1
ASPEST (ours)
37.
71.61=0022
38.4410.7
77.69110.1
77.85
88.2810..
Results of comparing ASPEST to the baselines on DomainNet R-C, Amazon Review and Ollo. The mean and std of each metric over three random runs are reported (mean=std)) The labeling budget M is 500. All numbers are percentages. Bold numbers are superior results.
Table 4:
Baseline comparisons. We compare ASPEST with two existing selective prediction methods: SR and DE with various active learning methods. The results in Table (complete results on all datasets for all metries and different labeling budgets are provided in Appendix F F.3) show that ASPEST consistently outperforms the baselines across different image, text and tabular datasels. For example, for MNIST- ASPEST improves the AUACC from 79.36% to 88. 83. when the labeling budget (M) is only 100. When A 500, for DomainNet R-*C, ASPEST improves the AUACC from 68.85% to 71.61%; for Amazon Review, ASPEST improves the AUACC from 76.63% to 77.69%; for Otto, ASPEST improves the AUACC from 87.89% to 88.28%.
5.3 Analyses and Discussions
In this section, we analyze why the key components checkpoint ensembles and self-training in ASPEST can improve selective prediction and perform ablation study to show their effect.
Checkpoint ensembles can alleviate overfitting and overconfidence. We observe that in active selective prediction, when fine-tuning the model on the small amount of selected labeled test data, the model can suffer overfitting and overconfidence issues and ensembling the checkpoints in the training path can effectively alleviate these issues (see the analysis in Appendix F.10
Self-training can alleviate overconfidence. We observe that the checkpoint ensemble constructed after fine-tuning is less confident on the test data Ux compared to the deep ensemble. Thus, using the softmax outputs of the checkpoint ensemble as soft pseudo-labels for self-training can alleviate overconfidence and improve selective prediction performance (see the analysis in Appendix F.11
Ablation studies. Compared to Margan,,n, ASPEST has two additional components: checkpoint ensemble and self-training. We perform ablation experiments MNIST and DomainNet to analyze
10

---

Published in Transactions on Machine Learning Research (03/2024)
Dataset
MNIST -SVHN
DomainNet R-C
Metric
AUACC
AUACC
Labeling Budget
100
500
500
1000
DE Margin
78.59:1144
94.31
68.85-00.4
10..3.30.3
ASPEST without self-training
78.09+1.3
94.25=0.4
69.59=0.2
72.45=0011
ASPEST without checkpoint cusemble
83.7812.99
96.54110.2
69.94-0.1
72. 72.20:10.4
ASPEST (x=0.1)
83.77+1.7
96.01=0.4
70.35=0.2
72.89+0.4
ASPEST (n-0.5)
83,9911.3
96.24:002
70.92=0.3
73.3740..
ASPEST (y=0.6)
85.17+1.3
96.24+002
70.96-0.2
73.06+001
ASPEST (==0.8)
85.40=2..3
96. 96.74=0.1
71.05=0.2
72.99:00.3
ASPEST (n=0.9)
88.84:1:1.0
96.62:10.2
71.61:10.2
73.27:10.2
ASPEST (x=0.95)
87.67+1.3
96.74+0.1
71.03=033
73.38+002
Table 5: Ablation study results for ASPEST, The mean and std of each metric over three random runs are reported (tteanstdd). All numbers are percentages. Bold numbers are superior results.
the effect of these. We also study the effect of the threshold r, in self-training. The results in Table 5 show for MSIST, adding the checkpoint ensemble component alone (ASPEST without self-training) does not improve the performance over DE+ Margin, whereas adding the self-training component alone (ASPEST without checkpoint ensemble) can significantly improve the performance. For DomainNet, both checkpoint ensemble and self- teafning have positive contributions. For both cases, ASPEST (with both self-training and checkpoint ensemble) achieves much better results than DE+ Margin or applying those components alonc. We also show the performance is not highly sensitive to y while typically setting larger v) (e.g. rj 0.9)
yields better results.
Integrating with UDA. To study whether incorporating unsupervised domain adaption (UDA) techniques into training could improve act selective prediction, we evaluate DE with UDA and ASPEST with UDA in Appendix F,12 Our results show that ASPEST outperforms (or on par with) DE with UDA, although ASPEST doesn't 't utilize UDA. Furthermore, we show that by combining ASPEST with UDA, it might achieve even better performance. For example, on MNIST-SVHHN ASPEST with DANN improves the mean AUACC from 96.62% to 97.03% when the labeling budget is 500. However, in some cases, combining ASPEST with UDA yields much worse results. For example, MNIST-SVHN, when the labeling budget is 100, combining ASPEST with UDA will reduce the mean AUACC by over 4%. We leave the exploration of UDA techniques to improve active selective prediction to future work superior and robust UDA techniques can be casily incorporated into ASPEST to enhance its overall performance.
6 Conclusion
In this paper, we introduce a new learning paradigm called active selective prediction which uses active learning to improve selective prediction under distribution shift. We show that this new paradigm results in improved accuracy and coverage on a distributionally shifted test domain and reduces the need for human labeling. We also propose a novel method ASPEST using checkpoint ensemble and self-training with low labeling cost. We demonstrate ASPEST's effectiveness over other baselines for this new problem setup various image, text and structured datasets. Future work in this direction investigate unsupervised hyperparancter tuning on test data, online data streaming, or further minimizing the labeling effort by designing time-preserving labeling interfaces.
Broader Impact Statement
The proposed framework yields more reliable predictions with more optimized utilization of humans in the loop. One potential risk of such a system is that the humans in the loop yield inaccurate or biased labels, our framework might cause them being absorbed by the prodictor model and the selective prediction mechanism, and eventually the outcomes of the system might be inaccurate and biased. We leave the methods for inaccurate label bias detection to future work.
11

---

Published in Transactions on Machine Learning Research (03/2024)
References
Jordan T Ash, Chicheng Zhang, Akshay Krishnamurthy, John Langford, and Alckh Agarwal. Deep batch active learning by diverse, uncertain gradient lower bounds. arXiv preprint arxiv 036600 2019.
Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenen- baum, and Boris Katz. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. Aduances neural information processing systems, 32, 2019.
William Beluch, Tim Genewein, Andreas Nürnberger, and Jan M Köhler. The power of ensembles for active learning in image classification. In Proceedings of the IEEE conference computer vision and pollern recognilion, pp. 9368-9377, 2018.
Wendy Kan Benjamin Bossan, Josef Feigl. Outo group product classification challenge, 2015.
Markus M Breunig, Hans-Peter Kriegel, Raymond T Ng, and Jörg Sander. Lof: identifying density-based local outliers. In Proceedings of the 2000 ACM SIGMOD international conference on Management of data, pp. 93 104, 2000.
Jiefeng Chen, Frederick Liu, Besim Avci, Xi Wu, Yingyu Liang, and Somesh Jha. Detecting errors and estimating accuracy on unlabeled data with self-training ensembles. Advances in Neural Information Processing Systems, 34:14980 14992, 2021.
Gordon Christie, Neil Fendley, James Wilson, and Ryan Mukherjee. Functional map of the world. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6172-6188 2018.
Ching-Yat Yao Chuang, Antonio Torralba, and Stefanie Jegelka. Estimating generalization under distribution shifts via domain-invariant representations. arXiv preprint arXin:2007.03511 2020.
Luke N Darlow, Elliot J Crowley, Antreas Antoniou, and Amos J Storkey. Cinic-10 is not imagenet or cifar-10. arXiv preprint arxiv:1810 03505, 2018.
Sanjoy Dasgupta. Two faces of active lcarning. Theoretical computer science, 412(119:1767-1781, 2011.
Jacob Devlin, Ming-Wae Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. orXiv preprinl urXiv:1810.000 04805, 2018.
Melanie Ducoffe and Frederic Precioso. Adversarial active learning for deep networks: margin based approach. arXiv preprint 1XXi:1802.0084 2018.
Ran El-Yaniv et al. On the foundations of noiso-free selective classification. Journal of Machine Learning Research, 11),), 2010.
Stanislav Fort, Huiyi Hu, and Balaji Lakshminarayanan. Deep ensembles: A loss landscape perspective. arXiv preprint ar Xiv: 2911.99222222 0.7777, 2019.
Bo Fu, Zhangjie Cao, Jianmin Wang, and Mingsheng Long. Transferable query selection for active domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7272-7281, 2021.
Giorgio Fumera and Fabio Roli. Support vector machines with embedded reject option. In International Workshop Support Vector Machines, pp. 68-89 82. Springer, 2002.
Yarin Gal, Riashat Islam, and Zoubin Ghahramani. Deep bayesian active learning with image data. In International Conference on Machine Learning, pp. 1183-1192. PMLR, 2017.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Nomain-adversarial training of neural networks. The journal of machine learning research, 17(1):2096-201 -2300, 2016.
12

---

Published in Transactions on Machine Learning Research (03/2024)
Yonatan Geifman and Ran El-Yaniv. Selective classification for deep neural networks. Advonces in neural information processing systems, 30, 2017.
Yonatan Geifinn and Ran El-Yaniv. Selectivenet: deep neural network with an integrated reject option. In International conference machine learning, pp. 2151-2155 PMLR, 2019.
lan J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples.
arXiv preprint 4rXio:1412.6572, 2014.
Yves Grandvalet and Yoshua Bengio. Semi-supervised learning by entropy minimization. Advances in neural information processing systems, 17, 2004.
Federica Granese, Marco Romanelli, Daniele Gorla, Catuscia Palamidessi, and Pablo Piantanida. Doctor: A simple method for detecting misclassification errors. Advances in Newral Information Processing Systems, 34:5669 5681, 2021.
Guy Hacohen, Avihu Dekel, and Daphna Weinshall. Active learning on a budget: Opposite strategies suit high and low budgets. orXiv preprint arXim:2202.0277 2022.
Awni Hanmun, Carl Case, Jared Casper, Bryan Catanzaro, Greg Diamos, Erich Elsen, Ryan Prenger, Sanjeev Satheesh, Shubho Sengupla, Adam Coates, et al. Deep speech: Scaling up end-to-end speech recognition. arXiv preprint Xiv: 141 12.12.5660 5567, 2014.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016a.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European conference on computer vision, pp. 630-645. Springer, 2016b.
Matthias Hein, Maksym Andriushchenko, and Julian Bitterwolf. Why relu networks yield cinf-conee prodictions far away from the training data and how to mitigate the problem. In Proceedings of the IEEE/CVF Conference on Compuler Vision and Pallern Recognition, pp. 41 50, 2019.
Martin E Hellman. The nearest neighbor classification rule with reject option. IEEE Transactions on Systems Science and Cybernetics, ((3):179-185, 1970.
Dan Hendrycks and Kevin Gimpel. baseline for detecting misclassified and out-of-distribution examples in neural networks. arXiv preprint arXiv:1610.022 021166 2016.
Gao Huang, Yixuan Li, Geoff Pleiss, Zhuang Liu, John E Hoperoft, and Kilian Q Weinberger. Snapshot ensembbles: Train 1, get m for free. arXiv preprinl ar xiv: 77000000000 2017a.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolu- tional networks. In Proceedings of the TEFE conference on computer vision and pattern recognilion, pp. 4700 4708, 2017b.
Sueng-Jun Huang, Rong Jin, and Zhi-Hua Zhou. Active learning by querying informative and representative examples. Advances in neural information processing systems, 23, 2010.
Fredrik D Johansson, David Sontag, and Rajesh Ranganath. Support and invertibility in domain-invariant representations. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 527- 536. PMLR, 2019.
Amita Kamath, Robin Jia, and Percy Liang, Selective question answering under domain shift. arXiv preprint arXiv:2006.090 09462, 2020.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv:1412.0m 6980, 2014.
ar preprint
Andreas Kirsch, Tom Rainforth, and Yarin Gal. Test distribution-aware active learning: A principled approach against distribution shift and outliers. arXio preprint arXiv:2166.1171 2021.
13

---

Published in Transactions on Machine Learning Research (03/2024)
Sosuke Kobayashi, Shun Kiyono, Jun Suzuki, and Kentaro Inui. Diverse lottery tickets boost ensemble from a single pretrained model. arXiv preprint or Xrx:22205. 1 1183, 2022.
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, et al. Wilds: A benchmark of in- the-wild distribution shifts. In International Conference on Machine Learning, pp. 5637 5664. PMLR, 2021.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncer- tainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017.
Yann LeCun. The MNIST database of handwritten digits. 1998. URL mnist/
Yann LeCun, Bernhard Boser, John Denker, Donnie Henderson, Richard Howard, Wayne Hubbard, and Lawrence Jackel. Handwritten digit recognition with prok-prrong network. Advances in neural information processing systems, 2, 1989.
Dong-Hyun Lee et al. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop challenges in representation learning, ICML, volume 3, pp. 896, 2013.
Xiaofeng Liu, Chaehwa Yoo, Fangxu Xing, Hyejin Oh, Georges El Fakhri, Je-Won Kang, Jonghye Woo, ct al. Deep unsupervised domain adaptation: A review of recent advances and perspectives. APSIPA Transactions on Signal and Information Processing, 11(1), 2022.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint ar Xiv: 1169,, 2019.
Mingsheng Long, Zhangjic Cao, Jianmin Wang, and Michael I Jordan. Conditional adversarial domain adaptation. Advances in newral information processing systems, 31, 2018.
Andrew McCallum, Kamal Nigam, et al. Employing em and pool-based active learning for text classification. ICML, volume 98, pp. 350 358. Citeseer, 1998.
Mohammad Moghimi, Serge J Belongie, Mohammad Saberian, Jian Yang, Nuno Vasconcelos, and Li-Jia Li. Boosted convolutional neural networks. In BMVC, volume 5, pp. 6, 2016.
Hussein Mozannar and David Sontag. Consistent estimators for learning to defer to an expert. In Interna- tional Conference on Machine Learning, pp. 7076-7087 7087. PMLR, 2020.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised fcature learning. 2011.
Jianmo Ni, Jiacheng Li, and Julian McAuley. Justifying recommendations using distantly-lbbeld reviews and fine- fne-grained aspects. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference natural language processing (FMNLP--ICCNLP)) pp. 188 197, 2019.
Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, David Sculley, Sebastian Nowozin, Joshma Dillon, Balaji Takshminarayanan, and Jasper Snoek. Can you trust your model's uncertainty? evaluating predictive uncertainty under datasct shift. Advances in neural information processing systems, 32, 2019.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine. Learning Research, 12:2825 2830, 2011.
14

---

Published in Transactions on Machine Learning Research (03/2024)
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE International Conference on Compuler Vision, pp. 1406 1415, 2019.
Viraj Prabhu, Arjun Chandrasekaran, Kate Saenko, and Judy Hoffman. Active domain adaptation via clustering imcortainty-wcigheed embeddings. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 5505-8514, 2021.
Stephan Rabanser, Anvith Thudi, Kimia. Hamidich, Adam Dziedzic, and Nicolas Papernot. Selective classi- fication via neural network training dynamics. arXiv preprinl arXiv:2205.1355 2022.
Shiori Sagawa, Pang Wei Koh, Tony Lee, Irena Gao, Sang Michael Xie, Kendrick Shen, Ananya Kumar, Weihua Hu, Michihiro Yasunaga, Henrik Marklund, et al. Extending the wilds benchmark for unsupervised adaptation. arXiv preprint arXiv:2112, 100000011.005000, 2021.
Kuniaki Saito, Donghyun Kim, Stan Sclaroff, Trevor Darrell, and Kate Saenko. Semi-supervised domain adaptation via minimax entropy. In Proccedings of the IEEE/CVF International Conference Computer Vision, pp. 8050 8058, 2019.
Mohammadreza Salehi, Hossein Mirzaei, Dan Hendrycks, Yixuan Li, Mohammad Hossein Rohban, and Mohammad Sabokrou. A unified survey on anomaly, novelty, open-set, and out-of-distribution detection: Solutions and future challenges. arXiv preprint arXiv:2110. 14051, 2021.
Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. arXiv preprint arXi:: 00489, 2017.
Burr Settles. Active learning literature survey. 2009.
Samarth Sinha, Sayna Ebrahimi, and Trevor Darrell. Variational adversarial active learning. In Proceedings of the IEEE/CVF International Conference on Compuler Vision, pp. 5972 5981, 2019.
Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi-supervised learning with con- sistency and confidence. Advances in neural information processing systems, 53:596-608, 2020.
Jong-Chy Su, Yi-Hsuan Tsai, Kihyuk Sohn, Buyu Liu, Subhransu Maji, and Manmohan Chandraker. Active adversarial domain adaptation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 739 748, 2020.
Vladimir Vapnik. Statistical learning theory. Wiley, 1998. ISBN 978-0-471-03003-4.
Feng Wang, Guoyizhe Wei, Qiao Liu, Jinxiang Ou, Hairong Lv, al. Boost neural networks by checkpoints. Advances in Newral Information Processing Systems, 34:19719 19729, 2021.
Colin Wei, Kendrick Shen, Yining Chen, and Tengyu Ma. Theoretical analysis of self-training with deep networks on unlabeled data. In International Conference on Learning Representations, 2020.
Wanqian Yang. Making Decisions Under High Stakes: Trustworthy and Expressive Bayesion Deep Learning. PhD thesis, 2020.
Huaxiu Yao, Caroline Choi, Bochuan Cao, Yoonho Lee, Pang Wei Koh, and Chelsca Finn. Wild-time: A benchmark of in-the-wild distribution shift over lime. arXiv preprinl arXie:2211.1/228, 2022.
David Yarowsky, Unsupervised word sense disambiguation rivaling supervised methods. In 33rd annual meeting of the association for computational linguistics, pp. 189-196, 1995.
Michelle Yuan, Hsuan-Tien Lin, and Jordan Graberra Cold-start active learning through self-supervised language modeling, arXiv preprint arXiv:2010. 1rXt::2010.09535, 2020.
Eric Zhao, Anqi Liu, Animashree Anandkumar, and Yisong Yue. Active learning under label shift. In International Conference on Artificial Intelligence and Stalistics, pp. 3412-3420. PMI.R 2021.
15

---

Published in Transactions on Machine Learning Research (03/2024)
Xiatian Zhu, Shaogang Gong, et al. Knowledge distillation by on-the-fly native ensemble. Advances in neural information processing systems, 31, 2018.
16

---

Published in Transactions on Machine Learning Research (03/2024)
Supplementary Material
In Section A discuss some additional related work. In Section B we give the definitions of some evaluation
metrics. In Section [ we describe the baselines in detail. In Section [
we show the complete ASPEST algorithm and analyze its computational complexity. In Section L we provide the details of the experimental
sctup. In Section
we give some additional experimental results.
A More Related Work
Distribution shift.
Distribution shift, where the training distribution differs from the test distribution, often occurs in practice and can substantially degrade the accuracy of the deployed DNNs (Koh et al.) 2021
Yao al. 2022 Barbu et al.) 2019). Distribution shift can also substantially reduce the quality of uncer- tainty estimation (Ovadia al.) 2019), which often used for rejecting examples in selective prediction and selecting samples for labeling in active learning. Several techniques try to tackle the challenge caused by distribution shift, including accuracy estimation (Chen et al. 2021 Chuang et al. 2020 error detec- tion (Hendryeks & Gimpel| 2016 Grancse et al.] (021), out-of-distribution detection (Salchi ct al. 2021
Saito el al. 2019), selective prediction (Kamath et al. 2020 and active learning [Kiscch et al. 2021j. In our work, we combine selective prediction with active learning to address the issue of distribution shift.
domain adaptation (Ganin et al. 2016
Deep ensembles. Ensembles of DNNs (or deep ensembles) have been successfully used to boost predictive performance (Moghimi et al.) 2016 Zhu al., 2018)- Deep ensembles also be used to improve the predictive uncertainty estimation [Lakshminarayanan et al. 2017 Fort et al., 2019). [akshminarayanan et al.] (2017) shows that random initialization of the NN parameters along with random shuffling of the data points are sufficient for deep ensembles to perform well in practice. However, training multiple DNNs from random initialization can be very expensive. To obtain deep ensembles more efficiently, recent papers explore using checkpoints during training lo construct the ensemble (Wang el al. 2021 Huang et al. 2017a), or fine-tuning single pre-trained model to create the ensemble (Kobayashi the checkpoints during fine-tuning sourco-trained model via active learning the ensemble and further boost the ensemble's performance via self-training. We also use the ensemble's uncertainty measured by a margin to select samples for labeling in active learning.
2022). In our work, we use
Self-training. Self-training is a common algorithmic paradigm for leveraging unlabeled data with DNNs. Self-training methods train a model to pseudo-labels (i.c., predictions on unlabeled data made by a previously-learned model) to boost the model's performance (Yarowsky, 1995 Grandvalet & Bengio 2004; Lee et al., 2013: Wei al., 2020 [Sohn et al.] (2020). In this work, we use self-training improve selective prediction performance. Instead of using predicted labels as pseudo-labels as common practice in prior works, we use the average softmax outputs of the checkpoints during training as the pseudo-labels and self-train the models in the ensemble on them with the KI-Divergence loss to improve seleetive prediction performance.
B Evaluation Metrics
We have introduced the accuracy and coverage metrics in Section 3.2 The accuracy is measured on the predictions made by the model without human intervention while the coverage is the fraction of remaining unlabeled data points where we can rely on the model's prediction without human intervention. The accuracy and coverage metrics depend on the threshold т. The following evaluation metrics are proposed to be agnostic to the threshold
Maximum Accuracy at a Target Coverage. Given target coverage te, the maximum accuracy is defined as:
max T)) s.l. cov(Ss,7) 2 la
(11)
We denote this metric acc|cov 2 la-
17

---

Published in Transactions on Machine Learning Research (03/2024)
Maximum Coverage at Target Accuracy. Given target accuracy tu. the maximum coverage is defined as:
cou(fa, s.t. acd(fe,)) to
(12)
When T oc, we define cov(fs.7) 0 and aco(fa) 5) 1. We denote this metric as covlace 2 tu. Area Under Accuracy-Coverage Curve (AUACC). We define the AUACC metric as:
AUACC(F.) = [' ace((s::)dcow(f..)
(13)
We use the composite trapezoidal rule estimate the integration.
C
Baselines
We consider selective classification baselines Softmax Response (SR) |Geifman & Yanivy 2017 and Deep Ensembles (DF) TTakshminarayanan et al. 2017 and combine them with active learning techniques. We describe them in detail below.
C.1 Softmax Response
Suppose the neural network classifier where the last layer is a softmax. Lot f(x ks) be the soft response output for the k-th class. Then the classifier is defined f(x) arg max [(x k) and the selection scoring function is defined as g(x) maXkey f(x k), which is also known the Maximum Softmax Probability (MSP) of the neural network. Recall that with f and the seleetive classifier is defined in Eq ① We use active learning to fine-tune the model ʃ to improve selective prediction performance of SR on the unlabeled test dataset Ux. The complete algorithm is presented in Algorithm [ In our experiments, we always set À = 1. We use the joint training objective (22 to avoid over-fitting to the small labeled test set Lf_1B, Br and prevent the model from forgetting the souree training knowledge. The algorithm can be combined with different kinds of acquisition funetions. We describe the acquisition functions considered for SR below.
Uniform. In the t-th round of active learning, we select | data points as the batch B, from Ux \ 1 B, via uniform random sampling. The corresponding acquisition function is: a(B, 11)) = 1. When solving the objective (11 the tic is broken randomly.
Confidence. We define the confidence score I on the input x as
Scout f) = max
f(x k)
(14)
kEY
Then the acquisition function in the 1-th round of active learning is defined as:
Σ
Scont(x; f(-1)
(15)
XEB
That is we select those test examples with the lowest confidence scores for labeling. Entropy. We define the entropy score of f on the input as
Sentropy (xi
Σ -f(x k) log f(x k)
(16)
kEY
Then the acquisition function in the 1-th round of active learning is defined as:
a B, ft-1. = Σ
Sentropy (x; ft-1)
(17)
xEB
18

---

Published in Transactions on Machine Learning Research (03/2024)
That is we select those test examples with the highest entropy scores for labeling.
Margin. We define the margin score of f on the input x as
Smargin (x: f) f(x ()
max kEV
f(x
k)
(18)
8.1.
arg max kEy
f(x k)
(19)
Then the acquisition function in the t-th round of active learning is defined as:
a (1,
Σ Smargin (x; ft-1)
(20)
That is we select those test examples with lowest margin scores for labeling.
kCG. We use the k- Ceeeeer-Greedy algorithm proposed in (Sener & Savarese 2017 to select test examples for labeling in each round.
CLUE. We use the Clustering Uncertainty-weige select test. examples for labeling in each round. Following (Prabhu al. 2021 we set the hyper-parameter T 0.1 on DomainNet and set 1.0 on other datasets.
Embeddings (CLUE) proposed in (Prabhu et al. 2021
BADGE. We use the Diverse Gradient Embeddings (BADGE) proposed in Ash et al. 2019 to select test examples for labeling in each round.
Algorithm Softmax Response with Active Learning
Input: A training dataset D an unlabeled test dataset Ux, the number of rounds T, the labeling budget A.
M, a source-trained model f, an acquisition function a and a
Tet fo = f.
Let Bg 0.
Let (x) maxxey [x(x k).
for t = L. T do
Sclect batch B, with a size of m -1 from Ux for labeling via:
B. arg
max BCUx B.J B
a(B, Se-1,
(21)
Use an oracle assign ground-truth labels the examples in B, to get Br. Fine-tune the model fi-1 using the following training objective:
min Ex (cE(x, y; 0) X Bix.)ED lCE(x, v;
(22)
where is the model parameters of fr-1 and lce is the cross-entropy loss function. Let f end for
Output: The classifier f f= fr fr and the selection scoring function g = maxkey f(x k).
C.2 Deep Ensembles
It has been shown that deep ensembles can significantly improve selective prediction performance (Laksh- minarayanan et al., 2017), not only because deep ensembles are more accurate than a single model, but also because deep ensembles yield more calibrated confidence.
Suppose the ensemble model contains N models fN Let fi | k)) denote the predicted probability of the model f the k-th class. We define the predicted probability of the ensemble model f on the k-th
19

---

Published in Transactions on Machine Learning Research (03/2024)
class as:
N f(x [() N f(((1) 1t).
(23)
The classifier is defined as f(x) = arg max f(x | k) and the selection scoring function is defined as g(x) = maxkgy f(x | k). We use active learning to fine-tune each model f in the ensemble to improve selective prediction performance of the ensemble on the unlabeled test dataset Ux. Each model fi is first initialized by the source-trained model f, and then fine-tuned independently via Stochastic Gradient Decent (SGD) with different sources of randomness (c.g., different random order of the training batches) the training dataset D and the selected labeled Lest data. Note that this way to construct the ensembles is different from the standard Deep Ensembles method, which trains the models from different random initialization. We use this way to construct the ensemble duc to the constraint in our problem sctting, which requires us to fine-tune a given source-trained model f. Training the models from different random initialization might lead to an ensemble with better performance, but it much more expensive, especially when the training dataset and the model are large (e.g., training foundation models). Thus, the constraint in our problem setting is feasible in practice. The complete algorithm is presented in Algorithm 2 In our experiments, we always set X = N = 5, and n, 1000. We also use joint training here and the reasons are the same as those for the Softmax Response bascline. The algorithm can be combined with different kinds of acquisition functions. We describe the acquisition functions considered below.
Uniform. In the t-th round of active learning, we select [ data points as the batch B, from Ux \utzo Br via uniform random sampling. The corresponding acquisition function al (B, 11-1,9t-1) = When solving the objective (30) the tie is broken randomly.
Confidence. The confidence scoring function Scout for the ensemble model f is the same that in Eq. [4]
(f(x k) for the ensemble model is defined in Eq. (23)). The acquisition function in the t-th round of active learning is defined as:
a (B, = Σ
(cont(x; ft-1)
(24)
x
That is we select those test examples with the lowest confidence scores for labeling.
Entropy. The entropy scoring function Sentropy for the ensemble model f is the same as that in Eq.
16
The acquisition function in the 1-th round of active learning is defined as:
a(B, f1-1
Σ Sentropy (x; (x; ft-1),
(25)
xEB
That is we select those test examples with the highest entropy scores for labeling.
Margin. The margin scoring function Smargin for the ensemble model f is the same as that in Eq. 18
The acquisition function the t-th round of active learning defined as:
a(B, fo-1 = Σ
Smargin (x; fc-1)
(26)
x B
That is we select those test examples with the lowest margin scores for labeling.
Avg- KLD. The Average Kullback-Leibler Divergence (Avg-KLD) is proposed in (McCallum et al. 1998) as
a disagrecment mcasure for the model ensembles, which can be used for sample selection in active learning. The Avg-KLD score of the ensemble model f on the input x is defined as:
fx log (' kk) f(x k)
Swix: f)
Σ
(27)
kEy
Then the acquisition function in the 1-th round of active learning is defined as:
a f6-1191-11 =
Ski(x; ft-1),
(28)
xEB
20

---

Published in Transactions on Machine Learning Research (03/2024)
That is we select those test examples with the highest Avg-KLD scores for labeling.
CLUE. CLUE (Prabhu et al, 2021 is proposed for single model. Here, we adapt CLUE for the ensemble model, which requires a redefinition of the entropy function H(Y x) and the embedding function e(x) used in the CLUE algorithm. We define the entropy function as Eq. (16) with the ensemble model f. Suppose oi is the embedding function for the model f in the ensemble. Then, the embedding of the ensemble model ʃ the imput x is (x)......, (x)], which is the concatenation of the embeddings of the models ft ,....... f Following (Prabhu et al. 2021
set the
T 0.1 DomainNet and set T 1.0
other datasets.
BADGE. BADGE Assh et al. 2019) is proposed for single model. Here, we adapt BADGE for the ensemble
model, which requires a redefinition of the gradient embedding in the BADGE algorithm. Towards this end, we propose the gradient embedding Ir the ensemble model f as the concatenation of the gradient embeddings of the models foaelss'.... f!....,,N..
Algorithm Deep Ensembles with Active Learning
Input: A training dataset D"r, An unlabeled test dataset Ux, the number of rounds T, the total labeling budget M, a source-trained model f, an acquisition function a(B,f,9), the number models in the ensemble N, the number of initial training steps n,, and a hyper-parameter X. Let fd f for j = I.....N.
Fine-tune each model fá in the ensemble via SGD for n, training steps independently using the following training objective with different randomness:
min 83
B( (c
(29)
where 0 is the model parameters of f and fc is the cross-entropy loss function.
Let By = 0.
Tet ge (x) = maxxay fe(x ki).
for = 1, T do
Select batch B, with a size of m
11 from Ux for labeling via:
B, arg
max
(30)
BCU BCUX XCUX B
Use an oracle to assign ground-truth labels the examples B, to get. Br. Fine-tune each model fi_, in the ensemble via SGD independently using the following training objective with different randomness:
min 6
(x R. (x 9; X Bix lo y:
(31)
where Hi the model parameters of fi-1-
Tet Hi = و for =o.rj..... I....,N.
end for
Output: The classifier f = fr and the selection scoring function g = maxxey f(x k).
ASPEST Algorithm and its Computational Complexity
Algorithm 3 presents the overall ASPEST method. Figure 3 illustrates how the chockpoint ensemble and the pseudo-labeled set constructed in the proposed ASPEST. Next, we will analyze the computational complexity of ASPEST.
Let the complexity for one step of updating P and N be tu (mainly one forward pass of DNN); for one DNN gradient update step is lg (mainly one forward and backward pass of DNN); and for sample selection
is l, (mainly sorting test examples). Then, the total complexity of ASPEST would be O ON Re
4, + N
21

---

Published in Transactions on Machine Learning Research (03/2024)
Algorithm 3 Active Selective Prediction using Ensembles and Self-Training
Input: A training set Drr unlabeled test set Ux, the number of rounds T, the labeling budget M, the number of models N, the number of initial training steps na, the initial checkpoint steps ca. checkpoint epoch Ce. threshold 7, sub-sampling fraction p, and hyper-parameter X.
Let fů = for j = I.....N. Set N₂ = and P OnxK-
tune each fú for n, training steps using the following training objective:
min E ( y;
and update P and Ne using Eq. 6 every es training steps. for f t= 1.... do Select batch B, from Ux for labeling using the sample selection objective 7 Use an oracle to assign ground-truth labels to the examples in B, to get B,. Set N₂ = and P OnxK.
Fine-tune each fi-1 using objective (8), while updating and N using Eq (6 every ee training epochs.
Tet J = 1L-1-
Construct the pseudo-labeled set R via Eq and create Raub by randomly sampling to [p-n] data points from R. Train each f further via SGD using the objective (10 and update P and Ne using Eq 6 every ce training epochs.
9
end for
Output: The classifier f(x,) arg max Pi. and the selection scoring function g(x,) maxxey P.kk
Checkpoint ensemble
Train 6g epochs
Train Ce epochs
Pscudo-labeled set
feL1 Checkpoint
fe
feL1
(xL,
6.1
0.9
8G
ao
no
02
0.2
0.2
0.2
0.2
Checkpoint
Checkpoint
0.3
0.1
9.1
0.4
(Xige
0.55
025
0.0
Train Ca epochs
Train Ce epochs
(Xig²
fe-
f2
f2-1
0.0
0.02
0.0
0.0
0.98
Chockpoint
Checkpoint
Checkpoint
Average softmax output of checkpoints Ux
Figure 3: Illustration of the checkpoint ensemble and pseudo-labeled set construction in the proposed ASPEST.
ne + +e. -t +N N. Sater -tul). where ef is the number of fine-tuning epochs and es is the number of self-training epochs and b is the batch size. Although the training objectives include training on Drr, the complexity doesn't depend on the size of Dtr since we measure over Uf_,, Bt in training objective (8) and mcasure e, over Reub in training objective (10). In practice, we usually have t, & tg and tn < lg. Also, we set ' <,,,, < T Ey and txte, & (ef p). F. So the complexity of ASPEST e,
is ONN. ... tt.. Suppose the size of Dir is ner and the number of source training epochs Cp. Then, the complexity for source training ( Overall, the complexity of ASPEST would be much smaller than that of source training.
[ie. ep tt). In practice we usually have N - '' n ef & Bar Ey.
It should be noted that ASPEST shares the same computational complexity as the DarMargin baseline, which also has a computational complexity o (N-- eer While both methods utilize an ensemble approach, key distinction is that Da+Margin does not incorporate self-training, unlike ASPEST. In con- trast, the MR+Margin baseline, which does not utilize ensembles, has lower computational complexity of O(T -ee -.) (.))
22

---

Published in Transactions on Machine Learning Research (03/2024)
While Margin and ASPEST indeed have higher computational complexities compared to Margin due to the inclusion of ensembles, this does not necessarily translate to proportionally longer running times. With careful implementation and adequate computational resources, it is feasible to train ensemble models in parallel, potentially aligning the actual running times of ASPEST with those the baseline mothods, SR+Margin and DE+ Margin. This parallelization strategy could mitigale the increased computat ional ional demand, making the running times of ASPEST comparable to, if not competitive with, the baselines under similar resource conditions.
Details of Experimental Setup E
E.1
Computing Infrastructure and Runtime
We run all experiments with TensorFlow 2.0 NVIDIA A100 GPUs in the Debian GNU/Linux 10 system. We report the total runtime of the proposed method ASPEST each dataset in Table G Note that in implementation, we train models in the ensemble sequentially. However, it is possible to train models in the ensemble in parallel, which can significantly reduce the runtime. With the optimal implementation, the inference latency of the ensemble can be as low as the inference latency of a single model.
Dataset
Total Runtime
MNIST-SVHN
24 min
CIFAR-10-CCIIC-II
1 hour
FMoW
2 hour 48 min
Amazon Review
1 hour 34 min
DomainNet (R+C)
2 hours 10 min
DomainNet (R+P)
1 hour 45 min
DomainNet (R-+S)
1 hour 51 min
Outo
18 min
Table 6: The runtime of ASPEST when the labeling budget M = 500. We use the default hyper-parameters for ASPEST described in Section
E.2 Datasets
We describe the datasets used below. For all image datasets, we normalize the range of pixel values to [0,1].
The source training dataset D is MNIST LeCun 1998) while the target test dataset Ux is SVHN NNelzer et al. 2011). MNIST consists 28 grayscale images handwritten digits, containing in total 5,500 training images and 1,000 test images. We resize each image be 32x32 resolution and change them to be colored. We use the training set of MNIST as Dir and the test set of MNIST the source validation dataset. SVHN consists 32x32 colored images of digits obtained from house numbers in Google Street View images. The training set has 73,257 images and the test set has 26,032 images. We use the test set of SVHN as Ux.
CIFAR-10 CINIC-10. The source training dataset Dtr is CIFAR-10 Krizhevsky et al. 2009) while the target test dataset Ux is CINTC-10 (Darlow al. 2018). CIFAR-10 consists 32x32 colored images with ten classes (dogs, frogs, ships, trucks, etc.), each consisting of 5,000 training images and 1,000 test images. We use the training set of CIFAR-10 as D'r and the Lest set of CIFAR-10 as the source validation dataset. During training, we apply random horizontal flipping and random cropping with padding data augmentations to the training images. CINIC-10 is an extension of CIFAR-10 via the addition of downsampled ImageNet images. CINIC-10 has a total of 270,000 images equally split into training, validation, and test. In each subset. (90,000 images) there are ten classes (identical to CIFAR-10 classes). There are 9,000 images per class per subset. We use a subset of the CINIC-10 test set containing 30,000 images as Ux.
FMoW. We use the FMOW-WILDS dataset from Koh et al. 2021). FMoW-wilds is based on the Functional Map of the World dataset (Christie al. 2018), which collected and categorized high-resolution satellite
23

---

Published in Transactions on Machine Learning Research (03/2024)
images from over 200 countries based the functional purpose of the buildings or land in the image, over the years 2002-2018 The task is multi-class classification, where the input x is an RGB satellite image, the label y is one of 62 building or land use categories, and the domain d represents both the year the image was taken as well as its goographical region (Africa, the Americas, Oceania, Asia, or Europe). The FMoW dataset encompasses both temporal and geographical distribution shifts; covering diverse regions across various continents over a span of 16 years. The training set contains 76,863 images from the years 2002-2013. The In-Distribution (ID) validation set contains 11,483 images from the years 2002-2013. The OOD test set contains 22,108 images from the years 2016-2018. We resize each image to be 96 resolution to save computational cost. We use the training set as Der and the ID validation set as the source validation dataset. During training, we apply random horizontal flipping and random cropping with padding data augmentations to the training images. We use the OOD test set as Ux.
Amazon Review. We use the Amazon Review WILDS dataset from Koh et al.; 2021). The dataset comprises 539,502 customer reviews on Amazon taken from the Amazon Reviews datasct (Ni et al. 2019). The task is multi-class sentiment classification, where the input. is the text of a review, the label y is corresponding star rating from to 5, and the domain d is the identifier of the reviewer who wrote the revicw. The issuc of distribution shifts in the Amazon Review dataset stems from the diverse writing styles and perspectives of different reviewers. The training set contains 245,502 reviews from 1,252 reviewers. The In-Distribution (TD) validation set contains 46,950 reviews from 626 of the 1,252 reviewers in the training set. The tutt----istributionn (OOD) test set contains 100,050 reviews from another set of 1,334 reviewers, distinct from those of the training set. We use the training set as Dtr and the ID validation set as the source validation dataset. We use subset of the OOD test set containing 22,500 reviews from 300 reviewers Ux.
DomainNet. DomainNet (Peng et al. 2019 is a dataset of common objects in six different domains. All domains include 345 categories (classes) of objects such as Bracelet, plane, bird, and cello. We use five domains from DomainNet including: (1) Real: photos and real world images. The training set from the Real domain has 120,906 images while the test set has 52,041 images; (2) Clipart: a collection of clipart images. The training set from the Clipart domain has 33,525 images while the test set has 14,604 images; (3) Sketch: sketches of specific objects. The training set from the Sketch has 48,212 images while the test set has 20,916 images; (4) Painting: artistic depictions of objects in the form of paintings. The training set from the Painting domain has 50,416 images while the test set has 21,850 images. (5) Infograph: infographic images with specific objects. The training set from the Infograph domain has 36,023 images while the test set has 15,582 images. We resize each image from all domains to be 96x96 resolution to save computational cost. We use the training set from the Real domain as Die and the test set from the Real domain as the source validation dataset. During training, we apply random horizontal flipping and random cropping with padding data augmentations to the training images. We use the test sets from three domains Clipart, Sketch, and Painting as three different Ux for evaluation. So evaluate three shifts: Real->Clipart (R+CC), Real->Sketch (R-S), and Real Real-PPintinn (R-P)). We use the remaining shift Real Real-sInfograph (R-T)) as a validation dataset for tuning the
Otto. The Otto Group Product Classification Challenge (Benjamin Bossan 2015 is a tabular dataset hosted on Kaggle P The task is to classify each product with 93 features into 9 categories. Each target category represents one of the most important product categories (like fashion, electronics, etc). It contains 61,878 training data points. Since it only provides labels for the training data, we need to create the training, validation and test set. To create a test set that is from different distribution than the training set, we apply the Local Outlier Factor (LOF) (Breunig et al. 2000). which is an unsupervised outlier detection method, on the Otto training data to identify a certain fraction (e.g., 0.2) of outliers as the test set. Specifically, we apply the FocalOutlieruuctor function provided by scikit-learn (Pedregosa et al.) 2011) on the training data with a contamination of 0.2 (contamination value determines the proportion of outliers in the data set) to identify the outliers. We identify 12, 376 outlier data points and use them as the test set Ux. We then randomly split the remaining data. into training set Dir with 43, 314 data points and source validation set with 6, 188 data points. We show that the test sct indeed has a distribution shift compared to the source
24

---

Published in Transactions on Machine Learning Research (03/2024)
validation set, which causes the model trained on the training set to have drop in performance (see Table 7 in Appendix F.1
E.3 Details on Model Architectures and Training on Source Data
On all datasots, we use the following supervised training objective for training models on the source training set Dtr;
min B(xy)e (cE(x,y;0)
(32)
where fc is the cross-entropy loss and is the model parameters.
On MNIST-SVHNN, we use the Convolutional Neural Network (CNN) (Leunn et al. 1989) consisting of four convolutional layers followed by two fully connected layers with batch normalization and dropout layers. We train the model on the training set of MNIST for 20 epochs using the Adam optimizer (Kingmaa & Ba 2014h with learning rate of 10-3 and a batch size of 128.
On MIPAR-1O-CIIICCE training set CIFAR-10 for 200 epochs using the SGD optimizer with a learning rate of 0.1, a momentum of 0.9, and a batch size of 128. The learning rate is multiplied by 0.1 at the 80, 120, and 160 epochs, respectively, and is multiplied by 0.5 at the 180 epoch.
we use the ResNet-20 network (He et al., 2016b). We train the model on the
On the FMoW dataset, we use the N22 network (Huang et al. 2017b) pre-trained on ImageNet. We train the model further for 50 epochs using the Adam optimizer with a learning rate of 10-4 and batch size of 128.
On the Amazon Review dataset, we use the pre-trained RoBERTa base model (Liu et al. 2019 to extract the embedding of the input sentence for classification (i.e., RoBERTa's output for the [CLS token) and then build an eight-layer fully connected neural network (also known as a multi-layer perceptron) with batch normalization, dropout layers and T.2 regularization on top of the embedding. Note that we only update the parameters of the fully connected neural network without updating the parameters of the pre- trained RoBERTa base model during training (i.e., freeze the parameters of the RoBERTa base model during training). We train the model for 200 epochs using the Adam optimizer with learning rate of 10-3 and a batch size of 128.
On the NomainNet dataset, we use the ResNel-50 network (He el al. 2016a) pre-trained on ImageNet. We train the model further on the training from the Real domain for 50 epochs using the Adam optimizer with learning rate of 10-4 and batch size of 128.
On the Otto dataset, we use six-layer fully connected neural network (also known multi-layer percep- tron) with batch normalization, dropout layers and L2 regularization. We train the model on the created training set for 200 epochs using the Adam optimizer with a learning rate of 10-3 and a batch size of 128.
E.4 Active learning hyper-parament parameters
During the active learning process, we fine-tune the model the selected labeled test data. During fine-tuning, we don't apply any data augmentation to the test data. We use the same fine-tuning hyper- parameters for different methods to ensure fair comparison. The optimizer used is the same that in the source training stage (described in Appendix E.3 On -SIST+ we use a learning rale of 10-3, On CIFAR-1O-CCIIICII we use a learning rate of 5 x 10-3, On FMoW, we use a learning rate 10 4, On Amazon Review, we use a learning rate of 10-3; On DomainNet, we use learning rate of 10-4; On Otto, we use a learning rate of 10 On all datasets, we fine-tune the model for at least 50 epochs and up to 200 epochs with batch size of 128 and early stopping using 10 patient epochs.
25

---

Published in Transactions on Machine Learning Research (03/2024)
F Additional Experimental Results
F.1 Evaluate Source-Trained Models
In this section, we evaluate the accuracy of the sourco-trained models the source validation dataset and the target test dataset Ux- The models are trained the source training set D (refer to Appendix E.3 for the details of source training). The source validation data randomly sampled from the training data distribution while the target test data are sampled from different distribution than the training data distribution. The results Table show that the models trained on D always suffer a drop in accuracy when evaluating them on the larget lest datasel Ux.
Dataset
Source Accuracy
Target Accuracy
SANISTSVVIS
99.40
24.68
CEAAR-OO-CINI-JU
90.46
71.05
FMoW
46.25
38.01
Amazon Review
65.39
61.40
DemainNet (R-CC)
63.45
33.37
DomainNet (R~P)
63.45
26.29
DomainNet (R-s)
63.45
16.00
Otto
80.72
66.09
Table 7: Results of evaluating the accuracy of the source-trained models the source validation dataset and the target test dataset Ux. All numbers are percentages.
F.2 Evaluate Softmax Response with Various Active Learning Methods
To whether combining existing selective prediction and active learning approaches could solve the active selective prediction problem, evaluate the existing selective prediction method Softmax Response (SR) with active learning methods based uncertainty diversity. The results in Table g show that the methods based on uncertainty sampling (SR++Cnffieence, SR+ Entropy and MRaMarggn) achieve relatively high accuracy of ʃ, but suffer from the overconfidence issue (i.e., mis-classification with high confidence). The method based on diversity sampling (SR+kCG) doesn't have the overconfidence issue, but suffers from low accuracy f. Also, the hybrid methods based on uncertainty and diversity sampling (SR+CLUE and SR+BADGE) still suffer from the overconfidence issue. In contrast, the proposed method ASPEST achieves much higher accuracy of f, effectively alleviates the overconfidence issue, and significantly improves the selective prediction performance.
Method
Accuracy of f
Overconfidence ratio 4
AUACC1
Confidencc
45.29=3.39
16.91+2.22
64.1442.83
Entropy
46.38
36.84+18.96
65.88+4.74
SR+Margin
58.10+00.55
13.18+1.85
76.79++0.45
SR++CGG
32.6843.87
0.0410.01
48.8317.21
SR+CLUE
55.22+2.27
9.47
73.15
SR+BADGE
56.55+11.62
8.37+2..66
76.06+1.66
ASPEST (ours)
71.82+1.49
0.10+0.02
88.84+1.02
Table 8: Evaluating the Softmax Response (SR) method with varions active learning methods and the proposed ASPEST on MNI.T-SVHN, The experimental setup describe Section 5.1 The labeling budget 100. The overconfidence ratio is the ratio of mis- classiaasi unlabeled test inputs that have confidence 2 1 (the highest confidence). The mean and std each metric over three random runs are reported (meantstd). All numbers are percentages. Bold numbers are superior results.
26

---

Published in Transactions on Machine Learning Research (03/2024)
F.3 Complete Evaluation Results
We give complete experimental results for the baselines and the proposed method ASPEST on all dalasets in this section. We repeat each experiment three times with different random seeds and report the mean and standard deviation (std) values. These results are shown in Table 9 (MNIST-SVHN), Table 10 (CIFAR-
10 10-+CINIC-10), Table 11 (FMoW), Table 12 (Amazon Review), Table 13 (DomainNet R R-C)), Table 14
(Domain R->P), Table 15 (DomainNet R R- and Table To (Outo). Our results show that the proposed method ASPEST consistently outperforms the baselines across different image, text and structured datasets.
Dataset
-SHN
Metric
cot c00/000 2: 9.00
000
90% f
AUACC f
Labeling Budget
100
500
1000
100
500
1000
100
500
1000
SR+Uniform
0.00+0.00
51.46++3.77
73.57+00.9
58.03+1.5
76.69++112
84.39+0.2
74.08+1.5
88.80+0.8
93.57=0.2
SR+Confdence
0.00+0.0
55.3
82.22+1.3
47.66+33.4
79.02+0077
87.19+0.4
64.14++2.8
89.93+0066
94.62+0.2
SR+ Entropy
0.0010.0
0.00=0.0
75.08:2.4
17.9347.0
77.0941.00
81812007
65.88:1-.7
88, 19.0.8
93.37=0.5
SR+Margin
0.00+1..0
63.60:1:2.7
82. 09:1..3.33
61.39:10.5
80.961009
86.9720.2
76.79:10.5
91.24:0.5
94.82=0.1
SR+LOG
2.52+13
23.04::0.3
38.97:42.6
34.5744444
52.76:11.1
64.342488
48.83:7722
73. 73.65:1100
83.16=2.0
SR+CLUE
0.00++000
62.03::2.4
81. 11.19::111
57.35++1.9
79.55+00.8
06.28=005
72.72+11.9
90.98:0.5
94.99=0.2
SR BADGE
0.00::0.0
62.55:1444
82.30:12.8
50.82:11.7
79.49:11.6
06.96-209
76.06:1166
91.00:10.9
95.16=0.6
DE+Uniform
24.71+5.6
68.98++1.6
83.67=0.11
63.22++1.77
81.67+0.4
87 07.
1..36++1.7
92.47=0.2
95, 95.48=0.0
DE+Entropy
6.24+8.8
63.30++6.5
8..62++1.5
56.61+006
80.16+:2.0
08.05=0.5
72.51++1.5
91.21++1.4
0.5
DR+Condieenee
14.92+5.1
6.,87+1.4
89.41+0033
61.11+2.9
81.80+0.55
8..75-+0.1
75.85++3.0
9.216++02
96.19+0.11
MarMar
21.5913.8
77,8412.8
92.7520.3
6..881112
85.11::1.1
91.1720.1
78.59:1.4
94.3110066
97 97.00=0.0
Avg-LL
10.9844.66
61.4533.4
88 08.00:2222
14.80:1::6
78.21#11.
89 239
71.67::2.2
90.92:10.8
96. 06.23=0.44
DE+CLUE
1..34+1.4
69.23:11.9
1..0
59.47:11.3
81.04:099
86.78 06.78:0.4
76.88:11.00
92. 92.70:10.5
95.5620.2
DE+BADGE
2..02++.5
72.31+1.2
88 0.423:0044
61.23++1.9
82.69+0.5
89.15=002
77.65+t:1.9
93.38+0.2
96.51=0.1
ASPEST (ours)
52.10=4400
89.22++0.9
98.70=0.4
1..10=
89.62=0.4
93.92=0.3
88.84=100
96.62=0.2
98.06+001
Table 9: Results of comparing ASPEST to the baselines on MNITT-SVHN. The mean and std of each metric over three random runs are reported (mean=tstd). All numbers are percentages. Bold numbers are superior results.
Datasct
CIFAR-10 CIIIII100
Metric
ozn co//ace
Iooe 2 99%
aoc/ 2 90% T
AUACC T
Laheling Budget
500
1000
2000
500
1000
2000
500
1000
2000
SR+ UUifform
57.4310.2
57.151066
58.3710.7
75.6710.2
75.6010.1
01.33
59.77:0.0
89.8110.1
90.09-0.2
SR+ BR-Conndeeeee
57.9610.6
57.0510.7
61.11111
76.1010.2
76.8740.2
78.77=0.4
90.00:10.2
89.9210.2
90.9120.3
Enttrop
57.78+0.7
57.0741.4
61.074044
76.3740.3
76.7140.5
78.85-02
90.01+0022
89.9440.3
90.88+0.0
SR++ Margin
57.72+0.8
57, 7
61.7140.2
76.24:102
76.9040.2
78. 0.2
89.95-10.2
90. 14:0.1
91.02-0.0
SR kCG
57,90.1.55
57 77.8
60.36:0.3
75.59-10.1
75.73:10.
76.68:002
89.78:10.1
89. 99.70.0.22
90.41:0.2
SR: CLUE
57.29.1055
55. 35.89:10.5
62.28:1.2
75.74:0.2
76.68:10.3
78. 0.22
89.674:0.2
90. 90.15:10.1
91 01.03:0.11
3R+BADDE
58.58+006
58.63++0.3
61.95+0.4
76.33+005
76.58+00.1
78.26
90.05++0.2
90. 90.16+0.1
90. 90.99-0.0
DE Uniformm
58.0610.33
58.72400.1
59.5110.3
76.65:1.11
77.0610.2
77.4620.1
90.26:10.1
90. 10.45.0.1
90.73-0.1
DE-Entropy
58.9110.6
60.961:0.
63.85:0.2
77.6010.1
79.14:0.1
80.82:0.2
90.55:00.1
01.1640.1
91.00000
DEuCiddee
58.5310.3
61.03:00.6
61:42:1022
77.7310.2
79.00:10.1
80.87=0.0
90.53:1000
91. 91.1110.1
91.9620.0
Margin
58.76+005
61. +..
64.92+0.5
77.61+0:2
78.91400.
80.59+0.1
90.56++0.1
91.1140.1
91. 91.98+0.1
DE+ Avg-KLD
50.99-006
62.05.10.3
65.02-005
77.84
79.15:40.0
81.04=0.1
90.74:10.1
91 91.3040.1
92, 02.10
D CLUE
59.27:40.1
66:104
64.42:00.0
77.19:1011
78.3740.2
79. 0.1
90.44:10.1
91.03:10.1 1
91.74-0.0
DE+ BADGE
59.3740.1
61.611:.1
6.1.33:0.44
77.13:1.11
78.33:002
79. 03.43.033
90.49:10.1
91.12:10.0
91.78.0011
ASPEST (ours)
60.38:10.3
63.3441.2
66.81=0.3
78.23:10.1
79.4940.1
81.25=0011
90.95-10.0
91.60-0.0
92.33+10.1
Table 10: Results of comparing ASPEST to the baselines on CIPAR-IO--CINICII,
The mean and std of each
metric over three random runs are reported (mean-sstd). All numbers are percentages. Bold numbers are superior results.
F.4 Effect of combining selective prediction with active learning
Selective prediction without active learning corresponds to the case where the labeling budget M = 0 and the selectod set B* = 0. To make fair comparisons with selective prediction mothods without active learning, we define a new coverage metric:
cov" cov" (fa, = Ex~Ux l[o(x) 2 T AA $ Br
(33)
27

---

Published in Transactions on Machine Learning Research (03/2024)
Dataset
FMoW
Meuric
100e 2 70%
600eow 1000
70% T
AUACC
Laheling Budget
500
1000
2000
500
2000
500
1000
2000
SR: -Uniform
08,50.207
42.00
12.00:20.5
52.31:1.1
51.76:10.7
54,277:0.2
60.31:0.7
65.75:10.4
67.67.00.3
02.73:0.3
SR Coubieence
07.3140.3
12.2841.2
53.7240.7
52.24:10.1
55.5240.3
61.7620.4
65.5740.1
68. 03:0.5
73. 03.14:0.5
SR+ Entropy
37.42+0.3
42.08 +0.2
51.18+0.4
51.74+0.4
54.94+0.2
60.62+0.2
65.31+002
68.00+0.11
71.99+0.2
SR+ Margin
38, 18.40+1.4
44.67 40.7
15.68+1.5
0..8
66.66-00.4
62.
66.114-0.6
69. 69.12-40.4
0..
SR+ ROG
06.50---.8
39.76411.2
45.87--0.6
49.36
51.45:40.3
0..47=001
64.34:40.55
66.21 06.21:400 +006
69.00 63.63-022
SR: CLUE
38.65--0.7
44.50-11.8
54.71:00.5
52.23:40.4
55,54411.0
01.13:0.4
65,784-0,3
68. 76:20.99
73.80=0011
SR BADGE
10.474:..5
45.65:1.2
57.59=0.4
53.0841.0
56.634:03
63.57=0.2
66.74::0.8
69.43:0.6
74.7620.2
DE + Uniform
04.7440.4
51.3741.1
61.92:0.4
06.39:10.5
60.01:0.3
65.74
69.44:10.3
72. 02.48.0.55
77 77.02=0.1
DE + Entropy
13.70 0.,
50.5241.
62.73:0.4
06.29:10.3
60.3140.3
66.53:0022
69.02:40.1
72. 10:10.5
76.00 06.6522.2
DE + Confidencc
05.2340.6
50.113:0.9
641.99:0.33
57, 57.18:1044
60. 00.46:3033
67 0.000.
69.80:10.3
72. 11:04.
77 77.3720.1
DE Margia
06,3520.6
54..9
69.70:20.8
57.84:10.3
62. 0.:353
69 09.87:0.4
08:18:00.3
73 73.62.0.33
78.8820.4
DE+ Avg-KLD
06.29+-0.
53.63+-0.8
68, 0.9
57.75-+0.4
61 60++0.3
09.11=00.2
70.16 00.16:+00.1
73.09-4022
08.48=0.3
CLUE
4.
49.97:40.3
08.06--0.55
56.39--0.1
59.05440.1
63.23=0044
69.53440.0
71. 71.96440.1
75.72=0.3
DE BADGE
47.3940.7
53,8340077
06.45:20.88
57.71+0.4
61. 16:40.2
68. 13:0.4
70.59:0044
73. 40.4000033
78.66=001
ASPEST (ours)
53.05:10.4
59.86:10.4
76.52-0.6
61.18:40.2
65.18:0.2
72.86-0033
71.12:0.2
74.25-102
79.93:10.1
Table 11: Results of comparing ASPEST to the baselines on FMo.. The mean and std of each metric over three random runs are reported (tt).. All numbers are percentages. Bold numbers are superior results.
Dataset
Amazon Review
Metric
coo jace >> 80% 1
ace >> 80% 1
AUACC 1
Labeling Budget
500
1000
2000
500
1000
2000
500
1000
2000
SRtUoiffrmm
13.71411.33
24.10+3.3
24.87::2.6
65.13:40.8
66.33+0.6
06.26-003
1..71+1.5
73.6440.7
73.53+0.7
SR Confidenne
11.2848.9
17.96:4..0
33.10:1.4
65.15:10.7
66.29+0.4
68.
72.89+0.7
73.25+1077
76.17:0.2
SR& Entropy
5,5547.8
13.32-9.5
25.47:1.88
65.11:1111
66.5620.7
6.7
71.962166
72.5341.1
74.19:10.5
Margin
14484410.9
22.61:4.2
28.35:661
65.75:0.5
66.31
68.15:0.4
73.25=100
73.65::0.5
75.17:0.8
SR++CG
20.02+11.0
17.02+12.2
29.08+44.2
64.08++311
66. 1.5
66.63+1.0 1.0
72.34+3.2
74.35++077
74.49+1.0
SR+CCLUE
4.15+15.9
25.15+4.9
31.88-2.1
66.1740.4
66.30+0.4
67.12-0.7
73.43++0.4
74.07+0.7
73.29+0.9
SR++BADGH
22.5840.4
23.78:6.4
30.71444.6
66.29=004
66.31
68.58-007
73.80
74.00+E1.0
75.76-40.8
DE+Unfform
31.35214
33.1521.1
36
16.561188
68. 18.1111..4
08, 02.10:10.6
68, 0..8
76. 20.00
76.1610:4
77.07=0.3
DE+Entropy
31.7+1.4
36.20
1.629=116
10.33:11.7
18.1933..
09. 0.2
0..27=0.3
75.9820.4
77.10110.3
78.53=00.3
DE+Conlidence
15.12+188
34.48=144
00.46:0055
69.07=003
69. 09.47:10.2
71.08=0.2
0..63:0.2
76.87400.3
78.27=0.1
Da+Margin
33.42+1.3
35.03=113
41.20:10.4
68.45:20.3
09.30++02
70.88=0.1
0..1
76.911-0.3
78.31=0011
Drg-Ag-KEA KLD
33.03+1.5
38.55+332
1..75++1.8
68.63+003
69.95++0.4
71.10+0.3
76.21
77.62++0.6
78.62++0.3
DE+CLUE
33.9243.0
35.27+1.4
34.83++3.1
68 08.09
68.07+00.3
68.40+0.6
76.27+0.6
76.65+H0.3
76.69+0.7
DE+BADGE
32.23:3.7
36.18:1.5
10.58:13.3
08.31:10.4
08.8740.2
00.
76.1320.7
77.09110.2
78.44:0.5
ASPEST (ours)
38.4410.7
40.96+0.8
45.77+0.1
69.31
71.60-03 70.17+0.2
77.69+0.1
78.35+00.2
7..51+U.2
Table 12: Results of comparing ASPEST to the baselines on Amazon Review. The mean and std of each metric over three random runs are reported (mtdn All numbers are percentages. Bold numbers are superior results.
The range of cos"(fs, is [0,1 - 11 where M = [3 and n = |Ux|. If we use a larger labeling budget M for active learning, then the upper bound of cou' (fs. will be smaller. Thus, in order to beat selective classification methods without active learning, active selective prediction methods need to use a small labeling budget to achieve significant accuracy and coverage improvement. We still use the accuracy metric defined in
We then define new maximum coverage at a target accuracy to metric as: max +
(Jss( s.t. acc(fss T) 2 ta
(34)
We denote this metric as cow"lacs 2 tu. We also mcasure the accuracy of f on the remaining unlabeled test data for different approaches.
The results under these metrics are shown in Table 3 (MNIST -SHHS,-SVHN), Table 17 (CIFAR-1O-CCINIC and Otto), Table 18 (FMoW and Amazon Review) and Table 19 (DomainNet). The results show that the coverage achieved by the selective prediction baselines SR and DE is usually low and the active learning baselines Margin, kCG, CLUE and BADGE fail to achieve the target accuracy with a labeling budget of 1K.
28

---

Published in Transactions on Machine Learning Research (03/2024)
Datasct
DomainNet R +C (casy)
Meuric
1000 2 80%
a00 con 80% 7
AUACC
Laheling Budget
500
1000
2000
500
1000
2000
500
1000
2000
SR: +Uniform
26.56:20.6
27 0..88
29.86:20.0
43.63:20.4
45.5740.3
47.27=0.4
63,31:0.4
65. 65.11:0.5
66. 70-00.22
SR Coubieence
26,96:1.2
27.80:11.2
32.51=1.3
04.90:10.8
17 17.26:0.44
52.0420.8
641,00:0..6
65. 65.88:0.6
69.70 00.777..7
SR+ Entropy
25.44+1.0
27.49++0.4
33.51+1.1
44.46+0.7
40.3
52.25+00.5
63.52+00.6
65.72+0.2
70.03+0.5
SR+ Margin
26.28+1.2 2
27.7741.0
32.92-0.4
45.2441.0
47. 47.12400.7
52.29-00.4
64.377-0.8
63.97+00.6
70.01 +0.4
21.12440.3
79.-04
23.43:0..5
39. 19.40.1
40.504-0.4
41.11
58,88 55,88400.00
60.11=0.4
60.89=001
SR: CLUE
27.17400.8
29.78400.8
3-182200.6
44.57400.7
46.794:0.1
09.70=0.3
64,38::0066
66.4740.3
69. 09.59=0.1
SR BADCE
27.7840.8
30.78:0.6
36.00
36.00:0..6
45.36:0066
48. 18.43:6
53.00=0.4
64.90::0.5
67.5640.4
71.39=0.4
DE + Uniform
00.82+0.8
33.05::0.4
86.80:2022
48. 19:10.3
50.09:20.3
0..98:0.5
67 00:40.
69.31:40.3
71.61:0.4
DE + Entropy
29.1340.9
34.07:00.3
00.32.0.3
45 15.6740.4
51 66:00.2
57 0..2
67. 0.3333
70 00.02....2
74.61:0.2
DE + Confidence
09.90:0.88
33.73:10.2
00.80:002
15.60:0033
52.03:0.3
58. 03.10.1
67. 67.45:0.3
70. 19:10.2
74.80=0.1
DE DE+Margan Margin
3..82+1.
35 35.68:10.2
43.39:2077
50. 12:
19.19.4.4
59. 09.17=0.2
68, 08.85:00.4
71.29:00.3
76. 0.3
DE+ Avg-KLD
32,23+0,2
36.03 36.00---0.6
0..00+00.5
49.81++0.3
+-.3
58 0.1
68,73++0.2
71 01:40
76.73+0.2
+LUE
30.80+0033
33.04::0.4
35.52:
08.56:4003
49.914-0.3
51, 0:24
67.82:00
69. 10:40.2
70.62=0.2
DE BADGE
10.1641.3
36, 18::0.3
03.34:0.3
09.78:10.3
53.26:40.1
58 0.4
08,46:1:033
71.35:0022
76.37=0.3
ASPEST (ours)
37.38:101
39.98:10.3
48.29-1.0
54.56:10.3
56.95:1.1
62.69-002
71.61:0.2
73.27:0.2
77.40:004
Table 13: Results of comparing ASPEST to the baselines on DomainNet R-C. The mean and std of each metric over three random runs are reported (ttan-tstd). All numbers are percentages. Bold numbers are superior results.
Datasct
DomainNet R +P (medium)
Metric
cou pace 2 70%
ace cov 70% T
AUACC T
Labeling Budget
500
1000
2000
500
1000
2000
500
1000
2000
3R+Uniformm
21.01+1.0
21.35++0..
22.64++0.5
36.78+0066
37.18+0.2
38.20+0.4
51.87+00.7
52.31
53.34+0.4
SR CCnnbdence
20.64:0.6
25.00.
23.60::0.6
37.01:0.3
38. 46:0077
40.23 40.23:0.4
51.7740.3
53.33.10.8
54.8020.5
SR+ Entropy
00.76.0.7
22. 11:00.3
23. 03.56.0.3
37 0..
38.38:10.3
40.30=0.11
51.66:044
53.29
54.8120.2
SR+ Margin
21.43+0.4
23.29:1:..3
2..70=1100
37.21:10.2
39. 19:40.0.44
40.81=0.4
52.33:10.1
54.09:10.3
35.70=0.4
SR+LOG
17.33+0.4 4
17.62-40.2
18. 0.2
33.97:+0.3
34.12:00.1
0..
48.61:+0.5
48 0..6
49.25:2022
SR CLUE
21.15:0.66
22.40-00.5
218410077
36.96:10.2
37.9340.3
39.31=004
51.974
53.2 53.20:20.5
04.8420.5
SR+ BADGE
20.0740.3
22.211:0.5
21.22:0.2
36, 10:10.1
38, 38.1140.4
40. 0..550
50.99:10.0
53. 33.1040.4
55. 00.44.2.4
DE+Uniform
25.42+0.2
26.38:10022
28.83+0.3
40.83:+001
41.66:40.2
43.93:0022
55.86:+0.11
56.62++0.1
58. 0.22
Datrotr
25.74+0.4
27.11++0.4
30.39+0.1
41.34+0.1
42.92+0.3
45.92-003
56.06-+0.2
57.51+0..3
60.10+0.2
Confidence
26.6910.4
27.38.107
30.47.0.1
41.45.0.2
13.12.1033
45 05.8820.1
56. 13:10.2
57 08.333333
60.20:0.2
Margan
25.78
27.88:10.5
31.03:0.4
41.26:10.2
13:13:3
06.23:20.4
56. 06.23:10.2
57 57.90:10.3
60. 60.49-0.3
Avg-vg-LLD KLD
26.30+10.7
28.00:1:.11
3..974022
41 01.80:0033
43.1740.11
46.32=0.2
56. 56.65:40033
57 0..
60.8 00.82:0.2
DE+CLUE
25.38++066
26.65:40.44
27 0..89:0011
40.86:+0.3
41. 11.62:+0.2
42. 0.1
55.79:+0.4
56.65
37.71=0011
DE+BADGE
26.27+007
27.60:00.1
31:202
42.02:00.6
43.414:.2
16.
56.67:00.5
58 03:0..1
60.8120.1
ASPEST (ours)
29.69:1011
32.50+0.3
35.4660.6
44.96=0.1
46.77=0.2
49.42=0.1
58.74=0.0
60.36=0.0
62.84+0.2
The mean and std of each metric over three random runs are reported (meantsstd). All numbers are percentages. Bold numbers are superior results.
Table 14: Results of comparing ASPEST to the baselines on DomainNet R-P.
In contrast, the proposed active selective prediction method ASPEST achieves the target accuracy with a much higher coverage.
F.5 Effect of joint training
In the problem setup, we assume that we have access to the training dataset Dir and can use joint training to improve selective prediction performance. In this section, we perform experiments to study the effect of joint training and the effect of the loss coefficient À when performing joint training. We consider three active selective prediction methods: +Rargng (Algorithm 1 with margin sampling), DE+ (Algorithm 2
with margin sampling), and ASPEST (Algorithm 3). We consider X E (0, 0.5, 1.0, 2.0}. When À = 0, we don't use joint training; when À > 0, we use joint training. The results are shown in Table 20 From the results, we can see that using joint training (i.c., when À > 0) can improve performance, especially when the labeling budget is small. Also, setting a too large value for X (e.g., X 2) will lead to worse performance. Setting x= = 0.5 or usually leads to better performance. In our experiments, we simply set X by default.
29

---

Published in Transactions on Machine Learning Research (03/2024)
Dataset
DomainNet R +S (hard)
Meuric
co0/ 2 70%
a00 con 70% 7
AUACC
Labeling Budget
500
1000
2000
500
1000
2000
500
1000
2000
SR: -Unifor
12.1240.77
12.424004
15.88:0.2
27.01:10.6
27.7 37.74:30.3
0.3
11,12:00.8
41.80:0.2
46. 16.1720.3
SR Coubieence
11.06:1.1
11.4840.5
14,4941.5
16.5341144
27. 0..22.
31.7
10.26:1166
41.66::0.2
45. 15.46:111
SR+ Entropy
+0.91+0.3
12.45
14.66+006
26.84+00.5
28.72+0.5
31.07+0.6
40.47+00.6
42.61 +0.88
45.31 +0.4
SR+ Margin
12,23+0.4
13.06
15.31+0.4
27.87+00.2
29.19+40.4
31.51 +0.8
41 41.91400.3
43.2240.4
45.97+00.8
SR+ ROG
9.03+-02
9.76+002
11.4140,2
23.32:40.4
24.06440.4
25 03.68:0.4
36.63440.3
37.3740.4
39 00,80-033
SR: CLUE
02.3940.33
14.1741.0
15,8040.88
27 27.8240
29.6 29.68400.4
30 0..6220.8
42.00:40.4
44.19.00.7
45. 45.58=0.9
SR BADGE
12.1840.9
13.1341.
15.8340.7
27,6841.00
28,964307
32 0..00::.4
11.7241.11
43.28:0.9
46. 0.000
DE + Uniform
05,91+0.5
17.55.0044
21:33:0.3
31.37:00.5
32.3740.4
36. 02:12:2022
06.28:10.5
47.7 47.70:0.4
51.64:0.2
DE + Entropy
03.70.0.33
16.3110.55
19.58:0.4
30.38:0044
32,45 02.452222
36. 18:2022
04.79:00.5
47 15:0..
00.8720.3
DE Confideecc
13.73+0.2
16.211:0.2
19.22:0.4
30,55.0.3
33.02
06.29-00.5
15.05:10.3
47 17.00.0.0
50.8120.4
DE Margin
14,99:0.2
17.4540.4
21.74.0.7
31.67:0055
33.5140.5
37 0.3
06.38:20.5
48. 44.0.5
52. 02.2240.4
DE+ Neg-KLD
15.75++0.5
18.144+07
0..35-00.3
31 0..66-00.
33.79--0.2
37 37.96-0022
46. 26.49.-001
48.7740.3
53 03.02-003
DE+CLUE
14.76++0.5
17,38
19.75440.4
31 05.40044
32.58-40.2
34.61 0..4
15.80--0.3
47.74:4.1
50.09=002
DE DBTDBA BADGE
14.9740.1
17.4941033
21.71:0.3
31 0,2
33.46440.1
37 0.33
16.034:0.1
48.31:2011
52.33=0.2
ASPEST (ours)
17.86:10.4
20.42:10.4
25.87-0.4
35. 35.17:0.11
37.28.0.3
41.46
49.62:0.1
51.61-0.4
55.90:102
The mean and std of each metric over three random runs are reported (mean-tstd). All numbers are percentages. Bold numbers are superior results.
Table 15: Results of comparing ASPEST to the baselines DomainNet R-S.
Dataset
Otto
Metric
con/ar: por: 2 80%
000
80%
AUACC 1
Labeling Budget
500
1000
2000
1000
2000
500
1000
2000
SR+ Uniform
03.5840.7
64.00::0.4
67.49-0.9
73.50:00.3
73.5740.6
75.21
$4.46::002
84.61=0033
85.72-0.2
SR+ FR-Coniidecee
1.763+1.7
73.4140.6
84.19:0.5
75.96:0..5
77.574002
81.39=0.2
$5.91:10.3
86.8020.11
88.93=0.1
SR+ Entropy
67.79410.8
73.834110
83.124077
75.43:10.4
77.9140.3
81.07:02
85.41:40.3
86 96.9440.22
88 06.
SR Margin
68. 10:10.1.
74. 14.04.
82.53:0.2
75.52:00.0
77 77.6640.1
80.93:001
85,56:10.1
86 06.9940.1
88 03:
SR KCG
61.8410.7
12.90:1..
59.85:110
73.75.0.33
73.0340.2
71.902033
85.0810.2
84.6740.2
83.79=0.33
SR+CLUE
68.21+1.2
70.85:00.6
78. 08.26:0.9
76.26:0.5
76.324002
79.30 00.3020.3
65.82:10.3
86 .8.22.2.2
88, 68.5320.2
SR+ BADGE
67.23+1.0
73.52-0022
83.17+0.4
74.74+0033
77.43+0.2
81.20-0.2
85.41++0.3
87.10+0.2
89.25 2.3
DE 1E+UUe Uniform
70.74:0.5
72.20100.6
75.58:0.5
76.40:10.1
77. 77.064:02
78.35 08.3520.2
86.78:10.1
87 27.60.0.
88. 11:0.1
DE+ DE+Entropy
75.71+0.3
50.914:0.2
92. 02.62:0.22
78.44:0.11
80.29::0.1
04.05
07.87400.1
88. 88.7740.1
90.99=0.1
DE+Coniieence
75.52+0.2
81.69:40.7
02.15:0099
78.28:10.1
80.49::022
83.83=0..
07.84400.1
89.05=20.1
90.98=0.1
Margin
75.49:10.8
$1.36400.8
92. 02.4940.4
78.414:0.3
80.5044:2
84.06:0.2
87:492
89. 99.10:40.2
90.95=0.2
DE i Avg-KLD
75,91:0.2
80.97410.5
91.9110.8
78.50:10.1
80,3340:2
83. 0..20.22
87.80.0.0
89. 99.06:0..1
90.98=0.1
DE CLUE
69.66:10.5
70.5240..1
70.17:0.4
76.09.10.3
76.32:1..1
76.3 76.3120.2
86.67:10.1
87.1140.0
87 07.0620.1
DE +BADGE
73,2310.2
77.89110.6
86.32:10.5
77.33110.1
79.2110.3
82. 02.22.0.2
87.5510.1
88, 76200.1
90.5820.0
ASPEST (ours)
77.85:102
84.20:0.6
94.26-00.6
79.28:0.11
81.4020.1
84.62-0.1
88.25 88.28=0.1
89.6120.1
91.499:0.0
The mean and std of each metric over three random runs are reported tst). All numbers are percentages. Bold numbers are superior results.
Table 16: Results of comparing ASPEST the baselines on Otto.
Dataset
CIFAR CINICC0-
Otto
Method
0ov" ace 2 90%
Accuracy of f
coo Jace 2 80%
Accuracy of f
SR (M-0)
43.10.0
71.05.10.0
6..90-10.0
66.0910.0
Selective Prediction
DE (M 0)
56. 56.64
71.3310.11
67.6910.4
68.12.0.3
Margin (M=1000)
N/A
72.74+0.1
N/A
70.19+0.2
kCG (M=1000)
N/A
71.16+0.11
N/A
65.88+0.2
Active Tearning
CLUE 100)
N/A
72.32+0.2
N/A
68.64+0.2
BADGE 1000)
N/A
72.11:0.11
N7A
69.80+00.4
Active Selective Prediction
ASPEST (M=1000)
61.23-00.2
74.89=0.1
77.50=0.5
74.13:0.11
Table 17: Results CIFAR-10-CCIN
and Otto for studying the effect of combining selective prediction with
active learning. The mean and std of each metric over three random runs are reported (meantstd). All numbers are percentages. Bold numbers are superior results.
30

---

Published in Transactions on Machine Learning Research (03/2024)
Dataset
FMo
Amazon Review
Method
70%
Accuracy
coo"
Jace 80%
Accuracy of f
SR (M-0)
32. 92.39-11 1-000
38.01
26. 0.0
61 61.40.0.00
Seloctive Prodiction
DE (M
37 L0.
41.0810.00
35. 11.9
63.8910.1
Margin (M=1000)
N/A
45.17+0.55
N/A
62.64+0.6
kCG (M=1000)
N/A
40.20++0.3
N/A
62.03+00.3
Active Learning
CLUE [M=1000)
N/A
43.75-+0.7
N/A
62.32+0.4
BADGE 1000)0)
N/A
44.73.10.3
N/A
62.44:0.55
Active Selective Prediction
ASPEST (M=1000)
57.1510.4
52.42#0.1
39.14=0.8
65.47:10.2
Table 18: Results on FMoW and Amazon Review for studying the effect of combining selective prediction with active learning. The and std of each metric over three random runs are reported (meantsstd). All numbers are percentages. Bold numbers are superior results.
Dalases
DomntnNet (eauyy
DomsiaNet (mudiiu))
DomminNet R-S (lued)
Method
cos" lace 80% 7
Accuracy of
00e" (acc 70%
Accuracy of
0ou" jaoc 2 70%
Accuracy of
SR (M-0)
21.000000
33.3720.0
18, 16200.0
26. 0.:
T.1660.
15,90:20.0
Selective Prediction
(M
11100.2
Margin (M-100))
N7A
39.422088
N/A
23. 20.80
N7A
22.14
Active Learning
CLUE CLUB
(M=1000)
N7A
38.07:0.1
N7A
285 28.88:005
23.02-000
N7A
22.14:003
(M=1000)
Active Selective Presietioo
ASPEST (M=1000;
37.2420.3
47.76:20.1
31.0120.3
35.69-10.1
19.45:20.3
28.16:10.3
Table 19: Results on DomainNet R-C, R-P and R-S for studying the effect of combining selective prediction with active learning. The mean and std of each metric over three random runs are reported (meanttstd). All numbers are percentages. Bold numbers are superior results.
F.6 Effect the number of rounds
In this section, we study the effect of the number of rounds T in active learning. The results in Table 21 show that larger T usually leads to better performance, and the proposed method ASPEST has more improvement as we increase compared to Margin and Margiarg... Also, when T is large enough, the improvement becomes minor (or can even be worse). Considering that practice, we might not able set a large T due resource constraints, we thus
10 by default.
F.7 Effect of the number of models N in the ensemble
In this section, we study the effect of the number of models N in the ensemble for MarMn and ASPEST. The results in Table 22 show that larger N usually leads to better results. However, larger N also means a larger computational cost. In our experiments, we simply set N = 5 by default.
F.8 Effect of the upper bound in pseudo-labeled set construction
When constructing the pseudo-labeled set using Eq. (9, we exclude those test data points with confidence equal to 1. In this section, we study whether setting such an upper bound can improve performance. The results in Table 23 show that when the labeling budget small, setting such an upper bound can improve performance significantly. However, when the labeling budget is large, sctting such an upper bound may not improve the performance. Since we focus on the low labeling budget region, we decide to set such an upper bound for the proposed ASPEST method.
F.9 Ensemble accuracy after each round of active learning
We evaluate the accuracy of the ensemble model f. in the ASPEST algorithm after the t-th round active lcarning. Recall that ft contains N models ft....,ff J. and fe(x) = arg max + E Hl (x k). The results in Table 24 show that after each round of active learning, the accuracy of the ensemble model will be improved significantly.
31

---

Published in Transactions on Machine Learning Research (03/2024)
Dataset
DomainNet R-C (easy)
Metric
1UACC
AUACC 1
Labeling Budget
100
500
500
1000
SR+ Margin (A=0) 0)
71.90:1..1
90.56.10.8
60.05
60.34:11.2
SR + Margin (A = 0.5)
75.54+1.7
91.43++0.5
64.99+00.7
66.81+0..5
SR + Margin (A [A=1] 1)
76.79=t0.5
91.24+0.5
64.3710.8
65.91=0.6
SR+ Margin (A = 2)
72.71+2..5
90.80-F0.3
64.17:10.3
66.21+0.2
DE Margin (A 0)
77.12:10.5
94.26.10.5
66.86.10.5
69.29-10.6
DE Margin () 0.5)
79.35+1.4
94.22+0..
69.28+0.3
71.60+0.2
DE+ Margin (A=1)
78.59+1.4
94.31+0.6
68.85+0.4
71.29+0.3
DE Margin (A= = 2)
77.644+2.3
93.81+0.4
68.54400.
71.28+0.2
ASPEST (A = 0)
84.48+2.5
96.99=0.2
68.61
73.21
ASPEST (A = 0.5)
86.46+3.1
97.01+0.0
71.53+001
73.69++01
ASPEST (A -A=10 1)
88.84+1.0
96.62:0.2
71.61
73.27+0.2
ASPEST (A 2)
85.46=1.7
96.43=1..1
00.
73.02
Table 20: Ablation study results for the effect of using joint training and the effect of the loss coefficient X. The mean and of each metric over Uhree random rums are reported (ste) All numbers are percenlages.
Dataset
MNIST-SVHI
Dom R-+C (easy)
Metric
AUACC
AUACC 1
Labeling Budget.
100
500
500
1000
Margin (T-1)
63.10:12.7
75.4
65.16110.4
66.76-00.3
Margargin (T-2)
68.09413.1
87.45 1.6
64.64410.8
66.91:1011
Margan (T=5)
1.7
91.32-+0.5
64.35++0.2
66.76=0.3
MRrMargin (T=10)
76.79:10.5
91.24
64.87-10.8
65.91:1066
Margar (0)
1.5
90.34 1.3
63.65
66.08
Dargargin (T=1)
6..
82.74412.1
68.39-10.2
70.55:1000
Margin (T=2)
7..25+1100
90.90++1.0 1.0
68.79++0.2
70.55+005
Margargin (T=5)
78.41+0.2
03.26+003
68.80+00.2
71.21
Margin (T=10) 10)
7..59
94.31
68.85-100.4
1..
Margin (0=
76.84+104
94.67+0.2
68.50+10.5
7..
ASPEST (T=1)
6..53+1.0
80.72+1.5
69.444-0.1
7..
ASPEST (T=2)
75.08+1.4
89.7010.7
7..6
7..56=0.3
ASPEST (T=5)
81.57+
95.43+10.1
71.23+10.1
73.19:001
ASPEST (T=10)
8..84+1.0
96.62+0..
71.61+0.2
73.27+0.2
ASPEST (T=20)
91.26+0.9
97.32+0.1
70.57+0.4
73.32+0.3
Table 21: Ablation study results for the effect of the number of rounds T. The mean and std of each metric over three random runs are reported (meandstd). All numbers are percentages.
F.10 Empirical analysis for checkpoint ensemble
In this section, we analyze why the proposed checkpoint ensemble can improve selective prediction per- formance. We postulate the rationales: (1) the checkpoint ensemble can help with generalization; (2) the checkpoint ensemble can help with reducing overconfident wrong predictions.
Regarding (1); when fine-tuning the model on the small set of selected labeled test data, we hope that the fine-tuned model could generalize to remaining unlabeled test data. However, since the selected test set is small, we might have an overfitting issue. So possibly some intermediate checkpoints along the training path achieve better generalization than the end checkpoint. By using checkpoint ensemble, might get
32

---

Published in Transactions on Machine Learning Research (03/2024)
Dataset
-VHN
NomainNet R-C (easy)
Metric
AUACC1
AUACCT
Labeling Budget
100
500
500
1000
Margan (N-2)
67.4113.99
91.
65.82:10.5
67.72:10.4
Margin (N=3)
77.53+1.5
93.41+0.1
67.54+0.44
69.61++0.2
Margin (N (N=4)
74.46+22.7
03.65+0.3
68.09+0.2
70.65+0.3
Margin (N=5)
78.59+1.4
94.31
68.85:10.4
71.
Margin (N=6)
79.34+0.7
94.40=0.1
68.63+0.2
71.65+003
Margan (N=7)
80.30+1.5
93.97=0.2
69.41
71.78+0.3
Margan (N=8)
78.91+1.5
94.52+0.2
69.00++0.0
7..4
ASPEST (N=2)
8..38+1.2
96.26
69.14+00.3
71.36+0..3
ASPEST (N=3)
84.86+100
96.60=0.2
69.21
72.25+0.2
ASPEST (N=4)
84.94+0.3
96.76=0.1
70.68#0.2
73.09#0.2
ASPEST (N=5)
88.84+100
96.62=0.2
71.61
73.27+0.2
ASPEST (N=6)
84.51+0.5
96.66=0.2
71.20+0.2
73.42+0.3
ASPEST (N=7)
86.70+2.3
96.90=0.2
71.16:0.2
73.50+0.1
ASPEST (N=8)
88.59+0.9
97.
71.62::0.3
73.76+0.2
Table 22: Ablation study results for the effect of the number of models N in the ensemble. The mean and std of each metric over three random runs are reported (mt)n. All numbers are percentages.
Dataset
DomainNet R-+C (easy)
Metric
AUACC 1
AUACC 1
Labeling Budget
100
500
500
1000
ASPEST without upper bound
86.95
96.59+0.1
71.39+1.11
73.52+0.2
ASPEST
88.84+10
96.6210.2
71.6110.2
73.27
Table 23: Ablation study results for the effect of setting upper bound when constructing the pseudo-labeles set R in ASPEST. The mean and std of each metric over three random runs are reported (meanisstd). All numbers are percentages. Bold numbers are superior results.
ensemble that achieves better generalization to remaining unlabeled test data. Although standard techniques like cross-validation and early stopping can also reduce overfitting, they are not suitable in the active selective prediction setup since the amount of labeled test data is small.
Regarding (2), when fine-tuning the model on the small set of selected labeled test data, the model can get increasingly confident on the test data. Since there exist high-confidence mis-classified test points, incorporating intermediate checkpoints along the training path into the ensemble can reduce the average confidence of the ensemble on those mis-classified test points. By using checkpoint ensemble, might get an ensemble that has better confidence estimation for selective prediction on the test data.
We perform experiments on the image dataset MNIST-SVVHN and the text dataset Amazon Review to verify these two hypotheses. We employ one-round active learning with a labeling budget of 100 samples. We use the margin sampling method for sample selection and fino-tune a single model on the sclected labeled test data for 200 epochs. We first evaluate the median confidence of the model the correctly classified and mis-classified test data respectively when fine-tuning the model on the selected labeled test data. In Figure 4 we show that during fine-tuning, the model gets increasingly confident not only on the correctly classified test data, but also on the mis-classified test data.
We then evaluate the Accuracy, the area under the receiver operator characteristic curve (AUROC) and the area under the accuracy-coverage curve (AUACC) metrics of the checkpoints during fino-tuning and the checkpoint ensemble constructed after fine-tuning on the target test dataset. The AUROC metric is equivalent to the probability that randomly chosen correctly classified input has higher confidence score
33

---

Published in Transactions on Machine Learning Research (03/2024)
Metric
Ensemble Test Accuracy
Datasct
MNIST
>VHH
DomainNet +C (casy)
Labeling Budget
100
500
500
1000
Round 0
24.67
24.87
37.33
37.46
Round 1
24.91
43.80
39.61
39.67
Round 2
37.75
54.91
41.15
41.55
Round 3
45.62
64.15
41.97
43.24
Round 4
50.94
71.65
42.57
45.09
Round 5
56.75
77.23
43.85
45.62
Round 6
59.82
79.97
44.20
46.60
Round
63.10
81.43
45.02
47.51
Round 8
67.49
82.78
45.17
48.59
Round 9
69.93
84.70
45.80
48.66
Round 10
71.14
85.48
46.36
49.70
Table 24: Ensemble test accuracy of ASPEST after each round of active learning. All numbers percentages.
MNIST-SVHN
Amazon Review
1.0
0.9
Median Confidence
Median Confidence
0.8
0.7
0.90
0.6
0.85
0.5
0.4
Prediction Wrong
Prediction
0.3
Correct
Correct
0.75
Epoch
Epoch
Figure 4: Evaluating the median confidence of the model on the correctly classified and mis test data respectively when fine-tuning the model on the selected labeled test data.
than randomly chosen mis-classified input. Thus, the AUROC metric measure the quality of the confidence score for selective prediction. The results in Figure 5 show that in the fine-tuning path, different checkpoints have different target test accuracy and the end checkpoint may not have the optimal target test accuracy. The checkpoint ensemble can have better target test accuracy than the end checkpoint. Also, in the fine-tuning path, different checkpoints have different confidence estimation (the quality of confidence estimation measured by the metric AUROC) on the target test data and the end checkpoint may have the optimal confidence estimation. The checkpoint ensemble can have better confidence estimation than the end checkpoint. Furthermore, in the fine-tuning path, different checkpoints have different selective prediction performance (measured by the metric AUACC) on the target test data and the end checkpoint may not have the optimal selective prediction performance. The checkpoint ensemble can have better selective prediction performance than the end checkpoint.
F.11 Empirical analysis for self- seff-training
In this section, we analyze why the proposed sell-training can improve selective prediction performance, Our hypothesis is that after fine-tuning the models on the selected labeled test data, the checkpoint ensemble constructed less confident on the test data Ux compared the end checkpoints). Thus, using the softmax outputs of the checkpoint ensemble soft pseudo-labels for self-training can alleviate the overconfidence issue and improve selective prediction performance.
the deep ensemble (obtained by ensembling
34

---

Published in Transactions on Machine Learning Research (03/2024)
MNIST-SVIN
2.44
0.72
2.42
0.603
Accuracy
6.40
AUROC
AUACC
0.73
0:38
0.69
0.36
0.67
0.00
6.34
Mode
Mode
0:47
spent
Checepart
0.52
Finil checcpernt ensemble
0.65
Final checopent etsemble
checopo
25
50
100 Epoch
125
150
175
230
25
75
100 Epoch
125
150
175
200
0
23
50
100
125
150
175
Epoch
Amazon Review
Amazon Review
0.670
0.755
Accuracy
0.825
0.692
AUROC
AUACC
0.745
0.000
9.740
0.615
Mode
0.600
Model
Mode
Checepait
Creckpaine
Checpoor
checcpoint ennemb
Finsl checlpoor ensemble
6.300
Shackpart ersamble
0
25
75
120
125
150
175
200
25
50
75
103
125
150
175
200
25
50
75
100
125
150
175
703
Epoch
Epoch
Epoch
Figure 5: Evaluating the checkpoints during fine-tuning and the checkpoint ensemble constructed after fine-tuning on the target test dataset.
We perform experiments on the image dataset MNIST and the text dataset Amazon Review verity this hypothesis. To see the effect of self-training better, we only employ onc-round active lcarning (i.e., only apply one-round self-training with labeling budget of 100 samples. We visualize the histogram of the confidence scores the test data Ux for the deep ensemble and the checkpoint ensemble after fine-tuning. We also evaluate the recciver operator characteristic curve (AUROC) and the area under the accuracy-coverage curve (AUACC) metrics of the checkpoint ensemble before and after the self-training, We use the AUROC metric to measure the quality of the confidence score for selective prediction. The results in Figure P show that the checkpoint ensemble is less confident on the test data Ux compared to the deep ensemble. On the high-confidence region (i.e., confidence2 17. Recall that r) the confidence threshold used for constructing the pseudo-labeled set R. We set " 0.9 in experiments), the checkpoint ensemble is also less confident than the deep ensemble. Besides, the results in Table 25 show that after self-training, both AUROC and AUACC metrics of the checkpoint ensemble are improved significantly. Therefore, the self-training can alleviate the overconfidncce issue and improve selective prediction performance.
Dataset
MNIST-SVHI AUROCT
Amazon Review
Metric
AUACCT
AUROCT
AUACCT
Before self-training
73.92
66.75
67.44
76.24
After self self-training
74.31
67.37
67.92
76.80
Table 25: Evaluating the AUROC and AUACC metrics of the checkpoint ensemble before and after self-training All numbers are percentages.
F.12 Training with unsupervised domain adaptation
In this section, we study whether incorporating Unsupervised Domain Adaptation (UDA) techniques into training could improve the selective prediction performance. UDA techniques are mainly proposod adapt the representation learned on the labeled source domain data the target domain with unlabeled data from the target domain [Liu et al. 2022). We easily incorporate those UDA techniques into SR SAlA (Algorithm) 1
35

---

Published in Transactions on Machine Learning Research (03/2024)
MNIST-SVHN (Deep Ensemble)
MNIST-SVHN (Checkpoint Ensemble)
4000
4000
Prediction
Prediction
3500
Wrong
3500
Wrong
Correct
Correct
3000
3000
2500
2500
Count
Count
2000
2000
1500
1500
1000
1000
500
500
0.2
0.4
0.6
0.8
1.0
0.2
0.4
0.6
0.8
1.0
Confidence
Confidence
Amazon Review (Deep Ensemble)
Amazon Review (Checkpoint Ensemble)
3500
Prediction
3500
Prediction
Wrong
3000
Correct
3000
Correct
2500
Count
Count
1500
1500
1000
1000
500
500
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
Confidence
Confidence
Figure 6: Plot the histogram of the confidence scores on the test data Ux for the deep ensemble and the checkpoint ensemble after fine-tuning.
DE (Algorithm 2), and the proposed ASPEST (Algorithm 3 by adding unsupervised training losses into the training objectives.
We consider the method DE with UDA and the method ASPEST with UDA. The algorithm for DE with UDA
is presented in Algorithm and the algorithm for ASPEST with UDA is presented in Algorithm We consider UDA techniques based on representation matching where the goal is to minimize the distance between the distribution of the representation on D and that on Ux. Suppose the model f a composition of prediction function h and a representation function & (i.e., f(x) = h()()))) Then LUDA (x;
d(p ,Pxx)) which is a representation matching loss. We consider the representation matching losses from the -rate-of-the-are UDA methods DANN (Ganin et al. 2016) and CDAN [Long et al. 2018
We evaluate two instantiations of Algorithm DE with DANN and DE with CDAN, and two instantiations of Algorithm 5 ASPEST with DANN and ASPEST with CDAN. The values of the hyper-parameters are the same those described in the paper except that set n, = 20. For DANN and CDAN, set the hyper- parameter between the source classifier and the domain discriminator to be 0.1. The results are shown in Table 26
(MNIST-SVHN), Table 27 (CIFAR-IO-CCINIC-OO)) Table 28 (FMoW), Table 29 (Amazon (DomainNet. R-*),, Table 31
Review), Table 30
(DomainNet R-PP), Table 32
(DomainNet R-+) and
Table 33 (Otoo).
From the results, we can see that ASPEST outperforms (or on par with) DE with DANN and with CDAN across different datasets, although ASPEST doesn't UDA techniques. We further show that by combining
36

---

Published in Transactions on Machine Learning Research (03/2024)
ASPEST with UDA, it might achieve better performance. For example, MNNST-SSVHN, ASPEST with DANN improves the mean AUACC from 96.62% to 97.03% when the labeling budget is 500. However, in some cases, combining ASPEST with DANN or CDAN leads to much worse results. For example, on MNIST-SSVHN when the labeling budget 100, combining ASPEST with DANN or CDAN will reduce the mean AUACC by over 4%. It might be because those cases, DANN or CDAN fails align the representations between the source and target domains. Existing work also show that UDA methods may not have stable performance across different kinds of distribution shifts and sometimes they can even yield accuracy degradation (Johansson et al.) 2019 Sagawa et al. 2021 So our findings align with those of existing work.
Algorithm DE with Unsupervised Domain Adaptation
Input: A training dataset D An unlabeled test dataset Ux, the number of rounds T, the total labeling budget M, a source-trained model f, an acquisition function afB,f.9). the number models in the ensemble N, the number of initial training epochs ns, and a hyper-paramcter
Leu Hd = for j I.....,N
Fine-tune cach model fg in the ensemble via SGD for n, training epochs independently using the following training objective with different randomness:
min 03
E(x (xE(x, y: + (UD:((",Ux:0+)
(35)
where LUDA is a loss function for unsupervised domain adaptation. Let Bo = 0.
Let (x) = maxkey f((x k).
for t= 1, T do
Select batch B, with a size of m [ from Ux for labeling via:
B. arg
max BCUX
a(B, 9t-1
(36)
Use an oracle to assign ground-truh labels to the examples in B, to get B,. Fine-tune each model fi-1 in the ensemble via SGD independently using the following training objective with different randomness:
min 85
[x lCE(x, yi 0') + X Exxy)DD (cE(x,y;0) + LUDA(D" (UDa((",Uxx;0) (37)
where AJ is the model parameters of fi-1
Let f. = end for
Output: The classifier f f= fr and the selection scoring function g = maxkey f(x k).
37

---

Published in Transactions on Machine Learning Research (03/2024)
Algorithm 5 ASPEST with Unsupervised Domain Adaptation
Input: A training set Dr", unlabeled test set Ux, the number of rounds T, the labeling budget M, the number of models N, the number of initial training epochs n2, checkpoint epoch Ce a threshold 7, a sub-sampling fraction p, and a hyper-paramedt A.
Tet Jd = f for j = I. Set Nº and P OnXK-
Fine-tune each f for n, training epochs using the following training objective:
min 80
(x,)) (xE(x, y: 0*) LUDA(D", ,Ux;0')
(38)
where LUDA is a loss function for unsupervised domain adaptation. During fine-tuning, update P and Ne
using Eq. 66 every Ce training epochs.
for t= -...... T do
Select batch B, from Ux for labeling using the sample selection objective 7
Use an oracle to assign ground-truth labels to the examples in Be to get Be.
Set N₂ ===00 0 and P = OnxK.
Fine-tune each Hi, using the following training objective:
min During fine-tuning, update P and No using Eq 6 Let f/. = f/-11-
E (x,),,, (x, y: 0² + X- (x (cE(x, 02) LUDA(D" (xxx()"
(39)
every ce training epochs.
Construct the pseudo-labeled set R via Eq 9 and create Raub by randomly sampling up to [p- n] data points from R. Train each f further via SGD using the objective (10] and update P and No using Eq 6 every Co
training epochs.
end for
Output: The classifier f(x;) arg max Pi.. and the selection scoring function g(x,) = -maxkey P.Kk
Dataset
MNIST
Metric
conface 2 900%
accicge > 90% T
AUACC
Labeling Budget
100
500
1000
100
500
1000
100
500
1000
DE with DANN
Uniform
2..274118
72.78112.0
87.05-4055
63.95 .4
8-.99
88.6440.22
50.37-10.7
93.25-10.4
96.05-10.1
DE with DANN
Eatropy
11.33+8.2
74.04+2.2
91.06-+ 1.4
58.28-+2.
83.64+0.99
90.41+0.5
74.62++1.6
93-45+0-3
96.47+0.2
DE with DANN
Coubdence
15.6846.3
76.34+3.1
83.96=1.2
61.32=3.0
85.02+0.9
91.6140.4
76.43+3.0
19.8540.6
96.9720.3
DE with DANN
Margin
30.6442.1
83.444-0.9
06.17
66.79-009
87.30+00.4
92.7140.2
82.1440.8
95.40++0.3
07.60-+0.1
DE with DANN
Avg-KLD
22. 22.3043.00
78. 13+2.1
03.42-
03:22-2.0
86-40-0.8
91.4740.6
78.88+1.6
94 04.25+0.5
07.02-0.2
DE with DANN
CLUB
16.12=13.6
72.27
86.71
61.70=277
82.72411
58. 16:0.2
17.4743.4
93. 33.33003
96.21=000
DE with DANN
BADGE
10.19
83.83112
90.91111
63.93:1.1
85.27
90. 15:05
19.82:11
91.5820.3
96.8021011
DE with CDAN
Unitorma
28. 10218
73.15241
87.302006
63.9522.7
83. 10:0033
58 86.603
30.28:222
93.312001
16.12202
DE with CDAN
Entropy
6.9419.8
74381115
10.77
50.00:122.3
81.110044
10.32.10.6
76.0112.0
93, 03.48.0.3
96.3810.2
DE
Confidence
13.47
75.15.128
02.77
84.622..0.9
91.21.10.3
16.19.12.8
93.62.10.6
96.63.10.1
DE with CDAN
Margin
22.4443.3
$1.844--.5
96.07-0.2
62.80-33.8
86.71+41.0
92.64-+0.
78.69-+2.6
04.89-+0-5
97.57+0.0
DE with CDAN
Avg-KLD
20.23+4.1
10.62+117
93. 13-2.5
62.23-2.7
86.34-t0.6
91.30++1.0
17.68+2.6
94.81+0044
96.97-0044
DE with CDAN
CLUE
74726.4
72. 72.6142.99
87, 87.22=0.2
57.82=2.9
82, 1.550011.3
$8.62-00.1
73.33-22.3 3
93. 00.388t0.7
96.31=0.0
RADGE
26.88+3.5
79.21440.1
02.50-0.7
65.60+11.7
85.32++0.1
91.18-+0.4
51.10++1.3
94.73-40.1
97.17+0.2
ASPEST (ours)
52.10=10
89. 39.2221119
98, 08.70220.1
76.1011.5
89.622004
93.9220.3
88 88.812110
96. 96.62220.2
98.06220.1
ASPEST wish DANN (ours)
37,9012.
91.61.006
00.39:204
69.45:1.77
90.70.10.3
94.42.10.4
84.55.11.0
97.03.10.1
98.23.10.1
ASPEST with CDAN [ours]
30.97
91.3910..6
99.50110.3
67.58
90.6010.33
94.46100.2
52.2013.3
96.95100.1
98.26110.1
Table 26: Results of evaluating DE with UDA and ASPEST with UDA on MNIST The mean and sid of each metric over three random runs are reported (meantstd). All numbers are percentages. superior results.
Bold numbers are
38

---

Published in Transactions on Machine Learning Research (03/2024)
Dataset
CIFAR-IO-CINES
Metric
coolace 2 90%
ace|cor 2 90%% T
AUACC
Labeling Budget
500
1000
2000
500
1000
2000
500
1000
2000
DE with DANN
Uniform
58.85=0.3
59. 59.39+0.2
00.01+0.1
77.06+0.2
07.33+0.2
77.84=0.1
90. 10.1000.11
90.60+00.1
90.73=0.1
DE with DANN
Eatropy
59.42-0.4
60.864-0.
64.5240.3
78.1440.2
79.20+0.1
81.31+0..1
90.72+0.0
91-064-0.1
92.02-00.0
DE with DANN
Confidence
09.44-066
61 01.0840.3
65.1240.2
78. 194001
79.38+0000
81.20+0.1
90.73=01
16. 01.20:001
92.06:000
DE with DANN
Margin
39.1=03
20
65.5840.4
78. 1540.0
19:25:0.1
81.06
90,70:01
11:00:01
92.11=000
DE with DANN
Avg-KLD
0.5500055
62.01:::11
63.082002
78.3240.1
79.31=0.1
81.020000
90 89:00
11.310000
92.11:0000
DE with DANN- + CLUE
0.5
61 691112
64.08..0.2
77,8410.2
78.35:1022
79.3810.1
90. 70.7510022
91.0740.1
91.6320.00
DE with DANN
BADGE
60.18 0.4
62.15.10.2
65.31 1..66
77.70.0.11
78.54.0.1
79.61..0.2
90.72.0011
91.19110.1
91.86.10.1
DE with CDAN
Uniform
58.72 0.2
59.49.1..5
60.2
77.16.10.0
17.52.10.1
77.9010.1
90.451...1
90.66.10..0
90.78.10.1
DE with CDAN-
Entropy
58.73-04 0.4
60.82+-0.5
64.45+0.2
77.95++0.1
79.20++0.1
81.0440011
90.57+0.1
91. 10400.
91.86+0.11
DE with CDAN
Conbdence
50.10-0.6
61.0340.6
04.60+0.2
71.924000
70:20+0:2
80.070000
90.00=000
91. 10+0.2
91.00:0000
DE with CDAN
Margin
59.88005
9
61,824044
78.0920.3
79:02=022
80.822011
900 00.73:011
17:02
91.9520.1
DE with CDAN
Avg-KLD
00.51
61. 71::1.5
63.032003
28.20220.2
79, 29:22.022
81.1520.1
90 80.8500.00
91. 1920.1
92.07=0.1
DE with CDAN CLUE
60.12 0.5
61.77100.3
64.06.10.2
77.8810.11
18.3810.2
79.42
09.4210.2
90.7310011
91. 91.0810.1
91.64.0.0
DE with CDAN BADGE
60.28 0.7
61. 1..0..
10.3
77.68.10.2
78.53.0.11
79.84..02
90.73
11.17.1.00
91.95.10.1
ASPEST (ours)
60.38 0.3
63.34.10..2
60.81.0.33
78.23.10.1
79.49.10.1
81.25.10..
90.951.0.0
91.60.10.0
92.33.1.11
ASPEST with DANN (ours)
61.69+0.2
63.58-40.4
66.81 -0.4
78.68-0.1
79.68+00.1
81.42+0.1
91.16+0.11
91.66-0.1
92.37+0..1
ASPEST with CDAN (ours)
61.00002
0.:00
06.78#0.1
78.3640.1
19.6120.1
81.49=0.0
91.1==0.0
01.3740.1
92.41=0.0
Table 27: Results of evaluating DE with UDA and ASPEST with UDA on std of each metric over three random runs are reported (meanttstd). All numbers are percentages. are superior results.
The mean and
Bold numbers
Datsset
FMoW
Metrie
coelace 2 70%
ac0 COU 2 70% T
AUACO
Labeling Budget
5100
1000
2000
500
10010
DE with DANN
Uniform
46.11-0.6
51.7740.3
62.76+0.5
57.62-0.3
60.67-0.4
66-21+0.2
70.17-0.3
72.46+0.3
76.83+0.2
DE with DANN
Entropy
44.36=0.7
48. 1940.3
00.32+0.8
66.78+0.11
59.51=0.0
65.76+0.3
09 00:000000
71.0240.2
76.154033
DE DANN-
Confideuce
1146=005
1
61477203
57.0120.3
60.5120.3
66.61:0.1
69.1 69.14:01
11.5020.1
75.70220.1
DE with DANN-
Margin
08.0-0.4
54.5510.5
70.1120.4
59.0740.2
62.7920.2
70.0220.1
70.76-0.1
73.291002
78.25.10.1
DE with DANN-
+ Avg-KLD
1842-0.1
55 9.
68.73 1.1
59.0620.2
63. 03.122002
09 41:01
70.84=011
73.8340.1
17.91=0.4
DE with DANN
CLUE
11.110.6
15.15::02
19.0220.5
56.0120.3
56.822022
38 38.6620.3
69 69.11:022
70. 162022
71.662022
DE DANN-
BADGE
0.5
51441115
07.99109
58.61 10.2
61 61.6700.0
68.7110.5
71.17
73.611011
78.65.0.33
DE with CDAN
Uniform
46.08 0.7
51.9240.88
62.87+0.2
67.45-+0.1 1
60.73-0.4
66. 66-19-0.2
09 09-93-0.3
72.6740044
76.87+0.1
DE with CDAN-
Entropy
44.420033
49. 3240.11
60. 01.333
66.83+0.11
60:01:0:2
65 8620.2
09. 18.10.
71.31+0.3
75.41:03
DE with CDAN-
Confideuce
01.75-0.1
11:11:1111
1.0
57 57.0920.11
60 0..5020.2
66 94101
09.2720.1
11.6020.2
16.132203
DE with CDAN-
Margin
47.48-077
54 74.4840.7
70.26+0.9
08.95+0.4
02.08+0.3
70.10:0.4
70.65+0.3
73. 03.46-0.22
78.34-0033
DE with CDAN-
Avg-KLD
1843-022
68 932060
59.3620.2
62.71=0.2
09 04.542022
71:12002
73.364002
17.972022
DE with CDAN-
CLUE
110920.3
46.112.0.5
08.9020.1
55.7820.3
56,9820.2
58, 162022
09.032001
70.024022
71.3120.1
DE CDAN BADGE
07.93-02
51.11102
60.5
58. 18.1610.1
61.811011
66.361022
70.91:022
73.6310.1
78.5210.2
ASPEST (ours)
53.0540.4
59.86-40.4
76.52-0.6
11-10-0.2
65.1840.2
72.86-0.3
71.12-022
74.25-0.2
79.93+0.1
ASPEST with DANN (ours)
51.02=0.9
58 63+1.1
72.97=0.9
61. 10:0.55
64.92+0.4
71.21=04
71.
71.03=0.3
73. 79.49.
17.84=033
ASPEST with CDAN (ours)
51.40 0.6
58.214-066
73.94+0.6
61.38-00
85.0440.2
71.63+0.2
71.1740.1
73.59+0.1
0..04+0.2
Table 28: Results of evaluating DE with UDA and ASPEST with UDA on FMoW. over three random runs are reported (sean All numbers are percentages. Bold numbers are superior results.
The mean and std of each metric
39

---

Published in Transactions on Machine Learning Research (03/2024)
Dataset
Amazon Review
Metric
coefare 80%
ace 2 80% 1
AUACCT
Labeling Budget
500
1000
2000
500
1000
2000
500
1000
2000
DE with DANN-
L niformm
38.55=3.3
37 1,
39.2141.9
89.0620066
68. 08.9120.1
69.412022
77.52-0.7
77.034044
77.70:0.2
DE with DANN-
Eatropy
38.22 2.3
41. 8..88
41.3711.3
69.48.10.3
70.7110.3
71.5510.2
77 77.49.0.5
78.39100.
78.58.1011
DE with DANN + Confidence
38.01 1.0
38.3642255
3..
60.45-10.1
70.16:102
71.44.0.2
77.54:0.2
77.58.0.55
18.48.10.3
DE DANN
Margin
36.82-1.3
1..3
41.98+1.5
00.35+0..3
60.63+0.3
71.27+0.2
77.30-0.3
77.2340.3
78.34+0.3
DE with DANN
ALg
37.15 2.9
38.21 3..2111.3
42.46.11.4
69.38.10.4
89.79.0.2
71.2110.2
17.2
77.7210.3
18.68.10.3
DE with DANN
CLUE
40.23 1.00
34.7144-.8
31.38+00.9
68.95-4-0.7
08.07 0.2
67 07.44-00.3
77.62+1.00 1.0
76.27 4-0.6
75.60-10.2
DE with DANN
BADGE
37.51-1.8
37 37.00+-0.3
41.6242..3
68.96+40.4
00.27--0.1
70.20+0044
77 77.20-00.4
77.21400.1
78.31+0.55
DE with CDAN
Uniform
37.81-033
37. 831.2.7
39. 09.3240.8
68 93.
69, 16:0.7
69.5 0.3
77.1620.1
77.30
77.74203
DE with CDAN-
Entropy
37.99 0.8
37. 37.681111
42.55.10.9
69.54.0.3
70.01 10.2
71.5210.2
77.5210.2
77.61.10.1
78.63.10.1
DE with CDAN-
Confidence
35.76 0.9
38.891-2.8
41.43:12.1
63.24-10.0
70.45-10.4
71.5040.44
77.08:0.22
77.82.00.4
78.47
DE CDAN
Margin
37.08-20
4.0
1..18+1.3
00.50+0.3
80.80-+0.4
71.29+0.0
77.50+0.5
77.31+0.3
78.46-0.3
DE with CDAN-
Avg. WLD
37.85
40.71100.9
44.35.10.9
69.41 10.3
70.29.1011
71.2810.2
77.2810.5
78.11.10.2
78.86.10.2
DE with CDAN-
CLUE
34.85 2.7
1-.03+-1.3
30.70++0.4
68.70++0.3
67.84-+0.1
67.12+00.
76.95+-0.7
76.234+0.4
75.36++0.4
DE with CDAN- BADGE
30.47-022
30. 1.11
41.64+0.9
00.33++0.0
00.34:002
70.58+00.2
77.52-0.2
77.40++0.2
78.2440.3
ASPEST (ours)
38.41-077
40 861008
43.7740.1
69.31:0033
70.17:0022
71.6020.2
77. 77.6920.1
78.35:1022
79.51
ASPEST with DANN (ours)
40.22 0.5
41.991144
45.84_0.1
69.42.10.1
70.30.0.1
71.5
78.00.10.1
78.34110.3
79.43.10.1
ASPEST with CDAN (ours)
40.02
0.5
40.66-10.6
44.95-00.4
60.50+0.1
70.37+0.2
71.42+0.0
77 7..80-0.1
78.57
79.25 +0.0
Table 29: Results of evaluating DE with UDA and ASPEST with UDA on Amazon Review. The mean and std
of each metric over three random runs are reported (mean All numbers are percentages. Bold numbers are superior results.
Dataset
Domainnes R-CC (rasy)
Metric
coeloce 2 80%
000 00e|cou 2 80% 1
AUAOC
Labeling Budget
500
1000
2000
500
1000
2000
500
1000
2000
DE with DANN
U Uifform
33,53=0.5
36 282013
10 1311011
50.37=0.5
52, 19:
35.15=0.1
69 09.1
70.98202
13.50:003
DE with DANN-
Eatropy
28.66=1.0
31. 4
12.7720.7
48. 13:0.6
52,70:20.3
39.0110.2
66.6020.5
70.61201
75.45:002
DE with DANN-
Coufideucee
29.92-04
35. 1.0
43.33.0.4
48.61
53.36100.5
30.7210.3
67 2310.2
70.9210.5
75.89.10.3
DE DANN
Margin
35.19 0.3
0..6310.2
46.51 10.5
52.2011.33
55.60 10.2
60.9710.-
70.7010..
15.4110.1
77.24
DE with DANN
Avg.KLD
36.02 0.6
39.67+0.5
47.2040.8
63:00++0.3
56.75+0033
61.22+0.3
0.3
71.19-0.3
73.51--0.2
77.46-0.2
DE with DANN
+ CLUE
32.26 1.5
10.0
35.66.10.3
50.21
50.90.10.1
51.5010.1
69.17
70.20.10.2
70.82.0.11
DE with DANN
BADGE
35.27 0.5
38.88110.3
45.97 1007
12.15-10.3
54.80
60.03-10.3
70.65.-0.1
72.95.10.1
76.87.10.1
DE with CDAN-
Uniforma
33.19=0.6
36, 01.00.7
39
09.3240.2
50.46::0.2
51 0.1
35 2320.2 2
69: 32:2223
70. 70.86:0.3
73.55=002
DE with CDAN-
Entzopy
29.501005
33, 1.
02.212:0.5
48.0120.11
52.52:0033
58 25.9620.2
66, 62:22
70.28200.
75.33:20.1
DE with CDAN-
Confidencee
29.21 1.0
0..9210.6
43.36.10.4
48.4810.4
52.85.10.4
59.5810.4
66, 66..2.
70.61.10.4
75.93.10.3
DE CDAN
Margin
35.87 0.7
0..3710.4
10.4
52.58.001
55.28
61 6..201002
10.95:0.2
72.95 10..2
17.26-10.1
DE with CDAN
Avg-KD
9.22100.
40.08110
47.621044
62.95210.3
55.9832011
61.561002
71 29:3333
73.6010.1
17.582022
DE with CDAN-
CLUE
31.74 2.1
35.111002
35.87.0.5
49.99.10.2
51.3910.2
51.43.0.2
69.04
70.35.10..0
70.8210.3
DE with CDAN BADGE
34.74 0.5
38.68440.7
45.874
51.80-10.3
54.75-10.2
60.22
70.38--0.
72.90-40.2
76.25-10.2
ASPEST (ours)
37.38=0.1
39, 9840.3
18 1.0
51.5620.3
06.95=0011
62 22.6920.2 2
71 0..
73.27=002
77. 77.4020.1
ASPEST with DANN (ous))
37.4114.88
42. 1.11
19.7120.6
55.60=0.1
58 08.20-00.2
63.641002
71 882112
74.18211
78.09.0.00
ASPEST with CDAN (ours)
36.60
1.2
42.96.10.6
50.86.00
55.35.10.2
58.7110.2
63.851002
71.99.10.2
74.6010.2
78.45.0.3
Table 30: Results of evaluating DE with UDA and ASPEST with UDA on Domain R-C, The mean and std
of each metric over three random runs are reported (mean All numbers are percentages. Bold numbers are superior results.
40

---

Published in Transactions on Machine Learning Research (03/2024)
Dataset
DomainNer R-P (medium)
Metric
corfare 70%
aceioon 2 70% 1
AUACCT
Labeling Budget
500
1000
2000
500
1000
2000
500
1000
2000
DE with DANN
L niform
26.
28 34141.
30.63202
41. 11.9620.2
42.89:0.2
44.7320.1
37.0420.1
58. 10102
59.87=0011
DE with DANN
Eatropy
24.75 0.4
27.0210.5
50 10.002
40.29.10.4
42.34.10.
45.7810.2
55.1910.3
57. 12.13
60.21.0.1
DE with DANN
Confidence
22.41 0.9
27.033-0.6
31.3040.6
39.05-10.5
42.61
46.601.0.2
5..
57.35.10.3
60.93:10.4
DE DANN
Margin
20.16-0.1
30.58+0.3
33.6440.6
43.78+0.2
17+0.
47.00+0.4
76.00-
50.94+0.0
62. 02.1040..4
DE with DANN
ALD
29.52 0.1
31.1710.4
34.0910.3
43.8410.3
45.33.102
48.1810.2
56.8910.2
60.25.10.22
62.54.10.2
DE with DANN
CLUE
27.48 0.5
27.834-0.2
28.39-+0.5
42.05-40.3
42.34-402
42.65+-0.1
87.32-0.3
67.64410.2
57.99-100 -022
DE with DANN
BADGE
28.02 0.1
30.36+-0.2
33.864-0.3
43.35+0.1
44.86+0.1
47.04+003
58 08.38+0000
50.82440
62.26-40.2
DE with CDAN
Uniform
2..
28, 331-1.
29 95.9100.4
41.7740.3
42.85:0.2
44 03.310.44
36, 86:2444
58. 01.00
59.4240.1
DE with CDAN-
Entropy
24.91 0.4
26. 0.99
30.33.10.4
40.34.1.33
42.07.0..
45.7910.2
55. 55.38
56. 70.18.8..
60.21.10.2
DE with CDAN-
Confidence
24.58 0.7
27.11440.5
31.07
40.32-10.2
42.64-003
46.25--0.3
55.144:0.3
57.40
00.63-10.3
DE CDAN
Margin
28.33-0.1
30.1740.3
33.54+0.4
43.44+0.4
44.77+0.1
47.56+0.2
58.31+0.2
50.65++0.1
62. 02.17.0.2
DE with CDAN-
Avg. KLD
28.69 0.2
30.89100.9
34.3010.2
43.6410.2
45.34.0.2
48.2210.1
58.6010.1
60. 60.15.10.44
62.67.0.11
DE with CDAN
CLUE
27.52-044 0.6
27.964-0.2
28. 28.18-0
42.02-40.2
42.4440.1
42.67+00.
57.21+003
57.70-+0.1
08.04+00.3
DE with CDAN- BADGE
28.70-0.1
30.28+0011
33.7740.4
43.45-40.0
44.73-0.33
47 47.84+002
58.47=0.1
50.644-0.2
62.37-402
ASPEST (ours)
29.69-0.1
32. 50.40.3
35. 45.46.6..6
44.96200.1
46.77:0.22
49. 0.1
38. 38.74210.0
60). 60.36:1:000
62,84:0.2
ASPEST with DANN (ours)
31.7510.4
33.58
10.3
36.96.10.2
46.16.0..
47. 10.64.10.2
50.37 10.3
59 59.63.10.2
61.06.0.11
63.75.0.1
ASPEST with CDAN (ours)
30.39
0.4
33.57 F..3
37.53 0.7
45.90+-011
47.71
30.31
50.13+0.3
61.17:0.2
63.69-103
Table 31: Results of evaluating DE with UDA and ASPEST with UDA on DomainNet R-+P. The mean and std of each metric over three random runs are reported (meantstd)) All numbers are percentages. Bold numbers are superior results.
Dataset
DomaiinNe R-S (hard)
Metric
coeloce 2 70%
n0e 2 70% 1
AUAOC f
Labeling Budget
500
1000
2000
500
1000
2000
500
1000
2000
DE with DANN
U Uniform
17.55=0.1
19 8240.3
23.5720.4
32.6120.5
31.60:003
37.73=0.2
47.0020.5
19.92404
53.52=0.1
DE with DANN-
Eatropy
10,77-0.8
15. 58115
20.11
27. 27.7820.7
$1.09-20.2
36.1910.3
41.6920.7
15.62403
51.05.2044
DE with DANN-
Coufideucee
10.64 122
15. 15.22104
20.25.10.5
28.09.11.0
31.7610.3
36.8610.8
41.9411.
46. 19.19...3
51.48.007
DE DANN
Margin
17.00 0.7
2..4410.6
10.5
33.61
35.79.10.5
0.03
46.6710.1
10.0
55.6410.4
DE with DANN
Avg.KLD
18.02 1.0
21 2240022
25.4640.2
34.00+00.2
36.61+0.2
40.72+0.2
49.05-0.2
51.7940.2
55.95+00.2
DE with DANN
+ CLUE
15.77 03
18.1410.77
19.49.1.44
32, 12.10.10.1
33.42.10.3
34.5010.3
47.1810.2
48.63
50.03.10.3
DE with DANN
BADGE
16.84 0.9
20.584.0.3
25.11
33.97
36.20-10.2
40.01110.3.
48.87:-0.2
51.46-10.22
55.33-10.2
DE with CDAN-
Uniforma
17.33=055
19. 7
22. 22.9940.5
32 32.17=0.5
31.59=00.3
37 77.88202 2
47. 09255
50. 50.02+0.2
53.51=0.3
DE with CDAN-
Entzopy
12.48-03
19.191
20.2220.
28 28.8310.11
32 02.41:20.4
36.5710.1
42.9320.5
47 17.00233
51.212022
DE with CDAN-
Coufidencee
11.23 0.6
13.9310.1
18.45.1..3
28.67.10.3
31.35.1.44
35.5610.8
42.87
45.40..0.7
49.80.11.
DE CDAN
Margin
18.06 0.7
2..3010.3
25.05.0.
33.96.1.2
35.76.10.2
40.1110..1
49.15:0.11
50.92.00.1
10.27
DE with CDAN
AVg-KD
18.63-10
20. 801413
25.4910.9
34.19:1.44
36, 11:0222
40.532005
49. 45:45
$1.5810.1
55.7420.5
DE with CDAN-
CLUE
16.51 0.3
18.8210.1
19.4710.1
32.23.10.2
30.8910.4
3..3
47. 17.4
49. 10.2
49.98.10.3
DE with CDAN BADGE
17.52 0.8
21-484-0.5
25.35-10.4
33.53-10.5
16.19-10.4
4-03
45.67
51.65-10..3
55.62-10.3
ASPEST (ours)
07.86=0.1
20. 12
25.87 25.8720.4
35. 17.17.
37. 0.3
41. 162002 2
49. 62:2.11
51.61-2044
55.902002
ASPEST with DANN (ous))
16.35_1.2
23.18
28. 00.0020011
36.3621022
39.4010.4
42 02.91
30.58
53.75
57.25.10.1
ASPEST with CDAN (ours)
18,81111
22.9510.8
28.17
36.85
19.10.1022
43.25
51.14.0.3
53.47.0.2
57.2610.2
Table 32: Results of evaluating DE with UDA and ASPEST with UDA on DomainNet R-SS The mean and std
of each metric over three random runs are reported (mean All numbers are percentages. Bold numbers are superior results.
41

---

Published in Transactions on Machine Learning Research (03/2024)
Dutaset
Otto
Mearie
coelace 80%
ace 2 80% 1
AUACC
Labeling Budget
500
1000
2000
1000
2000
000
2000
DE with DANN
U Uniformn
70.35
72. 4.140.
75.6310.7
76. 12.103
77.0410.1
78.2510.1
86.6710.1
87. 16116.11
88.09.1011
DE with DANN
Eatropy
15.27 0.3
21.2510.11
02.23.10.3
78.14.0.11
$0.45.10.0
83.7310.1
87.7310.1
88.91 1..00
90.9010.1
DE
Coulideuce
7L.6-003
92372006
80.50:0000
83.61002
87.51101
90.9420.1
DE with DANN
Margin
75,47 0.4
52.561...7
91.86.1099
78.26.10.1
80.79.0.2
83.6110.3
87.8710.1
D0.0000
90.88.1011
DE with DANN
Avg. RLD
76.02 0.6
81.7810.4
01.8210.3
78.53-10.0
80.70.10.1
83.6810.00
87.99
1:0.0
10.17
90.90.1.1
DE with DANN
CLUE
69.63 0.4
68.0740.3
62.70400.6
75.81+0.3
76.4440.0
73.40+0.3
86.68-0.2
86.3140..1
84.80-0..2
with
BADGE
74.69-0.6
79.04:0.6
87.63+0.4
71.9700
19.57=0.3
00.00=0.1
82=0.1
88.9240.1
90.67
DE with CDAN
Uniforta
70.25 0.9
72 431004
75.21 10.7
76.0910.3
76.94.0.1
78.1310.1
86.5
86.5610.3
87. 1..1..2
87.90.1011
DE with CDAN
Eatropy
74.7300.6
80
92.3820.2
77.972022
50.502003
83.812022
17:011
88. 93.300
90.8
DE
Coubdeuce
7.88-0.6
80.51:0.3
83.65103
90.95.001
DE with CDAN
Margiu
76,68 1.0
81.5710.4
10.20.1.5
78.74.10.5
50.62.0.2
84.011022
88.08
88.85.10.2
91.09.10.0
DE with CDAN
Avg-KLD
75.88 0.4
81.82100.8
91.4311. 1.1
78.45:10.1
50.72.10.3
83.72100.3
87.92:00.2
60. 12.1..2
90.91 10.2
DE with CDAN
CLUE
69.86-04 0.5
67.79+-0.2
63.46-00.9
76.00+0.2
76.42-40.3
73.66+003
86.81+0011
86.25440.1
85.00++0.1
DE with
BADGE
74.68-04
404003
87.37+0.4
71.89+0.1
79.78=0.1
02.8600.11
78-001
$3.90+0.11
90.7240.1
ASPEST (ours)
17.85002
84.20-20.6
94. 26266
79.2840.1
81.4010.1
8-.6220.1
86 28=0.1
89.61=0.1
91.49-2.0
ASPEST with DANN (ours)
78.14#0.1
83.33013
93.612000
79.33=0.1
81 33.2320011
81.210011
88.3020.2
59.3200.1
91.262000
ASPEST with CDAN (ours)
77.75-033
83.681015
94.442033
79.2710.0
81.001022
84.7620.1
88.3520.1
30.591000
91.41:000
Table 33: Results of evaluating DE with UDA and ASPEST with UDA on Otto.
The mean and of each metric
over three random runs are reported (mtan-sstd). All numbers are percentages. Bold numbers are superior results.
42


Reviews:
[{"review_id": "6ECLOC8lfP", "reviewer": "Reviewer_FWsG", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is well-written, methods seems solid, experiments are shown on a wide range of datasets, and improvements are consistent. There is both theoretical and empirical analysis of the proposed problem setting and method. Results are carefully studied and analyzed.\n\nMy biggest issue with the paper is that I don't know if the problem is actually something we see in the real-world. The experiments seem to be on synthetic shifts from one dataset to another. It would be much more compelling if there is a unique real-world task where this problem setting and method is actually required, or performance would be close to 0% or random. This would significantly improve the paper and address my concerns.\n\nAlso, I am not an expert in this area, but I am curious whether the baselines quoted and compared in the paper are sufficient, or are they designed to trivially fail (i.e baselines for active learning combined with separate baselines for selective prediction, which obviously fail when applied to this new problem combining both challenges). If so, then there should be a much more significant effort towards testing and analyzing strong baselines for this new problem.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a new learning paradigm called active selective prediction that combines selective prediction (not predicting on uncertain points) and active learning (deciding which datapoints to obtain human labels for). The authors' new framework aims to query more informative sample that the model is uncertain on, and propose an approach called ASPEST that utilizes ensembles of model snapshots with self-training with their aggregated outputs as pseudo labels. They show that this approach is quite effective on several datasets for selective prediction, active learning, and distribution shifts.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "see above weaknesses on real-world setting where we see this problem setting, new experimental results on this setting rather than artificially paired transfer datasets, and issues regarding baselines", "broader_impact_concerns": "no concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "oBNPevhnC6", "reviewer": "Reviewer_8iAq", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n1. The method proposed by the authors -- ASPESTS -- produces strong results compared to reasonable baselines\n2. The paper presents quite thorough experiments and ablations of the settings of interest.\n\n## Weaknesses\nBroadly, my primary concerns with this paper is that quite a few choices are not well motivated (at least from my read of the paper). There are several choices that I question : \n1. Carving out active selective prediction as a niche, separate domain.  Specifically, the authors present *active selective prediction* as separate from active learning because the points are chosen not only to improve the model but also the selection scoring function. However, this strikes me as quite a cosmetic difference, since one can always re-define the utility function for active learning to fold in performance on the scoring function too. Thus the claim of **we introduce a new machine learning paradigm** seems overstated.\n2. The motivation presented for why active domain adaptation (as a standalone approach) would not cover improvements in the selective predictor is unclear to me.  Specifically, the authors mention early in the intro that distribution shift can cause the selection predictor to fail due to miscalibration. It stands to reason that doing active domain adaption would fix the calibration problem (or if it doesn't, the authors provide no evidence of this), thus requiring minimal change to the selective predictor.\n3. The authors state in section 3.1 **These two approaches to involve humans have different objectives and thus, their\njoint optimization to best use the human labeling resources is not straightforward** -- however, it is unclear whether these two objectives are actually in opposition and not complementary. It would be good to either have an empirical or theoretical exploration of the compatibility of the two objectives. \n4. Figure 2 was quite hard for me to understand. The distribution of already labelled data is not shown and thus makes it hard to understand why the decision bound changes the way the authors suggest when the marked points are selected. Also, the particular configuration shown (where diversity is the selection metric, and equal number of points are chosen from each cluster \"where cluster is a group of points with the same latent label\" but some clusters are bigger so one would apriori expect more points to be sampled from that cluster leading to a much different decision boundary change than the authors propose) seems contrived and I am unsure if the setup reflects the vast majority of typical problems.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes to tackle the active learning and selective prediction problems together in tandem. \nThey specifically investigate the setting of distribution shift where models trained on the source domain tend to be miscaliberated thus resulting in poor coverage of the selective predictor and resultant poor performance of the joint system.\nThe paper propose a combination of deep ensembling and self-training when leveraging labelled points queried from active learning, resulting in ASPEST which shows strong performance in the settings they investigate.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Clarifications to Figure 2 as requested in above section\n2. Positioning of the paper should be reconsidered, else a stronger case should be made for why active domain adaptation approaches  (along with an appropriately chosen threshold for selective prediction) cannot be used (or compared to) in the evaluation settings the paper presented.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "atFKLtkm3H", "reviewer": "Reviewer_HHqS", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: The idea of this integration method looks interesting, a combination of different strategies, each addressing different aspects of the challenge. Self-training helps in adapting the model to new distributions, active learning efficiently expands the model's understanding of the new domain with limited labeled data, and selective prediction maintains prediction reliability in the face of uncertainty. An integrated approach, combining these elements, is often more effective in tackling distribution shifts in real-world scenarios.\n\nWeaknesses:\n\n1. This workflow induces too many hyperparameters. Although the authors do sensitive analysis on some hyperparameters, it is still not appropriate to introduce too many hyperparameters in the active learning process, since the validation dataset is insufficient, and the data scenarios vary.  The hyperparameters should be self-adaptive. For instance, in the joint training, in previous stages, the labeled data are insufficient, and the ensembles should be less reliable.  \n\n2. The use of ensembles and self-training might increase computational requirements. The authors only provide the running time of their methods, not comparison.\n\n3. This work demonstrates effectiveness w.r.t. distribution shift on some specific datasets like MNIST and SVHN, it may not establish the generalizability of ASPEST across a more diverse range of domains and data types.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The work presents ASPEST (Active Selective Prediction using Ensembles and Self-training). This method incorporates active learning with selective prediction to enhance accuracy and coverage in cases where the distribution of test data diverges from that of the training data (distribution shift). ASPEST queries highly informative samples from the shifted target domain, which significantly boosts the model accuracy and coverage breadth. Key to ASPEST's design is the use of ensemble model checkpoints, coupled with self-training techniques that leverage aggregated outputs as pseudo-labels.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. See the weaknesses part in the previous section.\n\n2. Add a notion table.\n\n3. Figure 1 could not become an overview of the proposed model, I cannot see how it works under the distribution shift data scenarios.\n\n4. The evaluation metrics are a mess, sometimes it is ACC>90, ACC>80, and sometimes ACC>70.\n\n5.  Is epochs=20 conduct a sufficient model training?\n\n6. It is better to provide evaluation metrics vs. labeling budget curves for different methods instead of using tables.\n\n7. The proposed method seems cannot give enough distinguishable model performance on standard datasets (without distribution shift problems), can add statistical tests.", "broader_impact_concerns": "This work uses self-training, there is always a risk of inherent biases in the training data being perpetuated or amplified by the model and result in fairness issues. And such biases can be accumulated by increasing the rounds.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Although not all the reviewers view the problem of "active selective prediction" as novel necessarily, all the reviewers are in agreement with respect to the manuscript - leaning accept. All requested changes were either made or discussed to satisfaction of the reviewers. The extensive experiments clearly show the improved performance of the proposed approach, ASPEST, as compared to baseline methods. The authors made edits to the manuscript to clarify any points of confusion. Hence, the recommendation is to accept the paper as is.
Decision: Accept (poster)