Pareto Optimization for Active Learning under Out-of-Distribution Data Scenarios
Authors: Xueying Zhan
Zeyu Dai
Qingzhong Wang
Qing Li
Haoyi Xiong
Dejing Dou
Antoni B. Chan
Venue: Accepted by TMLR ()

Abstract: Pool-based Active Learning (AL) has proven successful in minimizing labeling costs by sequentially selecting the most informative unlabeled data from large pool and querying their labels from an oracle or annotators.  However, existing AL sampling schemes may not perform well in out-of-distribution (OOD) data scenarios, where the unlabeled data pool contains samples that do not belong to the pre-defined categories of the target task. Achieving strong AL performance under OOD data scenarios presents a challenge due to the inherent conflict between AL sampling strategies and OOD data detection. For instance, both more informative in-distribution (ID) data and OOD data in an unlabeled data pool would be assigned high informativeness scores (e.g., high entropy) during AL processes. To address this dilemma, we propose a Monte-Carlo Pareto Optimization for Active Learning (POAL) sampling scheme, which selects optimal subsets of unlabeled samples with fixed batch size from the unlabeled data pool. We formulate the AL sampling task as a multi-objective optimization problem and employ Pareto optimization based on two conflicting objectives: (1) the conventional AL sampling scheme (e.g., maximum entropy) and (2) the confidence of excluding OOD data samples. Experimental results demonstrate the effectiveness of our POAL approach on classical Machine Learning (ML) and Deep Learning (DL) tasks.

Published in Transactions on Machine Learning Research (06/2023)
Pareto Optimization for Active Learning under Out-of- Distribution Data Scenarios
Xueying Zhan"
sjahan2-c@mmy.citnn...e
City University of Hong Kong
Zeyu Dai The Hong Kong Polytechmic University
pe..n.ddrddrcomnee..........
Qingzhong Wang Baidu Research
gingzuaagg&onttlook.com
Qing Li The Hong Kong Polytechnic University
cgli@ccomppolyyu..ek edw.l...edu.kk
Haoyi Xiong Baidu Research
haogj.zionngff@iieee.ry
Dejing Dou BCG X
dejingdou@gmail.com com
Antoni B. Chan
eehhan@cityy..duu..
City University of Hong Kong
Reviewed OpenReview: https: openrvveepe net/ forum? id= dxmcepSSYF
Abstract
Pool-based Active Learning (AL) has proven successful in minimizing labeling costs by sequentially selecting the most informative unlabeled data from large pool and querying their labels from an oracle or annotators. However, existing AL sampling schemes may not perform well out-of-distribution (OOD) data scenarios, where the unlabeled data pool contains samples that do not belong to the pre-defined categories of the target task. Achieving strong AL performance under OOD data scenarios presents challenge due to the inherent conflict between AL sampling strategies and OOD data detection. For instance, both more informative in-distribution (ID) data and OOD data in an unlabeled data pool would be assigned high informativeness scores (e.g., high entropy) during AL processes. To address this dilemma, we propose a Clonte-Carla Pareto Optimization for Active Learning (POAL) sampling scheme, which selects optimal subsots of unlabcled samples with fixed batch size from the unlabeled data pool. We formulate the AL sampling lask as multi-objective optimization problem and employ Pareto optimization based on two conflicting objectives: (1) the conventional AL sampling scheme (c.g., maximum entropy) and (2) the confidence of excluding OOD data samples. Experimental results demonstrate the effectiveness of our POAL approach on classical Machine Learning (ML.) and Deep Learning (DL.) tasks.
1 Introduction
In practical applications, obtaining large amounts of unlabeled data is relatively easy, but the process of labeling them be expensive and tine-consuming (Shen et al. 2004). AL aims to solve this problem it
"Wok is completed while Xueying Zhan at Baidu Research, Big Data Lab. TTe first two authors Xueying Zhan and Zeyu Dai are equally contributed.

---

Published in Transactions on Machine Learning Research (06/2023)
99
Minteshi disaace
Tody 1SALceodeoeeaa
D
0500
(a) Entropy score of ID and OOD data
(b) Scatter plot of TD and OOD data
(c) POAL vs. True PO
Figure 1: (a) Entropy distribution for ID and OOD data during AL processes on EX8 dataset. (b) Scatter plot of the AL score (entropy) and ID confidence score (negative Mahalanobis distance) of unlabeled data, A larger ID score indicates that data is more likely to be ID data. (c) An example of the difference in the selection between our POAL and True PO (POAL uses Monte-Carlo for approxination, while True PO makes final selection after transverse the whole search space) on EX8 dataset, where each element of Pareto set is selected subset with fixed batch size.
achieves higher model performance with less training data by soquentially selecting the most informative instances and then querying their labels from oracles/an aanelelonnossssoss (Zhan et al. 2021b). However, current. AL methods have primarily been evaluated on well-studied, simple, and clean datasets (Kothawade et al.) 2021 like MNIST (Deng) 2012), CIFAR10 (Krizhevsky et al. 2209)) etc. However, in real-life scenarios, when collecting unlabeled data, is common for unrkltnree data, referred as out-of-domain (OOD) data, to be mixed in with the task-related data. For instance, when the task involves classifying images of digits, images of letters might also be present in the unlabeled data pool (Du et al. 2021]. Most AL methods are not robust to OOD data scenarios. For example, Karamcheti et al. (2021) has demonstrated empirically that collective outliers hurt AL performances under Visual Question A Answering (VQA) tasks. Furthermore, selecting and querying OOD samples that are irrelevant to the target task can lead to wasted labeling costs and reduce the effectiveness the AL sampling process{Du et al., 2021). In this paper, we focus on the latter type of OOD data during AL, where the unlabeled data pool contains instances that do not belong to the classes of the target classification task.
There is a natural conflict between AL sampling process and OOD data detection. Most AL methods, especially uncertainty-base measures, prefer selecting data that are hardest to be classified by the current basic learner, e.g., high entropy of predicted class probabilities. However, in AL, if a basic learner (e.g., Neural Network with softmax output) performs poorly on ID data, it is more likely to provide non-informative predicted probabilities (i.e., close to uniform probabilities) on OOD data (Vaze et al., 2022]. During AL processes, the basic learner not well-trained due to the insufficient labeled data, and insufficient epochs in the case of deep AL. Therefore, the samples sclected by AL may contain high informative ID and OOD samples. For instance, consider the Maximum Entropy (ENT) approach for AL, which is classic uncertainty- based based
method [Twiss & Catlett 1994 Shannon 2001) that selects data samples whose predicted class probabilities have the largest entropy. Meanwhile, ENT is also a typical OOD detection method high entropy of the predicted class distribution suggests that the input may be OOD (Ren et al. 2019). Fig. T shows an example on EX88 data (Ng) 2008), which further illustrates this conflict, that is, a large percentage of data with high entropy scores are OOD, and thus ENT-bbsccd AL will likely to sclect OOD data for labeling. Thus, additional measures are needed to detect OOD samples so that they are not selected for AL. Fig. D is an example shows that the negative Mahalanobis distance (Lee et al. 2018 has certain negative correlation with entropy and thus could be used as an ID) confidence score.
Although the OOD problem has been demonstrated to affect AL in realistic scenarios (Karamcheti et al., 2021), there are only a few studies about it (Kothawade et al.) 2021 Du et al., 2021). SIMILAR (Submodular Information Measures Basod Active Learning) (Kothawade et al. 2021) adopted the submodular conditional mutual information (SCMI) function as the acquisition function. They jointly model the similarity between
unlabeled and labeled TD data sets and their dissimilarity with labeled OOD data sets. The estimation
2

---

Published in Transactions on Machine Learning Research (06/2023)
might not be accurate initially since labeled ID and OOD data sets are insufficient in the early stages of AL. Additionally, calculating SCMI on large-scale data is
CCAT, (Contrastive Coding AL) (Du et al. 2021) needs to pre-train extra self-supervised models like SimCLR (Chen et al. 2020), and also introduces Syper-parameters to trado-off between semantic and distinctive scores, whose values affect the final performance (see Section 4.3 in (Du et al. 2021 These two factors limit the range of its application.
In this paper, we advocate simultaneously considering the AL criterion and ID confidence when designing AL sampling strategies to address the above issues. Since the two objectives conflict, we define the AL sampling process under OOD data scenarios as multi-objective optimization problem (Seferlis & Georgiadis, 2004). Unlike traditional methods for handling multiple-criteria-baadd AL, such weighed-sum optimization (Zhan et al.) 2022a) or two-stage optimization (Shen et al. 2004 Zhan et al. 2022a): , we propose & novel and flexible batch-mode Pareto Optimization Active Learning (POAL) framework. The contributions and summarization of this paper are as follows:
1. We propose AL under OOD data scenarios within multi-objective optimization framework.
2. Our framework is flexible and can accommodate different combinations of AL and OOD detection methods according to various target tasks. Our experiments use ENT as the AL objective and Mahalanobis distance as ID confidence scores.
3. Naively applying Pareto optimization to AL will result in a Pareto Front with a non-fixed size, which can introduce high computational costs. To enable efficient Pareto optimization, we propose a Monte- Carlo (MC) Parelo oplimization algorithm for fexed-size batch-mode AL. MCPOAL could also provide near-optimal Pareto curves, as shown in Fig. [T.
4. Our framework works well on classical ML and DL tasks. We propose pre-selecting and early-stopping techniques to reduce the computational cost of large-scale datasets.
5. Our framework has no trade-off ayper-parameter for balancing AL and OOD objectives. It is crucial since i) AL is data- iata-insufficient, there might be no validation set for tuning parameters; ii) ayper-parameter tuning in AL can be label-cxpensive since every change of hyper-parameter causes AL to label new data, thus provoking substantial labeling inefficiency (Ash el al. 2020
2 Related Work
We discuss the related work in the following categories separately. We briefly introduced pool-based AL, OOD and distribution shift. data problems, how OOD data influence AL processes and multi-objection optimization.
2.1 Pool-based Active Learning
Pool-based AL has been well-studied in recent years (Settles) 2009;
Zhan et al., 2021b Ren et al.
2021
Zhan et al. (022a) and widely adopted in various tasks (Duong et al. 2018 Yoo & Kweon 2019 Dor et al,
2020 Haussmann et al.) 2020). Most AL methods rely on fixed heuristic sampling strategies, which follow
two main branches: uncertainty- and representative/ddveessi /dvvvviitvvbtbeeede measures (Ren et al., 2021 Zhan et al.)
2022ab. Uncertainty-basee approaches select data that maximally reduce the uncertainty of the larget basic
learner (Ash et al.
2020 Typical uncertainty-best based measures that perform well on classical ML tasks
like Benery-b-Coomittee
(QBC) (Seung et al. 1992). Bayesian Active Learning by Disagreement (BALD) Houlsby et al. 2011] have also been generalized to DL tasks (Wang & Shang 2014 Gal et al.) 2017 Beluch
et al. 2018 Zhha et al. 2022a). Representative /taiiiivvddvviitteeeee 'ddiversity-baasee
methods like k-Means (Zhan et al. 2022a
and Core-Set approach (Sener& & Savarese 2018) select a batch of unlabeled data most representative of the set.
Uncertainty Soocetainty/reepreeeen wwwreseetativee-aaseeeeee measures could be combined with weighted-sum or multi-stage optimization
(Zhan et al. 2022a). Weighted-sum optimization combines multiple objectives through linear combination with trade-off weights. (Yin ct al. 2017) is a typical method that adopts weighted-sum optimization, selecting the most uncertain and least. redundant samples as well as the most. diverse. Two-stage (or multi-slage) optimization first ranks data or selects a subset based on one objective, then makes final decisions based on
the other objective (Shen et al.] (2004] Zhao et al., 2019 Zhan et al., (022a). (Ash et al., 2020) computes gradient embedding of unlabeled data in the first stage (uncertainty), then clusters by KMeans++ in the second stage (diversity).
3

---

Published in Transactions on Machine Learning Research (06/2023)
2.2 Out-of- of-Distribution Data and Distribution Shift
OOD data detection. Detecting OOD data is vital in ensuring the reliability and safety of ML systems in real-life applications (Yang et al.] 2021 since the OOD problem severely influences real-life decisions. For example, in medical diagnosis, the trained classifier could wrongly classity a healthy OOD sample as pathogenic (Ren et al. 2019 Cui et al. 2020). Existing methods compute TD/OOD confidence scores based the predictions of (ensembles of) classifiers trained ID data, e.g., the ID confidence can be the entropy of the predictive class dist.ribution (Ren et al. 2019). Hendrycks & Gimpel (2017) observed that a well-trained neural network assigns higher softmax scores to ID data than OOD data, i.e, predictions for ID data have lower entropy. Follow-up work ODIN (Liang et al.} 2017) amplifics the effectiveness of OOD detection with softmax score by considering temperature scaling and input pre-processing. Others (Lee el al. 2018) propose simple and effective methods for detecting both OOD and adversarial data samples based on Mahalanobis distance. Due to its superior performance compared with other OOD strategies (see Table 1 in (Ren et al., 2019 ), our POAL framework uses Mahalanobis distance as the primary criterion for caleulating the ID confidence score. Nonetheless, our framework is general, and any ID confidence score could be adopted.
How OOD influences AL methods? As discussed in Section [ uncertainty-based AL measures like ENT are not robust to OOD data scenarios since OOD data are naturally difficult to be classified. seprrssnttaivveddiennsttyy
methods are also not robust OOD data since outliers/OOn samples are far from ID data and thus more likely to be selected first to cover the whole unlabeled data pool. Combined AL strategies that integrate uncertainty and representative- data. Weighted-sum optimization will select data with high uncertainty and representativeness scores, which are likely to be OOD. The multi-stage optimization adopts uncortainty-based measures to select a subset and then uses eepresentalive-basee large proportion of OOD data will be selected in the first stage. To address these issues, we proposed our POAL, which can be adapted to various AL sampling schemos with OOD data. We conducted experiments multiple highly-cited AL methods to validate our viewpoints, as shown in Section Fig.
measures will also be susceptible to OOD
measures for the final selection (or vice versa). For both variants, a
Distribution shift. AL with O0D data scenarios and AL with distribution shift scenarios share a common characteristic of gap between the true underlying data distribution and the estimated distribution from labeled and unlabeled data (Zhan et al. 2022b). The main difference is in the specific nature of the distribution gap. Distribution shift (aka datasct shift), refers to the discrepancy between the data distributions of the training and testing sets (or true underlying data distribution). There are several branches of AL with distribution shift: 1) where both labeled and unlabeled data are from the same domain but the labeled set is not sampled independently and identically distributed (i.i.d.) from the true data distribution due to the data collection process in AL, often addressed using fixed heuristics for data sampling (Beygelzimer et a 2009 Sawade el al. 2010; Ganti & Gray 2012 Farquhar et al., 2021 Zhan et al., (022b);; and 2) where the distribution shift occurs between the unlabeled data pool and the target tasks, such as when the target task is to classify Domestic Shorthair LLonhhiir cats while the unlabeled data pool contains various types of cats. In the former case, some works model the discrepancy between the labeled data and the true underlying data distribution, leveraging techniques like importance sampling to train unbiased basic learners. Other approaches focus on modeling the difference between the labeled and unlabeled sets or the full data pool to inform AL sampling strategics. For instance, Shui et al. (2020) utilizes unlabeled data information by training a discriminator to distinguish between labeled and unlabeled data sets.
In the second case, some works model the discrepancy between source and target domain of abbeled/unlabeled data pools to select more harget-task-relate data samples. Mahmood et al. (021 minimizes the Wasserstein distance between the unlabeled set and the set to be labeled as AL sampling strategy. dd Mathelin et al. (2021) discusses AL under domain shift for general loss functions and provides a generalization bound of the target risk based on localized discrepancy distances between sample points. Zhao et al. (2021) presents an AL framework for addressing label shift problems by combining importance weighting and sub-sampling techniques. Distribution shifts can be further applied to train better basic classifiers (Imberg et al. 2020)) Farquhar et al. 2021 Zhan et al. 2022b). In these AL approaches under distribution shift, distribution shift is useful for training AL. However, in AL with OOD data scenarios, OOD samples are hard to utilize for
4

---

Published in Transactions on Machine Learning Research (06/2023)
model training since OOD samples are not in the classes of interest of the target classification task. Thus, the current research aims to detect OOD samples and avoid sampling them in the first place.
2.3 Multi-objective Optimization
Many real-life applications require optimizing multiple objectives that conflict with each other. For example, in the sensor placement problem, the goal is to maximize sensor coverage while minimizing deployment costs (Watson et al. 2004). Since there is no single solution that simultaneously optimizes each objective, Pareto optimization can be used to find a set of " Parreo optimal" solutions with optimal trade-offs of the objectives (Micttinen] 2012]. docision maker can then select a final solution basod on their roquirements. Compared with traditional optimization methods for multiple objectives (e.g,, weighted-sum (Marler & Arora 2010));
Pareto optimization algorithms are designed via different weta-heuristies without any trade-off parameters (Zhou et al. 2011 Liu et al.. 2021). In our work, besides optimization of the two conflicting objectives (AL score and ID confidence), we also need to perform batch-mode subset selection. Pareto Optimization Subset Selection (POSS) (Qian et al. 2015) solves the subset selection problem by optimizing two objectives simultaneously: maximizing the criterion function and minimizing the subset size. However, POSS does not support more than one criterion and cannot work with fized subset size, thus not suitable for our task. Therefore, we propose Monte-Carlo POAL, which achieves: 1) optimization of multiple objectives; 2) no extra trade-off parameters for tuning; 3) subset selection with fixed size.
3 Methodology
In this section we introduce the problem definition of AL under OOD data scenarios, and details of POAL.
3.1 Problem Definition and Overview
We consider a general pool-based AL process for a K-class classification task with feature space X, label space Y e K} under OOD data scenarios. We assume that the oracle can provide fixed number of labels, denoted as budget B. When an OOD sample, which does not belong to one of the K classes, is qucried, the oracle will return an "OOD" labe to represent data outside of the specified task. We aim to select the most informative B instances with fewer OOD samples to obtain the best classification performance. To reduce the computational cost, we consider batch-mode AL, where batches of samples with fixed size b are selected and queried. We denote the AL acquisition function as axx; A), where A refers to AL sampling strategy, and the basic learner /classifier as fx(x). We denote the current labeled set as DD {(x; y)) and the large unlabeled data pool as D., = ]x The labeled data are sampled i. over In-Distribution (ID) data D, i.e, Di C D, and N < M. Under OOD data scenarios, active learners may query OOD samples whose labels are not in D. To simplify the problem settings, we ignore the queried OOD data and only add ID samples to Dj. Although OOD data could be used to improve model robustness, ( et al., 2021 demonstrated that this requires very large-scale OOD/ /noiss data (> 160k samples). However, in the AL settings, the size of D; is limited, especially for DOOD, and thus we do not, consider using OOD data for training in this paper. Selecting high-informative ID samples while preventing OOD data selection benefits more the AL processes.
Motivated by the natural conflict between AL and OOD data detection, illustrated in Fig. Dp, propose POAL outlined in Fig. 2 In each AL iteration, we firstly utilize the clean labeled set that only contains ID data for training a basic classifier fx( and constructing class- coasicondiiional Gaussian Mixture Models (GMMs) for detecting OOD samples in Du- Based on classifier fo(x), AL acquisition function axx; A) and the GMMs, we calculate the informativenesss/ uuneeraaint score ((x,)) and ID confidence score Mxx)) for each unlabeled sample x; in Du. Let s E (0, 1 be a binary vector whose element 8; = indicates that the i-th unlabeled sample is selected, and s; indicates unselected. In each AL stage, the multi-objective optimization goal is to find an optimal subset s* with b selected samples, that simultancously maximizes the informativenes; wtatrvveeeeesnematannnt and ID confidence score:
In our implementation, the label of OOD data is -1.
5

---

Published in Transactions on Machine Learning Research (06/2023)
Update
Orack/AAnnotaoor
Query
eO
Monte-Carlo
Basic Icarner
acquiilia fuaction
Unlabeled set
Optimal subset binory representations with sire &
Miniman
Figure 2: Overview of POAL. In each iteration, we train the basic learner and estimate GMMs (in DL tasks, we estimate multivariate Gaussian distributions with low-/high feature levels) calculate the AL and Mahalanobis distance- baseed ID confidence score for unlabeled data. We then construct POAL by randomly generating candidate solutions with fixed batch size and updating the Pareto set iteratively to get the optimal solution.
s" arg s ((ss) 8.l.
Σ ===bb b.
(1)
where U(s) EM s((xx) is shorthand for the aggregated AL score for the selected samples, and likewise for M(s). u(-) and M(+) will be introduced in Sections 3.2 and 3.3
3.2 Active Learning Score
AL selects data by maximizing the acquisition function a(x; A), i.e., x" = arg maxxeD, a(x; A) Gaal al., 2017). In our work, optimization. Thus, inspired by (Fredlund et al. 2010 Zhan et al. 2022b), we convert the acquisition function to querying density function via U(x:)
the AL scores for the whole unlabeled pool Du for the subsequent Pareto
o(x;A) In our experiments, we use ENT as our basic AL sampling strategy, and the acquisition function OENT(() = - Ex=r Poly k/x) log k[x)),
((x:A)
where po(y/x) is the posterior class probability using classifier fo(x). Our framework is flexible because it can incorporate various AL sampling strategics if their acquisition function can convert to querying density function. In general, AL methods that explicitly provide per-sample scores (e-g., class prediction uncertainty) or inherently provide pair-wise rankings among the unlaboled pool (e.g., k-Means) can convert to a querying density. Thus, many uncertainty- measures like QBC (Seung et al. 1992 BALD (Houlsby let al. 2011 Gal al. (217), and Loss Prediction Loss (LPL) (Yoo & Kweon 2019) are applicable since they explicitly provide uncertainty information per sample. Also, k-Means and Core-Set. (Sener & Savarese 2018) approaches are applicable since they provide pair-wise similarity information for ranking. On the other hand, AL approaches that only provide overall scores for the candidate subsets, such as Determinantal Point
Processes (Bryık et al. 2019 Zhan et al. 2021a), are unable to be adopted in our framework.
3.3 In- In-Distribution Confidence Score via Mahalanobis Distance
Intuitively, ID unlabeled data be distinguished from OOD unlabeled data since the ID unlabeled data should be closer to the ID labeled data (Dr) in the feature space. One possible solution is to calculate the minimum distance between an unlabeled sample and the labeled data of its predicted pseudo label provided by f(0) (Du et al., 2021). If this minimum distance large, the unlabeled sample will likely be OOD and vice versa. However, calculating pair-wisc distances between all labeled and unlabeled data is computationally expensive. A more efficient method to summarize the labeled ID data via a data distribution (probability density function) and then compute the distances of the unlabeled data to the ID distribution. Mukhoti et al.

---

Published in Transactions on Machine Learning Research (06/2023)
2021 adopts GMM generative classifier to estimate the feature-space density, which is further used to capture epistemic uncertainty. Using GMMs on ID data to calculate TD) confidence score for OOD detection in AL tasks has been shown to be a feasible solution in (Mukhoti et al. 2021). Furthermore, motivated by LLe et al. 2018 we adopt GMMs to represent the data distribution, and then our ID confidence score is based on Mahalanobis distance. AT for classical ML. tasks focuses on training basic learner So(x) with fixed feature representations, while AL for DI. jointly optimizes the feature representation X' and classifier fo(x) simultaneously (Ren al., 2021). Considering that these differences can influence the data distributions and further inftuence the Mahalanobis distance calculations, adopt different settings for classical ML and DL.
Classical ML tasks. We represent the data distribution estimated from Dj with GMMs. Specifically, since we have labels of Dt, we represent each class k with a class-conditional GMM,
Ck (K) PGMM (xly = >
"e N((( (+) )
(2)
where (mé Me > are the mixing coefficient, mean vector and covariance matrix of the c-th Gaussian component with the k-th class, and Nx Σ) is a multivariate Gaussian distribution with mean and covariance Σ. The class-conditional GMM for class k is estimated from the labeled data for class k, D( (x:ly; = k) using a maximum likelihood estimation (Reynolds) 2009) and the EM algorithm (Dempster et al.) [977). We then define the ID confidence score M(x) as the Mahalanobis distance between the unlabeled sample and its closest Gaussian component in any class,
(R) (K)
MML(x)
max - ||x Me x
(3)
DL tasks. For DL, we follow (Lee et al. 2018 to calculate M(x), since it makes good use of both low- and high-level features generated by the deep neural network (DNN) and also applics calibration techniques, e.g., inpuu-preprocessing (Liang al.] 2018) and feature ensembles Lee et al. (2018) to improve the OOD detection capability. Specifically, denote (x) as the output of the E-th layer of DNN for input x. A class-conditional Gaussian distribution with shared covariance is estimated for each layer and for each class k by calculating the empirical mean and shared covariance (A!()) Ee):
Exerro f((x), Σ1= - E Ex (f(x)))(((())))))))()) - P (f=))( -
(4)
Next we find the closest class for each layer l via he arg ming [lfé(x) - P IB, To make the ID and OOD data more separable, an input pre-processing step is applied by adding a small perturbation to x,
* = x + =sigu((<]](((( aB 2 (gg)) ), where controls the perturbation magnitude" Finally, the ID confidence
score is the average Mahalanobis distance for & over the L layers,
-1 + E max; - -lff((*) p. 112
MnL((x) =
5
Both AL and ID confidence score calculations will suffer from problems of insufficient training data. Insufficient training data is a common issuc in AL, which can lead to bias in the AL score calculations. This bias problem is most prevalent in the initial AL iterations, as the insufficiently trained basic learners may provide inaccurate predictions. As the amount of training data increases, the bias problem caused by insufficient data gradually diminishes (Zhan et al. 2022b). For OOD detection, the Mahalanobis distance calculation is influenced by the capability of the basic classifier used as feature extractor for estimating the Mahalanobis distance (in DL tasks), and the size of the ID labeled set. More data results in more accurate ID) data distribution estimation, so ID/OOD confidence score calculation may also suffer from insufficient training data in the initial stages.
3.4 Monte-Carlo Pareto Optimization Active Learning
Our framework for AL for OOD data contains two criteria, AL score and ID confidence score. As discussed in Section 2 weighted-sum and multi(two)-stage optimization are widely used in multiple-criteria AL problems.
2In our experiments we follow LLee al. 2018) and e 0.01.
7

---

Published in Transactions on Machine Learning Research (06/2023)
However, both have drawbacks when applied to AL for the OOD data scenario. In weighted-sum optimization, the objective functions are summed up with weight f/ @WeighteddSmm (x) = -)((x) + (1 m)), e.g., adopted by CCAL (Du et al. 2021). This introduces an extra trade-off parameter for tuning, but the true proportion of OOD data in Du is unknown; thus, we cannot tell which criterion is more important. Furthermore, due to the lack of data, there is not enough validation data for properly tuning v. For two-stage optimization (aaneveld & van der Vlerk 1990 Ahmed et al., 2004 Bindewald et al. 2020 Salo et al., 2022). a subset of possible ID samples is first selected with threshold M(x) < 6, and then the most informative samples in the subset are selected by maximizing (x). However, suppose 5 has not been properly selected. In that case, we might i) select OOD samples in the first stage (when 5 is too large), and these OOD samples will be more likely to be selected first due to their higher AL score in the second stage; ii) select ID samples that are close to existing samples (when 5 is loo small), and due to the natural conflict between the two criteria, the resulting subset will be non-informative thus influencing the AL performance (Zhao et al. 2019
Considering this contradiction between AL and OOD criteria, is vital to develop a combined optimization strategy that does not require manual tuning of this trade-off. Therefore, we propose POAL for balancing U(x) and M(x) automatically without requiring trade-off parameters. Consider one iteration of the AL process, where we must select b samples from Du. The scarch space size is the number of combinations M choose b, C(M, b), and the search is an NP-hard problem in general. Inspired by (Qian et al. 2015). we use a binary vector representation of candidate subset: s E 10M where 8; = 1 represents i-th sample is selected, and 5; = 0 otherwise. For the unlabeled set Du, we denote the vector of AL scores as u = [xxx)]] and ID confidence scores as m [), The two criteria scores for the subset s are then computed as oy (s)ss= = s u and (s) s m The ranking relationships between two candidate subsets s and s' are follows:
s' I denotes that s' is dominated by s, such that both scores for s' are no better than those of s, i.e.,
ou (s') S (s) and (s S оM
s' < s denotes that s' is strictly dominated by s, such that s' has one strictly smaller score (c.g.,
(s') < ou((s) and one score that is not better (e.g.; (sr)') S osr(s)).
s and s' are incomporuble if both is not dominated by s' and s' is not dominated by s.
Our goal (sce Eq. D is to find an optimal subset solution s that is not dominated by other subset solutions, with Σ -x/=b, b. The large search space makes traversing all possible subset solutions impossible. Thus propose Monte- Carlo POAL for fixed-size subset selection. Monte-Carr POAL iteratively generates a candidate solution S at random, and checks it against the current Pareto set P = (,,,,,,. ..... J. If there are no candidate solution in P that strictly dominates s, then s is added to P and all candidate solutions in P that are dominated by s are removed. Tn this way, a Pareto set P is maintained to include the solution(s) that dominate all other candidate solutions so far, while the solution(s) in P are incomparable with each other.
3.4.1 Early-stopping
We adopt an early-stopping strategy to automatically terminate POAL when there no significant change in P after many successive iterations, which indicates that a randomly generated s has little probability of changing P since most non-dominated solutions are included in P (Saxena et al., 2016). Firstly, need to define the difference between two Parelo sets, P and P'. We represent each candidate solution as a 2-dimensional feature vector, v(s) = (our(s), (s)). (MM The two Pareto sets can then be compared via their feature vector distributions, V(P) = {E(S)) and V(P') = {u(s"))eEP) We employed the maximum mean discrepancy (MMD) (Gretton et al. 2006) to measure the difference:
MMD(P.P') || Σ EE v(s) - PPT
E o(st)l
1
(6)
MMD is based on embedding probabilities in reproducing kernel Hilbert space H. and here we use the RBF kernel with various bandwidths (i.e., multiple kernels). At iteration 1, we compute the mean and standard deviation (SD) of the MMD scores in a sliding window of the previous sw iterations,
M,
Σ 8w int-s++ MMD(P:-1,P)) S. - 810
1 E
(MMD(P..,PP)) M.)
(7)
STo keep the two eriteria consistent, we normalize the ID confidence scores: t max(m) m.
8

---

Published in Transactions on Machine Learning Research (06/2023)
If there is no significant difference in the mean and SD after several iterations, e.g., M, and St do not change within two decimal points, then assume that P has converged, and we stop iterating.
3.4.2 Final selection
After obtaining the converged set P, we need to pick one s E P as the final solution (i.e., final subset for querying). Selecting a final solution is an open question in multi-objective optimization. Typical strategies [Chaudhari et al. 2010) to obtain a final solution include: (i) reformulate the problem as a single objective problem using additional information as required by the pocision-makee such as the relative importance or weights of the objectives, goal levels for the objectives, values functions, etc. (ii) Decision makers interact with the optimization procedure lypically by specifying preferences between pairs of presented solutions. (iii) Output representative set of non non-dominated solutions approximating the Pareto front, e.g., regarding the candidate solutions as data points, performing clustering, and outputting the centroid as a final solution.
In our work, we regard candidate solutions as data points and use the cluster centroid as the final solution:
s' arg milggep Egep I|s -s| arg maXsEP Exeps' ST
where we use the property s= Eq. 8 is equivalent to selecting the solution with a maximum intersection with other non-dominated solutions, i.e., we find the solution whose selected samples are commonly used in all other solutions in P.
Each solution in P represents particular ID-OOD trade-off since appropriate choice of weight 7 in the weighted-sum objective will lead to its selection. Thus, the maximum intersection operation selects the samples that are common to all settings of the ID-OOD tradc-off. In this sense, POAL bypasses the tuning of ID-OOD trade-off syper-parameters by selecting samples that work well for all trade-ofls.
POAL with early stopping and final selection strategies is summarized Algorithm
Algorithm Monte-Carke POAL with early stopping under OOD data scenarios.
Require: Unlabeled data pool D, with size M, criteria U(x) and M(x), batch size b, maximum repeal time T, population interval Piavi sliding window size Sa and final decision function F.
Ensure: A subset D, of D, where ID.|=6. 1: Let s [0] p= {s} and t= for 1,..., 7 do
Generate a random solution with condition Σ st if fz P such that s < then
Q = [z ss.
6:
P (P u
7:
end if
[Terminate the loop early if the pareto set p converged.
8:
if t +
then Calculate MMD, according Equation if MOD(L, Pinv) = 0 then
10:
11:
Calculate M, and S; with interval Plav according to Equation 7 respectively M, ROUND(M,, 2), S, = ROUND(S,, 2). if M, = M, = Mi-.. and S = S =Sp-.. then
12:
13:
14: Break
15: end if
16: end if
17: end if
18:
end for
19: return arg max F(s).
9

---

Published in Transactions on Machine Learning Research (06/2023)
3.4.3 Pre-selection for large-scale unlabeled data pool
We next consider how to use POAL on large datascts. The search space of size C(M, b) is vast for a large unlabeled data pool. We propose an optional pre-selection technique to decrease the search size by pre-selecting subset of candidates. We select an optimistic subset Suub from Du, based on the dominant. relationships between different samples from the original unlabeled data pool. Firstly, an initial Pareto front Prre. set of non-dominated data points, is selected according to two objectives U(x) and M(x). Since the size of Ppee Pre is not fixed and might not meet our requirement of the minimum pre-sclected subset size Sm; we iteratively update Duub by firstly adding the data points in Ppe pre to Daub: and then excluding Daub from Du for the next round of selection. The iteration terminates when 'Dsubl 2 8m- The pre-selected subset size 8m is set according to personal requirements (i.e., the computing resources and time budget). In our experiment, we set 8m 6b. The pscudo-code of POAL with the pre-sclection strategy summarized in Algorithm 2
Algorithm Pre-selecting technique on large-scale datasets.
Require: Unlabeled data pool Du. criteria U and M, minimum size of subset 8m of pre-sclection.
Ensure: A pre-selected subset Dsub of D., with [Daubl 2 8m.
1: Let Dauh = 0. i= =.0. 0.
2: while (Daubl < Sm do
3: Let Dp = 0.
4: for in 0,...: [Du] do
5: if fx E Pp such that D2 x then
Q= {x E Dpl x 3 )
D3 = (D, \ U
8:
end if
9:
end for
10: D_ = Du,, \ Dp.
11: Duub = Doub U Dp.
12: end while
13: return Daub
3.4.4 POAL vs. POSS
Our Monto-Carke POAL is inspired by Subset Selection by Pareto Optimization (POSS) (Qian et al. 2015 POSS selects a subset S C V such that f is optimized with the constraint |S| < k, where V {Xi, (XI,.... X,, is set of variables, is criterion function, and & is positive integer. POSS is NP-hard in general (Davis et al. 1997p. POSS has two conflicting objectives: maximizing the criterion function and minimizing the subset sizc. POSS first initializes Pareto set P with random solution and then selects solution from P to generate& a new solution by flipping each bit with probability 1/n. The new solution would add to P if it is not strictly dominated by any solution in P, and the solutions in P that is dominated by the new solution would be excluded. POSS repeats the itcration T times, and is proved to achieve the optimal approximation guarantee of 1/e) with E[T] < 2ek2n expected running time (Qian el al. 2015
Compared with POSS, POAL solves a more challenging problem, which has two essential differences: the number of objectives (or critcria in AL) and the constraint. Firstly, although POSS is in the form of bi-objective optimization, it supports only one criterion function, e.g., only Lxxx), while another criterion is that the subset size does not exceed b (i.e., Σ, 8 S b). In contrast, POAL needs to solve multiple criterion functions. Secondly, POSS solves the problem with the constraint |S| < k, where as our setting involves fixed-size solutions [S] k. Due to these differences, the theoretical guarantees in POSS cannot hold if we just adapt the POSS algorithm to our problem because: 1) Lemma 1 in POSS (Qian et al., 2015) is extracted from the proof of Theorem 3.2 in (Das& & Kempa, 2011]. which solves Subsct Selection problem as single objective optimization problem with greedy algorithm. If we simply add another objective, the theoretical
*Gencral multi-objective evolutionary algorithms generate population of solutions to update the Pareto set P each iteration (Saxena et al. 2016), but our POAL only generates one solution each iteration, which hardly improves P. Thus, two successive Pareto sets P P- and Pi at the iterations Pinv and are compared our POAT..
10

---

Published in Transactions on Machine Learning Research (06/2023)
optimal approximation guarantee (Theorem 1 in (Qian et al. 2015) cannot hold; 2) the POSS algorithm finds some optimal solution with the size |S| < k, which does not satisfy our solution requirement.
Considering the above issues, we devise our POAL to randomly generate candidate solutions with fixed batch sizes. As a result, this 1) controls all candidate solutions in P to have the same fixed size; 2) compared with POSS, Monte Carlo POAL does not depend on previous selections, which keeps the method free from initialization problems. Due to these differences in how candidate solutions are generated and the number of criteria, the theoretical bound the number of iterations T needed for convergence of POSS (Qian et al. 2015) cannot be applied to MCPOAL. Thus, when should the iterations of our Monte-Carke Carlo POAL terminate? QQan et al. 2017) has proved that the Parcto set empirically converges much faster than E[T] (sec Fig. 2 in 2017) and (Saxena et al. 2016), we also propose an early-slopping technique. For the calculation of E E in POAL, see Section A in Appendix.
QQaa et al. 2017)). Based on (Qian et al.
4 Experiments
In the experiments we aim to: 1) evaluate the effectiveness of POAL on both classical ML. and DL tasks under various scales of OOD data scenarios; 2) compare different multi-objective optimization strategics, i.c., weighted-sum, two-stage, and Pareto optimization; 3) observe how OOD data influence typical AL methods as discussed in Section 2
4.1 Experimental Design
Datasets. For classical ML lasks, we use pre-processed data from LIBSVM
CChang & Lin 2011
Synthetic data: EX8 uses EX8a as TD data and EX8h as OOD data N g 2008
Real-life data: Vowel (Asuncion & Newman 2007 Aggarwal & Sathe 2015 has 11 classes, and we use 7
classes as ID data and the remaining for OOD data. Letter (Frey & Slate
1991 Asuncion & Newman
2007) has 26 classes, and we use 10 classes (a-j) as ID data and the remaining 16 classes (k-z) as OOD data. We also construct 16 datasets with incrcasing ID:OOD ratios, denoted as letter(a-k), letter(a- (a-)ra,,))).... letter(a-z).
For DL tasks, we adopt the following image datasets:
CIFAR10 (Krizhevsky et al.) 2009) has 10 classes, and we construct two datascts: CIFAR10-04 splits the classes with ID:OOD ratio of 6:4, and CIPAR10-06 splits data with ratio as 4:6.
CIFAR100 (Krizhevsky et al. 2009) has 100 classes, and we construct CIFAR100-04 and C1FAR100-06
using the same ratios as CIFARIO.
Baselines. POAL helps existing AL methods select high informative ID data samples while preventing OOD data selection. In our experiments, we adopt ENT
basic AL sampling strategy. We compare against three baselinn methods considering OOD data: CCAL (Du et al. 202Tb, Deep Deterministic Uncertainty (DDU) (Mukhoti et al. 2021) with GMM density and SIMILAR (with FLVMI submodular function) (Kothawade et al. 2021). SIMILAR has many variants with different submodular functions. According to the experiments in [Kothaaade et al. 2021), SIMILAR (FLCMI) perform better on OOD data scenarios, but it is memory- /time-consuming. We compared POAL with SIMILAR (FLCCT) in down-sampled CTFAR10 datasot, followed the experimental settings in (Kothwwade al., 2021 as shown in Section C.2 in Appendix. We compare against five normal AL methods without OOD detection: ENT, BALD (Gal et al. 2017) and LPL (Yoo &
Kweon 2019) uncertainty- measures; K-Means representative- based measure and BADGE (Ash ct al. 2020) as combined strategy. From the aspect of multi-objective optimization, we compared MCPOAL against two alternative combination strategies: weighted sum optimization (WeightedSunn-)) using weights = {1.0, 0.2), where 1) refers to the ratio between the coefficients for the AL objective and OOD objective, and two-stage optimization (TwoStage) using threshold 8 = mean(m). Additionally, we add random sampling (RAND) and Mahalanobis distance (MAHA) as basclines. Finally, we report the oracle results of ENT where only ID data is selected first. (IDEAL-ENT), which serves ideal performance of POAL
11

---

Published in Transactions on Machine Learning Research (06/2023)
Implementation details. The training/test split of the datasets is fixed in the experiments, while the initial label set and the unlabeled set is randomly generated from the training set. Experiments are repeated 100 times for classical ML tasks and 3 times for DL tasks. We use the same basic classifier for each dataset, as shown in Table 7 in Appendix. To evaluate model performance, we measure accuracy and plot accuracy vs. budget curves to present the performance change with increasing labeled samples. To calculate average performance, we compute the area under the accuracy-budgee curve (AUBC) (Zhan et al. 2021b), with higher values reflecting better overall performance under varying budgets.
BXX4
viwved
lnderook) (Dvini miis 18c1
Icterboo) (D raio 10
100000
2030
Budgei
Budget
Budget
Bulget
(a) EX8
vowcl
(c) letter(a-k)
letter(a-n)
lettetoorr IDD main- 108
lnteetane ID:00 milo- 10:15
lemeeta)) 1D0 1916
ketter
Bald incore
extionn
Bedget
Bodget
Budgrt
(c) letter(a-r)
(f) letter(a-w)
(g) letter(a-z)
(h) (AUBC)
Figure 3: (a)-(g) are accuracy vs. budget curves for classical ML tasks. The AUBC performances are shown in perentheses in the legend. To observe the effect of increasing ID:OOD ratio on letter datasets, we plot AUBC vs. dataset curves (h).
CIFARIS 240D-00D unto 6.4)
CIFARIO DOID.ODD rallo 4.5)
CIBARICO OOD ntio 6:1
CIFARIDO 20DDDCDDDDDDDDDD (e80-4551
Budget
Budgei
Budget
Budgett
(a) CIFAR10-04
(b) CIFARIO-06
(c) CIFAR100-04
CIFAR100-06
CXID det alrcttd CIFARIO 04
DGD dats odeected CIFARIO.O 06
OOD redected CIFARIO 04
OND deta odfected CIEARIOO 06
1L (11D odlected
Rulio COOD Sauplea sefected
www.plac selected
0100
120g
400
100
Budget
Budgi
(c) CIFAR10-04 OOD
(f) CIFAR10-06 OOD
(g) CIFAR100-04 OOD
(h) CIFAR100-06 OOD
Figure 4: Results of POAL with Pre-Selection on DL datasets. (a-d) are accuracy vs. budget curves. We present the mean and standard deviation (in parentehsis) of AUBC. (e-h) plot the ratio of OOD samples selected.
12

---

Published in Transactions on Machine Learning Research (06/2023)
4.2 Experimental Result Analysis
4.2.1 Overall Performance
We present the overall performance of classical ML and DL tasks in Fig. 3 and Fig. respectively, We next analyze the overall performance from different aspects:
Performance on classical ML tasks. Considering each dataset (see data visualization in Appendix B.2 the idcal performance of POAL is shown on the synthetic datasct EX8 (Fig. Ba), the ID data is non- non-lincarly separable, and the OOD data is far from ID data. These properties make the calculation of M very accurate. Meanwhile, OOD and ID data close to the decision boundaries have high entropy scores. The curve of our POAL is the closest to IDEAL-ENT, which indicates that POAL selects the most informative ID data while excluding the OOD samples. For real-life classical ML datasets like vowel and letter (Fig. [ POAL also demonstrates its superiority. In letter, we fixed the ID data (letters a-j) and gradually increase the OOD data (letters k-z), and plot the result in Fig. 3h. We observe that except for IDEAL-ENT, all methods are influenced by increasing OOD data, e.g., the AUBC of RAND is 0.718 in Fig. 3r, 0.692 in Fig. 31. Although our method is also influenced by increasing OOD data, it is still superior to all baselines.
The performance of POAL is influenced by the Mahalanobis distance calculation, we observe from Fig. Fig. [] and Fig. g that the capability of Mahalanobis distance limited by the distinction between ID data distribution and OOD data distribution. EX8 has distinct ID/OOD data distributions (see Fig. 7 thus the Mahalanobis distance well distinguishes ID and OOD data, and reachos the optimal performance (Fig. 9h, MAHA has the same curve with IDEAL-ENT). However, for vowel and letler dataset, the ID/OOD data distributions are not as distinct as EX8 (see Fig. 7p-r), thus the performance of MAHA is influenced. Furthermore, it affects the performance of our POAL It makes our POAL behaves less perfectly on wowel and letter datasets than on the EX8 dataset. Similar conclusions appear in (Ren et al. 2019
Performance on DL tasks. POAL also works on large-scale data with DL tasks. Fig. shows the bccuracy-budget curves (along with the AUBC (acc) values) for CIFAR10 and CIFAR100 with ID:OOD ratios 6:4 and 4:6, together with the number of OOD samples selected during AL processes. Compared with normal AL methods, POAL outperforms both uncertainty-, representative baselines like LPL, BALD, k-Means and BADGE on all tasks. As shown in Fig. Fp-h, the three AL under OOD data scenario methods, POAL, CCAL and SIMILAR effectively reduce the amount of OOD data that is selected. Both POAL and CCAL perform better than normal AL sampling schemes (i.c., ENT, LPL, BALD, k-Means, BADGE and RAND) on both CIFAR10-04 and CIFAR10-06 dalasets.
pppresentativeediivertt---bbbs /iveessittttybeeeddd and combined AL
POAL outperforms CCAL, SIMILAR (FLVMI) and DDU by large margins on more challenging situations (i.c., ID:OOD ratio is 4:6), POAL selects fewer OOD samples and achieves better AUBC performance (c.g., the AUBC value of POAL on CIFAR100-06 is 0.525, while the highest baseline SIMILAR is 0.451). SIMILAR selects data samples close to labeled ID data and dissimilar to OOD data. It prevents selecting the OOD data, but the selected ID data may not be informative enough. CCAL utilized weighted-sum optimization to balance the informative and distinctive selections, and the trade-off parameter heavily affects its performance (also see Section 4.3 in (Du et al. 2027 ). The idea of DDU (GMM) is similar to MAHA, the difference is DDU (GMM) does not use shared covariance as Eq. 4 while MAHA constructs class-conditional data distribution that follows the multivariate Gaussian distribution. Overall, DDU selects less OOD data than MAHA, especially in early stages. But both two methods select more OOD samples than POAT, CCAL and SIMILAR.
To observe the efficiency of each AL sampling strategy in our experiments of DL tasks, we summarized the overall performance across all models and datasets we adopted, including the mean and standard deviation (SD) of AUBC performance and running time. We record the running time from the start of the AL process to the output of the final basic learner. From Table [ we can observe that our model outperforms all baselines (except for the ideal model - -Deell-EELL---A IDEAL-ENT) in terms of AUBC performance. For running time, POAL searches the near-optimal solution from the subset level, CCAL first extract semantic and distinctive features then calculates the wise distance between large-scale unlabeled and labeled data. This results in the high computational costs of CCAL and POAL The running time of POAL could be reduced by decreasing the
13

---

Published in Transactions on Machine Learning Research (06/2023)
sliding window size sw, although there may be some variations due to the Monto-Carlo sampling in MC POAL. Among POAL, CCAT, and SIMILAR (FIMMT), which are designed specific to AL under OOD data scenarios, the running cost of our POAL is affordable.
Figure also demonstrates the conflict between ID and OOD data in the unlabeled data pool within the current learning stage. The slope of the OOD-budget curve shows the degree of conflict between TD and OOD data, where a larger slope indicates a more severe conflict (more OOD data is sampled). Figure 4e&g have lower OOD ratios compared to Figure Hich, and we observed that standard AL measures, such as ENT, generally have higher slopes in situations with larger OOD ratios. This suggests that the degree of conflict is hcavier in situations with larger OOD ratios. Additionally, we found that the slopes for all basclines, including POAL, are higher in previous learning stages and slow down in the middle and final stages, indicating that the degree of conflict decreases as the learning stage progresses (Figure I-hh). Also note that at the beginning, all types of AL approaches are likely to encounter data insufficiency problems. Therefore, we first compared our AL approach with passive learning (RAND) and then compared our proposed method with other AL strategies. Our POAL consistently outperformed the baselines on DL lasks. In summary, while insufficient. training data can be problem in AL under OOD data scenarios, our approach effectively addresses this issuc and performs well in comparison to other methods by selecting the near-optimal subsets in the current learning stages.
POAL vs. other multi-objective optimization strategies. We compared our Pareto optimization against weighted-summ, two-stage optimization, and each single objective, i.e, ENT and MAHA. Both ENT and MAHA are ineffective when used as a single selection strategy under OOD scenarios in Fig Bp. ENT always selects OOD data with higher entropy until all OOD data are selected (FX8 contains 210 OOD samples), and it performs even worse than RAND. MAHA prefers to select ID samples first (see Appendix Fig, 9 but the data samples with smaller Mahalanobis distance are also easily classified and far from the decision boundary, and thus not informative. POAL outperforms the other combination strategies, WeightedSum for different 77 values and TwoStage, on all tasks. This demonstrates that joint Pareto optimization better handles the two conflicting criteria. Considering WeightedSum optimization with different v values, we observe from Figure 3 that 17 = 1.0, i.e., treat each objective with the same importance, performs better than other 1) settings on various tasks. This observation is consistent with (Du et al. 2021 Ebert et al. 2012).
POAL vs. normal AL. We run experiments on normal AL methods, including uncertainty-based (ENT, LPL and BALD), diversily-based (k-Means) and combined methods (BADGE). Although LPL. performs fairly well on standard CIFAR10 and CIFAR100 datasets (Yoo & Kweon 2019 Zhan et al. 2022a in OOD sccnarios, OOD data have larger predicted loss values and thus inffuence the selection. Similar phenomena were observed with BALD and ENT. The performances of k-Means are close to RAND since both methods sampled ID/OOD data with the same ratio. Although k-Means will not select. OOD samples first like basede measures, it still failed to provide comparable performances. POAL performs well by selecting fewer OOD samples during the AL iterations, as shown in Fig. [e- BADGE performed similar to k-Means. The first stage of BADGE calculates the gradient embedding per sample, and use k-Means++ for clustering. These operations results in both high informative ID and OOD data are mixed together with high gradient values, therefore, BADGE failed to distinguish ID/OOD data. In DL tasks, we adopted pre-selection to deal with large-scale datasets. Although it makes the search space smaller, i.e. changes the global solutions to local solutions, POAL-PS still performs significantly better than baseline AL methods.
4.3 Visualization of the Subset Selection of POAL
A direct way to validate the efficacy of POAL is to measure the difference between the selected subset by POAL and true Pareto set. However, on normal datascts, the scarching space of the optimal subset selection is too large, as analysed in Section 3.4 4 il is a combination-level, C(M,b). Therefore, we use the subset, selected by typical Pareto optimization (the operation is the same as the operation of the first round of pre-selection technique in Scction 3.4 to represent the optimal subset selection instead. We visualize the subset selection on data and multi-objective space, as shown in Fig. 5 Using typical Pareto optimization will: i) select subset with non-fixed subset. size, and ii) possibly include many OOD data samples since the
14

---

Published in Transactions on Machine Learning Research (06/2023)
Table Overall comparison across all AL sampling strategies of all DL tasks, including mean and standard deviation (SD) of AUBC performance and running time (in seconds) across three repeated trials. The experimental environment deployed on DDU and MAHA are different from other methods. To maintain fairness, we use symbol "_" instead of running times.
CIFARIO-04
CIFAR10-06
AUBC
Time
AUBC
Time
RAND
0.7500 (0.0014)
10805.3 (148.0)
0.8080 (0.0008)
5217.7 (348.0)
IDEAL-ENT
0.8007 (0.0029)
14578.0 (838.2)
0.8737 (0.0019)
9398.0 (696.4)
POAL (Sw 20)
0.7620 (0.0033)
62082.7 (7371.9)
"0.8400 (0.0029)
34946.0 (4860.2)
POAL (sw = 10)
0.7613 (0.0024)
56107.0
(3364.2)
0.8403 (0.0029)
47362.0 (188.7)
OČAL
0.7543 (0.0009)
76129.3 (5853.7)
0.8090 (0.0037)
35715.3 (328.5)
SIMILAR (FLVMI)
0.7397 (0.0012)
13942.7 (116.2)
0.7947 (0.0037)
8550.0 (231.3)
DDU (GMM)
0.7207 (0.0038)
0.7957 (0.0025)
ENT
0.7350 (0.0029)
TO155.3 (872.3)
0.7960 (0.0057)
5485.7 (655.4)
MAHA
0.7313 (0.0021)
0.8013 (0.0021)
LPL
0.7510 (0.0071)
12384.7 (1555.9)
0.7873 (0.0103)
3783.3 (326.9)
BADGE
0.7437 (0.0029)
41646.0 (12723.5)
0.8063 (0.0037)
17539.3 (628.0)
KMeans
0.7440 (0.0014)
18785.7 (2571.3)
0.8100 (0.0022)
8627.3 (407.2)
BALD
0.7480 (0.0022)
10478.3 (679.7)
0.8060 (0.0022)
4158.3 (431.1)
TwoStage
0.7300 (0.0029)
8342.3 (1059.4)
0.7970 (0.0028)
5524.7 (821.3)
WeightedSum- 1.0
0.7337 (0.0019)
9489.7 (454.5)
0.8033 (0.0033)
2006.3 (13.8)
WeightedSum-5.00
0.7327 (0.0029)
11892.3 (1733.9)
0.8060 (0.0045)
7998.3 (81.6)
WeightedSSm-0.22
0.7340 (0.0045)
8249.3
(2019.2)
0.8063 (0.0012)
5751.0 (795.3)
CIFARI00-01
CIFAR
AUBC
Time
AUBC
Time
RAND
0.4560 (0.0016)
11563.3 (985.5)
0.4453 (0.0026)
11075.3 (1198.2)
IDEAL-ENT
0.5250 (0.0008)
17736.7 (1694.7)
0.5707 (0.0017)
19098.0 (1458.8)
POAL (s.. 20)
0.4807 (0.0009)
66880.7
(1583.5)
0.5253 (0.0005)
62762.0 (4277.4)
POAL (s... = 10)
0.4797 (0.0017)
4932.3 (1891.2)
0.5270 (0.0008)
55955.3 (5162.4)
OCAL
0.4400 (0.0008)
21253.0
(2ī71.9)
0.4467 (0.0024)
65303.0
(1676.6)
SIMILAR (FLVMI)
0.4377 (0.0026)
20732.7
(662.1)
0.4510 (0.0024)
13137.0 (396.7)
DDU (CMM)
0.4510 (0.0022)
0.4483 (0.0012)
ENT
0.4267 (0.0034)
T1202.3 (2484.7)
0.4100 (0.0036)
T0940.0 (1318.4)
MAHA
0.4283 (0.0026)
0.4090 (0.0008)
LPL
0.4140 (0.0037)
17797.0 (4093.8)
0.4087 (0.0042)
5633.0 (252.3)
BADGE
0.4530 (0.0000)
58877.7
(13876.3)
0.4430 (0.0008)
58046.3 (17296.9)
KMeans
0.4527 (0.0005)
349113
(1121.3)
0.4413 (0.0021)
17465.0 (3570.3)
BALD
0.4467 (0.0040)
14699.0 (2675.9)
0.4313 (0.0061)
4574.3 (140.5)
TwoStage
0.4344 (0.0021)
11484.3 (4360.1)
0.4137 (0.0017)
3234.3 (91.1)
WeightedSum-L1 1.0
0.4267 (0.0025)
12722.0 0 (4164.8)
0.4103 (0.0005)
9455.3 (468.5)
WeightedSum-5ö.
0.4287 (0.0039)
14822.3 (6249.7)
0.4073 (0.0050)
11309.3 (1303.4)
WeightedSum-0.2
0.4290 (0.0022)
11736.7 (4394.3)
0.4097 (0.0033)
8865.7 (1230.1)
entropy score of OOD data samples are relatively large. In contrast, POAL selects more informative ID data and less OOD samples based on current basic learners in Fig. 5
We conduct a toy experiments to show whether the subset selection generated by POAL similar to the output generated by True Pareto Optimization for subset selection, based on exhaustive search, as shown in Figures 6 and [... We down-sampled the unlabeled data pool to size of 50 and set the batch size as 5. The resulting search space is of size C(50, 5) = 2, 118, 760, which is acceptable. We traverse the whole search space and use the same Final-sclection method as in POAL to generate single final solution. As shown in Fig. D. we observe that the candidate Pareto set P generated by POAL is close to the selection by True Pareto Optimization. In v, the solutions generated by POAL are located on Pareto Curve (or very close to the Pareto Curve), in which Pareto Curve is generated by the True Parcto Optimization. Observing Fig- Gn-f, the subset selection generate by POAL is better. This is because True Pareto Optimization finally selects the subset, with higher entropy and lower TD score, which might select. OOD samples.
15

---

Published in Transactions on Machine Learning Research (06/2023)
detu Dinggtt Aae
Epasse
(a) Typical Pareto Optimization
(b) POAL
(c) Typical Pareto Optimization
(d) POAL
Figure 5: The subset selection generated by POAL and Typical Pareto Optimization. The selection results are presented data space and multi-objective space respectively. The experimental settings are the same as Fig.
Optiziatn
POAL
Eiti I8 posticc dea II empolaen dia
berdtiv da
ID pecairc das anuahen
xcicctd
(a) True Pareto Optimization
(b) Typical Pareto Optimization
(c) POAL
Pactu Optiniatio
Typica Paodo Optisiation
puthic
IB Phepp dis
bith
Ara
(d) True Pareto Optimization
Typical Pareto Optimization
(f) POAL
Figure 6: The subset selection generated by POAL, True Pareto Optimization (via exhaustive search) and Typical Pareto Optimization down-sampled EX8 dataset. The selection results are presented on data space and multi- objective space respectively. a-f are based on single data point level. The x-axis and y-axis are l and M respectively.
4.4 Sensitivity Analysis
In our work, we have a hyper-parameter 8 (sliding window size) control the early stopping condition. Large sw applies more strict carly stop condition and vice versa. If there is no significant change within Sm Pinv iterations, then we can stop early. In our experiments, set 8. = 20. In this section, we conduct. ablation study to see if less strict more strict conditions inftuence the final performance, 10 (less strict) does not affect the model performance (there is no significant difference of AUBC performance between 8., 20 and s = 10). Less strict. early stopping also usually requires less running time, e.g., in CIFAR100-04 dataset, we have 66,880.7 seconds average running time with sw 20 and 4.932.3 seconds average running time with sw = 10.
shown
in Table 1 3rd and 4th rows of each table. We found that
Besides Sw, we have another hyper-paramett sm, which controls the minimum pro-selected subset size. Sm is designed to adapt to various computational resource constraints, but not for controlling the AL/OOOD
16

---

Published in Transactions on Machine Learning Research (06/2023)
trade-off. Larger 8m can be used when more computational resources/time are available, while smaller Sm save computational cost but may lead to slightly sub-optimal solutions. In our experiments, we set = 6b based on our computational resource situation, which works well in practice.
Although the numbers of prree-pprameterss of POAL are similar to the weighted-sum optimization, their purposes are fundamentally different. In the weighted-sum optimization, hyper-parameters relate to the weighting of the two objectives, and thus significantly influence the result. In contrast, the two hyper- parameters of POAL mainly affects the initialization and stopping criteria, possibly resulting in sub-optimal approximate solutions. Note that we do not need to tune these parameters for various tasks.
about the efficient approximation (concerning computational resources), which
5 Conclusion
In this paper, we proposed an effective and flexible Monte Carlo POAL framework for improving AL when encountering OOD data the unlabeled data pool. Our POAL framework i) incorporates various AL sampling strategies; ii) handles large-scale datasets; iii) supports batch-mode AT, with fixed batch size; iv) does not require tuning a trade-off hyper-paramest and v) works well on both classical ML and DL tasks. Future work could improve the OOD detection criterion by introducing OOD-specifie feature representations and other techniques for better distinguishing ID/OOD distributions.
Author Contributions
The first two authors, Xucying Zhan and Zeyu Dai, are cqually contributed this paper. Xucying Zhan contributed the model implementation, draft manuscripi preparation, analysis and interpretation of results and study conception and design parts. Zeyu Dai contributed the study conception and design, analysis and interpretation of results and draft manuscript preparation parts.
Acknowledgments
The work supported by Strategic Research Grant from City University of Hong Kong (Project No. 7005840).
References
Charu C Aggarwal and Saket Sathe. Theoretical foundations and algorithms for outlier ensembles. Acm sigkdd explorations newsletter, 17(1):24-47, 2015.
Shabbir Ahmed, Mohit Tawarmalani, and Nikolaos V Sahinidis. A finite branch-and-boum algorithm for two-stage stochastic integer programs. Mathematical Programming, 100(2):355-377, 2004.
Jordan T. Ash, Chicheng Zhang, Akshay Krishnamurthy, John Langford, and Alekh Agarwal. Deep batch active learning by diverse, uncertain gradient lower bounds. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
Arthur Asuncion and David Newman. Uci machine learning repository, 2007.
William Н Beluch, Tim Genewein, Andreas Nürnberger, and Jan M Köhler. The power of ensembles for active learning in image classification. In Proceedings of the IEEE conference computer vision and pollern recognilion, pp. 9368 93777,77 2018.
Alina Beygelzimer, Sanjoy Dasgupta, and John Langford. Importance weighted active learning. In Proceedings of the 26th annual international conference machine learning, pp. 49 49-56, 2009.
Viktor Bindewald, Fabian Dunke, and Stelan Nickel. Modeling multi-stage decision making under incomplete and uncertain information. Technical report, Technical report, 2020.
Erdem Bayik, Kenneth Wang, Nima Anari, and Dorsa Sadigh. Batch active learning using determinantal point processes. Xio preprint ar x 1::699060000 996600976, 2019.
17

---

Published in Transactions on Machine Learning Research (06/2023)
Chih-Chung Chang and Chih-Jen Lin. Libsvm: a library for support vector machines. ACM transactions intelligent systems and technology (TIST), 2(3):1 2011.
PM Chaudhari, RV Dharaskar, and VM Thakare. Computing the most significant solution from pareto front obtained multi-objective evolutionary. International Journal of Advanced Computer Science and Applications, 1(4), 2010.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597- 1607. PMLR, 2020.
Yufei Cui, Wiguannan Yao, Qiao Li, Antoni B Chan, and Chun Jason Xue. Accelerating monte carlo bayesian prediction via approxinating predictive uncertainty over the simplex. IEEE Transactions Neural Networks and Learning Systems, 2020.
Abhimanyu Das and David Kempe. Submodular meets spectral: greedy algorithms for subset selection, sparse approximation and dictionary selection. In Proceedings of the 28th International Conference on International Conference on Machine Learning, pp. 1057 1064, 2011.
Geoff Davis, Stephane Mallat, and Marco Avellaneda. Adaptive greedy approximations. Constructive approzimation, 13(1):57 98, 1997.
Antoine de Mathelin, François Deheeger, Mathilde Mougeot, and Nicolas Vayatis. Discrepancy-basee active learning for domain adaptation. In International Conference Learning Represenlations, 2021.
Arthur P Dempster, Nan M Laird, and Donald B Rubin. Maximum likelihood from incomplete data via the em algorithm. Journal of the Royal Statistical Sociely: Series (Methonlologica)) 39(1):1-22 1977.
Li Deng. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, (6)4:1111142, 2012.
Liat Ein Dor, Alon Halfon, Ariel Gera, Eyal Shnarch, Lena Dankin, Leshem Choshen, Marina Danilevsky, Ranit Aharonov, Yoav Katz, and Noam Slonim. Active learning for bert: an empirical study. In Proceedings of the 2020 Conference on Empirical Melhods in Natural Language Processing (EMNLP), pp. 7949-799 1969,966, 2020.
Pan Du, Suyun Zhao, Hui Chen, Shuwen Chai, Hong Chen, and Cuiping Li. Contrastive coding for active learning under class distribution mismatch. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8927-8936, 2021.
Long Duong, Hadi Afshar, Dominique Estival, Glen Pink, Philip R Cohen, and Mark Johnson. Active learning for deep semantic parsing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 43 48, 2018.
Sandra Ebert, Mario Fritz, and Bernt Schiele. Ralf: reinforced active learning formulation for object class recognition. In 2012 IEEE Conference on Computer Vision and Patlern Recognition, pp. 3626 3633. IEEE, 2012.
Sebastian Farquhar, Yarin Gal, and Tom Rainforth. On statistical bias in active learning: How and when to fix it. arXiv preprint arXio:2101. 11665, 2021.
Richard Fredlund, Richard M Everson, and Jonathan E. Fieldsend. A bayesian framework for active learning. In The 2010 International Joint Conference on Neural Networks (IJCNN), pp. 1 IEEE, 2010.
Peter W Frey and David J Slate. Letter recognition using holland-style adaptive classifiers. Machine learning, 6(2):161 182, 1991.
Yarin Gal, Riashat Islam, and Zoubin Ghahramani. Deep bayesian active learning with image data. In International Conference on Machine Learning, pp. 1183 1192. PMLR, 2017.
18

---

Published in Transactions on Machine Learning Research (06/2023)
Ravi Ganti and Alexander Gray. Upal: Unbiased pool based active learning. In Artificial Intelligence and Statistics, pp. 422-431. PMTRR, 2012.
Arthur Gretton, Karsten Borgwardt, Malte Rasch, Bernhard Schölkopf, and Alex Smola. kernel method for the two-sample-problem. Advances in newral information processing systems, 19, 2006.
Willem K Klein Haneveld and Maarten H van der Vlerk. Stochastic integer programming: General models and algorithms. Annals of operations research, 85:39, 1999.
Elmar Haussmann, Michele Fenzi, Kashyap Chitta, Jan Ivanecky, Hanson Xu, Donna Roy, Akshita Mittel, Nicolas Koumchatzky, Clement Farabet, and Jose M Alvarez. Scalable active learning for object detection. In 2020 IEEE intelligent vehicles symposium (tv), pp. 1430- 1435. IEEE, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on compuler vision and pollern recognilion, pp. 770-778 2016.
Dan Hendrycks and Kevin Gimpel. baseline for detecting misclassified and out-of-listribution examples in neural networks. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.nett net, 2017.
Neil Houlsby, Ferenc Huszár, Zoubin Ghahramani, and Máté Longyel. Bayesian active learning for classification and preference learning. arXiv preprint arXiv:1112.5744 2011.
Henrik Imberg, Johan Jonasson, and Marina Fisk. Optimal sampling in unbiased active learning. In International Conference on Artificial Intelligence and Statistics, pp. 559-569. PMLR, 2020.
Siddharth Karamcheti, Ranjay Krishna, Li Fei-Fei, and Christopher D Manning. Mind your outliers! investigating the negative impact of outliers on active learning for visual question answering, arXiv preprint XXix:2107.02391 2021.
Andreas Kirsch, Sebastian Farquhar, Parmida Atighehchian, Andrew Jesson, Frederic Baachaud-Charron and Yarin Gal. Stochastic batch acquisition for deep active learning. arXiv preprint arXiv:2106. 12059, 2021.
Suraj Kothawade, Nathan Beck, Krishnateja Killamsetty, and Rishabh Iyer. Similar: Submodular information measures based active learning realistic scenarios. Advances in Neural Information, Processing Systems, 34, 2021.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of- distribution samples and adversarial attacks. Advances in neurol information processing systems, 31, 2018.
David D. Lewis and Jason Catlett. Heterogencous uncortainty sampling for supervised lcarning. In Machine Learning, Proceedings of the Eleventh International Conference, Rulgers Universily, New Brunswick, NJ, USA, July 10-13, 13, 1994, pp. 118-156. Morgan Kaufmann, 1994.
Shiyu Liang, Yixuan Li, and Rayadurgam Srikant. Enhancing the reliability of distt---iittibuution image detection in neural networks. arXio preprint Xrxin 1XX::1706.0000 706602690, 2017.
Shiyu Liang, Yixuan Li, and R. Srikant. Enhancing the reliability of neural networks. In 6th International Conference on Learning Representations, ICLR 2018, Vancoweer, BC, Canada, April 30 -aay 8, 2018, Conference Track Proceedings. OpenReview.net, 2018.
image detection in
Shengcai Liu, Ning Lu, Cheng Chen, Chao Qian, and Ke Tang. Hydratext: Multi-objective optimization for adversarial textual attack. arXiv preprint or arXhv:2111.01528, 2021.
Rafid Mahmood, Sanja Fidler, and Mare T Law. Low budget active learning via wasserstein distance: An integer programming approach. Xix preprint 2i::2106.02000 2021.
19

---

Published in Transactions on Machine Learning Research (06/2023)
R. Timothy Marler and Jasbir S Arora. The weighted sum method for multi-objective optimization: new insights. Structural and mullidisciplinary oplimizalion, 41(6):853 862, 2010.
Kaisa Miettinen. Nonlinear mullioojecclive oplimization, volume 12. Springer Science & Business Media, 2012.
Jishnu Mukhoti, Andreas Kirsch, Joost Amersfoort, Philip HS Torr, and Yarin Gal. Deep deterministic uncertainty: A simple baseline. arXiv e-prints, pp. arXiv- 2021.
Andrew Ng. Stanford cs229 machine learning andrew ng. 2008.
Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. Scikit-learn: Machine learning in python. the Journal of machine Learning research, 12:2825-2888 28300, 2011.
Chao Qian, Yang Yu, and Zui-Hua Zhou. Subset selection by pareto optimization. Advances in neural informalion processing systems, 28, 2015.
Chao Qian, Jing- ChesCChng Shi, Yang Yu, Ke Tang, and Zhi-Hua Zhou. Optimizing ratio of monotone set functions. In IJCAI, pp. 8606-2612 2017.
Jie Ren, Peter Liu, Emily Fertig, Jasper Snoek, Ryan Poplin, Mark Depristo, Joshua Dillon, and Balaji Lakshminarayanan. Likelihood ratios for out-of-distribution detection. Advances in Neural Information Processing Systems, 32, 2019.
Pengzhen Ren, Yun Xiao, Xiaojun Chang, Po- Yao Huang, Zhihui Li, Brij B Gupta, Xiaojiang Chen, and Xin Wang. survey of deep active learning. ACM Computing Surveys (CSUR), 54(9))1-40) 2021.
Douglas A Reynolds. Gaussian mixture models. Encyclopedia of biometrics, 741(659-663), 2009.
Ahti Salo, Juho Andelmin, and Fabricio Oliveira. Decision programming for mixed-integer multi-stage optimization under uncertainty. European Journal of Operational Research, (99(2)5500-56)) 2022.
Christoph Sawade, Niels Landwehr, Steffen Bickel, and Tobias Scheffer. Active risk estimation. In ICML, 2010.
Dhish Kumar Saxena, A. Sinha, João A. Duro, and Q. Zhang. Entropy-based termination criterion for multiobjective evolutionary algorithms. IEEE Transactions on Evolutionary Computation, 20(4):485-498, 2016.
Panos Seferlis and Michael C Georgiadis. The integration of process design and control. Elsevier, 2004.
Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Conada, April 30 May 2018, Conference Track Proceedings. OpenReview.net, 2018.
Burr Settles. Active learning litcrature survey. 2009.
H Sebastian Seung, Manfred Opper, and Haim Sompolinsky. Query by committee. In Proceedings of the fifth annual workshop Computational learning theory, pp. 287-294, 1992.
Claude Elwood Shamnon. A mathematical theory of communication. ACM SIGMOBILE mobile compuling and communications review, 5(1):3-55 55, 2001.
Dan Shen, Jie Zhang, Jian Su, Guodong Zhou, and Chew Lim Tan. Multi-criteria--asse active learning for named entity recognition. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics, 21-26 July, 2004, Barcelona, Spain, pp. 589-396. ACL, 2004.
Changjian Shui, Fan Zhou, Christian Gagné, and Boyu Wang. Deep active learning: Unified and principled method for query and training. In International Conference on Artificial Intelligence and Statístics, pp. PMI.R, 2020.
20

---

Published in Transactions on Machine Learning Research (06/2023)
Sagar Vaze, Kai Han, Andrea Vedaldi, and Andrew Zisserman. Open-set recognition: A good closed-set classifier is all you need. orXiv preprint arXiv:21 10. 10.06207, 2021.
Dan Wang and Yi Shang. A new active labeling method for deep learning. In 2014 International joint conference neural networks (IJCNN), pp. 112-119. TEFE, 2014.
Tean-Paul Watson, Harvey J Greenberg, and William E. Hart. multiple-objective analysis of sensor placement. optimization in water networks. In Critical transitions in water and environmental resources management, pp. 1-10. 2004.
Hongxin Wei, Lue Tao, Renchunzi Xie, and Bo An. Open-set label noise can improve robustness against inherent label noise. Advances in Newral Information Processing Systems, 34:7978 7992, 2021.
Jingkang Yang, Kaiyang Zhou, Yixuan Li, and Ziwei Liu. Generalized out-of-distribution detection: A survey. arxiv preprinl arXix:21 10.1 1394, 2021.
Changehang Yin, Buyue Qian, Shilei Cao, Xiaoyu Li, Jishang Wei, Qinghua Zheng, and Tan Davidson. Deep similarity-based batch mode active learning with apploration-exploiattiie In 2017 IEEE International Conference Data Mining (ICDM), pp. 575-584. IEEE, 2017.
Donggeun Yoo and In So Kweon. Learning loss for active learning. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 93 102, 2019.
Xucying Zhan, Qing Li, and Antoni B Chan. Multiplo-criteria based active learning with fixed-size determi- nantal point processes. arXiv preprinl arXie:2107.0000 2021a.
Xucying Zhan, Huan Liu, Qing Li, and Antoni B. Chan. A comparative survey: Benchmarking for pool-based active learning. In Proceedings of the Thirtieth International Joint Conference Artificial Intelligence, IJCAT-21, pp. 4679-4688 2021b.
Xueying Zhan, Qingzhong Wang, Kuan-hao Huang, Haoyi Xiong, Dejing Dou, and Antoni B Chan. A comparative survey of deep active learning. orXiv preprint NKi:22033.13550 2022a.
Xucying Zhan, Yaowei Wang, and Antoni B Chan. Asymptotic optimality for active learning processes. The 38th Conference on Uncertainty in Artificial Intelligence, 2022b.
Eric Zhao, Anqi Liu, Animashrce Anandkumar, and Yisong Yuc. Active learning under label shift. In International Conference on Artificial Intelligence and Statistics, pp. 3412-3420. PMLR, 2021.
Yu Zhao, Zhenhui Shi, Jingyang Zhang, Dong Chen, and Lixu Gu. A novel active learning framework for classification: Using weighted rank aggregation to achieve multiple query criteria. Pattern Recognition, 93: 581-602, 2019.
Aimin Zhou, Bo-Yan Yang Qu, Hui Li, Shi-Zheng Zhao, Ponnuthurai Nagaratnam Suganthan, and Qingfu Zhang. Multiobjective evolutionary algorithms: A survey of the state of the art, Swarm and evolutionary computation, 1(1):32-49, 2011.
21

---

Published in Transactions on Machine Learning Research (06/2023)
Appendix
A Computational Complexity Analysis of POAL
A.1 Monte-Carlo POAL
Our proposed Mlonte-Carlo POAL for fixed-size subset selection used unordered sampling with replacement. Denote the ground set A with size N. At each iteration, B (batch size, B < N) samples are randomly selected. So the search space contains = C combinations. The worst case of POAL is to find all Pareto optimal solutions P {si, $2, ... out of N samples). Let [P] = M (M S). Hence, the computational complexity is the expected number of iterations that all of the M Parcto optimal solutions have been selected at least once, which is calculated below:
$2.-.. } out of S combinations (each solution is subset of B distinct samples
At each iteration, there is a probability EzS that one particular solution s; is not selected. So after & iterations, there is probability ( that solution has not been selected. Thus, the probability that solution 8; has been selected at least once after k iterations is - ( For all M solutions, the probability that they have all been selected at least once after k iterations (1 - ( To oblain the probability that these M solutions have been selected at least once exactly after & iterations (i.e., not before & iterations), we need to subtract the probability that it is completed after k iterations: (1 ) (1 ) Hence, the expected number of iterations is:
B =
Σ k ((1
S 5-1
jby - -(1--11-1 -
5 )
)
8
k=0
5 - 1 yMM.
Σ (1
=
5
k=
For small datasets, the size is tolerable, so the ground set A is just the unlabeled dataset D, with size and the search space size 5= C In practice, we applied an carly-stopping strategy, which can obtain a good enough solution with much fewer iterations.
A.2 Pre-selection for large-scale datasets
We designed a pre-selection technique for large-scale datasets to reduce the search space of POAL. As introduced in the last paragraph in Section 3.4 we iteratively select Pareto optimal samples from the unlabeled data. The worst case is that all unlabeled samples are Parelo optimal samples, so the number of I n. The computational complexity of the pre-selection is O(n²).
comparisons is 0 1+2 ... (n
We pre-selected 8m aB samples out of n unlabeled samples (a is set according to the computing resources and time budget, we set a = 6 in our experiment) for the subsequent Monte-Carle POAL, Therefore, = aB < n, and the search space size S is reduced from C to C. Also, with an early-stopping strategy, our POAL can achieve good enough solution with an acceptable number of iterations in practice.
B Experimental Settings
B.1 Datasets
We summarize the datascts we adopted in our experiments in Table 2, including how to split the datascts (initial labeled data, unlabeled data pool, and test set), datasel information (number of categories, number of feature dimensions). and task-concerned information (maximum budget, batch size, numbers of repeated trials and basic learners adopted in each task). Primarily, we recorded the ID OOD ratio in each task. We down-sampled the letter dataset and controlled that each category only has 50 data samples in the training set.
22

---

Published in Transactions on Machine Learning Research (06/2023)
Table Datasets used the experiments.
Datneet
#
of
#
batchs
basic
peat
100
10
100
10
620
500
100
10
16
870
500
550
10
100
[
heter(e-o)
10
16
720
500
550
10:
100
LJ
heter(e-e)
10
16
770
500
040
10:
100
LR
beteerfogg/
10
16
30
720
500
050
10
10
100
LR
(rter(owe)
10
16
30
870
500
550
10
10
100
L.R
bsterfoos)
10
16
30
920
500
850
10: 0
10
100
LR
biter(a-1)
10
16
30
970
500
550
10: 11
10
100
LR
beler(oae)
10
16
30
1620
500
550
10 11
10
100
LR
letter(o-u)
10
1870
500
550
100
letter(osw)
10
1120
500
850
100
LR
teterroos)
1170
500
100
Nes18
CIFARzm--4
Ber
01F48/00006
49000
4000
20000
No518
DIMRR
RtxNe118
B.2 Visualization of Datasets
B.2.1 Classical ML tasks.
We visualize the datasets of classical MT. tasks using t-Distributed Stochastic Neighbor Embedding (t-SNE) 7 as shown in Fig. 7 We split ID/OOD data by setting OOD data samples as semitransparent grey dots to better observe the distinction between the ID and OOD data distributions. On EX8 dataset, there is a large distinction between ID and OOD samples, thus the calculation of Mahalanobis distance score would be very accurate, see Fig. Dh as example. In contrast, the gap between ID and OOD data is small, like letter(a-z), the ability of Mahalanobis distance score for distinguishing ID and OOD data will be limited, as shown in Fig. g
B.2.2 DL tasks.
We also visualize the discrepancy using MMD in DL tasks since the raw features of DL. tasks are hard to visualize, like Fig. 7 We calculate the MMD distance within the ID data and the MMD distance between the ID and OOD data. To extract the feature representations in DL lasks, we utilized a ResNet50 which is pre-trained on ImageNet dataser After extracting the features (the output of the penultimate layer of the pre-trained ResNet50) of ID and OOD samples, next caleulate the MMD scores, which is the same the MMD settings in our early-stopping technique. Calculating the MMD score between these ID and OOD data samples is memorr--cnsunninn (CIFAR10 and CIFAR100 have 50, 000 training sets). To calculate the ID-ID MMD score, we randomly sampled 1, 0000 x 2 instances in ID data and calculated the MMD score. We repeated this operation 100 times and took the average score. For calculating the ID-OOD score, we perform the same operation.
The MMD scores of CIFAR1O-04, CIFAR10-06, CIFAR100-04 and CIFAR100-06 are shown in Table 3 The distance between ID and OOD data are 1.5z/ 2.5x larger compared with the in-between distance of ID data, which indicates that using the Mahalanobis distance for distinguishing ID and OOD data is feasible on CIFAR10 and CIFAR100 datascts. Note that we did not calculate the MMD scores of ID-ID and ID-OOD during each stage of the AL process, since calculating the MMD scores on large-scale data with high- digeeniinnso feature representations is time- and memory-consuuinng
B.3 Baselines and Implementations
We introduce the baselines and the implementation details in this section.
html
httpe///w..at......iiiaaaaaaaaaa www.alaiiooooooooooooooomomoee.. www.et50......m html
23

---

Published in Transactions on Machine Learning Research (06/2023)
(a) EX8
(b) vowel
(c) letter(a-k)
(d) letter(a-I)
(c) letter(a-m)
(f) letter(a-n)
letter(a-o)
(h) letter(a-p)
Inteta
i1) lotter(a-q)
(i) letter(a-r)
(k) letter(a-s)
letter(a-t)
Inteeet
(m) lotter(a-u)
(n) letter (a-v)
(o) (tter(a-w)
(p) letter(a-x)
(q) letter(a-y)
(r) letter(a-z)
Figure 7: Visualization data sets in classical ML tasks by t-SNE. We set the semitransparent grey dots to represent. OOD data samples, the remaining colorful dots are ID data samples.
For the basic learner/classífee we adopted Gaussian Process Classifier (GPC), Logistic Regression (LR), and ResNet18 in our experiments. We did not choose GPC in the letter dataset since the accuracy-budget curves based GPC not monotonically increasing. The requirement of selecting a basic classifier is: that the
24

---

Published in Transactions on Machine Learning Research (06/2023)
Table MMD scores of ID-ID and ID-OOD data DL tasks.
ID-ID
ID-OOD
CIFAR10-04
0.00608
0.046445
CIFARJ0-06
0.00596
0.03565
CIFAR100-04
0.00603
0.01305
CIFAR100-06
0.00604
0.01659
basic classifier can provide soft outputs, that is, predictive class probabilities to calculate the uncertainty of each unlabeled sample, e.g., entropy. We use the implementation of the GPC with RBF kerne and L of scikit-lcamy library (Pedregosa et al. 2011) with default settings. For ResNet18, we employed ResNet18 (He et al. 2016) based on PyTorch with Adam optimizer (learning rate: le - 3) as the basic learner in DL tasks. In CIFARIO and CIFARI00 tasks, we set the number of training epochs as 30, the kernel size of the first convolutional layer in 18 is 3 (consistent with PyTTrrc CIFAR implementation Input pre-processing steps include random crop (pad 4), random horizontal flip (p 0.5) and normalization.
For the implementation of the classical ML baselines, we have introduced it in the main paper.
For the baselines in DL tasks, we use the implementation of DeepAL 10 Zhan et al. simple introductions of BALD, LPL and BADGE as follows:
2022a). We provide
Baycsian Active Learning by Disagrcements (BALD) (Houlsby et al., 2011 Gal et al.
2017): : chooses
the data points that are expected to maximize the information gained from the model parameters, i.e., the mutual information between predictions and model posterior.
Loss Prediction Loss (LPL) YYoo & Kweon 2019): it is loss prediction strategy by attaching small parametric module that is trained to predict the loss of unlabeled imputs concerning the target model by minimizing the loss prediction loss between predicted loss and target loss. LPI, picks the top b data samples with the highest predicted loss.
Batch Active learning by Diverse Gradient Embeddings (BADGE) (Ash al.) 2020 first measures uncertainty the gradient magnitude for the parameters in the output layer in the first stage; it then clusters the samples by k-Means++ in the second stage.
For CCAL, We utilize the open-source code implementation of CCAI IT We train SimCLR (Chen et al. 2020), the semantic/distinue two feature extractors with 700 epochs and a batch size of 32 on a single V100 GPU.
feature extractor, which is provided by CCAL's source code. We train the
We run all our experiments on single Tesla V100-SXM2 GPU with 16GB memory except for running SIMILAR (FLCMI) related experiments. Since SIMILAR (FLCMI) needs much memory. We run the experiments (SIMILAR on down-sampled CIFAR10) another workstation with Tesla V100-SXM2 GPU with 94GB memory in total.
C
Experiments
This section an extension of Section 4 in the main paper. We provide additional experimental results analysis and more experiments that do not appear in the main paper.
C.1 Complete Experiments of POAL on Classical ML Tasks
We present the complete accuracy vs. budget curves and numbers of OOD samples selected vs. budget curves during AL processes in Fig. and Fig. respectively.
http:///aaa....o..////////////as oag.aaaeee....i....e.oeoeaeaeiaa html httpp://sshuu.ccm/Si////////eeee htt.:thhuubcccoiu---a--e
25

---

Published in Transactions on Machine Learning Research (06/2023)
leterie 10:000
Ieterri-B ID,
Auste Eig
Bocke Tot
Tue
Prgeeit an o01
Budget
Bodget
Budget
Bulget
(a) EX8
(b) vowel
(c) letter(a-k)
(d) letter(a-l)
Ietortarr 10:000 exton
leturfa llb-oo rain- 10M4
lnderbbu)) (D-i000 main-
Ieteriopg) Ib rato
Budgel
Bodact
Bodact
Bulact
(c) letter(a-n)
(f) letter(a-m)
(g) letter(a-o)
(h) lettcr(a-p)
le ID:OOD rrio. 1877
brteetorr DD-OODDiiiin 1019
leterra eD-DO 10:9
Inaedion ID-000 rule- 19:10
Budget
Budact
Budect
Buddect
(i) lotter(a-q)
(i) latter(a-r)
(k) letter(a-s)
(I) letter(a-t)
lnterioo)) 1D:000 1 10:1
laterto-s) DD-0DD mie 10:12
Interfany ID-000 mile- 10:15
letertjes) DD000 mino 10:14
einc encioon
Thile liryy
Budget
Bodget
Budget
(m) lotter(a-u)
(n) letter (a-v)
(o) letter(a-w)
(p) letter(a-x)
latert-y) Db:00D
ID
Nectic 40
Bodact (q) letter(a-y)
Budact (r) letter(a-z)
Figure 8: Accuracy vs. budget curves for classical ML tasks. The AUBC performances are shown in parentheses in the legend.
26

---

Published in Transactions on Machine Learning Research (06/2023)
Nanbon
Bedgrt
(a) EX8
(b) vowel
(c) letter(a-k)
(d) letter(a-l)
lattarta
Inttvtti
Nandoe 0000
Nanbon GOSD
Tedggn
Bodgi
Bodge
Badgu
(e) letter(a-n)
(f) letter(a-m)
letter(a-o)
(h) letter(a-p)
Intetig)
Intreta-
42
COOOO
Natton OOTD
Nastan OOOD
Budge
Bodgt
Badge
Badget
(1) (otter(a-q)
(i) lotter(a-r)
(k) letter(a-s)
letter(a-t)
lamtrta
Intirta-dd
Narthea of OOOD vnriaa slaand
Santhe 0OC0
Bedget
(m) letter(a-u)
(n) letter(a-v)
(o) letter(a-w)
(p) letter(u-x)
Mashoo 0000
Bodgi
(q) letter(a-y)
(r) letter(a-z)
Figure 9: Numbers OOD samples selected vs. budget curves classical ML tasks during processes.
C.2 POAL SIMILAR (FLCMI)
In the main paper, we have compared the SIMILAR with FLVMI as submodular mutual information. Referring to its comparable performance CIFAR10 dataset (see Figure 5a and Figure 5b in (Kothawade
27

---

Published in Transactions on Machine Learning Research (06/2023)
2021 reference). We choose FLVMI as baselines in main paper since it is both time and memory efficient. However, FLVMI performs worse than FLCMM, since FL.CMI is specially designed for AL under OOD data scenarios. However, FLCMI is both time and consoryconsuuing, In (Kothawade et al.) they adopted down-sampled experiments. Although the partition trick could be applied to solve this tene/mmennor--oosuminng problem, the partition will influence the final performance due the randomness. To provide fair comparison. We conduct additional experiments on the down-sampled CTFAR10 dataset on SIMILAR (FLCMI), following the experimental settings in (Kothawade et al. 2021).
et al.
2021
The comparison experiments shown in Fig, 10 Besides POAL and SIMILAR (FLCMI), also provide more baselines as reference, i.c., IDEAL- -ENAL-ENT, ENT, Margin (Wang & Shang 2014) and RAND. As shown in Fig. 10 both SIMILAR (FLCMI) and POAL have better performance than normal AL sampling strategies. From the aspect AUBC evaluation metric, our model has comparable performance with SIMILAR, 0.667 vs 0.669. Nevertheless, we have lower standard deviation value than SIMILAR, so our POAL is more stable. From the aspect of Accuracy vs. Budget curves, in carly stages (c.g.. Budget 2,500), our POAL is better than SIMILAR (FICMTT) and in latter stages SIMILAR (FICCMT) exceeds POAL. The reason is that in SIMILAR (FLCMI), they calculate the similarities between the ID labeled set and the unlabeled pool and the dissimilarities between the OOD labeled set and the unlabeled pool. In the carly stages, the OOD data is insufficient. Thus SLMILAR(FLCM) experiment, we find that SIMILAR (FLCMT) only performs well when we have enough information on both and OOD data samples, which results in more OOD data sample selection. Our POAL only considers the distance between unlabeled samples and labeled ID samples, so we are more efficient preventing OOD sample selection. Additionally, POAT is datascts. However, SIMILAR is limited by the computation condition. Our method is also more time efficient. than SIMILAR (FLCMI), shown in Table 4 SIMILAR (FLCMI) runs five times longer than our POAL.
will select more OOD samples, as shown in Fig. [0.b. From this
widely adopted since could be adopted large-scale
CIFARI O-downsampled (ID:OOD ratio 4:10)
OOD data selected in CIFAR O0ddowssampedd
Number of OOD samples selected
POAL-PS ( (2567=00.01)
1750
POAL-PS
0.76
SIMILAR 659168095
SEMEARR
1500
ENE (.647-0002)
ENT
Randem (3 5522)102)
Racdon
1250
Accuracy
150
0.66
500
0.64
0.62
1500
2007
2500
3000
3000
1500
2000
2500
3002
3500
Budget
Budget
(a) Accuracy Budget
(b) OOD num vs. Budget
Figure 10: The comparative experiments between our model and SIMILAR on down-sampled CIFARIO dataset.
Table 4:
The and standard deviation of running time (in seconds) of the comparative experiments with 3 repeated trials between our POAL and SIMILAR.
Method
SIMILAR 6419.0 (109.9) 32837.7 (897.7)
POAL-PS
IDEAL-ENT
ENT
Random
Margin
Time
2225.0 (12.6)
1690.0 (15.3) 1507.7 (5.8) 1573.7 (9.8)
C.2.1
Enlarged Figures
To increase the readability of our experimental results, we enlarged the Figures
and as shown
Figures
11
13
and
14
28

---

Published in Transactions on Machine Learning Research (06/2023)
BX8
vowel
699
Accuracy
Accuracy
0.3
POALOO 545)
6.4
ISEAL-BRE IDEAL-ENNOOOO (NN)
ENTIO A02)
IDEAL-EN PEEAL-ENIIOSSS 554)
ENTO5119)
MARINO
041
WoightdSan 0.0 592)
WoigtedSan 00 525)
WinghooSan
035
Wiighnod8mm 00
Weighneeemmm3 517)
039
200
200
400
Budget
Budget
(a) EX8
(b) vowel
letten(a-k) ID:OOD ratio 10:1
letter(a-n) ID:OOD ratio-
0.3
0.9
6.15
675
6.30
Accurcy
PONL23.73)
Accuracy
6.55
POLL00.71
LSEAL 4N719.714
6.9
IDEAL (1111111
0.59)
MARNO 712)
MAHINO doy
Twndinget T16)
(i (84)
Weightadimm:
Weightedtm- a0 dM4
WiigheoSan
645
WiighteSam
Wingenn88n--533 205)
Wiightetmm- m9
C49
Budget
Budget
(c) letter(a-k)
(d) letter(a-n)
letter(a-r) ID:OOD ratio 10:8
letter(a-w) ID:OOD ratio 10:13
0.55
Accuracy
0.85
POAL(SA77)
Accuracy
POALIOAS
0.9
IDEAL
ENTIO
ENRV033))
ENTOO17))
RAND:0.545)
RAND:0.536
MAHUYO 652)
MABAOO 652)
Texdingeth
Wrightatiinn
WeightrdSem=
Waa.teadimm-1000 M8)
WeightedSemmm 419)
Wrightedinn-2200 2.9 5g
0.5
Wrighted5mm-3330 43)
Budget
Budget
(e) letter(a-r)
(f) letter (a-terr(-w)
letter(a-z) ID:OOD ratio 10:16
letter
0.9
Accuracy
AUBC
0.55
ENTIO 589)
RAND(6:2)
MAHAM0613
642
NAHIA
TouRage
Waigheediamm110 000 592)
w.s.ticdSum-22200 229.99
20
Budget
Datasct
(g) letter(a-z)
(h) letter(AUBC)
Figure 11: This a enlanged version of Figure our main paper.
29

---

Published in Transactions on Machine Learning Research (06/2023)
CIFARIO 04(DDOOD ratio- 6:4)
CIFAR1O 06(IDOOOD ratio 4:6)
Accuracy
SIILAR SIUND (AWAAIN
Accuracy
SINELAN (EMMEEETTTIOOO
EDE (7THHEEE
MANA- 210
MANA OERmIODIIy
BADCE/S N415 90))
BADGE 10.00 JUMM
Budget
Budget
(a) CIFARIO-04
(b) CIFAR10-06
CIFARIOU 04(DD:0OD ratio 6:4)
CIFARI00 06(DD:0D ratio 4:6)
Accuracy
COTA DHARCR)
CSAL DEIEUEO
Sodee CUCHAIIG
344 M.B.SAnnAAe
14AX 8AIHAX
300
1000
100.9
2008
2000
258
5000
7509
10000
12200
1500
1750
2000
Budget
Budget
(c) CIFAR100-04
(d) CIFAR100-06
OOD data selected CIFARIO 04
OOD data selected in CIFARIO
Ratio of O'ODD samples selected
Ratio of OOD samples selocted
DOM
KOGLAN (PI9M)
14
DOSLE
SIVILAR (IIIMO
DOU
DOOT
IREAL INC
ENTT
HALD
RALD
150
1291
1508
12:00
1100
(799)
2901
101
1240
13:0
10.00
1864
1486
92.0
Budget
Budget
(e) CIFARIO-04 OOD
(E) CIFAR10-06 OOD
data selected CIFAR100
OOD selected CIFARI00
Ratio OOO samples selected
Ratio OFO sumplees selected
64
POAL-PS
DOU
IRSAL INC
TRRAA INI
INT
INT
MARHA
RAND
K36aac
BADES
R4246
100
14:0
Budget
Budget
(g) CIFAR100-04 OOD
(h) CIFAR100-06 OOD
Figure 12: This a enlanged version of Figure our main paper.
30

---

Published in Transactions on Machine Learning Research (06/2023)
Typical Parclo Optimization
POAL
ID postiive data
ID posttive data
000 data
0OD data
Typical Pareto Optimization
(b) POAL
Typical Pareto Uptimizatoon
POAL
negalive mibslarohii distamnee
Degalive inbasarmobis distamoe
positive data
positive data
negative data
negative data
OOD data
OOD data
selected data
selected data
entropy
entropy
(c) Typical Pareto Optimization
(d) POAL
Figure 13: This a enlarged version of Figure
in our main paper.
31

---

Published in Transactions on Machine Learning Research (06/2023)
True Pareto Optimization
True Pareto Optimization
poslivve data
negative
megairee malrllnnbii distance
s-lecial dats
positive ID nsgative data
sclestal data
entropy
(a) True Pareto Optimization
(b) 'True Pareto Optimization
Typical Pareto Optimization
Typical Pareto Optimization
ID postitve data
negathre selected data
mogative medalamobis dastance
ID negative data
selected dat
catropy
(c) Typical Pareto Optimization
(d) Typical Pareto Optimization
ID positive datn nagative selected data
mogotive melleasbii distance
ID negative date
dais
selectal dats
entropy
(e) POAL
(f) POAL
Figure 14: This is enlarged version of Figure 6 in our main paper.
32

---

Published in Transactions on Machine Learning Research (06/2023)
C.2.2 Supplementary Experiments Stochastic Acquisition
To explore whether the stochastic acquisition strategy (Kirsch et al. 2021 can be used for AL under OOD delection scenarios, we consider simple experiment using the same experimental settings as Figure T We add Gumbel-distributed random perturbations based on (0; 8) to the AL score function ENT. The visualization of densitics with different 3 scttings shown in Figure 15 Adding random perturbations could indeed increase the number of ID samples selected, compared with selection with only ENT. Larger B settings could result in better performance, but it still could not distinguish ID and OOD samples, as there is still significant overlap between the ID and OOD score distributions.
ID date
1D abia
ID dam 00D
doD-daa
000 dale
20.6
631
3.10
155
43
Emroggy
Berogy
(a) 8-1
(b) 8
0.05
(c) 8-
Figure 15: Densities when adding Entropy score with Gumbel-distributeed random perturbations with different prper-parameter settings. The experimental settings are the same as Figure
33


Reviews:
[{"review_id": "o3lE9QTNZB", "reviewer": "Reviewer_UoNg", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The strength of the paper lies in its analysis of the contradictions of the active learning strategy with OOD data, the proposed method's ability to work without the need for tuning hyperparameters, and the paper's demonstration of good empirical performance. However, the paper's weaknesses lie in the lack of clarity regarding the methodological contribution compared to POSS. It is relatively straightforward to update POSS by including one more metric or fixing the size of s in POSS. The question here is whether the updated algorithm could still maintain the theoretical soundness of POSS. Therefore, the paper needs to provide more analysis on this aspect.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a solution for active learning with unlabeled data that may contain out-of-distribution (OOD) data. Traditional active learning methods tend to select OOD data, which is unuseful for further learning. To overcome this problem, the paper introduces a Pareto optimization method that considers two metrics: one metric for selecting high-entropy data and another metric for selecting in-distribution data.\n\nThe proposed method aims to balance the selection of high-entropy data and in-distribution data to achieve optimal performance. The high-entropy data is useful for improving the model's accuracy, while in-distribution data is beneficial for increasing the model's robustness. The proposed method selects a subset of data that satisfies both criteria by finding the Pareto front, which represents the optimal trade-off between the two metrics.\n\nThe experimental results show that the proposed method outperforms traditional active learning methods and selects fewer OOD data. The proposed method converges faster than the baselines, and the selected data subset achieves higher accuracy and better robustness. The paper concludes that the proposed method is effective in active learning scenarios with unlabeled data containing OOD data.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "More analysis on the contribution compared to POSS.\n\nCompared to existing works such as [Kothawade et al, 2021] or [Du et al., 2021], it is clear that the proposed method does not depend on hyper-parameters. However, it is not clear whether the method is still under the impact of insufficient training data. The paper is suggested to discuss this point. \n\nMinor:\nSec 3.1 should specify \\mathcal{D} is a data distribution without OOD data\n\nFigure 1: it is not clear in deep learning, in what learning phase the phenomenon shown can occur. Do they happen across all learning phases, or in the early, middle, or final learning phase? The paper is suggested to make this point clear. \n", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ZHLolIYRK6", "reviewer": "Reviewer_aHXg", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper is exceptionally well-written, full of innovative ideas and contributions, and is a pleasure to read. The presented work addresses an important issue in the field, as curated datasets are rare in the real word, yet overused in research, and most real-world datasets likely contain numerous outliers and OOD samples that should be excluded from training.\n\nThe method is well-supported and motivated, and the experiments demonstrate its effectiveness compared to the employed baselines. However, there are a few key points to address by the authors:\n\n1. The paper's use of entropy as the active learning (AL) objective and Mahalanobis distance as ID confidence scores is equivalent to Mukhoti et al. (2021)[^1], which emphasizes the challenges of using curated datasets for OOD and AL benchmarks, and that using entropy as OOD and AL metric confounds aleatoric and epistemic uncertainty as it is the sum of both (total uncertainty). Mukhoti et al. propose using a Gaussian Mixture Model (GMM) on ID data as an ID confidence score for OOD detection and AL. They also highlight the potential for feature collapse of feature-space density-based methods. This paper does not mention these concerns. They run experiments on active learning, and figure 1(a) in this paper is similar to figure 1 in that paper. The contributions already distinguish between the metrics and the proposed Pareto-based approximation, but the inclusion of Mukhoti et al. (2021) as prior art and discussion would be very helpful.\n\n2. Fredlund et al. (2010)[^2] may be an additional useful reference for query density functions.\n\n3. Both Zhan et al. (2022b) and Kirsch et al. (2021)[^3] suggest using acquisition scores as a sampling distribution. The former shows this for classical ML datasets, while the latter investigates this for deep learning, using BALD and entropy. In particular, Kirsch et al suggest normalized scores (instead of softmax normalized score) as a sampling distribution when using BALD or entropy scores.\n\n4. Given the above, it would be great to compare to DDU as an AL baseline. The main difference is the feature-space density being used  as a single objective for AL acquisitions. In the case of shared covariance matrices, it would be equivalent to the (inverted) Mahalanobis distance, but DDU does not use shared covariance matrices. DDU would be an important baseline, and similarly, an ablation of your method to see if the dual objective has a considerable impact empirically vs only using ID confidence scores would also be recommended.\n\n5. Stochastic acquisition, as opposed to top-K acquisition, might offer a stronger baseline, as it could potentially focus less on OOD data given that it samples from the tail, too. Analyzing how stochastic sampling improves BALD, ENT, and LPL would be informative. As you mentioned in §3.2, the sampling-based approach can be applied to any AL method that provides per-sample scores.\n\n6. Please consider enlarging all figures to enhance readability, especially figures 3, 4, 5, and 6.\n\nThank you for your work, and best wishes!\n\n[^1]: Mukhoti, Jishnu, et al. \"Deep deterministic uncertainty: A simple baseline.\" arXiv e-prints (2021): arXiv-2102.\n\n[^2]: Fredlund, Richard, Richard M. Everson, and Jonathan E. Fieldsend. \"A Bayesian framework for active learning.\" The 2010 International Joint Conference on Neural Networks (IJCNN). IEEE, 2010.\n\n[^3]: Kirsch, Andreas, et al. \"Stochastic Batch Acquisition for Deep Active Learning.\" arXiv preprint arXiv:2106.12059 (2021).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a method for pool-based batch active learning with datasets containing out-of-distribution (OOD) data, which is typically uninformative for the target task. The authors propose a trade-off between two objectives to distinguish between in-distribution (ID) and OOD data, as total uncertainty-based acquisition functions like entropy often confound the two. \n\nTo eliminate the need for selecting a hyperparameter for explicitly trading off between the objectives, the authors introduce an evolutionary Monte Carlo-based approach. This method samples candidate acquisition batches and selects a compromise candidate set expected to perform well across all trade-offs. \n\nThe proposed method is compared to various active learning baselines and demonstrates strong results on pool sets containing OOD data in both deep learning and classical settings.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See above.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "rFBjzh0bWZ", "reviewer": "Reviewer_FtG9", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n\n1. The writing of this paper is clear and easy to understand. The math notations are  consistent across the paper. \n2. The authors showed consistently better empirical performance compared to all the baselines of classical machine learning and deep learning models under OOD scenario. \n\nWeaknesses: \n\n1. Based on the current formulation, some unlabelled data may not be used to help learning the final model when their in-distribution (ID) confidence score is low. Especially at the beginning of active learning, it is possible that many data points can have lower ID scores, degrading the overall active learning performance. The authors may need to better discuss how to better balance exploration and exploitation at different learning stages. Is it not clear how to tell the difference between the actual OOD data from false ones due to the model uncertainty at the beginning. The authors may want to check on its potential effects on overall active learning performance. \n2. There has been some previous works studying active learning under distribution shift. For example, the problem setup of this paper can be considered as a special case of the general distribution shift problem in [1]. The proposed algorithm for best-subset selection problem is based on a modified POSS [2]. The authors may want to make their methodological contributions clearer. \n3.  The experiments on CIFAR10 and CIFAR100 showed that many baseline methods performed worse than random sampling, including CCAL, SIMILAR, which were designed under OOD scenario. I understand that the methods using the single objective without considering the distribution shift may not work in this setup. But if many baseline methods perform worse than the randomly queried data, this would make the experiment design suspicious. The authors should give more discussions about these results. \n4. For the results using the weighted sum multi-objective optimization, it cannot be inferred from the paper whether the scale of two optimization objectives is matched using the coefficients $\\{0.2, 1.0, 5.0\\}$. The authors may want to discuss the sensitivity of the active learning results with respect to these coefficients. \n5. In Section 4.2.1 and Section 4.4, the authors claimed the computation cost can be somehow reduced by decreasing the parameter $s_w$. However, in Table 1, the running time on CIFAR10-06 actually increases when $s_w$ decreases while on CIFAR100-04, the running time drastically decreases as $s_w$ decreases. Some analyses and discussions based on these results would be highly appreciated.\n\n[1] Zhao, Eric, et al. \"Active learning under label shift.\" International Conference on Artificial Intelligence and Statistics. PMLR, 2021.\n[2] Qian, Chao, Yang Yu, and Zhi-Hua Zhou. \"Subset selection by Pareto optimization.\" Advances in neural information processing systems 28 (2015).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies active learning to query the unlabeled data under out-of-distribution (OOD) data scenario, for which the authors formulated a multi-objective best subset selection problem with a sampling based adhoc solution to jointly maximize the informativeness and in-distribution confidence of queried data so that only the informative in-distribution data will be queried. ", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "\n1. The authors may need to better discuss the problem setting, motivation, and their methodological contributions compared to the existing works as detailed in the previous weakness 1-2. \n\n2. The authors may need to provide more appropriately designed and comprehensive empirical results to address the potential issues as detailed in the weaknesses. ", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies active learning with unlabeled data containing OOD data, and proposes a Pareto optimization approach for this problem. The proposed method balances the selection of high-entropy and in-distribution data to achieve good performance. The experimental results demonstrate that the proposed method outperforms traditional active learning methods and selects fewer OOD data. 

Overall, the reviewers recognize that this paper makes a solid contribution to a well-motivated problem. In the discussion stage, the authors provide further revisions and clarifications on: 
- discussions on related works, such as POSS, active learning with distribution shift, 
- impact of insufficient training data
- experimental comparison with pure ID-based baseline DDU and MAHA
- experiments on the conflict of ID and OOD data on stochastic acquisition measures
- etc,
which help the reviewers appreciate the results further. 

The authors are asked to incorporate the reviewers' feedback in the final version: specifically, adding experimental comparisons with DDU and MAHA for CIFAR datasets in the rebuttal to Reviewer aHXg, and further clarifications to Questions 3-5 and the follow-up questions of Reviewer FtG9. 
 
Decision: Accept (poster)