Pareto Optimization for Active Learning under Out-of-Distribution Data Scenarios
Authors: Xueying Zhan
Zeyu Dai
Qingzhong Wang
Haoyi Xiong
Dejing Dou
Qing Li
Antoni B. Chan
Venue: Submitted to ICLR 2023 (2023)
Keywords: active learning
pareto optimization
out-of-distribution

Abstract: Pool-based Active Learning (AL) has achieved great success in minimizing labeling costs by sequentially selecting the most informative unlabeled samples from a large unlabeled data pool and querying their labels from oracle/annotators. However, existing AL sampling schemes might not work well under out-of-distribution (OOD) data scenarios, where the unlabeled data pool contains data samples that do not belong to the pre-defined categories of the target task. Achieving good AL performance under OOD data scenarios is a challenging task due to the natural conflict between AL sampling strategies and OOD sample detection -- both more informative in-distribution (ID) data and OOD data in unlabeled data pool may be assigned high informativeness scores (e.g., high entropy) during AL processes. In this paper, we propose a sampling scheme, Monte-Carlo Pareto Optimization for Active Learning (POAL), which selects optimal subsets of unlabeled samples with \emph{fixed batch size} from the unlabeled data pool. We cast the AL sampling task as a multi-objective optimization problem and utilize Pareto optimization based on two conflicting objectives: (1) the typical AL sampling scheme (e.g., maximum entropy), and  (2) the confidence of not being an OOD data sample. Experimental results show the effectiveness of our POAL on classical Machine Learning (ML) and Deep Learning (DL) tasks.

Under review as a conference paper at ICLR 2023
PARETO OPTIMIZATION FOR ACTIVE LEARNING UNDER OUT-OF-DISTRIBULE DATA SCENARIOS
Anonymous authors
Paper under double-blind review
ABSTRACT
Pool-based Active Learning (AL) has achieved great success in minimizing labeling costs by sequentially selecting the most informative unlabeled samples from large unlabeled data pool and querying their labels from oracle/annotators. However, existing AL sampling schemes might not work well under out- of of-distribution (OOD) data scenarios, where the unlabeled data pool contains data samples that do not belong to the pre-defined categories of the target task. Achieving good AL. performance under OOD data scenarios is a challenging task due to the natural conflict between AL sampling strategies and OOD sample detection both more informative in in-distribution (ID) data and OOD data in unlabeled data pool may be assigned high informativeness scores (c.g., high entropy) during AL processes. In this paper, we propose a Monte- Canto Pareto Optimization for Active Learning (POAL) sampling scheme, which selects optimal subsets of unlabeled samples with fixed batch size from the unlabeled data pool. We cast the AL sampling task as multi-objective optimization problem and utilize Pareto optimization based on two confficting objectives: (1) the typical AL sampling scheme (e.g., maximum entropy) and (2) the confidence of not being an OOD data sample. Experimental results show the effectiveness of our POAL on classical Machine Learning (MI.) and Deep Learning (DI.) tasks.
INTRODUCTION
In real-life applications, huge amounts of unlabeled data easily obtained, but labeling them would be expensive (Shen et al. 2004). AL aims to solve this problem - achieves greater accuracy with less training data by sequentially selecting the most informative instances and then querying their labels from oracles/annotators (Zhan et al. 20211). Current AL methods have been tested on well-studied datasets (Kothawade et al. [021] like MNIST (Deng 2012 and CIFARIO (Krizhessky et al.12000) These datasets are simple and clean. However, in realistic scenarios, when collecting unlabeled data, unrelated data (i.e., out-of-domain data) might be mixed in with the task-related data,
(021). Most AL methods e.g., images of letters when the task is to classify images of digits (Du et al. are not robust to OOD data scenarios. For instance, Karamcheti et al. 202T) has demonstrated empirically that collective outliers hurt AL performances under Visual Question Answering (VQA) tasks. Meanwhile, selecting and querying OOD samples that are invalid for the target model will waste the labeling cost (Du al. 2021 and make the AL sampling process less effective.
There is a natural conflict between AL and OOD data detection. Most AL methods, especially uncertainty- based measures, prefer selecting data that are hardest to be classified by the current basic classifier (e.g., high entropy of predicted class probabilities). However, in AL, if a basic learner (e.g.. Neural Network with softmax output) performs poorly ID data, it is more likely provide non-informative predicted probabilities (i.e., close to uniform probabilities) on OOD data (Vaze et al., 2021). During AL processes, the basic learner not well-trained due insufficient labeled data, and insufficient epochs in the case of deep AL. Therefore, the samples selected by AL may contain both high informative ID and OOD samples. For example, consider the Maximum Entropy (ENT) approach for AL, which is classic uncertainty-bart 2001) that selects data whose predicted class probabilities have the largest entropy. Meanwhile, ENT also typical OOD detection method - high entropy of the predicted class distribution suggests that the input may be OOD (Ren et al., 2019). Fig. top) shows an example on the EX8 dataset (Ng 2008), which further illustrates this conflict large percentage of data with high entropy scores OOD, and thus ENT-based AL will likely select OOD data for labeling. Thus, additional measures are needed to detect OOD samples that they are not selected for AL.. For example, in Fig. [bottom),
method (Lewis & Catlett 1994 Shannon

---

Under review as a conference paper at ICLR 2023
the negative Mahalanobis distance (Lee et al., 2018 shows a certain negative correlation with entropy
and thus could be used as an ID confidence score.
Although the OOD problem has been demonstrated to affect AL in real-life applications (Karamcheti et al. 2021), there are only a
few studies this topic (Kothawade et al. 2021 Du et al. 2021
dee
SIMILAR (Submodular Information Measures Based Active Learn-
ing) (Kothawade al. (021) adopted the submodular conditional mutual information (SCMI) function as the acquisition function. They jointly model the similarity between unlabeled and labeled ID data sets and their dissimilarity with labeled OOD data sets. The estimation might not be accurate initially since both labeled II) and OOD data sets are insufficient. CCAL (Contrastive Coding AL. (Du et al. 202T) needs to pre-train extra self-supervised models like SimCLR Chen et al. 2020), and also introduces hyper-parameters to trade-off between semantic and distinctive scores, whose values affect the final performance (see Section 4.3 in (Du et al., (001]). These two factors limit the range of its application. We compared our work with SIMILAR and CCAL in detail and showed the superiority of our method in Appendices C C[and]] F3
Figure I: (top) Distribution of en- tropy for ID and OOD data dur- ing AL processes on EX8. (bot- tom) Scatter plot of the AL score (entropy) and ID confidence score (negative Mahalanobis distance) of unlabeled data. Larger ID score indicates data is more likely to he ID) data.
In this paper, advocate simultaneously considering the AL eri- terion and ID confidence when designing AL sampling strategies to address the above issues. Since the two objectives conflict, we define the AL sampling process under OOD data scenarios as a multi-objective optimization problem (Seferlis & Georgiadis (2004). Unlike traditional methods for handling multiple-
based AL, such as weighed-sum sum optimization (Zhan et al. 2022a) or two-stage optimization (Shen et al.) 2004, Zhan et al. 2022a), we propose a novel and flexible batch-mode Pareto Optimization Active Learning (POAL) framework. The contributions and summarization of this paper
follows:
1. We propose AL under OOD data scenarios within multi-objective optimization framework. 2. Our framework is flexible and accommodate different combinations of AL and OOD detection methods according to various target tasks. In our experiments, we use ENT as the AL objective and Mahalanobis distance as IDD confidence scores.
3. Naively applying Pareto optimization to AL will result in a Pareto Front with a non-fixed size, which can introduce high computational cost. To enable efficient Pareto optimization, propose a Monte-Carlo (MC) Pareto optimization algorithm for fixed-size batch-mode AL.
4. Our framework works well on both classical ML and DL tasks, and we propose pre-selecting and early- stopping techniques to reduce the computational cost on large-scale datasets.
5. Our framework has no trade-off hyper-paramett important since: i) AL is data-insufficient, there might be no validation set for tuning parameters; ii) hyper-parameter tuning AL can be label-expensive since every change hyper-parameter causes AL to label new data, thus provoking substantial labeling inefficiency (Ash et al. 2020).
for balancing AL and OOD objectives. is
RELATED WORK
Pool-based Active Learning. Pool-based AL has been well-studied in these years (Settles) 2009
Zhan et al. 2021b; Ren al. (2021) and widely adopted in various tasks (Duong al. [0088] Yoo &
Kweon 2019; Dor et al. 2020; Haussmann et al.) 2020). Most AL methods rely on fixed heuristic sampling strategies, which follow two main branches: uncertainty- and ippresentaiive/divvrsitt-aba based measures (Ren et al. 2021: Zhan et al. 2022a). Uncertainty-baser approaches select data that maximally reduce the uncertainty of the target basic learner (Ash et al. (020). Typical uncertainty- based measures that perform well on classical MI. tasks like -omomiite (QBC) (Seung ct al.) 1992). Bayesian Active Learning by Disagreement (BALD) (Houlsby et al. 201 1) have also been generalized to DL tasks (Wang& & Shang 2014 Gal et al. 2017 Beluch et al. 2018 Zhan et al.) 2022a). Representative/ddiversit based methods like k-Means (Zhan et al., 2022a) and Core-Set approach (Sener & Savarese 2018) select a batch of unlabeled data most representative of the set. Uncertainty- and representative-bon based measures could be combined with weighted- sum or multi- stage optimization (Zhan et al.) 2022a). Weighted-sum optimization combines multiple objectives
2

---

Under review as a conference paper at ICLR 2023
2017) is a typical method that adopts weighted-sum optimization, seleeting the most uncertain and least redundant samples as well as the most diverse. Two-stage (or mulli-stage)) optimization first ranks data selects subset based on one objective, then makes final decisions based on the other objective (Shen et al. 2004] ZZhao et al.
through linear combination with trade-off weights. (Yin et al.
2019 /Zhan al. (022a). (Ash et al. 2020) computes gradient embedding of unlabeled data in the first stage (uncertainty), then clusters by KMeans++ in the second stage (diversity).
Detecting OOD data is vital in ensuring the reliability and safety of ML systems in real-life applications (Yang et al. 202I) since the OOD problem severely influences real-life decisions. For example, in medical diagnosis, the trained classifier could wrongly classify a healthy OOD sample as pathogenic (Ren al. 2019). Existing methods compute ID/OOD confidence scores based on the predictions of (ensembles of) classifiers trained on ID data, e.g., the ID confidence can be the entropy of the predictive class distribution (Ren et al. 2019). Hendrycks & Gimpel (2017) observed that a well-trained neural network assigns higher softmax scores to II) data than OOD data, i.c. predictions for ID data have lower entropy. Follow-up work ODIN (Liang et al. 2017)) amplifies the effecctiveness of OOD detection with softmax score by considering temperature scaling and input pre-processing. Others (Lee et al. 2018 Cui et al. 2020) propose simple and effective methods for detecting both OOD and adversarial data samples based Mahalanobis distance. Due to its superior performance compared with other OOD strategies (see Table I in (Ren et al., (2019),, our POAL framework uses Mahalanobis distance the primary criterion for calculating the ID confidence score. Nonetheless, our framework is general, and any ID confidence score could be adopted.
Out-of-Distribution.
How OOD influence AL methods? We have introduced the primary categories of AL methods, the OOD problem, and how OOD data influence AL, like uncertainty-basee We next discuss whether other AL methods are also affected by OOD data. As discussed in Section [ uncertainty- AL. measures like ENT are not robust to OOD data scenarios since OOD data are naturally difficult to be classified. pypresentaiiveddivvrrtyyyy-ybi seaeiiiersstt--bdddddd methods are also not robust to OOD data since outliers/OOD samples are far from ID data and thus more likely to be selected first to cover the whole unlabeled data pool. Combined AL strategies that integrate uncertainty- and representative-base
measures (e.g., ENT).
measures will also be susceptible to OOD data. Weighted-sumg optimization will select data with high uncertainty and representativeness scores, which are likely to be OOD. The multi-stage optimization adopts uncertainty- based measures to select a subset and then uses representative--asee based measures for final selection (or vice versa). For both variants, large proportion of OOD data will be selected in the first stage. To address these issues, we proposed our POAL, which can be adapted to various AL sampling schemes with OOD data. We conducted experiments on multiple highly-cited AL methods to validate our viewpoints, as shown in Section 4 Fig. 4
Multi-objer Mujti-objective optimization. Many real-life applications require optimizing multiple objectives that conflict with each other. For example, in the sensor placement problem, the goal to maximize sensor coverage while minimizing deployment costs (Watson et al. 2004). Since there is no single solution that simultaneously optimizes each objective, Pareto optimization can be used find a set of "Pareto optimal" solutions with optimal trade-off' of the objectives (Miettinen, 2012). A decision maker can then select a final solution based on their requirements. Compared with traditional optimization methods for multiple objectives (c.g., weighted-sum (Marler & Arora, 2010). Pareto optimization algorithms are designed via different mnctahhcuisstics without any trade-ofi off parameters (Zhou et al. 2011 Liu et al. 2021). Besides optimization of the two conflicting objectives (AL score and ID confidence), we also need to perform batch-mode subset selection. Pareto Optimization Subset Selection (POSS) (Qian et al., 2015) solves the subset selection problem by optimizing two objectives simultaneously: maximizing the criterion function and minimizing the subset size. However, POSS does not support more than one criterion and cannot work with fixed subset size, thus not suitable for our task. Therefore, we propose Monte- Carll POAL, which achieves: 1) optimization of multiple objectives; 2) no extra trade-off parameters for tuning: 3) subset selection with a fixed size.
3 METHODOLOGY
We introduce the problem definition of AL under OOD data scenarios, and details of our POAL.
3.1 PROBLEM DEFINITION AND OVERVIEW
We consider a general pool-based AL process for a K-class classification task with feature space X, label space y {1, {1,.... K) under OOD data scenarios. We assume that the oracle can provide
3

---

Under review as a conference paper at ICLR 2023
Update
Gradeckunntate
Quers
Moatr Carls POAL
AL boakk
larner
Colowdd
Nearr repersanatioon wabsbbeh
Ainkean
Figure 2: Overview of POAT.. In cach iteration, we train basic learner and estimate GMMs (in DL. tasks, we estimate multivariate Gaussian distributions with low-/high- feature levels) to calculate the AL and Mahalanobis distance-based ID confidence score for unlabeled data. We then construct POAL by randomly generating candidate solutions with fixed batch size and updating the Pareto set iteratively to get the optimal solution.
fixed number of labels, denoted budget B. When an OOD sample is queried, the oracle will return an "OOD" tabePto represent data outside of the specified task. Let B be the number of labels the oracle can provide, i.e., the budget. We aim select the most informative B instances with fewer OOD samples to obtain the best classification performance. To reduce the computational cost, we consider badeh-mode AL, where batches of samples with fixed size are selected and queried. We denote the AL acquisition function as (x; A), where A refers to AL sampling strategy, and the basic learner/classifier as [o(x). We denote the current labeled set as Dr- = {(x; 3/1 i== and the large unlabeled data pool D., {x, J The labeled data are sampled i.i.d. over data space D, i.e., Di E 'D, and N < M. Under OOD data scenarios, active learners may query OOD samples whose labels are not y. To simplify the problem settings, we ignore the queried OOD data since it is not useful for the classification task, and thus we only add ID samples to D,
Motivated by the natural conflict between AL and OOD data detection like Fig. (bottom), we propose POAL as outlined in Fig. 2 In each AL iteration, we firstly utilize the clean labeled set that only contains ID data for training a basic classifier fx(x) and constructing class-conditional Gaussian Mixture Models (GMMs) for detecting OOD samples in D.. Based classifier fe(x), AL acquisition function axx; A) and the GMMs, we calculate the informativenes/uunertann U(x;) and ID confidence score M(x.) for each unlabeled sample x; in D.. Let (0, 1) M be binary vector whose element 8, = 1 indicates that the i-th unlabeled sample selected, and s; = 0 indicates unselected. In each AL stage, the multi-objective optimization goal is find an optimal subset s* with b selected samples, simultaneously maximizes the informaaivennes//nccrtiin and ID confidence score:
score
= arg (o.1aa (U(s), M(s)
s.t.
Σ 8 =
(1)
where (ds) = -M sx,( ) is shorthand for the aggregated AL score for the selected samples, and likewise for M(s). (()) and M(-) will be introduced Sections 3.2 and 3.3
3.2 ACTIVE LEARNING SCORE
AL selects data by maximizing the acquisition function a(x; i.c., x* arg maXxeD, a(x; A) (Gal et al. 2017). our work, we use the AL scores for the whole unlabeled pool Du for the
subsequent Pareto optimization. Thus, we convert the acquisition function to querying density
a(x.:A) Zhan et al. 2022b). In our experiments, we use ENT as our basic function (xx, = Σj= EyE, A();AA)
AL sampling strategy, and the acquisition function (xET((x --E poly = k|x) log Po(y k k[x)), where po(y) pe(y|x) (x) is the posterior class probability using classifier fo fx(... Our framework is flexible because it can incorporate various AL sampling strategies if their acquisition function can convert to querying density function. In general, AL methods that explicitly provide per- sample scores (e.g., class prediction uncertainty) or inherently provide pair-wise rankings among the unlabeled pool (e.g.. k-Means) can convert to a querying density. Thus, many uncertainty- measures like QBC (Seung et al. 1992), BALD (Houlsby et al., 2011; Gal et al., 2017), and Loss Prediction Loss (LPL) (Yoo & Kweon 2019) are applicable since they explicitly provide uncertainty information per sample. Also, k-Means and Core-Set (Sener & Savarese 2018) approaches are applicable since they provide pair-wise similarity information for ranking. On the other hand, AL
In our implementation, the label of OOD data is -1.
4

---

Under review as a conference paper at ICLR 2023
approaches that only provide overall scores for the candidate subsets, such as Determinantal Point Processes (Bıyık et al. 2019; Zhan et al. 2021a), are unable to be adopted in our framework.
3.3 IN- IN-DISTRIBUTION CONFIDENCE SCORE VIA MAHALANOBIS DISTANCE
Intuitively, ID unlabeled data can be distinguished from OOD unlabeled data since the ID unlabeled data should be closer to the ID labeled data (Di) in the feature space. One possible solution is to calculate the minimum distance between an unlabeled sample and the labeled data of its predicted pseudo label provided by ʃ(0) (Du et al. 2021). If this minimum distance is large, the unlabeled sample will likely be OOD and vice versa. However, calculating pair-wise distances between all labeled and unlabeled data is computationally expensive. A more efficient method is to summarize the labeled ID data via a data distribution (probability density functiinn and then compute the distances of the unlabeled data to the ID distribution. Since adopt GMMs to represent the data distribution, our ID confidence score is based on Mahalanobis distance, as motivated by (Lee et al. (2018). AL for classical ML tasks focuses training basic learner (x) with fixed feature representations, while AL for DL jointly optimizes the feature representation X' and classifier fo(x) simultaneously (Ren al., T02T). Considering that these differences can influence the data distributions and further influence the Mahalanobis distance calculations, we adopt different settings for classical ML and DL.
Classical MI tasks. We represent the data distribution estimated from Dj with GMMs. Specifically, since we have labels of Di, we represent each class & with a class-conditional GMM,
PX y = k)) Ck Te (6) (x(lu
(K) >
(2)
the mixing coefficient, mean vector and covariance matrix of the e-th Gaussian component with the k-th class, and (x Σ) is a multivariate Gaussian distribution with mean μ and covariance Σ. The class-conditional GMM for class & is estimated from the labeled data for class k. D(4) = [x, lv = k} using a maximum likelihood estimation (Reynolds, 2009) and the EM algorithm (Dempster et al. 1977). We then define the ID confidence score M(x) as the Mahalanobis distance between the unlabeled sample and its closest Gaussian component in any class,
where (R) (1s)
MMc >x(h))
Mx)(x) = max max. -||x- - 1 1".""
(3)
DL tasks. For DL, we follow (Lee al. 2018 calculate M(x). since it makes good use of both low- and high-level features generated by the deep neural network (DNN) and also applies calibration techniques, e.g., aaput-preprocessiee
(Liang et al., 2018) and feature ensembles Lee et al. (2018) to improve the OOD detection capability. Specificallly, denote fe(x) the output of the ě-th layer of DNN for input x. A class- classcooniiionaa Gaussian distribution with shared covariance estimated for each layer and for each class & by calculating the empirical mean and shared covariance (ji (R) Êx):
p
E
1 Σ, Exeree D(2 (fe(x) - (1) (ffe()) - (2( (4) Exergey fe(x), Ee = PPl
1D(1)
Next we find the closest class for each layer E via & arg min+ (x) 2 and OOD data more separable, an input pre- pre-processing step is applied by adding a small perturbation
1 To make the ID
(k)
to x, X x + esign( (oiign(Vxllfdee Vxl]f((x) - ui 1 ). where e controls the perturbation magnitude? Finally, the ID confidence score is the average Mahalanobis distance for x over the L layers.
MxL(x)= = Σ tm max; -1)
(1) 14 113
(5)
3.4 MONTE- PARETO OPTIMIZATION ACTIVE LEARNING
Our framework for AL for OOD data contains two criteria, AL and ID confidence score. As discussed in Section weighted-sum and multi(two)-stage optimization are widely used in multiple- criteria AL problems. However, both have drawbacks when applied to AL for the OOD data scenario. In weighted-sum optimization, the objective functions are summed up with weight m
In our experiments we follow (Le al. 2018) and set 0.01.
5

---

Under review as a conference paper at ICLR 2023
WeeghttedSum (x) (x((x) + (1 Mx), e.g., as adopted by CCAL (Du et al. 2021 This introduces an extra trade-off parameter 1) for tuning, but the true proportion of OOD data in Du is unknown; thus, cannot tell which criterion is more important. Furthermore, due to the lack of data, there is not enough validation data for properly tuning 7/- For two-stage optimization (Haneveld
& van der Vlerk 1999; Ahhedd al. 2004]] Bindewald et al.; 2020 [aal et al. 2022), a subset of possible ID samples is first selected with threshold M(x) < ò, and then the most informative b samples in the subset are selected by maximizing (()). However, suppose 6 has not been properly selected. In that case, we might i) select OOD samples in the first stage (when 5 is too large), and these OOD samples will be more likely to be selected first due to their higher AL score in the second stage; ii) select ID samples that are close to existing samples (when ó is too small), and due to the natural conflict between the two criteria, the resulting subset will be non-informative, thus influencing the AL performance (Zhao et al., 2019).
Considering this contradiction between AL and OOD criteria, developing a combined optimization strategy that does not require manual tuning of this trade-off is vital. Therefore, we propose POAL for balancing ((x and M(x) automatically without requiring trade-off parameters. Consider one iteration of the AL process, where we must select samples from Du. The search space size is the number of combinations M choose b, C(M, b). and the search is an NP-hard problem in general. Inspired by (Qian al. 2015), we use binary vector representation of candidate subset: {0, 1JM, where 8;- = 1 represents ž-th sample is selected, and 8; 0 otherwise. For the unlabeled set Dn.. we denote the vector of AL scores as u = [U(x)]] and ID confidence scores as m [M )1 j= The criteria scores for the subset are then computed ou (s) s" u and om (s) s m The ranking relationships between two candidate subsets s and s' are as follows:
s' denotes that s' is dominated by s, such that both scores for s' are no better than those of 8, i.e., (s') S. ou (8) and (sr(s') S (s).
s' < denotes that s' is strictly dominated by s, such that s' has one strictly smaller score (e.g..
ou (s') < (s)) and one score that is not better (e.g., our (s') S (Mr(s))
and s' are incomparable if both s is not dominated by s' and s' is not dominated by s.
Our goal (see Eq. D is to find optimal subset solution s that dominates the remaining subset solutions with Σ s b. The large search space makes traversing all possible subset solutions impossible. Thus we propose Monte-Cart POAL for fixed-size subset selection. Monte-Carlo POAL iteratively generates candidate solution s random, and checks it against the current Pareto set P = (8,1,82.... }. If there are no candidate solution in P that strictly dominates s, then s is added to P and all candidate solutions in that are dominated by s are removed. In this way, a Pareto set P is maintained to include the solution(s) that dominate all other candidate solutions so far, while the solution(s) in P are incomparable with each other
Early-stopping. We adopt an early-stopping strategy automatically terminate POAL when there is no significant change in P after many successive iterations, which indicates that a rundomly generated has little probability of changing P since most non-dominated solutions are included in P [axena et al./ 2016). Firstly, we need to define the difference between two Pareto sets, P and Pl. We represent each candidate solution s as a a 2-dimensional feature vector, e(s) = (s) (s)). The two Pareto sets can then be compared via their feature vector distributions, V(P) = (u(s)) and V(P')= = (o(ss))><P> eP.. We employed the maximum mean discrepancy (MMD) (Gretton et al. 2006) to measure the difference:
MMD(P, PP)
|
Σ SEP u(s)
1 PT
E Ev "(>')
(6)
MMD is based on embedding probabilities in reproducing kernel Hilbert space H. and here we use the RBF kernel with various bandwidths (i.e., multiple kernels). At iteration I, we compute the mean and standard deviation (SD) of the MMD scores in sliding window of the previous 5w iterations,
M, = 8a0 - int-Au+1 MMD(P.-1,P.) S
1 E (MMD(P.-1, P.) M.) (7) Aw
If there is no significant difference in the mean and SD after several iterations, c.g., M, and S, do not change within two decimal points, then we assume that P has converged, and we stop iterating.
"To keep the two criteria consistent, we normalize the ID confidence scores: m + max (m) *POAL is inspired by POSS (Qian al. 2015), but have essential differences, see Appendix for details.
m.

---

Under review as a conference paper at ICLR 2023
Final selection. After obtaining the converged set P. we need to pick one E P as the final solution (i.e., final subset for querying). Selecting a final solution is still an open question in multi -ujti-objective optimization (see Appendix C.3 for more details).
s' = arg mingee
Σ EvEp ||s' - -sll"= arg m ENEP S''s
(8)
where we use the property s = s' b. In (8) we select the solution with a maximum intersection with other -on-domin solutions, i.e., find the solution whose selected samples are commonly used in all other solutions in P. Each solution in P represents a particular ID-OOD trade-off since appropriate choice of weight rj in the weighted-sum objective will lead to its selection. Thus, the maximum intersection operation selects the samples that are common to all settings of the ID-OOD trade-off. In this sense, POAL bypasses the tuning of ID-OOD trade-off hyper-parameters by selecting samples that work well for all trade-offs.
Pre-selection for large-scale data. We next consider using POAL on large datasets. The search space of size C(M, b) is huge for large unlabeled data pool. We propose an optional pre-selection technique to decrease the search space size. We select an optimistic subset Duub from D.. based on the dominance relationships between different pairs of samples from the original unlabeled data pool. Firstly, an initial Pareto front P Prre. set of non-dominated data points, is selected according to two objectives U(x) and M(x). Since the size of Prre is not fixed and might not meet our requirement of the minimum pre-selected subset size 8m: we iteratively update Daub by firstly adding the data points in Prre to Duub. and then excluding Saub from D,, for the next round of selection. The iteration terminates when (sub 2 Sm- The pre-selected subset size 8m is set according personal requirements (i.e., the computing resources and time budget). In our experiment, we set 8m 66.
We summarized POAL with early-stopping and pre-selection in Aggrrthmssaand] the Appendix The computational analysis is in Appendix D. More discussions of POAL are in Appendix G
4 EXPERIMENTS
In the experiments aim to: 1) evaluate the effectiveness of POAL on both classical ML and DL tasks under various scales of OOD data scenarios; 2) compare different multi-objective optimization strategies, i.e., weighted-sum, two-stage, and Pareto optimization; 3) observe how OOD data influence typical AL methods as discussed in Section 2
4.1 EXPERIMENTAL DESIGN
Datasets. For classical ML tasks, we use pre-processed data from LIBSVM (Chang & Lin 2011
Synthetic data: EX8 uses EX8a as ID data and EX8b OOD data (g 2008
Real-life data: Vowel ( (Asuncion & Newman] 2007 Aggarwal & Sathe 2015) has 11 classes,
and we use 7 classes as ID data and the remaining for OOD data, Letter (Frey & Slate 1991 Asuncion & Newman, 2007) has 26 classes, and we use 10 (a-j) as ID data and the remaining
16 classes (k-z) as OOD data. We also construct 16 datasets with increasing ID:OOD ratios, denoted as letter(a-k), letter(a-Il..... letter(a-z).
For DL tasks, we adopt the following image datasets:
CIFARIO (Krizhevsky et al. 2009) has 10 classes, and we construct two datasets: CIFARIO-04 splits the classes with ID:OOD ratio of 6:4, and CIFAR10-06 splits data with ratio as 4:6. CIFAR 100 (Krizhessky et al., 2009) has 100 classes, and we construct CIFAR100-04 and CIFAR100-06 using the same ratios as CIFARIO.
Appendix E.2 shows visualizations of the ID and OOD data in these datasets.
Baselines. Our work helps existing AL methods select more informative ID data samples while preventing OOD data selection. In our experiments, we adopt ENT as our basic AL sampling strategy for POAL framework. We compare against two baseline methods considering OOD data: CCAL
(Du et al. 202T) and SIMILAR (with FLVMI submodular function) (Kothawade et al. (02T). We compare against five normal AL methods without OOD detection: ENT, BALD (Gal et al. 2017
7

---

Under review as a conference paper at ICLR 2023
and LPL (Yoo & Kweon 2019) uncertainty- measures; k-Means parresentative-basee measure and BADGE (Ash et al., 2020) as combined strategy. To show the effectiveness of our compared it against two alternative combination strategies: weighted sum optimization (WeightedSum) using weights yj = {0.2, 0.5, J.8} and two-stage optimization (TwoStage) using threshold ó mean(m), Additionally, we add random sampling (RAND) and Mahalanobis distance (MAHA) as baselines. Finally, we report the oracle results of ENT where only ID data is selected first (IDEAL-ENT), which serves as ideal performance of POAL.
Pareto optimization,
Implementation details. The training/test split of the datasets is fixed in the experiments, while the initial label set and the unlabeled set is randomly generated from the training set. Experiments are repeated 100 times for classical ML tasks and 3 for DL tasks. We use the same basic classifier for each dataset (details in Appendix E Table D To evaluate model performance, accuracy and plot accuracy vs. budget curves to present the performance change with increasing labeled samples. To calculate average performance, we compute the area under the accuracy-budge curve (AUBC) (Zhan et al. (021b). with higher values reflecting better overall performance under varying budgets. More details about the experiments are in Appendix E including data information, dataset splits, ID OOD ratios, basic learner settings, and AL Syyper-parametes. (budget B, batch size b, and initial label set size), etc.
measure
(a) EX8
(b) vowel
(c) letter(a-k)
(d) letter(a-z)
(e) letter(AUBC)
Figure 3: (a)-(d) are accuracy vs. budget curves for classical ML tasks. The AUBC performances are shown in parentheses in the legend. To observe the effect of increasing ID:OOD ratio on letter datasets, we plot AUBC vs.
dataset curves in (e). The complete figures (more letter datasets) are shown in Appendix F.
Figs. aadnn
Sudge
Sedgi
Dedge
(a) CIFARIO-04
(b) CIFARIO-06
(c) CIFAR100-04
(d) CIFAR100-06
absaed CIFARIO OI
abited CIPARIO
000 dats oricted CITARIN
ont wlieea CITARIO
(e) CIFARIO-04 OOD
(I) CIFARIO-06 OOD
CIFAR100-04 OOD
(h) CIFAR100-06 OOD
Figure 4: Results of POAL with Pre-Selection on DL datasets. (a-d) are accuracy vs. budget curves. We present the mean and standard deviation (in parentchsis) of AUBC, (c-h) plot the ratio of OOD samples selected.
4.2 RESULT ANALYSIS
Overall performance. We present the overall performance of classical ML and DL tasks in Fig. and Fig. respectively. Considering each dataset (see data visualization in App. E.2). the ideal performance of POAL is shown on the synthetic dataset EX8 (Fig. 3 the ID data non-linearly separable, and the OOD data is far from ID data. These properties make the calculation of M very accurate. Meanwhile, OOD and ID data close to the decision boundaries have high entropy scores. The curve of our POAL is the closest to IDEAL-ENT, which indicates that POAL selects the most informative ID data while excluding the OOD samples. For real-life classical MI. datasets like vowel
8

---

Under review as a conference paper at ICLR 2023
and letter (Fig. 3p-)), POAL also demonstrates its superiority. In letter, fixed the ID data (letters a-j) and gradually increase the OOD data (letters k-z). From Fig we observe that except for IDEAL-ENT, all methods are influenced by increasing OOD data, e.g., the AUBC of RAND is 0.718 in Fig. Bc. 0.62 in Fig. B3.. Although our method is also influenced by increasing OOI) data, it is still superior all baselines. POAL also works on large-scale data with DL tasks. Fig. 4 shows the accuracy-budee curves (along with the AUBC (acc) values) for CIFARIO and CIFARIO0 with ID:OOD ratios 6:4 and 4:6, together with the number of OOD samples selected during AL processes. POAL outperforms both uncertainty-. representaaive/iivesssty based and combined AL baselines like LPL, BALD, k-Means and BADGE on all tasks. We next analyze various aspects of the experiments.
POAL vs. SIMILAR and CCAL. These three methods consider OOD data when selecting samples. As shown in Fig. 4e-h, the three methods effectively reduce the amount of OOD data that is selected. POAL outperforms CCAL and SIMILAR (FLVMI) by large margins on more challenging situations ID:OOD ratio is 4:6), POAL selects fewer OOD samples and achieves better AUBC performance (e.g., the AUBC value of POAL on CIFAR100-06 is 0.525, while the highest baseline SIMILAR 0.451). SIMILAR selects data samples close to labeled ID data and dissimilar to OOD data. prevents selecting the OOD data, but the selected IDD data may not be informative enough. CCAL utilized weighted-sum optimization to balance the informative and distinctive selections, and the trade-off parameter heavily affects its performance (also see Section 4.3 in (Du et al. (0211). Note that SIMILAR has multiple implementations with various submodular mutual information functions, and to provide fair comparison, also compared the most effective submodular functions (FLCMI) on the down- sampled CIFARIO dataset, as shown in the Appendix F3
POAL vs. other multi-objective optimization strategies. We compared our Pareto optimization against weighted-sum, two-stage optimization, and each single objective, i.c., ENT and MAHA. Both ENT and MAHA are ineffective when used as single selection strategy under OOD scenarios in Fig ENT always selects OOD data with higher entropy until all OOD data are selected (EX8 contains 210 OOD samples), and performs even worse than RAND. MAHA prefers to select ID samples first (see Appendix Fig. 77. but the data samples with smaller Mahalanobis distance are also easily classified and far from the decision boundary, and thus not informative. POAL outperforms the other combination strategies, WeightedSum for different values and TwoStage, on all tasks. This demonstrates that joint Pareto optimization better handles the two conflicting criteria.
POAL vs. normal AL. We run experiments on normal AL methods, including uncertainty-basee (ENT, LPL and BALD), diversity-based (k-Means) and combined methods (BADGE). A Although LPL performs fairly well on standard CIFARIO and CIFARI0O datasets Yoo & Kweon 2019 Zhan et al. 2022a). in OOD scenarios. OOD data have larger predicted loss values and thus inluunnce the selection. Similar phenomena were observed with BALD and ENT. The performances of k-Means are close to RAND since both methods sampled ID/OOD data with the same ratio. Although k-Means will not select OOD samples first like uncertainty- based measures, it still failed to provide comparable performances. POAL performs well by selecting fewer OOD samples during the AL iterations, as shown in Fig. 4e) BADGE performed similar to M-Means The first stage of BADGE calculates the gradient embedding per sample, and use k-Means++ for clustering. These operations results in both high informative ID and OOD data are mixed together with high gradient values, therefore, BADGE failed distinguish ID/OOD data. In DL tasks, we adopted pre-selection to deal with large-scale datasets. Although it makes the search space smaller, i.e., changes the global solutions to local solutions, POAL-PS still performs significantly better than baseline AL methods.
Additional experiments including sensitivity analysis of sw. POAL with other AL methods, more multi-objective optimization methods, POAL vs. SIMILAR (FLCMI) are presented in Appendix P
5 CONCLUSION
In this paper, we proposed an effective and flexible Monte Carlo POAL framework for improving AL when encountering OOD data. Our POAL framework i) incorporates various AL sampling strategies; ii) handles large-scale datasets; iii) supports batch-mode AL with fixed batch size; iv) does not require tuning off arper-parameter and v) works well on both classical ML and DL tasks. Future work could improve the OOD detection criterion by introducing OOD-specific feature representations and other techniques for better distinguishing ID/OOD distributions.
9

---

Under review as a conference paper at ICLR 2023
ETHIC STATEMENT
There is a legitimate concern that POAL with pre-selection strategies (POAL-PS) on large-scale datasets that would induce bias problems. Since the Pre-Selection technique would firstly filter some unlabeled data samples in one AL iteration, and thus have probability of being selected. Future work should concern more about pre-selection techniques. What kind of high informative data samples may miss? Is the pre-selected subset enough representative?
REPRODUCIBILITY STATEMENT
For models, we provide an implementation POAL in supplementary material, including the classical ML task and DL task versions. For datasets, all datasets employed in this paper public datasets.
REFERENCES
Charu C Aggarwal and Saket Sathe. Theoretical foundations and algorithms for outlier ensembles. Acm sigkdd explorations newsletter, 17(1):24 2015.
Shabbir Ahmed, Mohit Tawarmalani, and Nikolaos Sahinidis. A finite branch-and-bounde algorithm for two-stage stochastic integer programs. Mathematical Programming, 100(2)3355-37) 2004.
Jordan T. Ash, Chicheng Zhang, Akshay Krishnamurthy, John Langford, and Alekh Agarwal. Deep batch active learning by diverse, uncertain gradient lower bounds. In 8th International Con- ference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
Arthur Asuncion and David Newman. Uci machine learning repository, 2007.
William H Beluch, Tim Genewein, Andreas Nürnberger, and Jan Köhler. The power of ensembles for active learning in image classification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 9368-9377, 2018.
Alina Beygelzimer, Sanjoy Dasgupta, and John Langford. Importance weighted active learning. In Proceedings of the 26th annual international conference machine learning, pp. 49-56, 2009.
Viktor Bindewald. Fabian Dunke, and Stefan Nickel. Modeling multi-stage decision making under incomplete and uncertain information. Technical report, Technical report, 2020.
Erdem Bıyık, Kenneth Wang, Nima Anari, and Dorsa Sadigh. Batch active learning using determi- nantal point processes. arXiv preprint arXiv: (00.:/9900007777 2019.
Chih-Chuna Chung Chang and Chih-Jen Lin. Libsvm: library for support vector machines. ACM transactions on intelligent systems and technology (TIST), 2 2011.
PM Chaudhari, RV Dharaskar, and VM Thakare. Computing the most significant solution from pareto front obtained in multi-objective evolutionary. International Journal of Advanced Computer Science and Applications, 144) 2010.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020.
Yufei Cui, Wuguannan Yao, Qiao Li, Antoni B Chan, and Chun Jason Xue. Accelerating monte carlo bayesian prediction via approximating predictive uncertainty over the simplex. IEEE Transactions on Neural Networks and Learning Systems, 2020.
Geoff Davis, Stephane Mallat, and Marco Avellaneda. Adaptive greedy approximations. Constructive approximation, 13(1):57-98, 1997.
10

---

Under review as a conference paper at ICLR 2023
Antoine de Mathelin, François Deheeger, Mathilde Mougeot, and Nicolas Vayatis. Discrepancy- based active learning for domain adaptation. In International Conference on Learning Representations, 2021.
Arthur P Dempster, Nan M Laird, and Donald Rubin. Maximum likelihood from incomplete data via the em algorithm. Journal of the Royal Statistical Society: Series B (Methodological), 39(1): 1-22, 1977.
Li Deng. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 29(6):141-142, 2012.
Liat Ein Dor, Alon Halfon, Ariel Gera, Eyal Shnarch, Lena Dankin, Leshem Choshen, Marina Danilevsky, Ranit Aharonov, Yoav Katz, and Noam Slonim. Active learning for bert: an empirical study. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 7949-7962, 2020.
Pan Du, Suyun Zhao, Hui Chen, Shuwen Chai, Hong Chen, and Cuiping Li. Contrastive coding for active learning under class distribution mismatch. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8927-8936, 2021.
Dheeru Dua and Casey Graff. UCI machine learning repository, 2017. URL http:///accii
ics. uci. eed.mmddmmmm
Long Duong, Hadi Afshar, Dominique Estival, Glen Pink, Philip R Cohen, and Mark Johnson. Active learning for deep semantic parsing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics ( (Volume 2: Short Papers), pp. 43 48, 2018.
Sebastian Farquhar, Yarin Gal, and Tom Rainforth. On statistical bias in active learning: How and when to fix it. arXiv preprint arXiv:2101.11660 2021.
Peter W Frey and David J Slate. Letter recognition using holland-style adaptive classifiers. Machine learning, 6(2):161-182, 1991.
Yarin Gal, Riashat Islam, and Zoubin Ghahramani. Deep bayesian active learning with image data.
In International Conference on Machine Learning, pp. 1183-1192 PMLR, 2017.
Ravi Ganti and Alexander Gray. Upal: Unbiased pool based active learning. In Artificial Intelligence and Statistics, pp. 422-4311 PMLR, 2012.
Arthur Gretton, Karsten Borgwardt, Malte Rasch, Bernhard Schölkopf, and Alex Smola. kernel method for the woo-sampee-rroblee Advances in neural information processing systems, 19, 2006.
Shravan Kumar Gupta. Comparison of multi-criteria decision making methods for selection of handmade carpets. Journal of Natural Fibers, 19(2):658 668,, 2022.
Willem K Klein Haneveld and Maarten H van der Vlerk. Stochastic integer programming: General models and algorithms. Annals of operations research, 85:39, 1999.
Elmar Haussmann, Michele Fenzi, Kashyap Chitta, Jan Ivanecky, Hanson Xu, Donna Roy, Akshita Mittel, Nicolas Koumchatzky, Clement Farabet, and Jose M Alvarez. Scalable active learning for object detection. 2020 IEEE intelligent vehicles symposium (fv). pp. 1430-1435. IEEE, 2020.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribation examples in neural networks. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017.
Neil Houlsby, Ferenc Huszár, Zoubin Ghahramani, and Máté Lengyel. Bayesian active learning for classification and preference learning. arXiv preprint arXiv: / 1/2.5745, 2011.

---

Under review as a conference paper at ICLR 2023
Henrik Imberg, Johan Jonasson, and Marina Axelson-Fisk. Optimal sampling in unbiased active learning. In International Conference on Artificial Intelligence and Statistics, pp. 559-569. PMLR, 2020.
Rishabh Iyer, Ninad Khargoankar, Jeff Bilmes, and Himanshu Asanani. Submodular combinatorial information measures with applications in machine learning, In Algorithmic Learning Theory, pp. 722-754. PMLR, 2021.
Siddharth Karamcheti, Ranjay Krishna, Li Fei-Fei, and Christopher D Manning. Mind your outliers! investigating the negative impact of outliers on active learning for visual question answering. arXiv preprint arXiv:2107.0233 2021.
Vishal Kaushal, Suraj Kothawade, Ganesh Ramakrishnan, Jeff Bilmes, and Rishabh Iyer. Prism: A unified framework of parameterized submodular information measures for targeted data subset selection and summarization. arXiv preprint arXiv:2103.000 00128, 2021.
Ksenia Konyushkova, Raphael Sznitman, and Pascal Fua. Learning active learning from data. Advances in neural information processing systems, 30, 2017.
Suraj Kothawade, Nathan Beck, Krishnateja Killamsetty, and Rishabh lyer. Similar: Submodular information measures based active learning in realistic scenarios. Advances in Neural Information Processing Systems, 34, 2021.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of-distributton samples and adversarial attacks. Advances neural information processing systems, 31, 2018.
David D. Lewis and Jason Catlett. Heterogeneous uncertainty sampling for supervised learning. In Machine Learning, Proceedings of the Eleventh International Conference, Rutgers University, New Brunswick, NJ, USA, July 10-13, 1994, pp. 148-156. Morgan Kaufmann, 1994.
Shiyu Liang, Yixuan Li, and Rayadurgam Srikant. Enhancing the reliability of out-of-distribution image detection in neural networks. arXiv preprint wxXiv:/706.026900 2017.
Shiyu Liang, Yixuan Li, and R. Srikant. Enhancing the reliability of out-of-distribution image detection in neural networks. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 2018, Conference Track Proceedings. OpenReview.net, 2018.
Shengcai Liu, Ning Lu, Cheng Chen, Chao Qian, and Ke Tang. Hydratext: Multi-objective optimiza- tion for adversarial textual attack. arXiv preprint arXiv:2111.01528 2021.
Rafid Mahmood, Sanja Fidler, and Marc Law. Low budget active learning via wasserstein distance:
An integer programming approach. arXiv preprint S..iv:2106.02966 2021.
Timothy Marler and Jasbir S Arora. The weighted sum method for multi-objective optimization:
new insights. Structural and multidisciplinary optimization, 41(6))853-86) 2010.
Kaisa Miettinen. Nonlinear multiobjectivve optimization, volume 12. Springer Science & Business Media, 2012.
Andrew Ng. Stanford cs229 machine learning andrew ng. 2008
Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathicu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, al. Scikit-learn: Machine learning in python. the Journal of machine Learning research, 12:2825-2830, 2011.
Chao Qian, Yang Yu, and Zui-Hua Zhou. Subset selection by pareto optimization. Advances in neural information processing systems, 28, 2015.
Chao Qian, functions. In IJCAI, pp. 2606-2612, 2017.
Shi, Yang Yu, Ke Tang, and Zhi-Hua Zhou. Optimizing ratio of monotone
12

---

Under review as a conference paper at ICLR 2023
Jie Ren, Peter J Liu, Emily Fertig, Jasper Snoek, Ryan Poplin, Mark Depristo, Joshua Dillon, and Balaji Lakshminarayanan, Likelihood ratios for -uitt--diittibuutin detection. Advances Neural Information Processing Systems, 32, 2019.
Pengzhen Ren, Yun Xiao, Xiaojun Chang, Po-Yao Huang, Zhihui Li, Brij Gupta, Xiaojiang Chen, and Xin Wang. A survey of deep active learning. ACM Computing Surveys (CSUR), 54(9):1-40, 2021.
Douglas A Reynolds. Gaussian mixture models. Encyclopedia of biometrics, T41(659-663), 2009.
Ahti Salo, Juho Andelmin, and Fabricio Oliveira. Decision programming for mixed-integer multi- stage optimization under uncertainty. European Journal of Operational Research, 299(2)5500-555, 2022.
Christoph Sawade, Niels Landwehr, Steffen Bickel, and Tobias Scheffer. Active risk estimation, In ICML, 2010.
Dhish Kumar Saxema, A. Sinha, João A. Duro, and Q Zhang. Entropy-based termination criterion for multiobjective evolutionary algorithms. IEEE Transactions on Evolutionary Computation, 20 (4):485-495) 2016.
Panos Seferlis and Michael C Georgiadis. The integration of process design and control. Elsevier, 2004.
Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018.
Burr Settles. Active learning literature survey, 2009.
Sebastian Seung. Manfred Opper, and Haim Sompolinsky. Query by committee, In Proceedings of the fifth annual workshop on Computational learning theory, pp. 287-294, 1992.
Claude Elwood Shannon. A mathematical theory communication. ACM SIGMOBILE mobile computing and communications review, 5(1):3-55, 2001.
Dan Shen, Jie Zhang, Jian Su, Guodong Zhou, and Chew Lim Tan. Multi-criteria-bassee active learning for named entity recognition. In M Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics, 21-26 July, 2004, Barcelona, Spain, pp. 589-596. ACL, 2004.
Changjian Shui, Fan Zhou, Christian Gagné, and Boyu Wang. Deep active learning: Unified and principled method for query and training. In International Conference on Artificial Intelligence and Statistics, 1308-1318, PMLR, 2020.
Samarth Sinha, Sayna Ebrahimi, and Trevor Darrell. Variational adversarial active learning. In Proceedings of the JEEE/CVF International Conference on Computer Vision, pp. 5972-5981, 2019.
Sagar Vaze, Kai Han, Andrea Vedaldi, and Andrew Zisserman. Open-set recognition: A good closed-set classifier is all you need. arXiv preprint arXiv:2110.06207, 2021.
Dan Wang and Yi Shang. A new active labeling method for deep learning. In 2014 International joint conference on neural networks (IJCNN), pp. 112-119. IEFE, 2014.
Jean-Paul Watson, Harvey J Greenberg, and William E Hart. A multiple-objective analysis of sensor placement optimization in water networks. In Critical transitions in water and environmental resources management, pp. 1-10. 2004.
Jingkang Yang, Kaiyang Zhou, Yixuan Li, and Ziwei Liu. Generalized out-of-distributioo detection: A survey. arXiv preprint 2rXiv:2110.1132: 2021.
Changchang Yin, Buyue Qian, Shilei Cao, Xiaoyu Li, Jishang Wei, Qinghua Zheng, and Ian Davidson. Deep similarity based batch mode active learning with International Conference Data Mining (ICDM), pp. 575-584. IEEF, 2017.
In 2017 IEEE
13

---

Under review as a conference paper at ICLR 2023
Donggeun Yoo and In So Kweon. Learning loss for active learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 93-102, 2019.
Xueying Zhan, Qing Li, and Antoni B Chan. Multiple-criteria based active learning with fixed-size determinantal point processes. arXiv preprint arXiv:2107.01622, 2021a.
Xueying Zhan, Huan Liu, Qing Li, and Antoni B. Chan. comparative survey: Benchmarking for pool-based active learning. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, pp. 4679 4686, 2021b.
Xueying Zhan, Qingzhong Wang, Kuan-hao Huang, Haoyi Xiong, Dejing Dou, and Antoni Chan. A comparative survey of deep active learning. arXiv preprint arXiv:2203 2022a.
Xueying Zhan, Yaowei Wang, and Antoni B Chan. Asymptotic optimality for active learning processes. In The 38th Conference on Uncertainty in Artificial Intelligence, 2022b.
Yu Zhao, Zhenhui Shi, Jingyang Zhang, Dong Chen, and Lixu Gu. A novel active learning framework for classification: Using weighted rank aggregation to achieve multiple query criteria. Puttern Recognition, 93:581-603 2019.
Aimin Zhou, Bo- Yang Qu, Hui Li, Shi-Zheng Zhao, Ponnuthurai Nagaratnam Suganthan, and Qingfu Zhang. Multiobjective evolutionary algorithms: A survey of the state the art. Swarm and evolutionarr computation, 1(1):32 49, 2011.
14

---

Under review as a conference paper at ICLR 2023
Appendix
Table of Contents
Pseudo Codes POAL
More Discussions of POAL and POSS
17
Related Work
19
AL. under OOD data scenarios
19
AL with data distribution shift
20
C.3 Final Selection in Multi-obecctive Optimization
20
D Computational Complexity Analysis of POAL
21
D. Monte-Carlo POAT
21
D.2
Pre-selection for large-scale datasets
21
E
Experimental Settings
22
E.1
Datasets
................................
22
Visualization of Datasets
................................
................................
22
Baselines and Implementations
24
F
Experiments
24
F.1
Experiments on Classical Tasks
25
F2
Experiments on DL Tasks
25
F3
Additional Experiments (1): POAL vs. SIMI (FLCMI)
25
F4
Additional Experiments (2): Sensitivity Analysis of Hyper parameters
27
F3
Additional Experiments (3): More objeiveee Optimization Strategies
28
F6
Additional Experiments (4): POAL Incorporated with other AL Methods
29
F7
Summarising Experiments Results on DL Tasks
29
F8
Visualization of the subset selection of POAL
30
G
Discussions of POAL
30
Discussions about How POAL Works
30
2
Limitations
30
A PSEUDO CODES OF POAL
We summarize the POAL with an early stopping technique and pre-selection operation in Algorithm| 1 and Algorithm 2 respectively. In Algorithm [T] The implementation of MMD the RBF kernel with various bandwidths with open-source implementation
B MORE DISCUSSIONS OF POAL AND POSS
In this section, we mainly introduce Pareto Optimization for Subset Selection POSS QQian et al. 2015) and how our POAL is inspired from POSS. We then discuss the major difference between POAL and POSS.
'The Py implementation of MMD is: htttps://gittub.com///
deep-transfed transfer---rrrrr wwwnrferr-eerrnnnee ob h/a//masttee /MADT//MUAAAAAAAAAA MF'SAN/ MA'SAN_33rre 3 mmd hmdd.yy .py
15

---

Under review as a conference paper at ICLR 2023
Algorithm 1 Monte-Carlo POAL with early stopping under OOD data scenarios.
Require: Unlabeled data pool D with size M, criteria (xx) and M(x), batch size b, maximum repeat time 7, population interval Pine, sliding window size 8u and final decision function F. Ensure: A subset D. of D,, where [D, =1=66 b. 1: Lets= = {(0)MM p = {s} and =
2: for in 1.... T do
3: Generate a random solution with condition Es; = b. then
4: if fl such that
5:
Q = 2 3ss
6: p= (P (s)..
7:
end if
(Terminate the loop early if the pareto set p converged.
8:
if' * 0 then
9:
Calculate MMD, according to Equation 5,
10:
if MOD(t,, Pmn) O then Calculate M, and St with interval Pan according to Equation respectively.
11:
12:
M, = ROUNDOMM 2), St ROUND(S.,2) 2).
13:
if M, M,-1
Mi-s. and S, S: = -S then
14:
Break
15:
end if
16: end if
17: end if
18:
end for
19:
return arg maxsep F(s).
Algorithm 2 Pre-selecting technique on large-scale datasets.
Require: Unlabeled data pool Du. criteria 2/ and M, minimum size of subset Sm of pre- mel-selection. Ensure: A pre-selected subset Sub of D, with suubl Sm-
1: Let suhh = 0. = 0.
2: while [D suh| <. 8m do
3: Let Dp =
4: for in 0, ... [Dul do
5:
if fx E D, such that Di < x then
6:
Q [x E D x < }
7:
Dp = (D, (1) U
8:
end if
9:
end for
DM D2, D,p.
11: Danh = Danh U
end while
13:
return Dsub-
We briefly introduce POSS first. POSS is originally applied to the subset selection problem, which is defined as follows:
aig f(S) s.t. S < k,
SCV
where V = [X1 Xx3 } is set of variables, f is criterion function and & is a positive integer. The subset selection problem aims select subset S C such that f is optimized with the constraint [S] < k where denotes the size of a set. The subset selection problem is NP-hard in general (Davis et al. 1997).
POSS solves the subset selection by separating the problem into two objectives, namely optimizing the criterion function and minimizing the subset size. Usually, the two objectives are conflicting. Thus the problem is transformed into multi-objective optimization problem:
min SCV
(f()),
16

---

Under review as a conference paper at ICLR 2023
Due to different trade-offs of objectives, multi-objective optimization algorithms need to find Pareto set P containing Pareto optimal solutions. Specifically, POSS first initializes with a random solution and then selects a solution from P generate new solution by flipping each bit with probability 1/n (n is the number of variables). The new solution will be added to P if is not strictly dominated by any solution in P, and the solutions in P that are dominated by the new solution will also be excluded. POSS repeats this iteration for T times, and it is proved to achieve the optimal approximation guarantee of (1 1/e) with E[T] I 2ek-n expected running time.
Our Monte-Carlo POAL is inspired by POSS, but there are two essential differences, as follows:
Firstly, although POSS is in the form of bi-objective optimization, it supports only one criterion function, c.g., just (x),, while the other criterion is that the subset size does not exceed b (i.c., L; 8/ S 6). However, our POAL needs to solve two different criterion functions, which is much harder problem.
Secondly, POSS solves the problem with the constraint |S| < flipping the bits of the solutions from the previous iterations, i.e., solutions in P. This operation may change the subset size, violating our setting of fixed-size solutions. Thus, we adopt the Monte-Carlo approach to generate fixed-size solutions unrelated to the solutions from the previous iterations. However, our POAL needs fixed-size subset solutions, which means the search space different.
POSS generates new solutions by
Due to these differences, we proposed Monte Carlo POAL for our AL under OOD data scenarios tasks. We randomly generate candidate solutions with fixed batch sizes. I) controls candidate solutions in P have the same fixed size; 2) compared with POSS, Monte Carlo POAL does not depend on previous selections, which keeps the method free from initialization problems.
POSS cannot satisfy our requirements. Due to these differences in how candidate solutions generated and the number of criteria, the theoretical bound on the number of iterations T needed for convergence of POSS (Qian et al. 2015) cannot be applied when should the iterations of our Monte-Carlo POAL terminate? We noticed in previous research QQan et al. 2017) that the Pareto set empirically converges much faster than ETT (see Fig. 2 in Qian et al. 2017)) Therefore, we propose an early-stopping technique, which has been introduced
Monte-Carla POAL method. Thus,
Section 3.4
C RELATED WORK
This section is an extension of the Section We next introduce methods that consider AL under OOD data scenarios and the relationshipddiffrrenee between the scenarios of OOD AL and AL with biased data. We also discussed the final selection problem in Multi-objective optimization.
C.I AL UNDER OOD DATA SCENARIOS
We discusse the related work that involves AL under OOD data scenarios. As mentioned in the main paper, there is little related work AL under OOD data scenarios. To the best of our knowledge, there are two published works that discussed AI. under OOD) data scenarios: Contrastive Coding for Active Learning (CCAL) (Du et al. (02T) and Submodular Information Measures Based Active Learning In Realistic Scenarios (SIMILAR) (Kothawade et al., 202I). We next briefly introduce these two papers and compare them with our work.
C.1.1 POAL vs. CCAL
Du et al. (021)) considers the class distribution mismatch problem in AL. Their goal is to select the most informative samples with matched categories. CCAL is the first work related to AL for class distribution mismatch. It proposed contrastive coding- based method, which extracts semantic and distinctive features by contrastive learning. Semantic feature refers to category-level features that can be exploited to filter invalid samples with mismatched categories. Distinctive features describe the individual level. It an AL task-specific feature to select the most representative and informative unlabeled data samples. CCAL achieves good performance well-studied datasets, e.g., CIFAR 10 and CIFAR 100 datasets. CCAL utilized self-supervised models like SimCLR and CSL It is excellent idea since it will extract semantic and distinctive features compared with normal
17

---

Under review as a conference paper at ICLR 2023
feature representations like the output of the penultimate layer of neural networks. However, it also brings limitations, that training a self-supervvsed model has both high computational and timing costs, especially on large-scale data sets. Based on the release code of CCAL distinctive feature extraction model (with 700 epochs and batch size 32, on CIFARI 10 dataset) on V100 GPU. Additionally, CCAL adopted weighted-sum optimization combine semantic and distinctive scores, whose acquisition function is as follows:
take 3 days to train
aCCAL
tanh (k5 semantic(xx) -t) Saistinctive (x),
where the threshold t is for selectively narrowing the semantic scores of samples, and & controls the slope of the tanh function. oCCAL introduces two hyper-panee ayper-parameter' for balancing the semantic and distinctive scores. As discussed in Section 4.3 in (Du et al. (02T), the choice of the two hyper- parameters will influence the final performance. The final shortcoming comes from calculating the distinctive to obtain representativeness information. CCAL needs pair-wise comparison among the whole data pool, whicih time and memory-consumin on large-scale datasets.
C.1.2 POAL SIMILAR
SIMILAR is an AL framework using the previously proposed submodular information measures (SIM) acquisition functions. Apparently, (Kothawade et al., 202T) is an application of SIM (Iyer et al. 2021 Kaushal et al. 202T). Kothawade et al. (2021) proposed many SIM variants to deal with realistic scenarios, e.g., data imbalance, rare-class, OOD, etc. AL under the OOD data scenario is a sub-task of their work. They adopted a submodular conditional mutual information (SCMI) function that best matches AL under OOD data scenario tasks, that is, Facility Location Conditional Mutual Information (FLCMI,, and the function is:
(A, = Σ
max(min 81g. max 8:g) - mAx 8ij. 0), JEP
JEA
JEQ
NEV
where they use the currently labeled OOD points as the conditioning, set and the currently labeled in-distribution (ID) points the query set Q. A is unlabeled data set. It jointly models the similarity between A and Q and their dissimilarity with P. The advantages and disadvantages of SIMILAR are both clear. However, the proposed FLCMI is memory-comee connuoor--oonuiingg.. and it cannot handle large-scale datasets. For instance, we cannot run experiments on the full CIFARIO dataset with one V100 GPU with 32GB memory (memory error is reported). In (Kothawade et al. 202T), they conduct OOD-related experiments on a downsampled version of CIFARIO. They downsample the dataset to size 15.6K (the size of the whole CIFAR10 dataset for training is 50K).
(Kothaaade et al. 202I) provided a partition trick to solve the (eeoryy-cossuman problem by randomly separating the unlabeled data into several pieces and running the AL algorithm on every piece. It is effective but will hurt the final performance, especially when the number of partitions is large. We do not use the partition trick in our experiments provide fair comparison. We conduct a time and memory-efficient submodular function FLVMI as the baseline in main experiments (see Fig 4). FLVMI has comparable performance experiments on the down-sampled CIFARIO dataset, following the same experimental settings in Kothawade et al. (202T). The function of FLVMI is:
the experimental part of (?). We also conduct
(A = Σ
min (max Sij, max S:z). JEA
JEQ
iE D.l
To sum up, the existing research AL under OOD data scenarios contains the following problems to be overcome: 1) time/memory cost on large-scale datasets; 2) additional trade-off hyper-parameters for balancing different (even conflict) criteria need hand-tuning We propose our POAL framework to address these issues. To solve the first problem, we propose pre-selection and early stopping techniques to reduce the search space to save time and memory cost. To solve the second problem, we apply Pareto optimization for balancing various (even conflict) objectives. Therefore, our proposed framework can more easily be adapted to various tasks.
tuned by extra validation set.
hhttps://gtthhi www.c/////CCC. -MI
18

---

Under review as a conference paper at ICLR 2023
C.2 AL WITH DATA DISTRIBUTION SHIFT
There are similarities between AL with OOD scenarios and AL with distribution shift scenarios. Both have a gap between true underlying data distribution and the estimated distribution (estimated from labeled and unlabeled data pool). The significant difference between Al techniques under OOD data scenarios and AL. techniques with distribution shift lies in the difference between OOD and distribution shift. Distribution shift (aka dataset shift) refers the discrepancy between the data distributions of the training and testing sets (or true underlying data distribution) (Zhan et al. 2022b). It causes principle problem during the model fitting step in AL since some regions with large densities in the unlabeled data pool may not be well represented by the labeled data. For AL with distribution shifts, some work like (Beygelzimer et al., 2009) Sawade et al. 2010; Ganti & Gray 2012 Farquhar et al. 202 1 Zhan et al.) 2022b) tried first to model the discrepancy between labeled data and true underlying data distribution based on techniques like importance sampling and then train unbiased basic learner(s). Some work model the difference/dissrepann between labeled set and unlabeled set (or the full data pool) to help construct AL sampling strategies. (Shui et al. (2020) utilizes the unlabeled data information by training a discriminator to distinguish labeled and unlabeled data sets. [Sinha et al. 2019) learns the distribution of labeled data in a latent space using a VAE. A binary adversarial classifter (discriminator) is then trained to predict unlabeled examples. Mahmood et al. 2021) minimizes the Wasserstein distance between the unlabeled set and the set to be labeled as AL sampling strategy. de Mathelin et al. 202 discuss AL for general loss functions under domain shift and further provide generalization bound of the target risk involving pairwise distances between sample points based on localized discrepancy distance. Distribution shifts can be
further applied to train better basic classifiers (Imberg et al. 2020 Farquhar et al. 202 1 Zhan et al.
2022b). However, in AL with OOD data scenarios, OOD samples are hard to be utilized since they are useless to model training since OOD samples are not in the classes of interest of the classification
task. Thus, the current research aims to detect OOD samples and avoid sampling them.
C.3 FINAL SELECTION IN MUL.TI-OBJECTIVE OPTIMIZATION
Selecting a solution from large Pareto set P is potentially intractable for a decision maker: It an open question and is still being discussed in the multi-objective optimization field. Previous work (Chaudhari et al.) 2010) has summarized several ways to obtain a final solution: (i) reformulate the problem as single objective problem using additional information required by the decision- makers, such as the relative importance or weights of the objectives, goal levels for the objectives, values functions, etc. (ii) Decision makers interact with the optimization procedure typically by specifying preferences between pairs of presented solutions. (iii) Output a representative set of non-dominated solutions approximating the Pareto front, e.g., regarding the candidate solutions as data points, perform clustering and outputting the centroid as final solution.
Considering our tasks: AL under OOD data scenarios, (i) needs external information, e.g., which criterion is more important? Alternatively, it needs the extra trade-off parameters of the AL/OOD criteria. Nevertheless, in our work, we aim to propose a general framework. We must consider the worst situation; we have no extra information. If extra information is accessible, we can use (i) to get the final solution. (ii) needs to manually pick a solution or provide the preferences, which is also inaccessible. (iii) is the only way to select final solution without additional information, which is why we adopt it.
D COMPUTATIONAL COMPLEXITY ANALYSIS OF POAL
D.1 MONTE-CARLO POAL
Our proposed Monte-Carlo POAL for fixed-size subset selection used unordered sampling with replacement. Denote the ground set AA with size N. At each iteration, B (batch size, B < NN samples are randomly selected. So the search space contains S = C combinations. The worst case of POAL is to find all Pareto optimal solutions P = {81, $2, ... out of S combinations (each solution is a subset of B distinct samples out of N samples). Let |P] M (M < S). Hence, the computational complexity is the expected number of iterations that all of the M Pareto optimal solutions have been selected at least once, which is calculated below:
19

---

n. The computational complexity of Under review as a conference paper at ICLR 2023
At each iteration, there is probability S that one particular solution s, is not selected. So after i iterations, there is probability ($z that solution s; has been selected. Thus, the probability (م For M solutions, the probability that they have all been selected at least once after &: iterations is (1
that solution 8; has been selected at least once after k iterations
)
To obtain the probability that these M solutions have been selected least once exactly after iterations (i.e., not before k iterations), we need to subtract the probability that it is completed after k 1 iterations: (1 (85 (8-1)AM (1 (S-1) j 1)M Hence, the expected number of iterations is:
S - y&M
S )
[
k (1
Σ 1- ( (-5 5 k=
S --1
My
For small datasets, the size is tolerable, so the ground set A is just the unlabeled dataset D,, with size n, and the search space size S C In practice, we applied an early-stopping strategy, which can obtain a good enough solution with much fewer iterations.
D.2 PRE-SELECTION FOR LARGE-SCALE DATASETS
We designed pre-selection technique for large-scale datasets to reduce the search space of POAL. As introduced the last paragraph in Section 3.4 we iteratively select Pareto optimal samples from the unlabeled data. The worst case is that all unlabeled samples are Pareto optimal samples, so the number of comparisons is 0 + + + ... (n 1) the pre-selection is On??).
We pre-selected 8m aB samples out of unlabeled samples (a is set according to the computing resources and time budget, we set a = 6 in our experiment) for the subsequent Monte-Carke POAL. Therefore, N aB n, and the search space size S is reduced from C C. Also, with early-stopping strategy, our POAL can achieve a good enough solution with an acceptable number of iterations in practice.
E EXPERIMENTAL SETTINGS
E.1 DATASETS
We summarize the datasets we adopted in our experiments in Table including how to split the datasets (initial labeled data, unlabeled data pool, and test set), dataset information (number of categories, number of feature dimensions), and task-concerned information (maximum budget, batch size, numbers of repeated trials and basic learners adopted in each task). Primarily, we recorded the ID : OOD ratio in each task. We down-samples the letter dataset and controlled that each category only has 50 data samples in the training set.
Here we list the licence of the datasets used in our experiments:
EX8a and EX8b (gg 2008): Not listed.
vowel (Asuncion & Newman 2007 Aggarwal & Sathe 2015
Dua & Graff
2017
Aucune
licence fournie.
letter (Frey & Slate 1991,
Asuncion & Newman
2007
Dua & Grafi
2017 Not listed.
CIFARIO and CIFARI00 Krizhevsky et al. 2009 MIT Licence.
20

---

Under review as a conference paper at ICLR 2023
Table 1: Datasets used in the experiments.
# classes
# inttial labeled
#
dimension
laholed pool
sine
repeat trials
learner
mum Budget
650
10
100
nowel
25
5035
254
500
336 7 192
10
100
CICC
10
30
520
500
550
10
10
100
T.R
leterfa--
letter(a-m)
10
30
620
10 :
10
100
LR
letert(a-s)
10
16
30
670
500
550
10 1:4
10
100
L.R
o)
10
30
720
500
550
10
10
100
LR
letter(a-p)
(ettertag))
10
30
720
500
10
10
100
LR
(eneeta-r)
10
16
30
870
500
550
10 :
10
100
LR
letter(a-t)
10
30
920
300
550
10
10
100
LR
(enoria
(etter(aa))
10
30
1020
500
10
100
LR
leteeta-))
10
30
1070
500
550
10: 12
10
100
LR
(etertan))
30
1120
100
(efer((-))
(etertay))
10
30
1220
500
550
10: 15
10
100
LR
leteria::i
10
16
30
1270
300
550
10:
10
100
LR
CIARTO--M
RecNert8
CIFARIG-6
x
x
1000
49000
CIFAR100-M
60
522
x
x
1000
49000
6000
25000
6:14
500
R8
CIFAR/00-06
40
32
1000
4900
4000
20000
4
500
ResNet18
sampled
E.2 VISUALIZATION OF DATASETS
E.2.1 CLASSICAL ML TASKS.
We visualize the datasets of classical ML tasks using t-Distributed Stochastic Neighbor Embedding (t- SNEA shown in Figure We split ID/OOD data by setting OOD data samples semitransparent grey dots to better observe the distinction between the ID and OOD data distributions.
E.2.2 DL TASKS.
We also visualize the discrepancy using MMD in DL tasks since the raw features of DL tasks are hard visualize, like Figure We calculate the MMD distance within the ID data and the MMD distance between the ID and OOD data. To extract the feature representations in DL. tasks, we utilized a ResNet50 which is pre-trained on ImageNet dataset After extracting the features (the output of the penultimate layer of the pre-trained ResNet50) of ID and OOD samples, we next calculate the MMD scores, which is the same as the MMD settings in our early-stopping technique. Calculating the MMD score between these ID and OOD data samples is
(CIFAR 10 and CIFAR100 have 50, 000 training sets). To calculate the ID-ID MMD score, we randomly sampled 1, 100 X 2 instances in ID data and calculated the MMD score. We repeated this operation 100 times and took the average score. For calculating the ID-OOD score, we perform the same operation. The MMD scores of CIFARIO-04, CIFAR /IFAR/0-06, ZIFAR/00-00 and CIFAR/OR /00-06 are shown in Table [ Note that we did not calculate the MMD scores of ID-ID and ID-OOD during each stage of the AL process, since calculating the MMD scores on large-scale data with high-dimensional feature representations is time- and memory-consuning.
Table MMD scores ID-ID and ID-OOD
tasks.
ID-II
ID-0OD
CIFARIO-04
0.00608
0,04645
CIFAR10-06
0.00596
0.03565
CIFARI00-0
0.00603
0,01305
CIFAR100-06
0.00604
0.01659
http:////mppooooo/oo/////////ssa fold.
1Smm.
50s
21

---

Under review as a conference paper at ICLR 2023
D
(a) EX8
(b) vowel
(c) letter(a-k)
(d) letter(a-I)
(c) letter(a-m)
( letter(a-n)
g letter(a-o)
(h) letter(a-p)
(i) letter(a-q)
letter(a-r)
(k) letter(a-s)
letter(a-t)
(m) letter(a-u)
(n) letter(a-v)
(o) letter(a-w)
(p) letter(a-x)
(q) letter(a-y)
(r) lotter(a-z)
Figure S: Visualization of data sets in classical ML tasks by t-SNE. We set the semitransparent grey dots represent OOD data samples, the remaining colorful dots are ID data samples.
E.3 BASELINES AND IMPLEMENTATIONS
We introduce the baselines and the implementation details in this section.
For the basic learner/classifier, we adopted Gaussian Process Classifier (GPC), Logistic Regres- sion (LR), and ResNet18 in our experiments. We did not choose GPC in the letter dataset since the accuracy-budget curves based selecting a basic classifier is: that the basic classifier can provide soft outputs, that is, predictive class probabilities to calculate the uncertainty of each unlabeled sample, e.g., entropy. We the
GPC are not monotonically increasing. The requirement of
22

---

Under review as a conference paper at ICLR 2023
implementation of the GPC with RBF kerne and LR T0 of scikit-learn library Pedregosa et al. 2011 1 with default settings. For 8 we employed ResNet18 (He et al. 2016) based on Py Torch with Adam optimizer (learning rate: 1e - 3) as the basic learner in DL tasks. In CIFARIO and CIFARI00 tasks, we set the number of training epochs as 30, the kernel size of the first convolutional layer in ResNet18 is 3 (consistent with PyTorch CIFAR implementatior [ Input pre-processing steps include random crop (pad- = 4), random horizontal flip (p = 0.5) and normalization.
For the implementation of the classical ML baselines, we have introduced it in the main paper.
For the baselines in DL tasks, we use the implementation of DeepAL. 12 Zhan et al. 2022a). We provide simple introductions of BALD, LPI. and BADGE as follows:
Bayesian Active Learning by Disagreements (BALD) (Houlsby et al. 2011 Gal et al. 2017 7
:
chooses the data points that are expected to maximize the information gained from the model parameters, i.e., the mutual information between predictions and model posterior.
a loss prediction strategy by attaching a small parametric module that is trained to predict the loss of unlabeled inputs concerning the target model by minimizing the loss prediction loss between predicted loss and target loss. L.L picks the top b data samples with the highest predicted loss.
Loss Prediction Loss (LPL. (Yo & Kweon 2
Batch Active learning by Diverse Gradient Embeddings (BADGE) (Ash et al.) 2020): first measures uncertainty the gradient magnitude for the parameters in the output layer in the first stage; it then clusters the samples by k-Mcans++ in the second stage.
We train SimCLR (Chen et al. 2020), the semantic/distinctive feature extractor, which is provided by CCAL's source code. We train the two feature extractors with 700 epochs and batch size of 32 on a single V100 GPU.
For CCAL, We utilize the open-source code implementation of CCAI 13
We run all our experiments on a single Tesla V100-SXM2 GPU with 16GB memory except for running SIMILAR (FLCMI) related experiments. Since SIMILAR (FL.CII) needs much memory. We run the experiments (SIMILAR on down-sampled CIFAR/O) on another workstation with Tesla V100-SXM2 GPU with 94GB memory in total.
F EXPERIMENTS
This section is an extension of Section 4 in the main paper. We provide additional experimental results analysis and more experiments that do not appear in the main paper due to the page limit.
F.1 EXPERIMENTS ON CLASSICAL ML TASKS
We present the complete accuracy vs. budget curves and numbers of OOD samples selected vs. budget curves during AL processes in 6 and Fi We observe from Figure 5 Figure (6 and Figure 7 that the capability of Mahalanobis distance is limited by the distinction between ID data distribution and OOD data distribution. EX8 has distinct ID/OOD data distributions (see Figure a), thus the Mahalanobis distance well distinguishes ID and OOD data, and reaches the optimal performance (Figure a. MAHA has the same curve with IDEAL-ENT) However, for vowel and letter dataset, the ID/OOD data distributions are not as distinct as EX8 (see Figure 5 b-r), thus the performance of MAHA is influenced. Furthermore, affects the performance of our POAL makes our POAL behaves less perfectly on vowel and letter datasets than on the EX8 dataset. Similar conclusions appear in (Ren et al. 2019). This our future work to find better ways to differentiate ID and OOD data distributions.
[og_atic-regress..oo
23

---

Under review as a conference paper at ICLR 2023
8:55
rovel
Jdkeraak EX0OD m21
ahotisl 10:00 Itd
(a) EX8
(b) vowel
(c) letter(a-k)
(d) letter(a-1)
Id bectl EDOOD reato 233
ldterieci E200I map- 1244
Idkcttaci EE-00 193
idkuttapi (E-do0 mas
(c) letter(a-n)
(f) (a-ten(am)
g letter(a-o)
(h) letter(a-p)
Tataciao Iho0n 197
Letertent 10:000 onto I6cl
linett at 1h:00D ealo 109
latatar) EXeOn odl 1930
Her EiA oiytte
B112 dihecta
(i) letter(a-q)
letter(a-r)
(k) letter(a-s)
(I letter(a-t)
Lezutti (Di00D 30:11
Leaurti-st ID:000 eato- 1011
latarta-ao 000 16:17
Leturtinst 1D:000 anto-1ile
Sedget
Bedge
Dedge
Dedge
(m) letter(a-u)
(n) letter(a-v)
(o) lctter(a-w)
(p) letter(a-x)
leautiost 1D:000 eato- 10:15
Setutioc cOD 10:t
Dedges
(q) letter(a-y)
(r) lettcr(a-z)
Figure 6: Accuracy vs. budget curves for classical ML tasks. The AUBC performances are shown in parentheses the legend.
F.2 EXPERIMENTS ON DL TASKS
The comparative results are shown in Figure 8 It is an enlarged version of Fig. with the same contents. We have moderately better performance than CCAL on CIFARIO-04 dataset, i.e., our POAL has 0.762 AUBC performance and CCAL is 0.754. Note that in Figure 8,c, POAL has similar efficiency with CCAL on preventing selecting OOD data samples on CIFAR10-04 dataset. In CIFARI0-06 task, as the OOD ratio increases, our POAL outperforms CCAL, i.c. our POAL has 0.84 AUBC performance while CCAL is only 0.819. And POAL selects less OOD samples than CCAL, as shown in Figure d.. On CIFARION datasets, the advantages are more significant.
24

---

Under review as a conference paper at ICLR 2023
DX
(a) EX8
(b) vowel
(c) letter(a-k)
(d) letter(a-I)
(e) letter(a-n)
( letten(a-m)
(g) letter(a-o)
(h) letter(a-p)
(i) letter(a-q)
i letter(a-r)
(k) letter(a-s)
(l) letter(a-t)
(m) letter(a-u)
(n) letter(a-v)
(o) lctter(a-w)
(p) lettcr(a-x)
(q) letter(a-y)
(r) lottcr(a-z)
Figure 7: Numbers of OOD samples selected budget curves for classical ML tasks during AL processes.
POAL selected less OOD data than CCAL. Both POAL and CCAL perform better than normal AL sampling schemes (i.e., ENT, LPL, BALD, h-Means, BADGE and RAND) on both CIFARIO-04 and CIFAR10-06 datasets.
F.3
ADDITIONAL EXPERIMENTS (1): POAL vs. SIMILAR (FLCMI)
In the main paper, we have compared the SIMILAR with FLVMI as submodular mutual information. Referring its comparable performance CIFARIO dataset (see Figure 5a and Figure 5b in (Kothawade et al. 202I) as reference). We choose FLVMI as baselines in our main paper since is both time and memory efficient. However, FLVMI performs worse than FL.MMI since FL.MM is
25

---

Under review as a conference paper at ICLR 2023
specially designed for AL under OOD data scenarios. However, FLCMI is both time and memory consuming. In (Kothawade et al., 2021). they adopted down-sampled experiments. Although the partition trick could be applied to solve this time/memory consuuiin problem. the partition will influence the final performance due to the randomness. To provide a fair comparison. We conduct additional experiments the down-sampled CIFAR10 dataset on SIMILAR (FI.CMI), following the experimental settings in (Kothawade et al., 2021).
The comparison experiments are shown in Figure 9 Besides our POAL and SIMILAR (FLCMI), we also provide more baselines as reference, i.e., IDEAL-ENT, ENT, Margin (Wang & Shang, 2014) and RAND. As shown in Figure both SIMILAR (FLCMI) and POAL have better performance than normal AL sampling strategies. From the aspect AUBC evaluation metric, our model has comparable performance with SIMILAR, 0.667 vs 0.669. Nevertheless, we have a lower standard deviation value than SIMILAR, so our POAL is more stable. From the aspect of Accuracy vs. Budget curves, in early stages (e.g., Budget- < 2,500), our POAL is better than SIMILAR (FLCMI) and in latter stages SIMILAR (FLCMI) exceeds POAL. The reason is that in SIMILAR (FLCMI), they calculate the similarities between the ID labeled set and the unlabeled pool and the dissimilarities between the OOD labeled set and the unlabeled pool. In the early stages, the OOD data is insufficient. Thus SIMILAR(FLCMI) will select more OOD samples, as shown in Figure Ib. From this experiment, we find that SIMILAR (FLCMI) only performs well when have enough information OOD data samples, which results in more OOD data sample selection. Our POAL only considers the distance between unlabeled samples and labeled ID samples, so we are more efficient in preventing OOD sample selection. Additionally, our POAL is more widely adopted since we could be adopted on large-scale datasets. However, SIMILAR is limited by the computation condition. Our method is also more time efficient than SIMILAR (FLCMI), as shown in Table SIMILAR (FLCMI) runs five times longer than our POAL.
both ID and
Table 3: The mean and standard deviation of running time (in seconds) of the comparative experiments with repeated trials between our POAL and SIMILAR.
Method Time
POAL-PS
SIMILAR
IDEAL-ENT
ENT
Random
Margin
6419.0 (109.9)
32837.7 (897.7)
222660
1690.0 (15.3)
1507.7 (5.8) 1573.7 (9.8)
F,4 ADDITIONAL EXPERIMENTS (2): SENSITIVITY ANALYSIS OF HYPER- PARAMETERS
In our main paper and Algorithm I we have a prper-parameter s. (sliding window size) to control the early stopping condition. Large sw refers there is no significant change within sw x Pmn iterations/oopulation then we can stop early. In our experiments, set s 20. In this section, we conduct ablation study to see if less strict or more strict conditions influence the final performance, as shown in Table We found that 8 = 10 (less strict) does not affect the model performance (there is no significant difference of AUBC performance between s = 20 and s 10). Less strict carly stopping also requires less running time, e.g., in CIFAR100-04 04 dataset, we have 66,880.7 seconds average running time with s. 20 and 4,932.3 seconds average running time with s. = 10.
more strict early stop condition and vice versa. If
Besides Sw, we have another hyper-parameter sm, which controlts the minimum pre-selected subset size. Sm is designed to adapt to various computational resource constraints but not for controlling the AL/OOD trade-off. Larger Sm can be used when more computational resources/time are available, while smaller 8m can save computational cost but may lead to slightly sub-optimal solutions. In our experiments, we set 8m = Gb based on our computational resource situation, which works well in practice.
Although the numbers of are fundamentally different. In the weighted sum optimization, hyper-parameters relate to the weight- ing of the two objectives, significantly influencing the result. In contrast, the two hyper-parameters of POAL are about the efficient approximation (concerning computational resources), which mainly affects the initialization and stopping criteria, possibly resulting in sub-optimal approximate solutions. Note that we do not need to tune these parameters for various tasks.
of POAL are similar to the weighted sum, their purposes
26

---

Under review as a conference paper at ICLR 2023
Table 4: The hyper-parameter parameter sensitivity of sliding window size Sw., including mean and standard deviation (SD)
AUBC performance running time.
CIFARIO-04
AUBC
Time (scconds)
POAL with Su = 20
0.7620 (0.0033)
62082.7 (7371.9) 56107.0 (3364.2)
POAL with Sw: 10
0.7613 (0.0024)
CIFARIO-06
AUBC
Time
POAL with 8w
0.8400 (0.0029)
34946.0 (4860.2)
POAL with & = 10
0.8403 (0.0029)
47362.0 (188.7)
CIFART00-04
AUBC
Time
POAL with sw 20
0.4807 (0.0009)
66880.7 (I583.5)
POAL with 8 10
0.4797 (0.0017)
4932.3 (1891.2)
CIFARI00-06
AUBC
Time
POAL with & =
20
0.5253 (0.0005)
62762.0 (4277.4)
POAL with Su: 10
0.5270 (0.0008)
55955.3 (5162.4)
F.5 ADDITIONAL EXPERIMENTS (3): MORE MULTI-OBJECTIVE OPTIMIZATION STRATEGIES
In the main paper, we have introduced weighted optimization, widely adopted multiple- criteria/objective optimization problems. There is another available similar method called Weighted- Product Model (WPM). WPM is a popular multi-criteria decision-making (MCDM) method, similar the weighted-sum optimization model. The main difference between weighted-sum and weighted- product is that instead of adding in the main mathematical operation, it uses multiplication. We wrote WPM in AL with OOD data scenario as as WheightProd = arg maX U(s)" x ((s))1m)) We conduct single experiments show the performance on classical ML tasks (i.e., EX8 dataset). There is no significant difference between weighted-sum and weighted-produer optimization (Gupta 2022) (especially on subset selection, since the ranking of data samples are similarly produced by weighted-sum and weighted product). The experimental results are shown in Table 5 using weighted product optimization similar weighted-sun optimization.
seighted-product).
The results
Table 5: The comparison between weighted-sum and weighed-product optimization method. We test the
performance on EX8 dataset.
Method
AUBC (acc)
WeightedSum 11 0.5
0.692
WeightedSum 7) 0.2
0.685
WeightedSum 77 0.8
0.685
WeightedProd 77 -0.5
0.688
WeightedProd 77 0.2
0.688
WeightedProd 7 = 0.8 0.691
F.6 ADDITIONAL EXPERIMENTS (4): POAL INCORPORATED WITH OTHER AL METHODS
To present the flexibility of our POAL, e.g., is able to incorporate more AL sampling schemes, we conduct simple experiment classical ML tasks, we incorporate QBC (Seung al. 1992) and LAL (Konyushkova et al.) 2017). these two methods come from high-cited AL-related publications. The selection of basic AL sampling strategies according the comparative survey (Zhan al. 2021b), in which both QBC and LAL show competitive performance across multiple tasks and AL sampling strategies. We repeat the trials ten times in each experiment. The results are presented in Figure 10 We conduct experiments on EX8 and vowel datasets. This experiment shows that our POAL flexible to handle various AI. sampling strategies.
F.7 SUMMARISING EXPERIMENTS RESULTS ON DL TASKS
To observe the efficiency of each AL sampling strategy in experiments, summarized the overall performance across all models and datasets we adopted, including the mean and standard deviation (SD) of AUBC performance and running time. We record the running time from the start
27

---

Under review as a conference paper at ICLR 2023
Table 6: Overall comparison across all AL sampling strategies of all DL tasks, including and standard deviation (SD) AUBC performance and running time across three repeated trials.
CIFARIO-04
CIFARIO-06
AUBC
Time (seconds)
AUBC
Time (seconds)
RAND
0.7500 (0.0014)
10805.3 (148.0)
0.8080 (0.0008)
5217.7 (348.0)
IDEAL-ENT
0.8007 (0.0029)
14578.0 (838.2)
0.8737 (0.0019)
9398.0 (696.4)
POAL (5 20)
0.7620 (0.0033)
62082.7 (737I.9)
0.8400 (0.0029)
34946.0 (4860.2)
POAL (8 = 10)
0.7613 (0.0024)
56107.0 (3364.2)
0.8403 (0.0029)
47362.0 (188.7)
CCAL
0.7543 (0.0009)
76129.3
0.8196 (0.0037)
335713.3 (328:3)
SIMILAR (FLVMI)
0.7397(0.0012))
13942.7(116.2)
0.7947(0.0033
8550.0(231.3)
ENT
0.7350 (0.0029)
(8723.3
0.7960 (0.0057)
5485.7 (655.4)
LPL
0.7510 (0.0071)
12384.7 (1555.9)
0.7873 (0.0103)
3783.3 (326.9)
BADGE
0.7437 (0.0029)
41646.0 (12723.5) (127233))
0.8063 (0.0037)
17539.3 (628.0)
K.Means
0.7440 (0.0014)
18785.7 (2571.3)
0.8100 (0.0022)
8627.3 (407.2)
BALD
0.7480 (0.0022)
10478.3 (679.7)
0.8060 (0.0022)
4158.3 (431.1)
TwoStage
0.7300 (0.0029)
8342.3 (1059.4)
0.7970 (0.0028)
5524.7 (821.3)
WeightedSum-I.00
0.7337 (0.0019)
9489.7 (454.5)
0.8033 (0.0033)
2006.3 (13.8)
WeightedSum-5.0
0.7327 (0.0029)
11892.3 (1733.9)
0,8060 (0.0045)
7998.3 (81.6)
WeightedSum-002
0.7340 (0.0045)
8249.3 (2019.2)
0.8063 (0.0012)
5751. (795.3)
CIFAR 100-04
CIFAR 100-06
AUBC
Time (seconds)
AUBC
Time (seconds)
RAND
0,4560 (0.0016)
11563.3 (985.5)
0.4453 (0.0026)
11075.3 (I198.2)
DEAL-ENT
0.5250 (0.0008)
17736.7 (1694.7)
0.5707 (0.0017)
19098.0 (1458.8 8.)
POAL (8m 20)
0.4807 (0.0009)
"66880.7 (1583.5)
0.5253 (0.0005)
62762.0 (4277.4)
POAL (sw 10)
0.4797 (0.0017)
4932.3 (1891.2)
0.5270 (0.0008)
55955.3 (5162.4)
CCAL
0.4400
21253.0(217719)
0.4467(0.0024)
65303.0(1676.6)
SIMILAR (FLVMI)
(0.0026)
20732.7
0.4510(0.0024)
13137.0
ENT
0.4267 (0.0034)
11202.3 (2484.7)
0.4100 (0.0036)
10940.0 (1318.4)
LPL
0.4140 (0.0037)
17797.0 (4093.8)
0.4087 (0.0042)
5633.0 (252.3)
BADGE
0.4530 (0.0000)
58877.7 (13876.3)
0.4430 (0.0008)
58046.3 (17296.9)
KMeans
0.4527 (0.0005)
34944.3 (4121.3)
0.4413 (0.0021)
17465.0 (3570.3)
BALD
0.4467 (0.0040)
14699.0 (2675.9)
0.4313 (0.0061)
4574.3 (140.5)
TwoStage
0.4347 (0.0021)
11484.3 (4360.I)
0.4137 (0.0017)
3334.3 (91.1)
WeightedSuum-..0
0.4267 (0.0025)
12722.0 (4164.8)
0.4103 (0.0005)
9455.3 (468.5)
WeightedSum-5.0
0,4287 (0.0039)
14822.3 (6249.7)
0.4073 (0.0050)
11309.3 (1303.4)
WeightedSum-0.2
0.4290 (0.0022)
11736.7 (4394.3)
0.4097 (0.0033)
8865.7 (1230.1)
of the AL process to the output of the final basic learner. From Table 6 we can observe that our model outperforms all baselines (except for the ideal model IDEAL-ENT) in terms of AUBC performance. For running time, compared with normal AL sampling strategies, both POAL and CCAL are incomparable. Among POAL, CCAL and SIMILAR, which are designed specific to AL under OOD data scenarios, the running cost of our POAL is affordable. Especially, as mentioned in the previous section of hhper--aramene parameter sensitivity, the running timing cost our POAL can be reduced by loosing the early stop conditions (less ).).
F.8 VISUALIZATION OF THE SUBSET SELECTION OF POAL
direct way to validate the efficiency of POAL is to measure the difference between the selected subset by POAL and truly Pareto set. However, on normal datasets, the searching space of the optimal subset selection is too large, as analysed Section 3.4 it is a combination-level - CMM, b). Therefore, we use the subset selected by typical Pareto optimization (the operation is the same the operation of the first round of pre-section technique in Section 3.4 to represent the optimal subset selection instead. We visualize the subset selection data and multi-objective space, as shown in Figure Using typical Pareto optimization would i) select a subset with non-fit non-fixed subset size and ii) possibly include many OOD data samples since the entropy score of OOD data samples are relatively large.
28

---

Under review as a conference paper at ICLR 2023
POAL select more informative ID data less OOD samples based on current basic learners in Figure
We conduct toy experiments to show whether the subset selection generated by POAL is similar to the output generated Truly Pareto Optimization for subset selection. We down-sampled the size unlabeled data pool to 50 and set the batch size as the searching space is C(50, 5) 2, 118, 760, which is acceptable. We transverse the whole searching space and use the same Final-selection method as in POAL generate single final solution. As shown in Figure [2 We could observe that the candidate Pareto set P generated by POAL is close to the selection of Truly Pareto Optimization. In Figure 12 g. the solutions generated by POAL are located on Pareto Curve (or very close to Pareto Curve), in which Pareto Curve is generated by the truly pareto optimization. Observing Figure I2 the subset selection generate by POAL is better. This is because Truly Pareto Optimization finally selects the subset with higher entropy and lower ID score, thus might select OOD samples.
G DISCUSSIONS OF POAL
G.1 DISCUSSIONS ABOUT How POAL WORKS
In our work, choosing Mahalanobis distance-based measure to calculate ID confidence score has many reasons. Besides the superiority of OOD detection performance as introduced in (Ren et al. 2019), similar to combined strategies in AL, which has been introduced in Section 2 in the main paper, we combine uncertainty-basee and distance- basee criteria together. This allows us analyze the data in different aspects: task-agnostic aspect, refers to the uncertainty-basee measure (e.g.. ENT) and geometric aspect, refers the relative distance pair-wise similarity of unlabeled data samples. This guarantees a comparable performance under various OOD data scenarios.
We have demonstrated the effectiveness of POAL on various ratio of OOD data scenarios, as shown in Figure We have also conduct simple experiment on clean unlabeled data pool (no OOD data), POAL still works. We conduct a simple experiment on the vowel dataset, where we use all training data with 11 classes as ID data and there is no OOD data. (Note that in the AL+OOD setting, only 7 ID classes are used). On this pure ID data, the AUBC (acc) of ENT is 0.454, and our POAL is 0.465. Compared with normal AL, our POAL still maintains good performance when there are no OOD samples in the unlabeled data pool. This is because even if there are no OOD data in the unlabeled pool, POAL still be able to detect some outliers based on Mahalanobis distance and prevent selecting them, thus making better subset selection for AL (avoiding overfitting on outliers).
Our POAL is easily to be extended to any number of score functions, by extending the dominance relationship in Section 3.4jin in the main paper, and the vector scores for the early stopping criteria. Thus, it is possible adopt multiple AL or ID scores.
G.2 LIMITATIONS
Currently, our POAL has several limitations: (1) The accuracy of Mahalanobis distance-basee ID confidence score calculation; and (2) the Monte-Carlo sampling scheme our POAL for seeking non- dominated fixed-sized subset solutions is not high-efficiency. In future work, we will try more suitable feature representations to construct more distinct ID-/OOD- data distributions and more convenient features like semantic and distinctive feature representations in (Du et al. (002)). Regarding the efficiency of the Monte-Carlo sampling scheme, although many sampling schemes like (adaptive)
importance sampling and metropolis-hastimg sampling, these methods might suffer from initialization problems. Since the newly generated possible candidate solutions would rely on previous selections (e.g., P), which easily fall into local minima. Monte-Carlo sampling has no such problem. In future work, we will try to propose more efficient sampling methods for finding non-dominated subset solutions.
sampling would be more efficient than Monte-Caid -Crllo
29

---

Under review as a conference paper at ICLR 2023
CIFARIO 040DD:00D ratio 6:4)
CIFARIO 06LI::0D 4:6)
Accuracy
POAL-P OUTAMAORI
Accuracy
WIBE AN DELEEE
ETEAL INE MENEORT
SECELA CTANE TISAUCI
Rundim JLTHAARO
LIT PUTTOEE IET
ANL7 pubbirooo
2059
7590
18000
190E 1150
Budget
Budget
(a) Accuracy vs, Budget
(b) Accuracy vs. Budget
OOD data selected in CIFARIO
OOD data selected CIFAR
Ratio of OOD samples selocted
Ratio of OOD samples se lected
02
POAL-PS
TOAL-P5
0.-
SINELAR (TLASA))
CCAL
SIMBLAM (TION
CCAL
IDEAL ENT
IDBAL-ENT
ENT
LPL
LPL
RALD
BALD
KAGcaz
RADGE
BADKIE
Budget
Budget
(c) OOD samples Budget
(d) OOD samples vs. Budget
CIFARI00 04(DD:0OD ratio 6:4)
CIFARIO0 06(ID:0OD ratio 4:6)
56
Accuracy
Accuracy
SVEARHRUNN
CTAL DURUURR
Gradir (Ho03)))
Cliom E0A330)
900
2000
200
19:00
Budget
Budget
(c) Accuracy vs. Budget
Accuracy vs. Budget
OOD data selected in CIFARIO0 04
OOD data selected in CIFARI00 06
Ratio of OOD samples selected
Ratio of (OD samples les selected
84-
(PLMM)
02-
ENT
TDEAL INT
IDEAL-INT
ENT
RAND
RAND
02
LPL
LPL
BALT
BALD
2000
2008
2000
2006
2200
1564
10.00
12000
17400
11000
2003
Budget
Budget
(g) OOD samples vs. Budget
(h) OOD samples vs. Budget
Figure 8: The comparative experiments between our POAL-PS and baseline methods CIFARIO and CIFAR100 datasets. This is enlarged version of Fig. Ain the main paper.
30

---

Under review as a conference paper at ICLR 2023
PEARROOddwnsaaee (ID:OOD ratio 4:100
OOD data selected in CIFARIO-dowisamplled
POALL-IS (1U6T10.801)
Number of DOD samples selected
1190
2OAL-15 SIMILAR EDSALANT
0.78
SINILAR (JEE))
IREAL-ENT 2239911.004
6/6
2N1 Randes Maga
Bankeer (1.00 662)
Comples
0.72
Klagla (3.6 $5410,904)
Accuircy
6.70
6/90
390
391
6.0
2008
Budget
Budget
(a) Accuracy vs. Budget
(b) OOD num vs. Budget
Figure 9: The comparative experiments between model and SIMILAR down-sampled CIFARIO dataset.
vowel
0.57
Accuracy
Accuracy
670
POAT__AAA $22)
LAU
LAL:B439)
0075
TOAL_ (.0
POAAL DBN STT)
QUC1L177)
Budget
Budget
(a) EX8: Accuracy vs. Budget
(b) Vowel: Accuracy vs. Budget
EX8
vowel
PoaL LAI
Numberss of OOD samples selected
POAL_IAL
09"
Numbers stmppees
Budget
Budget
(c) EX8: OOD num vs. Budget
(d) Vowel: OOD num vs. Budget
Figure 10: Flexibility: POAL incorporates various AI. sampling strategies, including LAL and QBC.
31

---

Under review as a conference paper at ICLR 2023
Typical Partu Opiniaation
POAL
podlisc
purkive
aegitive
000 dots
dista
(a) Typical Pareto Optimization
(b) POAL
Typical
POAL
Regaaii salh Mahaaabbb Snune
oegazive
ID postiive dats negotive OOD dats sclasted
ID posiive dats zegative OOI dats
(c) Typical Pareto Optimization
(d) POAL
Figure 11: The subset selection generated by POAL and Typical Pareto Optimization. The selection results are presented on data space and multi-objective space respectively. The experimental settings are the same as Figure in the main paper.
32

---

Under review as a conference paper at ICLR 2023
sarie
Eyical
mapil
(a) Truely Pareto Optimization
(b) Typical Pareto Optimization
(c) POAL
of Wrarrrppaaaee
(d) Truely Pareto Optimization
(c) Typical Parcto Optimization
(f) POAL
Pareto Front
sum of negative mahalanobis distance
0,200
0.175
0.150
0.125
0.100
0.075
Truely pareto set
POAL candidate sot
0.050
Final selection of truely
POAL
1.50
1.75
2.00
2.25
2.50
2.75
3.00
3.25
3,50
of entropy
(g) POAL vs. Truely PO
Figure 12: The subset selection generated by POAL, Truely Pareto Optimzation and Typical Pareto Optimization on down-sampled EX8 dataset. The selection results are presented data space and multi-objective space respectively. Figure a-f are hased on single data point level, Figure-g is based on subset level, where each element of pareto set is selected subset with fixed batch size. The x-axis and y-axis are UI and M respectively.
33


Reviews:
[{"review_id": "ztlAahoLFVO", "reviewer": "Reviewer_SDAU", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studied active learning under out-of-distribution data, which could be unreliable with the existence of OOD data. This paper proposed a multi-objective loss to simultaneously control the data uncertainty and simultaneously filter the OOD data. Empirical results justified the proposed approach\n", "summary_of_the_review": "This paper studied active learning under out-of-distribution data, which could be unreliable with the existence of OOD data. The setting is quite novel for me and extensive empirical evaluations are done. However, this reviewer believes that there are several fundamental misconceptions in the problem setup. Besides, important related works are not discussed/compared, making this paper fall short of the acceptance bar. \n\n-------------------------------------\n### Post rebuttal after discussions\n\nI would like to appreciate the effortful responses by authors. I further spend some time to further re-read the manuscripts and related work. The following is my additional feedback.\n\n**About robust active learning.** In general, this paper aims to solve robustness in the context of active learning, where unlabelled data could be unreliable. However, checking the traditional active learning papers, these issues have been theoretically or practically discussed such as paper [1-4]. I should say, at least this setting is not entirely novel in active learning. \n\n**About theoretical contribution** This paper aims to propose a novel practical method in active learning. This part is not applicable. \n\n**About methodology and empirical contribution** Since this paper aims to propose a novel practical method, the requirements in this part should be significantly strong to achieve the acceptance bar. Unfortunately, after rechecking the paper, I would feel this is insufficient. In terms of practice, this paper evaluated several standard and relatively simple benchmarks such as tabular data and CIFAR100. However, recent **practical** active learning papers generally should evaluate very large scale, high-dimensional and complex datasets such as ImageNET or Open Images v6 such as [5-6]. Given the current paper is purely empirical, I do think such kinds of experiments are required. As for methodology, I would think the current version consists of different known components in different domains such as Pareto Optimization, out-of-distribution detection, etc. Why choose a specific approach in these domains? Why not other well-known methods in OOD detection? For addressing these, additional ablation studies on large-scale data such as ImageNET is necessary. \n\nOverall, since this paper aims to propose a novel practical method, however, the empirical/practical are not sufficient to achieve the acceptance bar. Without strong, sufficient and convincing empirical results in a very-large, complex and challenging dataset, it is quite hard to convince the community to adopt it practically. \n\nI hope my additional notes could further improve the paper quality.\n\n[1] Robust Active Learning Strategies for Model Variability\n\n[2] Corruption Robust Active Learning. Neurips 2021\n\n[3] Robust Interactive Learning. JMLR 2012\n\n[4] Active Learning with Logged Data. ICML 2018\n\n[5] Towards Robust and Reproducible Active Learning using Neural Networks CVPR 2022\n\n[6] Batch Active Learning at Scale. NeurIPS 2021\n\n\n[Please note that I am not saying CIFAR 10/100 is not useful. If the paper mainly proposes novel theories in understanding robust active learning, it is perfectly fine to merely evaluate these simple datasets as a proof of concept. In contrast, if this paper is empirical or methodological, strong empirical validations on challenging dataset is required in modern active learning.]  \n", "strength_and_weaknesses": "### Pros\n- This paper considered a new setting in active learning, when unlabeled data contains out-of-distribution data (or distribution shift), the uncertainty or diversity based idea could mostly capture this unreliable data, leading to a significant performance drop.\n- Empirical results and ablations support the method.\n- In general, this paper is clearly written.\n\n### Cons:\n- [Main concern] I do think this paper has a clear misconception of out-of-distribution data in the context of active learning, where a rethinking on the problem setting would be important.  \n- About related work. In fact, active learning under distribution shift has been recently discussed in a recent deep active learning framework. Surprisingly, there is even no discussion on these related works.\n\n### Comments on cons\n\n1.  I do think the out-of-distribution data in active learning is vaguely defined. The problem settings in the paper assume that unlabelled data could contain different distributions than labelled data (or OOD data). Thus we need to filter these query. However, this is not necessarily correct in active learning. \n- (1) In real-world active learning, the label annotation by **human** could provide the information of OOD data. For example,  labeled dataset = digits, and OOD data is classification cat/dog. Well, when machine query the image of cat, **human** will return cat rather than digits information. From this perspective, there is no need to conduct the pre-selection. If the data is indeed OOD (with different semantic information), human will return ground truth label, then we could easily detect outlier. \n- (2) If the OOD is defined similar semantic information (e.g, cat/dog classification in-doors and out-doors), querying these points could be beneficial or even **improve** the robustness of the prediction (see comment 2, related work section). I would think this is the problem of vague definition of OOD data, what is the formal definition in OOD data in active learning? OOD data share the same semantic information or not ? since in active learning, the data is assumed to ask **human** to query the ground truth label, there are no similar issues in the conventional out-of-distribution data generalization/detection. \n\n2. Another important aspect is the lack of related work. As far as I know,  the distribution shift and active learning has been recently empirical and theoretically studied. For example paper [1-3] studied active learning as a distribution shift problem (labelled data distribution and unlabelled data distribution). Specifically, they assume the unlabeled data shared similar semantic information (e.g, always dog/cat but different background rather than digits). Related theories/practice are developed. I do think these are important to discuss and compare.  \n\n[1] Deep active learning: Unified and principled method for query and training. Aistat 2020\n\n[2] Discrepancy-Based Active Learning for Domain Adaptation. ICLR 2022\n\n[3] Low-Budget Active Learning via Wasserstein Distance: An Integer Programming Approach. ICLR 2022\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "oUaicojIKS", "reviewer": "Reviewer_hK87", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "When the unlabeled data pool contains out-of-distribution (OOD) samples, active learning (AL) becomes challenging as OOD samples may often be confused with informative in-distribution (ID) samples.\nThis paper aims to address this issue by proposing a novel AL scheme that can outperform existing schemes under OOD scenarios.\nFor this purpose, they propose a Monte-Carlo Pareto Optimization for Active Learning sampling scheme, called (POAL).\nPOAL aims to batch-select an effective subset of samples from the unlabeled data pool, thereby efficiently improving the model accuracy.\n", "summary_of_the_review": "This paper proposes POAL, a Pareto optimization scheme for active learning under out-of-distribution data scenarios.\nWhile the proposed idea is moderately novel and leads to modest AL performance improvement in the presence of OOD data in the unlabeled data pool, it is shown to consistently outperform a number of baseline AL schemes under OOD scenarios.\n--------\nEvaluation scores have been updated after reviewing the authors' response.\n", "strength_and_weaknesses": "Considering that the goal of AL to select informative unlabeled data points for labeling and the goal of OOD data point detection may conflict with each other, POAL takes a multi-objective optimization scheme to identify and select data points that are located on the Pareto frontier such that they jointly maximize the AL score (for being informative) and the confidence score for being an ID data point (hence less likely to be OOD data points).\n\nIn this work, the authors take the maximum entropy (ENT) approach for assessing the informativeness of a data point and use the negative Mahalanobis distance to measure ID confidence.\nFor computationally efficient Pareto optimization and batch selection, the paper proposes Monte-Carlo POAL for fixed-size batch selection, where Monte-Carlo sampling iteratively generates a candidate solutions at random and compares them against the current Pareto set to efficiently update the set of Pareto optimal data points.\n\nThe performance of POAL has been evaluated for a traditional ML model as well as a deep learning (DL) model, where performance evaluation results show somewhat modest yet consistent improvement over a number of existing AL schemes.\n\n\n1.  As a performance bound, the authors show the AL performance that can be attained by oracle + ENT, where only ID data points are selected based on maximum entropy principles.\nHowever, since AL selection and OOD detection conflict with each other and can confound the performance evaluation results, it would be important to show how the AL schemes are also combined with oracle OOD detection.\nThis would provide insights into what performance improvement may be attained by improving OOD detection and to what extent the AL performance can be enhanced by selecting a specific AL scheme.\n\n\n2. While the paper proposes Mont Carlo POAL to make the Pareto optimization computationally more tractable, there are no results showing how scalable the proposed scheme is.\nThere should be a comprehensive evaluation of POAL in comparison with other AL schemes in terms of their scalability.\n\n\n3. There should be further investigation and discussion regarding the OOD detection performance and the overall AL performance in OOD scenarios.\nFor example, while SIMILAR sometimes outperforms PAL in terms of OOD detection, its overall AL performance appears to fall behind POAL.\nSometimes POAL outperforms SIMILAR in terms of OOD detection, which again leads to better AL performance.\nThere should be ablation experiments to decouple these confounding factors arising from the conflict between AL selection vs. OOD detection, which will be informative for understanding the merits of the proposed method POAL and what factors contribute to its improved performance.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "OTGyAuNLZT", "reviewer": "Reviewer_6RQ4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper explores the active learning problem under OOD data scenarios and incorporates AL and OOD objectives within a multi-objective optimization framework to balance their conflict. Specifically, they propose a Monte-Carlo Pareto optimization mechanism to enable efficient Pareto optimization, which selects optimal subsets of unlabeled samples with fixed batch size from the unlabeled data pool. The proposed framework is flexible and can apply to various combinations of AL and OOD sample selection methods, extensive experiments on ML and DL tasks demonstrate its effectiveness.", "summary_of_the_review": "Generally, I find this paper tackled an interesting problem with a multi-objective optimization framework for active learning under ODD setting. Yet, it still requires clarification and some solid empirical support before warranting acceptance of this paper.", "strength_and_weaknesses": "Strengths:\n1. This paper analyses how OOD data affects the effectiveness and efficiency of the active learning methods, and points out the conflict between AL sample selection objective and OOD detection objective.\n2. This paper casts the AL sample selection under OOD data scenarios as a multi-objective optimization problem, which can automatically balance the conflict between AL and OOD objectives without tuning hyper-parameters. Experiments show its superior to other optimization strategies. \n3. This paper proposes an efficient Monte-Carlo Pareto optimization algorithm for fixed-size batch-mode AL, which avoids searching a non-fixed-size Pareto Front and saves computation cost. Besides, this paper additionally adopts a pre-selection technique and an early-stopping strategy for large-scale datasets to improve efficiency. \n4. The proposed optimization framework is flexible, experiments of POAL incorporated with multiple AL methods on ML and DL datasets validate its effectiveness and generality. \n\nWeaknesses:\n1. The proposed Monte-Carlo Pareto optimization mechanism for select Pareto subsets is somewhat similar to the Pareto embedding mechanism in [a], the authors can make some comparisons and analysis.  \n2. How to deal with the selected OOD samples? Will they be recognized by the Oracle/Annotator and then be discarded, or be wrongly annotated?\n3. It will be better to visualize the selected samples on one dataset or a toy example to verify that they are truly Pareto subsets.\n4. How MC POAL performs on large-scale experiments?\n\nRefs:\n[a] Karlson Pfannschmidt and Eyke Hüllermeier: Learning Choice Functions via Pareto-Embeddings. KI 2020: 327-333\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject