Not All Out-of-Distribution Data Are Harmful to Open-Set Active Learning
Authors: Yang Yang
Yuxuan Zhang
XIN SONG
Yi Xu
Venue: NeurIPS 2023 poster (2023)
Keywords: Out-of-Distribution
Active Learning

Abstract: Active learning (AL) methods have been proven to be an effective way to reduce the labeling effort by intelligently selecting valuable instances for annotation. Despite their great success with in-distribution (ID) scenarios, AL methods suffer from performance degradation in many real-world applications because out-of-distribution (OOD) instances are always inevitably contained in unlabeled data, which may lead to inefficient sampling. Therefore, several attempts have been explored open-set AL by strategically selecting pure ID instances while filtering OOD instances. However, concentrating solely on selecting pseudo-ID instances may cause the training constraint of the ID classifier and OOD detector. To address this issue, we propose a simple yet effective sampling scheme, Progressive Active Learning (PAL), which employs a progressive sampling mechanism to leverage the active selection of valuable OOD instances. The proposed PAL measures unlabeled instances by synergistically evaluating instances' informativeness and representativeness, and thus it can balance the pseudo-ID and pseudo-OOD instances in each round to enhance both the capacity of the ID classifier and the OOD detector. %Meanwhile, PAL measures unlabeled instances by synergistically evaluating instances' informativeness and representativeness, which can more effectively estimate the values of instances. 
Extensive experiments on various open-set AL scenarios demonstrate the effectiveness of the proposed PAL, compared with the state-of-the-art methods. The code is available at \url{https://github.com/njustkmg/PAL}.

Not All Out-of-Distriuution Data Are Harmful to Open-Set Active Learning
Yang Yang', Yuxuan Zhang', Xin Song", Yi Xu Nanjing University of Science and Technology Baidu Talent Intelligence Center, Baidu Inc
Dalian University of Technology
(yyang, xuan faa__hang] @ @njust.edu.en, songxin06@ baiiiicoo,i..uommm yxu@dlut.edu.on
Abstract
Active learning (AL) methods have been proven to be effective way to reduce the labeling effort by intelligently selecting valuable instances for annotation. Despite their great success with in-distribution (ID) scenarios, AL methods suffer from per- formance degradation in many real-world applications because out-of-distribution (OOD) instances always inevitably contained in unlabeled data, which may lead to inefficient sampling. Therefore, several attempts have been explored open-set AL by strategically selecting pure ID instances while filtering OOD instances. How- ever, concentrating solely on selecting pseudo-ID instances may cause the training constraint of the ID classifier and OOD detector. To address this issue, we propose a simple yet effective sampling scheme, Progressive Active Learning (PAL), which employs a progressive sampling mechanism to leverage the active selection of valuable OOD instances. The proposed PAL measures unlabeled instances by synergistically evaluating instances' informativeness and representativeness. and thus it can balance the pseudo-ID and pseudo-OOD instances in each round to enhance both the capacity of the ID classifier and the OOD detector. Extensive experiments on various open-set AL scenarios demonstrate the effectiveness of the proposed PAL, compared with the state-of oofee-offhll of-the-art methods. The code available at /https://ggttee thub. wwtpsthtuu.oooonnmem commnjjstkmg//P/..
1 Introduction
Deep learning has achieved great success in many tasks such image classification [4]. article prediction (35), and graph classification (32). The main factor underlying excellent performance lies in the availability of extensive dataset with manual annotations, which can be costly in real-world applications [16). To reduce the high labeling cost, active learning an effective approach that involves iteratively selecting valuable instances from the unlabeled data pool and querying their labels for model retraining [4).
The basic assumption shared by various existing active learning methods is that all unlabeled instances are collected from the ID domain [T] 28), showing that the labeled and unlabeled instances share the same distribution. However, this assumption is unrealistic in real-worid scenarios unlabeled instances are primarily gathered from the open world, where massive instances belong to the OOD domain. Taking the image classification task as example, the goal is to classify mammals, but considerable proportion of the unlabeled instances consists of unknown classes (i.c., OOD domain), such bees, apples, and tables. Once these images selected for labeling, it would be waste of the annotation budget since they are not used for training the ID classifier. Therefore, it is important
Corresponding author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

to detect ID and OOD instances before the annotation in open-set AL, which is not applicable in traditional active learning methods. Consequently, the main challenge open-set AL revolves around effectively selecting valuable IDD instances for training classifier and distinguishing the OOD instances.
ny seleccteed CDD cata (PAL) (LOOSA) (Corwent)
Mac: 09%
0.7
Annotation
Selection
Selection Recall
0.6
Yes
0.5
Unlabeled Data
OOD Detector
OOD data?
A 0.3
ID Classifier
0.2
Traditional open-set AL
Annolation
Selection
7
Round
(a) The power of OOD instances.
(b) The proposed PAL process
Figure I: (a) the effectiveness of actively querying valuable pseudo-OO1 instances. ID Selection Recall (higher is better): the ratio of selected ID instances to the total number of ID instances. Acc: testing accuracy; w/: with; w/o: without. Experiments are conducted on CIFAR-100 using the first 20 classes as seen classes. We employ (1) the traditional active learning method Coreset; (2) the existing open-set AL approach LfOSA that only selects pseudo-ID instances; and (3) the proposed PAL that actively queries pseudo-OOD instances enhance the training of the detector and classifier. (b) the distinction between the proposed PAL process and the traditional open-set AL process (in the dashed rectangle). The proposed PAL includes an additional module (the process in red) that carefully leverages detected OOD instances during the model training progress.
To address the challenges of open-set AL. few studies have proposed 112 22 23), which commonly attempt to effectively filter the OOD instances in the unlabeled pool by adopting an extra OOD detector, thus increasing the II) purity of instances in the query set. Their goal is to optimize the selection of II) instances while eliminating OOD instances during each query round, aiming to exclusively add ID instances for model training. In other words, these methods mainly focus on learning the ID classifier, consequently restraining the resolution of the OOD detector. In contrast, we believe that not all OOD instances are harmful, and valuable OOD instances are also significant for both the classifier and detector training. For example, as shown in [ we observe that actively selecting OOD instances based their importance (i.e., PAL method) can achieve higher classification accuracy and selection recall, which is attributed to the promotion of ID purity and OOD filtering. Therefore, valuable OOD instances promptly strengthen the boundary between ID and OOD instances, and further improve the ID) purity for classifier retraining.
To this end, we propose a simple yet effective Progressive Active Learning (PAL) scheme that strategically introduces valuable pseudo-ID and pseudo-OOD instances for enhancing both the II) classifier and OOD detector. Different from existing open-set AL methods that solely focus on selecting pseudo-ID instances, our approach goes a step further by actively incorporating valuable pseudo-OOD instances, which allows us to explicitly retrain the OOD detector in conjunction with ID instances, resulting in a mutual promotion of the detector and classifier. The proposed PAL process is illustrated in Fi. Besides, to comprehensively measure the unlabeled instances. we propose to estimate the informativeness of instances by calculating entropy using the predictions from one-vs-all (OVA) classifiers, while evaluating the representativennss of instances with the automatically learned meta-weights.
2 Related Work
Active Learning. Active Learning aims to train a classifier by querying labels of the most valuable
instances in the unlabeled data, which can greatly reduce the manual labeling cost (7). The core idea
of existing active learning approaches is to ensure that selected instances can significantly improve
model performance, and various effective sampling strategies are designed (9 24 31 37). The initial
sampling criteria is uncertainty-based sampling [. which prefers to select instances that maximally
reduce the model uncertainty. For example, [9] proposed an uncertainty measure that generalizes
margin- based uncertainty; 37 employed the pairwise gradient length metric of informativeness.
2

---

Furthermore, various representativeness based methods are proposed, which prefer to select instances that match the data distribution. For example. [24] estimated the expected reduction in error that directly optimizes expected future error; [T] designed batch active learning with diverse gradient embeddings, which sampled groups of instances that are disparate and high magnitude. However, this family of approaches usually assumes in-distribution settings that the unlabeled instances are drawn from the same distribution of labeled examples, which is not suitable for effectively filtering OOD instances during the query round.
Several recent approaches have attempted to carry out the open-set AL contrastive AL framework to learn two contrastive coding models for calculating informativeness
23
introduced
and representativeness of an example: (22) introduced an auxiliary network model the per-example
max activation value distribution with a gaussian mixture model, which can dynamically select the
instances with the highest probability from known classes the unlabeled set; (23) employed the
meta-learning technique to find the best trade-off between purity and informativeness: [22 selected
set that maximize the distance on entire unlabeled data while minimizing the distance to the identified OOD instances. However, we find that these attempts are limited in filtering OOD instances, as they usually assign greater weight to maintain ID purity, thus causing the learning constraint of the ID classifier and OOD detector. Consequently, it is vital to develop effective approaches that achieve comprehensive balance between the ID classifier and the OOD detector.
-istif-distribution Defection. OOD detection aims to identify instances outside the target do- main [27] 30 39), i.e., ID domain, which is vital for securing the model's safety and interpretabil- ity [5] Existing methods always calculate instances' confidence scores based on the predictions of the classifier trained on IDD instances such as the entropy of the prediction distribution (2] 18
333. For example, (18) detected OOD instances with softmax score by considering temperature scaling and input pre-processing; 22 introduced a new model layer OpenMax, which used activation vectors to estimate the probability of deep network failure. To build more robust detector, several approaches attempted to utilize the generative adversarial network to generate pseudo-OOD instances for assisting learning. For example, [1T] proposed task-aware variation adversarial AL that considered data distribution of both label and unlabeled pools; [8] confirmed that generative model accuracy may not always be positively correlated with OOD detection performance. Nonetheless, the advantage of open-set AL lies in the existence of OOD instances within the unlabeled pool, making the fine- ane-grained usage of OOD instances possible.
ID Labled
Train
Retrain
Query
Prediction
8
g
+
D query
Share Parameter
Unabled Set
Un
000
S ID
DOD,
query
query
D
ID, query
OOD,
query
Figure 2: Framework of the proposed Progressive Active Learning (PAL). We initialized the ID classifier and OOD detector with the ID labeled data. During the query rounds, we explicitly query both pscudo-ID and pseudo-OOD instances with the designed sampling criterion to enhance the ID classifier and OOD detector.
3

---

3 Proposed Method
Problem Setting. We denote by the labeled data Dj = {(x] y i=1. where each x belongs to the C' known labels y E R and the unlabeled data D, = {x] 5=1 = (D DDOD), where D denotes the ID unlabeled data and DOOD represents the OOD unlabeled data, D DOOD = 0. In other words, an unlabeled instance xj may belong to an unknown class. In active learning, a human oracle is requested to label query set Di Quurry at (-th query round, which consists of OOD query set Douery and ID query set Diorr (i.e., Dr Query query U Duuery) Considering the restrictions on the annotation budget, the task of traditional open-set AL to construct the query set that contains more ID instances for assisting ID classifier building.
DDDDDDODD
Approach Overview. The framework of the proposed PAL is illustrated in Figure 2 including two key components: 1) importance score considering both informativeness and representativenss; 2) progressive sampling mechanism that precisely manages the ID and OOD instances within the query set. Diverging from traditional open-set AL. attempts, we adopt strategy of actively selecting the valuable pseudo-ID and pseudo-OOD instances to enhance the capacities of the classifier and detector simultaneously. In the following subsections, we first introduce the calculations of the importance score and then give the details progressive sampling. Finally, we present a theoretical understanding of using OOD instances.
3.1 Sampling Criteria
Uncertainty weight. To investigate the informativeness, we employ the one-vs-all (OVA) classi- fiers (26) to learn boundary between ID and OOD instances. In detail, OVA classifiers include C sub-classifiers, each one to distinguish whether the instance belongs to the corresponding class or not, e.g., the prediction of c-th sub-classifier can be denoted as (x')
(x(x)) E R where g represents the feature encoder, f denotes the c-th classifier. That is to say, (x (p"(p- =
1|x'), pp = ) where (p = 1|x') and ("(p = 0jx') denotes the probability of instance being an ID and an OOD respectively for class c. The overall loss of the OVA classifiers can be formulated as:
N LOVA Σ (logp " 11x)) + min log (p 0)x))), An
where Vx denotes the class of instance x Following 26).. we adopt the hard-negative sampling technique for sub-classifiers. The core idea of OVA classifiers is that each sub-classifier outputs a distance representing how far the input is from the corresponding class [26). Therefore, the sub- classifiers are effective in identifying OOD instances. To be specific, the probability of instance x be ID be calculated by gID = p'(p 1 1xx") log (p 1 1xx"), where é a given unlabeled instance, and larger s represents more likely the instance to be an ID instance.
arg max ( = ))x Actually, &D can express the informativeness
Meta-weight. In addition D we design representativeness criterion by learning the meta- weight of each unlabeled instance. Since the OOD instance in unlabeled set leads to a distribution mismatch, inspired by (5). we aim to automatically learn the weight of each unlabeled instance via tracking the effect of the supervised learning model to prevent OOD instances, which can be formulated as the following bi-level optimization problem:
N;
Esup (x,, y (ww)),
min
(2)
No lomp (x θ) Σ Wylann(3 O),
s.t. Ō(w) = arg min
Σ
1=1
where ④ = (0,, Coup (X,,
- yf log f (ge, (x)) the supervised loss, Lun (xX). θ) -c Se. (ge, (xx)) log fēr (ge, (x/)) is the unsupervised loss, and fo denotes the ID classifier. The problem 2
captures the process of learning the optimal model parameter (w)

---

through empirical risk minimization using labeled and weighted unlabeled instances, the model is then evaluated on Dr to optimize the weights, aiming to enhance the reliable performance. Thus, the learned w is considered as the measurement of representativeness, with higher values indicating that ID instances are more representative.
3.2 Progressive Sampling
Open-set AL aims to precisely select as many IID instances as possible from the unlaheled pool. With the 8 ID and w, we select the first b instances with the highest probability as the query set:
un (x" = w + uss ID
(3)
where ja is the ID selection, we empirically find that valuable OOD can enhance the detector and further boost ID selection. Therefore, we propose a simple yet effective progressive sampling. To be specific, in each round, we actively select b instances including N pseudo-ID instances and query pseudo-OOD instances, noting that we select OOD instances with the lowest un um(x)). At the first query, we set N NOOD >> NID query After that, the OVA classifiers are extended C + classifiers, and the (x(x")
However, unlike existing open-set AL attempts that only concern
query is given by
un(x") = = { + p(1 s pslD,
S)))
if +1
otherwise
3.3 Classifier Training
Based on the newly labeled data Dr query: we retrain both the ID classifier and OOD detector. In detail, the current ID labeled data can be represented as Dj Di-1 U UD where D denotes the accumulated ID labeled data, DiDery denotes the ID labeled data from the detected ID data. The ID classifier f is learned by minimizing the supervised loss [2f (xup(x). y!. 9). On the other hand, the current II) and OOD data for training OVA classifiers can be denoted as Di = Di-11 U Dr queryy
where - denotes the accumulated ID and OOD labeled data. Notably, the first round, the number of classes for detector f is C, and we train f with the initial Dj. In the subsequent rounds, the number of classes for detector f is expanded to C' + 1 by actively adding the OOD data. In each round, we retrain the f according loss Lova. The final target of open-set AL is the ID classification, and thereby we can adopt the f".
3.4 Theoretical Understanding
To theoretically understand the use of detected OOD instances in the OOD detector, we first give the following notations. Suppose that the samples (x, y) E (X,)) follows a unknown distribution P. where X' Rd is input space and
{0, 1} is output space. The label of ID instance is y = and the label of OOD instance is = 1. Let e R x y - R+ be the loss of interest. With the input and its corresponding label y. the expected loss is
L(ʃ) = Ep [e(((x)), v)}. Suppose we have a training dataset {(x1, v1),................ (xn: yn)} drawn from distribution P. then the empirical loss is Σ( = En ((f(xx)) wt). Let C(F) be some proper complexity measure of the family of hypothesis class F. We assume that the loss is Lipschitz with constant and the Rademacher
complexity (F) S V C(F) [10]. To simplify the analysis, we assume the distribution of ID and OOD samples is balanced and the number of OOD samples is the same as the number of ID samples in the training set. We suppose that the number of detected ID instances is m while the number of detected OOD instances is
n, and among the detected ID instances, the number of real ID instances my == am while the number real OOD instance mi := (1 (1-a a ) with 1. Then present the following theorem, whose proof can be found in the supplementary
where
0 < material.
Theorem For a Lipschitz loss & bounded by G, we have the following results with probability at least 1 - 8 simultaneously. For the proposed PAL method, the generalization error bound is
5

---

C(F)
log(1/6)
((PAa)) - [(SPAL) S 2
ev
(5)
2n
Vn
For the standard AL method, the generalization error bound is
C(F)
C(F)
log(2/8)
log(2/0)
C(SAL) [(SAL) ST.
am
V (1-a)m
2am
2 2(1 a)m
00
(6)
Vam
v(1 - (m
showing that the generalization error bound for PAL method better than the bound for standard AL method. That is to say, the use of detected OOD instances can improve the effectiveness of the OOD detector.
Remark. Since 0 < <
1 and < we know
Vn
Vam V(Iaam
Experiments
In this section, we aim to demonstrate the effectiveness of the proposed PAL. Due to the page limitation, more experimental results and details can be found in the supplementary material.
4.1 Experimental Setups
Datasets. We evaluate the efficiency of PAL on several image classification benchmarks, i.e., CIFAR-
10. CIFAR- 100 (13] and Tiny-lmagenet [36] datasets following standard open-set AL methods [12]
22]. The CIFAR-IO dataset consists 50,000 training images and 10,000 test images, with To classes and 5,000 images per class in the training set. The CIFAR-100 dataset has 100 classes and 50,000 training images and 5,000 test images, with 500 images per class in the training set. The Tiny-Imagenet dataset consists of 100,000 training images and 10,000 validation images, with 200 classes and 500 images per class in the training set. construct the open-set ALscenarios, following (22). set the proportion of ID classes as 20%, 30%, and 40% in experiments. For example, when the proportion set as 20% on CIFAR-10, 10, CIFAR-100, and Tiny-Imagenet, the first 2, 20, and 40 classes are considered ID classes and the last 80, and 160 classes considered as OOD classes.
Table Comparison of testing accuracy (%) on CIFAR-10, CIFAR- 100, and Imagenete datasets with ID proportion of 20%. The best results are highlighted in bold, and the second-best results are underlined.
Datasets
CIFAR 10
CIFAR- 100
Tiny- Imagemet
Rounds
3
5
1
3
9
1
3
9
Label Only
78.6
42.5
21.4
Random
88.1
93.1
95.9
97.4
44.6
49.2
52.2
54.5
56.7
22.9
26.8
32.7
37.8
39,2
Uncertainty
88.2
93.0
96.1
97.2
97.5
44.4
49.4
49.7
54.8
55.3
21.6
28.5
35.6
37.6
41.1
Certainty
88.1
90.8
91.6
92.4
93.0
45.0
50.7
53.1
54.4
54.9
21.7
27.9
35.4
39.8
44.7
Coreset
86.5
94.7
95.8
96.7
96.8
43.6
49.5
51.9
55.0
57.0
23.1
26.3
33.0
39.9
43.3
BALD
84.6
93.1
94.3
96.4
96.7
42.9
48.1
52.3
54.4
56.2
22.9
26.8
32.9
38.1
39.2
OpenMax
81.3
85.8
86.6
87.0
90.7
45.0
47.3
50.1
56.0
22.0
26.1
36.9
41.9
CCAL
88.9
94.3
96.2
97.4
97.7
45.1
50.9
53.4
57.2
60.4
23.2
28.5
35.6
40.6
44.9
MQnet
88.8
94.9
96.8
97.4
97.8
45.3
51.1
57.9
59.1
61.3
23.8
28.6
35.7
42.0
45.8
LFOSA
84.2
95.4
97.1
97.5
98.3
45.6
52.2
59.0
62.5
66.1
22.6
28.8
43.7
47.9
PAL
91.1
95.6
97.6
98.5
98.7
45.6
53.0
60.0
65.6
69.4
24.3
33.4
43.9
47.6
52.1
Baselines and Performance criteria. We compare PAL with three different types of methods: 1)
supervised method, i.e., Label Only. 2) traditional AL methods, i.e., Random, Uncertainty [17]
Certainty [20). Coreset (28) Openmax (2), and BALD (29 3) open-set AL methods, i.e., CCAL [3]
LfOSA [2] and MQnet [23]. We consider performance from two aspects: 1) ID classification, i.e.,
classification accuracy. 2) OOD detection during query round, i.c., precision and recall 22

---

Parameter setting, For all AL methods, following [2 we randomly sample 1 1%, 10% and 10% of the examples as the initial labeled set on CIFAR- 10, CIFAR- 100, and Imay-Imagenet datasets, respectively. To ensure fairness, we employ WideResNet (38) as the backbone for training all methods. Note that the labeled data only contains ID classes. In each AL. round, we train the model for 100 epochs, using the SGD optimizer with the momentum parameter of 0.9. The learning rate is initialized as 0.01 with a mini-batch size of 128, and the weight decay is set to be x 10-4. Additionally, the annotation budget per query round is limited to 1500 following [22] All experiments are implemented on a single NVIDIA V100 GPU.
4.2 Performance Comparison
Table I and Figure 3 record the ID classification and OOD detection results of PAL and other AL methods with the increasing number of queries, where the index (,, 7,9) denotes the query round.
ID classification. The results in Table show that: 1) AL methods perform better than the supervised method because the oracle will filter the OOD instances in the query set. However, the promotion is limited due to decreased ID instances, especially in complex datasets such as CIFAR- 100 and Tiny-Imagenet. 2) CCAL, LfOSA, and MQnet outperform traditional active learning methods because LfOSA selects more purified set for querying, including more ID instances and fewer OOD instances. LfOSA performs better than MQnet because MQnet maintains a balance between purity and informativeness of the query set that contains several OOD. 3) PAL consistently exhibits the best performance in each round on all datasets, especially on the Tiny-Imagene dataset. Compared to other open-set AL methods, PAL improvements about 7.5%, 3.9%. and 4. 4.2% under the 5, 7, and rounds respectively, indicating that PAL could better control the purity ID by constructing a more robust OOD detector.
lasde Orttinn
Selection Predtion
0.8
0.5
Selection Pr Precision
0.4
BAL
COLL
OCAL
0.3
0.4
FAL
NL
0.2
0.1-
0.1
Round
Round
Round
(a) CIFAR-10 10 Precision
(b) CIFAR- 110 Precision
(c) Ima-enmgeeet Precision
Selection Recall
Randerr
Cantret
Comtree
0.6
0.7-
fundor
Connec
Salection Remil
0.60
0.35
Unpistaaryy
Cartain
0.6
Moner
0.5-
POWN
CCAA
BALD
0.4
0.20
0.2
0.2
0.15
7
Round
Round
Round
(d) CIFAR- 10 Recall
(e) CIFAR-I00 100 Recall
(f) Tiny-Imagene Recall
Figure 3: The precision and recall comparisons of OOD detection on CIFAR-10, CIFAR-100, and Tiny-Imagenet datasets with ID proportion of 20%.
OOD detection. The results of Figure reveal that PAL consistently outperforms other baselines in terms of most selection precision and recall. For example. the proposed PAL achieves I 15.9% higher precision than LIOSA on Timy-Imagenet dataset in the last round, which validates that PAL can better eliminate the OOD instances by adopting the discriminative criteria and progressive sampling. Moreover, note that the precision of open-set set AL methods shows an initial increase followed by a subsequent decrease, for the reason that we are taking unreplaced query, so the remaining unlabeled data gradually reduced. Meanwhile, PAL can select more ID instances at the starting several rounds on C0 resulting in fewer ID instances left in the unlabeled set considering the relatively smaller dataset size of CIFAR-10. Thereby, PAI. achieves inferior precision compared to LIOSA in later rounds.
7

---

To further visualize the validity of pseudo-ID ID and pseudo-OOD instances, we compare the query purity of PAL with LfOSA in query rounds on three datasets, as illustrated in Figure | Although we actively select more OOD instances in the first round, the overall OOD instances are much fewer than LfOSA. Specifically, under the CIFAR-10, 10, LfOSA selected 52.2% OOD instances, whereas PAL only selected 40.3%. Besides, we find that the purities of pseudo-II and pseudo- -o0oo---Ooea are better than LIOSA, i.e., PAL achieves an overall 12.6% higher query purity compared to LIOSA on the
Tiny-Imagenet dataset, which validates the effectiveness of progressive building OOD detector.
22
Lissa
ID ID
000 OGD
123 Liesa
E ID ID
0OD G0D
22 Liosa
ID ID
000 DOD
PALL
12,000
000 ID
129 PA
D 0,000
0000 ID
XX FAL
1b cop
oDa ID
1400
1400
1400
1200
1200
1200
Numbar
1000
1000
1000
500
400
400
400
200
200
200
Round
Round
Round
(a) CIFAR-10
(b) CIFAR-100
(c) Tiny-Imagene
Figure 4: The query purity comparison between LfOSA and PAL. Query purity denotes the ratio of ID_ID instances to the total number of the annotation budget. ID ID denotes the true positive in the pseudo-ID instances, while ID ID_OOD represents the false positive in the pseudo-ID instances. Similarly, OOD_ _D denotes the false negative in the pseudo-OOD instances, while OOD _OOD represents the true negative in the pseudo-OOD instances. Note that LfOSA only queries the pseudo- ID instances.
4.3 Influence of Different ID Class Proportions
In this subsection, we evaluate the influence of different numbers of ID classes. The average results of accuracy, precision, and recall are presented in Figure 5 6 and respectively. In each figure, the first, second, and third columns represent the results on CIFAR-10,, CIFAR- and respectively, while the first and second rows represent the results with ID proportion 30% and 40%. Please note that the results of 20% are included in Table [ and Figure The results exhibit that the proposed PAL always outperforms other methods in terms of ID classification and OOD detection for all cases. For example, when ID proportion is set to 30%, PAL outperforms 3.6%, 3.3%, and 4.3% than the state-of-the-art method, i.e., LfOSA, in terms of accuracy, precision, and recall in the last round on CIFAR-100
0.98
0.96
0.92
EALD
Accumecy
Accursyy
CALD
0.
CL/A
0.55
0.35
COAL
0.84
0.86
0.50
LE744
0.30
0.82
0.25
0.80
9
5
Round
Round
Round
0.55
0.90
Unceratrr
0.40
Conte
0.63
Cemtut
Accuracy
0.85
0.55
Accuracy
0.35
CALD
0.80
0.20
0.45
0.25
0.70
0.49
3
Round
Round
Round
(a) CIFAR-IO Accuracy
(b) CIFAR-100 Accuracy
(c) Tiny-Imagenet Accuracy
Figure 5: Comparison of testing accuracy %)) on CIFAR-10 (first column), C0F (second
column), and Tiny-Imagenet (third column) datasets, with an ID proportion of 30% (first row) and 40% (second row).
8

---

0.7-
0.8
0.6
Selection Predfsion
Selection Precisioon
0.6
0.4-
COAL
ETOAA
0.4
0.3-
0.3
02
0.2
0.1
Round
Round
Round
0.8
Uncomantt
0.7
Selecion Preditoo
Cortins
C.7-
0.6
ROLL
CCA.
COL
0.5
COAL
0.5-
Seleciion
LICAS
0.4
0.4
0.4
0.3
02
0.2
Round
Round
Round
(a) CIFAR-IO Precision
(b) CIFAR-100 Precision
(c) Tiny-Imagenet Precision
Figure 6: The precision comparison of OOD detection on CIFAR-10 (first column), CIFAR- 100 (second column), and
(third column) datasets, with an ID proportion of 30% (first
row) and 40% (second row).
Salecton Recalt
0.6
BAID
0.5-
0.6-
Unsortante
Cartan
BALD
selection Recall
0.60
0.35
kanden
0.50
0.3
USSA
(455)
0.25
Mixi
0.2
0.2
0.1
Round
Round
Round
Tandon Cetoh
0.50
kander
Certers
Uimamanny
Selection Recall
SALD
Solection Recall
BALD
Selecton Recall
0.25
Gramst
0.20
Mexe
0.10
Round
Round
Round
(a) CIFAR-10 Recall
(b) CIFAR-100 Recall
(c) Tiny-Imagenet Recall
Figure 7: The recall comparison of OOD detection on CIFAR-10 (first column), CIFAR- 100 (second column) and Tiny-Imagenet (third column) datasets with ID proportion 30% (first row) and 40% (second row).
4.4 Ablation Study
Each Component of PAL. To analyze the contribution of each component of PAL, we conduct ablation study with ID proportion as 20%. The experimental results of classification accuracy are plotted in Figure g where w/o w denotes the PAL with only the informativeness criterion, and w/o 5 denotes the PAL with only the representativeness criterion. The results indicate that the representativeness criterion has a greater impact on open-set AL in realistic scenarios, for the reason that the representativeness criterion can better distinguish the challenge ID and OOD instances considering the instances' relationships.
Hyper-Parameter 1s. Considering that gID and w can capture different semanties (i.e., infor- mativeness and representativennsss) of unlabeled instances, set a with different values, i.c.

---

{0.2, 0.4, 0.6, 1}. to empirically investigate the impact of s SD and w. Table 2 shows the final round accuracy, precision, and recall on CIFAR-100 with ID proportion of 20%. PAL achieves the best ID classification accuracy and OOD detection recall with 0.8, which indicates that PAI. can leverage various semantic information of unlabeled instances and effectively improves ID selection by measuring instances' informativeness. Besides, representativeness can better ensure the true positivity of ID instances, allowing PAL to achieve superior precision at μ = 0.2.
0.00
0.0
0.43
004
0.00
0.40
0.55
640
0.30
5.00
0.29
Round
Round
Round
CIFAR-10
(h) CIFAR-100
(c) Tiny-Imagenet
Figure 8: Ablation study conducted CIFAR-10, CIFAR-100, and Tiny-Imagenet datasets with an ID proportion of 20%.
Table 2: Comparison of accuracy, precision, and recall across different values of μ in CIFAR-100, with ID proportion of 20%.
0.2
0.4
0.6
0.8
Accuracy (%)
66.2
67.2
68.5
69.4
67.6
Precision (%)
32.4
27.3
25.6
25.4
17.6
Recall (%)
67.0
69.2
71.0
73.8
69.7
Sampling Criteria. To evaluate the effectiveness of the proposed sampling criteria, we compare several classic metries for evaluating representativeness and informativeness. Table the results on CIFAR-10 10 and CIFAR-100 100 datasets, the PAL Core denotes replacing meta-weight with Coreset 28 which traditional metric method used for representativeness. PAL MSP and PAI._energy methods denote replacing the proposed informativeness criteria with MSP (6] and energy-based [9 criteria respectively. The results exhibit that PAL achieves superior performance on two datasets, revealing the effectiveness of meta- meta-weight in selecting instances.
Table 3: Comparison of testing accuracy (%) for PAr__Cree, PAL_MSP, PAI_energy, and PAL on CIFAR-10 and CIFAR-100 with an ID proportion of 20%.
Datasets
CIFAR-10
CIFAR-100
Rounds
I
3
5
7
9
3
5
7
PAL_Core
87.1
93.3
96.7
97.4
98.3
44.8
49.7
59.6
63.7
65.9
86.0
95.1
95.2
97.5
98.0
45.1
48.4
55.8
60.1
64.7
PAI_energy
85.6
93.7
94.7
97.0
97.9
45.32
52.5
58.6
64.5
66.2
PAL
91.1
95.6
97.6
98.5
98.7
45.6
53.0
60.0
65.6
69.4
5 Conclusion
In this paper, we have proposed PAL, a novel open-set AL method that employs simple but effective progressive sampling to enhance the performance of OOD detection by actively querying OOD instances at the initial round. PAL included informativeness (i.e., predictions of OVA detector) and representativeness (i.e., automatically learned metg-weights) to measure the importance of unlabeled instances. A theoretical result from the perspective of learning theory revealed that the detected OOD instances enhance the OOD detector. The effectiveness of PAI. was demonstrated with extensive empirical results across different scenarios, compared with the state-of-thh-arr methods. The limitation of the proposed PAL is only applicable to the classification task. In the future, we expect to extend PAL to other computer vision tasks such object detection.
Broader Impact. PAL improves the efficacy of active learning in real- world scenarios. Similar to other active learning methods, it effectively reduces the annotation cost for model training, while the privacy protection issue associated with the practical use of AL on realistic datasets deserves more attention.
10

---

Acknowledgments
The authors thank the anonymous reviewers for their helpful comments. This work is partially supported by National Key RD Program of China (2022YFF0112000) NSFC (62006118, 62276131), Natural Science Foundation of Jiangsu Province of China under Grant (BK20200460), Jiangsu Shuangchuang (Mass Innovation and Entrepreneurship) Talent Program, Young Elite Scientists Sponsorship Program by CAST, the Fundamental Research Funds for the Central Universities (NO.N12022028, No.30922010317, DUT No. 82232031).
References
[1] Jordan T. Ash et al. "Deep Batch Active Learning by Diverse, Uncertain Gradient Lower Bounds". In: ICLR. Addis Ababa, Ethiopia, 2020.
[2] Abhijit Bendale and Terrance E. Boult. "Towards Open Set Deep Networks". In: CVPR. Las Vegas, NV, 2016, pp. 1563-1572.
[3] Pan Du et al. "Contrastive Coding for Active Learning under Class Distribution Mismatch". In: ICCV. QC, Canada, 2021, pp. 8907-8916.
[4] Yarin Gal, Riashat Islam, and Zoubin Ghahramani. "Deep Bayesian Active Learning with Image Data". In: ICML. Sydney, Australia, 2017, pp. 1183-1192.
[5] Lan-Zhe Guo et al. "Safe deep semi-supervised learning for unseen-class unlabeled data". In: ICML Virtual Event, 2020, pp. 3897-3906.
[6] Dan Hendrycks and Kevin Gimpel. "A Baseline for Detecting Misclassified and Out-of- Distribution Examplesin Neural Networks". In: ICLR. Toulon, France, 2017.
[7] Sheng-Jun Huang. Rong Jin, and Zhi-Hua Zhou. "Active Learning by Querying Informative and Representative Examples". In: NeurIPS. Vancouver, Canada, 2010, pp. 892-900.
[8] Dihong Jiang, Sun Sun, and Yaoliang Yu. "Revisiting flow generative models for Out-of- distribution detection". In: ICLR. Virtual Event, 2022.
[9] Ajay J. Joshi, Fatih Porikli, and Nikolaos Papanikolopoulos. "Multi-class active learning for image classification". In: CVPR. Miami, Florida, 2009, pp. 2372-2379.
[10] Sham M. Kakade, Karthik Sridharan, and Ambuj Tewari. "On the Complexity Linear Prediction: Risk Bounds, Margin Bounds, and Regularization", In: NeurIPS. British Columbia, Canada, 2008, pp. 793-800.
[11] Kwanyoung Kim et al. ''Tas--warre Variational Adversarial Active Learning". In: CVPR.
Virtual Event, 2021, pp. 8166-8175.
[12] Suraj Kothawade al. "SIMILAR: Submodular Information Measures Based Active Learning In Realistic Scenarios". In: NeurIPS. Virtual Event, 2021, pp. 18685-18697.
[13] Alex Krizhevsky. "Learning Multiple Layers of Features from Tiny Images". In: Master's toronto...... wwwu""kri/ciiii cifar..i.r.....a///// .html
thesis (2009). URL: http::tttt///ttttt
[14] Alex Krizhevsky, Ilya Sutskever, and Geoffrey F. Hinton. "ImageNet Classification with Deep Convolutional Neural Networks". In: NeurIPS. Lake Tahoe, Nevada, 2012, pp. 1106-1114. [15] David Krueger et al. Distrii-n Generalization via Risk Extrapolation (REx)". In: ICML. Virtual Event, pp. 5815-5826.
[16] Yann LeCun, Yoshua Bengio, and Geoffrey E. Hinton. "Deep learning". In: Nature 521.7553
(2015), pp. 436-444.
[17] David D. Lewis and William A. Gale. "A Sequential Algorithm for Training Text Classifiers". In: ACM-SIGIR, Dublin, Ireland, 1994, pp. 3-12.
[18] Shiyu Liang. Yixuan Li, and R. Srikant. "Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks", In: ICLR. Vancouver, Canada, 2018.
[19] Weitang Liu et al. "Fnergy-based Out-of -ittrf-distribution Detection". In: NeurIPS. Virtual Event,
2020, pp. 21464-21475.
[20] Wenjie Luo, Alexander G. Schwing, and Raquel Urtasun. "Latent Structured Active Learning".
In: NeurIPS. Lake Tahoe, Nevada, 2013, pp. 728-736.
[21] Laurens van der Maaten and Geoffrey Hinton. "Visualizing Data using t-SNE". In: Journal of Machine Learning Research 9.11 (2008), pp. 2579-2605.
[22] Kun-Peng Ning al. "Active Learning for Open-set Annotation". In: CVPR, New Orleans,
LA, 2022, pp. 41-49.

---

[23] Dongmin Park et al. "Meta- Sitta-Quer--Neet Resolving CurityInformativenne Dilemma in Open-set Active Learning". In: NeurIPS. New Orleans, LA, 2022, pp. 31416-31429.
[24] Nicholas Roy and Andrew McCallum. "Toward Optimal Active Learning through Sampling Estimation of Error Reduction". In: /CML. Williamstown, MA, 2001, pp- 441-448.
[25] Kuniaki Saito, Donghyun Kim, and Kate Saenko. "OpenMatch: Open-set Consistency Reg- ularization for Semi-superviae Learning with Outliers". In: NeurIPS. Virtual Event, 2021,
pp. 25956-25967.
[26] Kuniaki Saito and Kate Saenko. "OVANet: One-vs-All Network for Universal Domain Adap- tation". In: ICCV. Montreal, Canada, 2021, pp. 8980-8989.
[27] M Salehi et al. "A Unified Survey Anomaly, Novelty, Open-Set, and Out of-Distribution Detection: Solutions and Future Challenges". In: Transactions on Machine Learning Research. 234 (2022).
[28] Ozan Sener and Silvio Savarese. "Active Learning for Convolutional Neural Networks: Core-Set Approach". In: ICLR. Vancouver, Canada, 2018.
[29]
Toan Tran et al. "Bayesian Generative Active Deep Learning... In: ICML. Long Beach, Califor-
nia, 2019, pp.
[30]
Wenjuan Xi et al. "Robust Semi-Supervised Learning for Self-learning Open- Open-World Classes". In: ICDM. Shanghai China, 2023.
[31] Yang Yang. De-Chual Zhan, and Yuan Jiang. "Learning by Actively Querying Strong Modal Features". In: IJCAI. New York, NY, 2016, pp. 2280-2286.
[32] Yang Yang et al. "Corporate relative valuation using heterogeneous multi-modal graph neural network". In: JEEE Transactions on Knowledge and Data Engineering 35.1 (2021), pp. 211- 224.
[33] Yang Yang al. "Learning Adaptive Embedding Considering Incremental Class", In: IEEE Transactions on Knowledge and Data Engineering 35.3 (2023), pp. 2736-2749.
[34] Yang Yang et al. "S2OSC: A holistic semi- sumi-super approach for open set classification", In: ACM Transactions on Knowledge Discovery from Data 16.2 (2021), 1), pp. 1-27.
[35] Yang Yang et al. "Semi-supervised multi-modal multi-instance multi-label deep network with optimal transport". In: IEEE Transactions on Knowledge and Data Engineering 33.2 (2019),
pp. 696-709.
[36] Leon Yao and John Miller. "Tiny imagenet classification with convolutional neural networks". In: CS 231N 2.5 (2015), p. 8.
[37] Xinge You. Ruxin Wang, and Dacheng Tao. "Diverse expected gradient active learning for relative attributes". In: IEEE transactions image processing 23.7 (2014), pp. 3203-3217,
[38] Sergey Zagoruyko and Nikos Komodakis. "Wide Residual Networks". In: BMVC. York, UK, 2016.
[39] Da- Wei Zhou, Yang Yang, and De-Chuar Zhan. "Learning classify with incremental new class". In: IEEE Transactions on Neural Networks and Learning Systems 33.6 (2021), pp. 2429- 2443.
12

---

The Supplementary of Not All Out-of-Distribution Data Are Harmful to Open-Set Active Learning
A Proof of Theorem 1
Firstly we prove the generalization error of the proposed PAL method. Following Theorem of [0 have
log(1/6)
((JPAL) [(SPAL) S 2T22 (F) ev
2n
where Ri (F) is the Rademacher complexity of a function class F and is the number of training samples. With the Rademacher complexity R., (FF S / /
x we give equation
Next, prove the generalization eπror of the standard AL method. To this end, denote by
P = P(xly = 0) the conditional probability for ID data and Pi = P(x = 1) the conditional probability for OOD data. Let Lf()) denote the loss from class j {0, 1}: Lf(s) Ep, (x([(x)))) v)).
and let (J) denote its corresponding empirical loss. Then by applying the standard analysis for each class j in Theorem 1 of [10], with probability - 6/2 have
(,(Aa) - ((A S 2LRM, (F) +cv log(2/0) 2mj
(j = 0,
(8)
Since the distribution of ID and OOD samples is balanced, we have ((SA))= Lo (SAL) ( due to the definitions of the loss functions. Similarly, due the number OO1) samples being the same as the number of ID samples in the training set, we know L(fAL) = 1/ (fALL) (fAL). Finally, by applying the union bound, have
log(2/8)
log(2/0)
((SAL) [(fAL. & 1.9m. (F) 1.2m. (F)
(9)
2
2mo
2
2m
With the Rademacher complexity Rmy (F)
C(3) my
for j 0, 1, complete the proof of equation
6 by plugging in = am and m² (1 m.
B Additional Implementation Details
Implementations. In the first round, enhance the effectiveness OOD detection, we set NID 300 and NOOD = 1200. To reduce the influence of the pseudo-OOD budget for the ID classifier, we query
query
set NID 1450 and NOOD 50 in the later rounds.
Table 4: Dataset Construction.
Number ID
OOD Ratio
Number of labeled data
Number unlabeled data
Dataset
Query size
classes
CIFAR- 10 (20%)
2
2:8 8
100
49900
1500
CIFAR-10 (30%)
3
3: 7
150
498550
1500
CIFAR-10 (10%)
4
4: 6
200
49800
1500
CIFAR-100 (20%)
20
20: 80
1000
49000
1,500
CIFAR- 100 (30%)
30:
1500
48500
1500
CIFAR- 100 (40%)
40
40 60
2000
48000
1500
Liny-lmagener (20%)
40
40 : 160
2000
48000
1500
Tiny-Imagenet (30%)
60
60 140
3000
47000
1500
Imagennt (40%)
80
80 120
4000
46000
1500
Description for Datasets. In Section 4, we provide a brief introduction to the construction of the open-set AL scenario. order provide more detailed explanation of the dataset construction, we use CIFAR- 100 with the ID proportion of 20% specific example. We consider the first 20 classes
13

---

as the ID classes. Then, we randomly sample 50 instances from each ID class to form the labeled set Dr, while the remaining instances are unlabeled data D.. As a result, the labeled and unlabeled sets comprise 1,000 and 49,000 instances respectively, as shown in Table 4
Description for baselines. Here's a brief introduction to the comparison methods in the experiment.
Random introduced randomly selecting instances from the unlabeled pool for labeling.
Uncertainty [17 employed entropy evaluate the uncertainty of instances.
Certainty 20 considered utilizing the weakly labeled data to query instances based on entropy of local marginals.
Coreset
28 proposed core-set selection with greedy k-center clustering to minimize the maximum distance between labeled data and unlabeled data.
BALLD 29 employed dropout as an approximation for Bayesian inference to facilitate active
sampling.
Openmax 2 introduced the activation vector to measure whether instance belongs to the
known classss.
CCAL introduced a framework of contrastive active learning, with two contrastive coding models employed to assess the informativeness and representativeness of each example.
MQnet 23 utilized meta-lcarning techniques to discover an optimal balance between purity and informativeness.
LFOSA [2 leveraged an auxiliary network equipped with gaussian mixture model to capture the distribution maximum activation values for each instance, enabling adaptive selection of high-poobability instances from known classes.
C Additional Experimental Results
Impact of the annotated OOD _OD_D<e data. Actually, the method of only adding additional data into OOD Detector, leading the annotated OOD_ID ID data (the annotated ID data after selection of the detected OOD data) not used in ID classifier of PAL. In this section, we aim to empirically investigate the impact of annotated OOD ID data. To this end, we modify the proposed PAL by adding the annotated OOD ID data into ID classifier and naming it as PAL+, which is illustrated in Figure
Annotation
Selection
Yes
Unlabeled Data
OOD Detector
OOD data?
Classifier
No
Traditional
open-set AL
Annotation
Selection
Figure 9: PAL+: PAL with the annotated OOD_ID data added to ID classifier (red dash line).
Considering experimental completeness, present the classification and OOD detection performance without and with using the OOD ID instances (i.c. PAL and PAL+), respectively. Note that the experiments are performed three datasets with ID proportion of 20%. The results in Figure 10 demonstrate that the proposed PAL and PAL+ are comparable, showing that the annotated OOD ID instances have minimal impact the overall performance of the model.
In addition, Table 5 lists the number of annotated OOD ID instances with their proportion in the detected OOD data, for round I, 3. 5, 7, and 9 on three data sets. The results in Table that the annotated OOD JD instances have a very small proportion the detected OOD instances, and this may also explain why the PAL and PAL+ comparable.
14

---

Table 5: The number (with its proportion) of annotated OOD ID instances used in PAL+.
Round
Seed
Dataset
3
5
7
CIFAR-10
5 (10%)
1 (2%)
0 (0%)
0 (0%)
I (2%)
seed=1
CIFAR-I00
5 (10%)
5 (10%)
(2%)
0 (0%)
0 (0%)
Tiny-Imagenet
3 (6%)
3 (6%)
2 (4%)
1 (2%)
0 (0%)
CIFAR-10 10
3 (6%)
0 (0%)
0 (0%)
0 (0%)
0 (0%)
seed=2
CIFAR- 100
(8%)
2 (4%)
(2%)
(2%)
0 (0%)
Tiny-Imagenet
6 (12%)
3 (6%)
(0%)
O (0%)
I (2%)
CIFAR-10
(2%)
0 (0%)
0 (0%)
O (0%)
0 (0%)
seed=3
CIFAR-100
(2%)
2 (4%)
(2%)
(2%)
0 (0%)
Imny-Imagenet
(4%)
3 (6%)
(4%)
0 (0%)
2 (4%)
PAL
0,90
PAL
ID Selection Recall
0.80
0.70
Accuracy
0.96
0.60
0.94
0.50
0.40
0.92
0.30
0.20
0.90
3
5
9
3
7
Round
Round
0.70
PAL
0.70
PAL.
0.65
0.60
Accuracy
0.60
Selection
0.50
0.55
0.40
0.50
0.30
0.20
3
Round
Round
0.55
0.45
0.50
PAL+
0.40
PAI
Accuracy
0.45
0.35
Selection
0.10
0.30
0.35
0.25
0.30
0.20
0.25
0.15
3
Round Figure 10: Comparison of classification and OOD detection performance on CIFAR-10 10 (first row), CIFAR-100 (second row) and Tiny-Imagent (third row) with an ID proportion of 20%.
Round
Visualization. To demonstrate the distribution of selected ID instances, we provide visual results of the iterative query round. In detail, we visualize the feature representations of the labeled and queried correct ID instances via t-SNE (21) on the CIFAR-10 10 dataset with an ID proportion of 40%. Note that green crosses represent the queried ID instances. Figure records the visualizations of PAL and state-of-the-art methods, i.e., Coreset and LfOSA. The figures reveal that PAL selects more ID instances than other methods, i.e., higher purity in different rounds. Moreover, from the perspective of data distribution, PAL would balance the informativeness and representativeness criteria to select meaningful ID instances for ID classifier training.
15

---

Airplane Nutomobile
Bird
Cat
Coreset
LIOSA
PAL
Figure 11: The t-SNE visualizations of Coreset, LIOSA, and PAL, considering the 1st (top row), 5th (middle row), and 9th (bottom row) rounds.
Compare with Open-Set Semi-Supervised Learning. Although both open-set/safe semi-supervised learning and open-set AL aim construct more effective ID classifier by reasonably utilizing unlabeled data, the essential difference lies in whether or not manual participation is necessary. Open-set/Safe semi-supervised learning aims to use all the pseudo-ID instances in unlabeled data according to the model's 's prediction, whereas open-set AL aims to select the most valuable pseudo-ID instances unlabeled data for manual labeling retrain the model without using all the unlabeled data. To evaluate the classification performance of PAI. and open-set semi-supervised learning methods, we have compared PAI. with two open-set semi-supervised learning methods, D (5) and OpenMatch [25] using the CIFAR-10 10 and CIFAR- 100 datasets with an ID proportion of 20% As illustrated in Table 6 the results show the proposed PAL also performs better than the open-set semi-supervised learning methods.
Table 6: Comparison testing accuracy (%) for L OpenMatch, and PAL on CIFAR-10 and CIFAR- 100 with an ID proportion of 20%.
Method
CIFAR-10
CIFAR-100
D
74.2
48.1
OpenMatch
82.1
66.5
PAL
98.7
69.4
Runtime Cost. To verify the runtime cost of the proposed PAL, evaluate it against several
baselines, including LS OpenMatch, Coreset, and LfOSA on CIFAR-I0 10 and CIFAR-100 with the ID proportion of 20%. As demonstrated in Table the results show that PAL has the second best runtime cost, outperforming DS OpenMatch, and LfOSA, but falling behind Coreset.
16

---

Table 7: Comparison of runtime cost for PAL and other baselines CIFAR-10 and CIFAR-100 with an ID proportion of 20%.
Method
CIFAR-10
CIFAR-100
DS
26.8h
43.4h
OpenMatch
37.7h
68.3h
Coreset
2.8h
3.6h
LfOSA
66.2h
77.8h
PAL
3.4h
4.4h
17


Reviews:
[{"review_id": "HZ1DSicmCR", "reviewer": "Reviewer_d3fV", "summary": "In this paper, the authors aims to improve open-set active learning, where unlabelled set may contains some open-set instances. To do this, they propose a new sampling scheme, called progressive active learning. Specifically, they use a progressive sampling method to select valuable OOD data for the tradeoff between pseudo-ID and pseudo-OOD instances in each round. Experiments on various datasets are conducted to show the effectiveness of the proposed method.", "questions": "Do you mean the positive effect of OOD data is only in detecting OOD examples? You write \"improving the ID purity in query sets\" in line 66.", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper is well-writing and easy-to-understand. I believe readers can easily get the core idea of the proposed method.\n\n2. The empirical performance is non-trivial. From Table 1, we can observe the improvement of the proposed method is significant.\n\n3. The motivation is clear. I agree that OOD data should not be simply filtered out for active learning.", "weaknesses": "1. The method is somewhat heuristic and not novel enough. In my view, both OVA classifier and meta-weight are not original contribution of this paper. The technical novelty of the progressive sampling is not sufficient to support the acceptance in this conference.\n\n2. The literature of leveraging OOD data to improve generalisation is missing, like OAT [1], ODNL [2] and Open-sampling [3]. The authors may need to discuss the relationship between the proposed method and these works.\n\n\n\n[1] Lee, Saehyung, et al. \"Removing Undesirable Feature Contributions Using Out-of-Distribution Data.\", ICLR, 2021.\n\n[2] Wei, et al. Open-set Label Noise Can Improve Robustness Against Inherent Label Noise. NeurIPS 2021\n\n[3] Wei, et al. Open-Sampling: Exploring Out-of-Distribution data for Re-balancing Long-tailed datasets. ICML 2022.\n\n---------\n\nAfter reading the responses from the authors, my concerns have been well addressed, so I lean towards acceptance for this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Q1: \" ...The technical novelty of the progressive sampling is not sufficient...\"\n\nA1: Open-set active learning aims to strategically select pure ID data and filter out OOD data, necessitating the presence of a robust OOD detector. The common practice of traditional open-set active learning methods, as depicted in Figure 1 (b) in the manuscript, usually relies on detected ID data to build the OOD detector. Subsequently, these methods primarily focus on selecting pseudo-ID instances, which would lead to the weakening of the OOD detector due to the lack of active selection of effective OOD data and affects the ID purity during the selection process. To address this challenge, we propose a simple yet effective sampling scheme that progressively selects pseudo-ID and pseudo-OOD instances in each round, especially adding valuable OOD data in the initial round. Thereby, it could enhance the capacity of the OOD detector and simultaneously promote the ID classifier by increasing the ID purity. We will highlight the contributions in the revision. \n\nOn the theoretical side, we provide a theoretical analysis of generalization error, which is commonly used in the literature of learning theory. Our theoretical analysis reveals that the proposed PAL has a better generalization error bound than the standard AL without using detected OOD data, showing the effectiveness of the proposed PAL. Please refer to **Global Response** for more details about the theoretical analysis. \n\n\nQ2: \"... literature of leveraging OOD data to improve generalization is missing...\"\n\nA2: Thanks for your suggestion. Unlike OAT, ODNL, and Open-sampling, which directly utilize OOD data to improve the ID classifier, our approach focuses on enhancing the OOD detector using detected OOD data, thereby controlling the ID purity and improving the ID classifier. We will include more discussion in the revision.\n\n\nQ3: \"Do you mean the positive effect of OOD data is only in detecting OOD examples...\"\n\nA3: Sorry for the confusion. The positive effect of selected OOD data can not only promote the OOD detector building for better detecting OOD data in subsequent phases but also improve the ID purity. We will make it clear and give more details in the revision."}, {"review_id": "fjYssgkPni", "reviewer": "Reviewer_oop5", "summary": "This paper considers the open-set active learning problem, a sub-topic of active learning that focuses on non-iid settings. The authors constructed both an ID classifier and an OOD detector to implement open-set active learning. Specifically, they proposed a sampling scheme to ensure a balance of ID and OOD samples, which helps both the ID classifier and the OOD detector. The authors conducted a series of experiments, and the results of these experiments verified the effectiveness of the proposed method.", "questions": "The proposal uses meta-learning to calculate meta-weights for obtaining representative metrics of the samples. Compared to previous methods such as clustering and coreset, does this complex method introduce too much computational overhead? Does it have any special advantages? In the experiment, can we obtain good results using traditional representative metric methods?\n\n--- \nThanks for the clarifications. Most of the concerns have been addressed. I will keep my score.", "limitations": "The authors have provided a discussion on the broader impact.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The considered problem is important and highly relevant to the machine learning community. \n2. The proposal is simple and reasonable. The authors suggest that out-of-distribution (OOD) samples can also help the process of open-set active learning from the perspective of OOD detection. They use a simple and effective balanced sampling approach to alleviate the learning process differences and improve the performance of open-set active learning. This proposal suggests that actively annotating some OOD samples can enhance the OOD detector and help open-set active learning. This supports the claim of the title \"Not All Out-of-Distribution Data Are Harmful\" and extends the previous focus on mainly querying ID samples.\n3. The experiments are sufficient, and the effectiveness of the proposal has been clearly verified.", "weaknesses": "CCAL is also a well-known open-set active learning method, which should be discussed and compared in the experiments.  \n[CCAL] Contrastive active learning under class distribution mismatch. TPAMI’22  \n\nIt is regretful that there is no relevant theoretical analysis to further support this work.   \n\nMiscellaneous:  \nThe results in Table 1 are too dense. Consider adjusting the layout.\nAlso, consider using the same text font in Figures 3, 4, 5, 6, and 7 as in the main body.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Q1: \"CCAL is also a well-known open-set active learning method, which should...\"\n\nA1: We have compared the proposed PAL with CCAL on CIFAR-10 and CIFAR-100 with the ID proportion of 20%. The results in Table 1 reveal that PAL outperforms CCAL, for the reason that CCAL does not actively use OOD data like the existing open-set AL method, which may lead to bias in instances' similarity calculation for unlabeled data.\n\nTable 1: Comparison of classification accuracy (%) for CCAL and PAL on CIFAR-10 and CIFAR-100 with an ID proportion of 20%. \n\n|       | CIFAR-10 |       |       |       |       | CIFAR-100 |       |       |       |       |\n| :---: | :------: | :---: | :---: | :---: | :---: | :-------: | :---: | :---: | :---: | :---: |\n| Round |    1     |   3   |   5   |   7   |   9   |     1     |   3   |   5   |   7   |   9   |\n| CCAL  |  88.90   | 94.35 | 96.20 | 97.55 | 97.70 |   45.10   | 50.90 | 53.40 | 57.20 | 60.45 |\n|   PAL    | **91.14** | **95.59** | **97.59** | **98.49** | **98.74** | **45.65** | **53.04** | **60.04** | **65.64** | **69.39** |\n\nQ2: \"...no relevant theoretical analysis to further support...\"\n\nA2: As mentioned in the **Global Rebuttal**, we have provided a theoretical analysis to support the proposed PAL method. The generalization results reveal that PAL has a better generalization error bound than the traditional AL method, showing the effectiveness of the proposed PAL. We will provide more comprehensive details about the theoretical analysis in the revision.\n\n\n\nQ3: \"...results in Table 1 are too dense...\"\n\nA3: Thank you. We will modify the table and improve the readability in the revision.\n\n\n\nQ4: \"...does this complex method introduce too much computational overhead? Does it have any special advantages...\"\n\nA4: Actually, the core idea of active learning is to control the budget of manual annotations to maximize the advancements in model performance. Although the introduction of meta-weights increases training costs, the performance has been improved by selecting more effective instances. Moreover, we have replaced the meta-weight with Coreset to validate the effectiveness, and the PAL with Coreset method, i.e., Core_PAL. The results in Table 2 show that Core_PAL performs worse than PAL, revealing the effectiveness of meta-weight in selecting instances.\n\n\nTable 2: Comparison of classification accuracy (%) for Core_PAL and PAL on CIFAR-10 and CIFAR-100 with an ID proportion of 20%. \n\n|          | CIFAR-10 |       |       |       |       | CIFAR-100 |       |       |       |       |\n| :------: | :------: | :---: | :---: | :---: | :---: | :-------: | :---: | :---: | :---: | :---: |\n|  Round   |    1     |   3   |   5   |   7   |   9   |     1     |   3   |   5   |   7   |   9   |\n| Core_PAL |  87.09   | 93.29 | 96.74 | 97.44 | 98.29 |   44.79   | 49.75 | 59.59 | 63.74 | 65.99 |\n|   PAL    | **91.14** | **95.59** | **97.59** | **98.49** | **98.74** | **45.65** | **53.04** | **60.04** | **65.64** | **69.39** |\n\n"}, {"review_id": "4Qh4FzSe1e", "reviewer": "Reviewer_k1sx", "summary": "This manuscript studies open-set active learning and points out that concentrating solely on selecting pseudo-ID instances may cause the training imbalance of the ID classifier and OOD detector. To address this issue, this manuscript proposes a simple yet effective sampling scheme, dubbed Progressive Active Learning (PAL). Extensive experiments on various open-set AL scenarios demonstrate the effectiveness of PAL.", "questions": "How is N^{ID}_{query} selected? Is the sample with maximum of un(x^u_j) chosen into N^{ID}_{query}? The sample with maximum of un(x^u_j) indicates strong certainty in these samples. Perhaps using model-generated pseudo-labels would be accurate enough, without the need to waste resources on querying these relatively certain samples.", "limitations": "Please see Weaknesses.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1.This manuscript points out that concentrating solely on selecting pseudo-ID instances may cause the training imbalance of the ID classifier and OOD detector and proposes Progressive Active Learning (PAL) to solve the problem.\n\n2.Extensive experiments on various open-set AL scenarios demonstrate the effectiveness of PAL.", "weaknesses": "1.The manuscript lacks corresponding theoretical analysis.\n\n2.The manuscript lacks an introduction to related work on open-set semi-supervised learning and safe semi-supervised learning, such as [1], [2], and [3].\n\n3.In line 39-41, “the main challenges in open-set AL revolve around effectively selecting valuable ID instances for classifier training and distinguishing the OOD instances”, Similar viewpoints have already been proposed by [2]. \n\n4.The framework in Fig. 2 is confusing and the flow description is not clear enough. For example, how is f^c trained?\n\n5.In every query, how are N^{OOD}_{query} and N^{ID}_{query} set up?\n\n6.The description of the retraining process is not clear. What is the final loss? Is f^c used for the final testing?\n\n7.In line 136, entropy is used for obtaining s^{ID}. Has the author tried other metrics such as MSP in [4] or energy in [5]?\n\n8.The manuscript lacks important baselines, such as [1], [2], [3]. The manuscript needs to be compared with open-set semi-supervised learning and safe semi-supervised learning methods. \n\n9.The manuscript mentions the need for multiple rounds of query and training, so the runtime of the manuscript needs to be compared with the baseline.\n\n[1] Safe deep semi-supervised learning for unseen-class unlabeled data.\n[2] Safe-Student for Safe Deep Semi-Supervised Learning with Unseen-Class Unlabeled Data.\n[3] Openmatch: Open-set semi-supervised learning with open-set consistency regularization.\n[4] A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks.\n[5] Energy-based out-of-distribution detection.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Due to the space limit, we put all tables in the **attached one-page PDF**.\n\nQ1: \"...lacks corresponding theoretical analysis...\"\n\nA1: We have provided a theoretical analysis in the **Global Rebuttal** to support the proposed PAL method. We theoretically show that PAL has a better generalization error bound than the traditional AL method, which means PAL has better generalization ability than traditional AL. We will include more comprehensive theoretical details in the revision.\n\nQ2: \"... lacks an introduction to related work on open-set semi-supervised learning and safe...\"\n\nA2: Although both open-set/safe semi-supervised learning and open-set active learning aim to construct a more effective ID classifier by reasonably utilizing unlabeled data, the essential difference lies in whether or not manual participation is necessary. Open-set/Safe semi-supervised learning aims to use all the pseudo-ID instances in unlabeled data according to the model's prediction, whereas open-set active learning aims to select the most effective pseudo-ID instances in unlabeled data for manual labeling to retrain the model without using all the unlabeled data.  We will provide more discussion in the Related Work section.\n\nQ3: \"...Similar viewpoints have already been...\"\n\nA3: Open-set active learning aims to strategically select pure ID data and filter out OOD data, and thus a powerful OOD detector is important. The common practice of traditional open-set active learning methods, as shown in Figure 1 (b) in the manuscript, always relies on ID data to construct the OOD detector and then concentrates solely on select pseudo-ID instances, which would lead to the weakening of the OOD detector due to the lack of active selection of effective OOD data and affects the ID purity during the selection process. To address this challenge, we propose a simple yet effective sampling scheme that is progressively selecting pseudo-ID and pseudo-OOD instances in each round, especially adding valuable OOD data in the initial round. Thereby, it could enhance the capacity of the OOD detector and simultaneously promote the ID classifier by increasing the ID purity. Besides, we also provide a theoretical analysis to support why the proposed PAL is better than traditional AL. Specifically, the theoretical results show that the proposed PAL has a better generalization error bound than traditional AL, showing the effectiveness of the proposed PAL. Please refer to **Global Response** for more details about the theoretical analysis.\n\nQ4: \"... the flow description is not clear enough...how is $f^c$ trained? \"\n\nA4: Sorry for the confusion. Specifically, in the first round, the number of classes for detector $f^c$ is C, and we train $f^c$ with the initial $D_l$. In the subsequent rounds, the number of classes for detector $f^c$ is expanded to C+1 by actively adding the OOD data. In each round, we retrain the $f^c$ according to $l_{ova}$ loss with $D_{l_{\\text{all}}}^t$. The algorithmic pseudocode is included in Appendix A.\n\nQ5: \"...$N_{query}^{OOD}$ and $N_{query}^{ID}$ set up...\"\n\nA5: Thanks. In our experiments, the label budget is set to be 1500 in each round following [Ref 1, Ref 2], i.e., $ |D_{query}^{t}| = N_{query}^{OOD_{t}} + N_{query}^{ID_{t}}=1500$. Specifically, in the first round, to enhance the effectiveness of OOD detection, we set $N_{query}^{ID} = 300$ and $N_{query}^{OOD} = 1200$. In the following rounds, to select more valuable ID instances, we set $N_{query}^{ID} = 1450$ and $N_{query}^{ID} = 50$. From Figure 4 in the manuscript, we can conclude that the query purity of PAL is superior to the comparison method, which reveals that we can autonomically select useful pseudo-OOD and pseudo-ID according to our assignment.\n\n\n\n[Ref 1] Pan Du et al., Contrastive active learning under class distribution mismatch, ICCV2021\n\n[Ref 2] Kun-Peng Ning et al., Active Learning for Open-set Annotation, CVPR2022\n\nQ6: \"...retraining process is not clear...\"\n\nA6: Sorry for the confusion. Actually, the final loss is described in line 174. The final target of open-set active learning is the ID classification, and thereby we can adopt the $f^o$. Besides, we also give the detection result for supplementary validation using $f^c$. We will modify the descriptions in the revision. \n\nQ7: \"...tried other metrics such as MSP in [4] or energy in [5]...\" and  \"...lacks important baselines, such as [1], [2], [3]...\"\n\nA7: We have replaced $s^{ID}$ with the MSP/energy-based method and compared PAL with D3SL and OpenMatch. We conducted experiments on CIFAR-10 and CIFAR-100 with the ID proportion of 20%. The results are presented in **Table 3 and Table 4 of the attached one-page PDF** . Furthermore, by employing the official code of D3SL and conducting hyperparameter tuning with values such as meta_lr={6e-5, 3e-4, 2e-3}, un_batch_size={100, 128, 256}, and lr_weight={1e-3, 1e-2, 3e-4}, the results showed in Table 4 signify the most favorable outcomes attained. The results in Table 3 show that by using $s^{ID}$ metric, PAL achieves the best performances on CIFAR-10 and CIFAR-100 datasets, compared with PAL with MSP/energy. The results in Table 4 show that PAL outperforms D3SL and OpenMatch on CIFAR-10 and CIFAR-100 datasets.  \n\nQ8: \"...the runtime of the manuscript needs to be compared with the baseline.\"\n\nA8: We have included a comparison of running times for PAL and other baselines on CIFAR-10 and CIFAR-100 with an ID proportion of 20% in **Table 5 of attached one-page PDF** . The results show that PAL has the second fast runtime, which is faster than D3SL, OpenMatch and LfOSA, but is slower than Coreset.\n\n\nQ9: \"How is $N_{query}^{ID}$ selected...\"\n\nA9: Actually, the value of $un(x_{j}^u)$ represents the uncertainty, which comprehensively considers the instance's representativeness and informativeness. Therefore, we select the first b examples with the highest probability, as described in Line 155. We will give more details in the revision."}, {"review_id": "UmOVu0xE6u", "reviewer": "Reviewer_ACvU", "summary": "This paper proposes an active learning approach for open-set learning. The proposed approach does not try to avoid sampling OOD data as some of the prior work did. Instead, it samples some of the OOD data intentionally to enhance the OOD detector.", "questions": "None", "limitations": "None", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "1.The proposed idea is easy to implement and understand.\n2.The proposed problem, open-set active learning, is meaningful to me.\n", "weaknesses": "1.The overall framework is trivial to me. It is a traditional active learning method adapted to an OOD detector and an ID learner. \n2.The essential part of active learning, which is the design of the sampling criterion, is neither novel nor plausible to me. For the uncertainty weight, it is merely the prediction score of the model proposed by [27]. What is more, this score tends to approximate the likelihood of a data point being an in-distribution sample. But I do not think this score reveals the informativeness/ sample importance in active learning. Different data points can have very different contributions to learning, the proposed sampling score does not seem to have the ability to differentiate them. The meta-weight can be seen as a loss-based active sampling criterion for an OOD detector, which is trivial to me. \n3.All the proposed methods are empirical, there is no quantitative analysis to support the effectiveness of the proposed methods. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Q1: \"...framework is trivial to me...\"\n\nA1: Open-set active learning aims to strategically select pure ID data and filter out OOD data, and thus a powerful OOD detector is important. The common practice of traditional open-set active learning methods, as shown in Figure 1 (b) in the manuscript, always relies on ID data to construct the OOD detector. Subsequently, these methods primarily focus on selecting pseudo-ID instances, which would lead to the weakening of the OOD detector due to the lack of active selection of effective OOD data and affects the ID purity during the selection process.  To address this challenge, we propose a simple yet effective sampling scheme that progressively selects pseudo-ID and pseudo-OOD instances in each round, especially adding valuable OOD data in the initial round. Thereby, it could enhance the capacity of the OOD detector and simultaneously promote the ID classifier by increasing ID purity. Besides, we also provide a theoretical analysis to support why the proposed PAL is better than traditional AL. Specifically, the theoretical results show that the proposed PAL has a better generalization error bound than traditional AL, showing the effectiveness of the proposed PAL. Please refer to **Global Response** for more details about the theoretical analysis. \n\n\nQ2: \"... the uncertainty weight, ... the meta-weight...\"\n\nA2: Sorry for the confusion. We represent the informativeness of samples by measuring the uncertainty of the most confident ID class. This is accomplished by utilizing the prediction of the OVA classifier. Besides, considering the divergent distributions of ID and OOD data inspired by [Ref 3,  Ref 4], we utilize meta-weight to measure the representativeness of each sample by assessing its similarity to the established data distribution. Instances with higher meta-weights are deemed to be more representative as they match better with the existing data distribution. Besides, following the comments of Reviewer oop5, we replaced the meta-weight with Coreset (we denote the new method by Core_PAL) to validate the effectiveness of the meta-weight by conducting experiments on CIFAR-10 and CIFAR-100 datasets. The results are listed in the following table and show that the Core_PAL performs worse than PAL, revealing the effectiveness of meta-weight in selecting instances.\n\nTable 2: Comparison of classification accuracy (%) for Core_PAL and PAL on CIFAR-10 and CIFAR-100 with an ID proportion of 20%. \n\n|          | CIFAR-10 |       |       |       |       | CIFAR-100 |       |       |       |       |\n| :------: | :------: | :---: | :---: | :---: | :---: | :-------: | :---: | :---: | :---: | :---: |\n|  Round   |    1     |   3   |   5   |   7   |   9   |     1     |   3   |   5   |   7   |   9   |\n| Core_PAL |  87.09   | 93.29 | 96.74 | 97.44 | 98.29 |   44.79   | 49.75 | 59.59 | 63.74 | 65.99 |\n|   PAL    | **91.14** | **95.59** | **97.59** | **98.49** | **98.74** | **45.65** | **53.04** | **60.04** | **65.64** | **69.39** |\n\n[Ref 3] Hugo Larochelle et al., OOD-MAML: Meta-Learning for Few-Shot Out-of-Distribution Detection and Classification, NeurIPS2020\n\n[Ref 4] Cesar Almecija et al., Uncertainty-Aware Meta-Learning for Multimodal Task Distributions, ICLR2023\n\n\nQ3: \"...proposed methods are empirical, there is no quantitative analysis...\"\n\nA3: In fact, we don't fully understand this question. We will try our best to answer this question, and any additional discussion is welcome. First, we have provided a theoretical analysis (please refer to **Global Response** for details) to show the effectiveness of the proposed PAL. Additionally, we have performed a repeatability experiment in Appendix B. As shown in Figure 10 in the supplementary material, the central line represents the mean value, while the upper and lower limits correspond to its standard deviations for each color. The results demonstrate that our PAL method exhibits consistently high stability. \n\n"}]

Author rebuttal:
We sincerely thank the PC, SAC, ACs, and reviewers for handling and reviewing our paper. All constructive and valuable comments are helpful in further improving our paper. 

Since most reviewers have mentioned the lack of theoretical analysis, we provided a generalization analysis. Our theoretical results reveal that the proposed PAL has a better generalization error bound than traditional AL, showing the effectiveness of the proposed PAL. We include the details as follows.

We also include additional experimental results in attached ***one-page PDF*** file. 


======== Theoretical Analysis ========

To theoretically understand the use of detected OOD data in the OOD detector, we first give the following notations. Suppose that the samples $(\mathbf{x},y)\in(\mathcal{X},\mathcal{Y})$ follows a unknown distribution $\mathcal{P}$, where $\mathcal{X} = \mathbf{R}^d$ is input space and $\mathcal{Y} = \\{0,1\\}$ is output space. The label of ID data is $y=0$, and the label of OOD data is $y=1$. Let $\ell:\mathbf{R}\times\mathcal{Y}\longrightarrow \mathbf{R}^+$ be the loss of interest. With the input $\mathbf{x}$ and its corresponding label $y$, the expected loss is $\mathcal{L}(f) = \mathbb{E}_{\mathcal{P}}[\ell(f(\mathbf{x}),y)]$. 

Suppose we have a training dataset $\\{(\mathbf{x_1},y_1), \dots, (\mathbf{x_n},y_n)\\}$ drawn from distribution $\mathcal{P}$, then the empirical loss is $\mathcal{\widehat L}(f) = \frac{1}{n}\sum_{i=1}^{n}\ell(f(\mathbf{x_i}),y_i)$. Let $C(\mathcal{F})$ be some proper complexity measure of the family of hypothesis class $\mathcal{F}$. We assume that the loss is Lipschitz with constant $L$ and the Rademacher complexity $\mathfrak{R}_n(\mathcal{F}) \le \sqrt{\frac{C(\mathcal{F})}{n}}$. To simplify the analysis, we assume the distribution of ID and OOD samples is balanced and the number of OOD samples is the same as the number of ID samples in the training set. We suppose that the number of detected ID data is $m$ while the number of detected OOD data is $n-m$ where $m<n$, and among the detected ID data, the number of real ID data is $m_0 := \alpha m$ while the number of real OOD data is $m_1 := (1-\alpha)m$  with $0<\alpha<1$. 

**Theorem 1.** For a Lipschitz loss $\ell$ bounded by $c$, we have the following results with probability at least $1-\delta$ simultaneously.

For the proposed PAL method, the generalization error bound is
\begin{align}
    \mathcal{L}(f_{PAL})- \mathcal{\widehat L}(f_{PAL}) \le 2L\sqrt{\frac{C(\mathcal{F})}{n}} + c\sqrt{\frac{\log(1/\delta)}{2n}} \le O\left(\frac{1}{\sqrt{n}}\right).
\end{align}
For the standard AL method, the generalization error bound is

\begin{align}
    \nonumber \mathcal{L}(f_{AL})- \mathcal{\widehat L}(f_{AL}) \le & L\sqrt{\frac{C(\mathcal{F})}{\alpha m}} + L\sqrt{\frac{C(\mathcal{F})}{(1-\alpha)m}} + \frac{c}{2}\sqrt{\frac{\log(2/\delta)}{2\alpha m}} + \frac{c}{2}\sqrt{\frac{\log(2/\delta)}{2(1-\alpha)m}} \le O\left(\frac{1}{\sqrt{\alpha m}} + \frac{1}{\sqrt{(1-\alpha)m}}\right),
\end{align}
where $0<\alpha<1$ and $n<m$.

**Remark.**   Since $0<\alpha<1$ and $n<m$,  we know $\frac{1}{\sqrt{n}} \le \frac{1}{\sqrt{\alpha m}} + \frac{1}{\sqrt{(1-\alpha)m}} $, showing that the generalization error bound for PAL method is better than the bound for standard AL method. That is to say, the use of detected OOD data can improve the effectiveness of the OOD detector. 

**Proof Sketch.** First, we proof the generalization error of the proposed PAL method. Following Theorem 1 of [R1], we have
\begin{align}
    \nonumber \mathcal{L}(f_{PAL})- \mathcal{\widehat L}(f_{PAL}) \le 2L\mathfrak{R}_n(\mathcal{F})+ c\sqrt{\frac{\log(1/\delta)}{2n}},
\end{align}
where $\mathfrak{R}_n(\mathcal{F})$ is the Rademacher complexity of a function class $\mathcal{F}$ and $n$ is the number of training samples. Withe the Rademacher complexity $\mathfrak{R_n}(\mathcal{F}) \le \sqrt{\frac{C(\mathcal{F})}{n}}$, we give equation (1). 

Next, we will proof the generalization error of the standard AL method. To this end, we denote by $\mathcal{P} = \mathcal{P}(\mathbf{x}|y=0)$ the conditional probability for ID data and $\mathcal{P_1} = \mathcal{P}(\mathbf{x}|y=1)$ the conditional probability for OOD data. Let $\mathcal{L_j}(f)$ be the loss from class $j \in \\{0,1\\}$: $\mathcal{L_j}(f) = \mathbb{E_{\mathcal{P_j}}}[\ell(f(\mathbf{x}),y)]$, and let $\mathcal{\widehat L_j}(f)$ be its corresponding empirical loss. 
Then by applying the standard analysis for each class $j$ in Theorem 1 of [R1], with probability $1-\delta/2$ we have 
\begin{align}
    \nonumber \mathcal{L_j}(f_{AL})- \mathcal{\widehat L_j}(f_{AL}) \le 2L\mathfrak{R_{m_j}}(\mathcal{F})+ c\sqrt{\frac{\log(2/\delta)}{2m_j}}, (j = 0,1).
\end{align}

Since the distribution of ID and OOD samples is balanced, we have 
$\mathcal{L}(f_{AL}) = \frac{1}{2} \mathcal{L_0}(f_{AL}) + \frac{1}{2} \mathcal{L_1}(f_{AL})$ due to the definitions of the loss functions. Similarly, due to the number of OOD samples is same as the number of ID samples in training set, we know $\mathcal{\widehat L}(f_{AL}) = \frac{1}{2}\mathcal{\widehat L_0}(f_{AL}) + \frac{1}{2}\mathcal{\widehat L_1}(f_{AL})$. Finally, by applying the union bound, we have
\begin{align}
    \nonumber \mathcal{L}(f_{AL})- \mathcal{\widehat L}(f_{AL}) \le L\mathfrak{R_{m_0}}(\mathcal{F})+ + L\mathfrak{R_{m_1}}(\mathcal{F})
    +\frac{c}{2}\sqrt{\frac{\log(2/\delta)}{2m_0}} + \frac{c}{2}\sqrt{\frac{\log(2/\delta)}{2m_1}}.
\end{align}

With the Rademacher complexity $\mathfrak{R}_{m_j}(\mathcal{F}) \le \sqrt{\frac{C(\mathcal{F})}{m_j}}$ for $j=0,1$, we complete the proof of equation (2) by plugging in $m_0 = \alpha m$ and $m_1=(1-\alpha)m$.

[R1] Sham M Kakade, Karthik Sridharan, and Ambuj Tewari. On the Complexity of Linear Prediction: Risk Bounds, Margin Bounds, and Regularization. In Advances in Neural Information Processing Systems, pages 793–800, 2009.

Meta-review: This paper was reviewed by four experts and received one Borderline Accept, two Weak Accept and one Accept as the ratings. All the reviewers agreed that the paper addressed a problem relevant to the machine learning community; it is well-written, and the ideas are easy to understand. They have also mentioned that the empirical results are encouraging and show that the performance improvement achieved by the proposed method is significant. 

The reviewers had raised concerns about the theoretical analysis of the method, which was provided by the authors in the rebuttal. Concerns were also raised about the novelty of the paper and its computation cost, which were answered convincingly by the authors in the rebuttal. One of the reviewers had mentioned that the proposed technique should be compared against CCAL, a well-known method for open-set active learning; the authors have conducted additional experiments, and reported the results in the rebuttal. 

The reviewers, in general, have a positive opinion about the paper and its contributions. Most of them have been satisfied with the author rebuttal, have raised their scores and have recommended acceptance. Based on the reviewers’ feedback, the decision is to recommend the paper for acceptance to NeurIPS 2023. The reviewers have provided some valuable suggestions, particularly about the discussion on open-set SSL in the Related Work section and a clear description of the loss function used for training. The authors are encouraged to address these in the final version of their paper. We congratulate the authors on the acceptance of their paper!
Decision: Accept (poster)