Title: Not All Out-of-Distribution Data Are Harmful to Open-Set Active Learning
Venue: NeurIPS 2023
Accepted: yes
Keywords: Out-of-Distribution

Active Learning
Abstract: Active learning (AL) methods have been proven to be an effective way to reduce the labeling effort by intelligently selecting valuable instances for annotation. Despite their great success with in-distribution (ID) scenarios, AL methods suffer from performance degradation in many real-world applications because out-of-distribution (OOD) instances are always inevitably contained in unlabeled data, which may lead to inefficient sampling. Therefore, several attempts have been explored open-set AL by strategically selecting pure ID instances while filtering OOD instances. However, concentrating solely on selecting pseudo-ID instances may cause the training constraint of the ID classifier and OOD detector. To address this issue, we propose a simple yet effective sampling scheme, Progressive Active Learning (PAL), which employs a progressive sampling mechanism to leverage the active selection of valuable OOD instances. The proposed PAL measures unlabeled instances by synergistically evaluating instances' informativeness and representativeness, and thus it can balance the pseudo-ID and pseudo-OOD instances in each round to enhance both the capacity of the ID classifier and the OOD detector. %Meanwhile, PAL measures unlabeled instances by synergistically evaluating instances' informativeness and representativeness, which can more effectively estimate the values of instances. 
Extensive experiments on various open-set AL scenarios demonstrate the effectiveness of the proposed PAL, compared with the state-of-the-art methods. The code is available at \url{https://github.com/njustkmg/PAL}.

Not All Out-of-Distribution Data Are Harmful to Open-Set Active Learning

Abstract

Active learning (AL) methods have been proven to be an effective way to reduce the labeling effort by intelligently selecting valuable instances for annotation. Despite their great success with in-distribution (ID) scenarios, AL methods suffer from performance degradation in many real-world applications because out-of-distribution (OOD) instances are always inevitably contained in unlabeled data, which may lead to inefficient sampling. Therefore, several attempts have been explored open-set ALby strategically selecting pure ID instances while filtering OOD instances. However, concentrating solely on selecting pseudo-ID instances may cause the training constraint of the ID classifier and OOD detector. To address this issue, we propose a simple yet effective sampling scheme, Progressive Active Learning (PAL), which employs a progressive sampling mechanism to leverage the active selection of valuable OOD instances. The proposed PAL measures unlabeled instances by synergistically evaluating instances' informativeness and representativeness, and thus it can balance the pseudo-ID and pseudo-OOD instances in each round to enhance both the capacity of the ID classifier and the OOD detector. Extensive experiments on various open-set AL scenarios demonstrate the effectiveness of the proposed PAL, compared with the state-of-the-art methods. The code is available at https://github.com/njustkmg/PAL .

1 Introduction

Deep learning has achieved great success in many tasks such as image classification [14], article prediction [35], and graph classification [32]. The main factor underlying its excellent performance lies in the availability of an extensive dataset with manual annotations, which can be costly in real-world applications [16]. To reduce the high labeling cost, active learning is an effective approach that involves iteratively selecting valuable instances from the unlabeled data pool and querying their labels for model retraining [4].

The basic assumption shared by various existing active learning methods is that all unlabeled instances are collected from the ID domain [1, 28], showing that the labeled and unlabeled instances share the same distribution. However, this assumption is unrealistic in real-world scenarios as unlabeled instances are primarily gathered from the open world, where massive instances belong to the OOD domain. Taking the image classification task as an example, the goal is to classify mammals, but a considerable proportion of the unlabeled instances consists of unknown classes (i.e., OOD domain), such as bees, apples, and tables. Once these images are selected for labeling, it would be a waste of the annotation budget since they are not used for training the ID classifier. Therefore, it is important

to detect ID and OOD instances before the annotation in open-set AL, which is not applicable in traditional active learning methods. Consequently, the main challenge in open-set AL revolves around effectively selecting valuable ID instances for training classifier and distinguishing the OOD instances.

1

3

5

7

9

Round

0.2

0.3

0.4

0.5

0.6

0.7

ID Selection Recall

Acc: 57%

Acc: 66%

Acc: 69%

w/ selected OOD data (PAL)

w/o OOD data (LfOSA)

w/o OOD data (Coreset)

( a ) The power of OOD instances.

OOD

Detector

Unlabeled

Data

Selection

Selection

ID Classifier

Annotation

Annotation

OOD

data?

Yes

No

Traditional

open-set AL

Figure 1: (a) the effectiveness of actively querying valuable pseudo-OOD instances. ID Selection Recall (higher is better): the ratio of selected ID instances to the total number of ID instances. Acc: testing accuracy; w/: with; w/o: without. Experiments are conducted on CIFAR-100 using the first 20 classes as seen classes. We employ (1) the traditional active learning method Coreset; (2) the existing open-set AL approach LfOSA that only selects pseudo-ID instances; and (3) the proposed PAL that actively queries pseudo-OOD instances to enhance the training of the detector and classifier. (b) the distinction between the proposed PAL process and the traditional open-set AL process (in the dashed rectangle). The proposed PAL includes an additional module (the process in red) that carefully leverages detected OOD instances during the model training progress.

To address the challenges of open-set AL, a few studies have proposed [12, 22, 23], which commonly attempt to effectively filter the OOD instances in the unlabeled pool by adopting an extra OOD detector, thus increasing the ID purity of instances in the query set. Their goal is to optimize the selection of ID instances while eliminating OOD instances during each query round, aiming to exclusively add ID instances for model training. In other words, these methods mainly focus on learning the ID classifier, consequently restraining the resolution of the OOD detector. In contrast, we believe that not all OOD instances are harmful, and valuable OOD instances are also significant for both the classifier and detector training. For example, as shown in Figure 1 (a), we observe that actively selecting OOD instances based on their importance (i.e., PAL method) can achieve higher classification accuracy and selection recall, which is attributed to the promotion of ID purity and OOD filtering. Therefore, valuable OOD instances can promptly strengthen the boundary between ID and OOD instances, and further improve the ID purity for classifier retraining.

To this end, we propose a simple yet effective Progressive Active Learning (PAL) scheme that strategically introduces valuable pseudo-ID and pseudo-OOD instances for enhancing both the ID classifier and OOD detector. Different from existing open-set AL methods that solely focus on selecting pseudo-ID instances, our approach goes a step further by actively incorporating valuable pseudo-OOD instances, which allows us to explicitly retrain the OOD detector in conjunction with ID instances, resulting in a mutual promotion of the detector and classifier. The proposed PAL process is illustrated in Figure 1(b). Besides, to comprehensively measure the unlabeled instances, we propose to estimate the informativeness of instances by calculating entropy using the predictions from one-vs-all (OVA) classifiers, while evaluating the representativeness of instances with the automatically learned meta-weights.

2 Related Work

Active Learning. Active Learning aims to train a classifier by querying labels of the most valuable instances in the unlabeled data, which can greatly reduce the manual labeling cost [7]. The core idea of existing active learning approaches is to ensure that selected instances can significantly improve model performance, and various effective sampling strategies are designed [9, 24, 31, 37]. The initial sampling criteria is uncertainty-based sampling [4], which prefers to select instances that maximally reduce the model uncertainty. For example, [9] proposed an uncertainty measure that generalizes margin-based uncertainty; [37] employed the pairwise gradient length as a metric of informativeness.

Furthermore, various representativeness-based methods are proposed, which prefer to select instances that match the data distribution. For example, [24] estimated the expected reduction in error that directly optimizes expected future error; [1] designed batch active learning with diverse gradient embeddings, which sampled groups of instances that are disparate and high magnitude. However, this family of approaches usually assumes in-distribution settings that the unlabeled instances are drawn from the same distribution of labeled examples, which is not suitable for effectively filtering OOD instances during the query round.

Several recent approaches have attempted to carry out the open-set AL [3, 12, 22, 23]. [3] introduced a contrastive AL framework to learn two contrastive coding models for calculating informativeness and representativeness of an example; [22] introduced an auxiliary network to model the per-example max activation value distribution with a gaussian mixture model, which can dynamically select the instances with the highest probability from known classes in the unlabeled set; [23] employed the meta-learning technique to find the best trade-off between purity and informativeness; [12] selected a set that maximize the distance on entire unlabeled data while minimizing the distance to the identified OOD instances. However, we find that these attempts are limited in filtering OOD instances, as they usually assign greater weight to maintain ID purity, thus causing the learning constraint of the ID classifier and OOD detector. Consequently, it is vital to develop effective approaches that achieve a comprehensive balance between the ID classifier and the OOD detector.

Out-of-distribution Detection. OOD detection aims to identify instances outside the target domain [27, 30, 39], i.e., ID domain, which is vital for securing the model's safety and interpretability [15, 34]. Existing methods always calculate instances' confidence scores based on the predictions of the classifier trained on ID instances such as the entropy of the prediction distribution [2, 18, 33]. For example, [18] detected OOD instances with softmax score by considering temperature scaling and input pre-processing; [2] introduced a new model layer OpenMax, which used activation vectors to estimate the probability of deep network failure. To build a more robust detector, several approaches attempted to utilize the generative adversarial network to generate pseudo-OOD instances for assisting learning. For example, [11] proposed task-aware variation adversarial AL that considered data distribution of both label and unlabeled pools; [8] confirmed that generative model accuracy may not always be positively correlated with OOD detection performance. Nonetheless, the advantage of open-set AL lies in the existence of OOD instances within the unlabeled pool, making the fine-grained usage of OOD instances possible.

Query

Train

Retrain

⊕

�

�

Prediction

1

C

...

2

�

� 퐈

Un

1

C

...

C+1

OOD ID

ID Labled set

...

Unabled Set

...

퐼�

푂푂�

+

�

=

Share Parameter

� �

� �

⊕

�

푂푂�

⊕

3 Proposed Method

Problem Setting. We denote by the labeled data D l = { ( x l i , y l i ) } N l i =1 , where each x l i belongs to the C known labels y l i ∈ R C and the unlabeled data D u = { x u j } N u j =1 = {D ID u , D OOD u } , where D ID u denotes the ID unlabeled data and D OOD u represents the OOD unlabeled data, D ID u ∩D OOD u = ∅ . In other words, an unlabeled instance x u j may belong to an unknown class. In active learning, a human oracle is requested to label query set D t query at t -th query round, which consists of OOD query set D OOD t query and ID query set D ID t query (i.e., D t query = D OOD t query ∪ D ID t query ). Considering the restrictions on the annotation budget, the task of traditional open-set AL is to construct the query set that contains more ID instances for assisting ID classifier building.

Approach Overview. The framework of the proposed PAL is illustrated in Figure 2, including two key components: 1) importance score considering both informativeness and representativeness; 2) progressive sampling mechanism that precisely manages the ID and OOD instances within the query set. Diverging from traditional open-set AL attempts, we adopt a strategy of actively selecting the valuable pseudo-ID and pseudo-OOD instances to enhance the capacities of the classifier and detector simultaneously. In the following subsections, we first introduce the calculations of the importance score and then give the details of progressive sampling. Finally, we present a theoretical understanding of using OOD instances.

3.1 Sampling Criteria

Uncertainty weight. To investigate the informativeness, we employ the one-vs-all (OVA) classifiers [26] to learn a boundary between ID and OOD instances. In detail, OVA classifiers include C sub-classifiers, each one is to distinguish whether the instance belongs to the corresponding class or not, e.g., the prediction of c -th sub-classifier can be denoted as p c ( x l ) = f c ( g ( x l )) ∈ R 2 , where g represents the feature encoder, f c denotes the c -th classifier. That is to say, p c ( x l ) := ( p c ( p = 1 | x l ) , p c ( p = 0 | x l )) , where p c ( p = 1 | x l ) and p c ( p = 0 | x l ) denotes the probability of instance being an ID and an OOD respectively for class c . The overall loss of the OVA classifiers can be formulated as:

̸

where y x l i denotes the class of instance x l i . Following [26], we adopt the hard-negative sampling technique for sub-classifiers. The core idea of OV A classifiers is that each sub-classifier outputs a distance representing how far the input is from the corresponding class [26]. Therefore, the subclassifiers are effective in identifying OOD instances. To be specific, the probability of instance x u to be an ID can be calculated by s ID = 1 + p ˆ c ( p = 1 | x u ) log p ˆ c ( p = 1 | x u ) , where ˆ c = arg max c p c ( p = 1 | x u ) . Actually, s ID can express the informativeness of a given unlabeled instance, and larger s ID represents more likely the instance to be an ID instance.

Meta-weight. In addition to s ID , we design a representativeness criterion by learning the metaweight of each unlabeled instance. Since the OOD instance in unlabeled set leads to a distribution mismatch, inspired by [5], we aim to automatically learn the weight of each unlabeled instance via tracking the effect of the supervised learning model to prevent OOD instances, which can be formulated as the following bi-level optimization problem:

where Θ = (Θ 1 , Θ 2 ) , ℓ sup ( x l i , y l i , Θ) = -∑ C c =1 y l i log f o,c Θ 1 ( g Θ 2 ( x l i )) is the supervised loss, ℓ un ( x u j , Θ) = -∑ C c =1 f o,c Θ 1 ( g Θ 2 ( x u j )) log f o,c Θ 1 ( g Θ 2 ( x u j )) is the unsupervised loss, and f o denotes the ID classifier. The problem (2) captures the process of learning the optimal model parameter ˆ Θ( ω )

through empirical risk minimization using labeled and weighted unlabeled instances, the model is then evaluated on D l to optimize the weights, aiming to enhance the reliable performance. Thus, the learned ω is considered as the measurement of representativeness, with higher values indicating that ID instances are more representative.

3.2 Progressive Sampling

Open-set AL aims to precisely select as many ID instances as possible from the unlabeled pool. With the s ID and ω , we select the first b instances with the highest probability as the query set:

where µ is the hyper-parameter. However, unlike existing open-set AL attempts that only concern ID selection, we empirically find that valuable OOD can enhance the detector and further boost ID selection. Therefore, we propose a simple yet effective progressive sampling. To be specific, in each round, we actively select b instances including N ID query pseudo-ID instances and N OOD query pseudo-OOD instances, noting that we select OOD instances with the lowest un ( x u j ) . At the first query, we set N OOD query ≫ N ID query . After that, the OV A classifiers are extended to C +1 classifiers, and the un ( x u ) is given by

3.3 Classifier Training

Based on the newly labeled data D t query , we retrain both the ID classifier and OOD detector. In detail, the current ID labeled data can be represented as D t l = D t -1 l ∪ D ID t query , where D t -1 l denotes the accumulated ID labeled data, D ID t query denotes the ID labeled data from the detected ID data. The ID classifier f o is learned by minimizing the supervised loss ∑ |D t l | i =1 ℓ sup ( x l i , y l i , Θ) . On the other hand, the current ID and OOD data for training OVA classifiers can be denoted as D t l all = D t -1 l all ∪ D t query , where D t -1 l all denotes the accumulated ID and OOD labeled data. Notably, in the first round, the number of classes for detector f c is C , and we train f c with the initial D l . In the subsequent rounds, the number of classes for detector f c is expanded to C +1 by actively adding the OOD data. In each round, we retrain the f c according to loss ℓ ova . The final target of open-set AL is the ID classification, and thereby we can adopt the f o .

3.4 Theoretical Understanding

To theoretically understand the use of detected OOD instances in the OOD detector, we first give the following notations. Suppose that the samples ( x , y ) ∈ ( X , Y ) follows a unknown distribution P , where X = R d is input space and Y = { 0 , 1 } is output space. The label of ID instance is y = 0 and the label of OOD instance is y = 1 . Let ℓ : R × Y -→ R + be the loss of interest. With the input x and its corresponding label y , the expected loss is L ( f ) = E P [ ℓ ( f ( x ) , y )] . Suppose we have a training dataset { ( x 1 , y 1 ) , . . . , ( x n , y n ) } drawn from distribution P , then the empirical loss is ̂ L ( f ) = 1 n ∑ n i =1 ℓ ( f ( x i ) , y i ) . Let C ( F ) be some proper complexity measure of the family of hypothesis class F . We assume that the loss is Lipschitz with constant L and the Rademacher complexity R n ( F ) ≤ √ C ( F ) n [10]. To simplify the analysis, we assume the distribution of ID and OOD samples is balanced and the number of OOD samples is the same as the number of ID samples in the training set. We suppose that the number of detected ID instances is m while the number of detected OOD instances is n -m where m<n , and among the detected ID instances, the number of real ID instances is m 0 := αm while the number of real OOD instance is m 1 := (1 -α ) m with 0 < α < 1 . Then we present the following theorem, whose proof can be found in the supplementary material.

Theorem 1 For a Lipschitz loss ℓ bounded by c , we have the following results with probability at least 1 -δ simultaneously. For the proposed PAL method, the generalization error bound is

For the standard AL method, the generalization error bound is

Remark. Since 0 < α < 1 and n < m , we know 1 √ n ≤ 1 √ αm + 1 √ (1 -α ) m , showing that the generalization error bound for PAL method is better than the bound for standard AL method. That is to say, the use of detected OOD instances can improve the effectiveness of the OOD detector.

4 Experiments

In this section, we aim to demonstrate the effectiveness of the proposed PAL. Due to the page limitation, more experimental results and details can be found in the supplementary material.

4.1 Experimental Setups

Datasets. We evaluate the efficiency of PAL on several image classification benchmarks, i.e., CIFAR10, CIFAR-100 [13] and Tiny-Imagenet [36] datasets following standard open-set AL methods [12, 22]. The CIFAR-10 dataset consists of 50,000 training images and 10,000 test images, with 10 classes and 5,000 images per class in the training set. The CIFAR-100 dataset has 100 classes and 50,000 training images and 5,000 test images, with 500 images per class in the training set. The Tiny-Imagenet dataset consists of 100,000 training images and 10,000 validation images, with 200 classes and 500 images per class in the training set. To construct the open-set ALscenarios, following [22], we set the proportion of ID classes as 20 % , 30 % , and 40 % in experiments. For example, when the proportion is set as 20 % on CIFAR-10, CIFAR-100, and Tiny-Imagenet, the first 2, 20, and 40 classes are considered as ID classes and the last 8, 80, and 160 classes are considered as OOD classes.

Baselines and Performance criteria. We compare PAL with three different types of methods: 1) supervised method, i.e., Label Only. 2) traditional AL methods, i.e., Random, Uncertainty [17], Certainty [20], Coreset [28], Openmax [2], and BALD [29]. 3) open-set AL methods, i.e., CCAL [3], LfOSA [22] and MQnet [23]. We consider performance from two aspects: 1) ID classification, i.e., classification accuracy. 2) OOD detection during query round, i.e., precision and recall [22].

Parameter setting. For all AL methods, following [22], we randomly sample 1 % , 10 % and 10 % of the examples as the initial labeled set on CIFAR-10, CIFAR-100, and Tiny-Imagenet datasets, respectively. To ensure fairness, we employ WideResNet [38] as the backbone for training all methods. Note that the labeled data only contains ID classes. In each AL round, we train the model for 100 epochs, using the SGD optimizer with the momentum parameter of 0.9. The learning rate is initialized as 0.01 with a mini-batch size of 128, and the weight decay is set to be 5 × 10 -4 . Additionally, the annotation budget per query round is limited to 1500 following [22]. All experiments are implemented on a single NVIDIA V100 GPU.

4.2 Performance Comparison

Table 1 and Figure 3 record the ID classification and OOD detection results of PAL and other AL methods with the increasing number of queries, where the index { 1 , 3 , 5 , 7 , 9 } denotes the query round.

ID classification. The results in Table 1 show that: 1) AL methods perform better than the supervised method because the oracle will filter the OOD instances in the query set. However, the promotion is limited due to decreased ID instances, especially in complex datasets such as CIFAR-100 and Tiny-Imagenet. 2) CCAL, LfOSA, and MQnet outperform traditional active learning methods because LfOSA selects a more purified set for querying, including more ID instances and fewer OOD instances. LfOSA performs better than MQnet because MQnet maintains a balance between purity and informativeness of the query set that contains several OOD. 3) PAL consistently exhibits the best performance in each round on all datasets, especially on the Tiny-Imagenet dataset. Compared to other open-set AL methods, PAL improvements about 7.5 % , 3.9 % , and 4.2 % under the 5, 7, and 9 rounds respectively, indicating that PAL could better control the purity ID by constructing a more robust OOD detector.

1

3

5

7

9

Round

0.2

0.4

0.6

0.8

Selection Precision

Random

Uncertainty

Certain

Coreset

BALD

Openmax

CCAL

MQnet

LfOSA

PAL

(a) CIFAR-10 Precision

1

3

5

7

9

Round

0.1

0.2

0.3

0.4

0.5

Selection Precision

Random

Uncertainty

Certain

Coreset

BALD

Openmax

CCAL

MQnet

LfOSA

PAL

(b) CIFAR-100 Precision

1

3

5

7

9

Round

0.1

0.2

0.3

0.4

0.5

0.6

Selection Precision

Random

Uncertainty

Certain

Coreset

BALD

Openmax

CCAL

MQnet

LfOSA

PAL

(c) Tiny-Imagenet Precision

1

3

5

7

9

Round

0.2

0.4

0.6

0.8

Selection Recall

Random

Uncertainty

Certain

Coreset

BALD

Openmax

CCAL

MQnet

LfOSA

PAL

(d) CIFAR-10 Recall

1

3

5

7

9

Round

0.2

0.3

0.4

0.5

0.6

0.7

Selection Recall

Random

Uncertainty

Certain

Coreset

BALD

Openmax

CCAL

MQnet

LfOSA

PAL

(e) CIFAR-100 Recall

1

3

5

7

9

Round

0.15

0.20

0.25

0.30

0.35

0.40

Selection Recall

Random

Uncertainty

Certain

Coreset

BALD

Openmax

CCAL

MQnet

LfOSA

PAL

(f) Tiny-Imagenet Recall

0.7

OOD detection. The results of Figure 3 reveal that PAL consistently outperforms other baselines in terms of most selection precision and recall. For example, the proposed PAL achieves 15.9 % higher precision than LfOSA on Tiny-Imagenet dataset in the last round, which validates that PAL can better eliminate the OOD instances by adopting the discriminative criteria and progressive sampling. Moreover, note that the precision of open-set AL methods shows an initial increase followed by a subsequent decrease, for the reason that we are taking an unreplaced query, so the remaining unlabeled data is gradually reduced. Meanwhile, PAL can select more ID instances at the starting several rounds on CIFAR-10, resulting in fewer ID instances left in the unlabeled set considering the relatively smaller dataset size of CIFAR-10. Thereby, PAL achieves inferior precision compared to LfOSA in later rounds.

To further visualize the validity of pseudo-ID and pseudo-OOD instances, we compare the query purity of PAL with LfOSA in query rounds on three datasets, as illustrated in Figure 4. Although we actively select more OOD instances in the first round, the overall OOD instances are much fewer than LfOSA. Specifically, under the CIFAR-10, LfOSA selected 52.2 % OOD instances, whereas PAL only selected 40.3 % . Besides, we find that the purities of pseudo-ID and pseudo-OOD are better than LfOSA, i.e., PAL achieves an overall 12.6 % higher query purity compared to LfOSA on the Tiny-Imagenet dataset, which validates the effectiveness of progressive building OOD detector.

1

3

5

7

9

Round

0

200

400

600

800

1000

1200

1400

Number

Lfosa

PAL

ID_ID

ID_OOD

OOD_OOD

OOD_ID

(a) CIFAR-10

1

3

5

7

9

Round

0

200

400

600

800

1000

1200

1400

Number

Lfosa

PAL

ID_ID

ID_OOD

OOD_OOD

OOD_ID

(b) CIFAR-100

1

3

5

7

9

Round

0

200

400

600

800

1000

1200

1400

Number

Lfosa

PAL

ID_ID

ID_OOD

OOD_OOD

OOD_ID

(c) Tiny-Imagenet

4.3 Influence of Different ID Class Proportions

In this subsection, we evaluate the influence of different numbers of ID classes. The average results of accuracy, precision, and recall are presented in Figure 5, 6, and 7 respectively. In each figure, the first, second, and third columns represent the results on CIFAR-10, CIFAR-100, and Tiny-Imagenet, respectively, while the first and second rows represent the results with ID proportion as 30 % and 40 % . Please note that the results of 20% are included in Table 1 and Figure 3. The results exhibit that the proposed PAL always outperforms other methods in terms of ID classification and OOD detection for all cases. For example, when ID proportion is set to 30 % , PAL outperforms 3.6 % , 3.3 % , and 4.3 % than the state-of-the-art method, i.e., LfOSA, in terms of accuracy, precision, and recall in the last round on CIFAR-100.

1

3

5

7

9

Round

0.80

0.82

0.84

0.86

0.88

0.90

0.92

0.94

0.96

Random

Uncertainty

Certain

Coreset

BALD

Openmax

CCAL

MQnet

LfOSA

PAL

1

3

5

7

9

Round

0.45

0.50

0.55

0.60

0.65

Random

Uncertainty

Certain

Coreset

BALD

Openmax

CCAL

MQnet

LfOSA

PAL

1

3

5

7

9

Round

0.25

0.30

0.35

0.40

0.45

Random

Uncertainty

Certain

Coreset

BALD

Openmax

CCAL

MQnet

LfOSA

PAL

1

3

5

7

9

Round

0.70

0.75

0.80

0.85

0.90

Random

Uncertainty

Certain

Coreset

BALD

Openmax

CCAL

MQnet

LfOSA

PAL

(a) CIFAR-10 Accuracy

1

3

5

7

9

Round

0.40

0.45

0.50

0.55

0.60

0.65

Random

Uncertainty

Certain

Coreset

BALD

Openmax

CCAL

MQnet

LfOSA

PAL

1

3

5

7

9

Round

0.20

0.25

0.30

0.35

0.40

Random

Uncertainty

Certain

Coreset

BALD

Openmax

CCAL

MQnet

LfOSA

PAL

(c) Tiny-Imagenet Accuracy

1

3

5

7

9

Round

0.2

0.4

0.6

0.8

Selection Precision

Random

Uncertainty

Certain

Coreset

BALD

Openmax

CCAL

MQnet

LfOSA

PAL

1

3

5

7

9

Round

0.1

0.2

0.3

0.4

0.5

0.6

Selection Precision

Random

Uncertainty

Certain

Coreset

BALD

Openmax

CCAL

MQnet

LfOSA

PAL

1

3

5

7

9

Round

0.2

0.3

0.4

0.5

0.6

0.7

Selection Precision

Random

Uncertainty

Certain

Coreset

BALD

Openmax

CCAL

MQnet

LfOSA

PAL

1

3

5

7

9

Round

0.2

0.3

0.4

0.5

0.6

0.7

0.8

Selection Precision

Random

Uncertainty

Certain

Coreset

BALD

Openmax

CCAL

MQnet

LfOSA

PAL

(a) CIFAR-10 Precision

1

3

5

7

9

Round

0.2

0.3

0.4

0.5

0.6

0.7

0.8

Selection Precision

Random

Uncertainty

Certain

Coreset

BALD

Openmax

CCAL

MQnet

LfOSA

PAL

(b) CIFAR-100 Precision

1

3

5

7

9

Round

0.3

0.4

0.5

0.6

0.7

Selection Precision

Random

Uncertainty

Certain

Coreset

BALD

Openmax

CCAL

MQnet

LfOSA

PAL

(c) Tiny-Imagenet Precision

1

3

5

7

9

Round

0.1

0.2

0.3

0.4

0.5

0.6

Selection Recall

Random

Uncertainty

Certain

Coreset

BALD

Openmax

CCAL

MQnet

LfOSA

PAL

1

3

5

7

9

Round

0.2

0.3

0.4

0.5

0.6

Selection Recall

Random

Uncertainty

Certain

Coreset

BALD

Openmax

CCAL

MQnet

LfOSA

PAL

1

3

5

7

9

Round

0.15

0.20

0.25

0.30

0.35

0.40

Selection Recall

Random

Uncertainty

Certain

Coreset

BALD

Openmax

CCAL

MQnet

LfOSA

PAL

1

3

5

7

9

Round

0.1

0.2

0.3

0.4

0.5

Selection Recall

Random

Uncertainty

Certain

Coreset

BALD

Openmax

CCAL

MQnet

LfOSA

PAL

(a) CIFAR-10 Recall

1

3

5

7

9

Round

0.2

0.3

0.4

0.5

0.6

Selection Recall

Random

Uncertainty

Certain

Coreset

BALD

Openmax

CCAL

MQnet

LfOSA

PAL

(b) CIFAR-100 Recall

1

3

5

7

9

Round

0.10

0.15

0.20

0.25

0.30

Selection Recall

Random

Uncertainty

Certain

Coreset

BALD

Openmax

CCAL

MQnet

LfOSA

PAL

(c) Tiny-Imagenet Recall

4.4 Ablation Study

Each Component of PAL. To analyze the contribution of each component of PAL, we conduct ablation study with ID proportion as 20 % . The experimental results of classification accuracy are plotted in Figure 8, where w/o ω denotes the PAL with only the informativeness criterion, and w/o S ID denotes the PAL with only the representativeness criterion. The results indicate that the representativeness criterion has a greater impact on open-set AL in realistic scenarios, for the reason that the representativeness criterion can better distinguish the challenge ID and OOD instances considering the instances' relationships.

Hyper-Parameter µ . Considering that s ID and ω can capture different semantics (i.e., informativeness and representativeness) of unlabeled instances, we set µ with different values, i.e.,

{ 0 . 2 , 0 . 4 , 0 . 6 , 0 . 8 , 1 } , to empirically investigate the impact of s ID and ω . Table 2 shows the final round accuracy, precision, and recall on CIFAR-100 with ID proportion of 20 % . PAL achieves the best ID classification accuracy and OOD detection recall with µ = 0 . 8 , which indicates that PAL can leverage various semantic information of unlabeled instances and effectively improves ID selection by measuring instances' informativeness. Besides, representativeness can better ensure the true positivity of ID instances, allowing PAL to achieve superior precision at µ = 0 . 2 .

1

3

5

7

9

Round

0.88

0.90

0.92

0.94

0.96

0.98

w/o

w/o

S ID

PAL

(a) CIFAR-10

1

3

5

7

9

Round

0.45

0.50

0.55

0.60

0.65

0.70

w/o

w/o

S ID

PAL

(b) CIFAR-100

1

3

5

7

9

Round

0.25

0.30

0.35

0.40

0.45

0.50

w/o

w/o

S ID

PAL

(c) Tiny-Imagenet

Sampling Criteria. To evaluate the effectiveness of the proposed sampling criteria, we compare several classic metrics for evaluating representativeness and informativeness. Table 3 records the results on CIFAR-10 and CIFAR-100 datasets, the PAL_Core denotes replacing meta-weight with Coreset [28], which is a traditional metric method used for representativeness. PAL_MSP and PAL_energy methods denote replacing the proposed informativeness criteria with MSP [6] and energy-based [19] criteria respectively. The results exhibit that PAL achieves superior performance on two datasets, revealing the effectiveness of meta-weight in selecting instances.

5 Conclusion

In this paper, we have proposed PAL, a novel open-set AL method that employs a simple but effective progressive sampling to enhance the performance of OOD detection by actively querying OOD instances at the initial round. PAL included informativeness (i.e., predictions of OVA detector) and representativeness (i.e., automatically learned meta-weights) to measure the importance of unlabeled instances. A theoretical result from the perspective of learning theory revealed that the detected OOD instances enhance the OOD detector. The effectiveness of PAL was demonstrated with extensive empirical results across different scenarios, compared with the state-of-the-art methods. The limitation of the proposed PAL is only applicable to the classification task. In the future, we expect to extend PAL to other computer vision tasks such as object detection.

Broader Impact. PAL improves the efficacy of active learning in real-world scenarios. Similar to other active learning methods, it effectively reduces the annotation cost for model training, while the privacy protection issue associated with the practical use of AL on realistic datasets deserves more attention.

Acknowledgments

The authors thank the anonymous reviewers for their helpful comments. This work is partially supported by National Key RD Program of China (2022YFF0712100), NSFC (62006118, 62276131), Natural Science Foundation of Jiangsu Province of China under Grant (BK20200460), Jiangsu Shuangchuang (Mass Innovation and Entrepreneurship) Talent Program, Young Elite Scientists Sponsorship Program by CAST, the Fundamental Research Funds for the Central Universities (NO.NJ2022028, No.30922010317, DUT No. 82232031).

References

Jordan T. Ash et al. 'Deep Batch Active Learning by Diverse, Uncertain Gradient Lower Bounds'. In: ICLR . Addis Ababa, Ethiopia, 2020.

Abhijit Bendale and Terrance E. Boult. 'Towards Open Set Deep Networks'. In: CVPR . Las Vegas, NV, 2016, pp. 1563-1572.

Pan Du et al. 'Contrastive Coding for Active Learning under Class Distribution Mismatch'. In: ICCV . QC, Canada, 2021, pp. 8907-8916.

Yarin Gal, Riashat Islam, and Zoubin Ghahramani. 'Deep Bayesian Active Learning with Image Data'. In: ICML . Sydney, Australia, 2017, pp. 1183-1192.

Lan-Zhe Guo et al. 'Safe deep semi-supervised learning for unseen-class unlabeled data'. In: ICML . Virtual Event, 2020, pp. 3897-3906.

Dan Hendrycks and Kevin Gimpel. 'A Baseline for Detecting Misclassified and Out-ofDistribution Examplesin Neural Networks'. In: ICLR . Toulon, France, 2017.

Sheng-Jun Huang, Rong Jin, and Zhi-Hua Zhou. 'Active Learning by Querying Informative and Representative Examples'. In: NeurIPS . Vancouver, Canada, 2010, pp. 892-900.

Dihong Jiang, Sun Sun, and Yaoliang Yu. 'Revisiting flow generative models for Out-ofdistribution detection'. In: ICLR . Virtual Event, 2022.

Ajay J. Joshi, Fatih Porikli, and Nikolaos Papanikolopoulos. 'Multi-class active learning for image classification'. In: CVPR . Miami, Florida, 2009, pp. 2372-2379.

Sham M. Kakade, Karthik Sridharan, and Ambuj Tewari. 'On the Complexity of Linear Prediction: Risk Bounds, Margin Bounds, and Regularization'. In: NeurIPS . British Columbia, Canada, 2008, pp. 793-800.

Kwanyoung Kim et al. 'Task-Aware Variational Adversarial Active Learning'. In: CVPR . Virtual Event, 2021, pp. 8166-8175.

Suraj Kothawade et al. 'SIMILAR: Submodular Information Measures Based Active Learning In Realistic Scenarios'. In: NeurIPS . Virtual Event, 2021, pp. 18685-18697.

Alex Krizhevsky. 'Learning Multiple Layers of Features from Tiny Images'. In: Master's thesis (2009). URL: https://www.cs.toronto.edu/~kriz/cifar.html .

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. 'ImageNet Classification with Deep Convolutional Neural Networks'. In: NeurIPS . Lake Tahoe, Nevada, 2012, pp. 1106-1114.

David Krueger et al. 'Out-of-Distribution Generalization via Risk Extrapolation (REx)'. In: ICML . Virtual Event, pp. 5815-5826.

Yann LeCun, Yoshua Bengio, and Geoffrey E. Hinton. 'Deep learning'. In: Nature 521.7553 (2015), pp. 436-444.

David D. Lewis and William A. Gale. 'A Sequential Algorithm for Training Text Classifiers'. In: ACM-SIGIR . Dublin, Ireland, 1994, pp. 3-12.

Shiyu Liang, Yixuan Li, and R. Srikant. 'Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks'. In: ICLR . Vancouver, Canada, 2018.

Weitang Liu et al. 'Energy-based Out-of-distribution Detection'. In: NeurIPS . Virtual Event, 2020, pp. 21464-21475.

Wenjie Luo, Alexander G. Schwing, and Raquel Urtasun. 'Latent Structured Active Learning'. In: NeurIPS . Lake Tahoe, Nevada, 2013, pp. 728-736.

Laurens van der Maaten and Geoffrey E. Hinton. 'Visualizing Data using t-SNE'. In: Journal of Machine Learning Research 9.11 (2008), pp. 2579-2605.

Kun-Peng Ning et al. 'Active Learning for Open-set Annotation'. In: CVPR . New Orleans, LA, 2022, pp. 41-49.

Dongmin Park et al. 'Meta-Query-Net: Resolving Purity-Informativeness Dilemma in Open-set Active Learning'. In: NeurIPS . New Orleans, LA, 2022, pp. 31416-31429.

Nicholas Roy and Andrew McCallum. 'Toward Optimal Active Learning through Sampling Estimation of Error Reduction'. In: ICML . Williamstown, MA, 2001, pp. 441-448.

Kuniaki Saito, Donghyun Kim, and Kate Saenko. 'OpenMatch: Open-set Consistency Regularization for Semi-supervised Learning with Outliers'. In: NeurIPS . Virtual Event, 2021, pp. 25956-25967.

Kuniaki Saito and Kate Saenko. 'OVANet: One-vs-All Network for Universal Domain Adaptation'. In: ICCV . Montreal, Canada, 2021, pp. 8980-8989.

MSalehi et al. 'A Unified Survey on Anomaly, Novelty, Open-Set, and Out of-Distribution Detection: Solutions and Future Challenges'. In: Transactions on Machine Learning Research. 234 (2022).

Ozan Sener and Silvio Savarese. 'Active Learning for Convolutional Neural Networks: A Core-Set Approach'. In: ICLR . Vancouver, Canada, 2018.

Toan Tran et al. 'Bayesian Generative Active Deep Learning'. In: ICML . Long Beach, California, 2019, pp. 6295-6304.

Wenjuan Xi et al. 'Robust Semi-Supervised Learning for Self-learning Open-World Classes'. In: ICDM . Shanghai China, 2023.

Yang Yang, De-Chuan Zhan, and Yuan Jiang. 'Learning by Actively Querying Strong Modal Features'. In: IJCAI . New York, NY, 2016, pp. 2280-2286.

Yang Yang et al. 'Corporate relative valuation using heterogeneous multi-modal graph neural network'. In: IEEE Transactions on Knowledge and Data Engineering 35.1 (2021), pp. 211224.

Yang Yang et al. 'Learning Adaptive Embedding Considering Incremental Class'. In: IEEE Transactions on Knowledge and Data Engineering 35.3 (2023), pp. 2736-2749.

Yang Yang et al. 'S2OSC: A holistic semi-supervised approach for open set classification'. In: ACM Transactions on Knowledge Discovery from Data 16.2 (2021), pp. 1-27.

Yang Yang et al. 'Semi-supervised multi-modal multi-instance multi-label deep network with optimal transport'. In: IEEE Transactions on Knowledge and Data Engineering 33.2 (2019), pp. 696-709.

Leon Yao and John Miller. 'Tiny imagenet classification with convolutional neural networks'. In: CS 231N 2.5 (2015), p. 8.

Xinge You, Ruxin Wang, and Dacheng Tao. 'Diverse expected gradient active learning for relative attributes'. In: IEEE transactions on image processing 23.7 (2014), pp. 3203-3217.

Sergey Zagoruyko and Nikos Komodakis. 'Wide Residual Networks'. In: BMVC . York, UK, 2016.

Da-Wei Zhou, Yang Yang, and De-Chuan Zhan. 'Learning to classify with incremental new class'. In: IEEE Transactions on Neural Networks and Learning Systems 33.6 (2021), pp. 24292443.

The Supplementary of Not All Out-of-Distribution Data Are Harmful to Open-Set Active Learning

A Proof of Theorem 1

Firstly, we prove the generalization error of the proposed PAL method. Following Theorem 1 of [10], we have

where R n ( F ) is the Rademacher complexity of a function class F and n is the number of training samples. With the Rademacher complexity R n ( F ) ≤ √ C ( F ) n , we give equation 5.

Next, we prove the generalization error of the standard AL method. To this end, we denote by P = P ( x | y = 0) the conditional probability for ID data and P 1 = P ( x | y = 1) the conditional probability for OOD data. Let L j ( f ) denote the loss from class j ∈ { 0 , 1 } : L j ( f ) = E P j [ ℓ ( f ( x ) , y )] , and let ̂ L j ( f ) denote its corresponding empirical loss. Then by applying the standard analysis for each class j in Theorem 1 of [10], with probability 1 -δ/ 2 we have

Since the distribution of ID and OOD samples is balanced, we have L ( f AL ) = 1 2 L 0 ( f AL )+ 1 2 L 1 ( f AL ) due to the definitions of the loss functions. Similarly, due to the number of OOD samples being the same as the number of ID samples in the training set, we know ̂ L ( f AL ) = 1 2 ̂ L 0 ( f AL ) + 1 2 ̂ L 1 ( f AL ) . Finally, by applying the union bound, we have

With the Rademacher complexity R m j ( F ) ≤ √ C ( F ) m j for j = 0 , 1 , we complete the proof of equation 6 by plugging in m 0 = αm and m 1 = (1 -α ) m .

B Additional Implementation Details

Implementations. In the first round, to enhance the effectiveness of OOD detection, we set N ID query = 300 and N OOD query = 1200 . To reduce the influence of the pseudo-OOD budget for the ID classifier, we set N ID query = 1450 and N OOD query = 50 in the later rounds.

Description for Datasets. In Section 4, we provide a brief introduction to the construction of the open-set AL scenario. In order to provide a more detailed explanation of the dataset construction, we use CIFAR-100 with the ID proportion of 20 % as a specific example. We consider the first 20 classes

as the ID classes. Then, we randomly sample 50 instances from each ID class to form the labeled set D l , while the remaining instances are unlabeled data D u . As a result, the labeled and unlabeled sets comprise 1,000 and 49,000 instances respectively, as shown in Table 4.

Description for baselines. Here's a brief introduction to the comparison methods in the experiment.

Random introduced randomly selecting instances from the unlabeled pool for labeling.

Uncertainty [17] employed entropy to evaluate the uncertainty of instances.

Certainty [20] considered utilizing the weakly labeled data to query instances based on entropy of local marginals.

Coreset [28] proposed core-set selection with greedy k-center clustering to minimize the maximum distance between labeled data and unlabeled data.

BALD [29] employed dropout as an approximation for Bayesian inference to facilitate active sampling.

Openmax [2] introduced the activation vector to measure whether a instance belongs to the known classes.

CCAL [3] introduced a framework of contrastive active learning, with two contrastive coding models employed to assess the informativeness and representativeness of each example.

MQnet [23] utilized meta-learning techniques to discover an optimal balance between purity and informativeness.

LfOSA [22] leveraged an auxiliary network equipped with a gaussian mixture model to capture the distribution of maximum activation values for each instance, enabling adaptive selection of high-probability instances from known classes.

C Additional Experimental Results

Impact of the annotated OOD_ID data. Actually, the method of only adding additional data into OOD Detector, leading the annotated OOD_ID data (the annotated ID data after selection of the detected OOD data) not used in ID classifier of PAL. In this section, we aim to empirically investigate the impact of annotated OOD_ID data. To this end, we modify the proposed PAL by adding the annotated OOD_ID data into ID classifier and naming it as PAL+, which is illustrated in Figure 9.

OOD

Detector

Unlabeled

Data

Selection

Selection

ID Classifier

Annotation

Annotation

OOD

data?

Yes

No

Traditional

open-set AL

Considering experimental completeness, we present the classification and OOD detection performance without and with using the OOD_ID instances (i.e. PAL and PAL+), respectively. Note that the experiments are performed on three datasets with an ID proportion of 20 % . The results in Figure 10 demonstrate that the proposed PAL and PAL+ are comparable, showing that the annotated OOD_ID instances have minimal impact on the overall performance of the model.

In addition, Table 5 lists the number of annotated OOD_ID instances with their proportion in the detected OOD data, for round 1, 3, 5, 7, and 9 on three data sets. The results in Table 5 show that the annotated OOD_ID instances have a very small proportion in the detected OOD instances, and this may also explain why the PAL and PAL+ are comparable.

Round

Seed

Dataset

Round

1

3

5

7

9

seed=1

CIFAR-10

5 (10

%

)

1 (2

%

)

0 (0

%

)

0 (0

%

)

1 (2

%

)

CIFAR-100

5 (10

%

)

5 (10

%

)

1 (2

%

)

0 (0

%

)

0 (0

%

)

Tiny-Imagenet

3 (6

%

)

3 (6

%

)

2 (4

%

)

1 (2

%

)

0 (0

%

)

seed=2

CIFAR-10

3 (6

%

)

0 (0

%

)

0 (0

%

)

0 (0

%

)

0 (0

%

)

CIFAR-100

4 (8

%

)

2 (4

%

)

1 (2

%

)

1 (2

%

)

0 (0

%

)

Tiny-Imagenet

6 (12

%

)

3 (6

%

)

0 (0

%

)

0 (0

%

)

1 (2

%

)

seed=3

CIFAR-10

1 (2

%

)

0 (0

%

)

0 (0

%

)

0 (0

%

)

0 (0

%

)

CIFAR-100

1 (2

%

)

2 (4

%

)

1 (2

%

)

1 (2

%

)

0 (0

%

)

Tiny-Imagenet

2 (4

%

)

3 (6

%

)

2 (4

%

)

0 (0

%

)

2 (4

%

)

1

3

5

7

9

Round

0.90

0.92

0.94

0.96

0.98

PAL

PAL+

1

3

5

7

9

Round

0.20

0.30

0.40

0.50

0.60

0.70

0.80

0.90

ID Selection Recall

PAL

PAL+

1

3

5

7

9

Round

0.45

0.50

0.55

0.60

0.65

0.70

PAL

PAL+

1

3

5

7

9

Round

0.20

0.30

0.40

0.50

0.60

0.70

ID Selection Recall

PAL

PAL+

1

3

5

7

9

0.25

0.30

0.35

0.40

0.45

0.50

0.55

PAL

PAL+

1

3

5

7

9

0.15

0.20

0.25

0.30

0.35

0.40

0.45

ID Selection Recall

PAL

PAL+

Round

Visualization. To demonstrate the distribution of selected ID instances, we provide visual results of the iterative query round. In detail, we visualize the feature representations of the labeled and queried correct ID instances via t-SNE [21] on the CIFAR-10 dataset with an ID proportion of 40 % . Note that green crosses represent the queried ID instances. Figure 11 records the visualizations of PAL and state-of-the-art methods, i.e., Coreset and LfOSA. The figures reveal that PAL selects more ID instances than other methods, i.e., higher purity in different rounds. Moreover, from the perspective of data distribution, PAL would balance the informativeness and representativeness criteria to select meaningful ID instances for ID classifier training.

．

臣

｀

，

一

在

十

＼补＋许

．

Airplane

玉翠免

�

Au

t

omobiIe

心热书社＋，血

■

Bird

为求 蹈

．

Ca

t

咕霓嗓啪冲

•

AL_sampIe

沿气＋尽审气`痒i,占却吨咄磕酴

亚毯§..,

吩崎翱｛

予

．

A

q

.

f

宁

上节

岔

坠

氛

µ

子

Coreset

LfOSA

PAL

Airplane

Automobile

Bird

Cat

AL_sample

嘈

Compare with Open-Set Semi-Supervised Learning. Although both open-set/safe semi-supervised learning and open-set AL aim to construct a more effective ID classifier by reasonably utilizing unlabeled data, the essential difference lies in whether or not manual participation is necessary. Open-set/Safe semi-supervised learning aims to use all the pseudo-ID instances in unlabeled data according to the model's prediction, whereas open-set AL aims to select the most valuable pseudo-ID instances in unlabeled data for manual labeling to retrain the model without using all the unlabeled data. To evaluate the classification performance of PAL and open-set semi-supervised learning methods, we have compared PAL with two open-set semi-supervised learning methods, DS 3 L [5] and OpenMatch [25] using the CIFAR-10 and CIFAR-100 datasets with an ID proportion of 20%. As illustrated in Table 6, the results show the proposed PAL also performs better than the open-set semi-supervised learning methods.

Peer reviews:
comments: SUMMARY:
This paper proposes an active learning approach for open-set learning. The proposed approach does not try to avoid sampling OOD data as some of the prior work did. Instead, it samples some of the OOD data intentionally to enhance the OOD detector.

STRENGTHS:
1.The proposed idea is easy to implement and understand.
2.The proposed problem, open-set active learning, is meaningful to me.

WEAKNESSES:
1.The overall framework is trivial to me. It is a traditional active learning method adapted to an OOD detector and an ID learner. 
2.The essential part of active learning, which is the design of the sampling criterion, is neither novel nor plausible to me. For the uncertainty weight, it is merely the prediction score of the model proposed by [27]. What is more, this score tends to approximate the likelihood of a data point being an in-distribution sample. But I do not think this score reveals the informativeness/ sample importance in active learning. Different data points can have very different contributions to learning, the proposed sampling score does not seem to have the ability to differentiate them. The meta-weight can be seen as a loss-based active sampling criterion for an OOD detector, which is trivial to me. 
3.All the proposed methods are empirical, there is no quantitative analysis to support the effectiveness of the proposed methods.

QUESTIONS:
None

LIMITATIONS:
None
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-06-23

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis manuscript studies open-set active learning and points out that concentrating solely on selecting pseudo-ID instances may cause the training imbalance of the ID classifier and OOD detector. To address this issue, this manuscript proposes a simple yet effective sampling scheme, dubbed Progressive Active Learning (PAL). Extensive experiments on various open-set AL scenarios demonstrate the effectiveness of PAL.\n\nSTRENGTHS:\n1.This manuscript points out that concentrating solely on selecting pseudo-ID instances may cause the training imbalance of the ID classifier and OOD detector and proposes Progressive Active Learning (PAL) to solve the problem.\n\n2.Extensive experiments on various open-set AL scenarios demonstrate the effectiveness of PAL.\n\nWEAKNESSES:\n1.The manuscript lacks corresponding theoretical analysis.\n\n2.The manuscript lacks an introduction to related work on open-set semi-supervised learning and safe semi-supervised learning, such as [1], [2], and [3].\n\n3.In line 39-41, “the main challenges in open-set AL revolve around effectively selecting valuable ID instances for classifier training and distinguishing the OOD instances”, Similar viewpoints have already been proposed by [2]. \n\n4.The framework in Fig. 2 is confusing and the flow description is not clear enough. For example, how is f^c trained?\n\n5.In every query, how are N^{OOD}_{query} and N^{ID}_{query} set up?\n\n6.The description of the retraining process is not clear. What is the final loss? Is f^c used for the final testing?\n\n7.In line 136, entropy is used for obtaining s^{ID}. Has the author tried other metrics such as MSP in [4] or energy in [5]?\n\n8.The manuscript lacks important baselines, such as [1], [2], [3]. The manuscript needs to be compared with open-set semi-supervised learning and safe semi-supervised learning methods. \n\n9.The manuscript mentions the need for multiple rounds of query and training, so the runtime of the manuscript needs to be compared with the baseline.\n\n[1] Safe deep semi-supervised learning for unseen-class unlabeled data.\n[2] Safe-Student for Safe Deep Semi-Supervised Learning with Unseen-Class Unlabeled Data.\n[3] Openmatch: Open-set semi-supervised learning with open-set consistency regularization.\n[4] A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks.\n[5] Energy-based out-of-distribution detection.\n\nQUESTIONS:\nHow is N^{ID}_{query} selected? Is the sample with maximum of un(x^u_j) chosen into N^{ID}_{query}? The sample with maximum of un(x^u_j) indicates strong certainty in these samples. Perhaps using model-generated pseudo-labels would be accurate enough, without the need to waste resources on querying these relatively certain samples.\n\nLIMITATIONS:\nPlease see Weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers the open-set active learning problem, a sub-topic of active learning that focuses on non-iid settings. The authors constructed both an ID classifier and an OOD detector to implement open-set active learning. Specifically, they proposed a sampling scheme to ensure a balance of ID and OOD samples, which helps both the ID classifier and the OOD detector. The authors conducted a series of experiments, and the results of these experiments verified the effectiveness of the proposed method.\n\nSTRENGTHS:\n1. The considered problem is important and highly relevant to the machine learning community. \n2. The proposal is simple and reasonable. The authors suggest that out-of-distribution (OOD) samples can also help the process of open-set active learning from the perspective of OOD detection. They use a simple and effective balanced sampling approach to alleviate the learning process differences and improve the performance of open-set active learning. This proposal suggests that actively annotating some OOD samples can enhance the OOD detector and help open-set active learning. This supports the claim of the title \"Not All Out-of-Distribution Data Are Harmful\" and extends the previous focus on mainly querying ID samples.\n3. The experiments are sufficient, and the effectiveness of the proposal has been clearly verified.\n\nWEAKNESSES:\nCCAL is also a well-known open-set active learning method, which should be discussed and compared in the experiments.  \n[CCAL] Contrastive active learning under class distribution mismatch. TPAMI’22  \n\nIt is regretful that there is no relevant theoretical analysis to further support this work.   \n\nMiscellaneous:  \nThe results in Table 1 are too dense. Consider adjusting the layout.\nAlso, consider using the same text font in Figures 3, 4, 5, 6, and 7 as in the main body.\n\nQUESTIONS:\nThe proposal uses meta-learning to calculate meta-weights for obtaining representative metrics of the samples. Compared to previous methods such as clustering and coreset, does this complex method introduce too much computational overhead? Does it have any special advantages? In the experiment, can we obtain good results using traditional representative metric methods?\n\n--- \nThanks for the clarifications. Most of the concerns have been addressed. I will keep my score.\n\nLIMITATIONS:\nThe authors have provided a discussion on the broader impact.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors aims to improve open-set active learning, where unlabelled set may contains some open-set instances. To do this, they propose a new sampling scheme, called progressive active learning. Specifically, they use a progressive sampling method to select valuable OOD data for the tradeoff between pseudo-ID and pseudo-OOD instances in each round. Experiments on various datasets are conducted to show the effectiveness of the proposed method.\n\nSTRENGTHS:\n1. This paper is well-writing and easy-to-understand. I believe readers can easily get the core idea of the proposed method.\n\n2. The empirical performance is non-trivial. From Table 1, we can observe the improvement of the proposed method is significant.\n\n3. The motivation is clear. I agree that OOD data should not be simply filtered out for active learning.\n\nWEAKNESSES:\n1. The method is somewhat heuristic and not novel enough. In my view, both OVA classifier and meta-weight are not original contribution of this paper. The technical novelty of the progressive sampling is not sufficient to support the acceptance in this conference.\n\n2. The literature of leveraging OOD data to improve generalisation is missing, like OAT [1], ODNL [2] and Open-sampling [3]. The authors may need to discuss the relationship between the proposed method and these works.\n\n\n\n[1] Lee, Saehyung, et al. \"Removing Undesirable Feature Contributions Using Out-of-Distribution Data.\", ICLR, 2021.\n\n[2] Wei, et al. Open-set Label Noise Can Improve Robustness Against Inherent Label Noise. NeurIPS 2021\n\n[3] Wei, et al. Open-Sampling: Exploring Out-of-Distribution data for Re-balancing Long-tailed datasets. ICML 2022.\n\n---------\n\nAfter reading the responses from the authors, my concerns have been well addressed, so I lean towards acceptance for this paper.\n\nQUESTIONS:\nDo you mean the positive effect of OOD data is only in detecting OOD examples? You write \"improving the ID purity in query sets\" in line 66.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}