SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Parameter setting. For all AL methods, following [22], we randomly sample  $1\%$ ,  $10\%$  and  $10\%$  of the examples as the initial labeled set on CIFAR-10, CIFAR-100, and Tiny-Imagenet datasets, respectively. To ensure fairness, we employ WideResNet [38] as the backbone for training all methods. Note that the labeled data only contains ID classes. In each AL round, we train the model for 100 epochs, using the SGD optimizer with the momentum parameter of 0.9. The learning rate is initialized as 0.01 with a mini-batch size of 128, and the weight decay is set to be  $5 \times 10^{-4}$ . Additionally, the annotation budget per query round is limited to 1500 following [22]. All experiments are implemented on a single NVIDIA V100 GPU.
Table 1 and Figure 3 record the ID classification and OOD detection results of PAL and other AL methods with the increasing number of queries, where the index  $\{1,3,5,7,9\}$  denotes the query round.
ID classification. The results in Table 1 show that: 1) AL methods perform better than the supervised method because the oracle will filter the OOD instances in the query set. However, the promotion is limited due to decreased ID instances, especially in complex datasets such as CIFAR-100 and Tiny-Imagenet. 2) CCAL, LfOSA, and MQnet outperform traditional active learning methods because LfOSA selects a more purified set for querying, including more ID instances and fewer OOD instances. LfOSA performs better than MQnet because MQnet maintains a balance between purity and informativeness of the query set that contains several OOD. 3) PAL consistently exhibits the best performance in each round on all datasets, especially on the Tiny-Imagenet dataset. Compared to other open-set AL methods, PAL improvements about  $7.5\%$ ,  $3.9\%$ , and  $4.2\%$  under the 5, 7, and 9 rounds respectively, indicating that PAL could better control the purity ID by constructing a more robust OOD detector.
(a) CIFAR-10 Precision
(d) CIFAR-10 Recall
Figure 3: The precision and recall comparisons of OOD detection on CIFAR-10, CIFAR-100, and Tiny-Imagenet datasets with ID proportion of  $20\%$ .
(b) CIFAR-100 Precision
(e) CIFAR-100 Recall
(c) Tiny-Imagenet Precision
(f) Tiny-Imagenet Recall
OOD detection. The results of Figure 3 reveal that PAL consistently outperforms other baselines in terms of most selection precision and recall. For example, the proposed PAL achieves  $15.9\%$  higher precision than LfOSA on Tiny-Imagenet dataset in the last round, which validates that PAL can better eliminate the OOD instances by adopting the discriminative criteria and progressive sampling. Moreover, note that the precision of open-set AL methods shows an initial increase followed by a subsequent decrease, for the reason that we are taking an unreplaced query, so the remaining unlabeled data is gradually reduced. Meanwhile, PAL can select more ID instances at the starting several rounds on CIFAR-10, resulting in fewer ID instances left in the unlabeled set considering the relatively smaller dataset size of CIFAR-10. Thereby, PAL achieves inferior precision compared to LfOSA in later rounds.
To further visualize the validity of pseudo-ID and pseudo-OOD instances, we compare the query purity of PAL with LfOSA in query rounds on three datasets, as illustrated in Figure 4. Although we actively select more OOD instances in the first round, the overall OOD instances are much fewer than LfOSA. Specifically, under the CIFAR-10, LfOSA selected  $52.2\%$  OOD instances, whereas PAL only selected  $40.3\%$ . Besides, we find that the purities of pseudo-ID and pseudo-OOD are better than LfOSA, i.e., PAL achieves an overall  $12.6\%$  higher query purity compared to LfOSA on the Tiny-Imagenet dataset, which validates the effectiveness of progressive building OOD detector.
(b) CIFAR-100
Figure 4: The query purity comparison between LfOSA and PAL. Query purity denotes the ratio of ID_ID instances to the total number of the annotation budget. ID_ID denotes the true positive in the pseudo-ID instances, while ID_OOD represents the false positive in the pseudo-ID instances. Similarly, OOD_ID denotes the false negative in the pseudo-OOD instances, while OOD_OOD represents the true negative in the pseudo-OOD instances. Note that LfOSA only queries the pseudo-ID instances.
(c) Tiny-Imagenet
4.3 Influence of Different ID Class Proportions
In this subsection, we evaluate the influence of different numbers of ID classes. The average results of accuracy, precision, and recall are presented in Figure 5, 6, and 7 respectively. In each figure, the first, second, and third columns represent the results on CIFAR-10, CIFAR-100, and Tiny-Imagenet, respectively, while the first and second rows represent the results with ID proportion as  $30\%$  and  $40\%$ . Please note that the results of  $20\%$  are included in Table 1 and Figure 3. The results exhibit that the proposed PAL always outperforms other methods in terms of ID classification and OOD detection for all cases. For example, when ID proportion is set to  $30\%$ , PAL outperforms  $3.6\%$ ,  $3.3\%$ , and  $4.3\%$  than the state-of-the-art method, i.e., LfOSA, in terms of accuracy, precision, and recall in the last round on CIFAR-100.
(a) CIFAR-10 Accuracy
Figure 5: Comparison of testing accuracy(%) on CIFAR-10 (first column), CIFAR-100 (second column), and Tiny-Imagenet (third column) datasets, with an ID proportion of  $30\%$  (first row) and  $40\%$  (second row).
(b) CIFAR-100 Accuracy
(c) Tiny-Imagenet Accuracy
(a) CIFAR-10 Precision
(b) CIFAR-100 Precision
(c) Tiny-Imagenet Precision
Figure 6: The precision comparison of OOD detection on CIFAR-10 (first column), CIFAR-100 (second column), and Tiny-Imagenet (third column) datasets, with an ID proportion of  $30\%$  (first row) and  $40\%$  (second row).
(a) CIFAR-10 Recall
(b) CIFAR-100 Recall
(c) Tiny-Imagenet Recall
Figure 7: The recall comparison of OOD detection on CIFAR-10 (first column), CIFAR-100 (second column) and Tiny-Imagenet (third column) datasets with ID proportion as  $30\%$  (first row) and  $40\%$  (second row).
Each Component of PAL. To analyze the contribution of each component of PAL, we conduct ablation study with ID proportion as  $20\%$ . The experimental results of classification accuracy are plotted in Figure 8, where w/o  $\omega$  denotes the PAL with only the informativeness criterion, and w/o  $S^{ID}$  denotes the PAL with only the representativeness criterion. The results indicate that the representativeness criterion has a greater impact on open-set AL in realistic scenarios, for the reason that the representativeness criterion can better distinguish the challenge ID and OOD instances considering the instances' relationships.
Hyper-Parameter  $\mu$ . Considering that  $s^{ID}$  and  $\omega$  can capture different semantics (i.e., informativeness and representativeness) of unlabeled instances, we set  $\mu$  with different values, i.e.,
$\{0.2, 0.4, 0.6, 0.8, 1\}$ , to empirically investigate the impact of  $s^{ID}$  and  $\omega$ . Table 2 shows the final round accuracy, precision, and recall on CIFAR-100 with ID proportion of  $20\%$ . PAL achieves the best ID classification accuracy and OOD detection recall with  $\mu = 0.8$ , which indicates that PAL can leverage various semantic information of unlabeled instances and effectively improves ID selection by measuring instances' informativeness. Besides, representativeness can better ensure the true positivity of ID instances, allowing PAL to achieve superior precision at  $\mu = 0.2$ .
(b) CIFAR-100
Figure 8: Ablation study conducted on CIFAR-10, CIFAR-100, and Tiny-Imagenet datasets with an ID proportion of  $20\%$ .
Table 2: Comparison of accuracy, precision, and recall across different values of  $\mu$  in CIFAR-100, with an ID proportion of  ${20}\%$  .
μ	0.2	0.4	0.6	0.8	1
Accuracy (%)	66.2	67.2	68.5	69.4	67.6
Precision (%)	32.4	27.3	25.6	25.4	17.6
Recall (%)	67.0	69.2	71.0	73.8	69.7
Sampling Criteria. To evaluate the effectiveness of the proposed sampling criteria, we compare several classic metrics for evaluating representativeness and informativeness. Table 3 records the results on CIFAR-10 and CIFAR-100 datasets, the PAL-Core denotes replacing meta-weight with Coreset [28], which is a traditional metric method used for representativeness. PAL_MSP and PAL_energy methods denote replacing the proposed informativeness criteria with MSP [6] and energy-based [19] criteria respectively. The results exhibit that PAL achieves superior performance on two datasets, revealing the effectiveness of meta-weight in selecting instances.
Table 3: Comparison of testing accuracy (%) for PAL-Core, PAL_MSP, PAL_energy, and PAL on CIFAR-10 and CIFAR-100 with an ID proportion of  $20\%$ .
Datasets Rounds	CIFAR-10	CIFAR-100
1	3	5	7	9	1	3	5	7	9
PAL_Neutral	87.1	93.3	96.7	97.4	98.3	44.8	49.7	59.6	63.7	65.9
PAL_MSP	86.0	95.1	95.2	97.5	98.0	45.1	48.4	55.8	60.1	64.7
PAL_Energy	85.6	93.7	94.7	97.0	97.9	45.32	52.5	58.6	64.5	66.2
PAL	91.1	95.6	97.6	98.5	98.7	45.6	53.0	60.0	65.6	69.4
In this paper, we have proposed PAL, a novel open-set AL method that employs a simple but effective progressive sampling to enhance the performance of OOD detection by actively querying OOD instances at the initial round. PAL included informativeness (i.e., predictions of OVA detector) and representativeness (i.e., automatically learned meta-weights) to measure the importance of unlabeled instances. A theoretical result from the perspective of learning theory revealed that the detected OOD instances enhance the OOD detector. The effectiveness of PAL was demonstrated with extensive empirical results across different scenarios, compared with the state-of-the-art methods. The limitation of the proposed PAL is only applicable to the classification task. In the future, we expect to extend PAL to other computer vision tasks such as object detection.
Broader Impact. PAL improves the efficacy of active learning in real-world scenarios. Similar to other active learning methods, it effectively reduces the annotation cost for model training, while the privacy protection issue associated with the practical use of AL on realistic datasets deserves more attention.
The authors thank the anonymous reviewers for their helpful comments. This work is partially supported by National Key RD Program of China (2022YFF0712100), NSFC (62006118, 62276131), Natural Science Foundation of Jiangsu Province of China under Grant (BK20200460), Jiangsu Shuangchuang (Mass Innovation and Entrepreneurship) Talent Program, Young Elite Scientists Sponsorship Program by CAST, the Fundamental Research Funds for the Central Universities (NO.NJ2022028, No.30922010317, DUT No. 82232031).
[23] Dongmin Park et al. "Meta-Query-Net: Resolving Purity-Informativeness Dilemma in Open-set Active Learning". In: NeurIPS. New Orleans, LA, 2022, pp. 31416-31429.
[24] Nicholas Roy and Andrew McCallum. "Toward Optimal Active Learning through Sampling Estimation of Error Reduction". In: ICML. Williamstown, MA, 2001, pp. 441-448.
[25] Kuniaki Saito, Donghyun Kim, and Kate Saenko. "OpenMatch: Open-set Consistency Regularization for Semi-supervised Learning with Outliers". In: NeurIPS. Virtual Event, 2021, pp. 25956-25967.
[26] Kuniaki Saito and Kate Saenko. "OVANet: One-vs-All Network for Universal Domain Adaptation". In: ICCV. Montreal, Canada, 2021, pp. 8980-8989.
[27] M Salehi et al. “A Unified Survey on Anomaly, Novelty, Open-Set, and Out of-Distribution Detection: Solutions and Future Challenges”. In: Transactions on Machine Learning Research. 234 (2022).
[28] Ozan Sener and Silvio Savarese. "Active Learning for Convolutional Neural Networks: A Core-Set Approach". In: ICLR. Vancouver, Canada, 2018.
[29] Toan Tran et al. "Bayesian Generative Active Deep Learning". In: ICML. Long Beach, California, 2019, pp. 6295-6304.
[30] Wenjuan Xi et al. "Robust Semi-Supervised Learning for Self-learning Open-World Classes". In: ICDM. Shanghai China, 2023.
[31] Yang Yang, De-Chuan Zhan, and Yuan Jiang. "Learning by Actively Querying Strong Modal Features". In: *IJCAI*. New York, NY, 2016, pp. 2280-2286.
[32] Yang Yang et al. "Corporate relative valuation using heterogeneous multi-modal graph neural network". In: IEEE Transactions on Knowledge and Data Engineering 35.1 (2021), pp. 211-224.
[33] Yang Yang et al. "Learning Adaptive Embedding Considering Incremental Class". In: IEEE Transactions on Knowledge and Data Engineering 35.3 (2023), pp. 2736-2749.
[34] Yang Yang et al. "S2OSC: A holistic semi-supervised approach for open set classification". In: ACM Transactions on Knowledge Discovery from Data 16.2 (2021), pp. 1-27.
[35] Yang Yang et al. "Semi-supervised multi-modal multi-instance multi-label deep network with optimal transport". In: IEEE Transactions on Knowledge and Data Engineering 33.2 (2019), pp. 696-709.
[36] Leon Yao and John Miller. "Tiny imagenet classification with convolutional neural networks". In: CS 231N 2.5 (2015), p. 8.
[37] Xinge You, Ruxin Wang, and Dacheng Tao. "Diverse expected gradient active learning for relative attributes". In: IEEE transactions on image processing 23.7 (2014), pp. 3203-3217.
[38] Sergey Zagoruyko and Nikos Komodakis. "Wide Residual Networks". In: BMVC. York, UK, 2016.
[39] Da-Wei Zhou, Yang Yang, and De-Chuan Zhan. “Learning to classify with incremental new class”. In: IEEE Transactions on Neural Networks and Learning Systems 33.6 (2021), pp. 2429–2443.
The Supplementary of Not All Out-of-Distribution Data Are Harmful to Open-Set Active Learning
Firstly, we prove the generalization error of the proposed PAL method. Following Theorem 1 of [10], we have
$$
\mathcal {L} \left(f _ {P A L}\right) - \widehat {\mathcal {L}} \left(f _ {P A L}\right) \leq 2 L \Re_ {n} (\mathcal {F}) + c \sqrt {\frac {\log (1 / \delta)}{2 n}}, \tag {7}
$$
where  $\Re_n(\mathcal{F})$  is the Rademacher complexity of a function class  $\mathcal{F}$  and  $n$  is the number of training samples. With the Rademacher complexity  $\Re_{\mathfrak{n}}(\mathcal{F})\leq \sqrt{\frac{C(\mathcal{F})}{n}}$ , we give equation 5.
Next, we prove the generalization error of the standard AL method. To this end, we denote by  $\mathcal{P} = \mathcal{P}(\mathbf{x}|y = 0)$  the conditional probability for ID data and  $\mathcal{P}_1 = \mathcal{P}(\mathbf{x}|y = 1)$  the conditional probability for OOD data. Let  $\mathcal{L}_j(f)$  denote the loss from class  $j\in \{0,1\}$ :  $\mathcal{L}_j(f) = \mathbb{E}_{\mathcal{P}_j}[\ell (f(\mathbf{x}),y)]$  and let  $\widehat{\mathcal{L}}_j(f)$  denote its corresponding empirical loss. Then by applying the standard analysis for each class  $j$  in Theorem 1 of [10], with probability  $1 - \delta /2$  we have
$$
\mathcal {L} _ {j} \left(f _ {A L}\right) - \widehat {\mathcal {L}} _ {j} \left(f _ {A L}\right) \leq 2 L \Re_ {\mathrm {m} _ {j}} (\mathcal {F}) + c \sqrt {\frac {\log (2 / \delta)}{2 m _ {j}}}, \quad (j = 0, 1). \tag {8}
$$
Since the distribution of ID and OOD samples is balanced, we have  $\mathcal{L}(f_{AL}) = \frac{1}{2}\mathcal{L}_0(f_{AL}) + \frac{1}{2}\mathcal{L}_1(f_{AL})$  due to the definitions of the loss functions. Similarly, due to the number of OOD samples being the same as the number of ID samples in the training set, we know  $\widehat{\mathcal{L}}(f_{AL}) = \frac{1}{2}\widehat{\mathcal{L}}_0(f_{AL}) + \frac{1}{2}\widehat{\mathcal{L}}_1(f_{AL})$ . Finally, by applying the union bound, we have
$$
\mathcal {L} \left(f _ {A L}\right) - \widehat {\mathcal {L}} \left(f _ {A L}\right) \leq L \Re_ {\mathrm {m} _ {\mathrm {o}}} (\mathcal {F}) + L \Re_ {\mathrm {m} _ {1}} (\mathcal {F}) + \frac {c}{2} \sqrt {\frac {\log (2 / \delta)}{2 m _ {0}}} + \frac {c}{2} \sqrt {\frac {\log (2 / \delta)}{2 m _ {1}}}. \tag {9}
$$
With the Rademacher complexity  $\Re_{m_j}(\mathcal{F})\leq \sqrt{\frac{C(\mathcal{F})}{m_j}}$  for  $j = 0,1$ , we complete the proof of equation 6 by plugging in  $m_0 = \alpha m$  and  $m_{1} = (1 - \alpha)m$ .
Implementations. In the first round, to enhance the effectiveness of OOD detection, we set  $N_{query}^{ID} = 300$  and  $N_{query}^{OOD} = 1200$ . To reduce the influence of the pseudo-OD budget for the ID classifier, we set  $N_{query}^{ID} = 1450$  and  $N_{query}^{OOD} = 50$  in the later rounds.
Table 4: Dataset Construction.
Dataset	Number of ID classes	ID : OOD Ratio	Number of labeled data	Number of unlabeled data	Query size b
CIFAR-10 (20%)	2	2 : 8	100	49900	1500
CIFAR-10 (30%)	3	3 : 7	150	49850	1500
CIFAR-10 (40%)	4	4 : 6	200	49800	1500
CIFAR-100 (20%)	20	20 : 80	1000	49000	1500
CIFAR-100 (30%)	30	30 : 70	1500	48500	1500
CIFAR-100 (40%)	40	40 : 60	2000	48000	1500
Tiny-Imagenet (20%)	40	40 : 160	2000	48000	1500
Tiny-Imagenet (30%)	60	60 : 140	3000	47000	1500
Tiny-Imagenet (40%)	80	80 : 120	4000	46000	1500
Description for Datasets. In Section 4, we provide a brief introduction to the construction of the open-set AL scenario. In order to provide a more detailed explanation of the dataset construction, we use CIFAR-100 with the ID proportion of  $20\%$  as a specific example. We consider the first 20 classes
as the ID classes. Then, we randomly sample 50 instances from each ID class to form the labeled set  $\mathcal{D}_l$ , while the remaining instances are unlabeled data  $\mathcal{D}_u$ . As a result, the labeled and unlabeled sets comprise 1,000 and 49,000 instances respectively, as shown in Table 4.
Description for baselines. Here's a brief introduction to the comparison methods in the experiment.
- Random introduced randomly selecting instances from the unlabeled pool for labeling.
- Uncertainty [17] employed entropy to evaluate the uncertainty of instances.
- Certainty [20] considered utilizing the weakly labeled data to query instances based on entropy of local marginals.
- Coreset [28] proposed core-set selection with greedy k-center clustering to minimize the maximum distance between labeled data and unlabeled data.
- BALD [29] employed dropout as an approximation for Bayesian inference to facilitate active sampling.
- Openmax [2] introduced the activation vector to measure whether a instance belongs to the known classes.
- CCAL [3] introduced a framework of contrastive active learning, with two contrastive coding models employed to assess the informativeness and representativeness of each example.
- MQnet [23] utilized meta-learning techniques to discover an optimal balance between purity and informativeness.
- LfOSA [22] leveraged an auxiliary network equipped with a gaussian mixture model to capture the distribution of maximum activation values for each instance, enabling adaptive selection of high-probability instances from known classes.
Impact of the annotated OOD_ID data. Actually, the method of only adding additional data into OOD Detector, leading the annotated OOD_ID data (the annotated ID data after selection of the detected OOD data) not used in ID classifier of PAL. In this section, we aim to empirically investigate the impact of annotated OOD_ID data. To this end, we modify the proposed PAL by adding the annotated OOD_ID data into ID classifier and naming it as  $\mathrm{PAL}+$ , which is illustrated in Figure 9.
Figure 9: PAL+: PAL with the annotated OOD_ID data added to ID classifier (red dash line).
Considering experimental completeness, we present the classification and OOD detection performance without and with using the OOD_ID instances (i.e. PAL and  $\mathrm{PAL + }$ ), respectively. Note that the experiments are performed on three datasets with an ID proportion of  $20\%$ . The results in Figure 10 demonstrate that the proposed PAL and  $\mathrm{PAL + }$  are comparable, showing that the annotated OOD_ID instances have minimal impact on the overall performance of the model.
In addition, Table 5 lists the number of annotated OOD_ID instances with their proportion in the detected OOD data, for round 1, 3, 5, 7, and 9 on three data sets. The results in Table 5 show that the annotated OOD_ID instances have a very small proportion in the detected OOD instances, and this may also explain why the PAL and  $\mathrm{PAL + }$  are comparable.
Table 5: The number (with its proportion) of annotated OOD_ID instances used in PAL+.
Seed	Dataset	Round
1	3	5	7	9
seed=1	CIFAR-10	5 (10%)	1 (2%)	0 (0%)	0 (0%)	1 (2%)
CIFAR-100	5 (10%)	5 (10%)	1 (2%)	0 (0%)	0 (0%)
Tiny-Imagenet	3 (6%)	3 (6%)	2 (4%)	1 (2%)	0 (0%)
seed=2	CIFAR-10	3 (6%)	0 (0%)	0 (0%)	0 (0%)	0 (0%)
CIFAR-100	4 (8%)	2 (4%)	1 (2%)	1 (2%)	0 (0%)
Tiny-Imagenet	6 (12%)	3 (6%)	0 (0%)	0 (0%)	1 (2%)
seed=3	CIFAR-10	1 (2%)	0 (0%)	0 (0%)	0 (0%)	0 (0%)
CIFAR-100	1 (2%)	2 (4%)	1 (2%)	1 (2%)	0 (0%)
Tiny-Imagenet	2 (4%)	3 (6%)	2 (4%)	0 (0%)	2 (4%)
Figure 10: Comparison of classification and OOD detection performance on CIFAR-10 (first row), CIFAR-100 (second row) and Tiny-Imagenet (third row) with an ID proportion of  $20\%$ .
Visualization. To demonstrate the distribution of selected ID instances, we provide visual results of the iterative query round. In detail, we visualize the feature representations of the labeled and queried correct ID instances via t-SNE [21] on the CIFAR-10 dataset with an ID proportion of  $40\%$ . Note that green crosses represent the queried ID instances. Figure 11 records the visualizations of PAL and state-of-the-art methods, i.e., Coreset and LfOSA. The figures reveal that PAL selects more ID instances than other methods, i.e., higher purity in different rounds. Moreover, from the perspective of data distribution, PAL would balance the informativeness and representativeness criteria to select meaningful ID instances for ID classifier training.
Coreset
LfOSA
PAL
Figure 11: The t-SNE visualizations of Coreset, LfOSA, and PAL, considering the 1st (top row), 5th (middle row), and 9th (bottom row) rounds.
Compare with Open-Set Semi-Supervised Learning. Although both open-set/safe semi-supervised learning and open-set AL aim to construct a more effective ID classifier by reasonably utilizing unlabeled data, the essential difference lies in whether or not manual participation is necessary. Open-set/Safe semi-supervised learning aims to use all the pseudo-ID instances in unlabeled data according to the model's prediction, whereas open-set AL aims to select the most valuable pseudo-ID instances in unlabeled data for manual labeling to retrain the model without using all the unlabeled data. To evaluate the classification performance of PAL and open-set semi-supervised learning methods, we have compared PAL with two open-set semi-supervised learning methods, DS $^3$ L [5] and OpenMatch [25] using the CIFAR-10 and CIFAR-100 datasets with an ID proportion of  $20\%$ . As illustrated in Table 6, the results show the proposed PAL also performs better than the open-set semi-supervised learning methods.
Table 6: Comparison of testing accuracy (%) for  $\mathrm{DS}^3\mathrm{L}$ , OpenMatch, and PAL on CIFAR-10 and CIFAR-100 with an ID proportion of  $20\%$ .
Method	CIFAR-10	CIFAR-100
DS3L	74.2	48.1
OpenMatch	82.1	66.5
PAL	98.7	69.4
Runtime Cost. To verify the runtime cost of the proposed PAL, we evaluate it against several baselines, including  $\mathrm{DS}^3\mathrm{L}$ , OpenMatch, Coreset, and LfOSA on CIFAR-10 and CIFAR-100 with the ID proportion of  $20\%$ . As demonstrated in Table 7, the results show that PAL has the second best runtime cost, outperforming  $\mathrm{DS}^3\mathrm{L}$ , OpenMatch, and LfOSA, but falling behind Coreset.
Table 7: Comparison of runtime cost for PAL and other baselines on CIFAR-10 and CIFAR-100 with an ID proportion of  $20\%$ .
Method	CIFAR-10	CIFAR-100
DS3L	26.8h	43.4h
OpenMatch	37.7h	68.3h
Coreset	2.8h	3.6h
LfOSA	66.2h	77.8h
PAL	3.4h	4.4h