Meta-Query-Net: Resolving Purity-Informativeness Dilemma in Open-set Active Learning
Authors: Dongmin Park
Yooju Shin
Jihwan Bang
Youngjun Lee
Hwanjun Song
Jae-Gil Lee
Venue: NeurIPS 2022 Accept (2022)
Keywords: Active Learning
Open-set Noise
Out-of-distribution Data

Abstract: Unlabeled data examples awaiting annotations contain open-set noise inevitably. A few active learning studies have attempted to deal with this open-set noise for sample selection by filtering out the noisy examples. However, because focusing on the purity of examples in a query set leads to overlooking the informativeness of the examples, the best balancing of purity and informativeness remains an important question. In this paper, to solve this purity-informativeness dilemma in open-set active learning, we propose a novel Meta-Query-Net (MQ-Net) that adaptively finds the best balancing between the two factors. Specifically, by leveraging the multi-round property of active learning, we train MQ-Net using a query set without an additional validation set. Furthermore, a clear dominance relationship between unlabeled examples is effectively captured by MQ-Net through a novel skyline regularization. Extensive experiments on multiple open-set active learning scenarios demonstrate that the proposed MQ-Net achieves 20.14% improvement in terms of accuracy, compared with the state-of-the-art methods.

Meta-Query-Net: Resolving Purity-Informativeness Dilemma in Open-set Active Learning
Dongmin Park', Yooju Shin', JJhhan Banga.3, Youngjun Lee', Hwanjun Song", Jae-Gil Lee
KAIST, NAVER AI Lab, $ NAVER CLOVA
(dongminpark, yooju. shin, youngjun. lee, jaegil) .astt.ac.aa ac.kr (jihwan. bang, hwanjun. onaverooo.....p....
Abstract
Unlabeled data examples awaiting annotations contain open-set noise inevitably. A few active learning studies have attempted to deal with this open-set noise for sample selection by filtering out the noisy examples. However, because focusing on the purity of examples in query set leads to overlooking the informativeness of the examples, the best balancing of purity and informativeness remains important question. this paper, to solve this purity-informatiieness dilemma open-set active learning, propose novel Meta Meta-Query Query-uerrreeeeee (MQ-NNrr that adaptively finds the best balancing between the two factors. Specifically, by leveraging the multi-round property of active learning, train MQ-Net using query set without an additional validation set. Furthermore, a clear dominance relationship between unlabeled examples is effectively captured by MQ-Net through novel skyline regularization. Extensive experiments on multiple open-set active learning scenarios demonstrate that the proposed MQ-Net achieves 20.14% improvement in terms of accuracy, compared with the state-of-the-art methods.
Introduction
The success of deep learning in many complex tasks highly depends on the availability of massive data with well-annotated labels, which are very costly to obtain in practice ([] Active learning (AL) one the popular learning frameworks to reduce the high numan-labeling cost, where a small number of maximally-informmatii examples are selected by query strategy and labeled by oracle repeatedly (2). Numerous query (i.e., sample selection) strategies, mainly categorized into uncertainty- based sampling [3] 4 and diversity- sampling [6. 8), have succeeded in effectively reducing the labeling cost while achieving high model performance.
Despite their success, most standard AI. approaches rely strict assumption that all unlabeled examples should be cleanly collected from pre-defined domain called in-distribution (IN), even before being labeled [9. This assumption is unrealistic in practice since the unlabeled examples are mostly collected from rather casual data curation processes such as web-crawling, Notably, in the Google search engine, the precision of image retrieval is reported to be 82% average, and it is worsened to 48% for unpopular entities [T0 IT. That is, such collected unlabeled data naturally involves open-set noise, which is defined set of the examples collected from different domains called out-of distributiii (OOD) [1].
In general, standard AL approaches favor the examples either highly uncertain in predictions highly diverse in representations as a query for labeling. However, the addition open-set noise makes these two measures fail to identify informative examples; the OOD examples also exhibit high uncertainty and diversity because they share neither class-distinetive features nor other inductive biases with IN examples [5. As result, active learner is confused and likely to query the OOD examples to
"Corresponding authors.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Unlabeled Examples ( (OIN, DOOD]
LL focused -A --CAL (HP-focused)
MQ-Net
65%
HP-LI
HP-HI
High
Tout Accuracy
Test Accuracy
53%
55%
Puantyy
45%
45%
Low
35%
dominance
HI do-doiinnecee
HP domommacae
HI dom
LP-L1
LP-HI
35%
25%
Law
High
Informativeness
Rounds
Rounds
(a) Purity-informativeness Dilemma.
(b) 10% Open-set Noise.
(c) 30% Open-set Noise.
Figure 1: Motivation of MQ-Net: (a) shows the purtty-infrrmativenss dilemma for query selection in open-set AI; (b) shows the AL. performances of a standard AL method (HI-focused), I.I. [5], and an open-set AL method (HP-focused), CCAL [3] along with our proposed MQ-Net for the ImageNet dataset with noise ratio of 10%; (c) shows the trends with a noise ratio of 30%.
a human-annotator for labeling. Human annotators would disregard the OOD examples because they are unnecessary for the target task, thereby wasting the labeling budget. Therefore, the problem of active learning with open-set noise, which we call open-set active learning, has emerged as new important challenge for real-world applications.
Recently, a few studies have attempted to deal with the open-set noise for active learning 13 16
They commonly try to increase the purity of examples in a query set, which is defined the proportion of IN examples, by effectively filtering out the OOD examples. However, whether focusing on the purity needed throughout the entire training period remains a question. In Figure [Ta), aa), let's consider an open-set AL task with a binary classification of cats and dogs, where the images of other animals, e.g., horses and wolves, are regarded as OOD examples. It clear that the group of high purity and high informativeness (HP-HH) is the most preferable for sample selection. However, when comparing the group of high purity and low informativeness (HP-LI) and that of low purity and high informativeness (L.--HI,, the preference between these two groups of examples is not clear, but rather contingent on the learning stage and the ratio of OOD examples. Thus, we coin a new term "purityinfformativenss dilemma" to call attention to the best balancing of purity and informativeness.
Figures bb) and (c) illustrate the purity-informaaiiennss dilemma. The standard AL approach, LL puts more weight on the examples of high informativeness (denoted as HI-focused), while the existing open-set AL approach, CCAL [13] puts more weight on those of high purity (denoted as HP-focused). The HP-ffcused approach improves the test accuracy more significantly than the HI-focused one at earlier AL rounds, meaning that pure as well as easy examples are more beneficial. In contrast, the HI-focused approach beats the HP- focused one at later AL rounds, meaning that highly informative examples should be selected even at the expense of purity. Furthermore, comparing low OOD (noise) ratio in [ and a high OOD ratio in Figure the shift from HP-dominance to HI-dominance tends occur later at a higher OOD ratio, which renders this dilemma more difficult.
In this paper, to solve the purity-informativnnss dilemma in open-set AL, we propose a novel meta- model Mieta-Query-Ner (MQ-Net) that adaptively finds the best balancing between the two factors. A key challenge is the best balancing is unknown in advance. The meta-model is trained to assign higher priority for in-distribution examples over OOD examples as well as for more informative examples among in-distribution ones. The input to the meta-model, which includes the target and OOD labels, obtained for free from each AL round's 's query set by leveraging the multi-round property of AL. Moreover, the meta-model is optimized more stably through novel regularization inspired by the skyline query [17 [X] popularly used multi-ohjective optimization. As result, MQ-Net can guide the learning of the target model by providing the best balancing between purity and informativeness throughout the entire training period.
Overall, our main contributions are summarized as follows:
1. We formulate the surity-infrrnativennss dilemma, which hinders the usability of open-set AL in woal-wrlld applications.
2. As our answer to the dilemma, we propose a novel AL framework, MQ-Net, which keeps finding the best trade-off between purity and informativeness.
Extensive experiments on CIFAR1O, CIFAR 100, and ImageNet show that MQ-Net improves the classifier accuracy consistently when the OOD ratio changes from 10% to 60% by up to 24%.

---

2 Related Work
2.1 Active Learning and Open-set Recognition
Active Learning is a learning framework to reduce the human labeling cost by finding the most infor- mative examples given unlabeled data p [9]. One popular direction is uncertainty-basee sampling. Typical approaches have exploited prediction probability, egg,, soft-max confidence (20) 3] mar- gin [21], and entropy [22]. Some approaches obtain uncertainty by Monte Carlo Dropout on multiple
forward passes [23 24 (2). LL [S] predicts the loss of examples by jointly learning a loss prediction module with a target model. Meanwhile, diversity-basee sampling has also been widely studied. To incorporate diversity, most methods use clustering 6 [6] or coreset selection algorithm 77]. Notably, CoreSet (7 finds the set of examples having the highest distance coverage on the entire unlabeled data. BADGE 8 is a hybrid of uncertainty- and diversity-
sampling which uses k-means++ clustering the gradient embedding space. However, this family of approaches not appropriate for open-set AL since they do not consider how to handle the OOD examples for query selection.
Open-set Recognition (OSR) is detection task recognize the examples outside of the target domain [2]. Closely related to this purpose, OOD detection has been actively studied (27). Recent work can be calegorized into classiferr-dependett, density-based, and self-supervised approaches. The classifierddeeenent approach leverages a pre-trained classifier and introduces several scoring functions, such as Uncertainty 228, ODIN (29]. mahalanobis distance (MD) [00, and Energy (31. Recently, ReAct [32] shows that rectifying penultimate activations can enhance most of the aforemen- tioned daesidier-dependent OOD scores. The density-based -aseed approach learns an auxiliary generative model like a variational auto-encoder to compute likelihood- OOD scores [33 34 35] Most self-supervised approaches leverage contrastive learning [36
37 38 CSI shows that contrasting with distributiionally-siitee augmentations can considerably enhance the OSR performance (36]
The OSR performance of classifierdeependent approaches degrades significantly if the classifier performs poorly (39). Similarly, the performance of density-based and self-supervised approaches heavily resorts to the amount of clean IN data 36). Therefore, open-set active learning is a challenging problem to be resolved by simply applying the OSR approaches since it is difficult to obtain nigh-quality classifiers and sufficient IN data at early AL rounds.
2.2 Open-set Active learning
Two recent approaches have attempted to handle the open-set noise for AL 13 16.. Both approaches try to increase purity in query selection by effectively filtering out the OOD examples. CCAL [I3] learns two contrastive coding models each for calculating informativeness and OODness of an example, and combines the two scores using heuristic balancing rule. SIMILAR [16] selects pure and core set of examples that maximize the distance on the entire unlabeled data while minimizing the distance to the identified OOD data. However, we found that CCAL and SIMILAR than standard AL methods, since they always put higher weights on purity although informativeness should be emphasized when the open-set noise ratio is small in later AL rounds. This calls for developing a new solution to carefully find the best balance between purity and informativeness.
often worse
3 Purity-Informativennss Dilemma in Open-set Active Learning
3.1 Problem Statement
Let DIN and Doon be the IN and OOD data distributions, where the label of examples from DOoD does not belong to any of the k known labels Y= {yr ) Then, an unlabeled set is mixture of IN and OOD examples, U = {XIN, XOOD), ,, i.e., XIN ~ DIN and XOOD ~ DOOD. In the open-set AL, a human oracle is requested to assign known label y an IN example cost CJN. while an OOD example x E XOOD is marked as open-set noise with a labeling cost CoOD-
XIN with a labeling
AL imposes restrictions on the labeling budget b every round. It starts with a small labeled set St. consisting of both labeled IN and OOD examples. The initial labeled set St improves by adding a small but maxinally-inffrraatv labeling cost for So by the oracle does not exceed the labeling budget Hence, the goal of open-set AL is defined to construct the optimal query set Sz. minimizing the loss for the wnseen target IN data. The difference from standard AL is that the labeling cost for OOD examples is introduced, where the labeling budget is wasted when OOD examples are misclassified as informative ones.
labeled query set SQ per round, i.e., S + Sy USQ, where the
3

---

Formally, let C()) be the labeling cost function for given unlabeled set; then, each round of open-set AL formulated find the best query set SQ as
SQ argmin
E B(D.))ETIN (s,us)))))
Se: ((SQ)5b
(I)
where C(SQ) = Σ (1[pEXI]CCIN + [[ENoon]COODD) aESq
Here, f(;; O US, denotes the target model trained on only IN examples in St U U.. and fets is a certain loss function, e.g., cross-entropy, for classification. For each AL round, all the examples in So are removed from the unlabeled set U and then added to the accumulated labeled set St, with their labels. This procedure repeats for the total number r of rounds.
3.2 murrt--nfformativennss Dilemma
An idcal approach for open-set AL would be to increase both purity and informativeness of query set by completely suppressing the selection of OOD examples and accurately querying the most informative examples among the remaining IN examples. However, the ideal approach is infeasible because overly emphasizing purity query selection does not promote example informativeness and vice versa. Specifically, OOD examples with low purity scores mostly exhibit high informativeness scores because they share neither class-distinctive features nor other inductive biases with the IN examples [14 [5]. We call this trade-off in query selection as the irrityiinformaiivenss dilemma, which is our new finding expected to trigger lot of subsequent work.
To address this dilemma, we need to consider the proper weights of purity score and an informative score when they are combined. Let Pax) be a purity score of example which can be measured by any existing OOD scores, e.g., negative energy [31] and T(x:) be an informativeness score of an example æ from any standard AL strategies, e.g., uncertainty B and diversity [26]. Next, supposing
} is tuple of available purity and informativeness scores for an example x. Then, a score combination function & ((2z), where 2n (P(x), [(x)), is defined return an overall score that indicates the necessity of æ being included in the query set.
2n = (2 [x))
Given unlabeled examples x; and Ty., if P(::) > P(ry) and I(x;) > I(æ;), ). it is clear favor Ij over zÿ based on ((2,)) > ((Zz,), However, due to the iriityinformativeness dilemma. if P(x;)) > P(x,)) and I(x;) < I(x;) or P(x;) < P(xy)) and T(x;) > I(xy), it is very challenging to determine the dominance between ((z,,) and ((Zzy), In order to design ( mainly focus
on leveraging meta-learning, which is more agnostic approach to resolve the dilemma other than
several heuristic approaches, such as linear combination and multiplication.
Meta-Query-Net
We propose meta-model, named Meta- Netta-Query-Net (MQ-NNet), which aims to learn meta-score function for the purpose of identifying a query set. In the presence of open-set noise, MQ-Net outputs the meta-score for unlabeled examples to achieve the best balance between purity and informativeness in the selected query set. In this section, we introduce the notion of a velf-validation set to guide the meta-model in supervised manner and then demonstrate the meta-objective of MQ-Net for training. Then, we propose a novel skyline constraint used in optimization, which helps MQ-Net capture the obvious preference among unlabeled examples when a clear dominance exists. Next, we present way of converting the purity and informativeness scores estimated by existing methods for use in MQ-Net. Note that training MQ-Net is not expensive because builds light meta-model on a small self-validation set. The overview of MQ-Net is illustrated in Figure
4.1 Training Objective with Self-validation Set
The parameters contained in MQ-Net (f;;w) w) is optimized in a supervised manner. For clean supervision, validation data is required for training. Without assuming a hard-to-obtain clean validation set, propose
a self-validation set, which is instantaneously generated in every AL round. In detail, we obtain a labeled query set Sq by the oracle, consisting of a labeled IN set and an identified OOD set in every round. Since the query set SQ unseen for the target model O and the meta-model w at the current round, we can exploit it as a self-validation set to train MQ-Net. This self-van validaaiittionn set eliminates the need for a clean validation set in meta-learning.

---

Given the ground-truth labels in the self-validation set, it feasible to guide MQ-Net to be trained resolve the purity-ifformativnnss. dilemma by designing an appropri- ate meta-objective. It is based on the cross-entropy loss for classification because the loss value of training exam- ples has been proven to be effective in identifying high informativeness examples [5] The conventional loss value by a target model e is masked to be zero if x E XOOD since OOD examples useless for AL,
Update d
MQ-Net Ow>
Unlabeled
Labeld Sr.
=
(onn
Updklew
1())
Selfvvliicatonn
Quary
Sq
Figure 2: Overview of MQ-Net.
Iace (2) -1 (o==))
where / is a true binary IN label, i.e. 1 for IN examples and O for OOD examples, which can be reliably obtained from the self-vaaidation set. This masked loss, Emce. preserves the informativeness of IN examples while excluding OOD examples. Given self-validation data Sq, the meta-objective defined such that MQ-Net parameterized by w outputs a high (or low) meta-score (zz,; w) if an example z's masked loss value is large (or small).
((5))= = -E Σ max 00 (Sign(emoe moe (x;)) (#e..) w) B(2z) w) + n)
iESQ JESQ
(3)
s.t. VX,, Ty if P(x;)= > P(z,) and I(x,) > I(z;), then P(zz, W)
D PZzzi w),
where v) > 0 is a constant margin for the ranking loss, and Sign(a, b) is an indicator function that returns +1 if a b, 1 if a = and - otherwise. Hence, (Zz,.i w) is forced to be higher than (Z2,;; w) if Emco (mce(T;) > (mze(T;); in contrast, ((2,, w) is forced to be lower than ((2z, w) if Ince (xi) < Exnce(Ig (Iz). Two OOD examples do not affect the optimization because they do not have any priority between them, i.e., mcce (2,) (zy))
In addition to the ranking loss, we add regularization term named the skyline constraint (i.e,, the second line) in the meta-objective Eq. 33. which is inspired by the skyline query which aims to narrow down a search space in a large-seale database by keeping only those items that are not worse than any other [7 (8). Specifically. in the case of P(x;) P(zz)) and I (z;) I(zz). )) the condition & ( w) > 4 (Zayi w) must hold in our objective, and hence we make this proposition as the skyline constraint. This simple yet intuitive regularization is very helpful for achieving meta-model that better judges the importance of purity or informativeness. We provide an ablation study on the skyline constraint in Section
4.2 Architecture of MQ-Net
MQ-Net is parameterized by a multi-layer perceptron (MLP), widely-used deep learning architecture
for meta-learning [4) A challenge here is that the proposed skyline constraint in Eq. (3) does not hold
with standard MIP model. To satisfy the skyline constraint, the meta-score function (1:: w) should
be a monotonic non-decreasing function because the output (meta-score) of MQ-Net for an example I; must be higher than that for another example zj if the two factors (purity and informativeness) of x; are both higher than those of zj. The MLP model consists of multiple matrix multiplications with non-linear activation functions such as ReLU and Sigmoid. In order for the MLP model to be monotonically non -ecreecas,n all the parameters in for ((: w) should be non negative, as proven by Theorem
Theorem 4.1. For any MLP meta-model w with non-decreasing activation functions, a mela-score function & (2: Rd + R holds the skyline constraints if w 0 and z(E Rd) > O where > is the component-wise inequality.
Proof. An MLP model is involved with matrix multiplication and composition with activation functions, which are characterized by three basic operators: (1) addition: h(z) f(z) + g(=), (2) multiplication: h(2)= = f() x 9(2), and (3) composition: h(z) f o g(2). These three operators are guaranteed to be non non-decreasing functions if the parameters of the MLP model are all non- negative, because the non-negative weights guarantee all decomposed scalar operations in MLP to be non-decreasing functions. Combining the three operators, the MLP model ((2; w), where w > 0, naturally becomes a monotonic non-decreasing function for each input dimension. Refer to Appendix for the complete proof.
5

---

In implementation, non-negative weights are guaranteed by applying a ReLU function to meta- parameters. Since the ReLU function is differentiable, MQ-Net can be trained with the proposed objective in an end-to-end manner. Putting this simple modification, the skyline constraint is preserved successfully without introducing any complex loss-based regularization term. The only remaining condition is that each input of MQ-Net must be a vector of non-negative entries.
4.3 Active Learning with MQ-Net
4.3.1
Meta-input Conversion
MQ-Net receives スn = (P(x), I(x)) and then returns a meta-score for query selection. All the scores for the input of MQ-Net should be positive to preserve the skyline constraint, i.e., = 2 0. Existing OOD and AL query scores are converted to the meta-input. The methods used for calculating the scores are orthogonal to our framework. The OOD score (-)) is conceptually the opposite of purity and varies in its scale; hence, we convert it to purity score by P(z) where Normalizo(-) the z-score normalization. This conversion guarantees the purity score to be positive. Similarly, for the informativeness score, we convert an existing AL query score (--) to I(x) = (sspmmmazl ((x))). For the z-score normalization, we compute the mean and standard deviation of (xx) or ((x) over the unlabeled examples. Such mean and standard deviation are iteratively computed before the meta-training, and used for the z-score normalization at that round.
(z))))
4.3.2 Overall Procedure
For each AL round, target model is trained via stochastic gradient descent (SGD) mini-batches sampled from the IN examples in the current labeled set Sr.- Based on the current target model, the purity and informative scores are computed by using certain OOD and AL query scores. The querying phase is then performed by selecting the examples So with the highest meta-scores within the labeling budget b. The query set Sq used as the self-validation set for training MQ-Net the current AL round. The trained MQ-Net is used at the next AL round. The alternating procedure of updating the target model and the meta-model repeats for a given number r of AL rounds. The pseudocode of MQ-Net can be found in Appendix B
5 Experiments
5.1 Experiment Setting
Datasets. We perform the active learning task on three benchmark datasets; CIFARIO 41 CI-
FAR100 [11, and ImageNet [42). Following the 'split-dataset' setup in open-world learning litera- we divide each dataset into two subsets: (1) the target set with IN classes and (2) the noise set with OOD classes. Specifically, CIFAR10 is split into the target set with four classes and the noise set with the rest six classes; CIFAR 100 into the two sets with 40 and 60 classes; and ImageNet into the two sets with 50 and 950 classes. The entire target set is used as the unlabeled IN data, while only a part of classes in the noise set is selected as the unlabeled OOD data according to the given noise ratio. In addition, following OOD detection literature [28 33 we also consider the cross-datased setup, which mixes a certain dataset with two external OOD datasets collected
ture 13 16
from different domains, such LSUN 444 and Places365 [45] For sake of space, we present all the results on the cross-dataset setup in Appendix D
Algorithms. We compare MQ-Net with random selection, four standard AL, and two recent open-set AL approaches.
Standard AL: The four methods perform AL without any processing for open-set noise: (1) CONF 33 queries the most uncertain examples with the lowest softmax confidence in the prediction, (2) CORESET 7 queries the most diverse examples with the highest coverage in the representation space, (3) LL [5] queries the examples having the largest predicted loss by jointly learning a loss prediction module, and (4) BADGE g considers both uncertainty and diversity by querying the most representative examples in the gradient via k--mean+++ clustering [46).
Open-set AL: The two methods tend to put more weight on the examples with high purity: (1) CCAL I3 learns two contrastive coding models for calculating informativeness and OODness, and then it combines the two scores into one using heuristic balancing rule, and (2) SIMILAR [66 selects a pure and core set of examples that maximize the distance coverage on the entire unlabeled data while minimizing the distance coverage to the already labeled OOD data.

---

RANDOM CONE
CORESET LL
BADGE OCAL SIMILAR
-ORS
10% Notise
20% Noise
40% Noise
60% Noise
40
90
90
U5
Accuracy
80
80
80
75
25
70
10
5
5
Round
Round
Round
Round
(a) Accuracy comparison over AL rounds on CIFARIO with open-set noise of 10%, 20%, 40%. and 60%.
70
10% Naise
40% Noise
60% Noise
Accuracy (%)
60
50
40
60
10
30
10
5
9
5
Round
Round
Round
Round
(h) Accuracy comparison over AL rounds on CIFAR 100 with open-set noise of 10%, 20%, -10. and 60%.
10% Naise
60
20% Noise
40%.ooss Noise
60% Noise
30
Accuracy
50
40
00
30
30
30
Round
Round
Bound
(c) Accuracy comparison over AI. rounds on ImgeNet with open-set noise of 10%, 20%, 40%, and 60%.
Figure 3: Test accuracy over AL rounds for CIFARIO, CIFARIOO, and ImageNet with varying open-set noise ratios.
For all the experiments, regarding the two inputs of MQ-Net, mainly CSI 33 and LL [5] for calculating the purity and informativeness scores, respectively. For CSI, as in CCAL, we train a contrastive learner on the entire unlabeled set with open-set noise since the clean an-distribution set is not available in open-set AL. The ablation study in Section 3.4 shows that MQ-Net is also effective with other OOD and AL scores its input.
Implementation Details. We repeat the three tegpnntrainnn,, querying, and labeling- of AL. The total number r of rounds is set 10. Following the prior open-set AL setup [6], we set the labeling cost CIN = 1 for IN examples and COOD = for OOD examples. For the class-split setup, the labeling budget b per round is set 500 for CIFARIO/100 and 000 for ImageNet. Regarding the open-set noise ratio T, we configure four different levels from light to heavy noise in (10% 20%, 40%, 60%). In the case of 0% (no noise), MQ-Net naturally discards the purity score and only uses the informativeness score for query selection, since the self-validation set does not contain any OOD examples. The initial labeled set is randomly selected uniformly at random from the entire unlabeled set within the labeling budget b. For the architecture of MQ-Net, we use 2-layer MLP with the hidden dimension size of 64 and the Sigmoid activation fuction. We report the average results of five runs with different class splits. We did not use any pre-trained networks. See Appendix
C for more implementation details with training configurations. All methods are implemented with PoTorc 1.8.0 and executed on a single NVIDIA Tesla V100 GPU. The code is available at atttps://githan com/kaist kaist- st- dmlab/MQNet.
5.2
Open-set Noise Robustness
5.2.1
Results over AL Rounds
Figure SigureBiilusten illlstrates the test accuracy of the target model over AL rounds on the two CIFAR datasets. MQ-Net achieves the highest test accuracy in most AL rounds, thereby reaching the best test accuracy the final round in every case for various datasets and noise ratios. Compared with the two existing open-set AL methods, CCAL and SIMILAR, MQ-Net shows a steeper improvement in test accuracy over rounds by resolving the murit-innfrrmatiennss dilemma query selection. For example, the performance gap between MQ-Net and the two open-set AL. methods gets larger after the sixth round, as shown in Figure b)), because CCAL and SIMILAR mainly depend on purity in query selection, which conveys less informative information to the classifier. For better classifier, informative
7

---

Table Last test accuracy (%) at the final round for CIFARIO, CIFARI00, and ImageNet. The best results are in bold, and the second best results are underlined.
Datasets
CIFARIO (4:6 split)
CTFAR 100 (40:60 split)
ImageNet (50:950 split)
Noise Ratio
10%
20%
40%
60%
10%
20%
40%
60%
10%
20%
40%
60%
Non-AL
RANDOM
89.83
89.06
87.73
85 64
60.88
59.69
55.52
47.37
62.72
60,12
54.04
48.24
CONF
92.83
91.72
88.69
85,43
62.84
60.20
53.74
45.38
63.56
62.56
51,08
45,04
Standard
CORESET
89.12
86.50
48.33
63.64
62.24
AL
89.41
86.95
65.08
56.27
48.49
63.28
61.56
BADGE
91.73
89.27
86.83
62.54
55.07
47.60
64.84
61.48
54.04
Open-sel
CCAL
90.55
89,99
88.87
87.49
61.20
61.16
56.70
50.20
61.68
60.70
56.60)
51.16
AI.
SIMILAR
89.92
89.19
68.53
87.38
60.07
59.89
56.13
50.61
63.92
61.40
56.48
52.84
Proposed
MQ-Net
93.10
92.10
91.48
89.51
66.44
64.79
58.96
52.82
65.36
63.08
56.95
54.11
% improve over 2nd beat
0.32
0.40
2.32
2.32
2,09
1.17
3.99
4.37
0.80
1.35
0.62
2.40
76 improve over the least
3.26
3.33
4.78
5.97
3.92
11.49
Patty (CS))
Slope -0. 74
Slope 82
Slope: -1.07
Skepe: -1 21
Round
Intommateeniss (
Intomiativniins ILL)
Informittvenees (LL)
Incocnativeeess (LL))
(a) The output of MQ-Net over AL rounds (Round 3, and with 10% noise.
Partyccs
Slopes
Sloge -0.97
Slopes -0. 81
Skope
10% Noise
20% Noise
40% Noise
60% Nais:
Informstvvness (L.))
Infomaatvestsn ILL)
Informadiveniss (LL)
T (LL))
(b) The final round's output of MQ-Net with varying noise ratios (10%, 20%. 40%, and 60%).
Figure 4: Visualization of the query score distribution of MQ-Net on CIFARI00. x- and y-axis indicate the normalized informativeness and purity scores, respectively. The background color represents the query score of MQ-Net; the red high, and the blue is low. Gray points represent unlabeled data, and blue and red points are the IN and OOD examples in the query set, respectively. The slope of the tangent line on lowest scored example in the query set displayed together; the steeper the slope, the more informativeness is emphasized in query selection.
examples should be favored at a later AL round due to the sufficient number of IN examples in the labeled set. In contrast, MQ-Net keeps improving the test accuracy in later AL round by finding the best balancing between purity and informativeness in its query set. More analysis of MQ-Net associated with the murit-informmatvennss dilemma discussed in Section 5.3
5.2.2 Results with Varying Noise Ratios
Table summarizes the last test accuracy at the final AL round for three datasets with varying levels of open-set noise. Overall, the last test accuracy of MQ-Net is the best every case. This superiority concludes that MQ-Net successfully finds the best trade-off between purity and informativenes: in terms of AL accuracy regardless of the noise ratio. In general, the performance improvement becomes larger with the increase in the noise ratio. On the other hand, the two open-set AL approaches are even worse than the four standard AL approaches when the noise ratio is less than or equal to 20%. Especially, in CIFAR 10 relatively easier than others, CCAL and SIMILAR are inferior to the non-robust AL method, L.L,, even with 40% noise. This trend confirms that increasing informativeness is more crucial than increasing purity when the noise ratio is small; highly informative examples are still beneficial when the performance of classifier is saturated the presence open-set noise. An in-depth analysis on the low accuracy of the existing open-set AL approaches in a low noise ratio is presented in Appendix

---

Table 3: Efficacy of the self-validation set.
Table 2: Effect of the meta inputs on MQ-Net.
Dataset
CIFAR (4:6 split)
Ratio
10%
20%
40%
60%
Noise Ratio
Dataset
10%
CIFARIO (4:6 split)
20%
40%
60%
MQ-Net
Query
93.10
92.10
91.48
89.51
Random
92.10
91.75
90.88
87.65
Stundard AL
BADGE
92,80
91.73
89.27
86.83
Open-set AL
CCAL
90.55
89.99
88.87
87,49
Table 4: Efficacy of the skyline constraint.
CONF-ReAc
93.21
91.89
89.54
67,99
CONFeCSI
93.28
92.40
91.43
89.37
Noise Ratio
10%
20%
40%
60%
Mel
LL-RcAct
92.34
91.85
90.08
88.41
w/ skyline
93.10
92.10
91.48
89.51
1L-CSI
93.10
92.10
91.48
89,51
MQ.Net w/o skyline
87.25
86.29
83.61
81.67
5.3 Answers to the Purit--nfoormativeness Dilemma
The high robustness of MQ-Net in Table and Figure 3 is mainly attributed to its ability to keep finding the best trade-off between purity and informativeness. Figure Fa (a) illustrates the preference change of MQ-Net between purity and informativeness throughout the AL rounds. As the round progresses, MQ-Net automatically raises the importance of informativeness rather than purity; the slope of the tangent line keeps steepening from -0.74 to - 1..1. This trend implies that more informative examples are required to labeled when the target classifier becomes mature. That is, as the model performance increases, 'fewer but highly-informativy than 'more but less- informavvvte examples in terms of improving the model performance. Figure bb) describes the preference change of MQ-Net with varying noise ratios. Contrary to the trend over AL rounds, as the noise ratio gets higher, MQ-Net prefers purity more over informativeness.
examples are more impactful
5.4 Ablation Studies
Various Combination of Meta-input. MQ-Net can design its purity and informativeness scores by leveraging diverse metries in the existing OOD detection and AL literature. Table 2 shows the final round test accuracy on CIFAR 10 for the four variants of score combinations, each of which is constructed by a combination of two purity scores and two informativeness scores; each purity score is induced by the two recent OOD detection methods, ReAct (32) and CSI (36). while each informativeness score is converted from the two existing AL methods, CONF and LL. "CONF-ReAct" denotes variant that uses ReAct the purity score and CONF the informativenes: score.
Overall, all variants perform better than standard and open-set AL baselines in every noise level. Refer to Table for detailed comparison. This result concludes that MQ-Net can be generalized over different types of meta-input owing to the learning flexibility of MLPs. Interestingly, the variant using CSI as the purity score is consistently better than those using ReAct. ReAct, a classifier-dependent OOD score, performs poorly earlier AL rounds. A detailed analysis of the two OOD detectors, ReAct and CSI, over AL rounds can be found in Appendix F
Efficacy of Self-validation Set. MQ-Net can be trained with independent validation set, instead of using the proposed self-validation set. We generate the independent validation set by randomly sampling the same number of examples the self-validation set with their ground-truth labels from the entire data not overlapped with the unlabeled set used for AI. As can be seen from Table 3 it is of interest
that our self-validation set performs better than the random validation set. The two validation sets have major difference in data distributions; the self-validation set mainly consists of the examples with highest meta-scores among the remaining unlabeled data per round, while the random validation set consists of random examples. We conclude that the meta-score of MQ-Net has the potential for constructing a high- -ugh-ty validation set in addition to query selection.
Efficacy of Skyline Constraint. Table demonstrates the final round test accuracy of MQ-Net with or without the skyline constraint. For the latter, standard 2-layer MLP is used as the neta-network architecture without any modification. The performance of MQ-Net degrades significantly without the skyline constraint, meaning that the non-constrained MLP can easily overfit to the small-sized self-validation set, thereby assigning high output scores on less-pure and less-informative examples. Therefore, the violation of the skyline constraint in optimization makes MQ-Net hard to balance between the purity and informativeness scores in query selection.
Efficacy of Meta-objective. MQ-Net keeps finding the best balance between purity and informative- ness over multiple AL rounds by repeatedly minimizing the meta-objective in Eq. 3 To validate its

---

Table 5: Efficacy of the meta-objective in MQ-Net. We show the AL performance of two alternative balancing rules compared with MQ-Net for the split-dataset setup on CIFARIO with the open-set set noise ratios of 20% and 40%.
Dataset
Noise Ratio
Round
7
8
9
10
P(z) + [()
61.93
73.82
76.16
80.65
82.61
85.73
87.44
88.86
89.21
89.49
20%
P(a) I(a)
61.93
71.79
78.09
81.32
84.16
86.79
88.74
89.89
90.54
91.20
CIFARIO
MQ-Net
61.93
73.82
80,58
84.72
87.31
89.20
90.52
91.46
91.93
92.10
(4:6 split)
P(z) Z(z)
59.31
72.50
75.67
78.78
81.70
83.74
85.08
86.48
87.47
88.86
40%
P(a) I[m)
59.31
66.37
73:57
77.85
81.37
84.22
86.80
88.04
88.73
89.11
MQ-Net
90.07
90.92
91.48
efficacy, compare it with two simple alternatives based heuristic balancing rules such as linear combination and multiplication, denoted as P(x)) + Z(x) and P(x) I(x), respectively. Following the default setting of MQ-Net, we use LL for P(x) and CSI for [(x).
Table shows the AL performance of the two alternatives and MQ-Net for the split-dataset setup on CIFARIO with the noise ratios of 20% and 40%. MQ-Net beats the two alternatives after the second AL round where MQ-Net starts balancing purity and informativeness with its meta-objective. This result implies that our meta-objective successfully finds the best balance between purity and informativeness by emphasizing informativeness over purity at the later AL. rounds.
5.5 Effect of Varying OOD Labeling Cost
The labeling cost for OOD examples could vary with respect to data domains. To validate the robustness of MQ-Net on diverse labeling scenarios, we conduct an additional study of adjusting the labeling cost COOD for the OOD examples. change with four different labeling costs (i.e., 0.5, 2, and 4). The two standard AL methods, CONF and CORESET, and two open-set AL methods, CCAL and SIMILAR. are compared with MQ-Net. Overall, MQ-Net consistently outperforms the four baselines regardless of the labeling cost. Meanwhile, CCAL and SIMILAR are more robust to the higher labeling cost than CONF and CORESET; CCAL and SIMILAR, which favor high purity examples, query more IN examples than CONF and CORESET, they are less affected by the labeling cost, especially when is high.
Table Effect of varying the labeling cost COOD. We measure the last test ac- curacy for the split-dataset setup on CI- FARI0 with an open-set noise ratio of 40%. The best values are bold.
(ablerossmmmarzz the performance
GOOD
CONF
CORESET
90.59
89.12
85.32
81.22
CCAL
90.25
88.87
88.16
87.25
SIMILAR
91.05
88.69
87.95
86.52
MQ-Net Net
92.52
91.48
89.53
87.36
6 Conclusion
We propose MQ-Net, novel meta-model for open-set active learning that deals with the purity- informativeness dilemma. MQ-Net finds the best balancing between the two factors, being adaptive to the noise ratio and target model status. A clean validation for the meta-model is obtained for free by exploiting the procedure of active learning. A ranking loss with the skyline constraint optimizes MQ-Net make the output legitimate meta-score that keeps the obvious order of examples. MQ-Net is shown to yield the best test accuracy throughout the entire active learning rounds, thereby empirically proving the correctness of our solution to the purity-informativenss. dilemma. Overall, we expect that our work will raise the practical usability of active learning with open-set noise.
Acknowledgement
This work was supported by Institute of Information Communications Technology Planning Evaluation (ITPP) grant funded by the Korea government (MSIT) (No. 2020-0-00862, DB4DL: High- Usability and Performance In-Memory Distributed DBMS for Deep Learning). The experiment was conducted by the courtesy of NAVER Smart Machine Learning (NSML) [47).
10

---

References
[1] Hwanjun Song, Minseok Kim, Dongmin Park, Yooju Shin, and Jae-Gil Lee. Learning from noisy labels with deep neural networks: A survey. IEEE Transactions Neural Networks and Learning Systems, pages 1-19, 2022.
[2] Burr Settles. Active learning literature survey. Technical Report 1648, University of Wisconsin- Madison, Department of Computer Sciences, 2009.
[3] Dan Wang and Yi Shang. A new active labeling method for deep learning. In IJCNN, pages 112-119, 2014.
[4] William H Beluch, Tim Genewein, Andreas Nürnberger, and Jan M Köhler. The power of ensembles for active learning in image classification. In CVPR, pages 9368-9377, 2018.
[5] Donggeun Yoo and In So Kweon. Learning loss for active learning. In CVPR, pages 93- 2019.
[6] Hieu T Nguyen and Arnold Smeulders. Active learning using clusclusteiing In ICML, 2004.
[7] Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: core-set approach. In ICLR, 2018.
[8] Jordan T. Ash, Chicheng Zhang, Akshay Krishnamurthy, John Langford, and Alekh Agarwal. Deep batch active learning by diverse, uncertain gradient lower bounds. In ICLR, 2020.
[9] Pengzhen Ren, Yun Xiao, Xiaojun Chang, Yao Huang, Zhihui Li, Brij B Gupta, Xiaojiang Chen, and Xin Wang. A survey of deep active learning. ACM Computing Surveys (CSUR), 54(9):1-40, 2021.
[10] Ahmet Uyar and Rabia Karapinar. Investigating the precision of web image search engines for popular and less popular entities. Journal of Information Science, 43(3):378-333 392, 2017.
[11] Mehrdad CheshmehSohrabi and Elham Adnani Sadati. Performance evaluation of web search engines in image retrieval: An experimental study. Information Development, pages 1-13, 2021.
[12] Chuanxing Geng. Sheng-jun Huang, and Songcan Chen. Recent advances in open set recogni- tion: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(10):3614 3631, 2021.
[13] Pan Du, Hui Chen, Suyun Zhao, Shuwen Cha, Hong Chen, and Cuiping Li. Contrastive active learning under class distribution mismatch. IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 1-13, 2022.
[14] Hung- Yu Tseng, Hsin- Ying Lee, Jia-Bin Huang, and Ming-Hsuan Yang. Cross-domain few-shot classification via learned feature-wise transformation. In ICLR, 2020.
[15] Hongxin Wei, Lue Tao, Renchunzi Xie, and Bo An. Open-set label noise can improve robustness against inherent label noise. In NeurIPS, 2021.
[16] Suraj Kothawade, Nathan Beck, Krishnateja Killamsetty, and Rishabh Iyer. Similar: Submodular information measures based active learning in realistic scenarios. In NewrIPS, 2021.
[17] Christos Kalyvas and Theodoros Tzouramanis, A survey of skyline query processing. arXiv preprint
2017.
[18] Ke Deng, Xiaofang Zhou, and Heng Tao Shen. Multi-source skyline query processing in road networks. ICDE, pages 796-805, 2007.
[19] Dongmin Park, Dimitris Papailiopoulos, and Kangwook Lee. Active learning is a strong baseline for data subset selection. In Has it Trained Yer? NeurIPS 2022 Workshop.
[20] David D Lewis and Jason Catlett. Heterogeneous uncertainty sampling for supervised learning. In ICML, pages 148-156. 1994.

---

[21] Dan Roth and Kevin Small. Margin Margin-based based active learning for structured output spaces. In ECML. pages 413-424, 2006.
[22] Ajay/ Joshi, Fatih Porikli, and Nikolaos Papanikolopoulos. Multi-class active learning for image classification. In CVPR, pages 2372 2779, 2009.
[23] Yarin Gal, Riashat Islam, and Zoubin Ghahramani. Deep bayesian active learning with image data. In ICML, pages 1 1192, 2017.
[24] Andreas Kirsch, Joost Van Amersfoort, and Yarin Gal. BatchBALD: Efficient and diverse batch acquisition for deep bayesian active learning. In NewrIPS, 2019.
[25] Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In ICML, pages 1050-1059, 2016.
[26] Gui Citovsky, Giulia DeSalvo, Claudio Gentile, Lazaros Karydas, Anand Rajagopalan, Afshin Rostamizadch, and Sanjiv Kumar. Batch active learning scale. In NeurIPS, 2021.
[27] Jingkang Yang, Kaiyang Zhou, Yixuan Li, and Ziwei Liu. Generalized out-of-distribution detection: A survey. arXiv preprint 1rXiv:2110.1133: 2021.
[28] Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples neural networks. ICLR, 2017.
[29] Shiyu Liang, Yixuan Li, and Rayadurgam Srikant. Enhancing the reliability of out-of- distribution image detection in neural networks. In ICLR, 2018.
[30] Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting -uitr--ditt samples and adversarial attacks. In ICLR, 2018
[31] Weitang Liu, Xiaoyun Wang, John D Owens, and Yixuan Li. Energy-based distrr-dittriuution detection. In NeurlPS, 2020.
[32] Yiyou Sun, Chuan Guo, and Yixuan Li. React: Out-of-distribution detection with rectified activations. NeurlPS, 2021.
[33] Jie Ren, Peter J Liu, Emily Fertig, Jasper Snoek, Ryan Poplin, Mark Depristo, Joshua Dillon, and Balaji Lakshminarayanan. Likelihood ratios for out-of-distribttion detection. In NeurlPS, 2019.
[34] Joan Serrà, David Álvarez, Vicenç Gómez, Olga Slizovskaia, José F Núñezz and Jordi Luque. Input complexity and out-of-distribution detection with likelihood generative models. In ICLR, 2019.
[35] Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Gorur, and Balaji Lakshminarayanan. Do deep generative models know what they don't know? In ICLR, 2019.
[36] Jihoon Tack, Sangwoo Mo, Jongheon Jeong, and Jinwoo Shin. CSI: Novelty detection via contrastive learning on distributionally shifted instances. In NeurlPS, pages 11839-11852, 2020.
[37] Jim Winkens, Rudy Bunel, Abhijit Guha Roy, Robert Stanforth, Vivek Natarajan, Joseph R Led- sam, Patricia MacWilliams, Pushmeet Kohli, Alan Karthikesalingam, Simon Kohl, et al. Con- trastive training for improved out-of-distribution detection. arXiv preprint arXiv:2007.05566 2020.
[38] Vikash Sehwag, Mung Chiang, and Prateck Mittal. SSD: A unified framework for self- supervised outlier detection. In ICLR, 2021.
[39] Sagar Vaze, Kai Han, Andrea Vedaldi, and Andrew Zisserman. Open-set recognition: A good closed-set classifier all you need. In ICLR, 2021.
[40] Jun Shu, Qi Xie, Lixuan Yi, Qian Zhao, Sanping Zhou, Zongben Xu, and Deyu Meng. Meta- Weight-Net: Learning an explicit mapping for sample weighting. In NeurIPS, 2019.
12

---

[41] Alex Krizhevsky. Learning multiple layers of features from tiny images. Master's thesis, University of Toronto, 2009.
[42] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In CVPR, pages 248-255, 2009.
[43] Kuniaki Saito, Donghyun Kim, and Kate Saenko. OpenMatch: Open-set semi-supervised learning with open-set consistency regularization. In NeurIPS, 2021.
[44] Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. LSUN: Construction of large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv: 1 100.. 503.06666,,6... 2015.
[45] Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A
10 million image database for recognition. IEEE Transactions Pattern Analysis and Machine Intelligence, 40(6):1452-144,, 2017.
[46] David Arthur and Sergei Vassilvitskii. k-means++: The advantages of careful seeding. Technical report, Stanford, 2006.
[47] Hanjoo Kim, Minkyu Kim, Dongjoo Seo, Jinwoong Kim, Heungseok Park, Socun Park, Hyunwoo Jo, KyungHyun Kim, Youngil Yang, Youngkwan Kim, Nako Sung, and Jung- Woo Ha. NSML: Meet the MLaaS platform with a real-world case study, arXiv preprint arXiv:1810.09957 2018.
[48] Pan Du, Suyun Zhao, Hui Chen, Shuwen Chai, Hong Chen, and Cuiping Li. Contrastive coding for active learning under class distribution mismatch. In ICCV, pages 8927-8936, 2021.
[49] Dongmin Park, Hwanjun Song. MinSeok Kim, and Jae-Gil Lee. Task-agnostic undesirable feature deactivation using out-of-distributioo data. In NewrIPS, pages 4040-4052, 2021.
[50] Saehyung Lee, Changhwa Park, Hyungyu Lee, Jihun Yi, Jonghyun Lee, and Sungroh Yoon. Removing undesirable feature contributions using out-of-distribution data. arXiv preprint arXiv:2101.0633, 2021.
[51] Jungbeom Lee, Seong Joon Oh, Sangdoo Yun, Junsuk Choe, Eunji Kim, and Sungroh Yoon. Weakly supervised semantic segmentation using out-of-distribution data. In CVPR, pages 16897-16906, 2022.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See the supplementary material.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See the supplementary material.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Sections 4
(b) Did you include complete proofs of all theoretical results? [Yes] See the supplementary material.
3. If you experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material as a URL)) [Yes] See Section and the supplementary material.
13

---

(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section and the supplementary material.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Section 5
4. If you are using existing assets (c.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] See Section
(b) Did you mention the license of the assets? [No]
(c) Did you include any new assets either in the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you' 're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "ze5JTrzyHqh", "reviewer": "Reviewer_joSH", "summary": "This paper introduced the open-set noise problem that might exist in active learning, illustrated the purity-informativeness dilemma and proposed a meta-query (MQ) net as a plugin in normal active learning processes. This paper is well-written before experimental part and easy to follow.", "questions": "Most of the questions are listed in previous part.\n\nQuestions:\n\n1. The definition of open-set noise (see previous part).\n\n2. The motivation/example of purity-informativeness dilemma contains a contradiction (see pervious part). In my view, in ideal case, it is purity rather than informativeness should be emphasized in latter AL processes.\n\n3. In line 63-64, the author said \"The input to the meta-model, which includes the target and OOD labels, is obtained for free from each AL round’s query set by leveraging the multi-round property of AL.\" This is an advantage, learned from the queried ID and OOD samples. But learning is for updating the META model to better output $\\Phi(<P(x), I(x)>;w)$, instead of using it to get better $P(x)$ and $I(x)$, it feels like it is just to train a classifier, but in deep learning tasks, the feature representation and classifier are jointly trained. \n\n4. In line 95, why choose classifier-dependent approach to get a meta-model, what is the motivation? Did the author compared with other density- and self-supervised  based methods?\n\n5. In line 277-278, why use CSI and LL for calculating the purity and informativeness scores? Especially LL, some researches show that LL is not stable due to the joint training with LossNet in some tasks. \n\n6. Is mq-net joint trained with basic classifier (e.g., ResNet18) or not like LL?\n\n7. Equation 3, the situation is both $P(x_i)> P(x_j)$ and $I(x_i) > I(x_j)$, this is apparently to find pareto fronts if one regard it as pareto-optimization problem. Could the author provide some discussions of the situation the number of pareto front set (the data samples that satisfies both $P(x_i)> P(x_j)$ and $I(x_i) > I(x_j)$) less than batch size (labeling budget b in main paper) in active learning process?\n\n8. Is the ResNet18 pre-trained?\n\n9. Did the author conduct repeat trials per experiment? (no error bar problem, mentioned in previous part)\n\n10. The experimental results of the CCAL, SIMILAR and standard AL approaches (mentioned in previous part)\n\n11. In line 284-285, the author already defined the cost of querying ood data samples, why it is not an evaluation metric in later experimental result analysis? It is important if the author could present the cost of querying ood data samples, the less cost means that the labeling budget is less wasted.\n\nHope the authors could provide convincing responses in rebuttal, I will increase my score if the author could persuade me.\n\nSmall typo: Line 261, \"OOD\" not \"ODD\".\n\n\n\n\n\n", "limitations": "The authors adequately addressed the limitations and potential negative societal impact of their work.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: The idea of adding a mq net into normal active learning is interesting, it just like LL4AL, which add an extra plugin/network to estimate whether the unlabeled data set belong to ood data or not. The design is both effective and flexible. Additionally, the L(SQ) inherently contains multi-objective optimization (pareto optimization) design ($P(x_i)> P(x_j)$ and $I(x_i) > I(x_j)$ for finding pareto fronts). It is interesting.  \n\nWeaknesses: there are still several problems left in this paper. \n- Firstly is the definition of the open-set noise problem. I think it is just ID/OOD problems, which is properly defined as class mismatch in CCAL and OOD data scenarios in SIMILAR, there is no need to create a new concept and called it open-set noise problem. This concept is wider, for instance, some instances are really belongs to the ID data distribution but contains noise in x, it is not OOD data but it still contains noise. But the author only conduct experiments follows class mismatch settings in the experimental part. \n- Secondly is the motivation of the purity-informativeness dilemma,  1) the author provide an example in Figure 1. This example is not convincing enough since LL4AL and CCAL are both non-typical methods, LL4AL is jointly trained with a LossNet and CCAL use SimCLR/CSI for extracting features, they are not comparable. Additionally, the example only show the first 10 rounds and show low-noise (10% and 20 % ood rate) situations; 2) Why can't we maintain purity all the time and at the same time acquire high informativeness? In addition, if there is a method to achieve the ideal/ Optimal effect, the proportion of OOD sample in unlabeled data pool will naturally be higher and higher, and more attention should be attached to purity. \n- Thirdly is the experiments, there is no error bar of the conduct experiments. The experimental results on CCAL and SIMILAR are indeed very strange, on low noise situations (10% and 20% OOD data rate), are even worse than typical uncertainty-based sampling strategy (e.g., CONF), I have contacted the authors of CCAL and SIMILAR to asked them if their models would perform worse than typical uncertainty-based measures like (CONF, ENTROPY), the author of SIMILAR paper said \"If there is low-noise then it should only be less challenging and the performance should at least be consistent and better than MARGIN.\". CCAL author show me their new experiments on low-noise data scenarios, also better than typical uncertainty-based measures. Since the author didn't provide the code (only provide the pdf version of appendix), I cannot check the implementation. Is it fair comparison?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "YGrD3gy-Gtp", "reviewer": "Reviewer_enpC", "summary": "Active learning typically involves querying and labeling points according to their informativeness, e.g. some notion of uncertainty or diversity. However, when there is out-of-distribution (OOD) data in the unlabeled dataset, such criteria may result in querying these OOD points, which wastes the labeling budget. Recent work has focused on improving the purity of queried points by filtering out all the OOD data and selecting informative in-distribution (IN) points. This paper challenges the notion that filtering OOD data needs to be done to the same extent at each round of active learning. Instead, it may be true that at later rounds, informative but impure OOD data points could be useful to query. There is hence a purity-informativeness dilemma that presents a dynamic tradeoff in the querying process. The paper introduces a meta-model, Meta-Query-Net (MQ-Net) that learns a score that is a function of both purity and informativeness scores updated after each round of AL using the labeled self-validation set already created from previous rounds. They also enforce a skyline constraint to encode ordering of the scores and implement it via nonnegative constraints. Empirically, MQ-Net outperforms both AL that doesn’t account for OOD data, and AL that is OOD aware but static throughout rounds. They also show how MQ-Net gradually prioritizes informativeness more than purity in later rounds, and perform ablation studies with varying purity/informativeness scores, removing the skyline constraint, and replacing the self-validation set with a random validation set.", "questions": "**Q1.** Given that MQ-Net requires training a model at each round, how long does MQ-Net take to run versus other baselines like CCAL and SIMILAR? \n\n\n**Q2.** A running example of what the OOD data and informative versus non-informative data looks like would be helpful. For instance, Figure 1a explains the purity-informative dilemma, but it is not clear what the dataset and task are.\n\n\n**Q3.** Equation 1 is defined as the optimal query set approach, but is not mentioned otherwise in the paper. It is not clear what the purpose of presenting the upper limit of a query set is. Also, the cost constraint in equation 1 is used in MQ-Net but is not mentioned in that section.\n\n**Q4.** The intuitive interpretation of $L(S_Q)$ was not clear. The main idea from previous sections was that we want a loss that emphasizes informativeness more in later rounds. How does this loss function do that? In particular, it would be good to just have some further discussion about the optimal ranking learned by $L(S_Q)$, case by case. E.g., OOD examples are always scored lower than IN examples, but amongst OOD examples they are roughly ranked by their dot product of purity and information.\n\n**Q5.** Typo: lines 52-53, “The HP-focused approach improves….than the HP-focused one at earlier AL rounds”\n\n\n\n\n", "limitations": "N/A", "rating": 7, "confidence": 2, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n_Significance_: The presence of OOD data can indeed confound standard querying approaches in AL and is an important problem in real datasets. It is unclear how to prioritize potential signal in such data dynamically, which this paper takes a principled step towards.\n \n_Originality_: the idea of using a self-validation set to dynamically meta-score points is novel to me and can inspire new ways of adaptively combining different metrics on unlabeled data.\n\n_Quality_: thorough experiments and ablations convince me of MQ-Net’s utility. \n\n**Weaknesses**\n\n_Quality_: it seems like MQ-Net would be more computationally expensive than previous approaches since a model is trained at each round. It would be interesting to see how much longer MQ-Net takes. \n\n_Presentation_: as someone who does not publish in this area, I had a few comments about clarity (in questions below).\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Wm4PWQptoE", "reviewer": "Reviewer_wpJ5", "summary": "The paper discusses the purity-informativeness dilemma for open-set active learning, where excessively increasing purity can lead to loss of informativenss. As mitigation they propose a new meta-heuristic, which learns to monotonically combine a purity and an informativeness metric, called MQ-Net. MQ-Net is trained on the acquired query sets alternatingly to utilizing it for query selection.", "questions": "Major questions:\n* could you additionally provide the standard deviation across the five different runs?\n* what computing infrastructure was used? In particular, I wonder about the resource requirements for experiments conducted on the ImageNet dataset\n* how do you do the z-score normalization, i.e., over which parts do you compute mean and standard deviation?\n\n\nMinor questions:\n* Figure 4: it appears that all red OOD samples received a high purity score - shouldn't it be vice-versa?\n* in Theorem 4.1., the constraint that the activation function needs to be monotonically non-decreasing is not mentioned (it is however in the appendix). This is an important requirement, and while most commonly used activation functions are monotonically non-decreasing, some, e.g., Swish, are not\n* in the problem statement, you introduce different costs for annotation of OOD and IN examples. To me it is not clear what would motivate this. In the experiments, you then set same costs for OOD and IN labeling, as in related work.\n\nMinor comments:\n* L52-53: One of the \"HP\" should be something else\n* L80: directions -> direction\n* L89: k-MEANS -> k-MEANS++\n* L115: more higher -> higher\n* L120: delimma -> dilemma\n* L187: a -> the\n* L324-325: this could make a good plot, e.g., x=AL round, y=noise ratio, color=slope\n* L363: legit -> legitimate / valid\n* Figure 4 is very small and thus hard to read", "limitations": "Yes, although the discussion is only present in the appendix.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n* the paper addresses a practically relevant problem\n* the proposed heuristic intuitively makes sense, and is compatible with different choices of purity or informativeness measures\n* there is small overhead of the heuristic and it seems to be easy to implement\n\nWeaknesses:\n* in contrast to the answers in the checklist, no code is provided. It is also unclear how hyperparameters of the MQ-Net were chosen, and there is no report about the resource usage.\n* without reporting standard deviations, it is hard to assess the significance of the empirical results - since the authors already report running the experiments five times with different random seeds it is unclear why they did decide to do so; in addition there is also no comparison against the performance of a random acquisition baseline - having it would justify the necessecity of using an active learning strategy at all\n* some parts of the paper could be improved in readability, e.g., in the problem statement, $X_{IN}$ is initially defined as set of in-distribution samples (without their label), but later one, $T_{IN}$ coming from the same distribution $D_{IN}$ contains pairs of (input, target).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "MSq1tHfJ8Uw", "reviewer": "Reviewer_s9UD", "summary": "This paper proposed the \"purity-informativeness dilemma\" problem and a meta-learning algorithm - MQ-Net to adaptively balance the purity and informativeness scores for better sample selection. The experimental results verify the reasonability of the concerned problem and the proposed solution. ", "questions": "As said in the weakness part:\n\n1. Is it necessary to train the MQ-Net in each round? What if we train it in the first round (or first several rounds) and fix it in the remaining round?\n\n2. Figure 4 indicates that the function learned by the MQ-Net is fairly simple. How do the MQ-Net's performances differ from those simple alternatives?\n ", "limitations": "The authors mentioned one limitation: they only consider one purity score and one informativeness score as input. They left the multi-score input as future work. \n\nHowever, the multi-score version is intuitive by adding more input dimensions to MQ-Net. Since the authors had already computed many scores in  Section 5.4, the multi-score version is very convenient to implement. Furthermore, when more scores are used, there would be a score selection problem in MQ-Net. The solution to that problem will increase the quality of the paper. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths \n\n1. The \"purity-informativeness dilemma\" problem is essential, and the analysis is convincing. \n2. The design and implementation of MQ-Net is reasonable. \n3. The active learning architecture is clean and clear, with significantly better performance.  \n\n# Weakness \n\n1. The paper does not discuss the benefit of training the MQ-Net in each round.\n2. The performance comparison between MQ-Net and other simple alternatives is not discussed. Those alternatives may include  heuristic rules like $\\frac{1}{(P(x) - 1) ^2 + (I(x) - 1)^2}$, logistic regressions, and ranking SVM.  \n3. The architecture of MQ-Net is not clearly stated. The activation functions are not reported, and the layer number is only reported in the appendix. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept