Deep Active Learning in the Open World
Authors: Tian Xie
Jifan Zhang
Haoyue Bai
Robert D Nowak
Venue: Accepted by TMLR ()

Abstract: Machine learning models deployed in open-world scenarios often encounter unfamiliar conditions and perform poorly in unanticipated situations. As AI systems advance and find application in safety-critical domains, effectively handling out-of-distribution (OOD) data is crucial to building open-world learning systems. In this work, we introduce ALOE, a novel active learning algorithm for open-world environments designed to enhance model adaptation by incorporating new OOD classes via a two-stage approach. First, diversity sampling selects a representative set of examples, followed by energy-based OOD detection to prioritize likely unknown classes for annotation. This strategy accelerates class discovery and learning, even under constrained annotation budgets. Evaluations on three long-tailed image classification benchmarks demonstrate that ALOE outperforms traditional active learning baselines, effectively expanding known categories while balancing annotation cost. Our findings reveal a crucial tradeoff between enhancing known-class performance and discovering new classes, setting the stage for future advancements in open-world machine learning.

Published in Transactions on Machine Learning Research (02/2025)
Deep Active Learning in the Open World
Tian Xie
Man.. Dwrisc. edu jifan Gcs, wisc. edu batha Scs. wisc. edu rdnowak Stnowak-rrisc..om edu
Jifan Zhang
Haoyue Bai
Robert Nowak
University of Wissonein-Madioor
Reviewed on OpenReview: https: // openreview. net/ forun? id= HkmymFPODD
Abstract
Machine learning models deployed in open-world scenarios oflen encounter unfamiliar con- ditions and perform poorly in unanticipated situations. As Al systems advance and find application in safety-critical domains, effectively handling ont-of-distribution (OOD) data is crucial to building open-world learning systems. In this work, we introduce ALOE, a novel active learning algorithm for open-worke environments designed to enhance model adapta- tion by incorporating new OOD) classes via a two-stage approach. First, diversity sampling selects representative set of examples, followed by OOD detection with GradNorm scores to prioritize likely unknown classes for annotation. This strategy accelerates class discovery and learning, even under constrained annotation budgets. Evaluations on three long-tailed image classification benchmarks demonstrate that ALOE outperforms traditional active learning baselines, effectively expanding known categories while balancing annotation cost. Our find- ings reveal crucial tradeoff' between enhancing known-class performance and discovering new classes, setting the stage for future advancements in open-world machine learning. Code of the work will be be uploaded at httt:://ggghhooo.ooofffffcttttta
1 Introduction
Modern machine learning models have achieved remarkable progress by leveraging large amounts of labeled data (LeCun et al.] 2015 He et al.] 2016 Khosla et al. 2020). Despite this success, most models are developed for soosed-wrlde settings, assuming that both training and test data originate from the same distribution. However, this assumption does not align with real-world environments, where models are inevitably encounter oul-of-distribbttion (OOD) data with previously unseen classes (Hendrycks Gimpel 2022 Hendrycks al.] 20022 Salehi al. (022).. Constrained by their fixed class boundaries, traditional models often struggle to generalize effectively to these novel classes, limiting their adaptability in open-world scenarios. Furthermore, obtaining human supervision for these novel examples in open- world scenarios is often time-consuming and costly, posing significant challenge to model adaptation and improvement. Active learning, which iteratively selects the most informative examples for labeling, has emerged as a promising approach to address the expensive nature of gathering human supervision. By prioritizing examples that provide the most signíficant information gain, active learning can enhance the model's learning efficiency while reducing the need for extensive human annotation. This approach particularly valuable open- world scenarios, where the high annotation cost and time-consuming nature of labeling make traditional supervised learning methods impractical.
Despite its potential, existing active deep learning algorithms (sce Zhan et al. (0022 Zhang et al. (022aa) for overviews) have rarely been studied under open-world scenarios, particularly those involving novel classes and imbalanced data distributions. In this work, we addresses this gap by developing a novel active learning algorithm that intogrates OOD detection techniques with GradNorm scores to handle the complexity of open- world environments. Our method designed for multi-class classification tasks the open-world, where the model encounters both known and unknown classes after deployment. Of the few works that study

---

Published in Transactions on Machine Learning Research (02/2025)
active learning under open-world settings, existing methods are often tailored to specific vision tasks, which result in highly specialized algorithm designs. When these methods are simplified for more generic problems like classification, they often reduce to basic uncertainty or diversity sampling techniques. In contrast, our approach in this paper offers a more comprehensive and versatile solution. By bridging OOD detection and diversity in our sampling stralegy, we provide more comprehensive sampling stralegy compared to approaches that focus on only one of these aspects.
We propose ALOE (Active Learning in Open-world Environments), two-stage algorithm tailored for open- open-world active learning. ALOE addresses the unique challenges of class discovery in open-world settings by dynamically incorporating new OOD classes through a structured sampling strategy: In the first stage, ALOE performs diversity sampling to select a representative set of examples from the data pool. This sampling ensures broad coverage of the data distribution, capturing a range of potential new classes and concepts. By focusing on diversity, the algorithm increases the probability of identifying and learning from rare or infrequent classes that may otherwise be overlooked in a random sampling approach. The second stage leverages GradNorm scoring function to rank examples within each cluster, prioritizing instances most likely to belong to unknown OOD classes. The OOD detection with GradNorm scores provides unified framework that distinguishes between in-distribution (ID) and OOD examples with greater resilience to model overconfidence. By focusing annotation efforts on these high-priority examples, ALOE accelerates the discovery and learning of new classes, even when the annotation budget is limited.
To empirically validate our approach, we conducted experiments on long-tail imbalanced image classification datasets. This choice of datasets is motivated by the prevalence of long-tail distributions in real-world scenarios, where rare classes concepts are often underrepreeented. Such imbalanced distributions make random sampling ineffective in discovering unknown classes, particularly those with small sample sizes. Our experimental results demonstrate the effectiveness of our approach. Compared with random sampling, on ImageNet-LTT, ALOE saves 70% of annotation cost to achieve the same accuracy. In six out of six experimental sottings 5.222 and 5.2.3), we observed that our algorithm performs the best comparing to all baseline experiments. These highlights underscore the potential of our method to significantly enhance model adaptation in open-world environments.
Lastly, our work reveals novel tradeoff between improving performance on known classes and discovering new ones. This finding opens up important avenue for future research, as balancing these competing objectives is crucial for developing truly adaptive Al systems. In summary, our proposed active learning algorithm offers a promising solution for adapting machine learning models to new, previously unseen con- ditions in open-world environments. By efficiently incorporating unknown instances and minimizing human annotation efforts, our approach paves the way for more robust and adaptable AT systems capable of adapting to dynamic, real-world scenarios.
2 Related Work
The carliest machine learning methods typically employed passive learning in a closed-world setting, where the model passively received training data and all data was fully labeled. However, in modern research and practice, it is common to encounter situations where f) some all of the data is unlabeled, ii) the unlabeled data contains new classes, ti) it is necessary to actively select training data. Our research combines these aspects and proposes novel approach tailored to these scenarios.
2.1 World Learning
Open-worke learning addresses the challenge of operating in dynamic environments where the model starts with a set of known classes and must detect and manage instances from unknown classes. This paradigm involves both labeled and unlabeled data, with a mixture of known and unknown classes, requiring the
model to either classify examples into known categories or recognize them as novel (Rizve et al. 2022 Zhu
et al., 2024 Xiao et al. 2024). Unlike novel class discovery that focuses solely on discovering new object categories (Fini al. 2021 Zhong et al. 2021b Han et al. Zhong al.) 2021a| Roy al. (2022). open-world learning must also correctly identify instances from previously known classes, making the task more complex.
2

---

Published in Transactions on Machine Learning Research (02/2025)
Open- woold semi-supervised learning (Cao al. 2022 Sun et al. 2024 generalizes semi-supervised learning by considering scenarios where the data includes both labeled and unlabeled instances from known and unknown classes. The model must learn to classify known classes and identify novel ones from unlabeled data. In contrast to traditional supervised scttings, the ability to generalize to unseon classes is crucial. Open- set recognition shares similarities with open-world learning by allowing the model lo reject novel instances during testing (Ge et al., 2017: Sun et al. 2020 Geng et al. 2020). However, open-world recognition extends this by requiring the model to incrementally learn and incorporate these novel classes into the set of known
categories (Boult et al., 2019 Bendale & Boult) 2015). Open-world contrastive learning further enhances this by learning compact representation spaces that facilitate both the classification of known classes and the discovery of novel ones (Sun & Li. 2022).
Open-world learning represents a significant advancement due to its capacity for active instance selection. In this setting, the model actively selects instances from unknown classes for labeling, allowing it to contin- uously expand the set of known classes as new data is labeled. Earlier work by [Bendale & Boult (2016) laid the foundation for open-world recognition; introducing the Nearest Non-Outlier (NNO) algorithm and estab- lishing evaluation protocols for managing novel classes in continual learning environment. More recently, comprchensive review by Zhu et al. (2024) highlighted three core components for open-world systems: rejecting unknowns, discovering novel classes, and incrementally learning from them. These components form the basis of many modern approaches to open-world learning. Our research extends this by integrating open-world learning with long-tail distributions and active sample selection, enabling the model to not only handle class imbalance but also dynamically evolve as new classes emerge.
2.2 Out-of- of-Distiibuutin Detection
Out-of-distribution (OOD) detection is a fundamental challenge for machine learning models deployed in open-world environments. It involves identifying whether inputs belong to unknown classes that the model has not encountered during training. The overconfidence of neural networks when handling out-of- distribution data was first revealed in Nguyen et al. (2015). Research in OOD detection has taken several main directions: post-hoc methods that devise scoring functions for detecting OOD inputs (Liu et al.) 2020)
Loe ct al.] 2018 Sun et al., 2022). training-time rogularization methods that leverage additional auxiliary
OOD datasels to address OOD delection (Hendrycks et al., 2018 Van Amersfoort et al.) 2020 Kalz-Sanuels
et al. multiview contrastive losses Khosla et al. 2020: Chen et al. 2020 for OOD detection (Winkens et al. 2020
2022 Bai et al. 2023 2024], and approaches exploring representation learning, such as exploring
Schwag et al. 2021 Ming et al.] (022)..
In particular, post-hoc methods address OOD detection by deriving test-time OOD scoring functions for pre-trained classifier. These methods include maximun softmax probability (Hendrycks & Gimpell 2016).
distance- based scores (Lee et al.] 2018 (Snn et al., 2022), energy-based scores (Liu et al.] 2020), activation rectification {Sun et al. 2021), ViM score (Wang et al. 2022a), gradient-based scores (Huang et al. 2021) among others. The GradNorm (Huang et al. 2021) score considers the gradient information providing a direct measure of how uncertain the model is with respect to its parameters. this work, we employ the GradNorm score for input examples to identify OOD data and cluster patterns, with the aim of effectively discovering and learning new classes from the data distribution.
2.3 Deep Active Learning
Active learning studics the problem of minimizing annotation cost while training high performance models. Active learning methods typically follows a sequential and adaptive procedure, where the algorithms first trains model based on the annotated examples far followed by annotating more examples selected from large number of unlabeled examples. The algorithm strategically chooses from the unlabeled examples for annotation, typically relying on uncertainty, diversity or expected model change types of metrics.
Uncertainty based strategies chooses examples that are determined to be the most uncertain for the model trained on the labeled sot thus far. This includes many of the traditional uncortainty metrics such as margin, entropy and confidence scores (ewis & Gale. 1994 Tong & Koller 2001 Balcan et al. 2006 Settles 2009;
Kremer al. 2014 Wang et al. 2022b). More advanced approaches for deep learning also includes Bayesian
3

---

Published in Transactions on Machine Learning Research (02/2025)
Beluch et all Diversity based strategies aim to choose set of unlabeled examples that are maximally lly different in an embedding space. This is usually achieved by clustering and covering techniques or greedy optimization of some global submodular diversity objective (Sener & Savarese 2017 Geifman & El- Eanyy 2017 Byyk et al.
uncertainty estimation and adversarial training Gal et al. 2017 Ducoffe & Precioso 2018
2018).
2019 Citovsky et al. 2021 Bhatt et al. 2024). In this paper, we explore a wide range of these diversity-
methods in conjunction with OOD detection to discover new classes. Lastly, existing approaches also take a
combination of uncertainty and diversity metrics, often predicting the expected model change if an example
is labeled (Ash et al. 2019 2021 Wang et al., 2021 Elenter et al.
2022 Mohamadi et al. 2022).
Active learning has recently focused much attention on the closcly related field of open set learning KKram--
cheti et al.
2021 Ning et al. 2022 Zhang et al.] 2022 Bai et al. 2024 Salaei et al.) 2024 Yang et al. 2024;
Mao et al., 2024). In open set learning, the primary objective is to classify all examples of unknown con- cepts into a single out-of-distribution class. Our study, however, addresses the more challenging open-world scenario. In this setting, we aim for the learner to not only identify unknown concepts but also to further learn and classify these examples into their appropriate concept categories.
The challenge of missing categories during initial training commonly arises when class sizes follows long tail distributions. In this paper, we study this exact realistic setting of open world learning with long tail data distribution. A large body of deep active learning literature has focused on the imbalance dataset
settings (Aggarwal et al., 2020 Kothawade et al.) 2021 Emam et al., 2021 Coleman et al. 2022 Jin et al.
2022
Cai 2022:
Nuggehalli et al.) 2023 Zhhang et al. 2024b (ooltani et al.) 2024 [hhang & Nowak (2024).
However, none of these work have studied the open world setting, where a subset of the classes are initially
unknown to the algorithm. Below, we survey the few task-specific active learning algorithms that are proposed for woen-world scenarios.
2.4 Open- World Active Learning
Overall, there has been few attempts in studying active learning under an open world setting. Ma et al. (2024) also studies the active learning setting where some classes do not have labeled examples. However, their paper assumes the knowledge of the total number of classes beforchand, making their study closer to the traditional active learning than truly addressing the open world challenge, where the total mumber of classes is agnostic to the learner. As we will discuss Scction f not knowing the number of classes poses a challenging tradeoff between exploring new classes and learning existing classes well.
Other existing literature shares our setting but targets specific tasks. Chen et al 2023 propose OpenCRB for worn-world 3D object detection, proposing an uncertainty-based scoring method that leverages the spatial distribution and density of data points for 3D point clouds. In the more general set tings, this work essentially reduces to simple uncertainty methods, which we will show to be less effective than our methods in open world settings. Zamzmi et al. (2022) utilizes a simple diversity- based method, k-medoids clustering, in annotating a diverse set of images for cchocardiography view classification. As we will show, our method that combines OOD detection and diversity- based met is superior than only considering diversity alone. In Section 5.2.1 and Table 2 provide details of the OOD scores we conduct experiments on.
3 Problem Setup
We consider pool-based batched active learning setting in an open-world scenario. The learner has access to a large pool of unlabeled data X = {#1, #2,,,,y ) and the true label distribution is defined by an unknown ground truth function f X + {1, 2, ..., K}, where K is the total number of classes in X. At the beginning the labeling process the labeled set L is set to be empty. At each iteration t, the learner selects a small batch of B unlabeled examples, (x:"
(t) ) C X, from the pool. The learner observes the labels {f {x=(x()) ) and adds these examples to L, the set of labeled examples available for training so far. In an open world setting, examples in L may only cover a fraction of the classes, which we denote by K S [K], the known classes. After cach batch query, the learner trains |KT classification model fo on L, which is used to inform the selection of the next batch of examples for labeling. Furthermore, we let & (z) denote the softmax distribution ion score based on the classification model ft.
4

---

Published in Transactions on Machine Learning Research (02/2025)
This iterative process continues, with the goal of selecting the most informative examples to annotate in each round, minimizing the number of labeled examples needed to achieve good generalization performance. The generalization performance is measured by the balanced accuracy on all K classes. If some of the classes do not have any annotated example, the accuracy for such classes are considered 0. Therefore, it crucial to annolate a wide array classes, while also learning the known classes well based on the annotated examples.
3.1 Out-of- of-Diitriibtiin (OOD) Scores
During each step t of the active learning process, recall K'' represents the set of classes that have at least one labeled example. When training a model using the labeled dataset, examples from classes not in IC (i.c., [K') are considered out-of-distribution (OOD) by definition. Traditional OOD detection research focuses on identifying OOD examples within a test set. In this study, we use an OOD scoring function, denoted an example and is any neural network classification model. This function Mxx, f) provides mcasure of how likely an example is to be in-distribution (ID) versus OOD. A higher OOD score suggests a greater likelihood that the example is OOD, which in lurn is more likely to belong to unknown class. We apply these OOD scoring techniques to determine which unlabeled examples are most likely to belong to unknown OOD classes.
as (xx, f), where
4 Methodology
To address the open world active anno- tation problem, we focus on two key ob- jectives: i) maximizing the selected ex- amples' coverage of OOD classes; i)) op- timizing the labeling budget efficiently collect examples from these OOD classes. A straightforward approach would be to simply annotate examples with the high- est OOD scores during each round of an- notation. However, our analysis in Fig- ure reveals that examples with high OOD scores often cluster within a small subset of OOD classes, making this ap- proach suboptimal for achieving broad coverage. This observation suggests the need for diversity- strategy to en-
Figure Visualization of highest OOD score unlabeled exam- ples after training on three known classes in CIFAR-100. As we can see, relying only on OOD scores for selection will encourage annotation of unlabeled examples in few (unknown) classes.
sure our annolated examples span a wide range of concepts.
In Section 4.1 we present ALOE, a two-stage approach that combines OOD scoring with diversity- active learning stratogies. ALOE first employs clustering-be basod diversity methods identify distinct groups of examples, then filters these clusters to retain only high-scoring OOD candidates for annotation. Our experiments show that applying diversity clustering before OOD filtering yields better results than the alternative stratogy (sce Scction 4.2) that exccutes these steps in the reverse order.
4.1 ALOE: Our Actively Learning Algorithm Under Open-world Environments
In this section, we describe our proposed algorithm for querying unlabeled data in pool-based batched active learning framework under open-world conditions. As detailed in Algorithm [ algorithm ALOE leverages OOD scores and k-means clustering method to ensure balance between identifying novel OOD examples and maintaining diversity within the queried batch.
The algorithm proceeds iteratively for T iterations, with each iteration comprising several key steps, as illustrated Figure Initially, deep neural network undergocs trained on the current labeled set L,-1- resulting in an updated model ft-1 that incorporates information from the newly labeled data. Subsequently, to ensure diversity in the selection process, the algorithm embeds the pool of unlabeled set X' using the neural
5

---

Published in Transactions on Machine Learning Research (02/2025)
Cluster Known Class Low OOD
Cluster 2 Known Class
Low OOD
Clustering
Cluster Unknown Class High OOD
Cluster 4 Unknown Class High OOD
Data Pool
Choose clusters
and * *: Labeled sample from known classes and Unlabeled sample from known unknown classes
Illustration of our algorithm ALOE in Algorithm [ Starting with a few labeled examples, the algorithm clusters all examples, idcally by their underlying classes. Each example's OOD score is calculated. In the bottom right plot, larger dots mean larger OOD score. Clusters with higher OOD ratios are prioritized for sampling to identify new classes. Labeled examples are then added the training set, and the process iterates, expanding the labeled pool with each batch.
Figure 2:
network features learned from f-11. The clustering method, k-means, is then applied to these embedded features. The number of clusters 2 max (B, |Acl) is set so that we obtain surplus of clusters to effectively filter out the in-distribution examples, where B is the batch size and [Ke] is the number of annotated classes in step t. Empirically throughout our experiments, we find the multiplier value to be a good and robust value. can be imagined that other clustering methods can be used. While will be discussed more in our experiments in Section 5.2.5 we stick to k-means when describing ALOE.
Following the clustering phase, we define the OOD cluster ratio as the proportion of predicted out-of- distribution (OOD) examples within each cluster, where predictions are made using an OOD scoring function ((xxf(111 : X + RR defined in Scction 5.2.1 To identify OOD examples, we establish a threshold at the 95%-TPR cutoff based on the in-distribution labeled examples. This threshold is commonly used in the OOD detection literature, and ensures at least 95% of the labeled examples are classified as ID. The clusters are then ranked based on their OOD cluster ratio. The top B clusters are selected for further processing, and from each of these selected clusters, the example with the highest OOD score is chosen form the final batch (4) of examples for annotation.
The iteration concludes with update phase where the labeled set L, is augmented with the newly labeled data, and these examples are correspondingly removed from the unlabeled pool U,. Through this iterative process, the labeled set gradually expands to include a diverse range of both in-distribution (ID) and out- of-distribution (OOD) examples, thereby enhancing the classifier's 's ability to generalize to both known and novel classes that were unknown at the beginning of the annotation process.
4.2 Reverse ALOE: Alternative Query Strategy
In addition to the approach described above, we also considered an alternative method reverse ALOE. As the name suggests, the order of using OOD scores and clustering methods are flipped. OOD scores were used first to select an initial set of candidates in this algorithm. Similarly with ALOE, we calculated a threshold t, which ensures a 95% true positive rate (TPR) in labeled set Lμ- The initial set are examples with OOD score larger than the threshold

---

Published in Transactions on Machine Learning Research (02/2025)
Algorithm 1 ALOE: Actively Learning in Open-world Environments
Input: Pool of examples X, initial labeled set L1 covering subset of total classes K1 C [K], initial
unlabeled U1 X L1, number of iterations T, batch size B.
Define: OOD scoring function Q X IR, which maps examples their likelihood of being OOD based the current labeled set L. for t= .. do
Fine-tune the large pretrained neural network on Le to obtain the model f(-1 Step 1: Embedding and Clustering for Diversity
Embed examples in unlabeled set Un using neural network features from ft-1 and apply k-means clustering with A == 2 max(B, [Ke)), resulting in clusters X, for each c c= c=
Step 2: Identify Clusters with High OOD Probability
Rank clusters based on the ratio of likely OOD examples to cluster size. For cach chuster Xe, the ratio of OOD examples is calculated as:
Exex. 1(2(x, ft=1)===- T)
(c) 7 FOOD
XXdl
where [ ft-1) > r) indicates whether the OOD score [(x, fz of an example æ exceeds the threshold 7, which ensures a 95% true positive rate (TPR) in the labeled set Lt, and [X,] is the total number of examples in cluster Xe. Pick B clusters with the highest OOD ratio.
Step 3: Query Examples from Chosen Clusters
From each of the B selected clusters, pick the example with the highest OOD score. This forms the query set ((++))
Annotate examples in X(+1) and update the labeled and unlabeled sets:
L L, ux(++), U111 + Ux(x(t1),
end for
Return: Fine-tune the pretrained model on the final labeled set Lr to obtain the final classifier (())
These candidates were then clustered with hods described in Section 5.2.5 to ensure diversity among the queried examples with the cluster number equal to batch size. In each cluster, the sample with highest OOD score is queried to be labeled. However, this method performed poorly in our experiments (Section 5.2.2 it did not select super diverse set of examples, reducing the overall efficiency of the active learning process.
5 Experiment
5.1
Experiment Setup
Dataset.
Open world challenges often arise from dataset imbalance with a large number of classes, where smaller classes are often unknown at the beginning of the annotation process. In this paper, we focus our ex- periment evaluations on common long tail imbalanced datasets with a large number of classes. Specifically, we utilize three image classification benchmark datasets, CIFAR100-LT (Alex 2009 ImageNet-LT (Deng et al. 2009) and Places365-IT (Zhou et al. 2017). The distribution of the three long-tailed datasets are given Appendix 7
Model. For evaluation, we follow the latest LabelBench framework (Zhang et al. 2024a). while introducing the new open world setting with dynamic number of classes at each iteration. Specifically, we fino-tune the pretrained CLIP ViT-B32 image encoder (Radford et al.) 2021 with a linear classification head attached. For every iteration of the active learning a algorithm, the model is reinitialized to the pretraining checkpoint and finetuned end-to-end on all labeled examples thus far.
In our experiment, we utilize the cold starting approach by reinitializing the model from the CLIP model checkpoint after every iteration of annotation. Speeifically, for labeled set La with |AC known classes, we
7

---

Published in Transactions on Machine Learning Research (02/2025)
attach linear head on the CLIP image encoder model with |AC]] outputs. We then finetune this model the labeled set La for each iteration 1.
Metric. To evaluate the performance of our method in the open-world active learning setting, we use two primary metrics: the number of annotated classes and the balanced accuracy. The number of annotated classes (denoted as |Kc] Section B is crucial as missing certain categories could hinder the model's per- formance when deployed in practice. On the other hand, balanced accuracy gives more wholistic view of the model's generalization performance. Specifically, given test set of examples (z)., (x'M V's) and model f mapping images to classes, the balanced accuracy is also known the average recall
K K
ACC
1
Σ 1((x() <:
k}
NE
where K is the total number of classes, N& is the number of examples in class k. The balanced accuracy simply averages the classificr's recall of predicting each class. Note that if the classifier does not prodict certain class, the accuracy for that class is 0.
Software and hardware. Our method is implemented with PyTorrc 2.2.0. All experiments are conducted on NVIDIA TITAN RTX for CIFAR100-LT and Placos365-LT, and NVIDIA A100 for ImageNet-LT.
Setup Summary. The experimental setup for each dataset is summarized Table [ We conduct experi- ments over 7' iterations, with a batch size of B and an initial labeled set L1 covering subset of classes K₁. For each of the following settings, we start with a small set of initially labeled classes ([K]l) to simulate a real-world open-world learning environment, where the model has limited knowledge of the complete label space at the beginning. The classes are chosen to be the largest [Ka] classes in size and the initial batch of labeled examples are distributed evenly across these classes.
Table 1: Experimental settings for each dataset
Dataset
Initial # Classess (KK,)
Total # Classes
#Iteration (T)
Batch Size (B)
CIFAR100
3
100
15
50
ImageNet-1KK
10
1000
10
1000
Places365
3
365
10
200
CIFAR100
10
100
15
50
C1FAR100
30
100
15
50
CIFAR100
50
100
15
50
Additionally, in the last three rows of Table I we show the settings of ablation studies on CIFAR100-LT by varying the initial number of labeled classes (IKa] 10, 30, 50) to assess the robustness of our algorithm across different scenarios. These studies highlight our method's ability to adapt to varying levels of initial knowledge while maintaining strong performance across different settings.
5.2
Main Results and Analysis
5.2.1
OOD Scoring Functions and Baseline Active Learning Algorithms
Before we present our experimental results, we introduce the OOD scoring functions and bascline active learning algorithms we use.
OOD Scoring Functions: In Table 2 we summarize the OOD scoring methods from previous literature, which are used throughout our rescarch. Energy (Liu et al. 2020) score provides global view of the uncertainty by aggregating information over all classes. These scores are particularly useful when the model outputs soft probabilities that span multiple classes. Margin (Scheffer et al.) 2001]) scores offer a more focused view by comparing only the top two predicted probabilities, making them sensitive to near-boundary decisions. GradNorm (Haang al., (021)) considers the gradient information, which can provide a more direct. measure of how uncertain the model is with respect to its parameters, especially in deep learning
8

---

Published in Transactions on Machine Learning Research (02/2025)
models. Mahalanobis distance [Lee al. 2018 is more geometric, considering how far given example is from the expected distribution of particular class. This distance metric is often used in embedding spaces. Gradient-based (Bai et al. 2024) score measures the model's sensitivity to input perturbations by calculating gradients, capturing how the model would change with slight variations.
Table 2: OOD scores used in the OOD score ablation study.
OOD Scores
Definition
Description
Energy Score
Anergy (s)
log Σ exp (F)
Aggregates log probabilities over all classes, providing measure of over- all uncertainty.
Margin Score
SMargin (x) Pimax (1) P Psooond (z)
The difference between the highest and becond-highest predicted prob- abilities, Fmax(1)) and Precond (x), fo- cusing on boundary cases.
GradNorm
GrradNorm(() = VeC(x; f)ll
Norm of the gradient of the loss func- tion L with respect to model pa- rameters 0, providing a measure of parameter-space uncertainty.
Mahalanobis Distance
9 Mahalannbbs (3) (2- [e0)² (2 Pc)
Distance from the class mean of em- beddings Pe. for correspouding class Σ is the class covariance matrix. Projection of the difference between sample gradients and their average onto the top singular vector of gra- dient matrix, made by stacking gra- dients of all data. V denotes the av- erage gradient across dataset.
Gradient-Based
Craddiet (VV((v((x;;) V,v)
Score
Baseline Active Learning Algorithms: We evaluated our algorithm against. several standard active learning baselines: Random, Margin, Coreset, Galaxy, Badge, ProbCover, TypiClust, and EOAL. Random is a straightforward baseline where examples are selected uniformly at random from the pool of unlabeled data, offering a comparison with non-strategic sampling. Margin (Schefler et al. 2001 is an uncertainty-based sampling method that selects examples based on the difference between the top two pre- dicted class probabilities. This approach aims to identify samples near decision boundaries, where the model is most uncertain. Coreset (Sener & Savarese 2017) is a diversity-based approach that seeks representa- tive subset of the unlabeled pool by minimizing the maximum Fuclidean distance between any point in the dataset and the nearest selected point in the subset. Badge (Ash et al. 2019) combines uncertainty and di- versity by selecting examples based on gradient embeddings to cover informative examples. Galaxy (Zhang et al. 2022) is specifically designed for imbalanced scenarios, using a balanced sampling strategy to improve performance on classes with fewer labeled examples, making well-suited for imbalanced datasets. Prob- Cover (Yehuda al.] 2022) adopts covering-based approach to maximize probabilistic coverage over the data distribution, effectively identifying critical examples under constrained labeling budget. It is impor- tant to note that ProbCover is highly sensitive to its hyperparameter. As active learning can only onnce when deployed in practice, we selected single value that is reasonably suitable across various scenarios. TypiClust (Hacohen et al.] 2022) is a low-budget active learning method that priorilizes selecting samples from dense regions of the feature space, ensuring representative coverage even with limited labeling resources. Entropic Open-set Active Learning (EOAL) (Safaci al. 2024) addresses open-set scenarios by employing entropy-based selection to handle both known and unknown classes effectively, ensuring robust performance in open-world settings. The time cost analysis of all baselines and ALOF is included in Appendix C
9

---

Published in Transactions on Machine Learning Research (02/2025)
5.2.2 Main Results Multiple Datasets
We now present the results of our algorithm applied to CIFARIOO-LT, ImageNet-LLT, and Places365-LT. As mentioned before, the long-tailed distribution of classes well-known challenge practical applications. Due to this issue, we often encounter open world learning scenarios where rare classes are likely to be missed during the initial annotation phase. In the following experiments, receive annotations from only small number of classes in the initial batch of annotation (three for CIFAR100-LT and Places365-LT, and len for ImageNeetLTT). In addition, our algorithm ALOE uses GradNorm for OOD scoreing function S and k-means for clustering method. We also include ablation studies around different initial number of classes, OOD scores, and clustering methods the later sections. In Table 3 and Table 4 balanced test accuracy on the three datasets are shown. Number of annotated classe and the plot of the results are summarized Table Table and Figure in Appendix B.
Table 3: Balanced test accuracy CIFAR100-LT and Places365-LT with different budget. (The standard error was calculated based on four trails.)
Dataset
CIFAR100-LT
Places365-LT
Budget
250
750
600
2000
Random
0.39240.008
0.55340.004
0.20840.007
0.275:10.004
Margin
0.445.1.000
0..43
0.217:1.005
0.287:10.004
Badge
0.447=0.000 .006
0.65220.006
0.231 #0.004
0.295=#0.006
Galaxy
0.436=:0.000
0.645+t0.00s
0.231 ::0.004
0.302
Coreset
0.449:40.012
0.61 70.010
0.238:10.003
0.304
ProbCover
0.346+0.00 .10
0.442+0.000
0.152 00007
0.2477=0.008
TypiClust
0.376# 0.51
0.509
0.199
0.275
EOAL
0.368++.00m 00s
0.578
0.211
0.277=+0000
ALOE(Ours)
0.479=0.007
0.650=0.04
0. 0.254+0.00
0.310=0.000
Table 4: Balanced test accuracy on ImageNet-LT with different budget. (The standard error was calculated based four trails.)
Dataset
ImageNet-Iie
Budget 3000
10000
Random
0.391+0.002
0.541
Margin
0.424
0.562
Badge
0.434410.004
0.581 #0.002
Galaxy
0.438
0.631110.o
Coreset
0.351+0.005
0.525
ATOOE(Ours)
0.51740.004
0.617:2..00
As shown in Table 3 and Table our method ALOE significantly outperforms all baseline methods in both balanced accuracy and the number of newly discovered classes. Starting with only three initially labeled classes, our method quickly expands the set of annotated classes, maintaining a higher discovery rate throughout the iterations. By leveraging the combination of OOD detection and diversity- ALOF is able to efficiently explore unknown classes while ensuring strong performance on the known classes. On ImageNet-LT, ALOE saves 70% of annotation cost to achieve the same accuracy comparing to random sampling. We also note that while some baselines occasionally match ALOE's performance, they struggle with consistency across different datasets. In contrast, ALOE consistently performs best among all methods on all datasets.
sampling,
We also experimented with Reverse ALOE, which doesn't 't show sufficient label elficiency comparing to other baselines. The Reverse ALOE result shown in Figure uses the combination of GradNorm OOD score and Coresct clustoring method, which behaves best in all combinations. While still competitive with some basclines, Reverse ALOE does not match the performance of ALOE. This performance gap likely stems from conducting OOD filtering first, which eliminates many classes of out-of-distributiio examples early in the
10

---

Published in Transactions on Machine Learning Research (02/2025)
process. As result, Reverse ALOE works with less diverse set of examples compared to ALOE, which preserves diversity by clustering examples before performing OOD filtering.
Interestingly, the advantage of ALOE becomes more pronounced with datasets that have a larger number of classes, espiccially on earlier iterations of experiment. ImageNet-LT, which has the highest mumber of classes (1,000), demonstrates a noticeable gap between ALOFF, and other baselines. For CIFAR100-LT and Places365-LT,, with 100 and 365 classes respectively, ALOE still performs well, although the difference is less striking. This trend reflects the inherent nature of ALOE, which excels at identifying new classes.
Another interesting observation is, on ImageNet-I.T, ALOE outperforms all baselines until the annotation budget around 7000. However, when most classes have been discovered, GALAXY slightly outperforms ALOE. Similarly, on Places365-LT, ALOE achieves strong performance in terms of balanced test accuracy, but did not find new classes as fast as Badge in later iterations. This reveals an interesting paradigm shift in active learning between discovering new categorics /cluusssrs of examples and learning the existing sengorris/cluster of examples. We discuss this phenomenon further in Section 6 ass novel challenge through the lens of open world scenarios.
5.2.3 Ablation Study of Initial Number of Annotated Classes
In Figure 3
we analyze the robustness of our algorithm by varying the initial number of annotated classes on CIFARIOO-LT. We experiment with three settings, including the initial number of annotated class being 10, 30 and 50 respectively. As observed in this experiment, ALOE consistently performs the best among all algorithms and is robust to different initial number of classes. It is also worth noting that despite maintaining strong performance, the improvement over the second best algorithm narrows slightly compare to the experiment in the previous section, where only three classes are initially labeled. This observation also corroborates our discussion the paradigm shift between discovering more classes and learning existing classes. As mentioned before, we further discuss this challenge in Section B
0.63
0.65
Balrrod Text Accurscy
0.0.
Ten Accuaryy
Balaned Test Accunty
0.0
0.55
0.55
0.50
0.50
0.45
0.45
0.40
0.85
0.35
0.50
0.30
0.35
Number Labels
Yamber Labels
Number Labels
IC0
100
100
Namber of Annorated Chea
00-
30-
Number of Amotrted Clases
30
Number Ammorated
80
Random
ProvCover
20
Margi
TypiClett
30
Burge
EOAL
50-
59
Galaay
ALOE (Ouss
m0
Coreset
3)
56
Number Labeis
Labels
Number of Labels
(a) KG]] 10
(b) [AG]] 30
(c) [AG = 50
Figure3 3: With different initial number of annotated classes, balanced test accuracy and number of annotated classes of CIFAR100-LT. (All subfigures share the same legend of the left bottom one. Shaded region represent the standard error conducted across four trials.)
11

---

Published in Transactions on Machine Learning Research (02/2025)
5.2.4 Ablation Study of OOD Scores
Figure 1 shows that different OOD scores affect the performance of our algorithm. On CIFARIO0-LT, Energy and GradNorm scores perform similarly. However, we used GradNorm to carry out our main experiments Section 5.2.2 since it direct reflection on the uncertainty of the model, espiecially deep learning models.
0.7
0.6
Number Aerocatee Camax
0.5
0.4
talancd Tes
Mahuinobis
Matalanchic
Magie
Magn
Gradart-boeed
Guslett based
Energy
GoafNorn
GracNorm
Number Labes
Labels
Figure 4: Ablation experiment for using different OOD scores (2z, f) in ALOE on CIFARIOO-LT, (Shaded
region represents the standard error conducted across four trials.)
5.2.5 Ablation Study of Clustering Methods
We also investigated various clustering methods. We experimented with a popular clustering method used in prior active learning algorithms. For instance, TypiClust (Hacohen et al. 2022] employs k-means, Coreset (Sener & Savarese 2017) utilizes k-center, and BADGE (Ash et. al. 2019) uses k-means We also implemented gaussian mixture models and mini-batch K-means as the clustering subprocedure. Both these are more computationally efficient than simple k-means, and obtain similar performance as the procedure using k-means. As shown in Figure this indicates that our algorithm is not highly sensitive to the choice of clustering algorithm. In our implementation, computational cost compared to other clustering methods. However, in most active lcarning scenarios, the selection cost is siguificantly lower than the cost of training the neural network. In the rare cases where k-maans becomes impractical, we recommend practitioners consider alternatives such as Gaussian Mixture Models (GMM) or mini-batch k-means.
used k-means despite its relatively higher
0.6
Namber Aovetatte Clesse
50
Biboood Text Aicuoay
40
0.5
73
0.4
60
0.3
GWM
lemansi-t
CVM l-masss
10
koaster
butch knsurn
0.2
30
230
40c
sio
Number of Label
"umber Labds
Figure 5: Ablation experiment for clustering methods on CIFAR100-LT with GradNorm OOD score. (Shaded region represent the standard error conducted across four trials.)
6 Balancing Novel Class Discovery and Known Class Learning
When running ALOE further larger amount of annotation budget, our experiments (Figure 6 demon- strate an intriguing phenomenon - bascline methods can even achieve slightly better performance than ALOE once most classes have been identified. Similarly, as mentioned in Section 5.2.3 on CIFAR100-LT, when the number of initial classes increases, the improvement gap of using ALOE narrows. These findings are expected, as ALOE uniquely prioritizes annotating diverse OOD classes, while baseline approaches focus on learning known ID classes. While this suggests the potential for an algorithm that excels at both tasks, achieving this dual objective presents significant challenges.
12

---

Published in Transactions on Machine Learning Research (02/2025)
Clan
Magh
(000:00e11
JLDE (Cw)
1.00
Figure 6: Balanced test accuracy and number of annotated classes CIFAR100-LT when batch size B = 200 and number of iteration T = 10. (Shaded region represent the standard error conducted across four trials.)
This challenge arises because the total number of classes is unknown to the learner. Even after most classes have been discovered in our ImageNet-Li" and Places365-LT experiments, the learner cannot determine whether significant numbers of classes remain undiscovered. Furthermore, our experiments demonstrate that ALOF, while perform the best during the class discovery phase, may not be the ideal strategy for learning in-distribution (ID) classes. This reveals a fundamental tradeoff in open-world active learning: the need to balance annotation between exploring novel classes and consolidating knowledge of existing ones.
Future research could approach this challenge from theoretical perspective by developing strategies that dy- namically alternate between exploration and consolidation. One promising direction is an iterative approach that first ensures discovery of classes above a certain size threshold, followed by focused learning of these identified classes. This process could then repeat with progressively lower the size thresholds, ultimately yiclding a model that performs well across classos of varying frequencics.
7 Conclusion
In this paper, we propose ALOE, a novel active learning algorithm specifically designed for open-world scenarios. Our approach leverages two-stage process that combines diversity-based sampling with out- of-distribution detection, enabling the discovery and learning of new classes in a dynamic environment. Through experiments three long-tailed datascts, ALOE demonstrates consistent and clear advantage over existing baselines in both balanced accuracy and class discovery. The results also highlight crucial tradeoff: while identifying new classes is important, effectively learning from previously discovered classes presents a competing challenge. This remains an unresolved issuc that warrants significant research attention.
Acknowledgement
This work has been supported by ONR MURI grant N00014-20-1-2877.
13

---

Published in Transactions on Machine Learning Research (02/2025)
References
Umang Aggarwal, Adrian Popescu, and Céline Hudelot. Active learning for imbalanced datascts. In Pro- ceedings of the IEEE/CVE Winter Conference on Applications of Computer Vision, pp. 1428 1437, 2020.
Krizhevsky Alex.
Learning multiple layers of features from tiny images. https://www. cs. toronto. pdf, 2009.
Jordan Ash, Surbhi Goel, Akshay Krishnamurthy, and Sham Kakade. Gone fishing: Neural active learning with fisher embeddings. Advances in Neural Information Processing Systems, 34:8927 8939, 2021.
Jordan T Ash, Chicheng Zhang, Akshay Krishnamurthy, John Langford, and Alekh Agarwal. Deep batch active learning by diverse, uncertain gradient lower bounds. arXiv preprint arXi::1906. 03671, 2019.
Haoyue Bai, Gregory Canal, Xuefeng Du, Jeongyeol Kwon, Robert D Nowak, and Yixuan Li. Feed two birds with one scone: Exploiting wild data for both distrrbiisti generalization and detection. In International Conference on Machine Learning, pp. 1454-1471. PMLR, 2023.
Haoyue Bai, Jifan Zhang, and Robert Nowak. Aha: Human-assisted out-of-distribution generalization and detection. Xiv preprint arXiv:2410.00 10.08000, 2024.
Maria-Florina Balcan, Alina Beygelzimer, and John Langford. Agnostic active learning. In Proceedings of the 23rd international conference on Machine learning, pp. 65 72, 2006.
William H Beluch, Tim Genewein, Andreas Nürnberger, and Jan M Köhler. The power of ensembles for active learning in image classification. In Proceedings of the IEEE conference on computer vision and pottern recognition, pp. 9368-9317 9377, 2018.
Abhijit Bendale and Terrance Boult. Towards open world recognition. In Proceedings of the IEEE conference on computer vision and pottern recognition, pp. 1893-1902, 2015.
Abhijit Bendale and Terrance E Boult. Towards open set deep networks. Proceedings of the IEEE conference on computer vision and pallern recognition, pp. 1563-1572, 1572, 2016.
Gantavya Bhatt, Yifang Chen, Arnay M Das, Jifan Zhang, Sang T Truong, Stephen Mussmann, Yinglun Zhn, Jeffrey Bilmes, Simon Du, Kevin Jamieson, et al. An experimental design framework for label-efficient supervised finetuning of large language models. arXiv preprint ar arXie:2401.0660 2024.
Erdem BIyIk, Kenneth Wang, Nima Anari, and Dorsa Sadigh. Batch active learning using determinantal point processes. Xiv preprint arXiv: 0797777 2019.
Terrance E Boult, Steve Cruz, Akshay Raj Dhamija, Manuel Gunther, James Henrydoss, and Walter J Scheirer. Learning and the unknown: Surveying steps toward open world recognition. In Proceedings of the AAAT conference on artificial intelligence, volume 33, pp. 0801-9807, 2019.
Xinmeng Cai. Active learning for imbalanced data: The difficulty and proportions of class matter. Wireless Communications and Mobile Computing, 2022, 2022.
Kaidi Cao, Maria Brbic, and Jure Leskovec. Open-world semi-supervised learning, In International Confer- ence on Learning Representations, 2022. URL http::///pppppeeeeeeeeeeee/r//es
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations, In International conference on machine learning, pp. 1597-1607. PMLR, 2020.
Zhuoxiao Chen, Yadan Luo, Zixin Wang, Zijian Wang, Xin Yu, and Zi Huang. Towards open world active learning for 3d object detection. arXie preprint arXiv:2310.100 10391, 2023.
Gui Citovsky, Giulia DeSalvo, Claudio Gentile, Lazaros Karydas, Anand Rajagopalan, Afshin Rostamizadeh, and Sanjiv Kumar. Batch active learning scale. Advances in Neural Information Processing Systems, 11111933-19444 2021.
14

---

Published in Transactions on Machine Learning Research (02/2025)
Cody Coleman, Edward Chou, Julian Katz-Samuels, Sean Culatana, Peter Bailis, Alexander C Berg, Robert Nowak, Roshan Sumbaly, Matei Zaharia, and T Zeki Yalniz. Similarity search for efficient active learning and search of rare concepts. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 6402-6410, 2022.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on compuler vision and pallern recognilion, pp. 248 288.. Teee, 2009.
Melanie Ducoffe and Frederic Precioso. Adversarial active learning for deep networks: a margin based approach. arXiv preprint arXiv:1802.09833 2018.
Juan Elenter, Navid NaderiAlizadeeh, and Alejandro Ribeiro. A lagrangian duality approach to active learn- ing. Xiviv preprint WwXi::2202.0400.. 2022.
Zeyad Ali Sami Emam, Hong-Min Chu, Ping- Yeh Chiang, Wojeiech Czaja, Richard Leapman, Micah Gold- blum, and Tom Goldstein. Active learning at the imagenet scale. arXiv preprint arXiv:2111. 11::2111112880, 2021.
Enrico Fini, Enver Sangineto, Stéphane Lathuilière, Zhun Zhong, Moin Nabi, and Elisa Ricci. A unified objective for novel class discovery. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.
2021.
Yarin Gal, Riashat, Islam, and Zoubin Ghahramani. Deep bayesian active learning with image data. In PMLR, 2017.
International Conference on Machine Learning, pp.
Zongyuan Ge, Sergey Demyanov, Zetao Chen, and Rahil Garnavi. Generative openmax for multi-class open set classification. In British Machine Vision Conference. BMVA Press, 2017.
Yonatan Geifman and Ran El- Yaniv. Deep active learning over the long tail. arXiv:1711.00041, 2017.
arXic preprint
Chuanxing Geng, Sheng-jun Huang, and Songean Chen. Recent advances in open set recognition: A survey. IEEE transactions on pallern analysis and machine intelligence, 43(1)):3614 3631, 2020.
Guy Hacohen, Avihu Dekel, and Daphna Weinshall, Active learning on a budget: Opposite strategies suit high and low budgets. orXiv preprint arXim:2202.0270 2022.
Kai Han, Sylvestre-Alvis Rebuffi, Sebastien Ehrhardt, Andrea Vedaldi, and Andrew Zisserman. Automat- ically discovering and learning new visual categories with ranking statistics. In International Conference on Learning Representations.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In
Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Dan Hendryeks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. arXiv preprint arXiv:1610.02136. 2016.
Dan Hendrycks and Kevin Gimpel. A bascline for detecting misclassified and out-of-distribution examples in neural networks. In International Conference on Learning Representations, 2022.
Dan Hendryeks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. arXiv preprint arXi::1812.0 046606 2018
Dan Hendrycks, Steven Basart, Mantas Mazeika, Andy Zou, Joseph Kwon, Mohammadreza Mostajabi, Jacob Steinhardt, and Dawn Song. Scaling out-of-distribution detection for real-world settings. In International Conference on Machine Learning, pp. -779. PMLR, 2022.
Rui Huang, Andrew Geng, and Yixuan Li. On the importance of gradients for detecting distributional shifts in the wild. Advances in Neural Information Processing Systems, 34:677-656 689, 2021.
15

---

Published in Transactions on Machine Learning Research (02/2025)
Qiuye Jin, Mingzhi Yuan, Haoran Wang, Manning Wang, and Zhijian Song. Deep active learning models for imbalanced image classification. Sonuledge-Baseee Systems, 257: 257:109817, 2022.
Siddharth Karamcheti, Ranjay Krishna, Li Fei-Fei, and Christopher Manning. Mind your outliers! inves- tigating the negative impact of outliers on active learning for visual question answering. arXie preprint w:Xiv2107.0239/ 2021.
Julian Katz- famzesa Julia B Nakhleh, Robert Nowak, and Yixuan Li. Training ood detectors their natural habitats. In International Conference on Machine Learning, pp. 10848-10865. PMLR, 2022.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishman. Supervised contrastive learning. Advances in neural information processing systems, 33:18661-18677 2020.
Suraj Kothawade, Nathan Beck, Krishnateja Killamsetty, and Rishabh Tyer. Similar: Submodular infor- mation measures based active learning in realistic scenarios. Advances in Neural Information Processing Systems, 34:18685 18697, 2021.
Jan Kremer, Kim Stoenstrup Pedersen, and Christian lgel. Active lcarning with support vector machines. Wiley Interdisciplinory Reviews: Data Mining and Knowledge Discovery, 4(4):313- 326, 2014.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, (215333)):433 444, 2015.
Kimin Lee, Kibok Lee, Honglak Lec, and Jinwoo Shin. A simple unified framework for detecting out- of-distribution samples and adversarial attacks. Advances in neural information processing systems, 31, 2018.
DD Lewis and WA Gale. A sequential algorithmfor training text classifiers. In SIGIR'94: Proceedings of the Seventeenth Annual International ACM-SIGIR Conference on Research and Development in Information Retrieval, organised by Dublin City University, pp. 3-12, 1994.
Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy-based out-of-distribution detection. Advances in neural information processing systems, 33:21464-21175, 2020.
Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella X Yu. Large-scale long- tailed recognition in an open world. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 2537 2546, 2019.
Shijie Ma, Fei Zhu, Zhun Zhong, Xu- Yao Zhang, and Cheng-Lin Liu. Active generalized category discovery. In Proceedings of the CREE/CV Conference on Computer Vision and Pattern Recognition, pp. 16890-16900, 2024.
Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 1180-4188, 2024. Ruiyu Mao, Ouyang Xu, and Yunhui Guo. Inconsistency-based data-centric active open-set annotation. In
Yifei Ming, Yiyou Sun, Ousmane Dia, and Yixuan Li. How to exploit hyperspherical embeddings for out- of-distribution detection? arXiv preprint arXiv:2203. 04450, 2022.
Mohamad Amin Mohamadi, Wonho Bae, and Danica J Sutherland. Making look-ahead active learning strategies feasible with neural tangent kernels. arXiv preprint arXiw:2206. 12569, 2022.
Anh Nguyen, Jason Yosinski, and Jeff Clune. Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 427-436, 2015.
Kun-Peng Ning, Xun Zhao, Yu Li, and Sheng-Jun Huang, Active learning for open-set annotation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 41-49, 2022.
Shyam Nuggehalli, Jifan Zhang, Lalit Jain, and Robert Nowak. Direct: Deep active learning under imbalance and label noise. ar Xio preprint or 2ii:2212.001990 2023.
16

---

Published in Transactions on Machine Learning Research (02/2025)
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748 8763. PMLR, 2021.
Mamshad Nayeem Rizve, Navid Kardan, Salman Khan, Fahad Shahbaz Khan, and Mubarak Shah. Openldn: Tearning to disoover novel classes for open-world semi-supervised learning. In European Conference on Computer Vision, pp. 382 401. Springer, 2022.
Subhankar Roy, Mingxuan Liu, Zhun Zhong, Nicu Sebe, and Elisa Ricci. class-incremental novel class discovery. In European Conference on Compuler Vision, pp. 317 333. Springer, 2022.
Bardia Safaei, VS Vibashan, Celso M de Melo, and Vishal M Patel. Entropic open-set active learning, In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 4686-4694, 2024.
M Salchi, H Mirzaci, D Hendrycks, Y Li, MH Rohban, M Sabokrou, et al. A unified survey on anomaly, novelty, open-set, and out of-distribution detection: Solutions and future challenges. Transactions on Machine Learning Research, 2022.
Tobias Scheffer, Christian Decomain, and Stefan Wrobel. Active hidden markov models for information extraction. In International symposium intelligent data analysis, pp. 309-318. Springer, 2001.
Vikash Schwag, Mung Chiang, and Prateek Mittal. Ssd: A unified framework for self-supervised outlier detection. Xiv preprinl. ar arXiv:2103. 1 12051, 2021.
Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. arXiv preprint vrXii:1705 00889 2017.
Burr Settles. Active learning litcrature survey. 2009.
Nasim Soltani, Jifan Zhang, Batool Salehi, Debashri Roy, Robert Nowak, and Kaushik Chowdhury, Learning from the best: Active learning for wireless communications, arXiv preprint wxXiv2240.048800 2024.
Xin Sun, Zhenning Yang, Chi Zhang, Keck Ling, and Guohao Peng. Conditional gaussian distribution learning for open set recognition. In Proceedings of the IEEE/CVP conference computer vision and pallern recognition, pp. 13480 13489, 2020.
Yiyou Sun and Yixuan Li. Opencon: Open-wordh contrastive learning. arXiv preprint arXiu:2208.02761, 2022.
Yiyou Sun, Chnan Guo, and Yixuan Li. React: Out-of-distribution detection with rectified activations. Advances in Neural Information Processing Systems, 34:144 157, 2021.
Yiyou Sun, Yifei Ming, Xiaojin Zhu, and Yixuan Li. Out-of-distribution detection with deep nearest neigh- bors. In International Conference on Machine Learning, pp. 20827. PMLR, 2022.
Yiyou Sun, Zhenmei Shi, and Yixuan Li. A graph-theoretic framework for understanding open-world semi- supervised learning. Advances in Neural Information Processing Systems, 36, 2024.
Simon Tong and Daphne Koller. Support vector machine active lcarning with applications to text classifica- tion. Journal of machine learning research, 2(Nov):45 66, 2001.
Joost Van Amersfoort, Lewis Smith, Yee Whye Teh, and Yarin Gal. Uncertainty estimation using a sin- gle deep deterministic neural network. In International conference on machine learning, pp. 9690-9700 PMLR, 2020.
Haonan Wang, Wei Huang, Andrew Margenot, Hanghang Tong, and Jingrui He. Deep active learning by leveraging training dynamics. arXie preprint 10.0011,, 2021.
17

---

Published in Transactions on Machine Learning Research (02/2025)
Haoqi Wang, Zhizhong Li, Litong Feng, and Wayne Zhang. Vim: Out-of-distribution with virtual-logit matching. In Proceedings of the IEEE/CVI conference on compuler vision and pollern recognilion, pp. 4921 4930, 2022a.
Jing Wang, Jie Shen, Xiaofei Ma, and Andrew Arnold. Uneertainty- active learning for read- ing comprehension. Transactions Machine Learning Research, 2022b. ISSN 2835-8856. URL
https:://openrevie http:p///ppppaeeeeeeeeeeeeeeeerr
Jim Winkens, Rudy Bunel, Abhijit Guha Roy, Robert Stanforth, Vivek Natarajan, Joseph R Ledsam, Pa- tricia MacWilliams, Pushmeet Kohli, Alan Karthikesalingam, Simon Kohl, et al. Contrastive training for improved out-of-distribution detection. arXiv preprint ar arXiv:2007.05666 2020.
Ruixuan Xiao, Lei Feng, Kai Tang, Junbo Zhao, Yixuan Li, Gang Chen, and Haobo Wang. Targeted repre- sentation alignment for open-world semi-supervised lcarning. In Proceedings of the IEEE/CVF Conference on Compuler Vision and Pallern Recognition, pp. 23072 23082, 2024.
Yang Yang, Yuxuan Zhang, Xin Song, and Yi Xu. Not all out-of-distribution data are harmful to open-set active learning. Advances in Neural Information Processing Systems, 36, 2024.
Ofer Yehuda, Avihu Dekel, Guy Hacohen, and Daphna Weinshall. Active learning through a covering lens. Advances in Newral Information Processing Systems, 0-22334--22617 2022.
Ghada Zamzmi, Tochi Oguguo, Sivaramakrishman Rajaraman, and Sameer Antani. Open-world active learn- ing for echocardiography view classification. In Medical Imaging 2022: Computer-Aided Diagnosis, volume 12033, pp. 138 148. SPIE, 2022.
Xueying Zhan, Qingzhong Wang, Kuan-hao Huang, Haoyi Xiong, Dejing Dou, and Antoni B Chan. A comparative survey of deep active learning. arXiv preprint arXiv:2203. 18450,00.11450 2022.
Jifan Zhang and Robert Nowak. Sieve: General purpose data filtering system matching gpt-4o accuracy at 1% the cost. arXiv preprint arXiv:2410.0221 02755,, 2024.
Jifan Zhang, Julian Katz-Samuuls, and Robert Nowak. Galaxy: Graph-based active learning at the extreme. arXiv preprint arXi::2202.0144 2022.
Jifan Zhang, Yifang Chen, Gregory Canal, Arnav Mohanty Das, Gantavya Bhatt, Stephen Mussmann, Yinglun Zhu, Jeff Bilmes, Simon Shaolei Du, Kevin Jamieson, et al. Labelbench: A comprehensive framework for benchmarking adaptive label-effiicntt learning. Journal of Data-centric Machine Learning Research, 2024a.
Jifan Zhang, Shuai Shao, Saurabh Verma, and Robert Nowak. Algorithm selection for deep active learning with inbalanced datasets. Advances in Neural Information Processing Systems, 36, 2024b.
Zhun Zhong, Enrico Fini, Subhankar Roy, Zhiming Luo, Elisa Ricci, and Nicu Sebe. Neighborhood con- trastive learning for novel class discovery. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10867-10875, 2021a.
Zhun Zhong, Linchao Zhu, Zhiming Luo, Shaozi Li, Yi Yang, and Nicu Sebe. Openmix: Reviving known knowledge for discovering novel visual categories in an open world. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 9462-9470, 2021b.
Bolei Zhou, Agala Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017.
Fei Zhu, Shijie Ma, Zhen Cheng, Xu- Yao Zhang, Zhaoxiang Zhang, and Cheng-Lin Lin Liu. Open-world machine learning: A review and new outlooks. Xixiv preprint arXio:2403.01759 2024.
18

---

Published in Transactions on Machine Learning Research (02/2025)
A Long Tail Datasets
CIFAR100. CIFAR1O0 contains 60,000 images across 100 classes, with 500 training images and 100 testing images per class. To create long-tailed version of CIFAR100, we use an exponential distribution where the number of examples per class N; is given by:
N₂ Noar;
where n is the total number of classes, No is the number of examples in the most frequent class, and the imbalance factor. In our experiments, we a 0.01, creating a highly imbalanced version of the dataset.
ImageNet. We use the ImageNet-1k subset, containing 1.28 million training images across 1,000 classes. Similar to CIFAR100, we create a long-tailed version of ImageNet using an exponential distribution with the same formula. Again, we set = 0.01 to generate the imbalance, resulting in a diverse distribution of images per class.
Places365. Places365 is a large-scale scene recognition dataset consisting over million images across 365 classes, designed for training and evaluating models on a wide variety of scene categories, including indoor and outdoor environments. Each class has up to 5,000 images for training, providing balanced dataset for scene classification tasks.
The Places365-LT we used in our experiments are based on the distribution described in Liu et al. (2019) This version follows an exponential distribution, where the number of examples per class ranges from a maximum of 4,980 images lo as few as images. The total 62500 images. The distribution was implemented by referring to the publicly available code associated with the paper.
The distribution of the three long-tailes datasets is visualized in Appendix Figure
503
5000
1250
Number
Clacs Number
3200
Nombbr
.00-
800
2000
300
600
2003
400
200
50
300
Cless Index
Class Index
Cless Index
(a) cifar L00-LT
(b) ImageNet-LL1
(c) Places365-L11
Figure 7: Number of images in each class of train datasets
B Experiments Results
C Per-batch Running Time Analysis of ALOE and Baselines
We analyze the per-batch rumning time of ALOE comparison to the mentioned basclines. Rocall that B is the batch size, N is the pool size, K is the number of classes, Q is the forward inference time of neural network on single example, and d is the feature dimension. Random Sampling is computationally the simplest, with time complexity of O(B), examples are chosen uniformly at random without any additional computation. Confidence Sampling methods, such as Margin Sampling and EOAL, require O(QN + KN + B Blog N), where N)QN) comes from forward passes on the pool, O(KN) from computing uncertainty scores (e.g., margins or entropy), and O log N) from selecting the top uncertain examples.
Diversity-fcussed methods like Badge and Coreset involve more computational overhead. Badge combines uncertainty and diversity through gradient embeddings, requiring O(QN+ Not + Bd3), where O(Nd²) arises
19

---

Published in Transactions on Machine Learning Research (02/202))
Daly
Mags
ALD2
DVL
310
Rride
ALCOUUM
X.08 F0me
1000
100
350
80
1750
1150
(a) CIFARI00-LT
(b) ImageNet-LT
(c) Places365-17
Figure Balanced test accuracy and number of annotated classes on CIFAR100-LT, ImageNet-LT and Places365-LT with all baselines and our algorithms. Reverse ALOE result shown on CIFAR100-LT, (Shaded region represent the standard error conducted across four trials.)
Table 5: Number of annotated classes on CIFAR100-LT and Places365-LT with different budget. (The standard error was calculated based on four trails.)
Datasct
CIFAR100-LT
Places365-LT
Budget
250
750
600
2000
Random
57.75+0.63
81.75=1.11
188.00+6.36
275.50=3.30
Margin
66.75+1.31
94.75+1.31
203.75+4.61
287.75+2.02
Badge
66.75:tu.75
96.00+0.62
209.75:1..77
295.25
Galaxy
68.5042..0
94.50.0.29
207 2..108
306.50 1....0
Coreset
66.25++1.70
91.0+0.71
2211.54+2.56
211275+1100
ProbCover
51.30:10.96
61.75.10.66
228. 088.00=5.05
TypiClust
56.00++1.08
13.25
181.00++1.4
265.00=22
EOAL
53.75 13.75.10.95
85.511.32
182.25
256.00 25.90013.92 13.92
ALOE(Ours)
73.5:10.94
96.1
230.50+1.80
304. 304.25:12.17
Table 6: Number of annotated classes on ImageNet-LT with different budget. calculated based on four trails.)
(The standard crror was
Dataset
ImageNet- LT
Budget
3000
10000
Random
584.7542.90
847.75+5.02
Margin
639.50=10.78
871.00+1.96
Badge
650.75+4.50
915.25+3.12
Galaxy
660.50411.86 84
989.00 089.0:11.29
Coreset
536.75 736.75455.02
824.25
ALOE(Ours)
789. 24.1.
077.0012.00
from gradient embedding computation and O(Bd²) from k-means++ clustering. Coreset solves a k-center problem, leading complexity of OQQN+ N exact form, although approximation methods reduce this O(QN + BNd). TypiClust is simpler, with O(QN BN), and ProbCover incurs O(QN + +N2d) due to pairwise coverage probability computations.
20

---

Published in Transactions on Machine Learning Research (02/2025)
For active learning algorithms in practice, time complexity is often not critical bottleneck because O(QN), representing the cost of forward passes on the pool, dominates all other terms. This ensures that met like ALOE remain computationally efficient while incorporating both diversity and OOD detection to achieve superior performance in open-world active learning scenarios.
21


Reviews:
[{"review_id": "39D1peFou7", "reviewer": "Reviewer_PXtB", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- Statements are mostly backed up with experimental results\n- Design choices of the algorithm are ablated\n\n**Weaknesses** \n- The motivation behind the algorithm is very high level. It could be difficult for the community to extract some takeaways from this paper, which can be built upon.\n- Baselines are rather old\n- Differences between the methods tend to be small, and the way the results are presented makes it difficult to compare them accurately. \n- What is the computational overhead of the algorithm? K-Means does not scale well with a very high number of samples (realistic scenarios in open-world active learning). The evaluation of lightweight clustering alternatives to improve scalability would be an interesting ablation study\n\nMore recent methods to compare to (I do not have a specific preference. I am sure there are even more recent methods available):\n\nDongmin Park, et al., \"Meta-query-net: Resolving purity-informativeness dilemma in open-set active learning\", NeurIPS 2022\n\nPan Du, et al., \"Contrastive active learning under class distribution mismatch\", TPAMI 2023\n\nYang Yang, et al., \"Not all out-of-distribution data are harmful to open-set active learning\", NeurIPS 2023", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose an active learning algorithm designed for open-world settings. During the active learning process the proposed method embeds samples by the currently trained neural network. These samples are clustered using k-means. Subsequently, the top-N clusters are selected based on the OOD score of the samples within the clusters. Lastly, the samples with the highest OOD scores are queried. \n\nThe authors demonstrate the effectiveness of their algorithm on several (longtail) datasets and additionally conduct ablation studies concerning the design choices (such as the used OOD metric or the clustering algorithm).", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Could the authors additionally provide a table with the final accuracies of the main result in the appendix? \n- More recent baselines would strengthen the claims\n- Could the authors provide a computational complexity comparison across methods?", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "s7pfxLbMsZ", "reviewer": "Reviewer_dXC5", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n* The paper is very clear and well written.\n* The proposed method is simple and easy to implement (Alg. 1).\n* The proposed method seems to perform well.\n\nWeaknesses:\n\n* W1: The experimental setup is solid overall, but I believe the paper would benefit significantly from more extensive experiments. Specifically, I expected additional results in the Appendix, where you investigate the paradigm shift discussed in Section 6 in much more detail. For instance, would similar observations hold if $f^t$ was not assumed to be pretrained? Expanding on this would provide a more comprehensive analysis (see point below also).\n* W2: There is a large body of work addressing proper evaluation of deep active learning methods (see, e.g., [1, 2]). A key insight from these studies is that the performance of different query strategies can vary significantly depending on factors such as: 1) the number of random seeds, and 2) the initialization of the prediction model $f^t$ (e.g., whether it is pretrained or not). Considering that this paper includes only four random repetitions and assumes a pretrained model, this raises some concerns. That said, I would not view this as grounds to reject a paper on (deep) active learning, provided the proposed method is sound and demonstrates strong performance. Nonetheless, I am curious to hear the authors' perspective on this.\n* W3: The paradigm shift discussed in Section 6 appears to primarily occur between the proposed method (ALOE) and the baseline Galaxy. I was hoping for a more detailed discussion to explain why this is the case, as it could provide valuable insights into the underlying reasons for this phenomenon. Why do we not, as far as I can tell, observe the paradigm shift with the other baselines?\n* W4: I wonder if the reason your method occasionally performs worse in later iterations might be due to a higher tendency to select outliers—specifically, those that are uninformative due to noise. This could negatively impact performance, particularly in datasets with a significant number of outliers. Do you think this could be the case? I suspect the extent of this effect may depend on the value of $k$. If so, an experimental study that verifies this would be interesting.\n* W5: In Figure 3c, why do you not continue until all classes are discovered for the Places365-LT dataset?\n* W6: It may be good to be a bit more detailed regarding the implementation of the baselines. In particular, how are distances computed for coreset (i.e., in what feature space)?\n\n**References**:\n\n[1] Carsten T. Lüth, Till J. Bungert, Lukas Klein, Paul F. Jaeger:\nNavigating the Pitfalls of Active Learning Evaluation: A Systematic Framework for Meaningful Performance Assessment. NeurIPS 2023.\n\n[2] Thorben Werner, Johannes Burchert, Maximilian Stubbemann, Lars Schmidt-Thieme:\nA Cross-Domain Benchmark for Active Learning. CoRR abs/2408.00426 (2024)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper explores open-world machine learning, focusing on effectively handling out-of-distribution (OOD) data. Specifically, it introduces an active learning approach that effectively discovers an unknown number of OOD classes. The study compares this method to common deep active learning strategies, highlighting a paradigm shift in active learning: from discovering new classes of examples to refining the understanding of existing classes.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "None for now, but please see points under weaknesses above.\n\nNote: I found some small typos, such as missing space before citations in the introduction. Also, there seems to be some space missing directly under the description of Figure 2.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "EOTipM80i1", "reviewer": "Reviewer_58BP", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The Out-Of-Distribution (OOD) problem is interesting and highly relevant to real-world applications.\n\nWhile the primary contribution lies in its practical applicability, the complexity of the proposed method is not addressed. Furthermore, the experimental results, as shown in the figures, do not effectively demonstrate the advantages of the proposed approach.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The Out-Of-Distribution (OOD) problem is interesting and highly relevant to real-world applications. The presentation is clear and easy to follow. However, I have some concerns:\n\nOut-of-Distribution Details: The out-of-distribution (OOD) setup requires further clarification. For instance, the problem setup appears to be online, where data with different classes arrives in batches or one by one. How is the target set defined in a streaming scenario? For example, if a machine learning model assigns low probabilities to all classes for a sample, this could indicate that the sample belongs to a new class or that the model struggles to identify an existing class. Is there a standardized approach for mapping such samples to a new class?\n\nComputational and Memory Complexity: What are the computational and memory complexity of the proposed algorithm? Specifically, does the use of k-means in each iteration limit its applicability to large-scale problems?\n\nPractical Scenario Analysis: Is there any analysis of the practical scenarios where different OOD scores from Table 2 are applicable? A deeper examination of these cases would enhance understanding.\n\nExperimental Results: While the main contribution of this work is the proposed algorithm for handling out-of-distribution, the experimental results do not convincingly demonstrate its advantage over existing methods. For example, as shown in Figure 3, the proposed methods, ALOE w/Energy and ALOE w/Gradnorm, often overlap with other methods. It is evident from the figures that the gap between the proposed methods and other approaches is not significant. If the authors wish to assert that the improvement is substantial, providing a table or conducting a relevant significance analysis is necessary.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Clarify the out-of-distribution (OOD) setup, including details such as how new labels are detected. Is there a standard definition being followed, or does this work consider alternative practical scenarios?\n\nInclude an analysis of the computational and memory complexity of the proposed method.\n\nProvide insights into when and how to use the different OOD scores presented in Table 2.\n\nPresent experimental results in tables or include a detailed analysis to effectively validate the proposed method's effectiveness.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers initially had concerns about the motivation for the proposed method, the scope of the experiments, and the baselines. All of these concerns have been sufficiently addressed in the rebuttal.
Decision: Accept (poster)