Language-assisted Feature Representation and Lightweight Active Learning For On-the-Fly Category Discovery
Authors: Anwesha Banerjee
Soma Biswas
Venue: Accepted by TMLR ()

Abstract: Contemporary deep learning models are very successful in recognizing predetermined categories, but often struggle when confronted with novel ones, constraining their utility in the real world. Identifying this research gap, On-the-fly Category Discovery aims to enable machine learning systems trained on closed labeled datasets to promptly discern between novel and familiar categories of the test-images encountered in an online manner (one image at a time), along with clustering the different new classes as and when they are encountered. To address this challenging task, we propose SynC, a pragmatic yet robust framework that capitalizes on the presence of category names within the labeled datasets and the powerful knowledge-base of Large Language Models to obtain unique feature representations for each class. It also dynamically updates the classifiers of both the seen and novel classes for improved class discriminability. An extended variant, SynC-AL incorporates a lightweight active learning module to mitigate errors during inference, for long-term model deployment. Extensive evaluation show that SynC and SynC-AL achieve state-of-the-art performance across a spectrum of classification datasets.

Published in Transactions on Machine Learning Research (September/2025)
Language-assisted Feature Representation and Lightweight Active Learning For On-the-Fly Category Discovery
Anwesha Banerjee
amweshahener@Hisccoo..
Deportment of Electricnl Engineering Indian Institute of Science, Bengaluru
Soma Biswas
somabisuua wwwsssssiissscc.oo
Department of Electrical Engineering Indian Institute of Science, Bengaluru
Reviewed on OpenReview: https: // openreview. net/ forum? id= ZohFoM8KO
Abstract
Contemporary deep learning models are very successful recognizing predetermined cat- egories, but often struggle when confronted with novel ones, constraining their utility in the real world. Identifying this research gap, On-the-fly Category Discovery aims to enable machine learning systems trained on closed labeled datasets to promptly discern between novel and familiar categories of the test-images encountered in an online manner (onc image at a time), along with clustering the different new classes as and when they are encountered. To address this challenging task, we propose SynC, a pragmatic yet robust framework that capitalizes on the presence of category names within the labeled datasets and the power- ful knowledge- base of Large Language Models to obtain unique feature representations for each class. It also dynamically updates the classifiers of both the seen and novel classes for improved class discriminability, An extended variant, SynC-AL incorporates lightweight active learning module to mitigate errors during inference, for long-term model deployment. Extensive evaluation show that SynC and SynC-AL achieve state-of-the-arl performance across a spectrum of classification datasets.
1 Introduction
Deep learning models have achieved remarkable success in visual recognition tasks, often surpassing human performance. However, these models typically rely on closed category sets and extensive annotated datasets, limiting their ability to recognize novel categories. This presents a challenge in real-world, where encountering new categories is inevitable. The abundance of unlabeled data increases the need for models that can generalize to new categories and identify similarities without extensive annotation. Generalized Category Discovery (GCD) Vazd 2T12) aims to classify samples in query set containing both known and unknown categories, based on a labeled support set of known categories. The challenge lies in accurately identifying new, unseen classes that share semantic similarities with the known ones. Still, the GCD task relies on (i) access to the data from test categories during training, which limits their ability to handle truly novel classes, and (ii) batch-wise processing during inference, which not desirable for real-time applications. To address the limitations of GCD, (Dn cf all, 2023) proposed a more challenging and realistic task, namely On-the- fly Calegory Discovery (OCD) (Figure M). OCD utilises closed-labeled set for training without requiring access to novel class data. Additionally, it performs inference for each query sample in online manner, and obviates the requirement of collecting batches of test data. This shift from the standard paradigm creates a agile and adaptable classification framework, better suited for real-world applications.
Here, we propose a novel framework termed SynC (Syncing image features and Language-assisted assisted represen- tations with Classifier Update), which is designed overcome these challenges. Since the model encounters images from novel classes during inference, is important that the model learns map input training images to semantically meaningful latent feature space, which is unique for each class without any apriori

---

Published in Transactions on Machine Learning Research (September/202g) 2002)
On-the-Fly Category Discovery with Light Weight Active
the-Fly Category Discovery (OCD) Training Phase
On-the-Fly Category Discovery (OCD)
Learnnng
Test Phase
Test Phase
Confidant prediction
Model
Model
Model
uncertain prediction
?
Individually Inferred
Individually Inforred
Novel Casses
Orscie
Known Coas
Novel Gince
Kaown Clas
Known Caas
Novel class
Dons the quary image match with any of the tep 3 prodictonss regrosrtttive imager ?
Noval Unlabaled Data
Unlabeled Data
Arplare
Known Chos
YES
NO
XQ
Labeled Data
Novel class
Known Classee
Novel Cianzee
(Kg)
Figure 1: Different stages of OCD: Left: Training, Middle: Testing and Right: Testing with Active Learning. We propose SynC and SynC-Al for handling both the testing scenarios.
knowledge about these classes. Towards this goal, as a pre-processing step, we utilize category names from the labeled dataset generate descriptions using the powerful knowledge-base of Large Language Models (LLMs). By providing prompts specific to each category, we obtain comprehensive textual representations that encode the semantic essence of each category. We use two tailored supervised contrastive loss functions to align the image embeddings with the semantic information embedded within their class-specific textual representations and class-wise prototypes. During inference, SynC categorizes the samples sequentially into previously encountered or novel classes by utilizing a classwwise adaptive threshold. By utilizing the ncarest mean class representatives as the classifiers, learned associations between visual and ual modalities are leveraged, thereby enhancing classification accuracy. During test-time, SynC utilizes the stored statistics of the classes encountered so far to periodically update all the classifiers simultaneously, thereby further improving their discriminability.
Furthermore, we propose an extended variant, SynC-AL, which incorporates lightweight active learning, al- lowing for elfective human intervention to correct errors that could otherwise propagate through the system. Even with minimal human involvement, SynC-AL can help to further boost the performance the frame- work. Experimental results demonstrate that the proposed framework achieves state-of-the-a performance across three widely used coarse-grained and four fino-grained classification datasots. The contributions of the work are follows:
1) We propose novel SynC framework which effectively utilizes language supervision to address the chal- lenging OCD task. We show that language supervision can be successfully leveraged for the OCD task, even when the class names of the novel categories are unknown.
2) We propose a test-time classifier update during inference to allow knowledge transfer between old and novel classes for improved discriminability.
3) To the best of our knowledge, this is the first work to propose strong baseline (SynC-AL) for On-the-
fly-Active Category Discovery (OACD) setting.
4) We establish state-of-the-art results for this challenging task several benchmarks.
2 Related Work
Here, we provide pointers to few related work the different concepts used in this work.
Generalized Category Discovery (GCD): GCD (S. Vazd 2022) involves classifying unlabeled data containing instances from both previously seen classes and entirely novel classes. Here, both unsuper- vised and supervised contrastive losses (C. Ting (022) are employed to fine-tune a Vision Transformer (ViT) (A. Kalesnikoy & Zhail, 2021), that was pre-trained using DINO Mathilar, 2021). This enhances the similarity between feature representations of various views of the same image and between different. images of the same class. PromptCAL, (Z. Sheng 20% introduced prompts to provide valuable semantic
2

---

Published in Transactions on Machine Learning Research 2Septe
supervision signal for the GCD task while AMEND (Banerjee et all, 20244 offers similar performance while using neighboorhood information. In Nan (212)), conceptual contrastive learning is utilized by the authors, which considers the relationship between instances and significantly improves clustering accuracy. SPTNet Hongjum, 2n proposes a two-stage adaptation approach that optimizes both model and data parameters by accounting for the spatial properties of image data. Tingzhang et al. in L.. Tingzhang (21121) propose a dual-context approach that enhances feature learning by integrating instance-level and cluster-level contextual information, thereby improving the identification and classification of categories in unlabeled datasets. Keon-Hee et al. in IP. Keon-Hee (2022) introduce novel problem setting 'Online Continuous Generalized Category Discovery". They also propose framework that enables models to continuously learn and adapt to new categories in streaming data using energy-guided discovery, variance-based feature augmentation, and contrastive loss to mitigate forgetting and enhance category separation. On-lhe-fly Category Discovery addressed in this work emphasizes real-time adaptation with instantaneous category discovery, requiring each sample to be inferred independently, whereas Online Continuous Generalized Cate- gory Discovery allows mulliple lest-lime samples and oflen incorporates memory buffers or replay stralegies.
On-the-fly Category Discovery: The authors of (Dn et all 2023) introduced the practical OCD task to enable trained models to instantancously recognize and classify novel categories, akin to human cognition. They propose a hash coding-based recognition model and a Sign-Magnitude <lIsentang..Ement (SMILE) architecture to mitigate intra-category variance, facilitating inductive learning and streaming inference for rapid adaptation to new information. Haiyang et al in Z. Haiyang (2022l) introduce a Proto- typical Hash Encoding (PHE) framework that enhances online discovery of new categories in streaming data by generating multiple prototypes per category to capture intra-category diversity and employing discriminative category encoding to improve hash code discriminstion, effectively addressing sensitivity issues in fine- ane-grained class discovery. Both the works relies of hash codes, hence they faces challenges related to the predefined length of hash codes for category descriptors, which is addressed in our work.
Self-supervision: SimCLR Ting 2020) proposes a framework for contrastive learning of visual representations, which learns effective features from unlabeled data by maximizing agreement between augmented views of the same image while minimizing agreement between views of different images. Supervised contrastive learning (K. Prannay,. 202) extends these methods by incorporating labeled data, where positive pairs consist of samples from the same class and negative pairs are from different classes. Our work leverages both supervised learning and contrastive learning to learn effective representations.
Zero-shot learning: CHiLS Novack el all 2023) improves CLIPs zero-shot accuracy on coarse classes by expanding each superclass into a set of finer hierarchical subclasses, sourced from taxonomies or generated via GPT-3, and then mapping subclass predictions back to their original superclasses through simple reweighting of superclass probabilities. This work demonstrates that leveraging predefined hierarchical label structures can substantially boost zero-shot accuracy within fixed taxonomy. However, it operates under the assumption that the full class hicrarchy is known in advance and docs not address settings involving truly novel categories. The authors of IP Sarafi (2122) use large language models to automatically generate customized, class-specific natural language prompts, conditioned on class names, attributes, and contextual cues, instcad of relying on fixed template phrasos. These tailored prompts, when used with vision-language models such as CLIP, lead to siguificantly improved zero-shot classification accuracy by capturing richer semantic nuances for each category. S3A (Zhang et all 2022)) addresses more realistic zero-shot setting in which neither labeled examples nor ideal vocabularies are available. It proposes a
pipeline, which extracts structural semantics from unlabeled images by clustering them, voting on candidate labels from a broad vocabulary, refining prompts with a language model, and realigning prototypes for preuuo-superiision The model then self-trains the CLIP image encoder using both individual and structural semantic alignment in a teacherstudent framework. However, S3A assumes access all unlabeled data upfront, whereas in the OCD setting, samples arrive in streaming manner and inference must be performed on-the-fly, making the two settings fundamentally different.
Textual supervision: In this paper (Gao et all 2022), the authors introduce KeyClass, a weakly supervised framework that automatically derives interpretable labeling functions from classlabel descrip-
3

---

Published in Transactions on Machine Learning Research )20ptember/2025)
tions and combines them via data programming to train text classifiers without any humanannotated documents Evaluated on both benchmark datasets and ICD-9 code assigmment for MIMIC-ITI notes, KeyClass achieves accuracy on par with fully supervised models, significantly reducing the need for costly manual labeling.
Active Learning: Traditionally, active learning is performed in closed-world setting, where the labeled and unlabeled data contains the same set of classes to maximize the performance of model with limited labeling budget (Scttles, 2009). The widely used pool-based methods (Z. Xucying,
202T) can be broadly categorized into two groups: (i) diversity-based methods Ozan, 2011 Jordan 1, 2021 choose samples
that effectively represent the entire dataset; (ii) uncertainty- methods (IR. Dam, 2006) prioritize selecting
samples that exhibit. high levels of predictive uncertainty. Hybrid methods (AA Sharat, 2020: HH. Sheng-Juni 2010) that combines both approaches for better performance have also been proposed. To overcome the issues of imbalanced classification performance and inconsistent confidence between old and new classes, inspired by active learning principles, Active Generalized Category Discovery (AGCD) (M. Shije, 20za)) is proposed. AGCD aims to improve GCD performance by strategically selecting valuable samples for labeling from an oracle, using an adaptive sampling strategy that considers novelty, informativeness, and diversity. Here, we propose an active learning strategy for the very challenging and realistic OCD setting.
3 Proposed Framework
In this work, we address the challenging OCD task, where the focus is on enabling machine learning models to adeptly identify previously unseen categories during test-time an online manner. The central concept revolves around training conducted on closed labeled dataset, subsoquently roquiring the model's capability discern novel and familiar categories instantaneously. The model is first trained on a support set Ds {(a;, = Xs Ys, encompassing S image samples 2; paired with corresponding labels ys- During inference, the model encounters query images from DQ = {x:) E Xq as they arrive one at time in an online manner. The test set DQ comprises both known (seen during training) and novel categories, i.e. the set of training labels is subset of the query labels (Ys C YQ). The task is to assign the query samples to either previously encountered classes or novel, unseen categories.
It is to be noted that, zero-shot learning assumes fixed set of target classes are known a priori. ZSL also uses external semantics (like attributes or text prompts) to label samples from unseen classes, whereas in case of On-the-fly Category Discovery, the model dynamically discovers now categorics relying only on semanntc information from the already seen classes.
The proposed framework SynC has two modules each, in both the train and test stages: The main modules of the training stage using examples from known classes arc: 1) Generation of unique LLM-assisted class representations and 2) Syncing image features with unique textual descriplion and class prototypes. The two modules of the test stage are: 1) Adaptive threshold for novel class discovery and 2) Test-time update of the classifiers. In addition, if human involvement is allowed during inference, we present another variant of our framework, SynC-AL, where a lightweight active learning module is incorporated to mitigate inherent inaccuracies from propagating further. Now, we describe the different modules in detail.
3.1 SynC Training Phase
Given training samples from known classes, the goal of SynC is to learn a mapping from the image to a semantic latent space such that the images of novel classes seen during testing will also be embedded in a semantically meaningful manner. We achieve this using the following two modules.
3.1.1 Generation of unique LLM-assisted class representations
We aim to learn a mapping from the images a semantically meaningful latent space, such that images belonging to the same class (known or novel) automatically gets mapped a unique location in the latent space. The dataset Dg contains labels yı corresponding each sample 2; Xs, which are used to obtain its
4

---

Published in Transactions on Machine Learning Research )20pt)mber/2025)
category name qt To obtain the unique text representation of each class, we utilize the powerful knowledge- base of the pre-trained Large Language Models (LLM). First, the LLM is supplied with prompts (11, Y2, ("1,"2, vd)
specific to each category name, to elicit distinctive descriptions for each category. Specifically, each prompt U; augmented with the category name q is given as input to the LLM, which generates a textual description Ta, i.e. 1 {1r + q) + Tz- For example, given the calegory name Black-footed Albatross, the prompts used for obtaining descriptions of this category are 'Describe the bird, Black-footed Albotross:", ' What are the identifying characteristics of the bird, Black-footed Albatross?' and 'Describe what the bird Black-footed Albotross looks like' Sarah, 2012). The details of all the prompts used are provided in Table I in Appendix A. Thus, for each category, d textnal descriptions T..... Ta) are acquired for d LLM prompts (v1, U2,,., va)- Su Subsequently, Centence-BERRI (INI, 2014) is employed to extract text embeddings from these descriptions, from which the mean lext. embedding is computed which is taken as the unique textual representation 4 for the class oj as seen in Figure 2
3.1.2 Syncing image features with unique textual description and class prototypes
The goal is to train a model on a set of classes, such that it (i) can discern whether a test (query) image belongs to one of the seen classes or a novel class, (ii) is able to group data from distinct novel classes separately and (iii) does not require any knowledge of the number of novel classes that will be encountered. We propose to leverage dlass-aamo-based text emboddings and the class-spocific image prototypes of the training data, since they are unique for each class, and automatically leads to the desired discrimination in the semantically meaningful latent space. Thus, the goal is to learn the mapping of the images (and their variations) to the unique, semantically meaningful textual descriptions learnt in the previous module, on the assumption that it will subsequently generalize and map images from novel classes to their respective unique locations in the latent space during testing. Specifically, given an image T2, the l2 normalized feature 2; is obtained by using feature extraction backbone f i.e. z; = f(z)). During training, we first utilize supervised contrastive loss to bring all images of the same class closer for improved discriminability. Specifically, for Vi, its augmentation and all samples within the mini-batch that belongs to the same class that of x; are considered as positive samples. The supervised contrastive loss (K. Prannay 2020) can be written as:
exp(z: zg/a)
C [() ( log Σn 1ns exp(z: zn/a)
(1)
where (i) denotes a set of indices of the augmentation and all the other labeled samples belonging to the same class as that of x; in the mini-batch B. a is the temperature used for scaling. To align the images of the same class with their unique textual description, we utilize the textual representation as an adjunct view for the anchor image z,. This association enhances the model's capacity to comprehend the intrinsic characteristics and semantic attributes of the depicted category. The textual supervised contrastive loss is given as:
exp(2: 4//a)) C - log Σn
(2)
n [mzze] exp(z: (n/a)
This alignment enhances the model's ability to establish meaningful connections between the visual and textual modalitics.
Furthermore, to enhance the clustering performance of the model, we propose to include the prototypical class representative as another positive sample within this framework of supervised contrastive loss. Specifically, for each class at, where C Ys, class-specific image prototype p/ is computed by taking the mean of all la normalized image features belonging to the class q- To facilitate the learning process and ensure that the image embeddings are more closely aligned with the corresponding class prototype, we utilize the class prototype p' as another positive pair. and the prototypical supervised contrastive loss is given as:
exp(2; p//a)
c - log In 1nzz exp(2 pm /ai
33)
This improves the overall clustering and classification performance. Thus, the unified training objective is:
Liotal L + +++ L + L
(4)
5

---

Published in Transactions on Machine Learning Research (September/2005 2022)
Generation unique LLM-asslated class representations
Syncing image with unique description and class prototypes
Adaptive Threshold
Test-tim update of the classifiers
integrated Light-weight Active Learning
Discovery
Training
Phase
Inference
Inforence
Inforonce during Inference phase in
mmge Syreeii
Training
Images class
Covariance all images in class
Σ
Phase
Test-lime update the classifiers
Class name
Mean of all images class
protolypical class representative p
with unique (ediu wooogeeee
C
ALL projection
Gachbaee
Ltotal
C
Akplann
Bit Palornoti
Labaled Datanet
(Kg
Augmentttion
4
Feature Deraction Backtonn
C
Class name
pud
BERT
Propp:
LLM
Ted desctppion
Mean text embedding
sackooyoat
embeddings
Figure Overall structure of our framework SynC and SynC-AT. The second module 'Syncing image features with unique textual description and class prototypes' is demonstrated in detail here. In the training phase, each image is passed through ViT backbone and MLP projection head extract image features. For each class c, multiple prompts are used to generate k textual embeddings whose mean defines unique LLM-based class representation 4, Simultancously, the mean of the image features for all samples in class forms the prototypical class representative pr. These are jointly used to align image and text features using the loss Crotal. During inference, use of adaptive thresholds helps discovery of instances from novel class, while light- active learning and test-time updates refine the classifier progressively.
Jointly minimizing Leotal ensures that the learned embeddings simultaneously capture visual consistency within classes, align closely with semantie textual descriptions, and cluster effectively around class prototypes.
3.2 SynC Inference Phase
During inference, the model trained as described above encounters individual samples sequentially. The task is to categorize them into previously encountered classes designate them originating from novel, unseen class. As novel classes discovered, in order to improve discriminability among all the classes, the classifiers are updated periodically using the stored statistics of all the classes encountered so far.
3.2.1 Adaptive Threshold for Novel Class Discovery
Given the integration of class prototypes as positive samples within the supervised contrastive loss framework, propose to employ class prototypes the classifter during testing. Given there labeled classes during training, cosine similarity between the dictionary of prototypical class representatives P .p..p.....,p of previously seen classes and the l normalized feature of the current image is calculated as simp <zi; P> To account for the variability among different classes, i.e. some classes may be more clustered compared to others which are more spread out, we utilize statistical modeling for setting adaptive thresholds for different classes, thus making more adaptable the data distribution of each class. Specifically, for each known class where = 21, the mean, Mc and standard deviation e of the cosine similarities between la normalized image features and their corresponding prototypical class representatives computed. The

---

Published in Transactions on Machine Learning Research (September/2025) 2202)
class specific threshold To for each class e is then determined follows:
To = Me - g * cc
(5)
where g is a hyperparameter. This ensures that the threshold is higher for the well clustered classes, and smaller for the spread out ones. The difference between the cosine similarity of the current image feature and the prototypical class representative pf is calculated and the sample is assigned to class e if the similarity exceeds the class-specific threshold. Otherwise, the sample is presumed to belong to a novel, unseen class. In this case, in the absence of any other information about the newly discovered category (like class name), we propose to store the l2 normalized feature z corresponding to the image sample as the classifier for this novel category. Hence, the first novel class representative is denoted as Z 2i. Consequently, the dictionary comprising of the prototypical class representatives of the classes seen so far is represented as (w', (p","",.,, .,?")..",.Z''.
Since we have limited access to samples from novel classes, we assign the threshold for an incoming novel class to be the same as the class-specific threshold of the known class that it is most similar to, based on their latent feature representations. This approach assumes that semantically similar classes will exhibit similar variations in their feature distributions.
3.2.2 Test-time update of the classifiers
During the testing phase, for the samples designated as belonging novel classes, the image feature is taken as the classifier, in the absence of their actual class names. We make two observations: (i) Depending upon the images encountered, these features may not be very good representatives of the corresponding class; (ii) This also leads to loss of discriminability when the old and new classes are considered together. To overcome these challenges, we propose to update the classifiers periodically, thereby sharing knowledge between all classes encountered so far.
Suppose, at a certain instant, m number of novel classes (m S N, where N is the total number of novel categories present in the query set DΩ) have been identified so far. The dictionary of classifiers at this point is given by (p', p², _2, -² ; ... ) For updating the classifiers, one straightforward solution is to store few training images of the classes encountered so far, but this may have storage/ porgge/ concerns. Instead, we utilize the means Me and covariances Ee for each class c, the underlying assumption is that each class can be modeled as a Gaussian distribution A(He, Ed) For the seen (training) classes, we can properly estimate the mean and covariances from the training data. Since we do not have access to many samples from the novel categories, we use the classifier feature ZJ where 2..,, m) as the mean feature for that category and the covariance of the most similar class (based on their latent fcatures), assuming that semantically similar classes will exhibit similar variations.Detale of experiments supporting the claim that "semantically similar classes will exhibit similar variations', is present in Appendix A.4.
Finally, features F. 2/, I,..., H are sampled from the distribution N(ee, Ee) for each class E 1, 1,,, L+ m, where H represents the number of generated features per class. Subsequently, the standard cross-entropy loss is used to adjust the classifiers by feeding these features into the classifier head, which is a single-layer perceptron with a dictionary of classifiers as its weights. This classifier adjustment results in better scparation of the classifiers with respect to cach other and is performed periodically after each new set of m novel classes is discovered.
4 SynC-AL: SynC integrated with Light Active Learning
In the On-the-fly Category Discovery (OCD) task, models must continuously adapt novel classes encoun- tered during inference. However, this real-time adaptability poses fundamental challenge: as the model incrementally processes individual samples, prediction errors (eg. misclassification of seen class instance as novel) can quickly accumulate, leading to compounding inaccuracies over tine as seen in Figure M Al- though the adaptive threshold strategy in SynC' mitigates some errors by dynamically adjusting decision boundarics, certain ambiguous or borderline samples remain inherently difficult for automated classification. Active Learning (AL) presents an attractive solution by strategically incorporating limited human supervi- sion to resolve these challenging cases. However, traditional AL approaches, which often involve extensive
7

---

Published in Transactions on Machine Learning Research 250pt)
human involvement, are impractical for real-time resource-constraanee applications. Thus, an effective AL method specifically tailored to the OCD task must be lightweight, efficient, and impose minimal cognitive load on human annotators.
Motivated by these practical constraints, we propose SynC-AL, a lightweight active learning extension of SynC. Unlike traditional AL methods that require extensive labeling or domain expertise, SynC-AL lever- ages simple binary comparisons with clas-rrerrsentaatiee images to efficiently resolve uncertain predictions. Additionally, we introduce Confusion Buffer to minimize redundant human queries, further reducing amno- tation costs without sacrificing accuracy. Through SynC-AL, we demonstrate that even minimal, intelligently directed human input can significantly enhance long-tery gengtermmenn performance in dynamic, real-world category dis- covery scenarios. Now we describe the approach in detail.
During testing, given an image sample z, with an (g-normalized feature representation 2,, its similarity with the prototypical class representatives (p',p² .p....... .... pt..p'ppppttee .p(++1) Zz, p( Zz, .... ....,p(/t p(z.m = zm) of proviously
seen classes (assuming m novel classes have already been encountered) is computed. If the maximum cosine similarity between 2; and any known class prototype pe is less than the corresponding class-specific threshold To, i.e.
max cos p²) < Tes
<E(I,... (l......+ton I.
the sample is considered a probable instance of novel category and is processed by the active learning module. Thresholding the cosine similarity for novel category discovery introduces potential inaccuracies, one common one being some instances of previously seen classes may not achieve a similarity score above their respective threshold Te, resulting in their misclassification as a novel class. To mitigate such errors and improve the long-term performance of the model, we introduce a light-weight active learning mechanism in our framework, which we term as SynC-AL.
Let Bactive denote the active learning budget, representing the maximum number of queries for which the model can request. human feedback. The total human effort depends not only on the budget Bactive, but also the complexity of the feedback requested. In our work, when query image I; is predicted to belong to novel class, the active lcarning module becomes functional. The straightforward human feedback that can be requested is the class of the impull sample, based on which it can be incorporated in one of the existing categories or classified as a new class. This approach has the following limitations:
i) It requires domain expertise to classify a given image into its class, specifically for fine-grained or specialized datasets like CUB-200-2011. Stanford Cars, Food 101, etc.
ii) It becomes especially harder when the number of classes of interest increases, eg, CUB-200-2011 dataset. has 200 classes.
Here, we propose specialized active learning module, that requires much simpler human feedback. When a query image x; is predicted to belong to a novel class, our proposed active learning module is engaged to refine the prediction by leveraging the models top-k most likely class predictions. The intuition is that usually, the correct class (if it belongs to one of the already seen classes) lies within the top predictions. In order to facilitate this, for each encountered class c, the model maintains representative image r", encapsulating the distinguishing features of that class. Let (zzp= denote the set of the top-k prodicted classes for 2;, where each class e (zop- has an associated representative image r The active learning module queries an oracle O to compare the query image æ; with each representative image re from the predicted set. The oracle provides a binary response for each comparison as follows:
0(x,)9)= Yes, if x; belongs to the same class as rd No, otherwise
If the oracle confirms that matches with any of the top-k predicted classes, the query image is assigned to class e. Conversely, if the oracle returns No for all k comparisons, the image is classified as belonging to a novel class. This active learning process is computationally efficient, as it only requires k comporisons, making it a lightweight and practical approach for real-time category discovery. This active lcarning module does not require significant domain knowledge, like names of fine-grained classes, etc., as the oracle is not tasked with identifying the exact class of the query image. Instead, it only needs to evaluate the similarity
8

---

Published in Transactions on Machine Learning Research (September/2025)
CIFARIO
CIFAR100
ImageNeet-100
CUB--20020111
Stanford Cars
Food101
Oxford Pets
5
50
100
51
19
10
100
200
196
101
Ds
12.5K
20.0K
31.9K
1.5K
2.0K
19.1K
0.9K
Dol
37.5K
30.0K
95.3K
4.5K
6.1K
56.6K
2.7K
Table 1: Details the datasets used, the standard split in terms of the number of labeled and unlabeled
classes
(Ys). (Ql)) and the number of images in the labeled and unlabeled set ((Ds). (Dql).
Components
CIFAR.
CIFAR.
Imagenct-
All
Old
New
All
Old
New
All
Old
New
SLC (Harligar (NYP))
41.54
58.29
33.29
44.36
58.98
15.10
32.92
86.55
5.22
MLDG (L. k 2018)
44.14
38.47
46.98
50.60
60.98
29.83
30.63
72.30
9.69
RankStat (EL Ku (0211)
42.14
49.26
38.59
35.00
44.01
16.98
31.06
73.30
9.83
WTA (U Xabur (17411)
43.12
34.52
47.42
40.83
52.89
16.72
30.84
72.92
9.68
SMILE cL
49.86
39.86
54.86
51.59
61.55
31.69
33.78
74.22
13:55
SynC
50,13
40.57
54.90
56.13
68.43
31.54
44.01
86.22
22.80
Table 2: Performance (accuracy %) of the proposed SynC framework on three coarse-grained datasets.
between the query image and the representative images, reducing the cognitive load and making the system more accessible for non-expert users.
Given the small active learning budget Bactive, its careful utilization crucial. To optimize its use, a repository of confusing samples is maintained to prevent redundant queries. When a sample 2; fails to surpass the class-specific threshold Te, but the active learning module confirms that it belongs to one of the previously seen classes, the sample considered confusing and difficult for the model to classify correctly. Formally, if
Confusion Buffer:
max cos(2;, < Te, but O(z,, = Yes & rf top predicted classes of Creens (E......))
(8)
where O(1,r r represents the oraclo-provided ground-truth label and Creen is the set of previously seen classes, the sample is stored in Confusion BufJer M, which has predefined fixed capacity.
To conserve the active learning budget, any new query image Ig from the test set DQ that doesn't 't cross the class- specific threshold T, is first cross-checked against the Confusion Buffer. Lot 2g be the Ig-normalized feature of zq. If the highest cosine similarity between 2g and any stored confusing sample 2m E M exceeds a similarity threshold Te, iee. Ma cos(2q, 2m) 2 Tr the query image is assigned to the corresponding class. This process helps classify confusing samples without invoking the active learning module. However, if Ty remains dissimilar to all stored samples, ie., max cos(2q. 2m) Te, then the active learning module is engaged, provided the query remains within the confines of the active learning budget. By incorporating the Confusion Buffer, unnecessary active learning queries are avoided, thereby optimizing the utilization of the limited budget Beetive. Detailed Algorithm is given in Appendix A.3.
5 Experiments
Extensive experimental evaluations are conducted on three generic and four tne-grained benchmark datasets. The generic datasets encompass CIFAR1O, CIFAR100 (Krizhevsky et al., 2009). and ImageNet- (IL Yon
glong 2020),, and the fino-grained datascts include Stanford Cars (K. Jonathan 20K). CUB (Welinder et all,
Oxford Pets (P.O0mkar MI, 20T2) and Food 101 (B. Lukas
(Semanticc Shift Benchmark).
These datasets collectively serve as standard benchmarks for assessing the performance and generalization capabilitics of our SynC framework for OCD setting and its extended version SynC-AL for OACD setting. From each dataset D, a subset of classes is sampled to constitute the set Ys denoted as the seen classes. Subsequently, 50% of the images associated with each seen class are randomly selected to form the labeled set Ds, while the remaining images constitute the unlabeled set DQ- Furthermore, it assumed that a disjoint validation set is available, where subset of labels is masked to obtain the unlabeled set. More dataset details given in Table III
9

---

Published in Transactions on Machine Learning Research (epttember/2225)
Oxford Pets
Food101
CUB-200-2011
Stanford Cars
Methods
Old
New
All
Old
New
All
Old
New
All
Old
New
SLC
35.5
41.3
33.1
20.9
48.6
6.8
28.6
44.0
20.9
14.0
23.0
9.6
RankStat
33.2
42.3
28.4
22.3
50.7
7.8
21.2
26.9
18,4
21.2
26.9
18.4
WTA
35.2
46.3
29.3
18.2
40.5
6.1
21.9
26.9
19.4
17.1
24.4
13.6
SMILE
41.2
40.7
24.0
54.6
8.4
32.2
50.9
22.9
26.1
16.2
PHE
48.3
53.8
45.4
29.1
64.7
11.1
36.4
55.8
27.0
31.8
61.9
16.8
SynC
61.65
69.46
57.55
31.09
60.72
15.98
45.33
54.27
40.85
24.60
34.80
19.55
Table 3: Performance (accuracy %) of the proposed SynC framework on four fine-grained datasets.
CIFAR 10
CIFAR 100
Imagenet-100
Methods
Al1
Old
New
All
Old
New
All
Old
New
Margin-based bascline
51.52
20.05
67.25
56.11
72.27
44.69
21.86
Uncertainty- baaed baseline
51.71
29.06
67.54
57.80
74.42
24.55
45.17
87.84
89.72
SynC-AL
62.12
39.65
06.36
57.96
72.67
28.56
46.00
88.04
24.88
Table 4: Experimental results showing the baselines of OACD setting and our framework 'Sync-AL' across three generic datasets.
Evaluation Protocol: For fair comparison with existing works (Du et all
Haiyang
KIKE!), we
employ Stnnet-Hungarian our evaluation protocol. The Strict-Hungarian protocol (S. Vaze 4iz calculates
the accuracy of the entire query set at once, thereby mitigating the possibility of clusters being redundantly utilized by both new and old categories. The clustering accuracy (ACC) is calculated by making use of the ground truth labels yi and the predicted cluster labels Jr:
Del Σ 10==((w)
ACC
max PEP(Y
DoO
where P (Y2) denotes the set of all possible permutations of the class labels in Do- The optimal assignment. is calculated using the Hungarian algorithm (W,) 19:5
402 we adopt B- (A. Kolesnikov & Zhail, 202T) as the feature extraction backbone, that has been pre-trained with DINO (C. Mathildd 2021) on Imagenet. A thrce-layer multi-layer perceptron (MLP) serves as the projection head, generating a 768-d feature vector as the out Fine-tuning the ViT backbone is limited to the last block. For all the experiments, a batch size 128 is utilized. The reported results have been obtained by training for 50 and 100 epochs on gaasee-rained and ino-grained datasets respectively. We use SGD with a momentum of 0.9, initial learning rate of 0.01 gradually annealed using a cosine scheduler. The active learning budget is kept to be triple the number of labeled classes. All experiments utilize single NVIDIA RTX A5000 GPU. We use 0.0001 as the constant threshold for the margin- OACD baseline. The constant threshold for uncertainty based OACD baseline is kept at 0.7. Confusing buffer size is kept at twice the size of active learning budget. The classifier alignment. frequency is kept at 10. The adaptive threshold parameter (computed based on validation set) is kept at 3 for Food 101, Imagenct- 100 and Stanford Cars datasets, for Oxford Pets and Cifar 10. CUB-200-201 and Cifar100 use g = 2.5 and 3.5 respectively. Additional implementation details are given in Appendix A.1.
Implementation details: As in recent works (S. Vazd
Z. Sheng
W
Experimental Results for OCD task: Table D. report the performance of SynC on the three coarse- grained and four fine-grained datasels using the "Strict- protocol. All denoles the overall accuracy
Oxford Pets
Food101
2UB-200-2011
Stanford
Methods
All
Old
New
All
Old
New
All
Old
New
All
Old
New
Margin-based baseline
60.56
81.12
49.75
32.50
54.95
21.05
46.28
67.47
85.67
25.92
44.52
16.72
Uncertainty-base baseline
62.13
62.67
61.84
32.37
57.15
19.78
46.57
71.40
34.19
25.29
26.72
19.62
SynC-AL
63.04
64.37
62.34
34.68
64.59
19.43
50.56
67.47
42.08
26.10
36.73
20.85
Table 5:
Experimental results showing the baselines of OACD setting and our framework 'Sync-AT' across
four fine-gue fin-ggriied datasets.
10

---

Published in Transactions on Machine Learning Research (September/2022
Figure 3: All the above samples are incorrectly marked as novel by SynC. Out of these examples, the samples correctly identified as belonging to previously-seen classes by SynC-AL are in green box, whereas samples that are wrongly marked by both SynC and SynC-AL. are shown in the red box.
while Old and New denote the accuracies for known and novel classes. The results of the other approaches are taken directly from (Dn et all 2023). SynC consistently outperforms the state-of-the-art approaches for all course-grained and three of the four fine-grained datasets, except Stanford Cars, making it an effective framework for OCD. Bold and underline represents the highest and second highest accuracy respectively.
Experimental Results for OACD task: Since there are no existing baselines for the On-lhe-fly Active Category Discovery (OACD) task, we compare the proposed SynC-AL framework with two strong baselines, which uses full supervision in active learning module in contrast to our extremely Light-weight active lcarming
module. Specifically, we report results on (i) Constant confidence and (Prrabhu el all 202T) (ii) Margin threshold. (Kie et al,, 2022) (Giu et all, 20221) Note that the active learning module is used to detect examples from novel classes, and thus standard baselines like random sampling, etc. cannot be utilized.
The performance of the two baselines and the proposed SynC-AL are reported in Tables EI, 5 for the course- grained and fine-grained datasets respectively. We observe that the proposed framework performs favorably with respect to the two baselines. For instance, while the OACD uncertainty-buased baseline attains 19. 19.73% accuracy on new classes in Food101, its old-class accuracy plummets to 57.15% Our SynC-AL framework not only maintains robust performance on new classes (19.43%) but also significantly enhances old-class retention (64.59%), achieving higher overall accuracy (34.68% vs. 32.50%). This pattern persists across other datasets such as CUB-200-2011, where SynC-AL increases the overall accuracy to 50.56% while also closing the gap between old and new classes.
Note that the two basclines uses complete supervision. Complete supervision implics that the oracle gives the class names of the query samples predicted as belonging to novel classes. So new class nodes are only created for the genuine new classes. In the proposed light- SynC-AL framework, if an example from seen class does not belong to the top-k predicted classes, it will be mistakenly considered class, leading to degraded performance. Also, unlike our framework, complete supervision requires more domain expertise. Still, the proposed light- SynC-AL. is able to correct several misclassified samples, that were mistakenly identified as novel by SynC, but actually belong to previously seen classes as shown in Figure H. This correction improves the accuracy of old classes, as these samples are reassigned to their correct categories. Few qualitative results are shown in Figure Figure E exhibits the growth of the confusion buffer while encountering samples during inference. It is important note that. the maximum size of the confusion buffer is bounded by the active learning budget, which typically constitutes small fraction of the total number of input samples. In SynC-AL, the buffer can grow up to the full active learning budget. However, we find that even a reasonably small confusion buffer can significantly enhance performance.1t It can be observed in figure E that the growth of the confusion buffer slows down as the model processes more samples.
novel
Table 6: Performance (accuracy %) of our framework SynC(w/o Classifier Alignment) with varying values of used for adaptive class-specific threshold compu- tation.
Table 7: Fdffect of frequency of classifier update. Per- formance (accuracy %) when classifiers are updated after a certain number of novel classes are discovered.
Classifier
CIFAR 100
CUB-200-200
Imagenet-
Stanford Cars
Alignment
All
Old
New
All
Old
New
All
Old
New
All
Old
New
10
56.13
68.43
31.54
49.44
49.07
49.63
3
42.96
89.45
19.60
23.36
30.50
19.82
20
55.51
66.83
32.87
45.33
54.27
40.85
3.5
43.76
89.85
20.59
20.69
25.37
18.38
30
55.64
66.95
33.02
43.12
50.33
39.51
4
43.81
87.70
21.75
20.68
28.87
16.62
11

---

Published in Transactions on Machine Learning Research (September/202) 2025)
Novel Clesses Predicted: Ground Truth va SynC-AL vs SynC
703
SySC-4L
30
1300
2010
5200
4003
Figure 4: Barplot showing how estimated number of novel classes improves using SynC-AL.
Additional Analysis: Here, conduct additional analysis of the different parameter choices.
1) Effect of Adaptive Threshold Parameter: Table T shows how different values of the hyperparameter in the adaptive threshold step (Eqn. (EI) impacts the balance between old and new class performance, with smaller g slightly favoring old-class accuracy (e.g., 89.45% for g=3 on Imagenet-100) and larger g slightly improving new-class accuracy (e.g., 21.75% for g=4). But the performance is quite robust with different. values of g.
Table 8: Performance (accuracy %) of our framework SynC with different text. encoder, namely SBERT and CLIP
Table 9: Effect of varying the number of text prompts used for generation ion of textual descriptions of each class.
CUB- SUBB-200-200
200-2011
Stanford Cars
Method
All
New
Old
New
Number of prompts
Oxford Pets
PHE SynC(SBERT)
36.4
55.8
27.0
31.3
61.9
16.8
All
Old
New
45.93
54.27
40.85
24.6
34.8
19.55
60.01
76.25
51.75
SynC(CLIP)
52.31
54.73
51.10
29.7
45.21
22.02
61.65
69.46
57.55
2) Effect of Classifier Alignment Frequency: We observe from Table Q that more frequent classifier alignment updates (every 10 novel classos) generally enhances performance both CIFAR-100 and CUB- 200-2011 datasels, particularly improving accuracy old classes. On the other hand, decreasing the update frequency slightly boosts accuracy on novel classes for CIFAR-100, suggesting a trade-off between preserving old-class accuracy and enhancing novel-class recognition.
3) Effect of different text encoders We conducted the same experiment on the CUB-200-2011 and Stanford Cars datasets using the same set of prompts, but with CLIP replacing SBERT for generating text embeddings as can be seen in Table 8. As expected, observe significant improvement in performance due to presence of the stronger text encoder.
4) Effect of text prompts used for description generation: The prompts are manually chosen and the performance of the framework do depend upon the nunber and quality of prompts. That being said, having more numbers of prompts increases the time taken for the pre-processing step. On the matter of quality of prompts, more particular and data-specific prompts perform better than generalized questions. For example, for the dataset CUB(a datasct consisting of 200 species of birds), 'What does the beak of the bird look like ?" will perform better than 'describe the bird.' All our results for the Oxford Pets da tateet are executed with 4 prompts. We then experimented by reducing the number of prompts, and visible decrease in performance is noticcable in Table B thus supporting our previous insights.
We analyze the contribution of each component of our framework (Table m.). Starting from a simple supervised contrastive baseline (C*), we progressively introduce textual contrastive loss (C*), prototypical contrastive loss (LP) and classifier alignment. Finally, we incorporate lightweight active Icarning (SynC-AL) with and without confusion buffer. This analysis clearly demonstrates that each additional component incrementally enhances overall performance, validating their importance.
5) Ablation Study:
12

---

Published in Transactions on Machine Learning Research (September/2022 2002)
Growth Confusion Ruffer Samples
Corfudan BuRerrte
1050
3300
2000
2000
3010
3500
4000
4503
Figure 5: Graph illustrating the change in the size of the confusion buffer as the model encounters samples during inference.
Method
L,
Lp
ClassiBerAlign. AL
Conf.Buffer
CIFAIt-10
Olil
AUB-20002001
Stanforrd Cars
All
New
Baseline (17 only)
x
x
37.04
67.24
21.94
90.22
46.53
22.04
21.08
31.05
16.15
only
38.32
57.04
29.26
36.65
56.53
25.18
21.87
32.43
16.64
12 only
39.06
34.05
41.67
33.89
53.33
24.15
23.2
35.6
18.55
E- E' 2'4+2 +-
6
x
49.85
39.80
54.88
37.12
56.67
27.32
21.92
27 64.64
19.08
1/ +Chessiier Aliion
SynC-AL. (AL, w/o Coaf. Budler
/
x
51.75
39.94
07.05 66
40.11
63.40
41.95
SynC-AL [Full model)
62.12
39.65
58.35
50.56
07.47
42.08
26.10
20.85
Table 10: Ablation study highlighting contributions of each component in SynC and SynC-AL frameworks.
CUB-200 220--2002
Estimated # #Novel Class
Estimated #class
Stanford Cars
Estimated
Estimated #class
Methods
All
Old
New
All
Old
New
# Noveel Class
SMILE (16bit)
31.9
52.7
21.5
824
924
27.5
52.5
15.4
798
896
PHE (16bit)
37.6
57.4
27.6
218
318
31.8
65.4
15.6
611
709
SMILE (32ba)
27.3
52.0
14.97
2046
2146
21.9
46.8
9.9
2855
2953
PHE (32bit)
38.5
59.9
27.8
374
474
31.5
61.0
15.8
661
762
SMILE (61bbt)
22.6
45.3
2810
2910
16.5
38.2
4890
4788
PHE (64bit)
38.1
60.1
27.2
393
493
32.1
66.9
15.3
819
917
SynC
45.33
54.27
40.85
395
495
24.60
34.60
19.55
427
525
SynC-AL
51.27
67.27
43.25
155
255
26.10
36.72
20.85
349
447
Table 11: Sync-AL provides a better estimate of the number of classes compared to SynC'. Correct number of classes are 200 and 196 for CUB and Stanford Cars respectively.
We hypothesize that the textual embedding space is limited for some datasels, especially in the case of Stan- ford Cars. The resolution needed for differentiating between 'Chevrolet Corvette ZR1 2012' 'Chevrolet Corvette Convertible 2012' is lacking in the textual embedding space provided by SBERT. Thus, it results in poor performance of our framework. We observe that the performance gain from using text embeddings on this dataset is minimal as seen in Table M. suggesting that stronger text encoder like CL.IP might. yield better results which we obtained in Table & The stronger encoder substantially improves accuracy on both old and new classes for Stanford Cars. Although richer class descriptions might boost performance further, we leave that exploration to future work. With CL.IP, our framework SynC ranks second only to PHE (Z. Haivang, 2024] on Stanford Cars; on every other datasct, however, it significantly outperforms PHE and all competing methods, even when using SBERT.
6) Comparison with SOTA:
The state-of-the-art OCD approaches PHE Z Haivang 2124 and
SMILE ctal, 2023) roly on hash codes to represent category descriptors, introducing inherent challenges associated with the predefined length of hash codes. This fixed-length representation limits the expressive- ness of category features, particularly in fine-grained settings where nuanced feature variations are crucial for distinguishing novel classes. In contrast, our proposed framework, SynC and SynC-AL, circumvents these constraints employing list of prototypes as classifiers, allowing for dynamic category representation without being coint by fixed bit-length encoding.
13

---

Published in Transactions on Machine Learning Research (0eptember/2025)
In addition, the lossy nature of hash-based representations in PHE and SMILE can potentially lead to un- wanted category merging fragmentation. Our prototype-based classification enables better estimation of novel category samples. As a result, our framework produces more accurate count of novel categories compared to the state-of-the-art approaches. We observe from Table M that both PHE and SMILE signifi- cantly overestimates the number novel classes (e.g., PHE (16-bit) predicts 611 novel classes and SMILE (16-bit) predicts 798 novel classes for Stanford Cars). In contrast, the proposed SynC framework provides a significantly more accurate estimate (395 for CUB and 427 for Stanford Cars), though there is still significant. room for improvement.
Morcover, both PHE and SMILE's performance is sensitive to hash code length. As seen in Table I.I increasing the bit-length from 16-bit to 64-bit does not lead to significant improvement in PHE performance, but results in an inflated number of estimated novel categories, highlighting the limitations of tash-based representations, especially for fine-grained category discovery.
Additionally, SynC-AL significantly outperforms SynC, benefiting from an active learning component that refines category boundaries and reduces overestimation errors. Specifically, SynC-AL estimates only 155 novel classes on CUB while the correct number of novel classes is 100, compared to 395 in SynC, indicating that the integration of active learning minimizes misclassification of ambignous instances as novel categories. Similar trends can be observed on Stanford Cars, reinforcing the effectiveness of our framework over fixed- length bash-based methods.
Limitations: One potential limitation of our framework is the reliance on LLM for obtaining the class representation. This might be challenge if the class names uncommon and may not have been present in LLM training. Another potential limitation arises when the generated text descriptions are highly similar to each other, observed in the case of the Stanford Cars dataset, where it's inherently difficult to describe different car models distinctly. Finally, if new classes gets predicted to old classes, then it will not be identified by our active learning framework.
6 Conclusion
In this work, we proposed a novel framework, SynC to address the challenging and realistic OCD task. We proposed a LLM-assisted unique semantic representation for learning the image mappings in addition to a periodic classifier update module for this task. Additionally, we also propose an extended version, SynC- AL which incorporates lightweight active learning, allowing minimal human intervention to correct errors and improve performance. Our proposed SynC and SynC-AL achieved slate-of-the-art performance across several benchmark datasets, significantly outperforming existing methodologies.
14

---

Published in Transactions on Machine Learning Research (September/2025) 2202)
References
K. Akshay L. John A. Alekh A. Jordan T, Z. Chicheng. Deep batch active learning by diverse, uncertain gradient lower bounds. International Conference on Learning Representatios((CLLB)) 2020.
D Weissenborn G. Heigold J. Uszkoreit L. Beyer M. Minderer M. Dehghani N. Houlsby S. Gelly T. Unterthiner
A. Kolesnikov, A. Dosovitskiy and X. Zhai. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations
2021.
A. Saket A. Chetan A. Sharat, A. Himanshu. Contextual diversity for active learning. In European Conference on Compuler Vision(ECCV) 16, 2020.
V. Gool Luc B. Lukas, G. Matthicu. Food-101 - mining discriminative components with random forests. In European Conference on Computer Vision (ECCV), 2014.
Anwesha Banerjee, Liyana Sahir Kallooriyakath, and Soma. Biswas. Amend: Adaptive margin and expanded neighborhood for efficient generalized category discovery. In Proceedings of the IEEE/CVF Winter Con- ference on Applications of Computer Vision (WACV), pp- 2101-2110, Jamuary 2024.
M. Ishan J. Hervé M. Julien B. Piotr J. Armand C. Mathilde, T. Hugo. Emerging properties in self- supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
N. Mohammad Н. Geoffrey C. Ting, K. Simon. A simple framework for contrastive learning of visual reprosontations. In International Conference on Machine Learning (ICML), 2020.
Ruoyi Du, Dongliang Chang, Kongming Liang, Timothy Hospedales, Yi-Zhe Song, and Zhanyu Ma. On-the- fly category discovery. In IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPPP pp. 11691-11700, 2023.
Chufan Gao, Mononito Goswami, Jieshi Chen, and Artur Dubrawski. Classifying unstructured clinical notes via automatic weak supervision. In Machine Learning for Healthcare Conference, pp. 673-690. PMLR, 2022.
Shurui Gui, Xiner Li, and Shuiwang Ji. Active test-time adaptation: Theoretical analyses and an algorithm. arxiv preprint 2rx:::2404.0005 050094, 2024.
E. Sebastien V. Andrea Z. Andrew H. Kai, R. Sylvestre-Allise. Autonovel: Automatically discovering and learning novel visual categories. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
Z. Zua H. Sheng-Jun, J. Rong. Active learning by querying informative and representative examples. A Advances in Neural Information Processing Systems(NeurIPS), 2010.
J. A. Hartigan. Clustering algorithms. John Wiley Sons, Inc., 1975.
Z. Yukun G. Bradley J. Xuhui, H. Kai. Joint representation learning and novel category discovery single-and multi-modal data. In IEEE/CVF International Conference on Compuler Vision (ICCV), 2021.
D. Jia F. Li K. Jonathan, S. Michael. 3d object representations for fine-grained categorization. In Proceedings of the IEFE international conference on compuler vision (ICCV) workshops, 2013.
W. Chen S. Aaron T. Yonglong I. Phillip M. Aaron L. Ce K. Dilip K. Prannay, T. Piotr. Supervised
contrastive learning, Advances in Neural Information Processing Systems(NeurIPS)) 2020.
Alex Krichevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
S. Yi-Zhe H. Timothy L. Da, Y. Yongxin. Learning to generalize: Meta-learning for domain generalization.
In AAAI conference on artificial intelligence, 2018.
S. Jiatao-C. Xinxiang Z. Bingchen H. Shaoguang L. Tingzhang, D. Mingxuan. Contextuality helps repre- sentation learning for generalized category discovery. In 2024 IEEE International Conference on Image Processing (TCTP), 2024.
15

---

Published in Transactions on Machine Learning Research (2pptember/2025)
Z. Zhun-Z. Yao L. Cheng-Lin M. Shijie, Z. Fei. Active generalized category discovery. IEEE/CVE Conference on Compuler Vision and Poltern Recognition (CVPR), 2024.
R. N. Centence-bert: Sentence embeddings using siamese bert-networks, arXie preprint arXie: 1908.1090.10004,, 2019.
Zachary Novack, Julian McAuley, Zachary Chase Lipton, and Saurabh Garg. Chils: Zero-shot image classi- fication with hierarchical label sets. International Conference on Machine Learning, pp. 16342-26300 PMLR, 2023.
S. Kyungwoo P. Gycong-Moon P. Keon--ee, L. Hakyung. Online continuous generalized category discovery. In European Conference on Compuler Vision (ECCV), 2024.
S. Nicu P. Nan, Z. Zhun. Dynamic conceptional contrastive learning for generalized category discovery. In IEEE/CVE Conference Computer Vision and Pattern (ecoonition(CVPP) 2023.
Z. Andrew J. C.P. Omkar M, V. Andrea. Cats and dogs. In 2012 IEEE Conference on Computer Vision and Pattern Recognition (Ctpiittiiio(RR).... CVPR), 2012.
L. Rosanne F. Ali P. Sarah, C. Ian. What does a platypus look like? generating customized prompls for zero-shot image classification. In JEFE/CVF International Conference on Computer Vision (ICCV), 2023.
Viraj Prablu, Arjun Chandrasekaran, Kate Saenko, and Judy Hoffman. Active domain adaptation via clustering uncortainty-weigee weiihht embeddings. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 1505-8514, 2021.
S. Kevin R. Dan. Margin-based active learning for structured output spaces. In European Conference Machine LearningEECML.), 2006.
S. Haim S. Н. Sebastian, O. Manfred. Query by committee. In Proceedings of the fifth annuua workshop Compulational learning theory, 1992.
S. Silvio S. Ozan. Active learning for convolutional neural networks: A core-set approach. International Conference on Learning (epreseentations((CLL)) 2018.
A. Vedaldi A. Zisserman Vaze, K. Han. Generalized category discovery. In IEEE/CVF Conference on Computer Vision and Pallern Recognition (CVPR), 2022.
Burr Settles. Active learning literature survey. 2009.
Phillip Yonglong, Dilip. Contrastive multiview coding. European Conference on Computer Vision, 2020.
Grant Van Horn, Steve Branson, Paul Van Horn, Ryan Farrell, Stefan Haber, Elijah Barry, Panos Ipeirotis, Pietro Perona, and Serge Belongie. inaturalist: Towards fine-grained biodiversity classification. In Pro- ceedings of the IEFE Conference Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 876 885, 2017.
K. Harold W. The hungarian method for the assignment problem. Noval research logistics quarterly, 1955.
H. Kai W. Hongjun, V. Sagar. Spinet: An efficient alternative framework for generalized category discovery with spatial prompt tuning. International Conference on Learning Representattinn//IILR]] 2024.
Q. Xiaojuan W. Xin, Z. Bingchen. Parametric classification for generalized category discovery: baseline study. In IEEE/CCI EVFF International Conference on Computer Vision (ICCV), 2023.
Peter Welinder, Steve Branson, Takeshi Mita, Catherine Wah, Florian Schroff, Serge Belongie, and Pietro
Perona. Caltech-uesd birds 200. 2010.
16

---

Published in Transactions on Machine Learning Research (Cpttember2025))
Binhui Xie, Longhui Yuan, Shuang Li, Chi Harold Liu, Xinjing Cheng, and Guoren Wang. Active learning for domain adaptation: An energy-based approach. In Proceedings of the AAAT conference on arlificial intelligence, volume 36, pp. 8708 8716, 2022.
L. Wenjing S. Nicu Z. Zhun 7. Haiyang, P. Nan. Prototypical hash encoding for on-the-fly fine-grained category discovery. In Advances in Neural Information Processing Systems (NeurIPS), 2024.
S. Zhiqiang N. Muzammal Guangyi K. Fahad Shahbaz Z. Sheng, K. Salman. Promptcal: Contrastive affinity learning via auxiliary prompts for generalized novel category discovery. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pallern Recognitioon (CVPRR) 2023.
L. Qing C. Antoni B Z. Xueying, L. Huan. A comparalive survey: Benchmarking for pool-based active learning. In International Joint Conference on Artificial Inteligence(LJCAA), 2021.
Sheng Zhang, Muzammal Naseer, Guangyi Chen, Zhiqiang Shen, Salman Khan, Kun Zhang, and Fahad Shah- baz Khan. S3a: Towards realistic zero-shot classification via self structural semantic aligument. In Pro- ceedings of the AAAI Conference on Artificial Intelligncee volume 38, pp. 7278 7286, 2024.
17

---

Published in Transactions on Machine Learning Research (tttembbe/22025)
Text prompts used for description generation
Datasets
Number Prompts
Describe what an O looks like
CIFAR10
3
Describe a(n) {}:
What are the identifying characteristics of a(n) {
Describe a photo of a(n) ():
Describe what a(n) { looks like:
CIFAR100
4
Describe a(n) ():
What are the identifying characteristics of a(n) {
Describe what a(n) ( looks like
A caption of an image of a(n) (::
ImageNNet-100
5
How can you identify a(n) ()? What does a(n) look like?
Describe an image from the internet of a(n) 0
Describe the bird {
CUB-200-2011
3
Describe what the bird ( looks like.
What are the identifying characteristics of the bird,a(n) {}?
How you identify a(n) {
Description of a(u) 0, a type of car
A caption of a photo of a(n) {
What are the primary characteristics of a(u) ()?
Stanford Cars
Description of the exterior of a(n) 0
What are the identifying characteristies of a(n) { a type of car?
Describe an image from the internet of a(n) 0
What does a(n) { look like?
Describe what a(n) O, type of looks like
Describe what a(n) { looks like Describe the color and shape of a(n) {
Food 101
4
What does a(n) O taste like?
what is the cuisine of a(n) ()
Describe what a(n) O looks like
Describe the key physical characteristics a(n) O for example:(size coat type colors, and distinctive features)
Oxford Pets
What is the typical temperament and interaction style a(n) 0 What is the origin or history of a(n) (
Table 12: Table denoting the prompts used for text generation using LLM
A Appendix
A.1
Implementation Details
We used
for the generation of our text descriptions from prompts appended with class- names. Following this, we used Sentence-BERT to generate our text embeddings using the prompts men- tioned in table T2.
A.2 Baselines pertaining On-the-fly Active Category Discovery
On-the-fly Active Category Discovery (OACD) requires that each sample be inferred instantancously, and the decision to request for human intervention must be made in real-time. Since inference occurs
18

---

Published in Transactions on Machine Learning Research (2ptemmber20255)
sample at time, traditional active learning strategies such random sampling, Chery-by-Commitee
(QBC)(S. H. Sebastian,
1992) cannot. be directly applied.
Baseline Sampling Strategies
A.2.1 Constant Confidence Threshold
We employ a constant confidence threshold as the sampling strategy. Given an image sample with an Iz-normalized feature representation 2, we compute its cosine similarity with the prototypical class reprosentatives "p..p",....p" of previonsly seen categories. The similarity score for class is defined as:
pp |==l||pp
cos(
E (I,...,)) L).
(10)
The highest similarity score across all previously seen categories is given by:
Smax
max Sc.
(11)
OE cE(1.....)
If Smax falls below predefined confidence threshold T, the sample is considered probable novel instance:
8max
<.
(12)
In such cases, we activate the active learning module to determine whether the sample belongs to an entirely new category or aligns with an existing one.
A.2.2 Constant Margin Threshold
In addition to the constant confidence threshold, we implement a constant margin threshold as an alternative sampling strategy. Instead relying on an absolute similarity threshold, we evaluate the relative difference between the highest and sccond-highest cosine similarities of the lg-normalized feature representation 2; with the prototypical class representatives.
Let the top two similarity scores be defined as:
8max
max cE (I (I......
82nd
max eE(I.......) Yarg
8p-
(13)
The margin between these two scores computed as:
As Smax 82nd
(14)
If this margin falls below predefined threshold Tm, the sample is considered ambiguous and likely belong to novel category:
As < Tin-
(15)
In such cases, we trigger the active learning module to determine whether the sample requires further supervision.
A.3 Algorithm
A.4
Semantically similar classes exhibit similar variations in their feature distributions
To evaluate whether semantically similar classes exhibit similar variations in their feature distributions, we perform the following experiment:
19

---

Published in Transactions on Machine Learning Research (September/2255)
Algorithm Light- Weight Active Learning module of SynC-AL
Input:Test set XQ prototypical class representative classifiers P = {p' , ,,,p'p
Output: prediction
while 2; X
Compute normalized feature žy from
each class c || do Compute cosine similarity sime between and p if sime 2 To then
Assign to class
prediction = class
end if end for if sime TeVe then
Check against Confusing Buffer M
if Similarity with confusing sample is high then Classify x, accordingly prediction index of the class
else
Use Active learning budget available if 2, known classes then Update Confusing Buffer M prediction index of the class
else
2: novel class. Update P accordingly prediction = last index of
end if end end if end while
First, for every pair of classes, we compute their text-embedding similarity, defined as the cosine similarity between their normalized lext embeddings. This quantifies how semantically similar the two classes are. Second, we compute covariance- based similarity by measuring the Frobenius norm of the difference between their feature covariance matrices and converting this distance into a similarity score (c.g., using an exponential transformation).
Finally, we compute agreement score. For each class, check whether its nearest neighbor (based text-embedding similarity) is among its top-3 most similar classes based on covariance similarity. The percentage of classes for which this condition holds is then reported for different datasets. For CIFAR-10, Oxford Pets, and Food-101, the agreement scores are found to be 90.0%, 78.4%, and 82.2%, respectively, supporting the underlying assumption.
For example, in the OxfordPets dataset, "American pit bull terrier' is dog breed whose text embeddings are close to that of 'Staffordshire bull terrier', and its top-3 most similar classes based on covariance similarity are 'Staffordshire bull terrier", 'boxer', and "pug".
A.5 Few experimental results
Here we have provided our full method, SynC( without textual regularization), and a comparison with other text encoders and SOTA PHE(Z. Haivang, 20221), for the dataset CUB-200-2011. From Table IX. it is evident that the proposed SynC' framework consistently outperforms the state-of-the-art method PHE on the fine- grained CUB dataset, even in the absence of textual guidance (i.c., when using vision-only input). Moreover, replacing SBERT with RoBERTa, whose pretraining corpus does not include image captions, still results in a notable performance gain. These findings indicate that the improvement arises from the models effective use of distributional semantics, rather than reliance on implicit visual priors within the language model.
Here we provide the results of our framework Synes performance on another fine-graine grainer dataset, Arachnida, subset of iNaturalist (Van Horn all, 2017) dataset.
20

---

Published in Transactions on Machine Learning Research (September/2025)
Text Encoder Variant
ALL
OLD
NEW
PHE
36.40
55.80
27.00
SynC(Vision only)
43.17
72.67
28.39
SynC(RoBERTa)
39.30
55.60
31.13
Sync(SBERT)
45.33
54.27
40.85
SynC(CLIP)
52.31
54.73
51.10
Table 13: Performance across Text Encoder Variants
Method
ALL
OLD
NEW
SMILE
29.90
57.90
12.20
PHE 37.00
75.70
12.60
SynC
43.24
70.97
30.87
Table 14: Performance of our framework SynC on 'Arachnida.
The results in Table [ show that our SynC framework surpasses the current state-of-the-an method, PHE. The dataset Arachnida includes fine-grained class names such as Loxosceles reclusa, Mastigoproctus gigan- teus, and Menemerus bivittatus, which are unlikely to be present in the pretraining corpus of SentenceBERIT. This suggests that the observed performance gains are not a result of any incidental visual-language align- ment in the text encoder. For all experiments, the adaptive threshold parameter is set to 3.5, and we use two prompts per known class to generate textual descriptions.
21


Reviews:
[{"review_id": "9dwBThCWRM", "reviewer": "Reviewer_MKmk", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n1. The paper is well written and easy to follow. \n2. The method performs well on the compared datasets, against the compared baselines. \n3. The experimental analysis appears to be robust, especially the ablation experiments. \n\n## Weaknesses\n1. **Formal definition of the problem and constraints:** The paper lacks a formal definition of the problem at hand. I would encourage the authors to express the assumptions and constraints of the problem at hand. For example, are we assuming that the images in novel categories are drawn from the same distributions (e.g., old classes: animal, novel classes: animals, or novel classes: cars). Similarly, the constraints must be clearly mentioned. For example, specify and describe the constraints that images cannot be stored due to storage or privacy concerns (Page 6), and other resource constraints (Page 7), while formulating the problem. The differences between Generalized, Online, and On-the-fly Category Discovery are unclear. I am not sure what the key differences between these different problems are. Finally, it is unclear what happens if labels are not of the same \"level\". For example, given an image of a horse, should it be categorized as a  horse or an animal? \n2. **Outputs:** My understanding is that the output of the method is not a class label for novel classes. Is my understanding accurate? How are class labels for novel classes generated? \n3. **Prompts:** It appears that the prompts $(v_1, \\dots, v_d)$ are manually chosen. How does the number and quality of prompts impact the method? \n4. **Experimental protocols:** \n    1. Why are the baseline methods in Table 2 and 3 different? Are the baselines latest?\n    2. It is unclear if the gains in some cases are significant. I recommend that the authors run each experiment with 3 different random seeds, and different samples, and numbers of known and unknown classes, to ascertain if the performance gains are significant. The means and standard deviations across these runs should be reported.\n    3. I am not sure what the strict Hungarian protocol entails. \n5. **Prior Work (minor):** \n    1. The idea of using LLMs for zero- (or few-) shot classification is not new. Some studies such as [1] have already used similar concepts before. \n    2. I am not sure what the downsides of having \"hash codes of predefined lengths\" are? \n6. **How present/future-proof are these methods?** Perhaps, the biggest issue I have is if the proposed methods perform on par with Large Vision Language Models. How do LVLMs such as ChatGPT or Claude perform on this task? Why do we need such a method?\n7. **Results:** Why is performance on the Stanford Cars dataset poor?\n8. **Hyper-parameters:** How are hyper-parameters for the method chosen? \n\n\n## References\n1. Gao, Chufan, et al. \"Classifying unstructured clinical notes via automatic weak supervision.\" Machine Learning for Healthcare Conference. PMLR, 2022.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes an image classification method, where the number and nature of classes are unknown. It is trained on a dataset on a subset of known classes. During inference, images are received one-by-one. Each each image can belong to one of the known classes, or to a novel class. The goal of the method is accurately classify images, as they appear, in an online setting. The authors propose a method which combines large language model based descriptions of existing class labels, contrastive learning, and active learning to accurately classify images in this setting. The method is evaluated against certain baselines, on a widely-used image classification benchmarks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please respond to the questions in the previous section. Please prioritized responding to questions / comments 4 -- 7.", "broader_impact_concerns": "I do not foresee any broader impact concerns. The paper also does not have a broader impact section.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "PcfR4sbqjv", "reviewer": "Reviewer_HWe6", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nOnline type learning, combined with active learning, is a smart strategy that shows better performance. The most prominent improvement is related to LLM's embeddings, which are aligned with the development of powerful LLM models.\n\nWeakness:\nThe paper does not reveal the whole picture of learning. The dimension of S-BERT is not emphasized, and some essential parts clarified are missing.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper explores On-the-Fly Category Discovery using LLM embeddings and active learning to address the uncertainty problem. They showed the framework applied in online learning, which is practical. The main losses are compromised by contrast losses using the embedded samples and prototypes of classes, as well as the LLM-assistant prototypes. The algorithm is almost clearly explained, but some issues need to be clarified. The experiments are thoroughly conducted with various aspects of the proposed algorithm. However, some issues need clarification. The contribution of this paper appears moderate, and I recommend identifying the main contributor to the improved performance. The LLM embedding or active learning? The LLM embeddings appear to be most prominent in the ablation study. It can be stated more impressively.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "$\\bullet$ The simulation or learning can check the following claim:\n1) This approach assumes that semantically similar classes will exhibit similar variations in their feature distributions (for the novel classes, we use the covariance matrix, the mean of which is most similar. This assumption looks strong. Therefore, some validations are required.   \n2) The size of the confusion buffer in active learning can be recorded in the learning phase. Please draw the graph illustrating the size of the confusion buffer. \n\n$\\bullet$ I have a question regarding formula (6); specifically, the right-hand side (RHS) of formula (6) has the index of $c$. However, there is no validated index $c$ on the left-hand side. I don’t know the dependency of $c$ on $\\tau_c$.\n\n$\\bullet$ I have a question about the section on Test-time update of the classifiers. At test time, the update of a classifier is performed using the cross-entropy method, which applies to the DNN classifier. However, it is not consistent with the classifier in Section 3.2.1 at first glance. Please clarify this issue.", "broader_impact_concerns": "This study can impact the vision learner in finding a new class. It is crucial in the auto-driving or other vision-based alert systems.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "RXoy40pdXP", "reviewer": "Reviewer_EQNJ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This is the first prototypical methods that demonstrate SOTA performance on OCD problem, outperforming the previous coding based methods. I personally consider this learning paradigm is more advantageous in robustness in that code learning is highly sensitive to the code length, making it difficult to determine in practice. However, to truly prove its edge, more experiments are required (see below).\n\nI hypothesize that the textual embedding space may be limited for some datasets, e.g., StanfordCars. It is recommended to conduct additional ablations on this dataset since it does not achieve SOTA performance on it; besides, we can try more text embedding models to gain deeper insights, e.g., how the performance will improve w/ CLIP textual encoder and degrade w/ weaker textual models. Meanwhile, how the performance will change w.r.t. the output dimensionality of the 3-layer projector? To prove that its relatively low sensitivity w.r.t. the previous coding based methods can be contributions.\n\nThe technical novelty is limited as most of its components are existing practices.\n\n\nNovelty: existing problem, problem of limited real-world scenarios, innovative combinations in proposed methods\nSignificance: concerns on generalizability on more datasets, good improvements on some datasets, medium or low future potential\nTechnical: Sound, Correct, Reasonable, Appropriate", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work proposes an integrated framework, SynC and SynC-AL, as strong baselines to address the OCD problem.\n\nSpecifically, it encompasses self supervised contrastive learning to prevent corruption of the representation, image-text contrastive learning to maintain representation generalizability, and prototypical learning to learn visual semantic structures.\n\nBesides, it utilizes class-specific statistical threshold as adaptive margin as the open-set classifier and novel-class classifier.\n\nIt proposes TopK comparison as cheap active learning query for test-time learning.\n\nIn contrast to previous coding based OCD methods, it can achieve SOTA performance on most of datasets.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Provide more in-depth investigations into the language utility: (1) How does language embedding space boost the performance? Is there any potential information leakage? (2) To what extent can linguistic representation benefit the semantic learning? (3) Are there any limitations on language representation? It would be appreciated if you could include an additional breakdown of ablations on the StanfordCars dataset for examination.\n\n\nHighlighting the difference between this work and conventional zero-shot learning is necessary, since their paradigms are highly homogeneous.\n\n\nLack discussions on related works using prompt augmentation for category semantics [1,2,3].\n[1] Chils: Zero-shot image classification with hierarchical label sets\n[2] What does a platypus look like? generating customized prompts for zero-shot image classification\n[3] S3A: Towards Realistic Zero-Shot Classification via Self Structural Semantic Alignment", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)