Prototypical Hash Encoding for On-the-Fly Fine-Grained Category Discovery
Authors: Haiyang Zheng
Nan Pu
Wenjing Li
Nicu Sebe
Zhun Zhong
Venue: NeurIPS 2024 poster (2024)
Keywords: Category Discovery
On-the-fly
Deep Hash
Primary area: machine_vision

Abstract: In this paper, we study a practical yet challenging task, On-the-fly Category Discovery (OCD), aiming to online discover the newly-coming stream data that belong to both known and unknown classes, by leveraging only known category knowledge contained in labeled data. Previous OCD methods employ the hash-based technique to represent old/new categories by hash codes for instance-wise inference. However, directly mapping features into low-dimensional hash space not only inevitably damages the ability to distinguish classes and but also causes ``high sensitivity'' issue, especially for fine-grained classes, leading to inferior performance. To address these drawbacks, we propose a novel Prototypical Hash Encoding (PHE) framework consisting of Category-aware Prototype Generation (CPG) and Discriminative Category Encoding (DCE) to mitigate the sensitivity of hash code while preserving rich discriminative information contained in high-dimension feature space, in a two-stage projection fashion. CPG enables the model to fully capture the intra-category diversity by representing each category with multiple prototypes. DCE boosts the discrimination ability of hash code with the guidance of the generated category prototypes and the constraint of minimum separation distance. By jointly optimizing CPG and DCE, we demonstrate that these two components are mutually beneficial towards an effective OCD. Extensive experiments show the significant superiority of our PHE over previous methods, e.g. obtaining an improvement of +5.3% in ALL ACC averaged on all datasets. Moreover, due to the nature of the interpretable prototypes, we visually analyze the underlying mechanism of how PHE helps group certain samples into either known or unknown categories. Code is available at https://github.com/HaiyangZheng/PHE.

Prototypical Hash Encoding for On-the-Fly Fine-Grained Category Discovery
Haiyang Zheng'; ; Nan Pul; Wenjing Li Nicu Sebe', Zhun Zhong2t
UUiiversity of Trento Heefii University of Technology
Abstract
In this paper, we study practical yet challenging task, On-the-fly Category Discov- ery (OCD), aiming to online discover the newly-coming stream data that belong both known and unknown classes, by leveraging only known category knowledge contained in labeled data. Previous OCD methods employ the hash-based technique to represent old/new categories by hash codes for instance-wise inference. How- ever, directly mapping features into low-dimensional hash space not only inevitably damages the ability to distinguish classes and but also causes "high sensitivity" is- sue, especially for fine-graide fine-grained classes, leading to inferior performance. To address these issues, we propose novel Prototypical Hash Encoding (PHE) framework consisting of Category-aware Prototype Generation (CPG) and Discriminative Category Encoding (DCE) mitigate the sensitivity of hash code while preserving rich discriminative information contained in high-dimension feature space, in a two-stage projection fashion. CPG enables the model fully capture the intra- category diversity by representing each category with multiple prototypes. DCE boosts the discrimination ability of hash code with the guidance of the generated category prototypes and the constraint of minimum separation distance. By jointly optimizing CPG and DCE, we demonstrate that these two components are mutually beneficial towards an effective OCD. Extensive experiments show the significant superiority of our PHE over previous methods, e.g., obtaining an improvement of +5.3% in ALL ACC averaged on all datasets. Moreover, due to the nature of the interpretable prototypes, we visually analyze the underlying mechanism of how PHE helps group certain samples into either known or unknown categories. Code is available https://github wwctuu...o//ttt.. HammHHaagnZhnng/HHE.
Introduction
While deep learning based machines have surpassed humans in visual recognition tasks [1, 2, 3, 4], their capability is often limited to providing closed-set answers, e.g., category names. In contrast, humans possess the ability to recognize novel categories upon first observation without knowing their category names. To bridge this gap, Novel/Generrlizee Category Discovery (NCD/GCD) techniques [5, 7, 8, 9, 10, 11] proposed to transfer knowledge from known categories to distinguish unseen ones. However, current NCD/GCD methods operate under an offline inference paradigm where the category discovery is often implemented by applying clustering unsupervised classification algorithms on a pre-collected batch of query data that needs to be discovered. This severely limits the practicability of NCD/GCD techniques in real-world applications, where the systems are expected to provide online feedback for every newly-coming instance.
To tackle this drawback, Du al. introduce the On-the-fly Category Discovery (OCD) task [12], which removes the assumption of a predefined query set and requires instance feedback with stream data input. OCD poses two primary challenges: 1) The requirement for real-time feedback is
"Equal contribution. t Corresponding authors.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

(a)
Pre-collected Ratch Data
Known Class
SMILE
Our PHE
OMfin Discovery
Offline
Hash Encoding
Level
Prototypical Hash Encoding
Clustering
dinss
VIT
VIT
Unknown
(b)
Online Discovery
Stream Data
Known Unknown
Class if
10010100
(c)
Methods
Ground Truth Class 200
Code Length
All Alc
(%)
Estimated
# Class
Accuracy
0.35
0.30
Instant Inforence
i
SMILF
Ours
12
12
3222
36.4
580
339
0.20
0.21
SMILE
10101001
SMILE
Ours
64
64
38.1
22.6
2910
493
0.15
0
20 20
Our PITE
60 30 100
High Sensitivity
Training Epoch
Figure 1: (a) Schema of Offline Category Discovery task (e.g. NCD [5] and GCD [8]. (b) Schema of On-the-fly Category Discovery task [12], studied in this paper. (c) Previous work (e.g., SMILE [12])
based on instance-level hash encoding. (d) Our PHE explores prototype-based hash encoding. (e) Performance comparison of PHE and SMILE and observation about "High Sensitivity".
incompatible with offline cluster-based methods [8, 9] (see Fig. (a)). 2) Due to uncertain open- world scenarios, the NCD/GCD methods (e.g., classifier based methods [13]), which assume the number of categories can be available/discovvred as a prior, are ineffective for OCD. As a solution, Du et al. propose the SMILE architecture [12], which employs the sign of an image feature (i.e., hash code) as its category descriptor to identify the corresponding category for online discovery. As illustrated in Fig. (c), SMILE directly mapps image features into low-dimensional hash space with an instance-level contrastive objective and regard that one hash code uniquely represents a category. Given this, although SMILE can derive category descriptors, it suffers from significant issue of "high sensitivity" for learned hash-form category descriptors and thus produces a significantly inaccurate number of categories well unsatisfied performance. As shown in Fig. (e), this drawback of SMILE is even more severe for fine- grained categories due to the small inter-class variations and large intra-class variations between samples. For example, given two images of the same bird category but with different poses, they commonly share very similar overall features. However, under SMILE, minor variations in single feature dimension can lead to opposite signs in their hash codes when the feature values are close to zero, classifying these two samples into distinct categories. Moreover, this issue becomes more pronounced the feature dimensionality increases, as demonstrated in Table 5.
To address this issue, we propose a new two-stage Prototypical Hash Encoding (PHE) framework to improve the intra-class compactness and inter-class separation of category descriptors while alleviating information loss due to dimension reduction. PHE is composed of Category-aware Prototype Generation (CPG) and Discriminative Category Encoding (DCE). First, CPG utilizes prototype-based interpretable classification models to learn multiple prototypes for each category and yields sparse representations with a probabilistic masking strategy. Then, DCE explicitly maps the prototypes of each category to corresponding category hash centers. By imposing hash codes approaching their hash centers, OCD models are encouraged to produce more accurate hash codes. Meanwhile, we design center separation loss ensure that different category hash centers maintain at least a Hamming distance of dmaz, where the d is derived from Gilbert- Varshamov bound in coding theory, illustrated in Fig. (d). Finally, we design tailor-made made OCD model inference method based on relaxed Hamming ball condition. Overall, the two-stage PHE framework enables OCD models maintain the diseriminabilities learned in high-dimension feature space and transfer them into low-dimensional encoding space, thereby enhancing accuracy for both seen and unseen categories. Extensive experiments conducted across multiple fineggrained benchmarks demonstrate that our approach substantially outperforms the SMILE architecture (see Fig. (c)). Our contributions are summarized as follows:
We propose new PHE framework to explicitly generate prototypical hash centers, which is beneficial for improving the intra-class compactness and the inter-class separation of hash codes.
Our PHE can effectively alleviate the "high sensitivity" issue of hash-based OCD methods.
We design a tailor-made center separation loss to further improve the discriminability of hash centers by constraining minimal separation distance derived from coding theory [14]. We also provide visual analyses to better understand the underlying mechanism of our PHE. Experiments on eight fine- -reinga datasets show that our method outperforms previous methods by a large margin, i.e., +5.3% improvement averaged on all datasets for all class accuracy.

---

2
Related Works
Novel Category Discovery (NCD), initially introduced by DTC [5]. aims to categorize unlabeled novel classes by transferring knowledge from labeled known classes. However, existing NCD methods [5, 6, 7] assume that all unlabeled data exclusively belong to novel classes. Generalized Category Discovery (GCD) is proposed in [8], allowing unlabeled data to be sampled from both novel and known classes. While existing NCD/GCD methods [6. 7, 8, 9, 13, 15, 16] have shown promising results, two key assumptions still impede their real-world application. Firstly, these models heavily rely on a predetermined query set (the unlabeled dataset) during training, limiting their ability to handle truly novel samples and hindering generalization. Secondly, the offline batch processing of the query set during inference makes these models impractical for online scenarios where data emerges continuously and the model requires instant feedback. To address these limitations, Du et al. introduced the On-the-Fly Category Discovery (OCD) [12]. which removes the assumption of a predefined query set and requires instance feedback with stream data input. They proposed the SMILE method, which identifies the category of each instance by the signature of its representation (a hash-form code). The comparison among different settings is shown in Table 1.
In this paper, we address the "high sensitivity" issue associated with hash-based category descriptors in SMILE, particularly in fine-grained scenarios. We encode category prototypes into hash centers, ensuring maximal separation between these centers. Additionally, we employ Hamming balls centered on these hash centers to represent each category, effectively mitigating the "high sensitivity" issue. Table I: Comparison between different category discovery settings. indicates that the number of new classes (CIs) set as the ground-trute or previously estimated.
Training Data
Test Data
Setting
Old Cls
New Cls
Require #Cls
Old
New Cls
Require # Cls
Online
NCD
Old+New*
X
Old+New"
X
GCD
Old+New*
Old+New X
OCD
X
Old
Old
Deep Hashing is popular method for large-scale image retrieval. It uses deep neural networks to learn a hash function that converts samples into fixed-length binary codes, ensuring similar samples share similar codes. Early methods, such HashNet [17], DPSH [18]. and DSH [19], optimize hash functions based on pairwise similarities or triplet-based similarities. Both approaches suffer from low training efficiency and insufficient data distribution coverage. By defining points as hash centers that are sufficiently spaced apart, methods like DPN [20], OrthoHash [21], and CSQ [22] largely enhance training efficiency and retrieval accuracy. However, in the worst case, hash centers derived from these methods can be arbitrarily close. To address this, Wang et al. [23] introduce the Gilbert- Varshamov bound from coding theory to ensure a large minimal distance between hash centers.
Drawing inspiration from deep hash methods, we employ hash center- approach for category encoding. Unlike deep hashing methods used in image retrieval tasks [22, 23]. which rely on predefined center points, our method generates category-specifie hash centers directly from category prototypes. This adaptation is particularly suitable for the category discovery task. Furthermore, we utilize Hamming balls centered on these hash centers to represent categories, effectively mitigating the "high sensitivity" issue associated with using hash-form descriptors in category discovery tasks.
Prototype-based Interpretable Models. Chen et al. [24] introduce the Prototypical Part Network (ProtoPNet), model designed for Interpretable classification. ProtoPNet features a set number of prototypical parts per class, enabling clear deeiiionnmmainn process. In addition, ProtoPNet offers a post-hoc analysis, in which it explains decisions for individual images by displaying all prototypes alongside their weighted similarity scores. This method employs multiple prototypes to represent a category, effectively distinguishing subtle differences between categories and demonstrating strong performance in fine- -en-ggrined classification. Numerous subsequent studies have adapted ProtoPNet for various applications such as medical image processing and explanatory debugging, among others [25, 26, 27, 28, 29]. Later, ProtoPFormer [30] further integrates the Vision Transformer as a backbone, utilizing both global and part prototypes for interpretable image classification.
In this paper, we utilize prototype-based models for representation learning and prototype acquisition. Unlike existing methods predominantly tailored for closed-set classification, our approach extends to the generation of discriminative hash codes from the learned category prototypes. This extension allows for broader applicability in handling new and unseen categories, i.e., open-set scenarios.
3

---

Linear Projector
P1
P2
Calegory
Prn
Hrf
h
h,p
Le
Backbome Network
Prototypes
Hamminng
Hash Centers
Activated Protodype
Sp
Shared
E(-)
Hy(--
CPG
DHE
dinse
Figure 2: Our PHE framework is composed of the CPG and DHC modules. First, CPG generates sateifry-specifie prototypes and prototype-guice instance representations. Then, DHC encodes the generated prototypes as hash centers to encourage the model to learn discriminative instance hash codes. Finally, depending on the Hamming distance between instance hash codes and hash centers, we can obtain instant feedback and online group instances into both known and unknown categories.
3 Prototypical Hash Encoding
Problem Setup. The setting of OCD is defined as follows. We are provided with a support set, denoted
as Ds {(x, yf M i= S X' x V's for training, and a query set, denoted as Dq (x,, y ) j= S X' x V' for testing. Here, M and N represent the number of samples in Dg and Dq. respectively. Vss and V' are the label spaces for the support set and query set, respectively, where V/s S Vo. We define classes in V's as known/old classes and classes in Vo/ / unknown/new classes. Only the support set Ds is used for model training. During testing. De includes samples from both known and unknown categories, which are inferred one by one, allowing for instant/online feedback.
Framework Overview. To achieve accurate and online category discovery. we design Prototypical Hash Encoding (PHE) framework, which mainly consists of Category-awaid Prototype Generation (CPG) module and a Discriminative Hash Encoding (DHE) module, illustrated in Fig. CPG aims at modeling diverse intra-category information and generating detegory-specinee prototypes for representing fine-grained categories. DHE leverages generated prototypical hash centers further facilitate discriminative hash code generation. Finally, based on a theoretically derived bounding of Hamming ball, we can determine the under-discovered category of instance and acquire instant feedback.
3.1 Category-aware Prototype Generation
SMILE [12] directly utilizes instance-level supervised contrastive learning on low-dimensional hash features for simultaneous representation learning and category encoding, This approach may result inadequate representation learning, as low-dimensional features struggle to capture complex data structures and patterss, especially in challenging fine- fine-graine grained scenarios (see Table 10).
To solve this issue, we choose perform representation learning upon the high-dimensional features instead of the low-dimensional hash features. Specifically, given batch of input images, X, the image features are denoted as Z = (x(((X)) E N where & represents the backbone, H. denotes linear head, and L represents the feature dimension. We use a prototype layer Ip to transform Z into similarity score vector R. with 9p containing m learnable prototypes (Pi, .2..... Pm. }. In the design of the prototype layer, CNN-hased methods [24] typically utilize the maximum pooled value from the similarity map, which is calculated between the feature map and the prototype, the similarity score. In this paper, to align with the Vision Transformer (ViT) [4] backbone commonly-use in OCD, we propose employ its cls token compute the prototype similarity score, inspired by ProtopFormer [30]. The similarity score Siy for the i-th sample to the j-th prototype is calculated as follows:
P,
1
+
Si+i = SD, (z.) log
(1)
P,
2
where is a small constant for numerical stability.
Probabilistic Masking Strategy. To encourage models to fully capture intra-category diversity, we assign prototypes equally to each category, thus m = k [>s] Given the prototype similarity
score to all prototypes in the prototype layer of i-th image s; = 1, S.mm]. fully
connected layer (FC) employed for supervised classification. To further discretize the prototypes,

---

avoid redundancy, and unleash the full potential of the prototypes, we utilize probabilistic masking strategy. This strategy masks some units of the similarity score according probability, thereby disabling the corresponding prototypes and leaving the remaining prototypes activated during training. Specifically, for each unit of similarity score S,, we mask it with a probability following the Bernoulli distribution B()). with 0 empirically set to 0.1. We use the following loss function to learn the image representations and prototypes:
Σ Ky, FC(((O) -s)),
(2)
[B] EEB
where B indicates the mini-batch of support set, l is the traditional cross-entropy loss and y, is the ground truth of image X₁.
Discussion. The benefits of our CPG module are threefold: i) Fine-Grained Category Distinction: Fine- Fine-grained categories often exhibit only subtle differences. Our CPG module allows for representing category with multiple prototypes, which can effectively capture and model both intra-class similarities and inter-class variances. ii) Category-specific Hash Center Generation: Based on the generated category prototypes of CPG, we further map them into low oow-dimensional hash features to serve as hash centers for each category. This helps in maintaining the distinctiveness of each category in the encoding space. iii) A New perspective for Classification Analysis: Beyond merely using hash encoding for category decisions, we can introduce a new prototype perspective for classification analysis. CPG enables visualizable reasoning process for classifying unseen categories, providing a more intuitive and interpretable method for understanding category distinctions.
3.2 Discriminative Hash Encoding
Category Encoding Learning. Given the set of learned category prototypes Po: for category q; in CPG, we map the mean vector of each category's 's prototypes Σ Pe./ k to its hash center, denoted as h = (4 Pc. /k)/k) R',, where H,, represents a linear head and L represent the feature dimension. The image feature 2, is mapped to a hash feature b;, where b, = H4 (z:). To ensure that image closely as possible share the same category descriptor, we
representations of the same category constrain each hash feature to be close to its corresponding category hash center and distant from other hash centers. We employ the following loss to optimize the hash features of the images:
C Σ Eyy (bm, h)).
(3)
where sim(b; bb, h) represents a pair-wise similarity vector consisting of the cosine similarities between the hash feature of image x; and all hash centers.
Hash Centers Optimization. Due to the subtle differences between fine-grained categories, different category hash centers may become closely similar even share identical hash codes. This hinders separability between fine-grained categories and leads to incorrect classification results. Therefore, aim to maximize the differences between hash centers for enhancing inter-class separation.
Given the hash center h, of category C, we denote its hash code h, where h, (h.) and sign(-) equals 1/-1 for positive/negative values. Since the sign function non-differentiable, we use a smoothed version of the sign function for back-propagation during training, defined as:
eaxT
GxT
sign'
(a)
(4)
where T is hyper-parameter that controls the smoothness of the sign function. Larger values of make the function more closely approximate the true sign function. In this paper, we set T ==3. 3. Consequently. h, sign" (h.). ). The difference between the i-th and j-th hash centers can be evaluated by the Hamming distance of their hash codes: llb, hy Hr L-h.-h dy Although we aim 2 to maximize the differences between hash centers, we cannor simply increase the Hamming distance between all hash centers, as this can lead model coveernneeee Since the encoding space is fixed, excessively distancing one hash code can inadvertentl bring it closer to another.
We design a center separation loss, Ceep. to ensure that the Hamming distance between any two hash centers is greater than equal to d, denoted as ll.. h2ll# d. The center separation loss is defined as:
Leep ΣΣ max(O, - |l,, - 1,11)),
(5)
5

---

Algorithm 1: Pscudocode for Inference of Our Method
Input:
Test data (query set) De contains both known and unknown categories, trained PHE model contains backbone E(), linear projector H-)-). protolype layer 9p(+) and linear projector Hx(()) dmaz-
Compute hash centers h for known categories;
// Hash Centers
Compute Hamming ball radius R = max( Ama= 1,
Build category list CH. add fi to Ch:
for each image x; D do
Compute hash code b. = sign Ha (H)(((())))
// Category Descriptor
for hash center hj E Cи do
Compute Harmning distance [lb; higl/:: if [|b, - h2ylı 1l S R then
Output pred ÿ, Cи inrrx((x))))
// Close to a known category
10
/ Different from existing categories */;
11
Add b. to C# Output pred SN [C#|]
// Create a new category
12
indicating that we only optimize the pairs of hash centers whose Hamming distance less than However, determining the appropriate value of dl is challenging. If d is too large, it can lead to model non-convergence; if it too small, can cause inadequate separation between hash centers. We choose maximum dmaw according to the Gilbert-Varshamer Varshamov bound [14] in coding theory, stated in Lemma 3.1.
Lemma 3.1 For binary symbols, there exists a set of hash codes of length L, (-1,1)², with a minimum Hammiing distance d, and number of hash codes Q that satisfies the following inequality:
2
(6)
Therefore, given a number of hash codes equal |)'sl, the maximum dmaw can be determined as:
Dis
2
dmae
Disl
26
E.mur
We apply this upper bound dyzaz seep Considering that we use smoothed version of the sign function, where computed values do not equate to precisely +1, impose quantization loss to constrain the values of hash codes to be close to +1:
Σ (1 h:).
(8)
The optimization loss L. for hash centers defined as a combination of the center separation loss Leep and the quantization loss Ly:
Le = Leep + Lag
(9)
3.3 Training and Inference
Model Training, During the model training process, the total loss is formulated as follows:
L = Ep L. 3 L,,
(10)
where a and 3 control the importance of center optimization and hash encoding, respectively.
Hamming Ball Based Model Inference. During on-the-fly testing, given input image 2, in the query set Do. use b, (H6 (Exx))) its category descriptor. Due to the introduction of the center separation loss, the Hamming distance between any two hash centers is not less than dmaz- We consider Hamming ball centered on the hash centers with radius of max( [m J, 1) to represent a category. Specifically, during inference, if the Hamming distance between b, and any existing hash center is less than equal to max() 2 1), we classify the image as belonging to the corresponding category that hash center. Otherwise, the image used to establish a new hash center and category. The pseudo-code for the inference is provided in Algorithm 1.

---

4 Experiment
4.1 Experiment Setup
Datasets. We have conducted experiments on eight fine-grained datasets, including CUB-200 [31], Stanford Cars [32], Oxford-IIIT Pet [33], Food-101 [34], and four super-categories from the more challenging dataset, iNaturalist [35], including Fungi, Arachnida, Animalia, and Mollusca. Following the setup in OCD [12], the categories of each dataset are split into subsets of seen and unseen categories. Specifically, 50% of the samples from the seen categories are used to form the labeled set Ds for training, while the remainder forms the unlabeled set Do for on-the-fly fly testing. Detailed information about the datasets used is provided in the Appendix. A.
Evaluation Metrics. We follow Du al. [12] and adopt clustering accuracy as an evaluation protocol,
formulated as ACC pool El I[y: C(v))), where y; represents the predicted labels and y: denotes the ground truth. The function C denotes the optimal permutation that aligns predicted cluster assignments with the actual class labels.
Implementation Details. For fair comparison, we follow OCD [12] and use the DINO DINO-pretrained ViT-B-16 [36] as the backbone. During training, only the final block of ViT-B-16 fine-tuned. In
our approach, the Projector (-) is a single linear layer with an output dimension set to L = 768. meaning the feature dimension and prototype dimension are equal to 768. Each category has A = 10 prototypes. The FC layer in Eq. 2 is non-trainable, which uses positive weights 1 for prototypes from the same category and negative weights -0.5 for prototypes from different categories. The Projector (-) consists of three linear layers with an output dimension set to L = 12, which produces 21 4096 binary category encodings. By default, we follow OCD to set this dimension for fair comparison. Additional experiments with varying L are reported in Sec. 4.4. The estimated values of dmaz in center separation loss Seep can be found in Appendix. A.2. The ratio a and B in the total loss are set to 0.1 and 3, respectively, for all datasets. More details and the pseudo-code for the training process can be found in the Appendix. A.2.
Compared Methods. Given that OCD is a relatively new task requiring instantaneous inference, traditional baselines from NCD and GCD are unsuitable for this setting. Consequently, we compared with the SMILE [12] along with three strong competitors in [12]: i) Sequential Leader Clustering (SLC) [37]] A classical clustering technique suitable for sequential data. ii) Ranking Statistics (RankStat) [38]: RankStat utilizes the top-3 indices of feature embeddings as category descriptors. iii) Winner-take-all (WTA) [39]: WTA employs indices of maximum values within feature groups as category descriptors. These three strong baselines are set following SMILE [12], and detailed implementation can be found in the Appendix. A.3.
4.2 Comparison with State of the Art
We conduct comparison experiments with the aforementioned competitors across eight datasets. The experimental results are reported Table 2. evident that the proposed PHE outperforms all tfte-offthhe-rt competitors across nearly all metrics. In particular, when compared with three strong baselines-SLe RankStat, and -TA-our method demonstrates significant improvements. Additionally, our method surpasses the top competitor, SMILE. by average of 5.4% in All classes accuracy on four common fine-grained datasets, and by an average of 5.1% in All classes accuracy on the four challenging datasets from the iNaturalist dataset. We achieve average improvement of 1 11.3% on Old classes across the eight datasets compared to SMILE, demonstrating the effectiveness of our method in alleviating the "high sensitivity" issue of hash- OCD methods. Importantly, our method consistently improves accuracy for unseen/new classes compared to SMILE. For instance, we achieve 4.1% 5% improvement on CUB-200 and a 4.4% improvement on the Oxford Pets dataset for new classes. This demonstrates that our method exhibits stronger generalization capabilities to new classes compared SMILE. This advantage becomes more pronounced as the dimensionality L of hash features increases, as discussed in Sec. 4.4.
4.3 Ablation Study
Components Ablation. We report an ablation analysis of the proposed components in our PHE on CUB dataset and Stanford Cars dataset, as shown in Table 3. "Without Cp" "refers to the removal of the
7

---

Table Comparison with the State of the Art methods. The best results are marked in bold, and the second best results are marked by underline.
CUB
Stanford Cars
Oxford Pets
Food101
Average
Method
All
Old
New
All
Old
New
All
Old
New
All
Old
New
All
Old
New
SLC [37]
31.3
48.5
22.7
24.0
45.8
13.6
35.5
41.3
33.1
20.9
48.6
6.8
27.9
46.1
19.1
RankStat [38]
27.6
46.2
18.3
18,6
36,9
9.7
33.2
42.3
28.4
22.3
50.7
7.8
25,4
44.0
16.1
WTA [39]
26.5
45.0
17.3
20.0
38.8
10.6
35.2
46.3
29.3
18.2
40.5
6.1
25.0
42.7
15.8
SMILE [12]
32.2
50.9
22.9
26.2
46.7
16.3
41.2
42.1
40.7
24.0
54.6
8.4
30.9
48.6
22.1
PHE (Ours)
36.4
55.8
27.0
31.3
61.9
16.8
48.3
53.8
45.4
29.1
64.7
11.1
36.3
59.1
25.1
Fungi
Arachnida
Animalia
Mollusca
Average
Method
All
Old
New
All
Old
New
All
Old
New
All
Old
New
All
Old
New
SLC [37]
27.7
60.0
13.4
25.4
44.6
11.4
32.4
61.9
19.3
31.1
59.8
15.0
29.2
56.6
14.8
RankStat [38]
23.8
50.5
12.0
26.6
51.0
10.0
31.4
54.9
21.6
29.3
55.2
15.5
27.8
52,9
14.8
WTA [39]
27.5
65.6
12.0
28.1
55.5
10.9
33.4
59.8
22.4
30.3
55.4
17.0
29.8
59.1
15.6
SMILE [12]
29.3
64.6
13.6
29.9
57.9
12.2
35.9
49.4
30.3
33.3
44.5
27.2
32.1
54.1
20.8
PHE (Ours)
31.4
67.9
15.2
37.0
75.7
12.6
40.3
55.7
31.8
39.9
65.0
26.5
37.2
66.1
21.5
Table 3: Ablation study on training components. The best results are marked in bold.
Table 4: Ablation study on training strategy. The best results are marked in bold.
CUB
SCars
CUB
SCars
Lp Le f
Methods
All
Old
New
All
Old
New
All
Old
New
All
Old
New
v
34.9
53.0
25.8
28.9
58.4
14.6
Fixed-h
35.4
54.7
25.8
30.0
61.5
14.8
v
32.0
43.4
26.4
24.1
40.2
16.3
Linear Cls
35.6
57.8
24.6
29.4
63.6
13.0
34.1
54.3
24.0
26.0
52.6
13.1
Supcon Cls
35.3
57.5
24.2
29.5
64.1
12.8
V v v 36.4
55.8
27.0
31.3
61.9
16.8
Ours
36.4
55.8
27.0
31.3
61.9
16.8
representation learning in CPG, where we use randomly initialized vectors which are further mapped to hash centers. This configuration results in an average reduction of 1. 1.95% in All classes accuracy across the two datasets. This variant of PHE shares the same architecture as SMILE, achieving clear decline on both datasets, demonstrating the effectiveness of our hash center-based category encoding method. It is noteworthy that removing L. causes a significant performance reduction, especially in seen categories- for instance, a 12.4% decrease in CUB. This decline due to the lack of Les which results in insufficient separation of hash centers, causing multiple old classes to share the same hash encoding and thereby being classified as the same. Additionally, removing L, also leads to performance degradation. This is because the hash encodings of features are not sufficiently close to the hash centers, resulting in inadequate learning hash features.
Strategy Ablation. In Table 4, we evaluate the training strategies of our method. "Fixed-h" refers to the use of handcrafted hash points that satisfy Eq. 7. Although this design maintains separation between hash centers, it may alter the relationships between categories learned in CPG, leading to sub-optimal outcomes. "Linear Cls." and "Supcon Cls." represent the use of simple linear classification and supervised contrastive learning classification methods for representation learning, respectively, where "Fixed-h" is also applied due to the absence of category prototypes. Although these variants perform well on seen categories, their generalization capabilities are inferior to our full prototype- method. This demonstrates the importance of integrating prototype learning to enhance generalization across unseen categories.
4.4 Evaluation
Evaluation on Hash Code Length. The hash code length L is crucial for category inference as it directly determines the size of the prediction space, which equals 2 A larger L value results in greater number of category encodings, making the "high sensitivity" issue more severe. We evaluate our method and SMILE with different hash code lengths Table 5. When small changes occur to (from 12 to 16). SMILE demonstrates stable results. However, with I, 32, SMILE experiences decrease in average all-classes accuracy by 5.1% on two datasets, and with 64, the decrease by 10.2% averaged on the two datasets. In contrast, method, which employs hash center- center-based

---

Table 5: Results with different hash code length L. The best results are marked in bold for each
CUB#200
Estimated
SCars# 196
Estimated
L
Methods
All
Old
New
#Class
All
Old
New
#Class
SMILE
31.9
21.5
27.5
52.5
15.4
896
16bit
Ours
37.6
57.4
27.6
318
31.8
65.4
15.6
709
SMILE
27.3
52.0
14.97
2146
21.9
46.8
9.9
2953
32bit
Ours
38.5
59.9
27.8
474
31.5
64.0
15.8
762
SMILE
22.6
45.3
11.2
2910
16.5
38.2
6.1
4788
64hit
Ours
38.1
60.1
27.2
493
32.1
66.9
15.3
917
Test Images
Training image where prototype comes from
Similarity Score
LeConte Sparrow
2.32
Conte
2.01
9.00
Sparrow
1.28
Seen
Homed Lark
1.78
Le Conte Sparrow
1.34
1.19
Homed Lark
1.18
3.54
Unseen
1.17
Figure 3: Case Study: Why is a Grasshopper Sparrow classified as a new category?
optimization and Hamming ball-based inference process, effectively mitigates the "high sensitivity" issue and maintains stable performance as L increases. Furthermore, when L is increased from 16 to 64, the estimated number of classes by SMILE increased by 1986 on the CUB dataset and 3892 on the Stanford Cars dataset, underscoring the impact of "high sensitivity" of hash-form category descriptors on accuracy. Conversely, our method exhibits remarkable stability.
images from the support whose latent representations is most similar to P, as the visualization for P,- During on-the-fly inference, the hash code functions as the category descriptor. Additionally, the learned prototypes allow us visually analyze why the model categorizes certain samples into known or unknown categories. Fig. 3 illustrates this reasoning process from the prototype perspective. On the left, an image labeled in green depicts Le Conte Sparrow, category recognized during training. Under it, red-labeled image represents Grasshopper Sparrow, new category. the center, we display the visualization of the prototypes, showing only five per class. The top five prototypes most similar to the green- lrbenel image are exclusively associated with the Conte Sparrow category, while those closest the reb-leeeled, unseen image span two different categories. The Grasshopper Sparrow exhibits significant body stripe similarities to the Le Conte Sparrow and shares head similarities with the Horned Lark. Upon calculating similarities with the top five activated prototypes, the similarity score for the green-labeled image to the Le Conte Sparrow category is 9.0, whereas the unseen Grasshopper Sparrow achieves similarity of 3.32 with the Le Conte Sparrow and 3.54 with the Horned Lark. This reasoning process is similar to how humans cognitively recognize new species. Given a new species, we use the entity
Visualization Analysis. We
characteristics of known categories to describe the unknown new category. Consequently, that exhibits high similarity to multiple known categories, rather than only one known category, is likely to be classified as previously unseen species.
ratios a and 3 in Eq. 10 illustrated Fig. We use All classes accuracy as the evaluation metric. and B control the relative importance of L. and L, during the training process, respectively. A lower value of a is found be preferable, as higher values can lead excessive changes in hash centers, thereby affecting training stability.
Hyper-parameters Analysis. 1)
The impact

---

a- CCB
8- B-CUB
Prototypes Numbers CUB
37
ACI (6)
All ACC (2))
36
36
96
35
34
34
34
0.2
0.4
0.8
1.0
2.0
3.0
10
a SCars
B B-SCars
Prototypes Numbers SCars
32.0
31.5
ACC)))
All ACC. (6)
31.5
31.0
31.0
31.0
All
30.5
30.5
30.5
30.0
50.0
30.0
0.2
0.4
0.8
2.0
3.0
10
Figure 4: Impact of hyype--araaeteet..
Regarding 3, the results generally stable; however, higher values show improved performance across both datasets. Consequently, we selected a = 0.1 and 3 3 for all datasets during training. 2) Additionally, examine the impact of the number of prototypes per class the CUB and Stanford Cars datasets, as shown in Fig. 4. When only one prototype per class used, performance is suboptimal, fails effectively represent the complexity of category. For instance, single bird species might exhibit different behaviors, such as flying or standing, that are not adequately captured by single prototype. More prototypes allow for better expression of the nuances within category, which is crucial in fine-grained classification. Therefore, we opt to utilize 10 prototypes per class across all datasets.
Hash Centers Analysis. We conduct an analysis of the distribution hash centers before and after training to further evaluate the impact of
the proposed hash center optimization loss,
Hash Centers Distribution CUB
analyze the Ham- ming distances between hash centers on
fe. Specifically,
1000
EpochO
800
dmax
Epoch200
the CUB dataset. As depicted in Fig. 5,
Count
prior to training, the hash centers, derived from randomly initialized prototypes, are distributed relatively uniformly. Notably, some centers exhibit Hamming distance of zero, indicating multiple centers sharing single hash code. After training, the Ham- ming distance between all hash centers is at least d-nas- This significant improvement demonstrates the effectiveness of L. in en- suring that multiple categories not share identical hash codes or reside excessively close to one another.
600
200
0
10
12
Harminng Distance
Figure Evolution of hash centers distribution during the training process.
5 Conclusion
In this paper, we introduce a Prototypical Hash Encoding (PHE) framework for fine-grained On-the- fly Category Discovery. Addressing the limitations of existing methods, which struggle with the high sensitivity of hash-form category descriptors and suboptimal feature representation, our approach incorporates prototype-based classification model. This model facilitates robust representation learning by developing multiple prototypes for each fine- ieeeggr category. We then map these category prototypes to corresponding hash centers, optimizing image hash features to align closely with these centers, thereby achieving intra-class compactness. Additionally, we enhance inter-class separation by maximizing the distance between hash centers, guided by the Gilbert- Varshamov bound. Experiments methods by large margin. Moreover, visualization study is provided to understand the underlying mechanism of our method.
eight fine-grained datasets demonstrate that our method outperforms previous
10

---

Acknowledgement. This work has been supported by the National Natural Science Foundation of China (62402157), the MUR PNRR project FAIR (PE00000013) funded by the NexIGenerationEU, the EU Horizon project ELJAS (No. 101 12023)))) the MUR PNRR project INEST-Intercooneceed Nord-Est Innovation Ecosystem (ECS00000043) funded by the NextGGenerationFU, and the EU Horizon project A14Trust (No. 101070190).
References
[1] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Communications of the ACM, 60(6):84-90, 2017.
[2] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recogmition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2016.
[3] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, 2017.
[4] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, 2021.
[5] Kai Han, Andrea Vedaldi, and Andrew Zisserman. Learning to discover novel visual categories via deep transfer clustering. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8401-8409, 2019.
[6] Zhun Zhong, Enrico Fini, Subhankar Roy, Zhiming Luo, Elisa Ricci, and Nicu Sebe. Neighbor- hood contrastive learning for novel class discovery. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10867-10875, 2021.
[7] Enrico Fini, Enver Sangineto, Stéphane Lathuilière, Zhun Zhong, Moin Nabi, and Elisa Ricci. unified objective for novel class discovery. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9284-99889 0292, 2021.
[8] Sagar Vaze, Kai Han, Andrea Vedaldi, and Andrew Zisserman. Generalized category discovery. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7492-7501, 2022.
[9] Nan Pu, Zhun Zhong, and Nicu Sebe. Dynamic conceptional contrastive learning for generalized category discovery. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
[10] Nan Pu, Wenjing Li, Xingyuan Ji. Yalan Qin, Nicu Sebe, and Zhun Zhong, Federated generalized category discovery. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 28741-28750, 2024.
[11] Haiyang Zheng, Nan Pu, Wenjing Li, Nicu Sebe, and Zhun Zhong. Textual knowledge matters: Cross-modality co-teaching for generalized visual class discovery. arXiv preprint arXiv:2403.07369. 2024.
[12] Ruoyi Du, Dongliang Chang, Kongming Liang, Timothy Hospedales, Yi-Zhe Song, and Zhanyu Ma. On-the-lly category discovery. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11691-111 11700, 2023.
[13] Xin Wen, Bingchen Zhao, and Xiaojuan Qi. Parametric classification for generalized category discovery: A baseline study. In Proceedings of the IEEE/CVE International Conference on Computer Vision, 2023.
[14] Rom Rubenovich Varshamov. Estimate of the number of signals error correcting codes. Docklady Akad. Nauk, SSSR, 117:739-741, 1957.

---

[15] Florent Chiaroni, Jose Dolz, Ziko Imtiaz Masud, Amar Mitiche, and Ismail Ben Ayed. Para- metric information maximization for generalized category discovery. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023.
[16] Wenbin An, Feng Tian, Qinghua Zheng, Wei Ding, DianYing, Wang, and Ping Chen. Generalized category discovery with decoupled prototypical network. In Proceedings of the AAAI Conference on Artificial Intelligence, 2023.
[17] Zhangjie Cao, Mingsheng Long, Jianmin Wang, and Philip S Yu. Hashnet: Deep learning to hash by continuation. In Proceedings of the IEEE international conference on computer vision, pages 5608-66111 1000-5667,2211,
[18) Wu-Jun Li, Sheng Wang, and Wang-Cheng Kang. Feature learning based deep supervised hashing with pairwise labels. In Subbarao Kambhampati, editor, Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence, IJCAI 2016, New York, NY, USA, 9-15 July 2016, pages 1711-1717. ЦCCA/AААI Press, 2016.
[19] Haomiao Liu, Ruiping Wang, Shiguang Shan, and Xilin Chen. Deep supervised hashing for fast image retrieval. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2064-2072, 2016.
[20] Lixin Fan, Kam Woh Ng, Ce Ju, Tianyu Zhang, and Chee Seng Chan. Deep polarized network for supervised learning of accurate binary hashing codes. In IJCAI, pages 825-831, 2020.
[21] Jiun Tian Hoe, Kam Woh Ng, Tianyu Zhang. Chee Seng Chan, Yi-Zhe Song, and Tao Xiang. One loss for all: Deep hashing with a single cosine similarity based learning objective. Advances in Neural Information Processing Systems, 34:24286-244 24298, 2021.
[22] Li Yuan, Tao Wang, Xiaopeng Zhang, Francis EH Tay, Zequn Jie, Wei Liu, and Jiashi Feng. Central similarity quantization for efficient image and video retrieval. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3083-3092, 2020.
[23] Liangdao Wang, Yan Pan, Cong Liu, Hanjiang Lai, Jian Yin, and Ye Liu. Deep hashing with minimaal-distace--spaaaaa hash centers. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 23455-23464, 2023.
[24] Chaofan Chen, Oscar Li, Daniel Tao, Alina Barnett, Cynthia Rudin, and Jonathan K Su. This looks like that: deep learning for interpretable image recognition. Advances in neural information processing systems, 32, 2019.
[25] Andrea Bontempelli, Stefano Teso, Katya Tentori, Fausto Giunchiglia, and Andrea Passerini. Concept-level debugging of part-prototype networks. arXiv preprint arXiv:2205. 2022.
[26] Jon Donnelly, Alina Jade Barnett, and Chaofan Chen. Deformable protopnet: An interpretable image classifier using deformable prototypes. Proceedings of the IEEE/CVE conference on computer vision and putlern recognition, pages 10265-10275, 2022.
[27] Monish Keswani, Sriranjani Ramakrishnan, Nishant Reddy, and Vineeth Balasubramanian. Proto2proto: Can you recognize the car, the way i do? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10233-10243, 2022.
[28] Meike Nauta, Ron Van Bree, and Christin Seifert. Neural prototype trees for interpretable fine-grained image recognition. In Proceedings of the IEEE/CVE conference on computer vision and patlern recognition, pages 19933-14943. 2021.
[29] Jiaqi Wang, Huafeng Liu, Xinyue Wang, and Liping Jing. Interpretable image recognition by constructing transparent embedding space. In Proceedings of the IEEF/CVF international conference on computer vision, pages 895-904, 2021.
[30] Mengqi Xue, Qihan Huang. Haofei Zhang, Lechao Cheng, Jie Song, Minghui Wu, and Mingli Song. Protopformer: Concentrating on prototypical parts in vision transformers for interpretable image recognition. arXiv preprint arXiv:2208.104: 10431, 2022.
12

---

[31] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uesd birds-200-2011 dataset. Computation & Neural Systems Technical Report, 2011.
[32] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine- grained categorization. In Proceedings of the IEEE/CVF International Conference Computer Vision Workshops, 2013.
[33] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2012.
[34] Lukas Bossard. Matthieu Guillaumin, and Luc Van Gool. Food-101-mining discriminative components with random forests. In European Conference on Computer Vision, 2014.
[35] Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 8769-8778, 2018.
[36] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in sele-supe vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021.
[37] John A Hartigan. Clustering algorithms. John Wiley & Sons, Inc., 1975.
[38] Kai Han, Sylvestre-Alvise Rebuffi, Sebastien Ehrhardt, Andrea Vedaldi, and Andrew Zisserman. Autonovel: Automatically discovering and learning novel visual categories. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(10):6767-6788 2021.
[39] Xuhui Jia, Kai Han, Yukun Zhu, and Bradley Green. Joint representation learning and novel cat- egory discovery on single-and multi-modal data. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 610-619, 2021.
13

---

Appendix
Implementation Details
14
A.1 Datasets Details and Evaluation Metric Details
14
A.2 Training Details
15
A.3 Compared Methods Details
15
A.4 Pseudo-code
15
B Additional Experiment Results and Analysis
16
B.1 Error Bars for Main Results
16
B.2 Inference on Different Input Sequences
16
B.3 Impact of Feature Dimension on SMILE Performance
17
B.4
Comparison with Deep Hash Methods
17
B.5
Computational Cost Analysis
17
B.6
Training Efficiency Analysis
18
B.7 Results of Different Dataset Splits
18
B.8 Compared to Prototype Learning Method
18
C Broader Impact and Limitations Discussion
19
Additional Visualization Analysis
19
A Implementation Details
A.1 Datasets Details and Evaluation Metric Details
Dataset Details. As outlined Table 6, our method evaluated across multiple benchmarks. We have introduced the iNaturalist 2017 [35] dataset to the On-the-Fly Category Discovery (OCD) task, demonstrating our method's effectiveness in handling challenging fine-grained datasets. The iNaturalist 2017 dataset, sourced from the citizen science website iNaturalist, consists of 675,170 training and validation images across 5,089 natural fine-grained categories, which include Plantae (plants), Insecta (insects), Aves (birds), and Mammalia (mammals), among others, spread across 13 cater-categories These super-categoria challenging dataset for fine-gr ame-grained classification. For evaluation, selected Fungi, Arachnida, Animalia, and Mollusca as the four super- super-categories. Following the protocol established in OCD [12], the categories within each dataset 50% the samples from the seen categories are used to form the labeled training set Ds, while the remainder forms the unlabeled Do for on-the-fly testing.
exhibit significant ontr-category variations, making each a
divided into subsets of seen and unseen categories. Specifically,
Table Statistics of datasets used in our experiments.
CUB
Scars
Pets
Food
Fungi
Arachnida
Animalia
Mollusca
Ysl
100
196
101
121
77
93
YQ
200
98
19
51
61
28
39
47
DDsl
1.5K
2.0K
0.9K
19.1K
1.8K
1.7K
1.5K
2.4K
DQ
4.5K
6.1K
2.7K
56.6K
5.8K
4.3K
5.1K
7.0K
Evaluation Metric Details. We employ clustering accuracy, specifically the Strict-Hungarian method as outlined OCD [12], as our evaluation protocol, common choice NCD/GCD tasks. Clusters ranked by size, and only the top-|Yol clusters retained; others are deemed misclassified. It
14

---

is crucial to outline our accuracy calculation for old and new classes. We perform the Hungarian assignment once for all categories Vig. and subsequently measure classification accuracy for the "Old" and "New" subsets. This setup follows the protocols in OCD [12] and GCD [12].
A.2
Training Details
Model Training Details. We employ the AdamW optimizer during training, using learning of le-4 for the backbone and le-3 for the projection head and prototype layer, with a weight decay of 0.05. We train the model for 200 epochs. The batch size is uniformly set at 128 across all datasets for consistent comparisons with leading methods. Training incorporates the use of Exponential Moving Average (EMA). Experiments were conducted using Tesla V100 and 1080Ti GPUs, with results reported as the mean of three runs.
Calculated dmaax. The value of dmux is calculated using Eq. 7 for all datasets and hash code lengths L. The specific values of dmux are reported as follows:
Table 7: Values of dmax for different hash code lengths L and datasets.
L
CUB
Scars
Pets
Food
Fungi
Arachnida
Animalia
Mollusca
12bit
3
4
3
16bit
4
5
5
4
5
32bit
10
10
12
11
11
11
11
64bit
24
24
27
25
25
26
25
25
A.3
Compared Methods Details
The setup for the comparative experiments follows OCD [12]. Detailed information is provided follows: Ranking Statisties (RankStat). [38] AutoNovel utilizes Ranking Statistics analyze sample relationships, particularly by using the top-3 indices of feature embeddings as category descriptors. This method aligns with the On-the-Fly Category Discovery (OCD) settings and poses strong challenge to hash-based descriptors. For a fair comparison, we use the same backbone (DINO-ViT-E ViT-B-16) for Ranking Statistics (RS) and retain only the fully-supervised learning stages, as no additional data can be used in the On-the-Fly Category Discovery (OCD) task. The embedding dimension set to 32, resulting a prediction space of C = 4, 906, comparable our method and SMILE, which uses a hash code length of L. = 12 and achieves prediction space of 212 4, 096. Winner-take-all (WTA). [39] To address potential biases towards salient features by Ranking alternative. WTA avoids reliance
Statistics, Winner-take-all (WTA) hash was proposed the global order feature embeddings and instead utilizes indices maximum values within divided feature groups. With 48-dimension embedding divided into three groups, WTA generates a prediction space of 16 4096, ensuring comparability for fair assessment. Sequential Leader Clustering (SLC). [37] We employ the same backbone and conducts fully supervised training the support set for SLC. For on-th on-the-fly fly testing. SLC utilizes features extracted from the backbone on the query set. We optimized the hyperparameters based the CUB dataset, applying these uniformly across other datasets to maintain consistency in our comparisons.
A.4 Pseudo-code
We have detailed the description of our PHE framework and the Hamming Ball Based Model Inference in the main text. The pseudo-code for the Hamming Ball Based Model Inference is provided in Algorithm 1, and the training process of our PHE framework is shown in Algorithm 2.
15

---

Algorithm 2: Pscudocode for PHE
Input: Training data (support set) Ds contains labeled data, image encoder E+). linear projection head H-(-), prototype layer (p(-), frozen layer FC and a linear projection head HA (+), hyper parameters (training epochs E1, prototypes per class k. et al.).
1M Model Training */
2 for each epoch 1... Ej do
for each batch (x,, v;) E B do
CPG module */;
(E(x(),))
// image feature // similarity score
S; = (p.).)
Compute Ep by Eq. (2) on and FC;
Trained prototypes Pe. for class Cj
// prototypes
DHC module */;
10
h, Ha(2 Po,k))
// hash centers // image hash feature
11
b, = (2;));
12
Compute L, by Eq. (3) on b, and all hash centers h;
13
Compute dmaa by Eq. (7);
// hash centers optimization upper bond Compute Ceep by Eq. (5) on dmaz and all hash centers h;
14
15
Compute Cg by Eq. (8) on all hash centers h;
16
Compute L. by Eq. (9) on Leep and C.
// hash center optimization loss
Compute the overall loss L by Eq. (10); Brok-proon and optimize 3, My, 9p. Hai
18
19
return Trained model PHE(.):
B Additional Experiment Results and Analysis
B.1 Error Bars for Main Results
In the main paper, we present the full results as the average of three runs to mitigate the impact of randomness. Detailed outcomes for our PHE, encompassing mean values and population standard deviation, are delineated in Table 8.
Table 8: Mean and std of accuracy in three independent runs.
Dataset
All
Old
New
CUB [31]
36.4=0.17
1.58
27.0
Stanford Cars [32]
31.3=0.49
61.9:1129
16.8:10.41
Oxford Pets [33]
48.3=0.96
53.8+:3224
45.4
Food101 [34]
29.1=0.25
64.7+0045
11.1=0.62
Fungi [35]
31.4=0.39
67.911.76
15.240.45
Arachnida [35]
37.0=0.34
75.7+-0.3
12.6++0.22
Animalia [35]
40.3=0.40
55.7+:2.16
31.8+
Mollusca [35]
39.9=0.66
65.0+22.22
26.5+
B.2 Inference on Different Input Sequences
We evaluated the accuracy of our PHE method CUB and Stanford Cars under different input sequences, as shown in Table 9. In the table, "fixed Sequences" indicates that the test data order shuffled (all results and comparisons in the main paper are tested this way), while "Random Sequences" indicates results obtained by randomly shuffling the test data order, with the results averaged over 10 runs. As can seen from table, our results are very stable and not affected by the input sequence order. This stability is mainly due to our hash center optimization, which ensures significant inter-class separation, and the use of Hamming balls based on hash centers for category prediction, which is very robust to different input sequences.
16

---

Table 9: Results with inference on different input sequences. The best results are marked in bold.
CUB
SCars
Methods
All
Old
New
All
Old
New
Fixed Sequences
36.4
55.8
27.0
31.3
61.9
16.8
Random Sequences
36.5
55,9
27.0
31.4
62.1
16.7
B.3 Impact of Feature Dimension on SMILE Performance
We analyze the performance of the SMILE method [12] with different feature dimensions in Table 10. In the table, "Offline" " and "Clustering Ace" refer to the clustering accuracy obtained by applying k-means clustering on the features of all test data, given a fixed number of categories (200 for CUB). From the table, it is evident that as the feature dimension decreases, the on-the-fly prediction accuracy significantly improves. This is because the "high sensitivity" issue of hash-form category descriptors becomes more pronounced at higher feature dimensions. On the other hand, as the feature dimension decreases, there is notable reduction in offline clustering accuracy, with decrease of 9.9% when reducing L from 256 to 12. This indicates that when the feature dimension is too low, it inevitably damages the model's ability to distinguish categories, resulting in poorer feature representation learning. Therefore, our PHE framework separates feature learning from category representation learning, which is one of the reasons why our PHE framework performs better.
Table 10: Online vs. offline predictions with different hash code lengths L in SMILE on CUB,
On-the-fly
Offline
L
All
Old
New
Clustering Acc
12bit
32.2
50.9
22.9
38.7
64bit
22.6
45.3
11.2
45.2
128bit
17.3
38.2
6.8
47.4
256bit
13.2
28.4
5.4
48.6
B.4 Comparison with Deep Hash Methods
We conduct comparative experiments using various deep hashing methods based our prototype learning model across the CUB and Stanford Cars datasets. The results, as detailed in Table 11, demonstrate the effectiveness of our PHE. Traditional methods like DPN, OrthoHash, and CSQ generate suboptimal hash centers, primarily designed for retrieval tasks with a focus on instance- level discrimination. Unlike these methods, our PHE incorporates the Varshhame bound to guide the learning of hash centers, specifically for category discovery. This strategy ensures that the hash centers are not only discriminable but also preserve the rich aategory-specifer information inherent in category-level prototypes. Furthermore, our PHE outperforms MDSH, which optimizes hash codes align with pre-defined static hash centers. In contrast, our approach dynamically maps hash centers from each category's 's prototypes, continuously refining these through end-to-end optimization. Additionally, our design incorporates
inference mechanism, which significantly reduces hash sensitivity. For instance, when the hash code length L is set to 32, the Hamming ball configuration outperforms the standard MDSH by an average margin of 5.9% across the two datasets.
Haaming--all--aan
B.5
Computational Cost Analysis
The computational cost of PHE is relatively low and primarily depends on the number of known categories, rather than directly correlating with the scale of the dataset. Specifically, each known category associated with a hash center, with hash centers h of shape [class nums, code length]. The main computational operations involve straightforward two-dimensional matrix multiplications. These include the dot product of features b, with a shape [batch size, code length], and hash centers, well as the dot product operations used in calculating Hamming distances between hash centers.
17

---

Table 11: Comparative results of various hashing methods on CUB and Stanford Cars datasets using 12-bi and 32-bit hash code lengths. "MC" denotes manually obtained centers compliant with the Gilbert-
Varshamov Gilbert-Varshamov
bound.
CUB 12bit
CUB 32bit
SCars 12bit
SCars 32bit
Methods
All
Old
New
All
Old
New
All
Old
New
All
Old
New
DPN [20]
22.2 38.0
14.2
12.5
11.1
13.2
18.8
36.1
10.5
12.8
20.4
9.1
OrthoHash [21]
30.0 49.2
20.5
13.6
24.8
8.0
19.6
37.2
11.1
13.2
20.0
9.8
CSQ [22]
26.1
45.3
16.5
23.1
44.1
13.0
CSQ-MC
26.4
50.6
14.3
26.9
61.8
10.0
MDSH [23]
34.3
57.6
22.8
27.4
40.8
20.7
28.8
60.2
13.7
25.8
47.8
15.2
MDSH+HHamming Ball
35.1
55.0
25.3
35.5
47.8
29.3
29.8
56.1
17.1
29.5
56.2
16.6
Ours
36.4
55.8
27.0
38.5
59.9
27.8
31.3
61.9
16.8
31.5
64.0
15.8
As detailed in Table 12, although the Food dataset is larger than both CUB and Stanford Cars, it includes only 100 categories. Consequently, the average training time per sample for the Food dataset is significantly lower than that observed for the CUB and Stanford Cars datasets.
Table 12: Comparison of training time per sample across CUB, Stanford Cars and Food datasets.
Dataset
CUB#200
SCars#198
Food#100
Number of training samples
1.5k
2.0k
19.1k
Training time minute
100.22
161.37
691.39
Training time per sample second
4.01
4.84
2.17
B.6 Training Efficiency Analysis
We provide a comparison of training times between our PHE and the state-of-the-ar method, SMILE, with results shown in Table 13. To ensure fairness, all experiments are conducted on an NVIDIA RTX A6000 GPU, with both algorithms trained over 200 epochs using mixed precision. The dataloader parameters are kept consistent across tests, with a batch size of 128. According to the table, our average training time across four datasets is 45.8 minutes shorter than that of SMILE. This improvement primarily due to the higher computational demands of SMILE's supervised contrastive learning approach, which processes two different views of samples for representation learning.
Table 13: Comparison of training times (in minutes)
Method
CUB
SCars
Food
Pets
SMILE 127.70
177.54
819.93
80.37
PHE (ours)
100.22
161.37
691.39
69.48
B.7 Results of Different Dataset Splits
We conduct experiments using different proportions of old category selection on the CUB and Stanford Cars datasets, as shown in Table 14, with all accuracy results reported and hash code length L set to 12. Our PHE outperforms SMILE by an average of 5.95% when 75% of old categories are selected and by 1.0% when 25% selected. This indicates that our PHE better models the nuanced inter-category relationships in fine-grained category divisions as the number of categories increases.
B.8 Compared to Prototype Learning Method
We conduct comparative experiments using the powerful prototype learning method SimGCD [13], based on our category encoding method. Due to the lack of unlabeled data, we remove the unsu- pervised loss component, T.a.- and the prototypes corresponding to new categories in SimGCD. As shown in Table 15, our of SimGCD for prototype learning, mapping the prototypes learned by
18

---

Table 14: Comparison with different known category split percentages.
Method SMILE
CUB-255
CUB-50%
CUB-75%
SCars-25%
SCars-50%
SCars-75%
19.9
32.2
41.2
12.6
26.2
37.0
PHE (ours)
21.2
36.4
46.5
13.3
31.3
43.6
SimGCD to hash centers for category encoding, yields very poor results on both datasets. "SimGCD- MC", which employs manually obtained centers satisfying the Gilbert- Varshamov bound along with features from the SimGCD projection head, shows an average improvement of 8.8% across the two datasets. This performance boost demonstrates that the prototypes learned by SimGCD are not suitable for category encoding in fine-grained scenarios. Our PHE offers two main advantages. First, unlike SimGCD, which learns only one prototype per category, our PHE generates multiple prototypes per class, effectively modeling intra-class variance of fine-grained categories. Second, unlike the prototypes in SimGCD, which serve as classifier weights, the prototypes in PHE can be explicitly visualized, providing additional insights into the model's behavior.
Table 15: Comparative results of different representation learning methods on CUB and Scars datasets.
"MC" denotes manually obtained centers compliant with the Gilbert- Varshamov bound.
CUB
SCars
Method
All
Old
New
All
Old
New
SimGCD [13]
25.0
49.9
12.6
21.9
38.5
13.9
SimGCD-MC
34.1
60.6
20.8
30.3
65.9
13.0
PHE (ours)
36.4
55.8
27.0
31.3
61.9
16.8
C Broader Impact and Limitations Discussion
Broader Impact. The On-the-fly Category Discovery (OCD) task designed to learn category differences from observed categories and then make real-time predictions across a broader range, including unknown categories. Our proposed Prototypical Hash Encoding (PHE) method for the OCD task holds potential for application in open-world scenarios. For instance, it can assist in botanical classification, where new plant species are discovered and need to quickly integrated into existing categories without retraining the entire system.
Limitations. Despite the superior performance of our Prototypical Hash Encoding (PHE) framework compared to existing methods, it still requires further research to enhance the accuracy of recognizing unknown categories. Current On-the-fly Category Discovery (OCD) methods generally achieve low accuracy in predicting new categories, as these categories were not encountered during training, posing significant challenges effective generalization. Future research should focus on improving the model's ability to recognize and categorize new, unseen categories.
Possible Solution in Future Work. Due the constraints of training data in the OCD setting, we are considering the integration of additional knowledge from pre-trained Large Language Models utilize LLMs establish bank of category attribute prototypes, which
(LLMs). First, are expected to be relevant across both known and unknown categories. Subsequently. during the real-time prediction process, we plan to employ LLMs and Vision-Language Pretrained Models (VLMs) to match these attribute prototypes with unknown categories. Ultimately, by integrating both instance and attribute features, aim for our PHE generate more precise predictions.
D Additional Visualization Analysis
We have conducted additional visualization analysis with images from the CUB dataset and Stanford Cars dataset, as shown in Fig. 6 to 11. As discussed in Sec. 4.4, in addition to using hash codes to represent categroies, our PHE framework allows for visual analysis from the perspective of prototype similarity to understand why certain images are identified as new classes, as depicted in Fig. 6, 7, and 10. Furthermore, even when images are misclassified, such assigning category to an old category as shown Fig. 8 and Fig. 11, we can still gain interesting insights from the prototype perspective. Firstly, these images from categories indeed exhibit high similarity
image from new
19

---

to the category where the activated prototypes belong to. Secondly, although the images from new categories show high similarity old ones, the computed similarity relatively lower compared to samples that truly belong to that old category. For example, in Fig. 8, the similarity values are 5.48 versus for the misclassified new category and the true sample of the old category, respectively.
Test Images
Training image where prototype comes from
Similarity Score
Warbling Vireo
1.32
Warbling Vireo
1.26
6.25
1.21
Seen
Orange Crraveed Warble
1.20
W Warbling Vireo
2.39
1.19
1.24
Orange Crowned Warble
1,21
3.58
Unseen
1.13
Figure 6: Case Study of the Cardinal and the Warbling Vireo.
Test Images
Training image where prototype comes from
Similarity Score
Rose Breasted Grosbeak
1.96
Rose Bressted Grobeak
1.87
9.34
Seen
Tree Spartow
1.17
Rose Breattdd Grosbeak
1.16
3.49
1.16
1.19
Tree Sparrow
2.38
Unseen
1.19
Figure Case Study of the Chestnut sided Warbler and the Rose breasted Grosbeak.
Test Images
Training image where prototype comes from
Similarity Score
1.87
1.84
9.14
Cape May
W Warbler
Cape May Warbler
1.73
Seen
1.19
1.16
5.48
Cape May Warbler
Unseen
Figure 8: Case Study of the Yellow Warbler and the Cape May Warbler.
20

---

Test Images
Training image where prototype comes from
Similarity Score
Marda Tribute SUV 2011
1.87
Mazda Tribute SUV 2011
1.87
8.44
1.65
Seen
Silveado Hytarid Crew Cab
1.23
Muzda 2011
Eries
2.46
1.23
1.28
Chevrolet
Silverudo
1.24
3.76
1500 Hybrid Crew Cab 2012
1.24
Figure 9: Case Study of the Dodge Dakota Club Cab 2007 and the Mazda Tribute SUV 2011.
Test Images
Training image where prototype comes from
Similarity Score
Porsche Panamera Sedan 2012
1.79
Porsche Panamera Sedan 2012
1.75
+
6.92
Seen
BMW Activellyrrid Sedan 2012
1.16
Porsche Panamera Sedan 2012
2.46
1.16
1.17
BMW ctivellybrid Sedan 2012
1.16
3.76
Unseen
1.16
Figure 10: Case Study of the Buick Verano Sedan 2012 and the Porsche Panamera Sedan 2012
Test Images
Training image where prototype comes from
Similarity Score
Volvo C30 Hatzhback
1.72
8.54
Volvo Hatchback 2012
...
1.89
Seen
Volvo C30 Hatchhack 2012
1.47
7.32
...
1.39
Inseen
Figure 11: Case Study of the Dodge Challenger SRT8 2011 and the Volvo C30 Hatchback 2012.
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The contributions designed for on-the-fly novel class discovery are clearly claimed in both abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Please refer Appendix. C.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
22

---

Justification:
Please refer to Sec. 3.2.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Please refer Sec. 4.1 and Appendix. A.2.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
23

---

Answer: [No]
Justification: Experiments were conducted on open datasets; please refer to Sec. 4.1 for details. Pseudo-code is provided in the Appendix. A.4. We will release the source code upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (uttps://nips.cm cc/
SaaideesCooeeuumiise
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results, See the NeurIPS code and data submission guidelines (https:
/e/ggudeee deuu/de///e Codeeubmiis aasion caisooooiccyy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Please refer to Sec. 4.1 and Appendix. A.2.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Please refer Appendix. B.I.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please refer to Appendix. A.2.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https :/te:////eeeee http///eerrpp.e.eee
Answer: [Yes]
Justification: Our research adheres to all aspects of the NeurlPS Code of Ethics, ensuring compliance with ethical standards throughout the study.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Please refer to Appendix.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
25

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e-g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI...
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g.. pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The scope of research does not involve data models with high risk for misuse. We utilize publicly available, non-sensitive datasets and models that do not require special safeguards for responsible release.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All third-party assets used our research, including datasets, and models, are properly credited in the manuscript. We have explicitly mentioned the licenses and terms of use for each asset, ensuring that all conditions are fully respected.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswi wwwode.hcoe...c. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
26

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This question does not apply our research no new assets were introduced. Our study solely relies on existing, publicly available resources, and thus, no additional documentation for new assets is necessary.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/codeemmodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This question does not apply to our research as it does not involve crowdsoure- ing experiments research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethies, workers involved in data collection, curation,
or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This question is not applicable to our research as no human subjects were involved.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
27

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
28


Reviews:
[{"review_id": "ojSu2WjYlr", "reviewer": "Reviewer_6xA4", "summary": "This paper focuses on On-the-fly Category Discovery (OCD), which is to determine novel categories during inference. OCD methods first compute the hash code of the image, this code then becomes a \"cluster index\", if not matching with existing code, then it is a novel category.\nHowever, the problem with the previous OCD method (SMILE) is that it becomes highly sensitive when the code length increases (difficult to match a new category due to the high possible combinations of hash code). \nInstead of contrastive learning in SMILE, this paper uses cross-entropy loss adapted with a minimal hash distance scheme as regularization to improve performance. Further, the paper also adds cross entropy loss in the feature space to avoid information loss when compressing features into hash code, the feature representation used is a prototype-based model (e.g., ProtoPFormer). \nOverall, it enhances SMILE by using better loss functions and different representations.", "questions": "In terms of the idea, I have no question, I believe the idea is simple and clever use of existing components. However, the lack of comparison to deep hashing methods makes me doubt the novelty and effectiveness of the proposed method/modules.", "limitations": "The author did mention limitations in the appendix which I also agree with.", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "originality: the idea itself is clever and simple, propose/reuse some modules to mitigate the problem in SMILE (to reduce the sensitivity problem). the hamming ball-based inference utilizes the benefit of GV bound.\n\nquality: analyzed the proposed method with many experiments.\n\nclarity: the paper is well-written\n\nsignificance: somehow significant as OCD is more practical in open-world scenario, and may help improve retrieval systems.\n\nthe one insight I got from this paper is that using hash code alone is sensitive when code length increases so it cannot be used for cluster index easily, and we need to make the hash code as compact as possible to reduce sensitivity (i.e., the same class must have the same hash code, by guiding the hash code into center, although already shown in multiple deep hashing papers such as CSQ [1], DPN [2], OrthoHash [3], [4]).\n\n[1] Li Yuan et al. Central Similarity Quantization for Efficient Image and Video Retrieval. CVPR 2020.\n[2] Lixin Fan et al. Deep Polarized Network for Supervised Learning of Accurate Binary Hashing Codes. IJCAI 2020.\n[3] Jiun Tian Hoe et al. One Loss for All: Deep Hashing with a Single Cosine Similarity based Learning Objective. NeurIPS 2021\n[4] Liangdao Wang et al. Deep Hashing with Minimal-Distance-Separated Hash Centers. CVPR 2023.", "weaknesses": "Comparison to deep hashing based methods are missing. There are many parts of the designed components similar to deep hashing such as the minimal hash distance formulation is adapted directly from [1]. Part of the loss objective is similar to cosine-similarity-based hashing methods like [2]. Another simple baseline is using ProtoPFormer + Deep Hashing method. From this paper, we do not know whether a naive application of deep hashing-based method is sufficient hence we cannot verify the novelty and effectiveness of the proposed method.\n\n[1] Liangdao Wang et al. Deep Hashing with Minimal-Distance-Separated Hash Centers. CVPR 2023. \n[2] Hoe et al. One Loss for All: Deep Hashing with a Single Cosine Similarity based Learning Objective. NeurIPS 2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for acknowledging that our idea is clever and simple, and for the positive comments on the writing and experiments. \n\n**Q1: More baseline with deep hash methods**. \n\nFollowing the meaningful suggestion, we have conducted additional comparative experiments with various deep hashing methods on CUB and SCars datasets. The results are reported and summarized as follows.\n\n|                   |          | CUB |-12bit|          | CUB|-16bit|          | CUB|-32bit|          | SCars|-12bit|          | SCars|-16bit|          | SCars|-32bit|\n| ----------------- | -------- | --------- | -------- | -------- | --------- | -------- | -------- | --------- | -------- | -------- | ----------- | -------- | -------- | ----------- | -------- | -------- | ----------- | -------- |\n| Methods           | All      | Old       | New      | All      | Old       | New      | All      | Old       | New      | All      | Old         | New      | All      | Old         | New      | All      | Old         | New      |\n| DPN[1]            | 22.2     | 38.0      | 14.2     | 17.6     | 24.1      | 14.4     | 12.5     | 11.1      | 13.2     | 18.8     | 36.1        | 10.5     | 14.9     | 25.3        | 9.8      | 12.8     | 20.4        | 9.1      |\n| OrthoHash[2]      | 30.0     | 49.2      | 20.5     | 25.2     | 42.1      | 16.7     | 13.6     | 24.8      | 8.0      | 19.6     | 37.2        | 11.1     | 15.0     | 24.5        | 10.4     | 13.2     | 20.0        | 9.8      |\n| CSQ[3]            | -        | -         | -        | 25.2     | 35.3      | 20.1     | 26.1     | 45.3      | 16.5     | -        | -           | -        | 25.4     | 54.8        | 11.2     | 23.1     | 44.1        | 13.0     |\n| CSQ-MC  | -  | -| -  | 29.3  | 56.4  | 15.7 | 26.4 | 50.6  | 14.3 | -   | -  | -  | 27.4 | 59.9        | 11.8     | 26.9     | 61.8        | 10.0     |\n| MDSH[4]| 34.3 | **57.6**  | 22.8| 31.9| 52.7| 21.5 | 27.4| 40.8 | 20.7| 28.8 | 60.2  | 13.7 | 27.7  | 53.4 | 15.3| 25.8| 47.8| 15.2 |\n| MDSH+Hamming Ball| 35.1 | 55.0 | 25.3 | 35.2 | 53.0 | 26.2 | 35.5 | 47.8 | **29.3** | 29.8 | 56.1| **17.1** | 29.9 | 57.2 | **16.5** | 29.5     | 56.2 | **16.6** |\n| PHE (ours) | **36.4** | 55.8 | **27.0** | **37.6** | **57.4**  | **27.6** | **38.5** | **59.9**  | 27.8 | **31.3** | **61.9** | 16.8 | **31.8** | **65.4** | 15.6| **31.5** | **64.0** | 15.8|\n\n*\"MC\" represents manually obtained centers that meet the GV bound. \"-\" means results were unavailable due to the requirements of the Hadamard matrix.\n\n**Summarization of Experimental Results.**\n\n1. **Comparison with methods[1-3] that generate suboptimal hash centers for modeling category information.** Unlike these methods[1-3], which are designed for retrieval tasks and mainly focus on instance-level discrimination, our PHE leverages the GV bound to constrain the learning of hash centers for category discovery. This approach allows the models to be flexible enough to preserve the rich category information contained in feature-level prototypes, while also generating discriminable hash category discriminators. As a result, PHE achieves better results.\n2. **Comparison with [4] that optimizes only hash code to align with pre-calculated hash centers.** Different from [4] that uses fixed hash centers, our hash centers are mapped from the prototypes of each category and then updated by end-to-end optimization, which can alter the relationships between categories learned in the CPG module, yielding better results compared to MDSH. For example, with code bits=12, our PHE surpasses MDSH by an average of 2.3% across two datasets. Besides the difference in hash center design, our Hamming-ball-based inference design effectively mitigates the hash sensitivity issue. For instance, when code bits=16, the MDSH-Hamming ball exceeds MDSH by an average of 2.75% across two datasets.\n\n**Q2: Comparison with OrthoHash [2] in terms of loss function**.\n\nWe agree with that both PHE and OrthoHash utilize cosine similarity to constrain hash features, yet there are significant differences in our design motivations and specific implementations:\n1. **Differences in design motivation**. OrthoHash focuses on instance retrieval and applies constraints in only the low-dimensional hash space, which tends to lose category-specific information (CSI), which is especially crucial for OCD tasks. In contrast, our hash feature constraint loss $L_f$ and prototype learning loss $L_p$ are applied in hash space and feature space, respectively. The motivation is to capture CSI  as rich as possible in high-dimensional feature space. We empirically find that this design can effectively mitigate CSI loss when projecting to hash space.\n\n2. **Differences in implementation**. **a) Hash center generation**. Our hash centers are mapped from category prototypes, which preserves the relationships between categories learned in the prototype feature space. OrthoHash uses a matrix whose row vectors are mutually orthogonal as hash centers, which might hinder modeling the relationship between similar categories in OCD tasks. **b) Cosine similarity**. We compute cosine similarity between hash features and hash centers without sign quantization. In contrast, OrthoHash calculates cosine similarity between continuous hash features and binary hash centers.  **c) Update of hash center**. Our centers can be optimized during the training process, while the centers in OrthoHash are fixed.\n\n*Our PHE effectively unifies representation learning and category encoding, achieving better OCD accuracy than OrthoHash. As shown in the table of **Q1**, PHE surpasses OrthoHash by an average of 9% when code bits=12.*\n\n[1] Deep Polarized Network for Supervised Learning of Accurate Binary Hashing Codes.\n[2] One Loss for All: Deep Hashing with a Single Cosine Similarity based Learning Objective.\n[3] Central Similarity Quantization for Efficient Image and Video Retrieval.\n[4] Deep Hashing with Minimal-Distance-Separated Hash Centers.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JuEKgvpUTg", "reviewer": "Reviewer_Pa9u", "summary": "This paper addresses the On-the-fly Category Discovery (OCD) task, which involves utilizing existing category knowledge to recognize both known and unknown categories in new data streams in real-time. To tackle the high sensitivity and suboptimal feature representation issues of existing methods when dealing with fine-grained categories, the paper proposes an innovative Prototypical Hash Encoding (PHE) framework. This framework uses a Category-aware Prototype Generation (CPG) module to represent each fine-grained category with multiple prototypes and employs a probabilistic masking strategy to encourage the model to fully capture intra-class diversity. The Discriminative Category Encoding (DCE) module maps the generated category prototypes to low-dimensional hash centers, optimizing image hash features to ensure intra-class compactness and inter-class separation. Additionally, a center separation loss function is designed to maintain a minimum Hamming distance between different category hash centers. Experimental results on multiple datasets confirm the superiority of this method.", "questions": "Apart from the issues mentioned in weakness part, there are some additional questions regarding implementation details:\n1. The article mentions that to encourage the model to fully capture intra-class diversity, K categories are equally allocated to each class. I would like to know how significant the choice of K is on the model's performance, and if there are experiments demonstrating the impact of different K values on the model.\n2. Similarly, when using the masking strategy, how does the variation in the value of θ affect the training and final performance of the model?\n3. What about the performance of different splits of old and new class, further analysis is needed regarding the size of different splits as well as whether the correlation of old and new classes essentially influence the final performance etc. Some in-depth insights are expected.", "limitations": "As mentioned in the appendix of the manuscript, although the proposed PHE framework outperforms existing methods in terms of performance, further research is needed to improve its accuracy in recognizing unknown categories. Additionally, the PHE method involves multiple hyperparameters that require careful tuning to achieve optimal performance, increasing the complexity of model debugging and optimization. Moreover, the weights of the component loss functions in the total loss function need to be appropriately set to balance the optimization objectives of different components, which may require extensive experiments to find the suitable weight combination.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The proposed PHE method demonstrates its superiority over existing state-of-the-art methods across multiple datasets. It effectively addresses the issues of large intra-class variance and small inter-class variance, while minimizing the information loss associated with dimensionality reduction. The paper employs visualization techniques to analyze the underlying mechanism by which PHE groups samples into known or unknown categories. Moreover, due to its optimization based on hash centers and inference process based on Hamming balls, the PHE method shows stable performance across different hash code lengths, effectively mitigating the \"high sensitivity\" problem. In contrast to the SMILE method, which exhibits significant accuracy degradation and instability with increasing hash code lengths, the PHE method maintains remarkable stability and consistency.", "weaknesses": "Although the proposed Prototype Hashing Encoding (PHE) framework outperforms existing methods in terms of performance, further research is needed to improve its accuracy in recognizing unknown categories. Additionally, due to the need to compute multiple hash centers and perform complex distance calculations, the computational cost of PHE is relatively high, especially on large-scale datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer’s recognition of our motivation and experimental results in addressing the hash sensitivity issue. We value the reviewer’s insightful comments and will include these into our final revision.\n\n**Q1: Improving accuracy in recognizing unknown categories.**\n\n1. **Limited room for improvement due to the unique challenge of OCD.** Compared with GCD/NCD tasks, OCD tasks **do not** use unlabeled data that might include unknown-category data for model training, which leads to a larger challenge for OCD methods to recognize unknown categories. We discussed the limitation in the main paper. Compared to the SOTA method, SMILE, our PHE achieves better accuracy for unknown categories. On the CUB dataset, our method exceeds SMILE by 4.1%/16% on 12/16 bits, respectively. Thus, as recognized by Reviewer ZQiT and dwc8, such an improvement might be non-trivial for OCD.\n\n2. **A possible solution in future work.** Due to the constraint of training data in OCD setting, we consider introducing additional knowledge from pre-trained Large Language Models (LLMs). Firstly, we can leverage LLMs to establish a bank of category attribute prototypes that are expected to be shared across both known and unknown categories. Then, during the on-the-fly prediction process, we plan to use LLM+VLM to match the attribute prototypes for unknown categories. Finally, by jointly considering the instance and attribute features, we hope that our PHE can generate more accurate predictions.\n\n**Q2: Computational cost for large-scale datasets**. \nWe agree that the computational cost of PHE is relatively high; however, the computational cost mainly depends on the number of known categories, which is not directly related to dataset scale. Specifically, each known category has a hash center, thus hash centers are represented by a tensor $\\mathbf h$ with a shape [num_class, bit]. Main calculations involve simple two-dimensional matrix multiplications, which include the dot product of features $\\mathbf{b}$ with a shape [batch_size, bit] and hash centers, $\\mathbf{b} * \\mathbf{h}$, as well as the dot product operations in the calculation of Hamming distances $\\mathbf{h} * \\mathbf{h}^\\text{T}$. Empirically, we have verified the analyses in the table below. It shows that although the Food dataset is larger in scale than CUB-200 and Scars-198, it contains only 100 categories. Therefore, the average training time per sample for the Food dataset is significantly lower than that for CUB and Scars.\n\n| Dataset              | CUB#200 | SCars#198 | Food#101 |\n| -------------------------------- | ------- | --------- | -------- |\n| number of training samples       | 1.5k    | 2.0k      | 19.1k    |\n| training time / minute           | 100.22  | 161.37    | 691.39   |\n| training time per sample / second | 4.01    | 4.84      | 2.17     |\n\n**Q3: The impact of different K values on the model**. We regard the \"K categories\" in the review comment as the number of prototypes per category, $k$. We have detailed the impact of this hyperparameter in Fig. 4 and Sec.3.4 of the main paper. Specifically, $k=1$ results in suboptimal performance, as it doesn't effectively represent the complexity of a category. Using larger $k$ captures the nuances within a category, which is crucial for fine-grained categories. However, when $k>5$, the improvement is minimal, as validated on two datasets. Therefore, K is not sensitive, and we choose a relatively large value of 10 for all datasets to avoid over-tuning this hyperparameter.\n\n**Q4: The impact of $\\theta$ in the masking strategy.** We have added experiments on the two datasets and reported results in the table below. The masking strategy helps reduce redundancy when using multiple prototypes, thus facilitating prototype learning. A smaller $\\theta$ value is found to be appropriate; within the range of (0, 0.2), the masking strategy can improve accuracy. When $\\theta$ exceeds 0.2, suboptimal results occur on both datasets. We did not fine-tune this parameter for each dataset but instead set it at 0.1 across all datasets. \n\n| Dataset| |CUB| | | SCars| |\n| - | -| - | - | - | - | - |\n| value of $\\theta$ |All |Old |New|All|Old|New|\n| 0| 35.5| 53.1| 26.6| 30.7| 60.3| 16.7|\n| 0.05| 36.1| 53.6| **27.4** | **31.6** | **63.4** | 16.3|\n| 0.1| **36.4** | **55.8** | 27.0| 31.3| 61.9| **16.8** |\n| 0.15| 36.0     | 54.3     | 26.9     | 31.4     | 62.2     | 16.6     |\n| 0.2| 35.3     | 52.7     | 26.6     | 30.1     | 59.8     | 15.7     |\n\n**Q5: Results of different dataset splits.** Good suggestion! We added experiments using different proportions of old category selection on the CUB and SCars datasets below, with all accuracy reported and code bits=12. Based on the results, when the proportion of selected old categories is 75%/25%, our PHE outperforms SMILE by an average of 5.95%/1.0%. This indicates that our PHE is more capable of modeling the nuanced inter-category relationships in fine-grained OCD, as the number of categories increases. Results across all datasets and more code bits will be supplemented in the revision.\n\n| Method | CUB-25% | CUB-50% | CUB-75% | Scars-25% | Scars-50% | Scars-75% |\n| -------------- | --------------- | --------------- | --------------- | ----------------- | ----------------- | ---------------- |\n| SMILE          | 19.9            | 32.2            | 41.2            | 12.6              | 26.2              | 37.0             |\n| PHE (ours)      | **21.2**        | **36.4**        | **46.5**        | **13.3**          | **31.3**          | **43.6**         |\n\n**Q6: Hyperparameter section.** 1) To avoid over-tuning hyperparameters, we use a fixed set of hyperparameters for all datasets (obtained based on CUB) to report accuracy, without tuning them individually for each dataset. 2) The hyperparameters we use are relatively robust within a certain range. In the loss function, the difference in loss scale determines the scale of the proportions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ft699s62gU", "reviewer": "Reviewer_ZQiT", "summary": "This paper introduces a novel framework called Prototypical Hash Encoding (PHE) for On-the-fly Category Discovery (OCD), which aims to discover both known and unknown categories from streaming data using labeled data of known categories. PHE first learns many prototypes for each category and then maps the learned prototypes to hash codes to distinguish samples from known or novel categories with a threshold.", "questions": "1. Compared with feature-level prototypes or prototypes after dimensionality reduction, what are the advantages of hash code-based prototypes? \n2. How is the training efficiency of the model compared to SMILE?\n3. How are hyperparameters selected without the validation set?", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-written and easy to follow.\n2. The proposed method addresses the limitations of previous hash-based OCD models by reducing sensitivity and preserving discriminative information through prototypes.\n3. The proposed method achieves significant improvements in accuracy across various datasets.", "weaknesses": "1. The proposed method is an improvement from the previous work [1], sharing the same core idea of utilizing hash codes for OCD, which limits the novelty of the paper.\n2. Despite the improved performance, the reasons why prototypes can address the sensitivity issue are not analyzed in depth.\n3. Compared with feature-level prototypes, the advantages of hash code-based category prototypes have not been demonstrated or experimentally verified.\n\n[1] Ruoyi Du, Dongliang Chang, Kongming Liang, Timothy Hospedales, Yi-Zhe Song, and Zhanyu Ma. On-the-fly category discovery. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11691–11700, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer’s recognition of our motivation to address the limitations of previous OCD models. We value the reviewer’s insightful comments and will incorporate these into our final revision.\n\n**Q1: Sharing the same core idea of utilizing hash codes for OCD**. We indeed use hash codes as category descriptors, which are also widely used in image retrieval and deep hashing methods. However, we argue that our innovation lies in: 1. Identifying the high sensitivity issue when applying hash codes to the challenging fine-grained OCD task, where only data from known categories is available. 2. We have introduced a new OCD framework, PHE, that explicitly achieves inter-class separability and intra-class compactness, compared to the SOTA method SMILE. 3. Unlike SMILE, we designed an on-the-fly inference method based on the Hamming ball. Our PHE method effectively mitigates the high sensitivity problem, for example, surpassing SMILE by 15.5% in terms of all accuracy on the CUB dataset with hash code bits = 64. This point has also been acknowledged by Reviewers 6xA4 and Pa9u.\n\n**Q2: The reasons why prototypes can address the sensitivity issue**.\nThanks for your insightful comments. We'd like to explain as follows.\n1. In fact, instead of depending on only prototypes themselves, we mitigate the sensitivity issue by 1) constraining the hash centers (mapped from feature-level prototypes) to be at least a Hamming distance of $d_{max}$ apart based on the Gilbert-Varshamov bound; 2) representing a category using a Hamming ball of radius $max(\\lfloor \\frac{d_{max}}{2} \\rfloor, 1)$.\n\n2. The role of prototypes is to: 1) achieve better representation learning; 2) map the category prototypes to category hash centers, unifying representation learning with hash encoding. The importance of prototype learning is evident from the variant without prototype learning loss $L_f$ in Tab.2 of the main paper. Removing the prototype learning loss $L_f$ results in an obvious accuracy drop on datasets in terms of all, old and new accuracy.\n\nWe will clarify this point more clearly in the introduction.\n\n**Q3: Feature-level prototypes vs.  Hash code-based prototypes**\n1. **Experimental results with a powerful feature-level prototype learning method.**\nFirstly, we would like to further explain that in the OCD setting, test data appear one-by-one, and the OCD task requires the model to instantly assign a category to the test data. Therefore, if only feature-level prototype learning is used, the method needs an additional online clustering approach to yield real-time category descriptors, which is also acknowledged in the original OCD paper [1]. Thus, we adapt SimGCD [2] ( an advanced prototype learning approach for GCD, which was also acknowledged by Reviewer dwc8) into OCD settings and implement two variant approaches: 1) SimGCD attached a online clustering approach SLC in [1] to achieve instance-wise inference. 2) SimGCD with our hash prototype framework. The experimental results below show that introducing PHE significantly improved all accuracy across two datasets. Meanwhile, our full method achieves better generalization ability, especially on new categories.\n\n| Dataset                                         |          | CUB      |          |          | Scars    |          |\n| ----------------------------------------------- | -------- | -------- | -------- | -------- | -------- | -------- |\n| Method                                          | All      | Old      | New      | All      | Old      | New      |\n| SimGCD [2] (only feature-level prototype learning) | 27.1     | 50.7     | 15.3     | 21.5     | 39.9     | 12.5     |\n| SimGCD [2] + hash code-based prototype learning             | 34.1     | **60.6** | 20.8     | 30.3     | **65.9** | 13.0     |\n| PHE (ours)                        | **36.4** | 55.8     | **27.0** | **31.3** | 61.9     | **16.8** |\n\n\n2. **Advantages of hash code-based features.** Hash code-based features can directly use the features’ signs (hash codes) as category descriptors, where features with the same category descriptor are considered to be of the same category. Therefore, compared to feature-level prototype learning, introducing a hash code-based design can directly optimize category descriptors, achieving higher accuracy in the OCD setting. According to the results in Table 1 of the main paper, where only feature-level prototype learning results (SLC) and hash code-based prototypes learning (our PHE) are compared, introducing hash code-based prototypes improved the average accuracy by 8.2% across eight datasets.\n\n**Q4: Training efficiency**. We provided a comparison of training times between our PHE and the SOTA method, SMILE, measured in minutes, as shown in the table below. To ensure fairness, all experiments were conducted on an NVIDIA RTX A6000 GPU. Both algorithms were trained for 200 epochs using mixed precision training. The dataloader parameters were consistent, with a batch size of 128 and num_workers set to 8. According to the table, our average training time across four datasets is less by 45.8 minutes compared to SMILE. This is primarily due to SMILE’s use of supervised contrastive learning with two views of samples for representation learning, which requires higher computational resources.\n\n| Method    | CUB    | Scars  | Food   | Pets  |\n| --------- | ------ | ------ | ------ | ----- |\n| SMILE     | 127.70 | 177.54 | 819.93 | 80.37 |\n| PHE (ours) | 100.22 | 161.37 | 691.39 | 69.48 |\n\n**Q5: Hyperparameter section.** To avoid over-hyperparameter tuning, we use a fixed set of hyperparameters which are obtained based on CUB to report accuracy for all datasets, without tuning them individually for each dataset.\n\nIf we have misunderstood any of your concerns, please let us know in the future comments.\n\n[1] On-the-fly category discovery. CVPR 2023.\n\n[2] Parametric Classification for Generalized Category Discovery: A Baseline Study. ICCV 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2vsjAOXnVG", "reviewer": "Reviewer_dwc8", "summary": "This paper proposes a prototypical deep hashing framework to address the fine-grained on-the-fly category discovery problem. The proposed method includes two main loss functions: first, distance minimization between the encoded hash features to the category-representative hash coding after projection $\\mathcal{H}_h$; second, enforcing minimal distance separation among hash encoding of different categories, which are enforced to be quantized to -1 and +1 for each dimension. The experimental results demonstrate the effectiveness of each loss function. The performance gains are considerable on all benchmarks compared with previous state-of-the-art baselines.", "questions": "- It would be clearer if the author could mention the technical practice of prototype-based visualization examples, though it may be proposed by the previous work, ProtopFormer.\n- In Table 3, how will the semi-supervised contrastive loss perform [1] in addition to the supcon and unsupcon baselines? These results matter to solidify the motivation for using the deep hashing framework.\n\n\n[1] Generalized Category Discovery\n\n[2] PromptCAL: Contrastive Affinity Learning via Auxiliary Prompts for Generalized Novel Category Discovery\n\n[3] Parametric Classification for Generalized Category Discovery: A Baseline Study\n\n[4] Deep hashing with minimal-distance-separated hash centers", "limitations": "As mentioned by the author, on-the-fly category discovery is a novel and challenging problem, therefore existing methods do not achieve satisfying performance on real-world tasks.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The performance gains of the proposed approach are considerable.\n- The proposed method is robust to the varying coding length.\n- The identified ‘sensitivity issue’ problem exists for on-the-fly category discovery.", "weaknesses": "- Writing: It is informal to leave all the related works in the appendix, which will confuse the readers on the contribution of this work. Besides, some expressions are not\n- Novelty: The technical contribution is limited in that the prototypical learning [3] is a mature practice in category discovery and the author seems to adapt the deep hashing method in [4] to this problem.\n- Motivation: The motivation of achieving the balance between instance discrimination and class discrimination, especially with prototypical learning in the category discovery field is not new [2,3]. However, this is accepted to some extent since on-the-fly category discovery is a new problem.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer’s recognition of our method’s good performance and its robustness to varying coding lengths (sensitivity issue). We value the reviewer’s insightful comments and will incorporate these into our final revision.\n\n**Q1: Writing**. Thanks for your constructive suggestions. We will **leave the section of related works to the main text**, and carefully check and improve the expressions in the revision.\n\n**Q2: Technical contribution**.\n\nWe argue that our contribution mainly lies in identifying and effectively mitigating the high sensitivity issue existing in current hash-based OCD methods, which was acknowledged by Reviewers 6xA4 and Pa9u as well. We provide elaborated discussions as follows.\n\n**1. Compared to prototype learning method (SimGCD [1])**, our PHE has two main advantages. **First**, unlike SimGCD that learns only one prototype for one category, our PHE generates multiple prototypes for one class, which is favorable for modeling intra-class variance of fine-grained categories. Empirically, we have supplemented experiments with the prototype learning method used in SimGCD to validate the superiority of our methods as shown in the table below.\n\n| Dataset       |          | CUB      |          |          | SCars    |          |\n| ------------- | -------- | -------- | -------- | -------- | -------- | -------- |\n| Method        | All      | Old      | New      | All      | Old      | New      |\n| SimGCD+PHE    | 25.0     | 49.9     | 12.6     | 21.9     | 38.5     | 13.9     |\n| SimGCD-MC+PHE | 34.1     | **60.6** | 20.8     | 30.3     | **65.9** | 13.0     |\n| PHE (ours)     | **36.4** | 55.8     | **27.0** | **31.3** | 61.9     | **16.8** |\n\nSpecifically, due to lack of unlabeled data, we removed the $L_{cls}^u$ and the prototypes corresponding to new categories in SimGCD. In the table, \"SimGCD+PHE\" indicates that we used SimGCD for prototype learning while also mapping the prototypes learned by SimGCD to hash centers for category encoding. This approach yielded very poor results on both datasets. \"SimGCD-MC+PHE\" refers to the use of manually obtained centers that satisfy the Gilbert-Varshamov bound and features from the SimGCD projection head for category encoding. Compared to mapping SimGCD’s prototypes to hash centers, the SimGCD-MC+PHE variant shows an average improvement of 8.8% on two datasets, demonstrating that the prototypes learned by SimGCD are not suitable for category encoding in fine-grained OCD scenarios.\n**Second**, Unlike the prototypes in SimGCD which are weights of classifiers, the learned prototype in PHE can be explicitly visualized, which provides additional perspective for interpreting the model's behavior, as illustrated in Fig.3 of the main text.\n\n**2. Compared with deep hash method (MDSH[2]),** our PHE has two main differences: **First**, unlike MDSH that pre-calculates hash centers and fixes these centers during training, our hash centers are derived by mapping from category prototypes and then are updated by end-to-end optimization, which better preserves the relationships between fine-grained categories learned in the prototype feature space. **Second**, we design an additional Hamming ball-based inference tailor-made for OCD, which effectively mitigates the sensitivity issues associated with using hash codes. Furthermore, we have supplemented experiments and the results in the following table verify the superiority of our PHE. We also provide the results of more hash-based baselines in **Q1** of Reviewer **6xA4.** \n\n| Datasets  |          | CUB      |          |          | SCars    |          |\n| --------- | -------- | -------- | -------- | -------- | -------- | -------- |\n| Methods   | All      | Old      | New      | All      | Old      | New      |\n| MDSH[2]   | 34.3     | **57.6** | 22.8     | 28.8     | 60.2     | 13.7     |\n| PHE (ours) | **36.4** | 55.8     | **27.0** | **31.3** | **61.9** | **16.8** |\n\n**Q3: Motivation of instance discrimination and class discrimination**\n\nWe agree that both OCD and GCD [3] need to learn good representations with “optimal balance between instance discrimination and class discrimination”. However, our primary motivation comes from the identification of the high sensitivity issue caused by applying hash codes in OCD tasks, which significantly hinders the effectiveness of the current hash-based OCD method. Motivated by this, we present the specific prototype design, loss implementation and Hamming-ball-based inference, to mitigate the high-sensitivity issue.\n\n**Q4: Technical practice of prototype-based visualization examples**. The process mainly consists of three steps: 1) Input an image to obtain its feature representation, while also capturing the attention map during forward propagation. 2) Select the top-k most similar (activated) prototypes from all prototypes. 3) Visualize the original samples/images corresponding to these prototypes. We will include detailed examples and process descriptions in the revision.\n\n**Q5: Will the semi-supervised contrastive loss (SSCL) perform?** Firstly, we'd like to clarify that in the OCD setting, only labeled data of known classes is available for model training, and unlabeled data appears only in an on-the-fly format during testing. Thus, the SSCL used in GCD[1] is incompatible with OCD. As for the experiments in Tab. 3, in fact, we did not include experiments with \"supcon and unsupcon baselines\". “Supcon Cls” represents the use of classification methods based on supervised contrastive learning for representation learning. According to the results, we find that although this variant performs well on seen categories, their generalization capabilities are inferior to our full prototype-based method.\n\n[1] Parametric Classification for Generalized Category Discovery: A Baseline Study. ICCV 2023\n\n[2] Deep Hashing with Minimal-Distance-Separated Hash Centers. CVPR 2023.\n\n[3] Generalized Category Discovery. CVPR 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank the ACs and reviewers for their considerable efforts in handling our paper.

We have appropriately addressed all concerns raised by the reviewers. These include providing more baselines with deep hashing methods (Reviewer #6xA4, #dwc8) and prototype learning methods (Reviewer #ZQiT, #dwc8), conducting additional ablation studies on hyper-parameters (Reviewer #Pa9u), explaining our choices of hyper-parameters (Reviewer #Pa9u, #ZQiT), comparing training efficiencies (Reviewer #ZQiT), and offering a clearer explanation of our contribution and motivation (Reviewer #ZQiT, #dwc8).

**Paper strengths acknowledged by reviewers:**

The motivation and idea are clear and simple (Reviewer #6xA4), the method is easy to follow (Reviewer #ZQiT), effectively mitigates the high sensitivity issues present in current On-the-fly Category Discovery models (Reviewer #6xA4, #Pa9u, #ZQiT, #dwc8), and demonstrates good performance (Reviewer #Pa9u, #ZQiT, #dwc8). The paper is well-written and easy to understand (Reviewer #6xA4, #ZQiT).

We expect the ACs and reviewers to fully consider the following factors when making the final decision: (1) A novel prototypical hash encoding framework for On-the-fly Category Discovery, which effectively mitigates the “high sensitivity issues” of current OCD methods and achieves significant performance improvements; (2) Thanks to the interpretable prototype learning, we provide a perspective for explaining the model's behavior when discovering categories. (3) Comprehensive responses to all the reviewers’ comments.

Please let us know if you have any additional questions or concerns. We are happy to provide further clarification.

Authors of Submission #524

Meta-review: This paper addresses the On-the-fly Category Discovery (OCD) task, which involves utilizing existing category knowledge to recognize both known and unknown categories in new data streams in real-time. A novel framework has been proposed to address this task. In general all the reviewers are positive about the contribution and performance of the proposed work. Thus, the recommendation is to accept the paper. The authors are strongly encouraged to address the comments in the camera-ready version.
Decision: Accept (poster)