Prototypical Hash Encoding for On-the-Fly Fine-Grained Category Discovery

Authors: haiyang  zheng
nan  pu
wenjing  li
nicu  sebe
zhun  zhong
Venue: NeurIPS_cc_2024_Conference
Published: 2024-04-24 09:48:16

Abstract: In this paper, we study a practical yet challenging task, On-the-fly Category Discovery (OCD), aiming to online discover the newly-coming stream data that belong to both known and unknown classes, by leveraging only known category knowledge contained in labeled data. Previous OCD methods employ the hash-based technique to represent old/new categories by hash codes for instance-wise inference. However, directly mapping features into low-dimensional hash space not only inevitably damages the ability to distinguish classes and but also causes ``high sensitivity'' issue, especially for fine-grained classes, leading to inferior performance. To address these drawbacks, we propose a novel Prototypical Hash Encoding (PHE) framework consisting of Category-aware Prototype Generation (CPG) and Discriminative Category Encoding (DCE) to mitigate the sensitivity of hash code while preserving rich discriminative information contained in high-dimension feature space, in a two-stage projection fashion. CPG enables the model to fully capture the intra-category diversity by representing each category with multiple prototypes. DCE boosts the discrimination ability of hash code with the guidance of the generated category prototypes and the constraint of minimum separation distance. By jointly optimizing CPG and DCE, we demonstrate that these two components are mutually beneficial towards an effective OCD. Extensive experiments show the significant superiority of our PHE over previous methods, e.g. obtaining an improvement of +5.3% in ALL ACC averaged on all datasets. Moreover, due to the nature of the interpretable prototypes, we visually analyze the underlying mechanism of how PHE helps group certain samples into either known or unknown categories. Code is available at https://github.com/HaiyangZheng/PHE.

Main paper: 
 introduction: 
 While deep learning based machines have surpassed humans in visual recognition tasks [1, 2, 3, 4] , their capability is often limited to providing closed-set answers, e.g., category names. In contrast, humans possess the ability to recognize novel categories upon first observation without knowing their category names. To bridge this gap, Novel/Generalized Category Discovery (NCD/GCD) techniques [5, 6, 7, 8, 9, 10, 11] are proposed to transfer knowledge from known categories to distinguish unseen ones. However, current NCD/GCD methods operate under an offline inference paradigm where the category discovery is often implemented by applying clustering / unsupervised classification algorithms on a pre-collected batch of query data that needs to be discovered. This severely limits the practicability of NCD/GCD techniques in real-world applications, where the systems are expected to provide online feedback for every newly-coming instance.To tackle this drawback, Du et al. introduce the On-the-fly Category Discovery (OCD) task [12] , which removes the assumption of a predefined query set and requires instance feedback with stream data input. OCD poses two primary challenges: 1) The requirement for real-time feedback is Figure 1 : (a) Schema of Offline Category Discovery task (e.g., NCD [5] and GCD [8] ). (b) Schema of On-the-fly Category Discovery task [12] , studied in this paper. (c) Previous work (e.g., SMILE [12] ) based on instance-level hash encoding. (d) Our PHE explores prototype-based hash encoding. (e) Performance comparison of PHE and SMILE and observation about "High Sensitivity". incompatible with offline cluster-based methods [8, 9] (see Fig. 1 (a) ). 2) Due to uncertain openworld scenarios, the NCD/GCD methods (e.g., classifier-based methods [13] ), which assume the number of categories can be available/discovered as a prior, are ineffective for OCD. As a solution, Du et al. propose the SMILE architecture [12] , which employs the sign of an image feature (i.e., hash code) as its category descriptor to identify the corresponding category for online discovery. As illustrated in Fig. 1 (c ), SMILE directly mapps image features into low-dimensional hash space with an instance-level contrastive objective and regard that one hash code uniquely represents a category. Given this, although SMILE can derive category descriptors, it suffers from a significant issue of "high sensitivity" for learned hash-form category descriptors and thus produces a significantly inaccurate number of categories as well as unsatisfied performance. As shown in Fig. 1 (e), this drawback of SMILE is even more severe for fine-grained categories due to the small inter-class variations and large intra-class variations between samples. For example, given two images of the same bird category but with different poses, they commonly share very similar overall features. However, under SMILE, minor variations in a single feature dimension can lead to opposite signs in their hash codes when the feature values are close to zero, classifying these two samples into distinct categories. Moreover, this issue becomes more pronounced as the feature dimensionality increases, as demonstrated in Table 5 .To address this issue, we propose a new two-stage Prototypical Hash Encoding (PHE) framework to improve the intra-class compactness and inter-class separation of category descriptors while alleviating information loss due to dimension reduction. PHE is composed of Category-aware Prototype Generation (CPG) and Discriminative Category Encoding (DCE). First, CPG utilizes prototype-based interpretable classification models to learn multiple prototypes for each category and yields sparse representations with a probabilistic masking strategy. Then, DCE explicitly maps the prototypes of each category to corresponding category hash centers. By imposing hash codes approaching their hash centers, OCD models are encouraged to produce more accurate hash codes. Meanwhile, we design a center separation loss to ensure that different category hash centers maintain at least a Hamming distance of d max , where the d max is derived from Gilbert-Varshamov bound in coding theory, as illustrated in Fig. 1 (d) . Finally, we design a tailor-made OCD model inference method based on a relaxed Hamming ball condition. Overall, the two-stage PHE framework enables OCD models to maintain the discriminabilities learned in high-dimension feature space and transfer them into low-dimensional encoding space, thereby enhancing accuracy for both seen and unseen categories. Extensive experiments conducted across multiple fine-grained benchmarks demonstrate that our approach substantially outperforms the SMILE architecture (see Fig. 1 (e) ). Our contributions are summarized as follows:• We propose a new PHE framework to explicitly generate prototypical hash centers, which is beneficial for improving the intra-class compactness and the inter-class separation of hash codes.Our PHE can effectively alleviate the "high sensitivity" issue of hash-based OCD methods. • We design a tailor-made center separation loss to further improve the discriminability of hash centers by constraining a minimal separation distance derived from coding theory [14] . We also provide visual analyses to better understand the underlying mechanism of our PHE. • Experiments on eight fine-grained datasets show that our method outperforms previous methods by a large margin, i.e., +5.3% improvement averaged on all datasets for all class accuracy. 
related works: 
 Novel Category Discovery (NCD), initially introduced by DTC [5] , aims to categorize unlabeled novel classes by transferring knowledge from labeled known classes. However, existing NCD methods [5, 6, 7] assume that all unlabeled data exclusively belong to novel classes. Generalized Category Discovery (GCD) is proposed in [8] , allowing unlabeled data to be sampled from both novel and known classes. While existing NCD/GCD methods [6, 7, 8, 9, 13, 15, 16] have shown promising results, two key assumptions still impede their real-world application. Firstly, these models heavily rely on a predetermined query set (the unlabeled dataset) during training, limiting their ability to handle truly novel samples and hindering generalization. Secondly, the offline batch processing of the query set during inference makes these models impractical for online scenarios where data emerges continuously and the model requires instant feedback. To address these limitations, Du et al. introduced the On-the-Fly Category Discovery (OCD) [12] , which removes the assumption of a predefined query set and requires instance feedback with stream data input. They proposed the SMILE method, which identifies the category of each instance by the signature of its representation (a hash-form code). The comparison among different settings is shown in Table 1 .In this paper, we address the "high sensitivity" issue associated with hash-based category descriptors in SMILE, particularly in fine-grained scenarios. We encode category prototypes into hash centers, ensuring maximal separation between these centers. Additionally, we employ Hamming balls centered on these hash centers to represent each category, effectively mitigating the "high sensitivity" issue. ✓ ✓ Old+New * ✗ ✓ Old+New * ✗ GCD ✓ ✓ Old+New * ✓ ✓ Old+New * ✗ OCD ✓ ✗ Old ✓ ✓ Old ✓Deep Hashing is a popular method for large-scale image retrieval. It uses deep neural networks to learn a hash function that converts samples into fixed-length binary codes, ensuring similar samples share similar codes. Early methods, such as HashNet [17] , DPSH [18] , and DSH [19] , optimize hash functions based on pairwise similarities or triplet-based similarities. Both approaches suffer from low training efficiency and insufficient data distribution coverage. By defining points as hash centers that are sufficiently spaced apart, methods like DPN [20] , OrthoHash [21] , and CSQ [22] largely enhance training efficiency and retrieval accuracy. However, in the worst case, hash centers derived from these methods can be arbitrarily close. To address this, Wang et al. [23] introduce the Gilbert-Varshamov bound from coding theory to ensure a large minimal distance between hash centers.Drawing inspiration from deep hash methods, we employ a hash center-based approach for category encoding. Unlike deep hashing methods used in image retrieval tasks [22, 23] , which rely on predefined center points, our method generates category-specific hash centers directly from category prototypes. This adaptation is particularly suitable for the category discovery task. Furthermore, we utilize Hamming balls centered on these hash centers to represent categories, effectively mitigating the "high sensitivity" issue associated with using hash-form descriptors in category discovery tasks.Prototype-based Interpretable Models. Chen et al. [24] introduce the Prototypical Part Network (ProtoPNet), a model designed for Interpretable classification. ProtoPNet features a set number of prototypical parts per class, enabling clear decision-making process. In addition, ProtoPNet offers a post-hoc analysis, in which it explains decisions for individual images by displaying all prototypes alongside their weighted similarity scores. This method employs multiple prototypes to represent a category, effectively distinguishing subtle differences between categories and demonstrating strong performance in fine-grained classification. Numerous subsequent studies have adapted ProtoPNet for various applications such as medical image processing and explanatory debugging, among others [25, 26, 27, 28, 29] . Later, ProtoPFormer [30] further integrates the Vision Transformer as a backbone, utilizing both global and part prototypes for interpretable image classification.In this paper, we utilize prototype-based models for representation learning and prototype acquisition. Unlike existing methods predominantly tailored for closed-set classification, our approach extends to the generation of discriminative hash codes from the learned category prototypes. This extension allows for broader applicability in handling new and unseen categories, i.e., open-set scenarios. E v k v Z m k N B B 4 K F n M C D Z W 8 v 0 h f U D R I B f 4 a T q o 1 t y 6 O w d a J V 5 B a l C g N a h + + V F C M k G l I R x r 3 f f c 1 A Q 5 V o Y R T q c V P 9 M 0 x W S M h 7 R v q c S C 6 i C f 3 z x F Z 1 a J U J w o W 9 K g u f p 7 I s d C 6 4 k I b a f A Z q S X v Z n 4 n 9 f P T H w d 5 E y m m a G S L B b F G U c m Q b M A U M Q U J Y Z P L M F E M X s r I i O s M D E 2 p o o N w V t + e Z VE E h D X n f h Q J T E u C O J J L i f s Q x Z C 7 F P X d 6 m e V 7 d 5 g L E g a 3 c h Z h h 8 F x Q H y C o F R o Z K D K k E E 5 Q Z A m V 2 l 1 i L x Q n u r f r J W O E v 8 f P l n i r p 9 E C r N U / y q 7 V n f F j L J V s + Z h r g o 7 F 2 W Q R 3 t k P A 6 9 E M U M B x J R K M T A t i L p J J B L g i h WQ n D o 1 n W O V 7 D C c = " > A A A C K 3 i c b Z D L S s N A F I Y n X m u 8 R V 2 6 C Z Z C 3 Z R E S n V Z F K H L C v Y C T Q i T y a Q d O r k w M x F K y P u 4 8 V V c 6 M I L b n 0 P J 2 2 8 t P W H g Z / v n M O c 8 7 s x J V w Y x p u y s r q 2 v r F Z 2 l K 3 d 3 b 3 9 r W D w y 6 P E o Z w B 0 U 0 Y n 0 X c k x J i D u C C I r 7 M c M w c C n u u e O r v N 6 7 w 4 y T K L w V k x j b A R y G x C c I C o k c 7 b J i B V C M E K T p d V a 1 k B e J U / W X t T I n 9 X / 4 H B 5 9 Y 0 c r G z V j K n 3 Z m I U p g 0 J t R 3 u y v A g l A Q 4 F o p D z g W n E w k 4 h E w R R n K l W w n E M 0 R g O 8 U D a E A a Y 2 + n 0 1 k y v S O L p f s T k C 4 U + p X 8 n U h h w P g l c 2 Z l v y x d r O f y v N k i E f 2 G n J I w T g U M 0 + 8 h P q C 4 i P Q 9 O 9 w j D S N C J N B A x I n f V 0 Q g y i I S M V 5 U h m I s n L 5 v u W c 1 s 1 B o 3 9 X K z X s R R A s f g B F S B C c 5 B E 7 R A G 3 Q A A v f g E b y A V + V B e Vq Y k U q r L i g h d V r A X a E K Y T C f t 0 M m F m Y l Q Q p 7 H j a / i p g u l u P V B n K T x 1 n p g 4 O f 7 z 2 H O + d 2 I E i 4 M Y 6 G U N j a 3 t n f K u + r e / s H h k X Z 8 0 u N h z B D u o p C G b O B C j i k J c F c Q Q f E g Y h j 6 L s V 9 d 3 q b + f 1 H z D g J g w c x i 7 D t w 3 F A P I K g k M j R b q q W D 8 U E Q Z r c p T U L j U J x o f 6 w d u o k 3 j c f O 0 n u u F 4 S p V / U 0 S p G 3 c h L X x d m I S q g q I 6 j z a 1 R i G I f B w J R y P n Q N C J h J 5 A J g i h O V S v m O I J o C s d 4 K G U A f c z t J D 8 1 1 a u S j H Q v Z P I F Q s / p 7 4 k E + p z P f F d 2 Z p v y V S + D / 3 n D W H j X d k K C K B Y 4 Q M u P v J j q I t S z 3 P Q R Y R g J O p M C I k b k r j q a Q A a R k O m q M g R z 9 e Rx i n J 7 1 R J R w T L o 4 Y h E f u E g Q R k P S l V Q y M o g 5 Q Y H L S N 9 9 v s v z / R f C B Y 3 C J z m N iR 2 g c U h 9 i p F U y D H e y q M A y Q l G L L 3 P K i P s R b J a X L J 2 5 q T + P 3 y y x l 0 / j R U O s p W 6 B w X i b F n w q j x d 8 e 7 c / 9 i J s l Z W d I y S W T First, CPG generates category-specific prototypes and prototype-guided instance representations. Then, DHC encodes the generated prototypes as hash centers to encourage the model to learn discriminative instance hash codes. Finally, depending on the Hamming distance between instance hash codes and hash centers, we can obtain instant feedback and online group instances into both known and unknown categories.N n A T e F t R A l s I i O Y 7 y P v A g n A Q k l Z k i I o W X G 0 k 4 R l x Q z o v Z P B I k R f k Z j M l Q y R A E R d j q b X g b L i n j Q j 7 h a o Y Q z u t q R o k C I a e C q y v y a Y j 2 X w 7 9 y w 0 T 6 t 3 Z K w z i R J M T z g / y E Q R n B / C u g R z n B k k 2 V Q J h T d V e I J 4 g j L N W H 5 U O w 1 p + 8 K X o 3 N a t R a z z W S 6 3 6 Y h w F c A E u Q Q V Y o A l a o A 0 6 o A s w + N S g V t W u t C + 9 r F / r t X m p r i 1 6 z s G v 0 J v f W F X C g w = = < / l a t e x i t >i R t T + s S A R w l M 0 I X 0 l A + Q T M U z m 0 0 t h U Z E x d E O u V i D h n K 5 3 J M g X Y u Y 7 q j K 7 p t j M Z f C v X D + W 7 t 0 w o U E U S x L g x U F u z K A M Y f Y V c E w 5 w Z L N l E C Y U 3 V X i D 3 E E Z b q w 7 I h 2 J t P 3 h a d i m n X z N p T t d C o L s e R A 1 f g G p S A D e q g A Z q g B d o A g 0 8 N a m X t R v v S i / q t b i 5 K d W 3 Z c w l + h V 7 / B l n b w o Q = < / l a t e x i t > 
prototypical hash encoding: 
 Problem Setup. The setting of OCD is defined as follows. We are provided with a support set, denoted asD S = {(x i , y s i )} M i=1 ⊆ X × Y Sfor training, and a query set, denoted as Framework Overview. To achieve accurate and online category discovery, we design a Prototypical Hash Encoding (PHE) framework, which mainly consists of a Category-aware Prototype Generation (CPG) module and a Discriminative Hash Encoding (DHE) module, as illustrated in Fig. 2 . CPG aims at modeling diverse intra-category information and generating category-specific prototypes for representing fine-grained categories. DHE leverages generated prototypical hash centers to further facilitate discriminative hash code generation. Finally, based on a theoretically derived bounding of the Hamming ball, we can determine the under-discovered category of instance and acquire instant feedback.D Q = {(x i , y q i )} N i=1 ⊆ X × Y Q for testing.SMILE [12] directly utilizes instance-level supervised contrastive learning on low-dimensional hash features for simultaneous representation learning and category encoding. This approach may result in inadequate representation learning, as low-dimensional features struggle to capture complex data structures and patterns, especially in challenging fine-grained scenarios (see Table 10 ).To solve this issue, we choose to perform representation learning upon the high-dimensional features instead of the low-dimensional hash features. Specifically, given a batch of input images, X, the image features are denoted asZ = H f (E(X)) ∈ R N × L,where E represents the backbone, H f denotes a linear head, and L represents the feature dimension. We use a prototype layer g p to transform Z into a similarity score vector s ∈ R m , with g p containing m learnable prototypes {p 1 , p 2 , . . . , p m }. In the design of the prototype layer, CNN-based methods [24] typically utilize the maximum pooled value from the similarity map, which is calculated between the feature map and the prototype, as the similarity score. In this paper, to align with the Vision Transformer (ViT) [4] backbone commonly-used in OCD, we propose to employ its cls token to compute the prototype similarity score, inspired by ProtopFormer [30] . The similarity score s ij for the i-th sample to the j-th prototype is calculated as follows:EQUATION)where ϵ is a small constant for numerical stability.Probabilistic Masking Strategy. To encourage models to fully capture intra-category diversity, we assign k prototypes equally to each category, thus m = k * |Y S |. Given the prototype similarity score to all prototypes in the prototype layer of i-th image s i = [s i→1 , s i→2 , ..., s i→m ], a fully connected layer (FC) is employed for supervised classification. To further discretize the prototypes, avoid redundancy, and unleash the full potential of the prototypes, we utilize a probabilistic masking strategy. This strategy masks some units of the similarity score according to probability, thereby disabling the corresponding prototypes and leaving the remaining prototypes activated during training. Specifically, for each unit of similarity score s i , we mask it with a probability following the Bernoulli distribution B(θ), with θ empirically set to 0.1. We use the following loss function to learn the image representations and prototypes:EQUATIONwhere B indicates the mini-batch of support set, ℓ is the traditional cross-entropy loss and y i is the ground truth of image x i .Discussion. The benefits of our CPG module are threefold: i) Fine-Grained Category Distinction: Fine-grained categories often exhibit only subtle differences. Our CPG module allows for representing a category with multiple prototypes, which can effectively capture and model both intra-class similarities and inter-class variances. ii) Category-specific Hash Center Generation: Based on the generated category prototypes of CPG, we further map them into low-dimensional hash features to serve as hash centers for each category. This helps in maintaining the distinctiveness of each category in the encoding space. iii) A New perspective for Classification Analysis: Beyond merely using hash encoding for category decisions, we can introduce a new prototype perspective for classification analysis. CPG enables a visualizable reasoning process for classifying unseen categories, providing a more intuitive and interpretable method for understanding category distinctions.Category Encoding Learning. Given the set of learned category prototypes P ci for category c i in CPG, we map the mean vector of each category's prototypes P ci /k to its hash center, denoted ash i = H h ( P ci /k) ∈ R L, where H h represents a linear head and L represent the feature dimension. The image feature z i is mapped to a hash feature b i , where b i = H h (z i ). To ensure that image representations of the same category as closely as possible share the same category descriptor, we constrain each hash feature to be close to its corresponding category hash center and distant from other hash centers. We employ the following loss to optimize the hash features of the images:EQUATIONwhere sim(b i , h) represents a pair-wise similarity vector consisting of the cosine similarities between the hash feature of image x i and all hash centers.Hash Centers Optimization. Due to the subtle differences between fine-grained categories, different category hash centers may become closely similar or even share identical hash codes. This hinders separability between fine-grained categories and leads to incorrect classification results. Therefore, we aim to maximize the differences between hash centers for enhancing inter-class separation.Given the hash center h i of category c i , we denote its hash code as ĥi , where ĥi = sign(h i ) and sign(•) equals 1/-1 for positive/negative values. Since the sign function is non-differentiable, we use a smoothed version of the sign function for back-propagation during training, defined as:EQUATIONwhere τ is a hyper-parameter that controls the smoothness of the sign function. Larger values of τ make the function more closely approximate the true sign function. In this paper, we set τ = 3. Consequently, ĥi = sign * (h i ). The difference between the i-th and j-th hash centers can be evaluated by the Hamming distance of their hash codes:|| ĥi -ĥj || H = L-ĥi• ĥj 2. Although we aim to maximize the differences between hash centers, we cannot simply increase the Hamming distance between all hash centers, as this can lead to model non-convergence. Since the encoding space is fixed, excessively distancing one hash code can inadvertently bring it closer to another.We design a center separation loss, L sep , to ensure that the Hamming distance between any two hash centers is greater than or equal to d, denoted as || ĥiĥj || H ≥ d. The center separation loss is defined as: [14] in coding theory, as stated in Lemma 3.1.EQUATIONLemma 3.1 For binary symbols, there exists a set of hash codes of length L, {-1, 1} L , with a minimum Hamming distance d, and a number of hash codes Q that satisfies the following inequality:EQUATIONTherefore, given a number of hash codes equal to |Y S |, the maximum d max can be determined as:EQUATION)We apply this upper bound d max to L sep . Considering that we use a smoothed version of the sign function, where computed values do not equate to precisely ±1, we impose a quantization loss to constrain the values of hash codes to be close to ±1:EQUATIONThe optimization loss L c for hash centers is defined as a combination of the center separation loss L sep and the quantization loss L q :EQUATIONModel Training. During the model training process, the total loss is formulated as follows:EQUATIONwhere α and β control the importance of center optimization and hash encoding, respectively.Hamming Ball Based Model Inference. During on-the-fly testing, given an input image x i in the query set D Q , we use bi = sign(H h (H f (E(x i )))) as its category descriptor. Due to the introduction of the center separation loss, the Hamming distance between any two hash centers is not less than d max . We consider a Hamming ball centered on the hash centers with a radius of max(⌊ dmax 2 ⌋, 1) to represent a category. Specifically, during inference, if the Hamming distance between bi and any existing hash center is less than or equal to max(⌊ dmax 2 ⌋, 1), we classify the image as belonging to the corresponding category of that hash center. Otherwise, the image is used to establish a new hash center and category. The pseudo-code for the inference is provided in Algorithm 1. 
experiment setup: 
 Datasets. We have conducted experiments on eight fine-grained datasets, including CUB-200 [31] , Stanford Cars [32] , Oxford-IIIT Pet [33] , Food-101 [34] , and four super-categories from the more challenging dataset, iNaturalist [35] , including Fungi, Arachnida, Animalia, and Mollusca. Following the setup in OCD [12] , the categories of each dataset are split into subsets of seen and unseen categories. Specifically, 50% of the samples from the seen categories are used to form the labeled set D S for training, while the remainder forms the unlabeled set D Q for on-the-fly testing. Detailed information about the datasets used is provided in the Appendix. A.1.Evaluation Metrics. We follow Du et al. [12] and adopt clustering accuracy as an evaluation protocol, formulated as ACC = 1|D Q | |D Q | i=1 I(y i = C(y i )), where y i represents the predicted labels and y i denotes the ground truth. The function C denotes the optimal permutation that aligns predicted cluster assignments with the actual class labels.Implementation Details. For a fair comparison, we follow OCD [12] and use the DINO-pretrained ViT-B-16 [36] as the backbone. During training, only the final block of ViT-B-16 is fine-tuned. In our approach, the Projector H f (•) is a single linear layer with an output dimension set to L = 768, meaning the feature dimension and prototype dimension are equal to 768. Each category has k = 10 prototypes. The FC layer in Eq. 2 is non-trainable, which uses positive weights 1 for prototypes from the same category and negative weights -0.5 for prototypes from different categories. The Projector H h (•) consists of three linear layers with an output dimension set to L = 12, which produces 2 12 = 4096 binary category encodings. By default, we follow OCD to set this dimension for fair comparison. Additional experiments with varying L are reported in Sec. 4.4. The estimated values of d max in center separation loss L sep can be found in Appendix. A.2. The ratio α and β in the total loss are set to 0.1 and 3, respectively, for all datasets. More details and the pseudo-code for the training process can be found in the Appendix. A.2.Compared Methods. Given that OCD is a relatively new task requiring instantaneous inference, traditional baselines from NCD and GCD are unsuitable for this setting. Consequently, we compared with the SMILE [12] along with three strong competitors in [12] : i) Sequential Leader Clustering (SLC) [37] : A classical clustering technique suitable for sequential data. ii) Ranking Statistics (RankStat) [38] : RankStat utilizes the top-3 indices of feature embeddings as category descriptors. iii) Winner-take-all (WTA) [39] : WTA employs indices of maximum values within feature groups as category descriptors. These three strong baselines are set following SMILE [12] , and detailed implementation can be found in the Appendix. A.3.We conduct comparison experiments with the aforementioned competitors across eight datasets. The experimental results are reported in Table 2 . It is evident that the proposed PHE outperforms all state-of-the-art competitors across nearly all metrics. In particular, when compared with three strong baselines-SLC, RankStat, and WTA-our method demonstrates significant improvements. Additionally, our method surpasses the top competitor, SMILE, by an average of 5.4% in All classes accuracy on four common fine-grained datasets, and by an average of 5.1% in All classes accuracy on the four challenging datasets from the iNaturalist dataset. We achieve an average improvement of 11.3% on Old classes across the eight datasets compared to SMILE, demonstrating the effectiveness of our method in alleviating the "high sensitivity" issue of hash-based OCD methods. Importantly, our method consistently improves accuracy for unseen/new classes compared to SMILE. For instance, we achieve a 4.1% improvement on CUB-200 and a 4.4% improvement on the Oxford Pets dataset for new classes. This demonstrates that our method exhibits stronger generalization capabilities to new classes compared to SMILE. This advantage becomes more pronounced as the dimensionality L of hash features increases, as discussed in Sec. 4.4.Components Ablation. We report an ablation analysis of the proposed components in our PHE on CUB dataset and Stanford Cars dataset, as shown in Table 3 . "Without L p " refers to the removal of the representation learning in CPG, where we use randomly initialized vectors which are further mapped to hash centers. This configuration results in an average reduction of 1.95% in All classes accuracy across the two datasets. This variant of PHE shares the same architecture as SMILE, achieving clear decline on both datasets, demonstrating the effectiveness of our hash center-based category encoding method. It is noteworthy that removing L c causes a significant performance reduction, especially in seen categories-for instance, a 12.4% decrease in CUB. This decline is due to the lack of L c , which results in insufficient separation of hash centers, causing multiple old classes to share the same hash encoding and thereby being classified as the same. Additionally, removing L p also leads to performance degradation. This is because the hash encodings of features are not sufficiently close to the hash centers, resulting in inadequate learning of hash features.Strategy Ablation. In Table 4 , we evaluate the training strategies of our method. "Fixed-h" refers to the use of handcrafted hash points that satisfy Eq. 7. Although this design maintains separation between hash centers, it may alter the relationships between categories learned in CPG, leading to sub-optimal outcomes. "Linear Cls." and "Supcon Cls." represent the use of simple linear classification and supervised contrastive learning classification methods for representation learning, respectively, where "Fixed-h" is also applied due to the absence of category prototypes. Although these variants perform well on seen categories, their generalization capabilities are inferior to our full prototype-based method. This demonstrates the importance of integrating prototype learning to enhance generalization across unseen categories.Evaluation on Hash Code Length. The hash code length L is crucial for category inference as it directly determines the size of the prediction space, which equals to 2 L . A larger L value results in a greater number of category encodings, making the "high sensitivity" issue more severe. We evaluate our method and SMILE with different hash code lengths in Table 5 . When small changes occur to L (from 12 to 16), SMILE demonstrates stable results. However, with L = 32, SMILE experiences a decrease in average all-classes accuracy by 5.1% on two datasets, and with L = 64, the decrease is by 10.2% averaged on the two datasets. In contrast, our method, which employs hash center-based optimization and a Hamming ball-based inference process, effectively mitigates the "high sensitivity" issue and maintains stable performance as L increases. Furthermore, when L is increased from 16 to 64, the estimated number of classes by SMILE increased by 1986 on the CUB dataset and 3892 on the Stanford Cars dataset, underscoring the impact of "high sensitivity" of hash-form category descriptors on accuracy. Conversely, our method exhibits remarkable stability.Visualization Analysis. We use an images from the support set whose latent representations is most similar to p j as the visualization for p j . During on-the-fly inference, the hash code functions as the category descriptor. Additionally, the learned prototypes allow us to visually analyze why the model categorizes certain samples into known or unknown categories. Fig. 3 illustrates this reasoning process from the prototype perspective. On the left, an image labeled in green depicts a Le Conte Sparrow, a category recognized during training. Under it, a red-labeled image represents a Grasshopper Sparrow, a new category. In the center, we display the visualization of the prototypes, showing only five per class. The top five prototypes most similar to the green-labeled image are exclusively associated with the Le Conte Sparrow category, while those closest to the red-labeled, unseen image span two different categories. The Grasshopper Sparrow exhibits significant body stripe similarities to the Le Conte Sparrow and shares head similarities with the Horned Lark. Upon calculating similarities with the top five activated prototypes, the similarity score for the green-labeled image to the Le Conte Sparrow category is 9.0, whereas the unseen Grasshopper Sparrow achieves a similarity of 3.32 with the Le Conte Sparrow and 3.54 with the Horned Lark. This reasoning process is similar to how humans cognitively recognize new species. Given a new species, we use the characteristics of known categories to describe the unknown new category. Consequently, an entity that exhibits high similarity to multiple known categories, rather than only one known category, is likely to be classified as a previously unseen species.Hyper-parameters Analysis. 1) The impact of the ratios α and β in Eq. 10 is illustrated in Fig. 4 . We use All classes accuracy as the evaluation metric. α and β control the relative importance of L c and L f during the training process, respectively. A lower value of α is found to be preferable, as higher values can lead to excessive changes in hash centers, thereby affecting training stability. Regarding β, the results are generally stable; however, higher values show improved performance across both datasets. Consequently, we selected α = 0.1 and β = 3 for all datasets during training.2) Additionally, we examine the impact of the number of prototypes per class on the CUB and Stanford Cars datasets, as shown in Fig. 4 . When only one prototype per class is used, performance is suboptimal, as it fails to effectively represent the complexity of a category. For instance, a single bird species might exhibit different behaviors, such as flying or standing, that are not adequately captured by a single prototype. More prototypes allow for a better expression of the nuances within a category, which is crucial in fine-grained classification. Therefore, we opt to utilize 10 prototypes per class across all datasets.Hash Centers Analysis. We conduct an analysis of the distribution of hash centers before and after training to further evaluate the impact of the proposed hash center optimization loss, L c . Specifically, we analyze the Hamming distances between hash centers on the CUB dataset. As depicted in Fig. 5 , prior to training, the hash centers, derived from randomly initialized prototypes, are distributed relatively uniformly. Notably, some centers exhibit a Hamming distance of zero, indicating multiple centers sharing a single hash code. After training, the Hamming distance between all hash centers is at least d max . This significant improvement demonstrates the effectiveness of L c in ensuring that multiple categories do not share identical hash codes or reside excessively close to one another. 
conclusion: 
 In this paper, we introduce a Prototypical Hash Encoding (PHE) framework for fine-grained On-thefly Category Discovery. Addressing the limitations of existing methods, which struggle with the high sensitivity of hash-form category descriptors and suboptimal feature representation, our approach incorporates a prototype-based classification model. This model facilitates robust representation learning by developing multiple prototypes for each fine-grained category. We then map these category prototypes to corresponding hash centers, optimizing image hash features to align closely with these centers, thereby achieving intra-class compactness. Additionally, we enhance inter-class separation by maximizing the distance between hash centers, guided by the Gilbert-Varshamov bound. Experiments on eight fine-grained datasets demonstrate that our method outperforms previous methods by a large margin. Moreover, a visualization study is provided to understand the underlying mechanism of our method. 6 , our method is evaluated across multiple benchmarks. We have introduced the iNaturalist 2017 [35] dataset to the On-the-Fly Category Discovery (OCD) task, demonstrating our method's effectiveness in handling challenging fine-grained datasets. The iNaturalist 2017 dataset, sourced from the citizen science website iNaturalist, consists of 675,170 training and validation images across 5,089 natural fine-grained categories, which include Plantae (plants), Insecta (insects), Aves (birds), and Mammalia (mammals), among others, spread across 13 super-categories. These super-categories exhibit significant intra-category variations, making each a challenging dataset for fine-grained classification. For our evaluation, we selected Fungi, Arachnida, Animalia, and Mollusca as the four super-categories. Following the protocol established in OCD [12] , the categories within each dataset are divided into subsets of seen and unseen categories. Specifically, 50% of the samples from the seen categories are used to form the labeled training set D S , while the remainder forms the unlabeled set D Q for on-the-fly testing. Evaluation Metric Details. We employ clustering accuracy, specifically the Strict-Hungarian method as outlined in OCD [12] , as our evaluation protocol, a common choice in NCD/GCD tasks. Clusters are ranked by size, and only the top-|Y Q | clusters are retained; others are deemed misclassified. It is crucial to outline our accuracy calculation for old and new classes. We perform the Hungarian assignment once for all categories Y Q , and subsequently measure classification accuracy for the "Old" and "New" subsets. This setup follows the protocols in OCD [12] and GCD [12] . 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: Please refer to Appendix. C.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: The scope of our research does not involve data or models with a high risk for misuse. We utilize publicly available, non-sensitive datasets and models that do not require special safeguards for responsible release. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: All third-party assets used in our research, including datasets, and models, are properly credited in the manuscript. We have explicitly mentioned the licenses and terms of use for each asset, ensuring that all conditions are fully respected. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.• For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification: This question does not apply to our research as no new assets were introduced.Our study solely relies on existing, publicly available resources, and thus, no additional documentation for new assets is necessary.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: This question does not apply to our research as it does not involve crowdsourcing experiments or research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: This question is not applicable to our research as no human subjects were involved.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Model Training Details. We employ the AdamW optimizer during training, using a learning rate of 1e-4 for the backbone and 1e-3 for the projection head and prototype layer, with a weight decay of 0.05. We train the model for 200 epochs. The batch size is uniformly set at 128 across all datasets for consistent comparisons with leading methods. Training incorporates the use of Exponential Moving Average (EMA). Experiments were conducted using Tesla V100 and 1080Ti GPUs, with results reported as the mean of three runs.Calculated d max . The value of d max is calculated using Eq. 7 for all datasets and hash code lengths L.The specific values of d max are reported as follows: The setup for the comparative experiments follows OCD [12] . Detailed information is provided as follows: Ranking Statistics (RankStat). [38] AutoNovel utilizes Ranking Statistics to analyze sample relationships, particularly by using the top-3 indices of feature embeddings as category descriptors. This method aligns with the On-the-Fly Category Discovery (OCD) settings and poses a strong challenge to hash-based descriptors. For a fair comparison, we use the same backbone (DINO-ViT-B-16) for Ranking Statistics (RS) and retain only the fully-supervised learning stages, as no additional data can be used in the On-the-Fly Category Discovery (OCD) task. The embedding dimension is set to 32, resulting in a prediction space of C 3 32 = 4, 906, comparable to our method and SMILE, which uses a hash code length of L = 12 and achieves a prediction space of 2 12 = 4, 096. Winner-take-all (WTA). [39] To address potential biases towards salient features by Ranking Statistics, Winner-take-all (WTA) hash was proposed as an alternative. WTA avoids reliance on the global order of feature embeddings and instead utilizes indices of maximum values within divided feature groups. With a 48-dimension embedding divided into three groups, WTA generates a prediction space of 16 3 = 4096, ensuring comparability for fair assessment. Sequential Leader Clustering (SLC). [37] We employ the same backbone and conducts fully supervised training on the support set for SLC. For on-the-fly testing, SLC utilizes features extracted from the backbone on the query set. We optimized the hyperparameters based on the CUB dataset, applying these uniformly across other datasets to maintain consistency in our comparisons.We have detailed the description of our PHE framework and the Hamming Ball Based Model Inference in the main text. The pseudo-code for the Hamming Ball Based Model Inference is provided in Algorithm 1, and the training process of our PHE framework is shown in Algorithm 2. We analyze the performance of the SMILE method [12] with different feature dimensions in Table 10 . In the table, "Offline" and "Clustering Acc" refer to the clustering accuracy obtained by applying k-means clustering on the features of all test data, given a fixed number of categories (200 for CUB). From the table, it is evident that as the feature dimension L decreases, the on-the-fly prediction accuracy significantly improves. This is because the "high sensitivity" issue of hash-form category descriptors becomes more pronounced at higher feature dimensions. On the other hand, as the feature dimension decreases, there is a notable reduction in offline clustering accuracy, with a decrease of 9.9% when reducing L from 256 to 12. This indicates that when the feature dimension is too low, it inevitably damages the model's ability to distinguish categories, resulting in poorer feature representation learning. Therefore, our PHE framework separates feature learning from category representation learning, which is one of the reasons why our PHE framework performs better. 11 , demonstrate the effectiveness of our PHE. Traditional methods like DPN, OrthoHash, and CSQ generate suboptimal hash centers, primarily designed for retrieval tasks with a focus on instancelevel discrimination. Unlike these methods, our PHE incorporates the Gilbert-Varshamov bound to guide the learning of hash centers, specifically for category discovery. This strategy ensures that the hash centers are not only discriminable but also preserve the rich category-specific information inherent in category-level prototypes. Furthermore, our PHE outperforms MDSH, which optimizes hash codes to align with pre-defined static hash centers. In contrast, our approach dynamically maps hash centers from each category's prototypes, continuously refining these through end-to-end optimization. Additionally, our design incorporates a Hamming-ball-based inference mechanism, which significantly reduces hash sensitivity. For instance, when the hash code length L is set to 32, the MDSH-Hamming ball configuration outperforms the standard MDSH by an average margin of 5.9% across the two datasets.The computational cost of PHE is relatively low and primarily depends on the number of known categories, rather than directly correlating with the scale of the dataset. Specifically, each known category is associated with a hash center, with hash centers h of shape [class nums, code length].The main computational operations involve straightforward two-dimensional matrix multiplications. These include the dot product of features b, with a shape [batch size, code length], and hash centers, as well as the dot product operations used in calculating Hamming distances between hash centers. We provide a comparison of training times between our PHE and the state-of-the-art method, SMILE, with results shown in Table 13 . To ensure fairness, all experiments are conducted on an NVIDIA RTX A6000 GPU, with both algorithms trained over 200 epochs using mixed precision. The dataloader parameters are kept consistent across tests, with a batch size of 128. According to the table, our average training time across four datasets is 45.8 minutes shorter than that of SMILE. This improvement is primarily due to the higher computational demands of SMILE's supervised contrastive learning approach, which processes two different views of samples for representation learning. We conduct experiments using different proportions of old category selection on the CUB and Stanford Cars datasets, as shown in Table 14 , with all accuracy results reported and hash code length L set to 12. Our PHE outperforms SMILE by an average of 5.95% when 75% of old categories are selected and by 1.0% when 25% are selected. This indicates that our PHE better models the nuanced inter-category relationships in fine-grained category divisions as the number of categories increases.We conduct comparative experiments using the powerful prototype learning method SimGCD [13] , based on our category encoding method. Due to the lack of unlabeled data, we remove the unsupervised loss component, L u cls , and the prototypes corresponding to new categories in SimGCD. As shown in Table 15 , our use of SimGCD for prototype learning, mapping the prototypes learned by SimGCD to hash centers for category encoding, yields very poor results on both datasets. "SimGCD-MC", which employs manually obtained centers satisfying the Gilbert-Varshamov bound along with features from the SimGCD projection head, shows an average improvement of 8.8% across the two datasets. This performance boost demonstrates that the prototypes learned by SimGCD are not suitable for category encoding in fine-grained scenarios. Our PHE offers two main advantages. First, unlike SimGCD, which learns only one prototype per category, our PHE generates multiple prototypes per class, effectively modeling intra-class variance of fine-grained categories. Second, unlike the prototypes in SimGCD, which serve as classifier weights, the prototypes in PHE can be explicitly visualized, providing additional insights into the model's behavior. Broader Impact. The On-the-fly Category Discovery (OCD) task is designed to learn category differences from observed categories and then make real-time predictions across a broader range, including unknown categories. Our proposed Prototypical Hash Encoding (PHE) method for the OCD task holds potential for application in open-world scenarios. For instance, it can assist in botanical classification, where new plant species are discovered and need to be quickly integrated into existing categories without retraining the entire system.Limitations. Despite the superior performance of our Prototypical Hash Encoding (PHE) framework compared to existing methods, it still requires further research to enhance the accuracy of recognizing unknown categories. Current On-the-fly Category Discovery (OCD) methods generally achieve low accuracy in predicting new categories, as these categories were not encountered during training, posing significant challenges to effective generalization. Future research should focus on improving the model's ability to recognize and categorize new, unseen categories.A Possible Solution in Future Work. Due to the constraints of training data in the OCD setting, we are considering the integration of additional knowledge from pre-trained Large Language Models (LLMs). First, we can utilize LLMs to establish a bank of category attribute prototypes, which are expected to be relevant across both known and unknown categories. Subsequently, during the real-time prediction process, we plan to employ LLMs and Vision-Language Pretrained Models (VLMs) to match these attribute prototypes with unknown categories. Ultimately, by integrating both instance and attribute features, we aim for our PHE to generate more precise predictions.We have conducted additional visualization analysis with images from the CUB dataset and Stanford Cars dataset, as shown in Fig. 6 to 11. As discussed in Sec. 4.4, in addition to using hash codes to represent categroies, our PHE framework allows for visual analysis from the perspective of prototype similarity to understand why certain images are identified as new classes, as depicted in Fig. 6 , 7, 9, and 10. Furthermore, even when images are misclassified, such as assigning an image from a new category to an old category as shown in Fig. 8 and Fig. 11 , we can still gain interesting insights from the prototype perspective. Firstly, these images from new categories indeed exhibit high similarity to the category where the activated prototypes belong to. Secondly, although the images from new categories show high similarity to old ones, the computed similarity is relatively lower compared to samples that truly belong to that old category. For example, in Fig. 8 , the similarity values are 5.48 versus 9.14 for the misclassified new category and the true sample of the old category, respectively.... • The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. Justification: Our research adheres to all aspects of the NeurIPS Code of Ethics, ensuring compliance with ethical standards throughout the study.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 


Decision: This paper addresses the On-the-fly Category Discovery (OCD) task, which involves utilizing existing category knowledge to recognize both known and unknown categories in new data streams in real-time. A novel framework has been proposed to address this task. In general all the reviewers are positive about the contribution and performance of the proposed work. Thus, the recommendation is to accept the paper. The authors are strongly encouraged to address the comments in the camera-ready version.

Reviews: review: paper_summary: paper_summary: This paper focuses on On-the-fly Category Discovery (OCD), which is to determine novel categories during inference. OCD methods first compute the hash code of the image, this code then becomes a "cluster index", if not matching with existing code, then it is a novel category.
However, the problem with the previous OCD method (SMILE) is that it becomes highly sensitive when the code length increases (difficult to match a new category due to the high possible combinations of hash code). 
Instead of contrastive learning in SMILE, this paper uses cross-entropy loss adapted with a minimal hash distance scheme as regularization to improve performance. Further, the paper also adds cross entropy loss in the feature space to avoid information loss when compressing features into hash code, the feature representation used is a prototype-based model (e.g., ProtoPFormer). 
Overall, it enhances SMILE by using better loss functions and different representations.
strength_weakness: strength_weakness: Comparison to deep hashing based methods are missing. There are many parts of the designed components similar to deep hashing such as the minimal hash distance formulation is adapted directly from [1]. Part of the loss objective is similar to cosine-similarity-based hashing methods like [2]. Another simple baseline is using ProtoPFormer + Deep Hashing method. From this paper, we do not know whether a naive application of deep hashing-based method is sufficient hence we cannot verify the novelty and effectiveness of the proposed method.

[1] Liangdao Wang et al. Deep Hashing with Minimal-Distance-Separated Hash Centers. CVPR 2023. 
[2] Hoe et al. One Loss for All: Deep Hashing with a Single Cosine Similarity based Learning Objective. NeurIPS 2021.
questions: questions: In terms of the idea, I have no question, I believe the idea is simple and clever use of existing components. However, the lack of comparison to deep hashing methods makes me doubt the novelty and effectiveness of the proposed method/modules.
limitations: limitations: The author did mention limitations in the appendix which I also agree with.
paper_summary: paper_summary: This paper addresses the On-the-fly Category Discovery (OCD) task, which involves utilizing existing category knowledge to recognize both known and unknown categories in new data streams in real-time. To tackle the high sensitivity and suboptimal feature representation issues of existing methods when dealing with fine-grained categories, the paper proposes an innovative Prototypical Hash Encoding (PHE) framework. This framework uses a Category-aware Prototype Generation (CPG) module to represent each fine-grained category with multiple prototypes and employs a probabilistic masking strategy to encourage the model to fully capture intra-class diversity. The Discriminative Category Encoding (DCE) module maps the generated category prototypes to low-dimensional hash centers, optimizing image hash features to ensure intra-class compactness and inter-class separation. Additionally, a center separation loss function is designed to maintain a minimum Hamming distance between different category hash centers. Experimental results on multiple datasets confirm the superiority of this method.
strength_weakness: strength_weakness: Although the proposed Prototype Hashing Encoding (PHE) framework outperforms existing methods in terms of performance, further research is needed to improve its accuracy in recognizing unknown categories. Additionally, due to the need to compute multiple hash centers and perform complex distance calculations, the computational cost of PHE is relatively high, especially on large-scale datasets.
questions: questions: Apart from the issues mentioned in weakness part, there are some additional questions regarding implementation details:
1. The article mentions that to encourage the model to fully capture intra-class diversity, K categories are equally allocated to each class. I would like to know how significant the choice of K is on the model's performance, and if there are experiments demonstrating the impact of different K values on the model.
2. Similarly, when using the masking strategy, how does the variation in the value of θ affect the training and final performance of the model?
3. What about the performance of different splits of old and new class, further analysis is needed regarding the size of different splits as well as whether the correlation of old and new classes essentially influence the final performance etc. Some in-depth insights are expected.
limitations: limitations: As mentioned in the appendix of the manuscript, although the proposed PHE framework outperforms existing methods in terms of performance, further research is needed to improve its accuracy in recognizing unknown categories. Additionally, the PHE method involves multiple hyperparameters that require careful tuning to achieve optimal performance, increasing the complexity of model debugging and optimization. Moreover, the weights of the component loss functions in the total loss function need to be appropriately set to balance the optimization objectives of different components, which may require extensive experiments to find the suitable weight combination.
paper_summary: paper_summary: This paper introduces a novel framework called Prototypical Hash Encoding (PHE) for On-the-fly Category Discovery (OCD), which aims to discover both known and unknown categories from streaming data using labeled data of known categories. PHE first learns many prototypes for each category and then maps the learned prototypes to hash codes to distinguish samples from known or novel categories with a threshold.
strength_weakness: strength_weakness: 1. The proposed method is an improvement from the previous work [1], sharing the same core idea of utilizing hash codes for OCD, which limits the novelty of the paper.
2. Despite the improved performance, the reasons why prototypes can address the sensitivity issue are not analyzed in depth.
3. Compared with feature-level prototypes, the advantages of hash code-based category prototypes have not been demonstrated or experimentally verified.

[1] Ruoyi Du, Dongliang Chang, Kongming Liang, Timothy Hospedales, Yi-Zhe Song, and Zhanyu Ma. On-the-fly category discovery. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11691–11700, 2023.
questions: questions: 1. Compared with feature-level prototypes or prototypes after dimensionality reduction, what are the advantages of hash code-based prototypes? 
2. How is the training efficiency of the model compared to SMILE?
3. How are hyperparameters selected without the validation set?
limitations: limitations: N/A
paper_summary: paper_summary: This paper proposes a prototypical deep hashing framework to address the fine-grained on-the-fly category discovery problem. The proposed method includes two main loss functions: first, distance minimization between the encoded hash features to the category-representative hash coding after projection $\mathcal{H}_h$; second, enforcing minimal distance separation among hash encoding of different categories, which are enforced to be quantized to -1 and +1 for each dimension. The experimental results demonstrate the effectiveness of each loss function. The performance gains are considerable on all benchmarks compared with previous state-of-the-art baselines.
strength_weakness: strength_weakness: - Writing: It is informal to leave all the related works in the appendix, which will confuse the readers on the contribution of this work. Besides, some expressions are not
- Novelty: The technical contribution is limited in that the prototypical learning [3] is a mature practice in category discovery and the author seems to adapt the deep hashing method in [4] to this problem.
- Motivation: The motivation of achieving the balance between instance discrimination and class discrimination, especially with prototypical learning in the category discovery field is not new [2,3]. However, this is accepted to some extent since on-the-fly category discovery is a new problem.
questions: questions: - It would be clearer if the author could mention the technical practice of prototype-based visualization examples, though it may be proposed by the previous work, ProtopFormer.
- In Table 3, how will the semi-supervised contrastive loss perform [1] in addition to the supcon and unsupcon baselines? These results matter to solidify the motivation for using the deep hashing framework.


[1] Generalized Category Discovery

[2] PromptCAL: Contrastive Affinity Learning via Auxiliary Prompts for Generalized Novel Category Discovery

[3] Parametric Classification for Generalized Category Discovery: A Baseline Study

[4] Deep hashing with minimal-distance-separated hash centers
limitations: limitations: As mentioned by the author, on-the-fly category discovery is a novel and challenging problem, therefore existing methods do not achieve satisfying performance on real-world tasks.

Comments: title: recommendation
comment: Dear Reviewer 6xA4, 

We really appreciate your valuable comments! We will add the important discussions mentioned above in the final manuscript and highlight them. 

Thank you once again for your efforts and contributions!
thank you for your response. the experiments now look more complete, as long as the author adds them into the final revision, I have no further questions. I am recommending a weak accept (6).
Dear Reviewer Pa9u,

We greatly appreciate your satisfaction with our responses! We will include the important discussions mentioned above in the final manuscript and highlight them. We truly appreciate your efforts!
Authors have partly addressed my concerns regarding model's stability and split variation, I'm willing to raise my score by one.
Dear Reviewer ZQiT,

Thank you for your positive feedback and for increasing the rating! We will include the important discussions you mentioned in the final manuscript.

We truly enjoy our interactions and greatly appreciate all your efforts!
Thanks for your reply, my concerns and questions have been partly addressed. I will raise my score by one.
Dear Reviewer dwc8,

We greatly appreciate your satisfaction with our responses, and very glad you increase the rating! We will add the above important discussions in the final manuscript and highlight them.

Thanks again for your valuable suggestions and comments. We enjoy communicating with you and appreciate your efforts!
Thanks for the reply. Most of my questions are well addressed. Although the author clarified their contributions and differences compared with previous works, my concern about the technical novelty and contribution of this work has been partly addressed. After rebalancing its weaknesses and strengths, I decide to raise my score by one.