Title: Prototypical Hash Encoding for On-the-Fly Fine-Grained Category Discovery
Venue: NeurIPS 2024
Accepted: yes
Keywords: Category Discovery

On-the-fly

Deep Hash
Abstract: In this paper, we study a practical yet challenging task, On-the-fly Category Discovery (OCD), aiming to online discover the newly-coming stream data that belong to both known and unknown classes, by leveraging only known category knowledge contained in labeled data. Previous OCD methods employ the hash-based technique to represent old/new categories by hash codes for instance-wise inference. However, directly mapping features into low-dimensional hash space not only inevitably damages the ability to distinguish classes and but also causes ``high sensitivity'' issue, especially for fine-grained classes, leading to inferior performance. To address these drawbacks, we propose a novel Prototypical Hash Encoding (PHE) framework consisting of Category-aware Prototype Generation (CPG) and Discriminative Category Encoding (DCE) to mitigate the sensitivity of hash code while preserving rich discriminative information contained in high-dimension feature space, in a two-stage projection fashion. CPG enables the model to fully capture the intra-category diversity by representing each category with multiple prototypes. DCE boosts the discrimination ability of hash code with the guidance of the generated category prototypes and the constraint of minimum separation distance. By jointly optimizing CPG and DCE, we demonstrate that these two components are mutually beneficial towards an effective OCD. Extensive experiments show the significant superiority of our PHE over previous methods, e.g. obtaining an improvement of +5.3% in ALL ACC averaged on all datasets. Moreover, due to the nature of the interpretable prototypes, we visually analyze the underlying mechanism of how PHE helps group certain samples into either known or unknown categories. Code is available at https://github.com/HaiyangZheng/PHE.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a prototypical deep hashing framework to address the fine-grained on-the-fly category discovery problem. The proposed method includes two main loss functions: first, distance minimization between the encoded hash features to the category-representative hash coding after projection $\\mathcal{H}_h$; second, enforcing minimal distance separation among hash encoding of different categories, which are enforced to be quantized to -1 and +1 for each dimension. The experimental results demonstrate the effectiveness of each loss function. The performance gains are considerable on all benchmarks compared with previous state-of-the-art baselines.\n\nSTRENGTHS:\n- The performance gains of the proposed approach are considerable.\n- The proposed method is robust to the varying coding length.\n- The identified ‘sensitivity issue’ problem exists for on-the-fly category discovery.\n\nWEAKNESSES:\n- Writing: It is informal to leave all the related works in the appendix, which will confuse the readers on the contribution of this work. Besides, some expressions are not\n- Novelty: The technical contribution is limited in that the prototypical learning [3] is a mature practice in category discovery and the author seems to adapt the deep hashing method in [4] to this problem.\n- Motivation: The motivation of achieving the balance between instance discrimination and class discrimination, especially with prototypical learning in the category discovery field is not new [2,3]. However, this is accepted to some extent since on-the-fly category discovery is a new problem.\n\nQUESTIONS:\n- It would be clearer if the author could mention the technical practice of prototype-based visualization examples, though it may be proposed by the previous work, ProtopFormer.\n- In Table 3, how will the semi-supervised contrastive loss perform [1] in addition to the supcon and unsupcon baselines? These results matter to solidify the motivation for using the deep hashing framework.\n\n\n[1] Generalized Category Discovery\n\n[2] PromptCAL: Contrastive Affinity Learning via Auxiliary Prompts for Generalized Novel Category Discovery\n\n[3] Parametric Classification for Generalized Category Discovery: A Baseline Study\n\n[4] Deep hashing with minimal-distance-separated hash centers\n\nLIMITATIONS:\nAs mentioned by the author, on-the-fly category discovery is a novel and challenging problem, therefore existing methods do not achieve satisfying performance on real-world tasks.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-24"}

comments: SUMMARY:
This paper introduces a novel framework called Prototypical Hash Encoding (PHE) for On-the-fly Category Discovery (OCD), which aims to discover both known and unknown categories from streaming data using labeled data of known categories. PHE first learns many prototypes for each category and then maps the learned prototypes to hash codes to distinguish samples from known or novel categories with a threshold.

STRENGTHS:
1. The paper is well-written and easy to follow.
2. The proposed method addresses the limitations of previous hash-based OCD models by reducing sensitivity and preserving discriminative information through prototypes.
3. The proposed method achieves significant improvements in accuracy across various datasets.

WEAKNESSES:
1. The proposed method is an improvement from the previous work [1], sharing the same core idea of utilizing hash codes for OCD, which limits the novelty of the paper.
2. Despite the improved performance, the reasons why prototypes can address the sensitivity issue are not analyzed in depth.
3. Compared with feature-level prototypes, the advantages of hash code-based category prototypes have not been demonstrated or experimentally verified.

[1] Ruoyi Du, Dongliang Chang, Kongming Liang, Timothy Hospedales, Yi-Zhe Song, and Zhanyu Ma. On-the-fly category discovery. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11691–11700, 2023.

QUESTIONS:
1. Compared with feature-level prototypes or prototypes after dimensionality reduction, what are the advantages of hash code-based prototypes? 
2. How is the training efficiency of the model compared to SMILE?
3. How are hyperparameters selected without the validation set?

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the On-the-fly Category Discovery (OCD) task, which involves utilizing existing category knowledge to recognize both known and unknown categories in new data streams in real-time. To tackle the high sensitivity and suboptimal feature representation issues of existing methods when dealing with fine-grained categories, the paper proposes an innovative Prototypical Hash Encoding (PHE) framework. This framework uses a Category-aware Prototype Generation (CPG) module to represent each fine-grained category with multiple prototypes and employs a probabilistic masking strategy to encourage the model to fully capture intra-class diversity. The Discriminative Category Encoding (DCE) module maps the generated category prototypes to low-dimensional hash centers, optimizing image hash features to ensure intra-class compactness and inter-class separation. Additionally, a center separation loss function is designed to maintain a minimum Hamming distance between different category hash centers. Experimental results on multiple datasets confirm the superiority of this method.\n\nSTRENGTHS:\nThe proposed PHE method demonstrates its superiority over existing state-of-the-art methods across multiple datasets. It effectively addresses the issues of large intra-class variance and small inter-class variance, while minimizing the information loss associated with dimensionality reduction. The paper employs visualization techniques to analyze the underlying mechanism by which PHE groups samples into known or unknown categories. Moreover, due to its optimization based on hash centers and inference process based on Hamming balls, the PHE method shows stable performance across different hash code lengths, effectively mitigating the \"high sensitivity\" problem. In contrast to the SMILE method, which exhibits significant accuracy degradation and instability with increasing hash code lengths, the PHE method maintains remarkable stability and consistency.\n\nWEAKNESSES:\nAlthough the proposed Prototype Hashing Encoding (PHE) framework outperforms existing methods in terms of performance, further research is needed to improve its accuracy in recognizing unknown categories. Additionally, due to the need to compute multiple hash centers and perform complex distance calculations, the computational cost of PHE is relatively high, especially on large-scale datasets.\n\nQUESTIONS:\nApart from the issues mentioned in weakness part, there are some additional questions regarding implementation details:\n1. The article mentions that to encourage the model to fully capture intra-class diversity, K categories are equally allocated to each class. I would like to know how significant the choice of K is on the model's performance, and if there are experiments demonstrating the impact of different K values on the model.\n2. Similarly, when using the masking strategy, how does the variation in the value of θ affect the training and final performance of the model?\n3. What about the performance of different splits of old and new class, further analysis is needed regarding the size of different splits as well as whether the correlation of old and new classes essentially influence the final performance etc. Some in-depth insights are expected.\n\nLIMITATIONS:\nAs mentioned in the appendix of the manuscript, although the proposed PHE framework outperforms existing methods in terms of performance, further research is needed to improve its accuracy in recognizing unknown categories. Additionally, the PHE method involves multiple hyperparameters that require careful tuning to achieve optimal performance, increasing the complexity of model debugging and optimization. Moreover, the weights of the component loss functions in the total loss function need to be appropriately set to balance the optimization objectives of different components, which may require extensive experiments to find the suitable weight combination.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on On-the-fly Category Discovery (OCD), which is to determine novel categories during inference. OCD methods first compute the hash code of the image, this code then becomes a \"cluster index\", if not matching with existing code, then it is a novel category.\nHowever, the problem with the previous OCD method (SMILE) is that it becomes highly sensitive when the code length increases (difficult to match a new category due to the high possible combinations of hash code). \nInstead of contrastive learning in SMILE, this paper uses cross-entropy loss adapted with a minimal hash distance scheme as regularization to improve performance. Further, the paper also adds cross entropy loss in the feature space to avoid information loss when compressing features into hash code, the feature representation used is a prototype-based model (e.g., ProtoPFormer). \nOverall, it enhances SMILE by using better loss functions and different representations.\n\nSTRENGTHS:\noriginality: the idea itself is clever and simple, propose/reuse some modules to mitigate the problem in SMILE (to reduce the sensitivity problem). the hamming ball-based inference utilizes the benefit of GV bound.\n\nquality: analyzed the proposed method with many experiments.\n\nclarity: the paper is well-written\n\nsignificance: somehow significant as OCD is more practical in open-world scenario, and may help improve retrieval systems.\n\nthe one insight I got from this paper is that using hash code alone is sensitive when code length increases so it cannot be used for cluster index easily, and we need to make the hash code as compact as possible to reduce sensitivity (i.e., the same class must have the same hash code, by guiding the hash code into center, although already shown in multiple deep hashing papers such as CSQ [1], DPN [2], OrthoHash [3], [4]).\n\n[1] Li Yuan et al. Central Similarity Quantization for Efficient Image and Video Retrieval. CVPR 2020.\n[2] Lixin Fan et al. Deep Polarized Network for Supervised Learning of Accurate Binary Hashing Codes. IJCAI 2020.\n[3] Jiun Tian Hoe et al. One Loss for All: Deep Hashing with a Single Cosine Similarity based Learning Objective. NeurIPS 2021\n[4] Liangdao Wang et al. Deep Hashing with Minimal-Distance-Separated Hash Centers. CVPR 2023.\n\nWEAKNESSES:\nComparison to deep hashing based methods are missing. There are many parts of the designed components similar to deep hashing such as the minimal hash distance formulation is adapted directly from [1]. Part of the loss objective is similar to cosine-similarity-based hashing methods like [2]. Another simple baseline is using ProtoPFormer + Deep Hashing method. From this paper, we do not know whether a naive application of deep hashing-based method is sufficient hence we cannot verify the novelty and effectiveness of the proposed method.\n\n[1] Liangdao Wang et al. Deep Hashing with Minimal-Distance-Separated Hash Centers. CVPR 2023. \n[2] Hoe et al. One Loss for All: Deep Hashing with a Single Cosine Similarity based Learning Objective. NeurIPS 2021.\n\nQUESTIONS:\nIn terms of the idea, I have no question, I believe the idea is simple and clever use of existing components. However, the lack of comparison to deep hashing methods makes me doubt the novelty and effectiveness of the proposed method/modules.\n\nLIMITATIONS:\nThe author did mention limitations in the appendix which I also agree with.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-13"}