Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: Authors propose a continuous setting of GCD called CCD where unlabeled data from novel and known categories appear in batches. And they also propose a framework/method to iteratively update representation & novel category discovery for CCD. In proposed framework, authors first perform novelty detection and discover novel categories & update representation on batch of unlabeled data via pairwise similarity of representation & regularize via static branch. Then based on estimate of current cluster assignments, they remove outliers of clusters to update exemplars and retrain representation with pseudo labels to merge categories, and finally merge static and dynamic representation with a moving avg. strategy.

strengths_and_weaknesses: Key contributions of paper is Proposal of CCD which useful continuous adaptation variation of GCD. And they propose a framework to address the setup and alleviate challenges of catastrophic forgetting, noisy gradient information available in CCD unlike GCD.

Strengths:
	Addresses an important problem of interest of practical value.
	Proposed framework could potentially be applied with various techniques.
	Multiple settings of experiments for evaluations and good analysis in experiments.

Weakness
	Lack of evaluation with different techniques, especially considering a continuous version of GCD or other semi/self-supervised representation updates.
    Lack of clarity in writing few details w.r.t setting and experiments, especially what and how if any additional data is used beyond exemplars. Also it is mentioned as limitation that number of novel classes at any time is known a-priori in supplementary, to make it clear it should be stated in main paper too while defining $L_{BCE}$ and in the setting.

Suggestion to Authors: 
    Proposed grow and merge framework is a useful generic solution and is potentially agnostic to techniques used for self-supervised representational learning and to updating representation within CCD iterations. 
    To decouple effectiveness of proposed framework and get better information w.r.t dependency on exact techniques especially w.r.t online vs offline loss in performance, it could be fruitful to adopt proposed framework to some other method too? 

E.g. [1] for GCD, a follow up to AutoNovel adopted in this paper or any other method for GCD could be another upper-bound to consider? This is a suggestive example, authors may choose other method of representation learning and updates to adopt proposed framework.
		For [1] may be worthwhile to consider performing an experiment without WTA in growing phase and use techniques from GCD, with and without appropriate $L_{SD}$.
		Where for an unlabeled dataset , update representation with self-supervised instance contrastive learning and also supervised contrastive learning leveraging small dataset?, and perform semi-supervised k-means++ applied by GCD for clustering.
	        On this representation, get final performance with and without retraining/additional fine-tuning using other techniques of pseudo labels as alternative to PLL as already evaluated in ablation study at each 't'.

Also if it is assumed number of novel classes is known a priori, consider an ablation study to analyze sensitivity i.e. if our estimate of novel classes is off by some value at each time step 't' and in  addition these details could potentially be better explained by stating them explicitly in both setting and experiments.

Writing: 
	In section 2.1 it is stated only unlabeled data is available and in line 60 it is stated small dataset is available for growing phase, please be explicit for readers to get a clear understand of setting. Is it source-free or exemplar only or some small replay buffer, etc. 
	In line 43, 44 it is stated known classes classification and novelty detection would need different types of features, and this is not necessarily true or would suggest rewriting  sentence. As ideally we need more discriminative features to classify known + unknown and only training only w.r.t known may not have sufficient features to classify unknown but both need better discriminative features as indicated by [2] (closed set accuracy is all you need for open set)
	Minor corrections: Line 266, Phrase should be replaced with phase.

References:
1. Generalized Category Discovery https://openaccess.thecvf.com/content/CVPR2022/papers/Vaze_Generalized_Category_Discovery_CVPR_2022_paper.pdf 
2. OPEN-SET RECOGNITION: A GOOD CLOSED-SET CLASSIFIER IS ALL YOU NEED? https://openreview.net/pdf?id=5hLP5JY9S2d


questions: In line 60 of page 2, 'we maintain a small set of training examples for each category for both known & discovered'. Can you comment and clarify size of this small dataset stored and all the steps where this dataset is leveraged, so is this small dataset not used for merging and only in growing as stated in line 62?
limitations: Currently framework is adopted to a certain choice of self-supervised representational learning, though framework is generic its effectiveness with other techniques is to be evaluated empirically.
And as authors pointed out this framework assumes no.of.novel categories in each time-step a priori but could be addressed in future work.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper advances the task of novel category discovery to a dynamic setting where unlabelled data are continuously fed into a model for category discovery, refered as continuous category discovery.
The paper also proposed a grow and merge framework to solve this problem by balancing between novel category discovery and merging the newly discovered information with the knowledge already know to the model.

strengths_and_weaknesses: S1: The proposed experimental setting is properly defined, and could be of interest for the novel category discovery community.

S2: The proposed grow and merge framework seems to perform well on the proposed experimental setting.

W1: The overall pipeline is complex, however I think the evaluation and discussion of the results are not enough, see Questions.

W2: The implicit assumption that the number of categories is known seems to be unrealistic.

Overall, I think this paper has its merit, but the defect outweight its merit, and the presentation of the work needs to be improved.

questions: Q1: In fig. 2, the paper aim to show that the features from the known categories and the novel categories are different, however, this visualization is not quantitative and could cause a strong bias if we only visualize a small subset of the dataset. Although I think the overall observation is correct, I would suggest to use the notion like uniformity and alignment [R1] to better show the main motivation of the paper.

Q2: In the sample sifting part, I did not quite understand the reasoning of find the maximum distance between a data point and the j nearest neighbors, isn't the maximum distance between a data point and its top-j neighbors always at the j-th nearest neighbor? In other word, is this measure always depends on the parameter j? I am also wonderring how about using a averaged distance between the data point and all its top-j nearest neighbor.

Q3: The setting assumes that the number of categories at time t is known, however this is an unrealistic assumption, because the novel categories are unknown, how can we assume we know the number of categories? In previous works on novel category discovery, there are methods that can be used to estimate the number of novel categories [R2], and so it makes sense for NCD methods to assume the category is known, however in the CCD case proposed in this paper, I would suggest the paper to include discussion of how to apply the algorithm in [R2] to estimate the number of categories in each time step in the CCD setting, and also include experiments of evaluate the proposed method using the estimated category number.

Q4: The method part of this paper is quite hard to understand, because it involves concepts that are not properly defined (such as exemplars and prototypes, the paper only states it use these terms as in iCaRL makes the paper not self-contained), and the framework seems to be quite complex, I would suggest to make it clear about every concept mentioned in the paper, and perhaps also include a method diagram to better demonstrate the idea of the framework.

Q5: The framework is quite complex, involving four losses, how to balance between these losses? The paper simply add them together without weighting, it would be great if we can have a study about the weighting between the four losses.

Q6: The experiments in Table 4 seem weird, the DRNCD method performed lowest on the MI setting, but the performance on SMI setting improves a lot, this is strange, could this be a sign that the setting are not stable, so that the variance of the experiments is large? Also, I would appreciate it if the paper could provide error bars on the experiments results.


Minor:

1. The paper is not very self-contained, the term exemplars and the initialization of exemplars and prototypes are used but not defined, the paper directly cites iCaRL for this, IMO, a good paper should be self-contained, at least the definition of these terms should be in the main text.
2. Typo at line 266 phrase -> phase


[R1] Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere, ICML 2020

[R2] Learning to Discover Novel Visual Categories via Deep Transfer Clustering, ICCV 2019

limitations: The most noticable limitation of the proposed method and setting is that the number of novel categories is assumed to be konwn, I would suggest the author to try to include a discussion on this and try to expand the algorithm in [R2] for the proposed setting.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper addresses the problem of continuous category discovery, which is a natural extension of novel category discovery by sequentially feeding part of the targeting datasets into the model. The method builds on top of existing methods for novel category discovery and continual learning, namely, AutoNovel[6] and iCaRL[14], with a framework trained with grow and merge phases, containing two branches. Experiments on three public datasets have been presented showing obvious improvement over the compared methods. 
strengths_and_weaknesses: Strengths:
+ The paper tacks the problem of continuous category discovery (NCD), which jointly considers the problem of novel category discovery and continual learning. This is a natural and interesting extension of NCD. 
+ Good performance is achieved on three public datasets, outperforming the compared baselines. 

Weaknesses:
- Though effective, the key components of the method are based on AutoNovel and iCaRL. The extra branches and grow-merge phases increased the complexity and meomery&computation cost for training. 
- Experiments are only carried out on three relatively small datasets with a relatively small number of new classes in each time step. Also, the data splits for different time steps for the MI case vary a lot, which seems strange (as specified in the supplementary), why not adopt a consistent proportion for all? Or perhaps more different proportions can be used? How the different classes in each time step are constructed? randomly sampling or predefined in some way? 
Only 10 classes are incremented at each time step, how about incrementing more classes? say 20, 50? For more challenging cases, a larger dataset could be more useful. It appears that more different datasets are used in the literature, e.g, CIFAR10, ImageNet,  Stanford-Cars FGVC-Aircraft in DRNCD [15].
- The new category number is assumed to be known in each incremental time step. 
questions: Please refer to the weaknesses above. In addition, 

- Figure 4 and the illustration need to be clarified. The text is not well aligned with Fig 4, making the explanation rather unclear. For example, what are these selected clusters, what are those gray dots, which time-step is it, which are the old classes, and which are the new classes?  
- As an additional comment to the weakness of the Experiments above, it would be more convincing to repeat on different random splits -- assuming the splits used in the experiments were randomly drawn, and also report the results on each split and each time step (more suitable for supplementary, but it would be useful to show how the method actually works in the intermediate steps)
- In Table 2 and 4, why DRNCD w/ LwF results are not reported, since it outperforms AutoNovel in NCD? But only the results of AutoNovel w/ LwF are reported.
- How was novelty detection threshold 0.6 determined?
limitations: Yes.
review_title: null

Meta-review: metareview: This paper proposes a method for continuous category discovery in novel category discovery tasks, assuming a dynamic setting in which unlabeled data are continuously fed to the model for category discovery. Specifically, it is a technique that balances the discovery of new categories with the merging of newly discovered information with knowledge already known to the model, and has shown its usefulness experimentally.  These results contribute to the progress of research in this field.
confidence: Certain
Decision: Accept