Grow and Merge: A Unified Framework for Continuous Categories Discovery
Authors: Xinwei Zhang
Jianwen Jiang
Yutong Feng
Zhi-Fan Wu
Xibin Zhao
Hai Wan
Mingqian Tang
Rong Jin
Yue Gao
Venue: NeurIPS 2022 Accept (2022)
Keywords: Novel Category Discovery
Incremental Learning
Continuous Learning

Abstract: Although a number of studies are devoted to novel category discovery, most of them assume a static setting where both labeled and unlabeled data are given at once for finding new categories. In this work, we focus on the application scenarios where unlabeled data are continuously fed into the category discovery system. We refer to it as the {\bf Continuous Category Discovery} ({\bf CCD}) problem, which is significantly more challenging than the static setting. A common challenge faced by novel category discovery is that different sets of features are needed for classification and category discovery: class discriminative features are preferred for classification, while rich and diverse features are more suitable for new category mining. This challenge becomes more severe for dynamic setting as the system is asked to deliver good performance for known classes over time, and at the same time continuously discover new classes from unlabeled data. To address this challenge, we develop a framework of {\bf Grow and Merge} ({\bf GM}) that works by alternating between a growing phase and a merge phase: in the growing phase, it increases the diversity of features through a continuous self-supervised learning for effective category mining, and in the merging phase, it merges the grown model with a static one to ensure satisfying performance for known classes. Our extensive studies verify that the proposed GM framework is significantly more effective than the state-of-the-art approaches for continuous category discovery.

Grow and Merge: A Unified Framework for Continuous Categories Discovery
Xinwei Zhang'; Jianwen Jiang"", Yutong Feng", Zhi-fan Wu², Xibin Zhao'
Hai Wan', Minggian Tang', Rong Jin?, Yue Gao
BBNRss, KLISS, School of Software, Tsinghua University
2 Alibaba Group
THUIBCS, BLBCI, Tsinghua University
sinwei-222 Emails. tsinghua. t edu.
{jianwen. jjw,, feln,utogggnn fyt, wuzhifan wzf, mingqian tmq.jinrong jinrong..jji. Balibaba-inc.o com, {zxb, wanhai, gaoygaa)tsiingnune edu.
Abstract
Although a number of studies devoted to novel category discovery, most of them assume a static setting where both labeled and unlabeled data are given at once for finding where unlabeled data are continuously fed into the category discovery system. We refer to it the Continuous Category Discovery (CCD) problem, which is significantly more challenging than the static setting. A common challenge faced by novel category discovery is that different of features needed for classification and category discovery: class discriminative features are preferred for classification, while rich and diverse features are more suitable for new category mining. This challenge becomes more severe for dynamic setting as the system is asked deliver good performance for known classes over time, and the same time continuously discover new classes from unlabeled data. To address this challenge, we develop framework of Grow and Merge (GM) that works by alternating between growing phase and merging phase: in the growing phase, it increases the diversity of features through a continuous self-supervised learning for effective category mining, and in the merging phase, it merges the grown model with a static one ensure satisfying performance for known classes. Our extensive studies verify that the proposed GM framework is significantly more effective than the state-of-thee th-t---rrttttt approaches for continuous category discovery.
categories. In this work, focus the application scenarios
Introduction
Human beings good grouping objects into category through clustering, and the definition of categories are continuously expanding and updated over time. Recent developments of intelligent visual systems
only distinguish pre-defined categories (1-5), but also discover new categories from unlabeled data, task that is known as novel category discovery [6.].
Existing works novel category discovery are limited to the static setting, where both labeled data (by known classes) and unlabeled data (with potential unknown categories) are given at once. In contrast, for real-world applications, unlabeled data continuously fed into the system for discovering new categories, making a significantly more challenging problem. Besides, current studies for novel category discovery often assume that all the unlabeled data belong the unknown
"Equal contributions. *Correpponing authors.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

new categories, which is generally not true in real applications, In this work, we examine the dynamic setup of novel category discovery where the system was initially given a set of data labeled by known classes, and unlabeled data are continuously streamed into the system for discovering new classes. The system is requested to consistently yield satisfying performance for known classes, and the same time, dynamically discover new categories from the streaming unlabeled data. We refer to it as Continuous Category Discovery, or CCD for short.
We illustrate the process of CCD in Figure ! It is comprised of two main stages: the initial stage where a classification model is trained by a set of labeled examples, and the continuous category discovery stage where new categories are continuously discovered from a stream of unlabeled data belonging to both known and unknown classes. intuitive approach address the dynamic nature of CCD is to combine the existing methods for open-ser set recognition [9- novel category discovery
12 6 13,, and incremental learning 14 15]. This however insufficient because our learning system has to accomplish two tasks at the same time, i.e., accurately classify instances into the known classes, and discover new categories from an unlabeled data stream. turns out that these two task models usually produce different types of features: discriminative features on known classes are preferred by classification model, while rich and diverse features are critical for identifying new classes, as illustrated in Figure 2 A simple combination of novel category discovery and incremental learning will fail to address the trade-off consistently over time, which is further verified by our empirical studies.
intii Stage
Continutus Catigory Diccovery Stage
Time step 1 Unlabeled
Time-slap
Labeled
Uniabeledd set
Continuous Datasets
Known
Classffer
Unknown
Overview of the Continuous Category Discovery (CCD). The continuous data stream is mixed with unlabeled samples from both known and novel categories. CCD requires to distinguish known categories, discover novel categories and merge the discovered categories into known set.
Figure 1:
To address the challenge of continuous category discovery, we propose framework of Grow and Merge, or GM for short. After pre-training a static model A over the labeled data, we will update model A with respect to unlabeled data stream by alternating between the growing phase and the merging phase: in the growing phase, we will increase the diversity of features by continuously training our model over received unlabeled data through combination of supervised and self- supervised learning; in the merging phase, we will merge the grown model with the static one by taking weighted combination of both models. By alternating between the growing and merging phases, we are able to maintain a good performance for known classes, and at same time, the power of discovering new categories. This is clearly visualized in Figure where the first two panels show that existing approaches can do well on one of the two tasks but not both, and last panel shows that features learned by the proposed GM framework works well for both tasks.
Finally, one of the common issues with continuous training is catastrophic forgetting. To alleviating the forgetting effect as we are growing the number of categories over time, we maintain small set of labeled samples from known categories and pseudo-labeled samples from novel categories. These selected examples are used in the growing phase to expand feature diversity for effective category discovery. Extensive experimental results show that our proposed method consistently shows satisfying performance under multiple practical scenarios compared with existing methods.
The main contributions of this paper are summarized as follows:
We study a new problem named continuous category discovery, or CCD, which better reflects the challenge of category discovery in the wild. It needs to simultaneously maintain good performance for known categories and the ability of discovering novel categories.
We propose a framework of grow and merge, or GM, for CCD, that able to resolve the confficts between the classification task and the task of discovering new categories.

---

Known
Categories
Novel
Categories
Maintaining Categories
Discovering Novel Categories
GM: Balance Muintaiaing Discovering
Figure 2: Features visualization model trained on known categories, model B trained for novel category discovery (based model A) and the proposed GM model.
We conduct experiments under four different settings to fully investigate the scenarios of novel category discovery in the wild. The proposed method shows less forgetting of known categories and better performance for category discovery compared to existing methods.
2 Problem Definition
In this section, we formulate the problem of Continuous Category Discovery (CCD). There are two main stages of the CCD problem, i.e., Initial Stage and Continuous Category Discovery Stage. The settings of each stage are introduced, together with the evaluation metrics of CCD problem.
2.1 Setting of Continuous Category Discovery
During the initial stage, a labeled training dataset Do Dirain, (oc y, ) provided train the model on the initial known category set C* = 21, }" where x is the initial training data, and 3 C is the corresponding label. The model expected classify categories in Co, and learn meaningful representation from the labels' semantic information.
During the continuous category discovery stage, serial of unlabeled training datasets {D) train Ir= sequentially provided, where D Dirain {= N' indicates the dataset at time t. Though unlabeled. we denote the known potential appeared categories until time f as C = {1, 2, .... K The model expected discover newly appeared unknown categories from D Prrain store the representations of the discovered novel categories and maintain the knowledge of the known categories.
2.2 Evaluation Metrics
At each time-step f, we evaluate the classification performance on the test dataset Di {(0). ls < t}, containing test samples from all the known previously discovered categories. For the newly appeared unknown categories, we evaluate the novel category discovery performance. The maximum forgetting metric M, and the final discovery metric M2 are designed for evaluation. To evaluate the performance of the clustering assignments, we follow the standard practise 66] [6] to adopt clustering accuracy on the known categories and the novel categories, denoted as ACC and ACC respectively. The maximum forgetting My is defined as the maximum value of the differences between ACCO known and ACCt for every and the final discovery Ma defined as the final cluster accuracy on novel categories, i.e. Ma ACCT povel- supplementary materials.
See more details in the

---

Intial Stage
Continuous Calegory Diecovery Stage
Labeled
Dasa
Untaboled
dlsarication
Gangerns
Mocee
Ouillon Baldrr
Dynamic
Dynamic
Dynamic
Dynamic
rodel unitiption
Phome
dstilationn
desitiation
Static
Static Brarch
Static Branch
Static
Branch
Branch
000
unidid semphe
Growing Phase
Merging
Viiging
prodicted foodd
Figure 3: The proposed Grow and Merge (GM) Framework. There exist growing and merging phases for each time-step, where dynamic branch is frrely updated for discovering new classes in the growing phase, and then unified with a frozen static branch the merging phase for maintaining classification on all appeared classes.
Method
3.1
An Overview of Grow and Merge Framework
The CCD problem requires both maintaining the performance known categories and discovering novel categories under the continuous learning scheme. For the compatibility of the two tasks, we propose Grow and Merge (GM) Framework, which is illustrated in Figure
During the initial stage, the model (-) is firstly pre-trained on the initial dataset with labeled data in both supervised and self-supervised manner. An exemplars set Ps- {Px,.} is constructed as well as the prototypes. The exemplar set contains representative samples for each class while the prototype (4g is the averaged feature vector of the exemplars from the k-th class. The exemplar set is constructed by iteratively picking several typical samples with the principle of maintaining the closest
center to the prototype (4e i.e. Рs. f- argmin Mk [<)(()) + Ei 60 (Pkj)] ||- The definitions of prototype and exemplars are adapted from iCaRL [15]. which enables us
-onaaarmttttic classifier that predicts label of a query sample from its nearest prototype's 's label.
During the continuous category discovery stage, a dual-brande architecture with a static branch (-) and a dynamic branch <D(() (-) is introduced for training, where both branches are initialized by the pre-trained model (-). At each time-step, GM includes two phases: Growing and Merging. In the growing phase, the novel category data are filtered out from the continuous data and the dynamic branch is trained for novel category discovery. Then the merging phase fuses the newly discovered and known categories into a single model for unified classification. Knowledge from the frozen static branch is distillated into the dynamic one to avoid forgetting previous classes.
It is noted that though GM maintains two branches during training, only one branch needed during testing.
Experimental results show significant performance of the GM framework, where the buan-b architecture and knowledge distillation succeed to balance the feature conflicts between continuous classification and novel categories discovery, and show better performance on both sub-tasks.
3.2 Growing: Enrich Features with Unlabeled Data
To achieve novel category discovery from the data stream with both labeled and unlabeled samples from known and unknown classes, we firstly filter out data of unknown classes with novelty detection, and then conduct novel category discovery based both the new data and the previous model.
Novelty Detection aims to identify the samples belonging to the novel categories in the current training dataset Pirain since there may also exist samples from previously known categories. With the maintained prototypes of known categories, the feature vectors of samples from known categories would lay closer to at least one prototype with a high probability. Then novel distance

---

dnovel is defined as the distance between the sample x and its nearest prototype, i.e. (z)) = min K d(ui, (i (zt)), where d(: the distance function. Samples with dnweel larger than a threshold are filtered out as novel samples for the following process.
Novel Categories Discovery and Learning from Static Branch. We optimize the dynamic branch (-) for novel category discovery, guided by both the novel samples and previously learned knowledge for rich feature representation. (i) For learning and discovering from the novel samples, we adopt an unsupervised representation learning method [6]. A cluster head g(-) is learned to assign samples &t to the clusters with novel category indices {K--1 + 1..., ) Such an assignment is supervised by the similarity 8ij between samples z² and æÿ with binary cross entropy loss, i.e.
A
LBCK
Σ ==
Big log cf cj +1 - 8ig log cf c))),
NTC
where N is the number of novel category samples at time t (specified or estimated). The similarity Sij is measured by the Hinner-Take-la hash (WTA), which equals one when samples share the same top-k channels of their feature vector, otherwise equals zero.
(ii) For learning from the static branch (.) with previously learned knowledge, we introduce static-dynamic distillation loss to transfer the representation learned from labeled data in the static branch into the dynamic one:
N LSD N Ed
N. No Σ d ( ob (x()))
1
(2)
The joint loss LECE and LSD enables the dynamic branch to discovering novel categories with richer feature representation, and the cluster head g(-) could assign new labels to the novel category data e!.
3.3
Merging: Unification of Categories and Branches
After the Growing phase, both categories and models contain two parts for the known and novel categories, respectively. The Merging phase reunifies them together for all the appeared categories.
3.3.1 Category Unification: Continuous Learning for Newly Discovered Categories
To merge the novel categories discovered in Growing phase into the classifier, we firstly sift samples to construct the exemplar sets of the novel categories, then refine the model representation using a pseudo label representation learning for classification on both known and novel classes.
Sample Sifting aims to filter out incorrectly samples based on their cluster confidence. The assigned class indices by 9()) are denoted as the pseudo- of the novel samples, which may inevitably have errors and lead less reliable prototypes of the novel categories. We sift out these potentially noisy samples via the local sample density, with the assumption that samples from the same class would lay tightly. Suppose NE)()) indicates the j nearest neighbors of E from the same class. The local sample density (2f) is defined the distance between & and its j-th nearest neighbors, i.e. (g)) = max=eN, (( z). Samples with larger G, are sifted out, then the left samples are denoted as ot with pseudo labels oz.
Pseudo Label Representation Learning. Based the sifted novel samples of the k-th novel category, we firstly construct the exemplar set P& = {Px,.) Without knowing the prototype μus i.e. the explicit center of the k-th category, we estimate it with the averaged vectors of sifted samples. For the purpose of continuous classification on all appeared classes, it is required to tighten the sparse representation distribution learned in the growing phase. Pseudo Label Learning (PLL) loss is introduced here. Compared with commonly-used Cross Entropy loss, PLL does not require an explicit classifier and thus does modify the deployed model structure during the continuous phase. The loss function LPU. aims to pull exemplars pull the exemplars ps: and the corresponding prototypes μa closer, and push pt away from the other prototypes Hy + k):
Rt
Pa
explof (Px.)) (x/T) ΣK
CPL
log
(3)
K
(15/T)
5

---

where r is a temperature hyper-parameter To avoid catastrophic forgetting, based on data in the latest exemplars, the static-dynamic distillation loss Eq. 2 is also employed here.
3.3.2 Branch Unification: Merging Multiple Branches into One Unified Model
During the above process, the dynamic branch is optimized to classify all appeared categories. In order to reduce computation cost and further utilize the representation of the static branch trained from labeled data, we merge the two branches into single one. In experiments, we found such branch merging can further improve the recognition performance. To fuse the representation Øs and D. the exponential moving average strategy (EMA) is applied:
% - abss + (1 a))0s,
where a is the momentum hyper-parameter control the weight of fusion, generally valued close to 1 (a = 0.99 in this paper).
3.4 Estimating the Number of Novel Categories
Grow and Merge framework is proposed solve the CCD problem. GM is a generic framework which focuses on how to implementing continuous classfication and novel classes discovery at the same time. Estimating the number of novel categories has been studied in literatures including [6], which can be easily employed in GMNet. For each time step f. the representations of the coming data ((x') ) mixed together with the exemplars The semi-Kmeans algorithm is applied on these mixed data with different number of clusters, and the cluster accuracy on the exemplars are evaluated based on the pseudo labels of the exemplars. The number of the clusters with the highest cluster accuracy is estimated as the number of novel categories.
4 Experiments
In this section, we design experiments in different scenarios to simulate the varying cases of real-world applications, which enables us fully investigate the performance and flexibility of the proposed method. Two main evaluation metrics, namely, maximum forgetting M and final discovery Ma are employed to measure the performance. CIFAR- representing the small, middle and large scale datasets are used in the experiments.
[17. CUB-200 [88 and ImageNet-
19 20
Experimental Scenarios. We formulate four different scenarios for experiments follows. (1) Class Incremental Scenario (CI): the data are only drawn from novel categories. (2) Data Incremental Sce- nario (DI): the data are only drawn from the known categories. (3) Mixed Incremental Scenario (MI): the data are drawn from both novel and known categories. (4) Semi-supervised Mixed Incremental Scenario (SMI): the data are drawn from both novel and known categories, and portion of the data are labeled, which is closed to the real-world application.
Table 1: The content of D Di under different scenarios DI, CI, MI and SMI.
Known Categories
Novel Categories
Labeled data
Unlabeled data
X
DI
Xx
MI
SMI
The settings of these scenarios are summarized in Table T We mainly focus on the standard scenario CI in this paper, while the experimental results DI, MI, and SMI are reported to demonstrate the flexibility of the proposed method. Each scenario contains == 3 time-steps of the continuous learning stage. Details of the scenarios could be found in supplementary materials.
Implementation Details. For experiments, we train the model 100 epochs using LACF + LSD loss and 100 epochs using LBCR + LSD + LPLI. loss for each time-step 1. The for EMA is set 0.99. The novelty detection threshold set to 0.6. 15 neighbors are selected for each sample during sample sifting and the top 50% samples with larger local density sifted out. The

---

Table 2: Experimental results under CI scenario (mean=sstd)..
CIFAR-100
CUB-200
ImageNet-100 My
Method
My 4
Me
My
Ma
Ma
lower- hound of M, toffine K-Means) apper-bound of Ma (offlive AuioNovel)
5.37:4-0.00
16,95:4:1.04
2.94
14.29+0,47
3.034:5.47
18.20-10.58
5.12:::.84
42.27:4:.97
7.57+:0.77
16.13+0..77
6.81++0.98
34,18:11.77
AutoNovel (online)
66.76-t:1.5
29.65:44.60
60.66-00.20
18.08-00.44
63.00+0.42
5.83+11.81
DRNCD (online)
14.99:1:1.9
8.53::2.35
42.74:0.28
6,30:10933
47.02-422118
5.00:11.37
AuteNovel (online) LWF
23.944:1.54
27.84:0.57
59.8220.58
23.23:44.81
61.31:0.37
7.59=10.55
DRNCD (online) LwF
26.87-+0.15
3.57:40.61
03.25
5.78
30.88:::1.57
6.78
iCaRL (fixed exemplars) LwF
34.79-+0.29
42.02++0.31
10.04-10.28
GM (Ours)
9.87-10.5
35.97++1.28
19.46 1.63
24.97+1881
8.30--0.42
27.24
Stanford-Cars
FGVC-Aireraf
ImageNet-200
Method
My 4
Ma
My L
Ma f
M,
Ma
lower-bound of M, (offline K K-Means)
2.41:1:059
10.23:10.32
3.86+0.71
15.10::1.25
3.1640.76
15.28:10.67
apper-bound of Ma loffline AutoNovel)
11.24:1:.59
20.69::3.28
6.95+22.0
27.67:00.86
5.98:0.67
24.62::0.47
AuteNovel (onlinc)
72.07:10.11
16.71:::140
53.231:0.00
26.93::1.48
60.79:10.61
10.2142.52
DRNCD (online)
50.70:1:329
2.39:10.84
38.89 +4.15
5.35::0.07
13.91=11.37
4,39-:1.20
AutoNovel (online) + L.WF
71.52+40.33
20.31+00.69
1..63++1.37
8.68+45.67
61.03-00.77
9.77++1.00
DRNCD (online) LwF
27.40:1:0.50
3.88:10.63
18.8914.155
5.61+3.00
27.40:0..4
6.7111.55
iCaLL (fixed exemplurs) LwF
24.31:10.64
22.60
10.55-+0.52
GM (Ours)
17.66:10.70
25.77:10.54
8.21
29.65-+1.63
7.50-+11.0
19.33-10.66
temperature T used in LPLL is set to 0.1. Mean Squared Error (MSE) Consistency Loss LMSE
N Σi (x)) -
(a) (ax(')) 'I2 / is also used during the continuous learning following AutoNovel [6]
where 2t! denotes the augmentation of a²e More details could be found in supplementary materials.
4.1
Comparisons with SOTA methods on CT scenario
We first investigate the performance of the proposed method under CI scenario. CIFAR- 100, CUB- 200, ImageNet- 100, Stanford- FGVC-Aircraft, and ImageNet-200 are used in this experiment. The maximum forgetting My and final discovery Ma are reported in Table Since there are few works about continuous novel category discovery and recent works about incremental learning may not fit the unsupervised scenario, we compare the proposed method with the lower bound of M and the upper bound of Ma. two recent methods for novel category discovery, i.e., AutoNovel [6] and DRNCD [16), and the combinations of the novel category discovery methods and continuous learning methods.
It is noticed that there is not exact lower/upper bound method under the CI scenario. Therefore, we conduct two methods under the offline setting, where all of the data containing both known and novel categories to the lower bound of M,, and the original AutoNovel method as the upper bound of Ma. It should be enhanced that the lower/upper bound results are for reference and could not exactly represents the best performance on both M, and Ma. Since recent novel category discovery and incremental learning methods usually contain multiple stages, we chose AutoNovel and LwF [14] and combine them for comparison based on the flexibility of implementation. It is observed that AutoNovel with LwF greatly improves the M, and M, compared to the online version of AutoNovel. However, such a simple combination not completely solves the fcature conflict caused by the two different tasks. The proposed GM framework divides and conquers the conflict of features and achieves the best results among the compared methods (that conduct classification and discovery training in one single phase) on all datasets, indicating the effectiveness of the two phases of growing and merging.
available in single stage. We present the results of K-means [2 a reference
Table 3: Experimental re- sults under DI scenario. tst.))
Table 4: Experimental results under MI and SMI scenario. istd))
MI
SMI
Method
My
Method
My 1
Mat
My L
Ma
Original Model
0
AutoNovel 6
63.25
5.74+40.17
60.98+1.277 1.27
3,961
DRNCD [6
43.81+11711
5.12:10.8
56.27=1.13
47.6741.66
Pscudo iCaRL
9.51=+0.35
AutoNovel + LwF 614
61.89:0.87
5.59+11.9
61.45=2.33
31.8411.06
DeepClustering
22
1.61::0.44
DRNCD + LwF 16 14
35.44:14.61
4.10-10.99
1.038-1001
$2,53-+1.92
A
A
24
0.30++0.07
GM novelty detection
4.74:10.78
10.61+3..77
8.43:0.45
31.42-10.6
GM
-23.68::0.27
GM
9.65:10.32
30.58-11.13
9.41=0311
35.52+1.21

---

Table 5: Estimation of the number of novel class in each time step, as well as the corresponding performances using the estimated number.
#Classes 2)
#Classes (f = 3) #Classes
M
Ma
CIFAR-100 (CT)
11
13
13
34.91 9.91-
ImageNet- 100 (CT)
12
13
13
9.19-00.1
26.26+11227
CIFAR-100 (MI)
14
13
0.33++0.59
28.06+0043
CIFAR-100 (SMI)
14
14
14
9.63
35.31
Intra-Closs Distances
Inter-Class Distances
1393
Intre
Before Grow
After Grow (b)
After Merge
Growing
Meriia
Grewitz
Merinu
Growitz
Merging
Figure 4: (a) The intra-/inter- class distance measured throughout the growing and merging phases. (b) Feature visualizations different time of GM, where the colored dots denote previously appeared classes, and the gray dots denote the newly appeared classes without labels.
4.2 Extended Experiments on More Complicated Scenarios
In this part, we extend GM to more complicated scenarios: DI, MI, and SMI scenarios. Experiments are conducted on CIFAR- 100 and the results are listed in Table Table Due to there no novel category in the incremental unlabeled data under DI, we remove the cluster head in the AutoNovel to form compared method AutoNovel". Incremental learning method iCaRI. [15] combined with pseudo label assignment and unsupervised learning method DeepClustering combined with finetuning on incremental data are also chosen for comparisons. As shown in Table after finetuning on the incremental unlabeled data, all comparison methods show the catastrophic forgetting effect, and the performances are even worse than the original model. GM avoids catastrophic forgetting and effectively utilizes continuous unlabeled data, further improving the final performance. In MI and SMI scenarios, incremental data are mixed with known and novel categories, which makes label assignment for incremental data more difficult. Nonetheless, shown in Table 4 GM can still effectively avoid catastrophic forgetting and achieve superior performance (especially on M2) with the help of two phases of growing and merging, compared the comparison methods.
4.3 Estimating the Number of Novel Categories
In this part, experiments with the unknown number of novel categories are conducted on CIFAR- 100 and Image 100 in CI, MI, and SMI scenario. The model should estimate the number of novel categories, discover the novel categories, and maintain the performance on the known categories during each time step. The experimental results are provided in Table It could be found that GM without knowing the ground-truth number of novelty classes can still outperform other methods using the ground-truth number of novelty classes. GM model only has 1.33 and 2.47 decreases under CI scenario, and no more than 4.71 decreases under more challenging MI and SMI scenarios, which is acceptable.
4.4 Ablation Studies
Previous experimental results indicate Grow and Merge framework can better accomplish the contin- uous learning and the novel category discovery multiple scenarios. Here we provide intra- and inter- class distance among features and the visualizations in different time of GM in Figure 4 It can be clearly observed that with the alternation of the growing and merging phases, the intra-class distance first increases and then decreases. The classes are relatively compact but overlapped with others before growing phase, loose after growing phase, and compact and separated with others after merging phase. The results are consistent to our motivation that the GM relaxes and tightens the features alternatively to resolve the feature conflicts caused by CCD task. The detailed analysis for all proposed key components of GM provided. Ablation studies are conducted the CIFAR-100 100

---

dataset under CI scenario, shown in Table 6 and Tabel 7 We also provide the experiments about estimating the number of novel categories in the supplementary material.
Growing Phase. We investigate the performance of introducing additional methods for representation learning in the growing phase. Combining with cross entropy loss (CE) of the exemplars degrades Ma. which validates that relaxing the constraint of memorizing previous category information can facilitate category discovery. Thus, we further introduce more relaxed self-supervised methods. Combing SimCLR 3 also degrades Ma. which could be attributed that pushing the representation of negative samples away hurts the performance. An alternative method SimSiam [25] achieves comparable performance without pushing negative samples, which also validates the importance of a relaxed representation in this phase.
Merging Phase. In the merge phase, PLL is employed to improve the representation. We evaluate the performance of the models replacing PLLL with other representation learning methods. CE represents cross entropy loss with the pseudo label given by the prototypes and SSL. denotes standard InfoNCE The experimental results demonstrate that, though these alternative methods could perform competitively on known classes, they sacrifice the performance of discovering novel classes.
loss 3 4
Model Merging. We evaluate the performance of the model without EMA, My increases for 55% and Ma decreases for 14%, which demonstrates the necessity of EMA. We also consider an alternative fuse method which assigns Øg & % after the update of % by Eq. 4 Ma is significantly affected with a decrease of 33%. Therefore, we use the fixed Φg in this paper.
Static-Dynamis Branch. The multi-branch aims to maintain known category information and discover novel categories on different branches. We first evaluate the performance when directly replacing the multi-branch with single one. The results show that M increases dramatically, which demonstrates the importance of the multi-branch to preserve already learned categories. We further investigate the components of the static-dynamic branch. Removing the initialization of branch 0% also degrades both My and Ma. For the proposed loss LSD aligning the feature space of Φs and results show that the model without LSD performs worse both M, and Ma for the separation of the feature spaces leading to the failure of the prototype mechanism. The model replacing LSD with DINO 26 increases My for 3% and decreases Ma for 6.3%.
Sifting Merging. Sifting aims to sift out samples with low cluster confidence given by the cluster head. Except for the criterion based on the adopted local geometry property, there are several potential criteria to sift samples. Confidence Ranking (CR) selects portion of all samples with the highest score Conditional Confidence Ranking (CCR) selects a portion of samples from each class with the highest score. Confidence Filtering (CF) selects the samples with the score higher than a given threshold. The results show that: (i) For My, the model without sifting and CR performs worse, and CCR, CF and the local sample density based method performs at almost same level. (ii) For Ma. the local sample density based method outperforms all of these compared methods.
Hyperparameter Aalysis of Static-Dynamic Branch. We also evaluate the effect of a in Eq. 4 shown in Figure 5 The hyper-parameter a controls the importance of de relative to &b- For the model with only EMA, both M and Ma decrease with the increase a, since the model with higher maintains more information from the known categories. The model with only LSD (i.e., the model
w/o. EMA shown in Tabel 6) performs worse than the proposed full model, which demonstrates that only the constrain of the distance between the feature space of dy and Φ% is not enough to transfer the knowledge from dy to of- The performances of the models with both EMA and LSD varies smoother, compared with the models with only EMA. Therefore, the models could achieve higher performances and be more robust by combining EMA and Lsp-
a.
b.
d.
35.0
40.0
20.0
14
17.5
30.0
32.5
12
35.0
37.5
32.5
27.5
125
30.0
25.0
025
0.5
0.00
0.00
0.50
0.75
80
0.5
077
0.00
Figure 5: The variation comparison of the performance of the models by varying a. a: My of the model with only EMA; b: Ma of the model with only EMA; e: M, of the model with both EMA and LsD: d: Ma of the model with both EMA and LSD-

---

Table 6: Results of ablation studies GM phases. (tt.
Table 7: Results of ablation studies Static- Dynamic Branch and Sifting in the Merging. (meantstd.)
My
Ma
GM
988:20.25
35.97
My
Ma
W CCE
9.54
31.37=0.19
9:8720
35.971
Growing Phose
Cap SSmCL
9.2420.08
32.29=1.4
w/o, Mult Brunch
33.46:0066
33.80.1.17
WTA+Cop4SSiSiin
9.56:20.06
34.81
Stati Dynamic Branch
wio. Initialiae do
PL.
8.3:20.30
34.48
wto. Esp
15:78:20422
11.71
Merging Phase
S PLL SSL
9.19:10:17
1.27=00225
31.95-0811
34,94-2.01
wio. Sifting
DINO
1000=:0:1
9.58::0.30
31.68+00.55
32.89
7.55:10222
1.80
Sifting the Merging
Scftng (CR)
9.11=:0.19
33.18 1.46
Model Morging
w/o. EMA
Update d5
11.75:11.55
17.72:22.12
31.18=2.15
24.34=2.66
Sifting (CCR)
Sifting (CFF
9.78=t0.06
9:742:0.58
0..86.10.55
31,93
5 Related Work
27 aims discover unknown classes from unlabeled data by leveraging pairwise similarity 16 28 or employing unsupervised clustering [I3] 29]. Most of these methods assume that there is a pure unlabeled set that only contains samples from unknown categories, which means the class space of unlabeled data is completely disjoint with the labeled set. And at test time, samples are all from novel classes. These assumptions limit their applications the real world. CCD aims to tackle novel class discovery under a more realistic situation in which the unlabeled data (i) are continuously confronted and (ii) contain both known and unknown classes.
Novel Class Discovery 688
Continual Learning needs the model to extend its ability to address new tasks when confronted with new data 38). The main challenge of continual learning is to avoid catastrophic forgetting [39 40),
i... the performance degradation on previously learned tasks when new ones are ingested. Existing methods be categorized into replay methods [13 42 regularization-basee
methods 14 43
44 and parameter isolation methods 45 Most of the methods assume there are available labeled data in each time-step. Nevertheless, the data in continuous streams are more likely to be unlabeled in reality, which means the category information is unknown and needs to be discovered.
Different from existing novel category discovery and continual learning. CCD requires the model to discover novel categories from the continuous unlabeled data while maintaining the classification performance on known and discovered classes.
6 Conclusion
In this paper, we study the continuous category discovery (CCD) problem where unlabeled data are continuously fed into the category discovery system. The main challenge of the CCD problem is that different sets of features are needed to classify known categories and discover novel categories. To address this challenge, we develop framework called Grow and Merge that alternatively updates the model in the growing phase and the merging phase. Extensive experimental results demonstrate that GM outperforms the existing methods with the less forgetting effect of known categories and higher accuracy for category discovery.
Acknowledgement
This research is sponsored in part by the Key-Area Research and Development Pro- gram of Guangdong Province under Grant 2020B010164001 and the NSFC Program (No. 11100000UUU92222221999944
This work was sup- ported by Alibaba Group through Alibaba Innovative Research Program.
References
[1] Wouter Van Gansbeke, Simon Vandenhende, Stamatios Georgoulis, Mare Proesmans, and Lue Van Gool. SCAN: learning to classify images without labels. In European Conference on Computer Vision, pages 268-285, 2020.
10

---

[2] David Berthelot, Nicholas Carlini, Ian J. Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raffel. Mixmatch: A holistic approach to semi-supervisee learning. In Advances in Neural Information Processing Systems, pages 5050-5060, 2019.
[3] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In Proceedings of the 37th International Conference Machine Learning, pages 1597-1607, 2020.
[4] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. Momentum contrast for unsupervised visual representation learning. In IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 17226-9735 2020.
[5] Yutong Feng, Jianwen Jiang, Mingqian Tang, Rong Jin, and Yue Gao. Rethinking supervised pre- pre-training for better downstream transferring. In International Conference on Learning Representations, 2022.
[6] Kai Han, Sylvestre- Alvise Rebuffi, Sebastien Ehrhardt, Andrea Vedaldi, and Andrew Zisserman. Autonovel: Automatically discovering and learning novel visual categories. IEEE Transactions on Puttern Analysis and Machine Intelligence, 2021.
[7] Enrico Fini, Enver Sangineto, Stéphane Lathuilière, Zhun Zhong, Moin Nabi, and Elisa Ricci. unified objective for novel class discovery. In IEEE/CVF International Conference Computer Vision, pages 9264-9272. 2021.
[8] Sagar Vaze, Kai Han, Andrea Vedaldi, and Andrew Zisserman. Generalized category discovery. CoRR, abs/2201. 0.0609,,.......... 2022.
[9] Abhijit Bendale and Terrance E. Boult. Towards open set deep networks. In JEEE Conference on Computer Vision and Pattern Recognition, pages 1572-1572, 2016.
[10] Chuanxing Geng, Sheng-Jun Huang, and Songcan Chen. Recent advances in open set recog- nition: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(10): 3614-3631, 2021.
[11] Zhi-Fan Wu, Tong Wei, Jianwen Jiang, Chaojie Mao, Mingqian Tang, and Yu-Feng Li. NGC: A unified framework for learning with open-world noisy data. In 2021 IEEE/CVE International Conference on Computer Vision, ICCV 2021, Montreal, QC. Canada, October 10-17, 2021, pages 62-76 71. IEEE, 2021.
[12] Kai Han, Sylvestre-Alviss Rebuffi, Sebastien Ehrhardt, Andrea Vedaldi, and Andrew Zisser- man. Automatically discovering and learning new visual categories with ranking statistics. In International Conference on Learning Representations, 2020.
[13] Kai Han, Andrea Vedaldi, and Andrew Zisserman. Learning to discover novel visual categories via deep transfer clustering. In JEEE/CVF International Conference on Computer Vision, pages 8401-8409, 2019.
[14] Zhizhong Li and Derek Hoiem. Learning without forgetting. In European Conference on Computer Vision, pages 614-229, 2016.
[15] Sylvestre-Alvise Rebulfi, Alexander Kolesnikov, Georg Sperl, and Christoph H. Lampert. iCaRL: Incremental classifier and representation learning. IEEE Conference on Computer Vision and Pattern Recognition, pages 5533-5542 2017.
[16] Bingchen Zhao and Kai Han. Novel visual category discovery with dual ranking statistics and mutual knowledge distillation. In Advances in Neural Information Processing Systems, pages 22982-22994, 2021.
[17] Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Tech Report, 2009.
[18] Peter Welinder, Steve Branson, Takeshi Mita, Catherine Wah, Florian Schroff, Serge Belongie, and Pictro Perona. Caltech-uesd birds 200. 2010.

---

[19] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115(3):211-253) 2015.
[20] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In European Conference on Computer Vision, pages 776-794, 2020.
[21] Hans Hermann Bock. Clustering methods: a history of k-means algorithms. Selected Contribu- fions in Data Analysis and Classification, pages 161-172, 2007.
[22] Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsupervised learning of visual features. In European Conference on Computer Vision, pages 139-156, 2018.
[23] Jay Yagnik. Dennis Strelow, David A. Ross, and Ruei-Sung Lin. The power of comparative reasoning. In IEEE International Conference on Computer Vision, pages 2431-2438, 2011.
[24] Xuhui Jia, Kai Han, Yukun Zhu, and Bradley Green. Joint representation learning and novel category discovery on single- and multi-modal data. In IEEE/CVF International Conference on Computer Vision, pages 590-599, 2021.
[25] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15750-15758, 2021.
[26] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in supe-supe vision transformers. In JEEE/CVF International Conference on Computer Vision, pages 9630-99009 1640,9960,000 2021.
[27] Bingchen Zhao and Kai Han. Novel visual category discovery with dual ranking statistics and mutual knowledge distillation. In Advances in Neural Information Processing Systems, pages 22982-22994, 2021.
[28] Xuhui Jia, Kai Han, Yukun Zhu, and Bradley Green. Joint representation learning and novel category discovery on single- and multi-modal data. In IEEE/CVF International Conference on Computer Vision, pages 590-599. IEEE, 2021.
[29] Zhun Zhong, Enrico Fini, Subhankar Roy, Zhiming Luo, Elisa Ricci, and Nicu Sebe. Neighbor- hood contrastive learning for novel class discovery. In IEEE Conference on Computer Vision and Pattern Recognition, pages 10867-10875, 2021.
[30] Matthias Delange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Ales Leonardis, Greg Slabaugh, and Tinee Tuytelaars. A continual learning survey: Defying forgetting in classification tasks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
[31] Quang Pham, Chenghao Liu, and Steven C. H. Hoi. Dualnet: Continual learning, fast and slow. In Advances in Neural Information Processing Systems, pages 16131-16144, 2021.
[32] Hyuntak Cha, Jacho Lee, and Jinwoo Shin. Co²L: International Conference on Computer Vision, pages (496-9905,, 2021.
Contrastive continual learning. In IEEE/CVF
[33] Fei Zhu, Zhen Cheng, Xu-Yar Yao Zhang, and Chenglin Liu. Class-incremental learning via dual augmentation. In Advances in Neural Information Processing Systems, pages 14306-14318, 2021.
[34] Hao Liu and Huaping Liu. Continual learning with recursive gradient optimization. In Interna- tional Conference on Learning Representations, 2022.
[35] Zichen Miao, Ze Wang, Wei Chen, and Qiang Qiu. Continual learning with filter atom swapping. In International Conference on Learning Representations, 2022.
[36] Divyam Madaan, Jaehong Yoon, Yuanchun Li, Yunxin Liu, and Sung Ju Hwang. Representa- tional continuity for unsupervised continual learning. International Conference on Learning Representations, 2022.
12

---

[37] Yu-Ming Tang, Yi-Xing Peng, and Wei-Shi Zheng. Learning to imagine: Diversify memory for incremental learning using unlabeled data. CoRR, abs/2204.08932, 2022.
[38] MohammadReza Davari, Nader Asadi, Sudhir Mudur, Rahaf Aljundi, and Eugene Belilovsky, Probing representation forgetting in supervised and unsupervised continual learning. CoRR, abs/2203. 13381, 2022.
[39] Anthony V. Robins. Catastrophic forgetting, rehearsal and pseudorehearsal. Connection Science, 723):123 1995.
[40] Ian J Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio. An empirical investigation of catastrophic forgetting gradient- based neural networks. CoRR, abs/1312.6211, 2013.
[41] Hanul Shin, Jung Kwon Lee, Jachong Kim, and Jiwon Kim. Continual learning with deep generative replay. In Advances in Neural Information Processing Systems, pages 2990-2999, 2017.
[42] David Lopez-Paz and Marc' Aurelio Ranzato. Gradient episodic memory for continual learning. In Advances in Neural Information Processing Systems 30, pages 6467-6476, 2017.
[43] Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning, pages 3987-3995, 2017.
[44] Sang-Woo Woo Lee, Jin-Hwa Kim, Jachyun Jun, Jung-War Woo Ha, and Byoung- Tak Zhang. Overcoming catastrophic forgetting by incremental moment matching. In Advances in Neural Information Processing Systems, pages 4652-4662, 2017.
[45] Arun Mallya and Svetlana Lazebnik. Packnet: Adding multiple tasks to single network by iterative pruning. In IEEE Conference on Computer Vision and Pattern Recognition, pages 7765-7773, 2018.
[46] Arun Mallya, Dillon Davis, and Svetlana Lazebnik. Piggyback: Adapting single network to multiple tasks by learning to mask weights. In Proceedings of the European Conference on Computer Vision, pages 72-88, 2018.
[47] Ju Xu and Zhanxing Zhu. Reinforced continual learning. In Advances in Neural Information Processing Systems, pages 907-916, 2018.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See supplementary materials.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See supplementary materials.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full of assumptions of all theoretical results? [N/A] Theoretical results are not included.
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex- perimental results (either the supplemental material or as URL.)? [Yes] See the supplementary materials.
13

---

(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See section and supplementary materials.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See supplementary materials.
4. If you are using existing assets (e.g., code, data, models) or curating/releusing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] See Section
(b) Did you mention the license of the assets? [N/A] The license of the assets are not provided their official website.
(c) Did you include any assets either in the supplemental material
URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [No]
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [No]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, applicable? [N/A] Crowdsourcing and human subjects are not used in this research.
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "lYU0uqmKV6y", "reviewer": "Reviewer_WTKt", "summary": "This paper addresses the problem of continuous category discovery, which is a natural extension of novel category discovery by sequentially feeding part of the targeting datasets into the model. The method builds on top of existing methods for novel category discovery and continual learning, namely, AutoNovel[6] and iCaRL[14], with a framework trained with grow and merge phases, containing two branches. Experiments on three public datasets have been presented showing obvious improvement over the compared methods. ", "questions": "Please refer to the weaknesses above. In addition, \n\n- Figure 4 and the illustration need to be clarified. The text is not well aligned with Fig 4, making the explanation rather unclear. For example, what are these selected clusters, what are those gray dots, which time-step is it, which are the old classes, and which are the new classes?  \n- As an additional comment to the weakness of the Experiments above, it would be more convincing to repeat on different random splits -- assuming the splits used in the experiments were randomly drawn, and also report the results on each split and each time step (more suitable for supplementary, but it would be useful to show how the method actually works in the intermediate steps)\n- In Table 2 and 4, why DRNCD w/ LwF results are not reported, since it outperforms AutoNovel in NCD? But only the results of AutoNovel w/ LwF are reported.\n- How was novelty detection threshold 0.6 determined?", "limitations": "Yes. ", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n+ The paper tacks the problem of continuous category discovery (NCD), which jointly considers the problem of novel category discovery and continual learning. This is a natural and interesting extension of NCD. \n+ Good performance is achieved on three public datasets, outperforming the compared baselines. \n\nWeaknesses:\n- Though effective, the key components of the method are based on AutoNovel and iCaRL. The extra branches and grow-merge phases increased the complexity and meomery&computation cost for training. \n- Experiments are only carried out on three relatively small datasets with a relatively small number of new classes in each time step. Also, the data splits for different time steps for the MI case vary a lot, which seems strange (as specified in the supplementary), why not adopt a consistent proportion for all? Or perhaps more different proportions can be used? How the different classes in each time step are constructed? randomly sampling or predefined in some way? \nOnly 10 classes are incremented at each time step, how about incrementing more classes? say 20, 50? For more challenging cases, a larger dataset could be more useful. It appears that more different datasets are used in the literature, e.g, CIFAR10, ImageNet,  Stanford-Cars FGVC-Aircraft in DRNCD [15].\n- The new category number is assumed to be known in each incremental time step. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "DyFdR-nYCL2", "reviewer": "Reviewer_kWya", "summary": "This paper advances the task of novel category discovery to a dynamic setting where unlabelled data are continuously fed into a model for category discovery, refered as continuous category discovery.\nThe paper also proposed a grow and merge framework to solve this problem by balancing between novel category discovery and merging the newly discovered information with the knowledge already know to the model.\n", "questions": "Q1: In fig. 2, the paper aim to show that the features from the known categories and the novel categories are different, however, this visualization is not quantitative and could cause a strong bias if we only visualize a small subset of the dataset. Although I think the overall observation is correct, I would suggest to use the notion like uniformity and alignment [R1] to better show the main motivation of the paper.\n\nQ2: In the sample sifting part, I did not quite understand the reasoning of find the maximum distance between a data point and the j nearest neighbors, isn't the maximum distance between a data point and its top-j neighbors always at the j-th nearest neighbor? In other word, is this measure always depends on the parameter j? I am also wonderring how about using a averaged distance between the data point and all its top-j nearest neighbor.\n\nQ3: The setting assumes that the number of categories at time t is known, however this is an unrealistic assumption, because the novel categories are unknown, how can we assume we know the number of categories? In previous works on novel category discovery, there are methods that can be used to estimate the number of novel categories [R2], and so it makes sense for NCD methods to assume the category is known, however in the CCD case proposed in this paper, I would suggest the paper to include discussion of how to apply the algorithm in [R2] to estimate the number of categories in each time step in the CCD setting, and also include experiments of evaluate the proposed method using the estimated category number.\n\nQ4: The method part of this paper is quite hard to understand, because it involves concepts that are not properly defined (such as exemplars and prototypes, the paper only states it use these terms as in iCaRL makes the paper not self-contained), and the framework seems to be quite complex, I would suggest to make it clear about every concept mentioned in the paper, and perhaps also include a method diagram to better demonstrate the idea of the framework.\n\nQ5: The framework is quite complex, involving four losses, how to balance between these losses? The paper simply add them together without weighting, it would be great if we can have a study about the weighting between the four losses.\n\nQ6: The experiments in Table 4 seem weird, the DRNCD method performed lowest on the MI setting, but the performance on SMI setting improves a lot, this is strange, could this be a sign that the setting are not stable, so that the variance of the experiments is large? Also, I would appreciate it if the paper could provide error bars on the experiments results.\n\n\nMinor:\n\n1. The paper is not very self-contained, the term exemplars and the initialization of exemplars and prototypes are used but not defined, the paper directly cites iCaRL for this, IMO, a good paper should be self-contained, at least the definition of these terms should be in the main text.\n2. Typo at line 266 phrase -> phase\n\n\n[R1] Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere, ICML 2020\n\n[R2] Learning to Discover Novel Visual Categories via Deep Transfer Clustering, ICCV 2019\n", "limitations": "The most noticable limitation of the proposed method and setting is that the number of novel categories is assumed to be konwn, I would suggest the author to try to include a discussion on this and try to expand the algorithm in [R2] for the proposed setting.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "S1: The proposed experimental setting is properly defined, and could be of interest for the novel category discovery community.\n\nS2: The proposed grow and merge framework seems to perform well on the proposed experimental setting.\n\nW1: The overall pipeline is complex, however I think the evaluation and discussion of the results are not enough, see Questions.\n\nW2: The implicit assumption that the number of categories is known seems to be unrealistic.\n\nOverall, I think this paper has its merit, but the defect outweight its merit, and the presentation of the work needs to be improved.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "AKsWkHClrrR", "reviewer": "Reviewer_oK8E", "summary": "Authors propose a continuous setting of GCD called CCD where unlabeled data from novel and known categories appear in batches. And they also propose a framework/method to iteratively update representation & novel category discovery for CCD. In proposed framework, authors first perform novelty detection and discover novel categories & update representation on batch of unlabeled data via pairwise similarity of representation & regularize via static branch. Then based on estimate of current cluster assignments, they remove outliers of clusters to update exemplars and retrain representation with pseudo labels to merge categories, and finally merge static and dynamic representation with a moving avg. strategy.\n", "questions": "In line 60 of page 2, 'we maintain a small set of training examples for each category for both known & discovered'. Can you comment and clarify size of this small dataset stored and all the steps where this dataset is leveraged, so is this small dataset not used for merging and only in growing as stated in line 62?", "limitations": "Currently framework is adopted to a certain choice of self-supervised representational learning, though framework is generic its effectiveness with other techniques is to be evaluated empirically.\nAnd as authors pointed out this framework assumes no.of.novel categories in each time-step a priori but could be addressed in future work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Key contributions of paper is Proposal of CCD which useful continuous adaptation variation of GCD. And they propose a framework to address the setup and alleviate challenges of catastrophic forgetting, noisy gradient information available in CCD unlike GCD.\n\nStrengths:\n\tAddresses an important problem of interest of practical value.\n\tProposed framework could potentially be applied with various techniques.\n\tMultiple settings of experiments for evaluations and good analysis in experiments.\n\nWeakness\n\tLack of evaluation with different techniques, especially considering a continuous version of GCD or other semi/self-supervised representation updates.\n    Lack of clarity in writing few details w.r.t setting and experiments, especially what and how if any additional data is used beyond exemplars. Also it is mentioned as limitation that number of novel classes at any time is known a-priori in supplementary, to make it clear it should be stated in main paper too while defining $L_{BCE}$ and in the setting.\n\nSuggestion to Authors: \n    Proposed grow and merge framework is a useful generic solution and is potentially agnostic to techniques used for self-supervised representational learning and to updating representation within CCD iterations. \n    To decouple effectiveness of proposed framework and get better information w.r.t dependency on exact techniques especially w.r.t online vs offline loss in performance, it could be fruitful to adopt proposed framework to some other method too? \n\nE.g. [1] for GCD, a follow up to AutoNovel adopted in this paper or any other method for GCD could be another upper-bound to consider? This is a suggestive example, authors may choose other method of representation learning and updates to adopt proposed framework.\n\t\tFor [1] may be worthwhile to consider performing an experiment without WTA in growing phase and use techniques from GCD, with and without appropriate $L_{SD}$.\n\t\tWhere for an unlabeled dataset , update representation with self-supervised instance contrastive learning and also supervised contrastive learning leveraging small dataset?, and perform semi-supervised k-means++ applied by GCD for clustering.\n\t        On this representation, get final performance with and without retraining/additional fine-tuning using other techniques of pseudo labels as alternative to PLL as already evaluated in ablation study at each 't'.\n\nAlso if it is assumed number of novel classes is known a priori, consider an ablation study to analyze sensitivity i.e. if our estimate of novel classes is off by some value at each time step 't' and in  addition these details could potentially be better explained by stating them explicitly in both setting and experiments.\n\nWriting: \n\tIn section 2.1 it is stated only unlabeled data is available and in line 60 it is stated small dataset is available for growing phase, please be explicit for readers to get a clear understand of setting. Is it source-free or exemplar only or some small replay buffer, etc. \n\tIn line 43, 44 it is stated known classes classification and novelty detection would need different types of features, and this is not necessarily true or would suggest rewriting  sentence. As ideally we need more discriminative features to classify known + unknown and only training only w.r.t known may not have sufficient features to classify unknown but both need better discriminative features as indicated by [2] (closed set accuracy is all you need for open set)\n\tMinor corrections: Line 266, Phrase should be replaced with phase.\n\nReferences:\n1. Generalized Category Discovery https://openaccess.thecvf.com/content/CVPR2022/papers/Vaze_Generalized_Category_Discovery_CVPR_2022_paper.pdf \n2. OPEN-SET RECOGNITION: A GOOD CLOSED-SET CLASSIFIER IS ALL YOU NEED? https://openreview.net/pdf?id=5hLP5JY9S2d\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept