Uncertainty-Aware Hierarchical Refinement for Incremental Implicitly-Refined Classification
Authors: Jian Yang
Kai Zhu
Kecheng Zheng
Yang Cao
Venue: NeurIPS 2022 Accept (2022)
Keywords: incremental learning

Abstract: Incremental implicitly-refined classification task aims at assigning hierarchical labels to each sample encountered at different phases. Existing methods tend to fail in generating hierarchy-invariant descriptors when the novel classes are inherited from the old ones. To address the issue, this paper, which explores the inheritance relations in the process of multi-level semantic increment, proposes an Uncertainty-Aware Hierarchical Refinement (UAHR) scheme. Specifically, our proposed scheme consists of a global representation extension strategy that enhances the discrimination of incremental representation by widening the corresponding margin distance, and a hierarchical distribution alignment strategy that refines the distillation process by explicitly determining the inheritance relationship of the incremental class. Particularly, the shifting subclasses are corrected under the guidance of hierarchical uncertainty, ensuring the consistency of the homogeneous features. Extensive experiments on widely used benchmarks (i.e., IIRC-CIFAR, IIRC-ImageNet-lite, IIRC-ImageNet-Subset, and IIRC-ImageNet-full) demonstrate the superiority of our proposed method over the state-of-the-art approaches.

Uncertainty-Aware Hierarchical Refinement for Incremental Implicitly-Refiie Classification
Jian Yang!* Kai 1 Kecheng Zheng²
Yang Cao
University of Science and Technology of China
Ant Group
3 Institute of Artificial Intelligence, Hefei Comprehensive National Science Center
(yang) 1 annn11112 1380maiil 1, zkzy@mai 11) .utc.. edu.cn z kcc1 oudd gmai c.com forrest Sustc. duste.ed.cnc
Abstract
Incremental implicitly-refiee classification task aims at assigning hierarchical la- bels each sample encountered different phases. Existing methods tend fail in generating hierarchy-invariant descriptors when the novel classes are in- herited from the old ones. To address the issue, this paper, which explores the inheritance relations in the process of multi-level semantic increment, proposes Hierarchical Refinement (UAHR) scheme. Specifically, our proposed scheme consists of a global representation extension strategy that enhances the discrimination of incremental representation by widening the corre- sponding margin distance, and hierarchical distribution alignment strategy that refines the distillation process by explicitly determining the inheritance relation- ship of the incremental class. Particularly, the shifting subclasses corrected under the guidance of hierarchical uncertainty, ensuring the consistency of the homogeneous features. Extensive experiments widely used benchmarks (ie.. IIRC-CIFAR, IIRC-ImagNNet-litt. MIRC-ImageNet--uusee and IIRC Imageett-Nee- full) demonstrate the superiority of our proposed method over the state-of-the-art approaches.
an Uncertainty-Awan
Introduction
In recent years, deep learning has made huge breakthroughs in the field of computer vision, matching or even surpassing human performance on some image recognition tasks [3]. However, learning multiple tasks [25] in sequential data (i.e., continual learning [20, 22]) remains major challenge, which requires models to have the ability to aggregate different learning objectives into a coherent whole over time.
When a new task comes with the increase of identified classes (i.e., joint training with all old and new data is too time- time-consuming and labor-intensive Furthermore, most data from past tasks are unavailable due to the data privacy. To adapt rapidly to the new scenarios, previous methods [26] try to adopt simple alternative that directly fine-tune the network with new data. However, this may severely degrade the performance of the old class due the bias of the feature extractor and classifier towards the new class [15], which is also known catastrophic forgetting.
learning),
To address the issue, existing methods [16] maintain the performance of the old class by preserving the representative samples (i.e., exemplar) and aligning the output distribution (i.e., knowledge dis-
"Co-first Author
"Corresooding Author
"Wor done during internship at Ant Group
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Uour Drch
Drlclow
The setting of IRC. Compared with the standard incremental process, superclasses and subclasses under the IIRC setting are mixed in the incremental process, and the labels of subclasses images are refined progressively, even for the same image.
Figure 1:
tillation). This joint optimization with incremental samples strikes balance between stability and plasticity, resulting in better predictions that fall into the label space of the old or new class.
However, is incremental learning always binary classification option that distinguishes be- tween the old and new class? In existing incremental learning settings (e.g., ImageNet in Fig. I())), each instance is arranged in a certain incremental stage in an exclusive manner. However, in real life, people's semantic understanding of the same instance may be gradually enriched as the learning process proceeds. As shown in Fig. 1(b), the semantics of the same image is refined from Garment to miniskirt the IIRC- ImageNet setting. This paper focuses on the incremental abil- ity to progressively learn and maintain multi-level semantic information, which is also known as Incremental amplicitlyReeinee Classification (i.e. IIRC) [1].
How to discern the semantic inheritance relationship in a hierarchical incremental scenery? We find that the incremental performance for multi-level recognition decreases dramatically when applying classical class-incremental learning methods directly to the IIRC setting. Our analysis mainly attributed to the following reasons. As shown in Fig. probability distributions of classes with inheritance relationships show an obvious consistency in the initial phase, the incremental subclass inherited from a certain old class gradually outgroups under the supervision of new labels. It destroys the integrity of the representation of the whole old class, thus losing the semantic relevance of the hierarchical labels. On the other hand, some incremental classes inherit from none of the existing classes, leading to feature confusion due to the lack of old supervision.
the hand, although the
To this end, we propose an Uncertainty-Aware Hierarchical Refinement (UAHR) scheme, which exploits the correlation of hierarchical distributions guide the optimization of incremental rep- resentation. Concretely, a global representation extension strategy is proposed to widen the dis- tribution distance among all new classes in the embedding space, enhancing their diseriminative properties. Furthermore, hierarchical distribution alignment strategy is further proposed to correct the optimization of the shifting subclasses by aligning with the distribution of the whole superclass, ensuring the consistency of the hicrarchical uncertainty. In this way, we use RBF mapping to ex- plicitly measure the distance in the feature space between the training samples and the class centers, quantifying the feature correlation. In the incremental phase, we calculate the entropy distribution of new classes in the old embedding space for the estimate of hierarchical uncertainty. The resulting differences and similarities are utilized to identify the multi-level semantic relationships, refining the subsequent distillation objectives. Comprehensive experimental results on IIRC-CIFRR, IIRC- ImageNet-lite, and IIRC-ImageNet-full datasets demonstrate the superiority of our method.
2
Related Work
2.1
Class-Ineremental Learning
Existing class incremental learning methods can be mainly divided into four types. Regularization- based methods preserve high-weight parameters by estimating the importance of individual model parameters [2, 12, 24], while allowing unimportant parameters to be updated flexibly to ensure the learning ability for new classes. However, the study of [20], it is shown that such methods have poor generalization performance in class incremental learning.

---

New Classes (he-Sulecar))
Seperctno 2
Seperchor
Sand Tew
(A) Initial phase Figure 2: Motivation of our method. The class with inheritance relationships has consistency in the initial distribution. As incremental learning proceeds, the subclasses shift away from the distribution of the corresponding superclasses and mix with new classes without subclasses.
(B) Incremental phase
Distillation-based methods encourage the model to learn tasks, while the model representation obtained by the current training data is forced to mimic the representation of the old model. The new model in [14] completes knowledge distillation by matching the output of the sigmoid function with the old model. LUCIR [10] mitigates forgetting by distilling the output of the softmax function with the temperature scale and L2 regularization. LwM [7] reduces forgetting by matching gradient attention maps from the highest-scoring labels. PODNet [8] performs knowledge distillation be- tween old and new models by retaining intermediate features pooled along width and height, and controls the balance between previous knowledge and new information, resulting in a more adaptive incremental representation.
Rehearsal-based methods work by storing a limited number of representative examples or adopting generative model for old class samples when training tasks. Incremental Classifier Represen- tation Learning (iCaRL) [16] retains a small number of samples for each class to approximate class centroids and make predictions based the nearest class average classifier. [18] generates pre- viously observed class samples by using generative adversarial networks (GANS) [9]. Structure- based methods [27] keep the learned parameters related to the previous classes unchanged, and assign new parameters in different forms, such as unused parameters, extra networks, etc., to learn new knowledge. DER [23] concatenates the extracted features by adding a new feature extractor in the incremental phase. A sparsity сттоr is adopted to encourage the model to compress parameters as much as possible. Different from the existing incremental methods focusing on the invariant seman- tic concepts, our method is designed for the progressive understanding of the incremental semantic information, facilitating better feature update and retention.
Our Method
3.1 Problem Description and Analysis
In real life, the human brain gradually enriches the semantic understanding of an object as the learning process advances. For example, only know that rats small animals in the young phase, but we understand that rats are rodents as we grow up. To simulate this process, we follow the incremental -mplicitly-refined classification task (IIRC), which is arranged into N phases in the order of data stream with hierarchical labels, denoted as {Ti, ... Tv' J. Different from the traditional incremental learning settings, the samples of incremental phases in IIRC may have one or two labels (i.e., hierarchical labels). Specifically, the hierarchical labels refer to that the one label of sample is subclass (i.e., hamster), and the other one a superclass (i.e., small animal). The number of samples with superclass labels is always larger than that with subclass labels, which increases with the number of subclasses. During training, we always follow an incomplete information setup, i.e. samples with multiple labels are only provided with the label in the current phase. We only use subset of superclasses in the first phase to train the model. Subsequent phases contain a mixture of superclasses and subclasses.
3

---

Old Model (Phase
Features
Old Classes
Class-level Uncertainty Estimate
Distribution Alignment
Distillatin
Eatrg
Hierarchical
Refinement
Global Representation Extension
Aggreation
Current Model
Novel Features
Closs Centroids
Nevel Classes
Figure 3: Our proposed Uncertainty-Awan Hierarchical Refinement scheme for IIRC.
We face two challenging problems in this setting: (1) How to judge the relationship among labels across phases, especially the superelass-subclass relationship? Moreover, how to guide the optimiza- tion of the novel class by means of the distribution relationship with superclass exactly? (2) How do reduce the feature confusion among these novel classes in the incremental phase? Therefore, We propose two strategies. (1) Global Representation Extension. We use the RBF kernel to calcu- late the uncertainty among all class centers, which is minimized for training during the incremental phase. (2) Hicrarchical Distribution Alignment. We firstly calculate the output entropy of the new class samples with the old model, estimating the uncertainty of the classes across phases. Then use the obtained uncertainty to discriminate the superclass- subclaass relationships and other relation- ships. Finally, update the old model's 's output to guide the new model's optimization direction. Thus, we can keep the distribution invariance between the subclass and the superclass.
3.2 Global Representation Extension
Optimization with only the data of new classes during the incremental phase causes great bias in the representation and classifier learning. To maintain the stability of the feature space, we use the representation distance
measure uncertainty. New class representation is optimized by comparing the output entropy with the target labels. At the same time, the stability of the old class representation ensured by maintaining the same output between the old and new models with the new samples. Specifically, we use the RBF distance [19] to calculate the representation extension loss. We use the RBF kernel to map representation infinite dimensional space for comparison, which is shown follows,
1 Ri
[h((x)cc ho(x=)
Ldiv
K (() jncar)
exp
(1)
202
where K-)) denotes the Radial Basis Function kernel (RBF kernel [11]), n, denotes the number of classes contained in the current batch, (-(+) denotes the backbone of our net, we can get the output features of the picture from it, nd denotes the output dimension after global pooling, and ho(:)c denotes the average feature value of a particular class c. The symbol j represents the class number with the smallest distance class in the feature space, which is not equal to c. The optimization of this loss forces the extension of the distance among the class centers, obtaining better discrimination in the class distribution. Subsequently, we use a linear layer get the entropy of the samples, then judge the class of these samples. We adopt the BCEWithLogissL.oo to optimize the learning, which is shown follows,
N Lets Σ ly (fg(sigmoid((f((ho(z))))) + (1 y:) log(1 (f
where yi denotes the ground truth label of sample i, N denotes the batch size of training, J;; denotes the is sample, and Sa(-) denotes the classifier of the current model. It should be noted that the labels here as well as the final output of the model contain only the classes of the current phase. The optimization of the loss allows the model to eventually judge the label of sample by whether

---

Algorithm Acquisition of Hierarchical Semantic Relationship
Input:The outputs of the samples of class old model output, the number of samples height, the number classes for each phase Stage, the global pooled features of the sample class on old model feat, and the number of the current phase idr.
Declare an initial label matrix Mors of size (height, idz).
2: for each j E [0, height] do for each l E [0, idæ] do
3:
4:
Smaz, Liaaz max Stage[k]] + 1]])
5: Morilj. k] = Lmas
6: end for
7: end for
8: Declare phase label statistics matrix Mpha of size (2, idx).
9: for each j [0, idæ] do
10:
Lphas Cpha countmax(MMrrll:
11:
if Lyha < hheggtt/ / then
12:
Lipha =11
13:
end
14:
MphalO, j] Lhhe
15:
Mphall, ]]= Cpha
end for
17:
For all labels in the current Mpha that are not -1, an equal number of corresponding sample features are randomly extracted from f'eal, and the standard deviation calculated for each, forming Stdpha-
Calculate the standard deviation std for the current feat, declared as Stde. 19: Super = argmin((Stdpho Std,l)
Output:the superclass of current class C: Super.
its output is greater than 0. And the closer the corresponding representation is to the class center, the larger the output value. Finally, we use the output value of the new sample on the old model as the supervised signal of the distillation process. Thus, the current model learns classes while maintaining the knowledge of the old model, which is shown below:
Lass BCEWithLogissLoss( fe (z)) notal, fā (bā(z)/a)),
(3)
where h-(-) denotes the backbone of the current model, fo(-) denotes the classifier of the current model, (-)-) denotes the backbone of the old model, (-) denotes the classifier head of the old model, is scale hyperparameter of distillation, and naad denotes the number of classes that have been learned, æ a training sample from the current phase. The optimization of this loss facilitates the stability of the old class feature distribution.
3.3 Hierarchical Distribution Alignment
In the incremental phase, focusing on the label relationships is the key to solving the IIRC prob- lem. Firstly, the new subclass samples can be mapped into the embedding space by the old model with fixed parameters, which close the corresponding superclass distribution. Then the label relationships can be inferred from the hierarchical uncertainty, which is calculated by counting the entropy of new samples on the old model across phases. the same time, we flexibly adjust distribution border to correct the shift of subclass samples and mitigate the confusion caused by the brother class.
Specifically, as shown in Fig. 3, we take the new class C' as an example. Firstly, we obtain all mapped features and corresponding output entropy of each old class on the old model with the new samples of class C. Then, according to the obtained output entropy, the ratio of each old class distribution calculated. This ratio is the hierarchical uncertainty of the current added class C' samples each old class label. The obtained hierarchical uncertainty is binarized to determine the corresponding old class with similar distribution (possibly superclass-subee sublllssssss or brother class relationship) to the current specific class C. Finally, We calculate the standard deviation of each similar old class with the same operation. Direct superclass-subclan relationships have closer standard deviation values on the feature distributions. So the similar old class with the closest std value is judged as
5

---

IRFCR
GRE
HDA
phase 0
phase
phase 10
phase 15
phase 21
78.35
26.48
21.27
18.81
17.78
77.04
26.75
21.46
19.38
18.32
77.06
29.31
24.73
22.42
18.38
77.53
30.11
25,31
23,56
19.05
Table 1: Ablation study of our method on IIRC-CIFAR.
the superclass of the current class C, and the rest of the classes are brother classes. The detailed procedure is shown in Algorithm 1.
After obtaining the label relationships, use the selective distribution alignment distillation mech- anism to guide the representation learning of models. According to the labeling relationship. we can achieve the alignment of the distribution and mitigate the confusion caused by the brother class. On one hand, when the features of the novel sample have a superclass shift, its output entropy value of the corresponding superclass on the old model is improved by a margin distance. Thus, the shifting samples are guided within the corresponding superclass distribution. On the other hand, the output entropy value of novel sample that lacks a superclass but falls within the old class distribution is subtracted by margin distance. Specifically, according to the label relationships, when the current related an old class, the highest output entropy by the sample of class on the old model is subtracted by a margin value. Moreover, when the current new class C is subclass of an old class, the output entropy value of the class C sample on the old model, which corresponds to the superclass, is added with one margin value. At the same time, the highest output entropy value of the non-superclass is subtracted by a margin value. Our selective distribution alignment distillation loss can be obtained by:
new class C
Lais = BCEWitt BCEWitthoogisssosss(/////// (x)) mota]. y"("),
(4)
where ne the new output of the old model, after performing our hierarchical distribution align- ment strategy. In the end, our complete loss is:
Call = Cots + Ldis + Ldiv * Y
(5)
where y denotes hyper-peee parameters for balancing the losses. Moreover, in our experiments, 7 is set as 10.0. A detailed description of the hyperparameter selection is shown in supplementary material B.2.
4 Experiments
4.1 Dataset and Settings
According to the semantic relevance among labels, CIFAR100 [13] and ImageNet [6] datasets are re- arranged form the two-level hierarchy datasets [1]. Each label starts as leaf node (i.e., subclass), and similar labels are assigned a common parent node (i.c., superclass). The integrated datasets are called IIRC-CIFAR and IIRC-ImagNNet-full. IRC--CIAR. Ten superclasses are set up, each with about 4 to 8 subclasses. In incremental phases, each new phase introduces five classes. IIRC- CIFAR involves 22 phases with ten preset class orders called phase configuration for multiple tests. IRC-ImageNNe--ull. In MIRC-ImaaeNet-fui
sixty-three superclasses are set up, and the number of subclasses that belong to one superclass varies greatly, from 118. There are a total of 35 phases, with 30 classes per phase. Five preset class orders are fixed for multiple tests. IIRC-ImageNet shorter, lighter version with just ten phases (with five task configume onnigurationss, which is referred as IIRC-I ImageNet together with IIRC-ImaeNNet-flll IIRC-ImageNet- Net--mueeeettteeff [21], as a simplified version of the IIRC-ImagNNet-ful.. involves ten superclasses and 100 subclasses.
The superclasses of each dataset are combined by extracting 40% samples from each subclass, while the corresponding subclasses are saved with 80% samples. That means that the superelasses and sub- classes share 20% samples. When the superclass contains more than eight subclasses, the samples of the subclasses are extracted in the proportion of number of eubclasees 40%. The training process follows the incomplete information setting, i.e., if sample has more than one label, only the label of the current phase is provided. The validation set also follows the incomplete information setting.

---

IIRC-CIFAR
Method
phase 5
phase
phase 15
phase 21
Common Uncertainly
25.70
20.89
19.55
15.32
Ours
30.11
25.31
23.56
19.05
Table 2: Comparison common uncertainty method IIRC-CIFAR.
After a certain number of training phases, the overall performance is evaluated on validation set and test set under the complete information setting, which contains all seen class labels.
Metrics. The evaluation metric uses the precision-weighted Jaccard similarity (PW-JS) measure the performance of the model on phase k after training on phase j. as follows:
1
YR n Pkal YA Pkal
VKu n R
Ryk-
X
(6)
In the formula j 2 k. VEi is the prediction value of the model for ieh samples of kes phase, Y&4 is the ground truth, and nig is the number of samples in this phase. Moreover, for the evaluation of the overall performance,
use the average precision-weighee Jaccard similarity of the model after training on phase j for all seen classes, as follows:
1
YnnY Y: Y,
Rj
X
YUY.
4.2 Ablation Study
To prove the effectiveness of our method, we conduct set of ablation experiments on the IIRC- CIFAR dataset. One the essential components in our scheme is the global representation exten- sion strategy, referred to the GRE component. The other is the hierarchical distribution alignment strategy, referred to as the HDA component. To verify the functionality ponents, we conducted experiments in the case of phase configuration 0 of IIRC-CIFAR. As shown in Table 1, the HDA component alone achieves an improvement of 0.6% in the last phase, while the GRE component alone achieves 0.54% in the last phase. Furthermore, with both components added, a 1.27% increase over baseline is achieved. proves that both components play a positive role in evaluating the overall incremental performance.
the GRE and HDA com-
In Fig. 6 (C), we show the incremental performance at each phase, which evaluated over the test samples at specific phase j. after training on that phase( using Equation 6). Such cure graphs evaluate the learning ability for new classes. We can see that the HDA component significantly improves the average performance while the GRE component further enhances the learning ability on top of the HDA component.
4.3 Analysis
The impact of global representation extension. To explore the impact of the global representation extension strategy, we visualize the t-SNE features of the baseline and our method. We select data from phase 0 and phase 4 of IIRC-CIFAR with phase configuration O. In the upper part of Fig. 4, we can observe an overlap among the representation distributions of the new classes in the baseline method. Our method weakens the overlap and expands the distribution distance among the novel class. Fig. 4 proves that the usage representation distance for the uncertainty estimate is adequate. Reducing this uncertainty leads to the distance extension between the individual class centers.
The impact of the hierarchical uncertainty. To explore the effect of hierarchical uncertainty, we first compare different uncertainty methods. A detailed version of Table 2 is presented in A.3 of the supplementary materials. As shown in Table. 2, the "Common Uncertainty" denotes an entropy- based uncertainty method widely used in the OOD [5]. We use it to construct label relationships. It can be seen from the Table. that our method achieves better experimental results, proving that the entropy output and the standard deviation of the features exhibit the most significant impact on the cross-hierarchical uncertainty estimation among the statistical properties of the superclass and the corresponding subclass.
7

---

Stuge
Baseline
Ours
Clasaes
Superclasses
Subclasses
Figure 4: The impact of our method on the representations. (1) The upper shows the effect of extending the distribution distance between the incremental classes. (2) The lower row shows the effect of the hierarchical distribution alignment.
(A Grouund Truth
(OiCaRL-CNN Figure 5: Confusion matrix for different methods.
(B) Tune
(D)Ours
To explore the effect of the hierarchical distribution alignment strategy, we visualize the t-SNE features for the baseline and our method on the IIRC-CIFAR dataset with phase configuration by selecting data from phase 0 and phase 11. In the bottom half of Fig. 4, we can see a significant shift between the distributions subclasses and corresponding superclasses in the baseline. While our method extensively corrects this shift and promotes the distribution of the novel subclasses falling the corresponding superclasses. demonstrates that adjusting the optimization direction of the feature distribution according to the labeling relationship helps the new model to learn better the representation relationship between the old and new classes and deepen it gradually.
4.4 Comparison with SOTA
To comprehensively measure the performance of our method under IIRC setting, we reproduces the classical methods (i.e., ER [17], AGEM [4], LUCIR [10], iCaRL-CNN [16], iCaRL-norm [16], Podnet [8]. HCV [21]) and compare with them.
be seen in Fig. 6 (A) that method achieves the best until the end among all methods with incomplete information in IIRC-CIFAR, As shown in Fig. 6 (D), our method also maintains the highest scores in larger IIRC-Imaenet--ltte demonstrating the effectiveness of our method. As shown in Fig. 6 (E), our method achieves the SOTA result on the IIRC-ImageNet- Subset dataset. It be shown in Fig. 6(F) that our method finally achieves the highest score IRCC-mageNNet-flll. More details are shown in A.5 of the supplementary materials.
Average PW-JS.
Performance new classes. The PW-JS values indicating the performance of the newly incremen- tal phase are shown in Fig. 6(B). can be seen that the ER algorithm and the AGM algorithm are superior in learning new classes. Combine with the performance in overall phases, they obtain an advantage in the new classes at the expense the performance in old classes. Among all beneficial algorithms for suppressing catastrophic forgetting, method achieves the best and outperforms existing methods in learning new classes.
We conduct an additional experimental set the confusion matrix, where we extract the data from the TRC-CIFAR with phases 0, 1, 2, 7, 12, and 17. They are combined and presented in Fig. 5. In the HRC task, the diagonal values in the confusion matrix represent the prediction of subclass

---

02
AGM
mm
0.5-
0.44
0.3
0.7
001
20
60
80
103
120
10
15
Number of Deserved Classes
Current Phase Number
0.8
Baselite
0.0
Baselite
0.7
Bacelite IEA
Bacellie CAl 1BA
0.6
0.-
0.5
0.4
0.3-
0.2
0.2
0.2
0
10
15
20
Current Phose Number
(C)
(D)
80-
Coll
0.4
60
05-
0.4-
0.1
0.8
20
Current Phase Number
Current Rhone Number
Figure 6: Curve graphs of the comparison experiments. (A) The overall performance comparison on the IIRC-IIAR dataset. (B) The performance of the incremental phase the IIRC-CIFAR dataset. (C) It shows the performance of two components of our method on the evaluation of new class learning ability. (D) The performance comparison on the IRC-ImageNet--lie (E) performance comparison on the IIRC- SICC-ImageNet-Subbsee dataset. (F)The performance comparison on the IIRC- HIRC-ImageNet-Fil dataset. indicates our re-implementation.
labels, while the bottom-left corner represents one of the superclasses. A more precise confusion matrix in the lower left corner represents better label relationship maintenance between subclass and superclass. As shown in Fig. 5, our method performs best in both the diagonal and lower left positions, proving the superiority of our method in maintaining label relationships.
Conclusion
This paper proposes a novel Uncertainty-Awart Hierarchical Refinement scheme for the IIRC task. global representation extension strategy is presented to enhance the discrimination of incremen- tal classes, and the tricky distillation process is refined with hierarchical distribution alignment strategy. Consequently, our method involves multi-level semantic scenery in incremental learning- Experimental results show the superiority of our method in both stability and plasticity.

---

Acknowledgments and Disclosure of Funding
Supported by the National Key R&D Program of China under Grant 2020AAA0105701, the Na- tional Natural Science Foundation of China (NSFC) under Grants 61872327, and the Major Special Science and Technology Project Anhui (No. 012223665049). This work was supported by Ant Group through Ant Research Intern Program.
References
[1] Mohamed Abdelsalam, Mojtaba Faramarzi, Shagun Sodhani, and Sarath Chandar. lire: In- cremental implicitly-refined classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11038-11047, 2021.
[2] Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuyte- laars. Memory aware synapses: Learning what (not) forget. In Proceedings of the European Conference Computer Vision (ECCV), pages 139-154, 2018
[3] Junyi Chai, Hao Zeng, Anming Li, and Eric WT Ngai. Deep learning in computer vision: A critical review of emerging techniques and application scenarios. Machine Learning with Applications, 6:100134, 2021.
[4] Arslan Chaudhry, Marc' Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Effi cient lifelong learning with a-gem. arXiv preprint arXiv: 18/w.00422 2018.
[5] Wenhu Chen, Yilin Shen, Hongxia Jin, and William Wang. A variational dirichlet framework for out-of-distribution detection. arXiv preprint arXiv:1811.03300 2018.
[6] Jia Deng, Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255 Iece, 2009.
[7] Prithviraj Dhar, Rajat Vikram Singh, Kuan-Chuan Peng. Ziyan Wu, and Rama Chellappa. Learning without memorizing. In Proceedings the IEEE/CVF conference computer vision and pattern recognition, pages 1138-5146, 2019.
[8] Arthur Douillard, Matthieu Cord, Charles Ollion, Thomas Robert, and Eduardo Valle. Podnet: Pooled outputs distillation for small-tasks incremental learning. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XX 16, pages 86-102. Springer, 2020.
[9] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communica- lions of the ACM, 63(1 13(114413,-14, 2020.
[10] Saihui Hou, Xinyu Pan, Chen Change Loy. Zilei Wang, and Dahua Lin. Learning a uni- fied classifier incrementally via rebalancing. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 131--39, 2019.
[11] Wladyslaw Kaminski and Pawel Strumillo. Kernel orthonormalization in radial basis function neural networks. IEEE Transactions on Neural Networks, 8()):1177-1183, 1997.
[12] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, An- drei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Bann Barwinska, et al. Overcoming catastrophic forgetting neural networks. Proceedings of the national academy of sciences, 1 114(13):3521-3526 2017.
[13] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[14] Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40(12):2935-2444 2017.
10

---

[15] Vinay Venkatesh Ramasesh, Aitor Lewkowyez. and Ethan Dyer. Effect of scale on catastrophic forgetting in neural networks. International Conference on Learning Representations, 2021.
[16] Selvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 2001-2010, 2017.
[17] David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne, Ex- perience replay for continual learning. Advances in Neural Information Processing Systems, 32, 2019.
[18] Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim. Continual learning with deep generative replay. Advances in neural information processing systems, 30, 2017.
[19] Joost Van Amersfoort, Lewis Smith, Yee Whye Teh, and Yarin Gal. Uncertainty estimation using a single deep deterministic neural network. In International conference on machine learning, pages 9690-9700. PMLR, 2020.
[20] Gido M Van de Ven and Andreas S Tolias. Three scenarios for continual learning. arXiv preprint arXiv: 1904.07734, 2019.
[21] Kai Wang, Xialei Liu, Luis Herranz, and Joost van de Weijer. Hev: Hicrarchy-consistengg verification for incremental implicitly-refined classification. arXiv preprint arXiv:2110.11148 2021.
[22] Kun Wei, Cheng Deng, Xu Yang, et al. Lifelong zero-shot learning.
[23] Shipeng Yan, Jiangwei Xie, and Xuming He. Der: Dynamically expandable representation for class incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Viston and Pattern Recognition, pages 3014-3023, 2021.
[24] Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intel- ligence. In International Conference on Machine Learning, pages 3987-3995. PMLR, 2017.
[25] Yu Zhang and Qiang Yang. A survey multi-task learning. IEEE Transactions Knowledge and Data Engineering, 2021.
[26] Kai Zhu, Yang Cao, Wei Zhai, Jie Cheng, and Zheng-Jun Zha. Self-promoted prototype refine- ment for few-shot class-incremental learning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 6801-6810, 2021.
[27] Kai Zhu, Wei Zhai, Yang Cao, Jiebo Luo, and Zheng-Jun Zha. Self-sustainnng representa- tion expansion for non-exemplar class-incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 9296-9305, 2022.
Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information on how answer these questions. For each question, change the default [TODO] [Yes], [No], or [N/A]. You are strongly encouraged to include justification to your answer. either by referencing the appropriate section of your paper or providing brief inline description. For example:
Did you include the license to the code and datasets? [Yes] See Section 2.
Did you include the license to the code and datasets? [No] The code and the data are proprictary.
Did you include the license to the code and datasets? [N/]]
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Supplementary Materials.
11

---

(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Supplementary Materials.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or URL)? [Yes] See Supple- mentary Materials.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Supplementary Materials.
4. If you are using existing assets (e.g., code, data, models) or curating/releasigg new assets...
(a) your work uses existing assets, did you eite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you'l re using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifi- able information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
12


Reviews:
[{"review_id": "vGAjZmbV8sI", "reviewer": "Reviewer_o6FB", "summary": "This paper proposes a method called UAHR to address the task of incremental implicitly refinement classification. Specifically, two components, Global Representation Extension (GRE) and Hierarchical Distribution Alignment (HDA) are proposed and integrated together to measure the uncertainty and align the distribution. Experimental studies on public datasets including IIRC-CIFAR and IIRC-ImageNet-lite show the effectiveness of the proposed UAHR scheme.", "questions": "While the paper is well motivated and the ablation study is extensive. There are still several issues with this paper.\n\n1, experiments issues\n\na, in table 1 of the ablation study, in phase 0, why the results of variants with additional one or two of the proposed components are worse than the results of the plain baseline? At the same time, adding GRE seems always harms the performance? How to interpret it?\n\nb, again, in subfigure c of fig. 5,  it seems the major improvement is from the refine component, and the DIV can hardly improve the final performance. This needs explanations.\n\nc, the comparison with SOTA methods is not sufficient. There should be quantitative results (not shown in figure). \n\nd, as subfigures a and b show, the results achieved by the proposed method are not strong, worse than several SOTA methods.\n\n2, missing details\n\na, in equation 5, why $l_{cls}$ and $l_{dis}$ share the same weight? How is $\\gamma$ determined?\n\nb, for the settings of ImageNet, both reference 18 and the following BMVC paper adopt the setting of 10 super classes and 100 subclasses, why do not you follow them to make a fair comparison?\n\n[ref a] HCV: Hierarchy-Consistency Verification for Incremental Implicitly-Refined Classification\n\n3, poor representation\n\na, there are many long sentences in the paper, making the paper very hard to follow. And there are many grammar issues. I will list some in the following, but they are definitely not all.\n\nb, the figures and tables in the submission should be self-contained. Each equation should be followed by a period or comma. \n\nc, minor: caption in fig. 2, \"The class with inheritance relationships have consistency\", L169, \"we flexibly adjusting \", L175, \"And then making the number divide by the number of ...\", L176, \"The label that final result is 1 has a ...\", caption in fig. 4, \"The upper row shows the effect of extend the ...\"", "limitations": "As indicated in the above sections, there are issues with experimental studies, representations, and missing details. For details, please see the above sections.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "There are several strengths of the proposed UAHR.\n\n+ The paper is well motivated.\n+ The ablation study is extensive.\n+ There are many intuitive visualizations of the intermediate results, enhancing the credit of the motivation.\n\nAlso, some weaknesses exist.\n\n- The experimental comparison with the SOTA methods is not sufficient.\n- The representation of the paper is poor.\n- Details are missing.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "ig8stBzEsVt", "reviewer": "Reviewer_wtSW", "summary": "This paper proposes an uncertainty-aware hierarchical refinement scheme for incremental implicitly-refined classification, exploring inheritance relations of multi-level semantic increment. It consists of a global representation extension module and a hierarchical distribution alignment module.", "questions": "1.\tMore comparison experiments should be conducted on other datasets such as IIRC-ImageNet-full to further verify the effectiveness of proposed method, which is commonly used in IIRC tasks.\n2.\tThe effectiveness of the GRE module is not verified in ablation studies, according to the results in Table 1. More detailed analyses should be given.\n3.\tMost part of the performance decrease happens in first 5 phases, focusing on how to maintain more robust performance in the beginning may lead to better results in latter phases.\n4.\tIn Table 2, the common uncertainty method should be introduced briefly, so the comparison can verify the effectiveness of proposed method. Numerical results of comparison experiments should be provided to better illustrate the improvement of proposed method.\n5.\tMore details should be provided as explanation for Figure 6, the present version cannot clearly show the improvement in maintaining label relations.\n", "limitations": "The authors should give more introduction and limitation analyses about the proposed method in realistic application scenarios.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1.\tThis paper introduces global representation extension and hierarchical distribution alignment to the IIRC task for better hierarchical invariance, and conducts experiments on IIRC-CIFAR dataset to verify its effectiveness.\n2.\tThis paper makes progress in a newly proposed task, IIRC, which focus on the robustness of models when processing data of different categories and different-grained classes.\n\nWeakness:\n\n1.\tThere is only one dataset for overall performance comparison experiments. More experiments and analyses on more datasets should be added to verify the effectiveness of the proposed method.\n2.\tThe effectiveness of separate module is not fully verified. The ablation results of the GRE module in Table 1 is lower than the baseline, more explanations should be given.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "LC5cWpTDRWd", "reviewer": "Reviewer_qkcE", "summary": "Inspired by recent Incremental Implicitly-Refined Classification (IIRC) works, this paper proposes an Uncertainty-Aware Hierarchical Refinement (UAHR) method, which explores the semantic correlation of different granularity levels of image labels to ensure the consistency assumption of the feature distribution of class from different granularity levels. Technically, a global representation extension strategy is developed to boost the inter-class discriminability of newly-added classes, and a hierarchical distribution alignment strategy is designed to ensure distribution consistency across different classes belonging to the same super-class. Experiments are conducted on IIRC-CIFAR and IIRC-ImageNet-lite datasets.", "questions": "**Major Issues**\n>+ Why the RBF kernel can effectively measure the uncertainty of inter-class features.\n>+ What are the differences between the KD method used in this paper and the commonly-used KD methods during the rebuttal period.\n>+ Inconsistency of experimental results in Figure 5.\n\n**Minor Comments**\n>+ In Eq. 1, the definition of K is missing.\n>+ Some typos, e.g., Line 145, Page 5. The mathematical symbols j should be edited as the math format rather than the text format.\n>+ In Line 270, Page 9, the citation number of SOTA methods (such as ER, AGEM, LUCIR) is missing.\n", "limitations": "Technical contribution: It is interesting that the authors find that, the current incremental training phase would destroy the feature distribution consistency between super- and sub-classes. However, the part about the global representation extension strategy in the paper is relatively weak, due to the following two reasons:\n- the authors did not explain why the RBF kernel can effectively measure the uncertainty of inter-class features.\n- The knowledge distillation (KD) loss between the current model and old model is a widely-used technique in the class-incremental learning field [Ref 1, 2, 3]. I feel that the author seems to directly apply the KD loss to solve the knowledge forgetting issue, and the KD loss is not adapted according to the IIRC task, which reduces my rating of technical contribution for this paper. Please explain the differences between the KD method used in this paper and the commonly-used KD methods during the rebuttal period. \n\nTechnical details:\n- The authors use the RBF kernel to calculate the uncertainty between different classes. Why choose the RBF kernel to calculate inter-class uncertainty? Will the effectiveness of the method decline without the RBF mapping. \n- Actually, Eq. 1 presents that the mean diversity across all novel class-centers is formulated. But different classes may have different importance for IIRC tasks. As a result, is a class-wise diversity uncertainty metric better?\n- In Line 187, Page 5, the authors state that the output entropy of the old classes needs to be subtracted by a margin distance value. It is unclear how to obtain such a margin distance.\n\nExperiments:\n- The work [Ref 4] was also conducted on the IIRC-CIFAR and IIRC-ImageNet-lite datasets. But this paper did not compare with the original paper’s results in the work [Ref 4]. Besides, the definition of the horizontal axis in Figure 5 is also inconsistent with that in Figures 3-5 in [Ref 4]. Please explain the reason for this inconsistency during the rebuttal period.\n\n[Ref 1] Semantic-aware knowledge distillation for few-shot class-incremental learning. CVPR-2021\n[Ref 2] Few-shot class-incremental learning via relation knowledge distillation. AAAI-2021\n[Ref 3] Class-Incremental Learning by Knowledge Distillation with Adaptive Feature Consolidation. CVPR-2022\n[Ref 4] IIRC: Incremental Implicitly-Refined Classification. CVPR-2021\n", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "1. This paper finds the feature distribution consistency across the classes with inheritance relationships. Further, to guide the optimization of the incremental learning process, this paper designs the hierarchical distribution alignment strategy to exploit the correlation of hierarchical class distributions. \n2. Ablation studies show the effectiveness of each proposed module.\n3. This paper is well-written, clearly presenting the main idea.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "9V8N4IL_Qea", "reviewer": "Reviewer_cq21", "summary": "This paper tackles the Incremental Implicitly-refined classification by proposing a framework that is based on Uncertainty-Aware Hierarchical Refinement. In different stages, label relationships are better leveraged by the proposed method for semantic reasoning, and the uncertainty further assists this inference process. The proposed method achieves competitive performance in two popular benchmarks.", "questions": "My major concern is about the incremental novelty of this method. In the few-shot learning community, both class hierarchy with global class representations and the entropy-assisted class margin control have been well exploited. \n\nFrom the reviewer's perspective, this paper should properly discuss these related methods with necessary statistical comparison. ", "limitations": "The limitations have been discussed in the appendix.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\n+ Good paper writing and motivation to construct the class hierarchy.\n\n+ Clear illustrations in Figure 4 and Figure 5 that facilitate understanding.\n\nWeaknesses are mainly related to the novelty:\n\n- The hierarchical class structure has been adopted in[1]. \n\n- The use of global representation has also been leveraged in [2]. \n\n- Entropy-based margin controller has been proposed in [3].\n\n[1] Large-Scale Few-Shot Learning: Knowledge Transfer With Class Hierarchy. CVPR 2019\n[2] Few-Shot Learning With Global Class Representations. ICCV 2019\n[3] Spot and Learn: A Maximum-Entropy Patch Sampler for Few-Shot Image Classification. CVPR 2019\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept