Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes an uncertainty-aware hierarchical refinement scheme for incremental implicitly-refined classification, exploring inheritance relations of multi-level semantic increment. It consists of a global representation extension module and a hierarchical distribution alignment module.
strengths_and_weaknesses: Strengths:

1.	This paper introduces global representation extension and hierarchical distribution alignment to the IIRC task for better hierarchical invariance, and conducts experiments on IIRC-CIFAR dataset to verify its effectiveness.
2.	This paper makes progress in a newly proposed task, IIRC, which focus on the robustness of models when processing data of different categories and different-grained classes.

Weakness:

1.	There is only one dataset for overall performance comparison experiments. More experiments and analyses on more datasets should be added to verify the effectiveness of the proposed method.
2.	The effectiveness of separate module is not fully verified. The ablation results of the GRE module in Table 1 is lower than the baseline, more explanations should be given.

questions: 1.	More comparison experiments should be conducted on other datasets such as IIRC-ImageNet-full to further verify the effectiveness of proposed method, which is commonly used in IIRC tasks.
2.	The effectiveness of the GRE module is not verified in ablation studies, according to the results in Table 1. More detailed analyses should be given.
3.	Most part of the performance decrease happens in first 5 phases, focusing on how to maintain more robust performance in the beginning may lead to better results in latter phases.
4.	In Table 2, the common uncertainty method should be introduced briefly, so the comparison can verify the effectiveness of proposed method. Numerical results of comparison experiments should be provided to better illustrate the improvement of proposed method.
5.	More details should be provided as explanation for Figure 6, the present version cannot clearly show the improvement in maintaining label relations.

limitations: The authors should give more introduction and limitation analyses about the proposed method in realistic application scenarios.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: Inspired by recent Incremental Implicitly-Refined Classification (IIRC) works, this paper proposes an Uncertainty-Aware Hierarchical Refinement (UAHR) method, which explores the semantic correlation of different granularity levels of image labels to ensure the consistency assumption of the feature distribution of class from different granularity levels. Technically, a global representation extension strategy is developed to boost the inter-class discriminability of newly-added classes, and a hierarchical distribution alignment strategy is designed to ensure distribution consistency across different classes belonging to the same super-class. Experiments are conducted on IIRC-CIFAR and IIRC-ImageNet-lite datasets.
strengths_and_weaknesses: 1. This paper finds the feature distribution consistency across the classes with inheritance relationships. Further, to guide the optimization of the incremental learning process, this paper designs the hierarchical distribution alignment strategy to exploit the correlation of hierarchical class distributions. 
2. Ablation studies show the effectiveness of each proposed module.
3. This paper is well-written, clearly presenting the main idea.


questions: **Major Issues**
>+ Why the RBF kernel can effectively measure the uncertainty of inter-class features.
>+ What are the differences between the KD method used in this paper and the commonly-used KD methods during the rebuttal period.
>+ Inconsistency of experimental results in Figure 5.

**Minor Comments**
>+ In Eq. 1, the definition of K is missing.
>+ Some typos, e.g., Line 145, Page 5. The mathematical symbols j should be edited as the math format rather than the text format.
>+ In Line 270, Page 9, the citation number of SOTA methods (such as ER, AGEM, LUCIR) is missing.

limitations: Technical contribution: It is interesting that the authors find that, the current incremental training phase would destroy the feature distribution consistency between super- and sub-classes. However, the part about the global representation extension strategy in the paper is relatively weak, due to the following two reasons:
- the authors did not explain why the RBF kernel can effectively measure the uncertainty of inter-class features.
- The knowledge distillation (KD) loss between the current model and old model is a widely-used technique in the class-incremental learning field [Ref 1, 2, 3]. I feel that the author seems to directly apply the KD loss to solve the knowledge forgetting issue, and the KD loss is not adapted according to the IIRC task, which reduces my rating of technical contribution for this paper. Please explain the differences between the KD method used in this paper and the commonly-used KD methods during the rebuttal period. 

Technical details:
- The authors use the RBF kernel to calculate the uncertainty between different classes. Why choose the RBF kernel to calculate inter-class uncertainty? Will the effectiveness of the method decline without the RBF mapping. 
- Actually, Eq. 1 presents that the mean diversity across all novel class-centers is formulated. But different classes may have different importance for IIRC tasks. As a result, is a class-wise diversity uncertainty metric better?
- In Line 187, Page 5, the authors state that the output entropy of the old classes needs to be subtracted by a margin distance value. It is unclear how to obtain such a margin distance.

Experiments:
- The work [Ref 4] was also conducted on the IIRC-CIFAR and IIRC-ImageNet-lite datasets. But this paper did not compare with the original paper’s results in the work [Ref 4]. Besides, the definition of the horizontal axis in Figure 5 is also inconsistent with that in Figures 3-5 in [Ref 4]. Please explain the reason for this inconsistency during the rebuttal period.

[Ref 1] Semantic-aware knowledge distillation for few-shot class-incremental learning. CVPR-2021
[Ref 2] Few-shot class-incremental learning via relation knowledge distillation. AAAI-2021
[Ref 3] Class-Incremental Learning by Knowledge Distillation with Adaptive Feature Consolidation. CVPR-2022
[Ref 4] IIRC: Incremental Implicitly-Refined Classification. CVPR-2021
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes a method called UAHR to address the task of incremental implicitly refinement classification. Specifically, two components, Global Representation Extension (GRE) and Hierarchical Distribution Alignment (HDA) are proposed and integrated together to measure the uncertainty and align the distribution. Experimental studies on public datasets including IIRC-CIFAR and IIRC-ImageNet-lite show the effectiveness of the proposed UAHR scheme.
strengths_and_weaknesses: There are several strengths of the proposed UAHR.

+ The paper is well motivated.
+ The ablation study is extensive.
+ There are many intuitive visualizations of the intermediate results, enhancing the credit of the motivation.

Also, some weaknesses exist.

- The experimental comparison with the SOTA methods is not sufficient.
- The representation of the paper is poor.
- Details are missing.
questions: While the paper is well motivated and the ablation study is extensive. There are still several issues with this paper.

1, experiments issues

a, in table 1 of the ablation study, in phase 0, why the results of variants with additional one or two of the proposed components are worse than the results of the plain baseline? At the same time, adding GRE seems always harms the performance? How to interpret it?

b, again, in subfigure c of fig. 5,  it seems the major improvement is from the refine component, and the DIV can hardly improve the final performance. This needs explanations.

c, the comparison with SOTA methods is not sufficient. There should be quantitative results (not shown in figure). 

d, as subfigures a and b show, the results achieved by the proposed method are not strong, worse than several SOTA methods.

2, missing details

a, in equation 5, why $l_{cls}$ and $l_{dis}$ share the same weight? How is $\gamma$ determined?

b, for the settings of ImageNet, both reference 18 and the following BMVC paper adopt the setting of 10 super classes and 100 subclasses, why do not you follow them to make a fair comparison?

[ref a] HCV: Hierarchy-Consistency Verification for Incremental Implicitly-Refined Classification

3, poor representation

a, there are many long sentences in the paper, making the paper very hard to follow. And there are many grammar issues. I will list some in the following, but they are definitely not all.

b, the figures and tables in the submission should be self-contained. Each equation should be followed by a period or comma. 

c, minor: caption in fig. 2, "The class with inheritance relationships have consistency", L169, "we flexibly adjusting ", L175, "And then making the number divide by the number of ...", L176, "The label that final result is 1 has a ...", caption in fig. 4, "The upper row shows the effect of extend the ..."
limitations: As indicated in the above sections, there are issues with experimental studies, representations, and missing details. For details, please see the above sections.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper tackles the Incremental Implicitly-refined classification by proposing a framework that is based on Uncertainty-Aware Hierarchical Refinement. In different stages, label relationships are better leveraged by the proposed method for semantic reasoning, and the uncertainty further assists this inference process. The proposed method achieves competitive performance in two popular benchmarks.
strengths_and_weaknesses: Strength:

+ Good paper writing and motivation to construct the class hierarchy.

+ Clear illustrations in Figure 4 and Figure 5 that facilitate understanding.

Weaknesses are mainly related to the novelty:

- The hierarchical class structure has been adopted in[1]. 

- The use of global representation has also been leveraged in [2]. 

- Entropy-based margin controller has been proposed in [3].

[1] Large-Scale Few-Shot Learning: Knowledge Transfer With Class Hierarchy. CVPR 2019
[2] Few-Shot Learning With Global Class Representations. ICCV 2019
[3] Spot and Learn: A Maximum-Entropy Patch Sampler for Few-Shot Image Classification. CVPR 2019



questions: My major concern is about the incremental novelty of this method. In the few-shot learning community, both class hierarchy with global class representations and the entropy-assisted class margin control have been well exploited. 

From the reviewer's perspective, this paper should properly discuss these related methods with necessary statistical comparison. 
limitations: The limitations have been discussed in the appendix.
review_title: null

Meta-review: metareview: Three reviewers are positive on this paper. Although the rating of one reviewer is borderline reject,  he is fairly confident in his assessment.  In effect,the authors have well addressed all the reviewers' concerns in the rebuttal.  So I suggest accepting this paper. 
confidence: Certain
Decision: Accept