Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: In this paper, the authors find that the main problem with the rehearsal methods is decision boundary overfitting to new classes. To solve this problem, they propose to combine the mixup and re-sampling strategy to synthesize adequate data used in knowledge distillation. Extensive experiment results are provided.
strengths_and_weaknesses: ### Strengths 

- The proposed method is technically sound. 

- This paper is well-organized and easy to follow.

- Extensive experimental results and visualizations are provided.

&nbsp;

### Weaknesses

- **The authors only provide results on small-scale datasets, e.g., CIFAR-100.** However, it is quite common to evaluate class-incremental learning on large-scale datasets. For example, iCaRl [12], BiC [8], LUCIR [10], Mnemonics [51], and PODNet-CNN [52] all provide result on ImageNet (full size, 1000 classes). Thus, it is not reasonable to ignore large-scale datasets for a top-tier conference submission. 

&nbsp;

- **The technical novelty of this paper is somewhat limited.** This paper is based on re-sampling and mixup. Both strategies have already been widely used in many related topics, such as long-tailed recognition and few-shot learning. Thus, the technical contributions of this paper are somewhat limited. 

&nbsp;

- **The state-of-the-art method is compared in Table 1.** For example, [A] archives much better performance than other baselines, but it is not compared in this paper. It seems the proposed method cannot achieve better performance than [A]. 

&nbsp;

[A] Yan, Shipeng, Jiangwei Xie, and Xuming He. "Der: Dynamically expandable representation for class incremental learning." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.


&nbsp;

=== **After rebuttal** ===

Thanks for the active responses from the authors. The authors provide some new results in the rebuttal period and address some of my concerns. However, my major concerns about large-scale datasets and the technical novelty still remain. Thus, I tend to keep my initial rating. 

questions: Please address the issues raised in the "weaknesses" part.
limitations: It would be better to add a section to discuss the potential negative societal impact.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper addresses the problem of class incremental learning with an emphasis on adjusting the decision boundary. The authors claim that the deficiency of old data and the imbalanced training mainly cause the ineffective decision boundary. The paper proposes Re-MKD, which combines re-sampling strategy and Mixup, and Incremental Influence Balanced (IIB) loss to tackle the aforementioned problem. The experimental results on various benchmarks show the effectiveness of the proposed framework.
strengths_and_weaknesses: - Strengths
    - Overall, the paper is well-written and clear. The authors’ claim that imbalanced training is critical issue for the ineffective decision boundary is well supported throughout the paper. Both proposed methods (Re-MKD and IIB loss) may not be super-novel, but they are reasonable solution for the problem. In particular, the motivation of the proposed solutions are clear. Re-MKD makes good use of the rate rebalancing of the old data and can populate many diverse examples from the old data. IIB also seems suitable for the class incremental learning since class incremental learning can be viewed as long-tail problem, where the old data is extremely scarce.

- Weaknesses
    - I have an issue with the claim made in the paper, which leave some room for improvement.
        - The authors repeatedly emphasize that data of newly added classes, which are OOD data from the perspective of the previously learned classes, harm the knowledge distillation process. However, the proposed framework still uses the mixed data of the newly-added data for knowledge distillation. It would be nice if the paper includes the ablation study about the 3 types of mixing classes mentioned in Line 158-159. In other words, I wonder how the overall incremental learning performance will be affected when we exclude the mixed data of the newly added classes from computing $L_{kd}$.
    - There are some serious formatting issues throughout the paper.
        - Font size of the captions and equations is significantly smaller than the original submission format. This should be revised.
    - Minor notation error
        - The notation $L$ is used multiple times in different contexts. In general, $L$ denotes the loss, but in Line 188, $L$ denotes the feature dimension. Moreover, $f$ in Line 191 also needs to be corrected since $f$ does not indicate the dimension of the feature vector.
questions: - As mentioned in the weakness section, I wonder how the overall incremental learning performance would be affected when we exclude the mixed data of the newly added classes when computing $L_{kd}$.
- Why is the IIB factor only multiplied to $L_{ce}$? Since $L_{kd}$ also can affect the overall decision boundary and IIB factor also considers $L_{kd}$ as in eq (10), IIB weighting factor may be helpful to $L_{kd}$ in eq (16), too. If possible, it would be nice to see how the final performance will be affected if IIB factor is also considered to $L_{kd}$. Or, if I missed something, please let me know.
- Conventionally, many CIL methods [9, 10, 52] utilize additional “class balanced fine-tuning (CBF).” It seems that the second phase of EDBL algorithm, which is balanced training, can be a good replacement for CBF, since CBF only utilizes small number of examples unlike the proposed framework. Thus, it would be nice if the ablation study of balanced training is added to the paper. Or, is “Re-MKD” row in Table 3 indicates the Re-MKD + CBF?
- Figure 3 can be improved if it contains more classes than two classes. Even though it is a minor issue for me, it seems demonstrating only a single case may weaken the claim of the paper.
limitations: The paper sufficiently deals with the limitation of the paper and broader impacts.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The authors address the problem of imbalance between old and new
classes in class-incremental learning due to the limited memory for
old classes.  For re-sampling, they added 3 types of Mixup instances:
among old classes, between old and new classes, and among new classes,
with proportionally more for the first 2 types to yield balanced
classes.  The loss function has cross entropy and knowledge
distillation.  They adapted influence balanced (IB) method to include
both the classification and knowledge distillation terms from the loss
function, which they call IIB.  They further use hyperparameter alpha
to weight the IIB factor for knowledge distillation (to decompose
knowledge distillation and cross entropy).  The loss function L_IIB is
lambda * L_ce / IIB, where lambda is class-wise reweighting.

The overall 2-phase EDBL algorithm combines mixing and IIB.  The first
phase has knowledge distillation.  The second phase has L_IIB, which
includes classification and knowledge distillation.

With 3 datasets, empirical results indicate the the proposed EDBL
generally outperforms a number of existing techniques.  An ablation
study also indicates the contribution of each of the proposed
components.

strengths_and_weaknesses: Strengths

The combination of Resampling with Mixup and IIB seem interesting.

Empirical results indicate the the proposed EDBL generally outperforms
a number of existing techniques.

Weaknesses

While the paper is generally well written, the motivation and
description of the overall 2-phase EDBL algorithm are not clear.  I
had to consult Algorithm 1 in the Appendix.  It could be improved by
describing and motivating the 2 phases first, and then the respective
loss functions that are involved.

Phase 2 can be considered as a superset of Phase 1--the motivation for
the need of Phase 1 could be further explained. (See questions below)

questions: According to Algorithm 1 in the Appendix, the main difference between
Phase 1 (MKD) and Phase 2 (Balancing training) is whether IIB is
incorporated or not.  Why not having only Phase 2?  A comparison of
the last 2 rows in Table 3 (if I understand correctly) seems to
indicate the benefit of adding Phase 1 (but seems to be not discussed
in Section 5.5.1).  Further insights would be important.

Eq 11: the first term seems to be missing 2* for f^t(x) and h. If so,
Eq 13 and Eq 15 seem to have similar issues.

Lines 223, 234, and 236: balancing training ->  balanced training?

Phases are used in both EDBL and CIL.  Using different terms could be
easier to separate the two concepts; e.g. stages in EDBL and phases in
CIL.

limitations: Limitations and negative societal impact were discussed.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper first points out that the scantness of exemplars for the old classes and OOD between the previous classes and the new classes
cause insufficient knowledge distillation (KD). Then it proposes a two-phase method (EDBL) for class incremental learning. In the first phase, re-sampling strategy and Mixup Knowledge Distillation are used to generate mixed data and to form a better decision boundary.  The second phase uses the incremental influence-balanced (IIB) method to mitigate the imbalanced learning problem.  Experiments show that EDBL outperforms baselines by a large margin.
strengths_and_weaknesses: Originality: The idea of generating mixed data by mixup using samples from old classes and new classes has been used in class incremental learning [1].  So that is not a new contribution. The introduction of the Influence Balance method and the decomposition of the Incremental Influence Balance factor are novel. The novelty of this paper is somewhat reduced.

[1] Class-Incremental Learning via Dual Augmentation, NeurIPS2021.

Quality: This paper is technically sound and the claims are supported. However, the experimental setup is flawed. The authors should discuss and compare more batch continual learning methods which use the KD mechanism [2,3,4].

[2] Dark experience for general continual learning: a strong, simple baseline, NeurIPS2020

[3] Co2L: Contrastive Continual Learning, ICCV2021

The method outperforms many baselines in the setting of CIFAR100 and Tiny-ImageNet. But I don't know if the EDBL method also has the lowest average forgetting rate.

Clarity: This paper is well organized. I suggest the author give a simple introduction for task/domain incremental learning in Section 2 and a brief comparison of other approaches like the regularization-based approach.

Significance: The EDBL method outperforms many baselines in the setting of CIFAR100 and Tiny-ImageNet.  And the idea of calculating the incremental influence-balance weighting factor for each sample and then using the factor to adjust the loss is interesting. 
questions: Question 1:  The EDBL method generates more mixed data for the two types of mixup among old classes and mixup between old
classes and new classes than the type of mixup among new classes. But the rate ($N/2$) set for the first two types is just an empirical value. Please try to investigate the performance of EDBL at different rates and illustrate the results. Also, what does the "tail and head class" mean (Line 162)?

Question 2: Can the EDBL method help the KD method to alleviate the forgetting problem further? Please calculate and compare the average forgetting rate of the EDBL method and other baselines. 

Question 3: The paper says the classification weighting factor is for learning new tasks. But it also calculates the classification weighting factor of the mixed data generated by the samples from old classes. Can the author explain the role of the classification weighting factor in this case?

Question 4: That is just a minor question: does the EDBL method also work well in the few-shot continual learning setting? In this setting, the model is over-fitting to the new data severely. And the IIB loss may have a significant impact in this setting.



limitations: I don't see any potential negative societal impact in this paper
review_title: null

Meta-review: metareview: The paper aims to use the mixup and re-sampling strategy to improve knowledge distillation. Reviewers find that the classification weighting factor is not a new task, and the technique and novelty is very limited. Because both the mixup and re-sampling strategy have been used in many settings. There is no techniqual contribution. The author is encouraged to improve the paper by considering the comments from the reviewers.
confidence: Certain
Decision: Reject