Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper investigates the Class incremental learning (CIL) problem by expressing the problem as a Within-task in-distribution (IND) class learning problem (WIP) and task-id prediction (TP) problem. The author theoretically induces that solving the two problems: WIP and TP, is a necessarily sufficient condition for solving CIL, and also shows that TP is closely related to the out-of-distribution detection (OOD) problem. Based on the investigation, the author proposes a new CIL technique using recent OOD algorithms (solving WIP and OOD simultaneously) and achieved superior CIL performance, supporting the author's claim that OOD is a critical factor for CIL. 
strengths_and_weaknesses: StrongPoints

(1) The theoretical investigation of CIL, starting from WIP and TP to OOD, seems reasonable and worth sharing with the community.

(2) The proposed simple CIL solution using OOD supports the claim of the author, that OOD which can also solve WIP, can sufficiently solve CIL problem. Moreover, comparision to many existing CIL problems increases the credibility of the author's suggestion.

WeakPoints

First, note that the reviewer is positive about this work, and this section is much like questions rather than criticism.

(1) The paper focuses on the traditional CIL problem set-up that the incoming data label is completely disjoint. However, a recent line of studies also introduces a more realistic, close-to real-world setting, where the incoming data label is overlapped, which called Blurry task setting [1][2]. When following the author's theoretical induction, the reviewer could not find specific problems restricting the induction to be applied in the Blurry task setting. Hence, applying the idea to the Blurry set-up will strengthen the author's claim.

[1] Rainbow memory: Continual learning with a memory of diverse samples. J Bang, H Kim, YJ Yoo, JW Ha, J Choi - Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021
[2] Online Continual Learning on a Contaminated Data Stream with Blurry Task Boundaries. J Bang, H Koh, S Park, H Song, JW Ha, J Choi - Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022

(2) For the testing datasets, recent CIL problems also use ImageNet to show their algorithm's performance larger than CIFAR 10/100 or tiny-ImageNet. The reviewer does not think that the tendency shown in the experiments will not change drastically, but checking it in ImageNet would much strengthen the credibility of the Experiments.
questions: See the Strongpoints/weakness page.
limitations: See the Strongpoints/weakness page.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper studies Class Incremental Learning (CIL) and shows that the CIL problem can be decomposed into two subproblems: within-task in-distribution prediction (WIP) and task-id prediction (TP). As a result, the CIL performance is bounded by WIP and TP performances. Moreover, the authors show that the TP subproblem itself can be bounded by the OOD detection problem, and thus by extension, CIL can be bounded by WIP and OOD.   

To test their theory, the authors add better OOD detection to several CL algorithms and report performance improvements.
strengths_and_weaknesses: **Strengths**:  


1. The paper is very well-organized and well-written. It is easy to follow the arguments.  


2. The analysis is mostly intuitive, and I appreciate the effort to understand CIL more theoretically.  




**Weaknesses**:  


1. I'm not entirely convinced about the bounds in the paper, and I think the theory is partially incomplete. Let's assume we have an oracle that, given a data point, can tell us the task id with zero loss. Also, let's assume if we train on each task individually, we have zero loss. Then by Theorem 1, it means that we can have zero CIL loss. However, this implies that we should have zero loss for the equivalent TIL problem (since we have task ids), which is not necessarily true. I think (but not sure) the theory holds if we assume we don't have shared parameters for tasks (e.g., parameter isolation methods) or tasks are all orthogonal. Otherwise, even if we know about task ids (e.g., TIL), we still can show the error cannot be bounded. As an example, let's say we have two tasks and we always have access to task ids, but we use the same parameters for both tasks. Then if we train on task 2 for a very long time (without using any CL algorithm), since our loss function only depends on the data of task 2, and since we have shared parameters, the parameters will change, and if we measure the loss on task 1, it can grow (similar to all other TIL works), unless we have additional assumptions. I am willing to discuss this further with the authors during the discussion period.


2. Discrete Task Assumption: Assumptions 1 and 2 assume that each data point belongs to one task exactly and tasks are discrete. However, in many real-world setups, the shift may be continuous.  



3. I have read the argument on Sec.4.2 (L274), but I still think the forgetting rate is an important metric that measures the backward transfer in many algorithms. I encourage the authors to compare the average forgetting for a few benchmarks.  



Overall, while I think this work is a nice first step, I think the theory is limited in its correct form, and more importantly, I think we need additional assumptions. I am willing to raise my score if my concerns are addressed.



--------
Update: 
After discussion with the authors, I believe they have addressed my concerns, and I raise the score.
questions: See above.
limitations: Adequate discussion.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: In this paper, the authors propose that the class-incremental learning problem can be decomposed into two sub-problems: within-task prediction and task-id prediction. They further prove that task-id prediction is correlated to out-of-distribution (OOD) detection. They design some class-incremental learning methods based on their theoretical results and provide experiment results on multiple datasets. 
strengths_and_weaknesses: Strengths:

- This paper provides in-depth theoretical analyses for class-incremental learning. This is interesting.

- Extensive experiment results on different baselines are provided. 

Weaknesses:

- All theoretical analyses are based on the assumption that all classes are independent and identically distributed. However, this might not be true in many real-world applications. For example, there are many classes that correspond to different species of dogs in ImageNet. If these classes occur in different incremental stages, they will significantly influence the task-id prediction. Thus, it is questionable how much these theoretical results can be generalized for real-world applications. 

- Some top-performing methods are not cited and compared. For example, [A] archives better performance on CIFAR-100 compared to the proposed method (see Figure 3 in [A]). 

- Many details haven’t been provided in the main paper. For example, the authors propose some new CIL techniques in Section 4. However, some important details, such as the number of additional parameters required by the proposed method, are not available. A NeurIPS submission needs to be self-contained, and the authors need to let the audience understander the proposed method without reading the supplementary. It doesn’t seem possible in the current version. 

- It would be better to provide a figure to show the overall framework of the proposed method. 

- This paper is poorly-organized and difficult to understand. For example, the authors mixed the proposed method with the experiment results in Section 4. 

-----

Overall, I think this is an interesting paper with extensive theoretical and experimental results. However, there are some unsolved issues as I mentioned in the “weaknesses” part. Thus, my overall rating is “borderline reject”. 

-----

[A] Yan, Shipeng, Jiangwei Xie, and Xuming He. "Der: Dynamically expandable representation for class incremental learning." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.

questions: as the comments are given in weaknesses.
limitations: not clear in this paper.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper studies class incremental learning(CIL) from a theoretical point of view.
It proposes to decompose CIL into within-task prediction(WIP) and task-id prediction(TP) correlating to out-of-distribution(OOD) detection. It further provides sufficient and necessary conditions for improving CIL performance, and proves that a good performance in WIP, TP, and OOD is necessary for a good performance of CIL. Experiments on a combination of existing CL and OOD methods validate the theoretical results.
strengths_and_weaknesses: Strengths:
+ The discussion of CIL performance from a theoretical point of view is interesting.

+ The theoretical results provide guidance on selecting a combination of CL and OOD methods for achieving better CIL performance.

+ The authors have conducted extensive experiments on four datasets and nine CL algorithms.

Weaknesses:
- Although the authors claim that they have proposed a new CIL method based on their theoretical analysis. It's not clear what the novelty of the proposed method. 
questions: All listed CIL methods are a combination of existing CL and OOD detection methods. It's not clear what are the new designed CIL methods.

limitations: The authors did not adequately addresses the limitations and potential negative societal impact of the work.
review_title: null

Meta-review: metareview: This paper studies the class-incremental setting of continual learning. The authors provide a theoretical justification for decomposing the problem in two: a) task-id prediction, and b) within-task prediction. Based on this insight, the authors propose two new continual-learning methods. 

The initial reviews were somewhat mixed and the reviewers raised several questions, especially on some of the theoretical arguments proposed in the paper. The authors provided detailed replies that were convincing to the reviewers and the reviewers now agree that this is a good paper that should be accepted. I would add that this contribution might prove significant as most existing CL algorithms are empirical in nature rather than grounded in theory. This is likely even more so for the more challenging class-incremental learning setting.

I am happy to recommend acceptance of this paper, congratulations! I also strongly encourage the authors to take into account the comments of the reviewers when preparing the camera-ready version of this work. In particular: 
- Several expert reviewers misunderstood key aspects of the paper upon first reviewing it. I do not have precise suggestions for improving the paper, but it would be useful to explicitly discuss some points raised by the reviewers in the paper or in the appendix. There are many different flavors of continual learning even within class-incremental and so clarifying the exact scope of the theoretical development might also help.
- In your replies to the reviewers, you mention not having enough time to study your methods on imagenet. Regardless of the performance of tiny-imagenet, my sense is that using the full imagenet would improve the (perceived) significance of the results.

confidence: Certain
Decision: Accept