Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper presents a theoretical analysis using SPCA, which predicts the performance of algorithm in advance. In addition, it propose a label optimization scheme to prevent the catastrophic forgetting with theoretical support (under the strong assumption of X are independent Gaussian random vectors with identity covariance).
strength_and_weaknesses: **Strengths**
- S1: (Even with strong assumptions) The theoretical analysis and a novel label optimization formulation seems novel.

**Weaknesses**
- W1: **Empirical validation is weak** -- Used benchmarks are all MNIST variants, thus in a small scale and relatively simple. Since the method is based on the strong assumption (See my summary), the empirical validation is quite weak.
- W2: **No discussion on the empirical comparison** -- Why the proposed method does not outperform the GEM in RMNIST.
- W3: **Presentation is somewhat unclear" -- Missing citation for the compared methods. Though the method of EWC, GEM, iCaRL and etc are quite well known, they should be properly referred either in the text or in the table.
- W4: **Recent replay-based methods (published in 2021-2022 -- see the list below) are largely ignored in review** - Some of them could be compared with but not all as they are evaluated in different set-ups.
  - Wang et al., "Continual Learning with Lifelong Vision Transformer", CVPR 2022
  - Hersche et al., "Constrained Few-shot Class-incremental Learning", CVPR 2022
  - Wan et al., "Continual Learning for Visual Search with Backward Consistent Feature Embedding", CVPR 2022
  - Tang et al., "Learning to Imagine: Diversify Memory for Incremental Learning using Unlabeled Data", CVPR 2022
  - Bang et al., "Rainbow memory: Continual learning with a memory of diverse samples", CVPR 2021
clarity,_quality,_novelty_and_reproducibility: Overall the paper is clearly written and the proposal is in high quality (though there is some concern in novelty). There are some suggestion to improve the clarity of the presentation (see my comments in weaknesses).
summary_of_the_review: Though the analysis and the proposed method is simple and limited in applicability, it presents an exact analysis on catastrophic forgetting and a novel label optimization formulation (though it's simple). The reviewer believe that this contribution is not trivial and meaningful to the community.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors propose to use supervised PCA to tackle catastrophic forgetting in continual learning. They provide a theoretical analysis of the proposed method. They develop a label optimization scheme. They validate the method on four datasets and three scenarios (task incremental learning, domain incremental learning and class incremental learning).
strength_and_weaknesses: Strength: 
1) using PCA and SPCA (or other dimensionality reduction methods) in continual learning can be beneficial, though it is not well motivated in this paper.  
2) experimental results suggest performance improvement. 

Weakness:
1) It is unclear why SPCA can prevent catastrophic forgetting in both the motivation and theoretical analysis.  
2) There is a view (e.g. see https://arxiv.org/pdf/2011.05309.pdf) that SPCA can be simply replaced with a task loss plus the traditional PCA loss producing even superior result than Barshan’s method. Task loss is readily given in continual learning. Thus this may suggest a traditional PCA loss added to continual learning may do a better job that the proposed method. 
3) Gradients Orthogonal Projection method is relevant to the proposed method, but is not analysis/compared sufficiently nor empirically.  
4) I don't understand why labels need to be optimised. Labels are given by the tasks/datasets. The distribution of the labels are, in general, inaccessible, and the dataset is a sample of the distribution. 




clarity,_quality,_novelty_and_reproducibility: The paper lacks clarity, which cast a shadow on quality and novelty. 

It would be better if the authors can explain how and why the theorem 1 and 2 can support the proposed method can prevent catastrophic forgetting. 
summary_of_the_review: I don't think the paper is ready to publish yet. 

Post-rebuttal:
The original form of the paper was unclear and confusing. The authors answered all my questions well and these issues can be addressed in the camera-ready version. Hence I have increased my score.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper propose a theoretical analysis of a SPCA-based continual learning algorithm using high dimensional statistics. Based on the theoretical analysis, they propose a label optimization process. Empirical evaluations highlight the effectiveness of the method.
strength_and_weaknesses: The theoretical result is sound and significant. While there exist some concerns as follows:
- Some mathematical notations seems inconsistent across the paper, making it hard to understand. E.g., the subscript order of c = [n11/n, …, n2t/n] in the appendix does not seem to be consistent with the previous description. Ds and Dc is not explained for readers.
- The formula of Q(z) is wrong. In the cumulative distribution function of standard normal distribution, The exponent of e should be -t^2/2. Whether this error will have an impact on subsequent theoretical proofs and experimental results.
- For the conclusion of Theorem 1, when the number of tasks t increases indefinitely, can the comparison of the information that SPCA-based methods and other continuous learning methods need to preserve be illustrated?
- Regarding the weight setting seems to be arbitrary, it is recommended to explain the rules for weight setting.
- The analysis of the experimental results is weak, e.g, in Table 1, why in DIL setting, OSCL substantially underperforms compared methods. In addition, since it is theoretically proven that the OSCL method avoids catastrophic forgetting, to illustrate this, it is recommended to include the average forgetting measure when presenting the results.
- Whether this method can be combined with a neural network?
- How to update the statistics in the online task environment？In my understanding, the dimension is increase when the number of tasks increase. How to use previous statistics to obtain current statistics?
clarity,_quality,_novelty_and_reproducibility: The  theoretical result is promising, while many details need to clarify, and the writting needs to improve. 
summary_of_the_review: Overall, this paper is interesting, since it provide a novel theoretical analysis of catastrophic forgetting. However, some important points (see above) need to clarify. The current version can not meet the requriement for publishing in ICLR.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduces a novel CL method based on Supervised Principal Component Analysis (SPCA). The main contributions can be fo summarized as:

(1) Proposing a simple continual learning algorithm, which is based on supervised principal component analysis.

(2) Providing a a theoretical analysis (exact classification error rather than bounds) using high dimensional statistics.

(3) Developing a develop a label optimization scheme.

(4) Several applications are presented
strength_and_weaknesses: 
Strengths:

(a) This paper provides extensive theoretical analysis.

(b) The proposed method seems to be effective in the provided experiments.

Weaknesses:
I am not very good at supervised principal component analysis. Therefore, I talk about my feelings of this work.

(a) A more detailed description of supervised principal component analysis (SPCA) is necessary. What is SPCA? How is SPCA connected with CL? 

(b) The description of proposed method is confusing. In the main file, no description of proposed method can be found, while they are placed in appendix. Clear description of proposed method is necessary.

(c) What does assumption 2 mean? What can we infer from it? 
clarity,_quality,_novelty_and_reproducibility: Poor clarity.
Good quality.
Nice originality of the work.
summary_of_the_review: I am not very good at supervised principal component analysis. I recommend marginally above the acceptance threshold. I will change my score according to other reviewer's comments.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes a new continual learning extension over the supervised PCA method. This works by re-weighing the class labels for the different datasets in a way that prevents catastrophic forgetting. The re-weighing scheme is theoretically derived and experimentally tested.

It is great to see a paper that has strong theoretical grounding and is not purely empirical. Although the theory may not generalise to non-linear more complex methods, such as neural networks, the authors were able to show that the linear method can be successfully applied over features extracted by neural networks. Achieving good performance in the continual learning setting for sPCA is important, because this method can run on computationally constrained devices.  
justification_for_why_not_higher_score: Although it is great to see a theoretically grounded paper with good empirical support for the derived method, the fact that the results are presented on variants of MNIST and with linear methods only limits the audience for this paper.
justification_for_why_not_lower_score: This paper is of value to the community, well rated by the reviewers and presents theoretically supported results.
Decision: Accept: poster