Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors proposed an approach for class incremental learning by leveraging the conditional of the labels given the data (covariates), which they decompose in terms of the labels given the task (consistent with task incremental learning) and the task given the data (task predictor), this without the need of a replay buffer or pseudo-rehearsal. The task prediction is realized incrementally via independent principal component analysis and quantified via Mahalanobis distance.
strength_and_weaknesses: The authors propose a relatively simple approach to address CIL tasks via estimating features for each task using iPCA and hard attention masking (i.e., the HAT core) to prevent catastrophic forgetting. The approach simple and intuitive, computationally efficient and leverages the hard attention mechanism in HAT as a means to mitigate catastrophic forgetting.

As a drawback, the authors seem to omit the existing literature in expansion-based approaches for continual learning. Further, expansion-based approaches have demonstrated excellent performance (at the time, so of which, do not consider transformer architectures) in terms of catastrophic forgetting at the expense of moderate parameter growth. For instance, the authors do not mention Mehta et al. 2021, a nonparametric network-expansion approach that also uses a Gaussian distribution and a likelihood similarity (with a Mahalanobis distance calculation at its core) to predict task labels. The latter, though probably will not be able to compete in terms of performance, underscores that the idea of using a summary in representation learning space and a similarity function as a means to identify tasks is not new, thus reducing the novelty of the proposed approach.

The experimental results are extensive and convincing, specially the fact that all baselines use the same architecture and that the transformer was trained to exclude classes similar to CIFAR and Tiny-ImageNet. The details of the experiments are clear and the ablation studies in Table 2 and 3 help understand the contribution of each component of the proposed approach.

As a minor note, Figure 2 is hard to see, so changing the visualization or simply replacing it by a Table will improve the presentation.
clarity,_quality,_novelty_and_reproducibility: The proposed approach, which is conceptually simple, is well justified, described and motivated. The paper is well written and experiments are comprehensive. Novelty is limited by similar approaches using distances over summaries in representation space. The authors facilitate reproducibility by providing a detailed account of the model, optimization, experimental settings and source code.
summary_of_the_review: The authors propose a conceptually simple approach for CIL based on incrementally estimated summaries over previous tasks via iPCA and the hard attention masking of HAT. The resulting approach is computationally efficient, leverages a pre-trained transformer encoder and delivers excellent performance.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper addresses continual learning where task IDs are not available.
The authors propose a learning method preventing catastrophic forgetting and providing task ID estimation capability.
The proposed method utilizes batches of features for each task generated from the generators obtained with incremental PCA. At this time, the OOD generator learned from past tasks is also used. The method estimates the ID for the current sample as that of the closest task in terms of the Mahalanobis distance from the mean and variance of the distribution of features of each task.
Experimental results on multiple public datasets demonstrate that the proposed method performs better than existing methods.
strength_and_weaknesses: *Strength
- Experimental results of the proposed method showed promising performance.

*Weaknesses
- The important assumption of why the class labels of tasks are disjoint is not well justified.
- The problem setting is unclear. It is described in about four lines in the Definition paragraph but without a specific flow. It is difficult to understand what they actually do. For example, I cannot see when task IDs are estimated, when tasks are switched, tasks come in altogether; they come once, they don't come again, or they come repeatedly. Will all data be discarded each time, or will it be discarded for each task?
- The motivation of Eq.2 is unclear. Why do the authors introduce pseudo feature vectors Z? The description of the method is also unclear. How can we generate y for Z? Where does B come from?
clarity,_quality,_novelty_and_reproducibility: *Clarity
- The subscripts for w_k,j,l are not defined.
- The motivation for the regularization in Eq.7 is unclear.
- Why do they use the OOD samples in the computation of WTP?
- It is better to show the results after each task as well as the average performance over them. The authors only provide average classification accuracy after the final task.

*Quality
- Please see the above comments.

*Novelty
- The proposed method seems to be novel.

*Reproducibility
- Code is available.
summary_of_the_review: The clarity is low. The motivation and details of the proposed method are unclear.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper considers class incremental continual learning, in which disjoint subsets of classes are introduced one at a time. A multi-head architecture is used which shares a pre-trained feature extractor (backbone). Forgetting is addressed by using HAT (Serra et al., 2018) which applies a separate binary mask to the backbone for each task (subset of classes).
The authors decompose the prediction into task-id prediction (TIP) probability and within-task prediction (WTP) probability. Both are estimated by making use of class-specific approximations of the features distribution.
The resulting method is shown to outperform competitive baselines and the design decisions are motivated with ablation experiments.

strength_and_weaknesses: I think that the method is comprised of insights which could be of interest to the community. Moreover, the experiment compare the method many baselines and conduct ablation studies.
One potential weakness is that the approach relies on a pre-trained backbone. Another issue is that the “Related Work” section does not describe and compare to the most closely related work, but rather just cites it.

clarity,_quality,_novelty_and_reproducibility: The problem setting is clearly defined and the method is well explained.
It is hard to judge the novelty of the method because, while related work is referenced, it is not described, thus, the difference from similar past work is not clear.
summary_of_the_review: The paper present  interesting ideas and satisfactory experiments. Conversely, I am uncertain about the novelty of the method.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper decomposes the CIL problem into two prediction steps, which are within-task prediction and task-id prediction. Several strategies are designed to tackle these sub-problems, e.g. HAT, PCA, single head classifier fine-tuning. Experimental results show that the proposed EWT works better than existing baselines on pre-trained transformer network.
strength_and_weaknesses: Strength:
1. This paper proposed a new perspective to decompose the CIL probability into WTH and TIP probability.
2. For the above two sub-problems, the paper proposes a simple yet effective solution based on HAT, PCA, and features replay.
3. The proposed EWT outperforms the existing methods without exemplar replay.

Weaknesses:
1. The pre-trained model used in the experiments still too strong for CIFAR and Tiny-/Sub-ImageNet.
2. The proposed methods could be highly dependent on the good feature representation obtained by pre-trained model. If not,  some experiments under normal settings are needed to prove it.
3. There is no quantitatively comparison of additional storage between EWT and other replay-based methods.
clarity,_quality,_novelty_and_reproducibility: I think the decomposition of CIL proposed in this paper is good, and the authors designed some significant strategies to solve it. But these  strategies are based on existing methods and may rely heavily on pre-trained models. The novelty is not enough.
summary_of_the_review: Based on the above comments, I tend to reject this paper. The main reason is that the experimental settings are insufficient to prove the effectiveness of the method.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper presents a framework for class-incremental continual learning in which for each class, the method incrementally maintains a distribution over activations. The class is then predicted by combining the scores of the within-task classifier with a class-conditional Gaussian classifier in activation space, to predict the task.

Strengths:
* Several of the reviewers praised the experimental evaluation, which compares to a large variety of competing methods. 
* The combination of incrementally learning the class mean activations and TIP prediction seems to be novel.

Weaknesses:
* A few reviewers mentioned the strong similarity to previous work (e.g., HAT)

* Key design decisions of the approach are not well motivated: 

 - For example, the network is separated between a feature extractor f_i, and a network g_i that is specific to each task. It is unclear whether f_i is shared across all the tasks (in which case, why subscript?) or if they are separate to each task (in which case, why separate f from g?). The paper says "Note
that although f_i’s are task specific but they are all learned in the same network and there are a lot of parameter sharing." which I am unable to understand. This issue is important for understanding why we expect HAT to be helpful here.

- I can understand why we want classifiers to be robust in general, but why is this problem more severe in the context of CIL? Why is the class-conditional Gaussian expected to be more robust to OOD data than simply predicting the task id using a separate network (perhaps with shared weights?)

- I share the reviewer's question about the use of synthetic data in task 1. The paper says "we use Z in the step 1 training because we want to produce better distributions", but I think that the question is *why* do we believe that incorporating pseudo feature vectors generated in this way will help learning. Is this simply a way of doing data augmentation? I believe that p(y | z, i) in (2) is undefined. I assume that this is just  p(y |z, i) = g_i(z), but this should be stated.

* As a minor point, although the overall setup of CIL is described in detail in previous work, I do sympathize with the comments of Reviewer vcxy that the description of CIL should be expanded slightly to make the paper more self-contained and help to motivate the method. e.g., make clear that the task boundaries are known, and when the final data arrives, we want to classify based on all the classes seen so far, which can be decomposed into task + class prediction. For example, the paper does not actually say, as far as I can tell, that the method is expected to process all of the training data from each task, and then move on to the next task. However, this did not play a major role in my decision.


justification_for_why_not_higher_score: Mostly the reject decision is based on the lack of motivation for key technical decisions. To some extent, I share the concerns about the similarity to previous work, but that is not as a major a concern; a combination of simple ideas can be useful if we understand why they should work, and they work well.
Decision: Reject