Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This work tackles incremental image classification problem, and proposes two techniques, ClassAug and SemanticAug, to improve the overall accuracy over all classes, including old and new classes. Results on two benchmarks, including CIFAR-100 and Tiny-ImageNet, are presented to demonstrate the superior performance.
main_review: Strength
  -	The method does not require to store example of old classes, and its performance is better than non-exemplar based methods and close to exemplar-based methods on two benchmarks.
  -	ClassAug seems to be an original idea with supporting evidence from spectral analysis on primary eigenvector components in the per-class features

Weakness
  -	One issue I find with the paper writing is this paper is not self-contained. In Table 1, a minimal introduction to iCaRL and CCIL is not missing, which makes it difficult to get the idea in section 3.1 before reviewing prior papers. Similarly, L169, minimal introduction to LwF-MC is not included.
  -	The proposed SemanticAug seems to be an incremental contribution as it resembles ISDA [24] a lot. While the discussions at L255 are valid, the adaptation of ISDA idea in incremental classification seems to be straight-forward.
  -	Some technical exposition is incomplete.
    o	Fig 2 (b), when data augmentation is used, such as mixup, cutmix, it often requires more training epochs to learn a better model. Can you clarify whether results with data augmentation in Fig 2 (b) are obtained with more training epochs?
    o	L217, how many augmented classes (i.e. m) are added? Any data sampling technique is used to balance original new classes and augmented new classes? how many images are generated for each augmented class?
    o	L239, for each old class, do you fix the M deep features once they are generated from a normal distribution with fixed mean/covariance? Since backbone is also updated as more new tasks are added, the mean/covariance will be outdated.
    o	L292, what is “herd” selection technique?

limitations_and_societal_impact: The chance of having negative social impact is low. So it is ok authors do not address them.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a class-incremental learning (CIL) method that addresses the representation-bias and classifier-bias observed in CIL. The goal is to learn transferable ans compact representations for incremental learning and leverage the distribution of old classes to avoid forgetting and to jointly learn a unified classifier. This paper also presents an interesting finding that regularization in CIL have a negative effect.
main_review: This paper explores important questions in CIL: i) which parts of the representations are tend to be forgotten, ii) how regularization impacts representation learning and iii) how to facilitate representation learning in CIL. To analyze the forgetting, they use spectral decompositions, obtain the eigenvalues and eigenvectors, and check the cosine similarity between eigenvectors. To learn the transferable representations, they propose to use class augmentations. To learn the classifier, semantic augmentation is used. Class augmentation and semantic augmentation are called as dual augmentation framework and it is the proposal of this paper. To the best of my understanding, this approach is novel and has not been used before. Besides this approach doesn't require examplers and replay.

It would be useful to clarify a few points. In section 3.1.1 it is mentioned that intuitively the corresponding angle could capture the representation shift between the old and updated feature extractors during incremental learning, and reflect the forgetting along different directions in feature space. It is not clear why intuitively this is correct. The complexity of dual augmentation framework is not analyzed in the paper. The experiments show that the results of no augmentation, only semantic augmentation, only class augmentation and their combination and it seems the combination of both augmentation outperforms the others by margin, but it is not clear how costly to run the combination, which augmentation is cheaper etc.
 
The experiments are performed on CIFAR-100 and Tiny-ImageNet. These datasets are called challenging but it is not explained in which sense they are challenging. To the best of my knowledge, they are standard benchmarking datasets for incremental learning and it would be useful to run additional experiments on real challenging datasets where there are data samples from different domains.   

The structure of the paper is nice, the problem, motivation and the goal is clear. But there are so many typos and grammatical mistakes that makes to follow the paper difficult. Some examples: p5, l161 two groups "of" eigenvectors, l170 firstly trained -> first trained, l185 simimlar -> similar, p7, l263 discribes -> describes, l266 foucs -> focus, diffrnt -> different, ... etc. 
limitations_and_societal_impact: It is not clear how this method is applicable to more challenging CIL problems where there is distribution shift and out-of-distribution data. It would be useful to add this to the discussion.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: The paper addresses two problems which occur in the class-incremental learning: representation bias and classifier bias. To address the representation bias, the authors propose to learn a less compact representation in each task. More concretely, they augment the number of existing classes by introducing random interpolations among real samples. Regarding classifier bias, the authors propose to generate an infinite number of features from of classes to maintain the decision boundary of previously learned classes. The insights into representation learning by analyzing the structural characteristics of the learned embedding space via spectral decomposition is interesting. The paper is relatively well written, although it lacks details in some aspects. The approach is moderately novel, being a combination of existing techniques. The experimental validation does not confirm that the proposed approach improve the current state of the art in class-incremental learning.
main_review: Here are my concerns:
1. The problems addressed in the current paper, namely representation bias and classifier bias, are not depicted clearly enough in Figure 1. I would recommend to improve the clarity of the figure.
2. Page 5, 'Feature Compression Perspective': what is the difference between the 'overall features' and 'learned features'? Why there is a difference in dimensionality in both cases?
3. Figure 4: According to section 3.1.2, the extracted features from f_(t-1) and f_t are further compressed, right? This step is not depicted in the figure.
4. An Algorithm to summarize all the steps of the approach would increase the clarity of the paper.
5. There are several typos in the paper, but most of them in page 7.

limitations_and_societal_impact: The current work does not present a negative societal impact.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors propose a non-replay class-incremental learning method that works by combining input mixup with pseudo-replay at feature level. The authors motivate this method by the means of a simple study on how feature space eigenvectors change while training. The proposed approach is evaluated against a collection of current continual learning methods and shows competitive performance.

main_review: 
In general, I found the technical core of this work to be very interesting:
- I found semanAug to be a simple yet very interesting approach. Recent Continual Learning literature often disregards feature pseudo-rehearsal to focus instead on input replay or generation. According to the results shown by the authors, this approach is deserving of further exploration.  
- The overall baseline proposed by the authors is very simple and yet manages to show remarkable performance, given its lack of a memory buffer. It is very remarkable to see it performing on par with memory-based methods, which up to now are generally regarded as the only viable solution for class-incremental learning.  

However, this work is characterized by several weaknesses, that I feel need to be addressed to increase its potential:  
- In the introduction, the authors make a distinction between classifier and representation bias. While the latter clearly refers to the mismatch between the features learned in past tasks vs the ones needed for new tasks, I cannot clearly understand what the authors mean by classifier bias. The definition provided at lines 32-33 seems to me a general description of the stability-plasticity dilemma, the issue at the core of continual learning and widely debated in literature [11, 17 in the paper]. I do not see this as a bias to be solved, but rather as a generally valid description of what happens when training a neural network on incremental data. Along the same line, the authors claim at line 128 that setting a trade-off between plasticity and stability hinders learning in the long run. Again, I see this as a choice that need to be made and that is not negative per se: whether it hinders training only depends on the produced amount of stability.
- The authors claim at lines 44-45 that “learning diverse and transferable representation is an important requirement […] which has been ignored by previous works”. I do not find this statement to be accurate: finding suitable and transferable representations has always been one of the cornerstones of continual learning approaches [13, 28 in the paper, Joseph et al. 2020, Chaudhry et al. 2020]  
- This paper adopts an unclear position towards Replay-based methods: on the one hand, the authors claim that they are deliberately targeting non-exemplar-based continual learning, on the other hand, they only adopt replay-based method for their preliminary experiment in section 3.1 and compare IL2A with several replay methods in section 4, so they are not limiting themselves to the aforementioned setting. Even though the method proposed by the authors requires retaining a previous snapshot of the model (which amounts to a memory footprint that may be comparable with replay methods), their non-exemplar-based proposal performing on par with rehearsal approaches is a significant result. However, I wonder if their proposed approach can be successfully coupled with replay and get to even better performance. Indeed, adding such a simple additional experiment could make the whole work even more relevant.
- A continual learning work that shares some similarities with the authors’ proposal is IL2M (Belouadah et al., 2019), which also makes use of previous class statistics to rectify the prediction scores and prevent representation bias. I think that adding a direct comparison with how this principle is differently exploited in this work could provide a comprehensive picture for the readers.  
- I found the experimental sections to be often unclear so I strongly suggest that the authors take further steps to improve it, with particular reference to the following points:  
    - I could not understand the meaning of the numbers contained in Table 1. Are they average accuracy values or incremental accuracy values? Are these numbers referring to different experiment (50-split dataset, 60-split dataset, etc.) or do they refer to the same experiment? What dataset is being used?  
    - What does “eigenvalues of covariance” mean at line 182? How exactly is the intrinsic dimension of features defined at line 188? When introducing a specific approach for evaluation, it could be useful to explicitly present it to the reader or at least introduce relevant references.  
    - Running continual learning experiments is often a complex matter, since there exist different experimental settings which are often incompatible. In order to evaluate whether experiments are fairly conducted, since the authors did not submit the code for review, but state they will publish it upon acceptance, I need more details on the way experiments were conducted. What codebase was used for the experiments, a new one created from scratch or was it based on the code from other works? Are all results computed anew for all methods or taken from other works? How was hyper-parameter selection conducted for the competitors, were they tuned at all or were original hyperparameters used?  
    - In the ablation study, I could not clearly understand how the baseline is constructed: how is the class-mean of old classes used to regularize the classifier (line 311)? Does this mean that only part of the semanAug loss is used? I could not understand this from the text.  
    - How is the forgetting measure at line 324 defined? There are several instances of such measure in CL literature, so I would need a citation to understand which one the authors are referring to.  

The following minor issues did not affect my evaluation directly, but I list them for the authors to consider:  
- I believe that adding variance values for the presented results (at least in the supplementary material) would be very useful to further simplify the understanding of the presented data.  
- Since the denominator in equation 3 is 1, the equation could be simplified.  
- In line 202, the authors define [56] “a very nice recent paper”, I would recommend keeping a neutral tone instead.  
- The comic sans typeface was used in most presented figures, which is unusual in scientific papers. A common sans-serif font would probably be more appropriate.  
- There are several typos that need fixing:  
Line 39: learning, and → learning and  
Line 49: is benefited from → benefits from  
Line 62: to storing → to store / storing  
Line 96: Classical strategies commonly synthetic → verb is missing  
Line 99: in complementary → in complement  
Line 129: our high idea → our high-level idea  
Line 136: naturally rises → naturally raises  
Line 140: How the regularization affect → How does the regularization affect  
Line 143: are tend to be forgetting → tend to be forgotten  
Line 149: is donated as → is denoted as  
Line 161: two groups eigenvectors → two groups of eigenvectors  
Line 185: We note that simimlar → We note that a similar  
Line 236: tast samples → task samples  
Line 236: wrong class → wrong classes  
Line 237: high level idea to → high-level idea is to  
Caption of figure 4: hans → hand  
Line 266: differnt → different  
Line 274: rest incremental step → remaining incremental steps  
Line 300: bais → bias  
Line 306: task → tasks  
Line 310: method that using → method using  
Line 311: and use the class → and class

(Joseph et al., 2020) Meta-consolidation for continual learning, NeurIPS 2020  
(Chaudhry et al., 2020) Continual Learning in Low-rank Orthogonal Subspaces, NeurIPS2020  
(Belouadah et al., 2019) Il2m: Class incremental learning with dual memory, ICCV 2019

limitations_and_societal_impact: This is a theoretical work in continual learning classification. This is a very abstract field that does not elicit any immediate application. As such, I feel that no discussion on potential societal impacts is necessary.
review_title: null

Meta-review: We thank the authors for the additional clarifications provided in their rebuttal, which resolved most of the concerns raised by the reviewers. The contributions are relatively simple, but convincingly shown to be very effective compared to more sophisticated approaches recently proposed in continual learning. These results provide new insights and will benefit the NeurIPS community. However, the reviewers also provided a list of improvements that should be incorporated in the revised version. The paper was not self-contained at times and needs to be polished in order not to hamper the understanding. Exposition of the experiments would also benefit of a revision.
Decision: Accept (Poster)