Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper introduces a new method of few-shot incremental learning with its regularization-based method motived by a subspace view.
Regularization terms to the linear layers of incremental classes are proposed to encourage the novel weight vector to be in the subspace spanned by those of the base classes, and semantic information from the text domain can further be leveraged to guide the interpolation within the base-class subspace. 
main_review: **Strengths**

- This paper attacks a very important and practical challenge. and the motivation of the proposed method is well presented.

- The idea of regularizing the novel weight vector using the base ones to prevent the network from capturing spurious correlations seems interesting. 

**Questions**

- I'm a bit confused by the regularization term imposed on the old class $R_{old}$. In the following session, since the feature extractor is fixed and I assume there are no available samples from the base classes, why the weight vectors of base classes are not fixed? And why this regularization term is important?

- While better quantitative results are reported in the experiment section, it remains unclear to me why restricting the novel feature vectors to be in the subspace of base classes can improve the overall performance. This motivation opposes some prior work on subspace-motivated continual learning framework, e.g., [1]. My main concern is why this regularization will not cause interference to base and other incremental classes, as the classification boundaries are all restricted to the subspace defined by the base classes, and I do not see any terms used to encourage a robust subspace obtained by the base classes only.

- Some visualizations, e.g. to the weight vectors are expected to further support the effectiveness of the proposed regularization terms, especially the semantic subspace regularization. 

- Missing discussions on some recent work, e.g., [2,3].

**Minor**

This paper demands additional rounds of proofreading. 

Non-exclusive examples:

Page 5 line 7 double 'append'

Section 2 paragraph 3 'learners.Existing' missing space

Section 3 paragraph 4 'Qi et al. (2018) initialize novel' -> 'initializes'

Section D.1 line 2 'with learning starting' -> 'with a learning rate starting'

[1] Continual Learning in Low-rank Orthogonal Subspaces, NeurIPS, 2020
[2] Few-Shot Incremental Learning with Continually Evolved Classifiers, CVPR, 2021
[3] Self-Promoted Prototype Refinement for Few-Shot Class-Incremental Learning, CVPR, 2021
summary_of_the_review: As a simple idea, more discussions and empirical supports are expected to make it more convincing. I'll raise my score if all the concerns are effectively addressed.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper leverages a simple but effective regularization technique for few-shot incremental class learning. The assumption is that at the first stage we have a lot of data and can train a good feature extractor and a network that can classify all the classes to a good degree. Given a few examples of a new unseen class, the authors enforce that the weights of the final layer of the classifier are close to the space that is spanned by the current classifier's last layer weights. They achieve state-of-the-art results on this very challenging task.
 

main_review: **Strong points**
Very thorough experiments. I liked the visualizations in Figures 2 and 3. This gives a very good understanding of the extent of the model and how it performs. I also liked the idea of adding classes in each session. The method is novel and intuitive for solving this problem. Finally, the paper is easy to follow and well-structured.

**Weak points**
This method does not work if the feature extractor cannot do a good job. An important part of training a new classifier is to learn the feature extractors. This approach misses this in both ways. First of all, when new classes are introduced, their features are not added to the feature extractor. Second, if the feature extractor does not focus on an important feature in a future class, the method is expected to fail.

**Review** 
Based on strong points and weak points, I vote for acceptance of the paper. I think it would be great if the authors can explore the points I mentioned in the weak points, but still, the paper itself is an interesting idea with very good experiments. In addition the idea of using text for further improving the regularization kind of weighted my decision more toward acceptance. It shows that there are interesting ways to develop this method.
summary_of_the_review: The paper proposes a method for using regularization on the final layer of a classifier to improve the performance on novel unseen classes with very few examples. Then delves into the proposed idea and studies it with very good and justifiable experiments and visualizations. As a result, I vote for acceptance.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes subspace regularization technique for incremental few-shot learning. The high-level idea is to find the basis vectors of the subspace spanned by the base classes, and then project the new classes into the subspace. The regularizer encourages the new class weights to be similar to the projected vectors with an L2 loss. Additionally, the paper also proposes to use semantic word vectors to perform regularization. Experimental results confirm that the proposed method is significantly better compared to Tao et al. (2020) and Chen & Lee (2021) either with an additional memory buffer or without, but it is unclear whether it is due to a better few-shot learning algorithm or simply a better pretrained checkpoint on the base classes. Moreover, the benefit of semantic regularization is not significant.
main_review: Strengths:
- The paper presents a simple model and strong experimental results.
- The experimentation seems solid. It has evaluated on single and multi-session setups, with or without a memory buffer.

Weaknesses:
- It is unclear where the most benefit of the proposed model comes from. Based on Figure 2, it seems that Chen & Lee (2021) has a much worse checkpoint for base classes, and perhaps as a result the novel class learning is also not as good. It would be good if the authors can experiment both Tao et al. (2020) and Chen & Lee (2021) with the same pretrained checkpoint to showcase the real benefit in the few-shot learning phase.
- The type of subspace regularization exists in non-incremental few-shot learning and I wonder if the authors can provide an empirical comparison on that. It is unclear why subspace regularization depends on the incremental aspect. See Simon et al. (2020), Devos & Grossglauser (2020), Yoon et al. (2019).
- The semantic regularization technique provides a small gain on the multi-session benchmark and very marginal gain on the single session benchmark. This makes the contribution less focused and I suggest only include this in an additional study in the experiment section.
- As pointed out by the authors, the current framework only aims at learning the linear classifier in a sequential manner, which limits the ways it can be applied.
- If the number of samples is small (as is the case for few-shot learning sessions), then one could afford to do joint training at each session and store all the support examples. Such a joint training baseline should be provided for comparison (as was provided in Chen & Lee).

References:
- Simon et al. Adaptive Subspaces for Few-Shot Learning. CVPR 2020.
- Devos and Grossglauser. Regression Networks for Meta-Learning Few-Shot Classification. ICML Workshop on Automated Machine Learning 2020.
- Yoon et al. TAPNet: Neural Network Augmented with Task-Adaptive Projection for Few-Shot Learning. ICML 2019.

summary_of_the_review: In conclusion, my main concern is that this work may not have the same starting checkpoint when comparing to prior works and therefore it is hard to evaluate its empirical success in the few-shot learning phase. Another main concern is that the relation between subspace and incremental learning is unclear, since subspace regularization could also be applied to standard few-shot learning, and therefore comparisons to subspace methods on standard few-shot learning seems required. Other aspects of this work look solid to me. My initial evaluation of this paper is “weak reject”.

Authors' response on joint training addressed my concern. However, my other concern about comparison to Tao 2020 and Chen & Lee 2021 was not fully addressed. That being said, I understand there might be difficulty reproducing the numbers. Overall, I increased my rating from 5 to 6 to acknowledge the effort in responding my review comments.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors tackle the problem of few-shot class incremental learning (FSCIL). In this setting, a learner is first trained on a set of base classes for which plenty of data is available. Then, it must learn from small sets of novel classes while retaining good performance in the whole sequence. In order to prevent the classifier from overfitting new classes, they authors propose to regularize novel class weights to keep them in the subspace of base classes, thus making the classifier rely on known features rather than new features that are probably spurious. In addition, they also show that it is possible to align new classes with base classes based on their similarity in a learned semantic space such as Glove. Experimental results on miniImagenet and tiered-Imagenet show that their method obtains better performance than previous state of the art.
main_review: Strengths
=======
* The method is technically sound, simple, and it improves the performance on multi-session FSCIL.
* The text is well-written.
* The authors provide enough details to reproduce their results and they promise to release the code public.

Weaknesses
==========
* The improvement on the single-session setup is not clear. However, I value that the authors were honest about it. In fact, in this setup simple fine-tuning seems to be very effective. 
* There are some relevant works that were not cited such as [A,B,C,D,E].
* Some previous work such as [C, Cheraghian et al.] evaluated their models on CUB and CIFAR. Could you clarify why you did not use the same evaluation as them?
* The main contribution (aligning new weights with previously learned ones) is not new (the authors already cite previous work), but the way they do it seems new to me. It would be good if you clarified what exactly in your work is proposed by you and what is borrowed from previous literature.

[A] Simon, Christian, et al. "Adaptive subspaces for few-shot learning." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.

[B] Tao, Xiaoyu, et al. "Topology-preserving class-incremental learning." European Conference on Computer Vision. Springer, Cham, 2020.

[C] Tao, Xiaoyu, et al. "Few-shot class-incremental learning." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.

[D] (Arxiv, Optional) Yan, Kun, et al. "Aligning Visual Prototypes with BERT Embeddings for Few-Shot Learning." arXiv preprint arXiv:2105.10195 (2021).

[E] Xing, Chen, et al. "Adaptive cross-modal few-shot learning." Advances in Neural Information Processing Systems 32 (2019): 4847-4857.

summary_of_the_review: The proposed method is simple and effective and the paper is well-written. On the other hand, the main contribution is small and I have some questions about the experimental setup and relationship with previous related work (see weaknesses). Overall, I think that the simplicity and effectiveness outweigh the negative aspects and **I would be inclined towards accept if the authors resolve my concerns and those of the other reviewers.**
review_title: null

Meta-review: The paper proposes a subspace regularization technique that encourages the new class weight vector to be in the subspace spanned by those of the base classes for few-shot class incremental learning. Even though similar techniques exist in few-shot learning literature, reviewers appreciate the simplicity of the method and thorough experiments. The authors have revised the paper to include missing references suggested by reviewers during the rebuttal. They were not able to add experiment comparisons to Tao et al. (2020) and Chen & Lee (2021) as requested by reviewer Vrap due to missing code release.  Please consider adding them in your draft later.
Decision: Accept (Poster)