Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a method to mitigate catastrophic forgetting in class-incremental few-shot learning (CIFSL). The key observation is that there may be many directions in the weight space that are flat with respect to the loss. If these directions can be identified, then updates may be freely made in these directions without affecting performance on the previous classes. In order to do this, a method called WaRP is proposed that utilizes the SVD of the network activations to construct a rotation such that the majority of directions are flat. For each subsequent incremental learning session, only a subset of the parameters with respect to the new basis are updated. The subset of parameters to be learned in each learning session are chosen based on a score criterion that relies on the magnitude of the gradients. Experiments on several standard CIFSL benchmarks demonstrate the improvements of the proposed WaRP method relative to baselines.
strength_and_weaknesses: Strengths
- Writing is clear and contains several interesting insights distributed throughout the paper.
- The method makes intuitive sense and should be easy to implement.
- The experiments are complete and contain a set of ablations justifying design decisions.
- The proposed WaRP method could have impact in a variety of transfer learning scenarios beyond CIFSL. Any setting where performance on previous tasks should be preserved can take advantage of the ideas behind WaRP.

Weaknesses
- The method seems to be primarily focused on preserving accuracy with respect to the base classes rather than the incrementally learned novel classes. In particular, the basis is computed only once after training on base classes, rather than updating after each session.
- After many learning sessions, it seems plausible that there could be a shortage of trainable parameters.
clarity,_quality,_novelty_and_reproducibility: The clarity is excellent. In particular, the introduction and Figure 1 motivate the problem well. The quality is also good, with complete experiments and ablations. With respect to novelty, the paper does not have a large amount of strictly novel material per se but rather pieces together various bits of knowledge into an elegant solution for catastrophic forgetting. In my view, this is a sufficient amount of novelty. Reproducibility also is good and code is included in the supplementary materials.

Some other comments:
- For the ablations over alpha, I found it slightly disorienting for the plots to focus on the end sessions only. Perhaps showing the performance increase or decrease relative to a baseline alpha might make the plots easier to parse?
- One aspect of the submission that could be improved is an analysis of the learned basis and masks. For example, are parameters masked evenly throughout the network, or are they concentrated in either the lower or upper parts of the network? If the latter, then this matches the intuition that lower-level representations can be fixed with fine-tuning performed on top. WaRP offers data-driven masking and thus is in a unique position to be able to answer these questions.

summary_of_the_review: Update post-rebuttal: The authors have addressed my remaining concerns with additional results and ablations. I will maintain my original rating.

---

Overall, this is a strong submission. It addresses a meaningful problem and provides an effective and elegant solution to tackle it. Experiments show the proposed WaRP method provides consistent accuracy improvements relative to baselines.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this work, the authors propose an algorithm to efficiently learn new classes in an incremental setup with a few-shot dataset at every iteration. Towards this, they propose WaRP - which first changes the basis of the parameter/weight space of the base network and then uses SVD to find parameters with less importance in that space. At every round, a fraction of parameters which are found to be unimportant (with a metric defined in this paper) are fine-tuned to learn new classes while the important parameters are kept fixed. Important parameters found after each iteration are added to the original set of important parameters. On standard few-shot class incremental learning (FSCIL) benchmarks, WaRP outperforms existing methods. 
strength_and_weaknesses: **Strengths**
* The narrative of the paper is crisp, easy to follow and enjoyable to read. Authors clearly motivate why changing the basis is important (instead of staying in the original parameter space) with an explanation in the two-dimensional space. After that, authors use the existing knowledge that activations reside in a low-dimensional sub-space to find unimportant parameters in this new basis and updates only those parameters for a given task while keeping the important parameters fixed. Every major component of the proposed algorithm is well motivated, clearly explained and intuitively makes sense. 
* Apart from only providing a more theoretical overview of the proposed method, authors also provide details on how the weight-space change of basis (re-parameterization) can be implemented for a fully-connected and convolutional layer. Authors also improve the computational cost of the method by organizing the parameters in a way such that standard automatic differentiation packages can be used.
* Experimental results are performed on multiple FSCIL benchmarks and the proposed method outperforms existing methods (on-average, ~1 point over the SOTA method F2M). Authors also provide reasonable ablation studies to showcase the importance of various components including weight space re-parameterization, amount of forgetting with the amount of parameters modified etc. - overall, the empirical evaluations properly justify the efficacy of the algorithm.

**Weaknesses/Clarifications needed**
* For me, a major missing piece in this paper is the lack of comparison against methods from the domain of neural network pruning. Although the authors compare against existing methods from the literature, but for me, an important baseline to compare this method would be to take a pruning algorithm e.g. any channel pruning method for CNN, use that to identify which channels are unimportant after every round and only fine-tune those channels for a given task. Apart from channel pruning, other methods like layer pruning or structured pruning are also relevant to this discussion.
* Modifying the basis by re-parameterization requires non-trivial amount of work specific to each neural network operator. Authors provide guidance on how to do it for Conv/FFN, but I'd like to understand if this process can be generalized and applied to other kind of operators - multi-head attention from a Transformers network architecture or a transposed convolution operator for example. This can potentially hinder the adaptability of this method to other type of tasks or architectures.
* I'd like to see a plot on how many parameters are staying unimportant after every round - this will help us understand how does this method scale to more number of tasks.
* Finally, some details on the overall computational cost increase (at the beginning to change the basis and then for every iteration) compared to doing a standard fine-tuning will help us understand the cost/performance trade-off better.


clarity,_quality,_novelty_and_reproducibility: **Clarity**

The narrative of the paper is clear and straightforward. Although the proposed method is somewhat complicated, authors do a great job in motivating each piece and explaining the underlying details of each component carefully. I appreciate the extra attention authors have paid to explain the overall procedure in a succinct manner (sec 3.3).

**Quality & Novelty**

I find this paper to be meeting the quality bar for a top-tier ML conference. The writing is crisp and pleasure to read, proposed method is well motivated and explained and finally, sufficient empirical results are provided to validate the hypothesis. The idea of moving the weight to a different basis and performing weight update in that space is unique and as a whole, there is enough novel contribution in this work.

**Reproducibility**

Authors have provided enough details pertaining to their proposed algorithm, backbone and other hyperparameters. My only minor concern regarding reproducibility is the difficulty in terms of extending this method to other neural network operators in future.



summary_of_the_review: Overall, I found the paper to be well written, clearly motivating the proposed method along with explaining each component in details and finally showcasing the efficacy of the method on standard benchmarks against current SOTA methods and via additional ablation studies. At this point, based on my opinion alone, I am leaning towards accepting this paper. I'd still like the authors to respond to my comments - especially regarding the channel pruning style baselines.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper introduces weight space rotation process into the few-shot class-incremental learning. Specifically, the proposed method can identify the important parameters and freeze them in the following sessions.
strength_and_weaknesses: Strengths: 

The paper is well-written. The motivation is clear and the method is technically correct. 

Weaknesses:

1). In experiments, the comparisons with the previous methods are not enough. The following papers are not compared, but they seem to outperform the proposed method:

[1] Few-shot class incremental learning by sampling multi-phase tasks. (TPAMI)

[2] Subspace regularizers for few-shot class incremental learning. (ICLR ‘22)

[3] Metafscil: A meta-learning approach for few-shot class incremental learning. (CVPR ‘22)

[4] Constrained few-shot class-incremental learning. (CVPR ‘22)

[5] Few-shot class incremental learning via entropy-regularized data-free replay. (ECCV ‘22)

[6] Few-shot class-incremental learning from an open-set perspective. (ECCV ‘22)

I notice that the authors give references for [2] (introduction) and [1] (related work), but I do not find a reasons to ignore them in the comparison in experiments.

2). Fixing part of the network, though seems to be novel in FSCIL, but has been studied in the very similar task CIL (Adaptive Aggregation Networks for Class-Incremental Learning, CVPR ‘21). The motivations are very similar. It would be better to have a discussion instead of claiming “first work that makes changes to the parameter space itself”.

clarity,_quality,_novelty_and_reproducibility: The paper is with a good clarity.
summary_of_the_review: The paper is well-written and has a clear motivation. But the experiments do not support the advantage of the proposed method. Comparison is not enough. Necessary discussion is lacking to support the novelty.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors propose a method to rotate the weight space in a way that newly acquired knowledge does not interfere with the previous one. In the rotated space, only few dimensions are important for previous tasks and thus, they can be frozen to encode new knowledge in the remaining weights. The proposed method outperforms all the considered the state-of-the-art baselines.
strength_and_weaknesses: Strengths
=======
* The proposed method is sound and generalizes the methods of Saha et al., 2021 and Wang et al., 2021.
* Results are state of the art
* The authors provide ablations and code, which makes their work more reproducible

Weaknesses
=========
* Given the generality of the method, it is not clear why it is only tested on class-incremental learning
* Only papers from 2021 are considered for comparison, missing works like [A]
* I found the notation complex and I had to read the paper multiple times to understand all the equations. I still have trouble with the relationship between Hilbert spaces and equation 4.

[A] Hersche, Michael, et al. "Constrained Few-shot Class-incremental Learning." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.



clarity,_quality,_novelty_and_reproducibility: Clarity
=====
* For the most part the paper is clear. However, I found the description of the method complex and it looks like it could be simplified.

Quality
=====
* The quality of the related work could be improved. For example [B] presents a method to rotate the weights of a neural network to perform EWC, which is related to your work.

Novelty
======
* The proposed method is incrementally based on the work of Saha et al., 2021 and Wang et al., 2021, which the authors acknowledge (which is a positive thing).

Reproducibility
===========
* The authors provide all the necessary tools to reproduce their method.

[B] Liu, Xialei, et al. "Rotate your networks: Better weight consolidation and less catastrophic forgetting." 2018 24th International Conference on Pattern Recognition (ICPR). IEEE, 2018. 
summary_of_the_review: The proposed method is sound and achieves good performance. My main concerns are about comparison with recent papers like [A], generality (why few-shot incremental and not class incremental, for instance), and the amount of novelty. Overall I think this is a good paper and I would be willing to raise my score after discussion.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: # Summary of Contribution

This paper describes a method for Few-Shot Class-Incremental Learning that is based on identifying directions in parameter space that are "flat" with respect to the loss function. The authors propose to identify such directions and describe an approach (called WaRP) based on Singular Value Decomposition to represent the parameter space of the model in a basis in which most directions are "flat". Experiments on several standard few-shot class-incremental learning benchmarks demonstrate the improved performance of WaRP over baselines and state-of-the-art approaches.

# Strengths

+ **Clarity**: The paper is very well-written and provides clear motivations and insightful observations throughout all stages of the technical development. The narrative gracefully shifts between theoretical and technical levels of description with illustrations of how basis change in parameter space can be implemented for linear and convolutional layers. Reproducibility is high given the clear technical descriptions (and provided code).

+ **Experimental Results and Analysis**: The experimental results are thorough and the ablations provided illustrate the importance of each element of the contribution, including an analysis of forgetting as a function of the amount of parameters modified.

+ **Insights and Intuitions**: The paper contributed something interesting to the discussion on continual learning in general through its insights and methodical analysis of the effect of basis changes on few-shot incremental learning. 

# Weaknesses

+ **Missing Comparisons with Recent Work**: Several reviewers commented that the paper lacks comparison with more recent work on few-shot continual learning. During the discussion phase the authors provided additional results (also justifying why it is difficult to compare with some published results due to architectural differences.

+ **Complexity**: Some reviewers commented at a few passages are difficult to comprehend on first reading, which however is perhaps to be expected given the theoretical nature of the central thesis of the contribution. In rebuttal the authors improved some aspects of this, and the supplementary appendices do a good job of providing additional details useful for deeper understanding. 

# Summary

The reviewers are nearly unanimous in their positive opinion of this paper. The main outstanding concern was the lack of comparison with several more recent works on few-shot CIL. New results were provided during the discussion phase, which however leaves the question of whether or not these additional results constitute a significantly different work that that originally submitted. However, the positive aspects of this contribution greatly outweigh the negatives and it deserves a place at ICLR.

justification_for_why_not_higher_score: I think this paper, given its excellent clarity of presentation and crisp narrative, could make a decent oral presentation. In the end, however, I think it is somewhat lacking in terms of novel elements. I would not be disappointed to see it bumped up to an oral. 
justification_for_why_not_lower_score: The reviewers were nearly unanimous in their praise for the paper, for the clarity of presentation and the insights it provides through a sustained crisp narrative. The experimental results are equally well thought-out and excecuted.
Decision: Accept: notable-top-25%