Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 9: Top 15% of accepted NeurIPS papers, strong accept
review_content: summary: This paper describes an approach to incremental few-shot learning based on
finding a good starting configuration in parameter space that leads to more
stable incremental few-shot learning with less forgetting. The authors propose
to learn an approximately flat region on the initial training set and to then
constrain few-shot incremental learning to remain within this flat region and
thus minimize forgetting. Experimental results are given on a wide range of
few-shot incremental benchmarks, including CIFAR-100, miniImageNet, and CUB-200.

main_review: This paper is very well-written with excellent motivations, empirical and
theoretical follow-through, and convincing experimental evaluation. I have
picked and poked and prodded at the work, but can find very little to improve. I
especially appreciated the empirical motivations of the work in section 3, as
well as the convergence analysis. A few that I think should be addressed:

1. The motivation of starting from a flat region and regularizing incremental
   learning based on it is reminiscent of Elastic Weight Consolidation, but
   motivated from a different starting point -- pun intended since EWC does not
   constrain the initial starting point, but rather only regularizes based on an
   approximation of the loss surface at an initial solution. It would be
   interesting to see how and why regularization using the Laplace approximation
   of the loss landscape (a la EWC) around a flat minima works when compared to
   clamping to remain in the flat region.
2. I do not find any discussion of the flat region bound $b$ used to determine
   the flat region. I expect a smaller $b$ leads to a smaller flat region and
   thus less plasticity, and a larger $b$ to a larger flat region and more risk
   of forgetting. It would be nice to see an ablation on this parameter to
   understand better this tradeoff.

**To summarize**: I really like this paper and I think it is novel approach that
adds something very interesting to the discussion on incremental learning.

**Post-rebuttal Summary**: I appreciate the author efforts to clarify and provide additional ablations. Other reviewers point to somewhat more critical blemishes and/or questions about the proposed approach, but I am still very positive about the work. The manuscript is very well-reasoned and well-written, and I think it adds something interesting to the discussion on few-shot IL. I do suggest that the authors consider carefully the comments from reviewer dLsD on carefully articulating the contribution of the work and the relationship between flat minima, few-shot learning, and incremental learning.  
limitations_and_societal_impact: Discussion of limitations and societal impact is entirely missing.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper introduces a novel approach for few-shot incremental learning, which searches for the flat local minima of the base training objective function and then fine-tune the model parameters within the flat region on new tasks. This way, the model could learn the new classes without forgetting the old ones. The approach is well-motivated and is scientifically sound.
main_review: The paper is well-documented and clearly written. The related work section covers the most relevant papers in the literature.  Extensive experiments have been performed and the approach was conveniently compared with the current state-of-the-art, demonstrating its superiority.
Here are my concerns:
1. Section 4.1: Definition 1, Condition 1: 'b_i' is not introduced. Who is 'epsilon'?
2. In Section 4.3, you make a series of assumptions. How realistic are they in practice? What measures do you take in order to guarantee that they are met?
3. How do you find the optimal parameter 'b'? What value do you use in your experiments? A study of the effect of the parameter 'b' on the performance would be interesting.

limitations_and_societal_impact: The paper does not have a negative societal impact.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The proposed method is a nice trial for tackling the issues in a challenging data setting --- class incremental learning with few-shot samples in every new class. The paper writing is clear to show that the converge of the proposed method is theoretically guaranteed and the performance shows better than some baselines --- incremental learning methods (but missing incremental few-shot learning methods introduced in Related Works [48, 42, 62, 9, 34, 61]). 
main_review: I have two concerns about this paper. One is the performance should be highly related to the discrepancy gap between base classes and the coming (new) classes in incremental learning phases but there is no analysis on this. The other one is the motivation is not very strong to use in the only setting of few-shot samples in new classes. Please check the detailed comments.

1. The performance looks good but this paper only shows its comparisons with some incremental learning methods (baselines methods implemented in a new incremenral few-shot settings). The comparisons to closely related works such as [48, 42, 62, 9, 34, 61] are missing totally. 

2. The performance of the models are related to the pattern differences between base classes and the coming (new) classes, which is not analyzed in the paper (as it used only one fixed setting of class splits). Given a larger difference between base and new classes, does the found flat minima (using base classes) still work for learning new classes? In addition, the experiments are a bit limited to small datasets. 

3. Flat minima regions guarantee a good generalization of the trained model to new classes. Is this especially helpful for incremental learning in few-shot settings? It is interesting to see its empirical performance on the standard settings of class incremental learning, e.g., using ImageNet which is large-scale or at least using Sub-ImageNet.




limitations_and_societal_impact: NA
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes an incremental few-shot learning method focusing on training the base model that can easily learn new classes, preserving the classification capability for the old classes. When training the base model, the proposed method, Finding Flat Minima (F2M) finds the flat minima where the loss function does not change a lot even if a random noise is injected. When the novel classes are given, F2M updates the feature extractor and clamp the parameters to make the feature extractor resides in the flat minima region. 
main_review: [Strength]
1.	The idea of finding flat minima to overcome the catastrophic forgetting and ensure the base performance is interesting. The proposed F2M outperforms the other incremental few-shot learning methods for all sessions except for the base classes training phase (t=1).
2.	The proposed F2M does not saves the data from prior sessions except for prototypes for inference. 
3.	The paper is well written and easy to follow. Also, the detailed ablation experiment results are provided to help understanding. 

[Weakness]
1.	Although the paper argue that proposed method finds the flat minima, the analysis about flatness is missing. The loss used for training base model is the averaged loss for the noise injected models, and the authors provided convergence analysis on this loss. However, minimizing the averaged loss across the noise injected models does not ensure the flatness of the minima. So, to claim that the minima found by minimizing the loss in Eq (3), the analysis on the losses of the noise-injected models after training is required.
2.	In Eq (4), the class prototypes before and after injecting noise are utilized for prototype fixing regularization. However, this means that F2M have to compute the prototypes of the base class every time the noise is injected: M+1 times for each update. Considering the fact that there are many classes and many samples for the base classes, this prototype fixing is computationally inefficient. If I miss some details about the prototype fixing, please fix my misunderstanding in rebuttal.
3.	Analysis on the sampling times $M$ and noise bound value $b$ is missing. These values decide the flat area around the flat minima, and the performance would be affected by theses value. However, there is no analysis on $M$ and $b$ in the main paper nor the appendix. Moreover, the exact value $M$ used for the experiments is not reported. 
4.	Comparison with single session incremental few-shot learning is missing. Like [42] in the main paper, there are some meta-learning based single session incremental FSL methods are being studied. Although this paper targets on multi-session incremental FSL with different setting and different dataset split, it would be more informative to compare the proposed F2M with that kind of methods, considering that the idea of finding flat minima seems valuable for the single session incremental few-shot learning task too.

* There is a typo in Table 2 – the miniImageNet task is 5-way, but it is written as 10-way.

-------------------------------------------------------------------------------------------------------------------------------------------
Post Rebuttal
- Reviewer clarified the confusing parts of the paper, and added useful analysis during rebuttal. Therefore, I raise my score to 6.

limitations_and_societal_impact: The authors did not addressed the limitations of  the paper.
review_title: null

Meta-review: After reading the paper and other reviews, and interaction with the authors via the rebuttal, all reviewers recommend to accept the paper. It's well written and motivated, and the proposed method, which is relatively simple, is reasonably evaluated in the context of incremental few-shot learning.
Decision: Accept (Spotlight)