Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The method augments rehearsal-based methods for continual learning. At the heart of the method is a measurement of the influence of each example on the stability and the plasticity of the algorithm. First, the authors introduce a method for estimating said influence. Second, they present a method for using the influence to regularise the parameter updates. Third, they show how to use the influence in order to select which examples of past tasks to store in memory.
strengths_and_weaknesses: The paper introduced an interesting direction of combining research on Example Influence with Continual Learning. It appears that the methods they introduce are novel and well motivated. Finally, the experiments show good improvement over the many rehearsal-based baselines.

However, I feel that the paper’s presentation can benefit from another iteration. There are many sentences which need to be improved and that at the moment hinder the readability of the paper. E.g. lines 44, 52, 190, and 203. Moreover, I was not left with the impression that the presentation of the background material, in particular on influence functions, was satisfactory. For instance, I am confused by eq. (6) and the use of iff. 
Apart from the presentation, it’d be good if the experiments also introduced baselines which represent other CL directions, s.a. regularisation-based methods.

questions: Do you anticipate that your method would work on sequences of tasks with different input domains? For instance, if you had FMNIST classification mixed with CIFAR10.
limitations: I did not see a discussion on the limitations of the method. I’d be curious to better understand the number of tasks the method can handle before it breaks, possibly as a function of the memory size.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper investigates the question of how important each example is for plasticity and stability in continual learning. The authors motivate their approach from the perspective of the Influence Function which was previously used as an explainability method. However, since computing the actual Influence Function requires computing an inverse of the Hessian, the authors decide to instead propose MetaSP, a two-level optimization technique, which aims to approximate the Influence Function. Equipped with this, one can find a loss balancing factor $\gamma$ which is Pareto optimal with respect to stability and plasticity, and then use this balanced loss for training as well as choosing which examples to keep in the buffer. Finally, the authors show an empirical evaluation on multiple datasets and perform additional analysis of the method.
strengths_and_weaknesses: Strengths:
- The idea of disentangling plasticity and stability through IF-like approach and then using this information to get better performance seems novel and intriguing.
- The empirical evaluation is quite thorough and the proposed method performs well. In particular, I appreciate including the training time as the algorithm seems to be computationally complex. 
- The ablation studies are useful and give a deeper insight into the method. It's interesting to see how the number of examples with a positive impact on plasticity and stability changes with time. 

Weaknesses:
- The major weakness of the paper from my perspective is that in the end, I'm not sure how well the Importance Function motivation connects with the final MetaSP algorithm. Intuitively, these two seem connected, but a theoretical or empirical argument confirming that would be very useful. In particular, do you know how well the MetaSP approach approximates the gradient from Equation (6)? Can we show their equivalence, either theoretically or by running some experiments on toy problems? The authors hint at this kind of result in Section 4.1, but no hard proof is given.
- The fact that the algorithm is actually applied only on the last 5 epochs out of 50 should be highlighted. This is only mentioned in a single subsection near the end of the paper but seems quite crucial. Have you checked how well the other methods perform in this scenario? For example, what happens if we use fine-tuning for 45 epochs and then ER for 5 epochs? 
- The presentation could be improved, as the paper is hard to read at times. There are some minor mistakes (see the Minor comments section below) and in general I think the paper could be more self-contained. In particular, checking the previous works was necessary for me to understand Section 5.1 (SP Pareto Optimal), as the authors do not list the KKT conditions and do not really explain what the result of Eq (10) is. Grammar could also be improved, but for the most part, the mistakes do not make it harder to understand the paper so this is not an important issue.
- I'm a bit skeptical whether K-Means on the input space learns anything more than just basic background colors. Have you checked how the examples get clustered? What happens if you do not perform clustering at all (i.e. 1 cluster) or just cluster the examples randomly?
- The paper does not list limitations although in the checklist the authors claim that it does.


Minor comments:
Line 22: "robust CL system should achieve outstanding S while keeping a stable P" - that statement might be a bit controversial as it depends on your desiderata - you might want a model that is plastic but can remember the past moderately well.
Line 96: What do you mean by performance? You are using the $p$ symbol which suggests probability, but that's a bit confusing.
Line 120: "Hessian [...] assumed as positive definite" - that's a pretty strong assumption if we're not close to the minimum (i.e. during the training).
Line 121: "Always out-of-memory" - that's a bit informal, given enough storage or a small number of parameters I think this operation should be tractable. Of course, I agree that in most cases this statement is true, but I would suggest using more precise wording.
Line 168: might be useful to state those KKT conditions
Line 225: I would suggest, when possible, reusing metrics from previous CL papers, as introducing new metrics make it difficult to compare between papers.
Figure 3: At what point is each of these plots generated? I.e. these metrics are dependent on the parameters $\theta$, so which $\theta$ are we using? The one from the end of the current task?
Figure 4: I don't quite understand this figure. Why are there three bars per task and why there are two separate plots? 
Figure 5: How was the Pareto front plotted?
Table 2: I assume you mean "training time" rather than "implementation time"?


**EDIT:** Authors' responses and improvements made to the paper convinced me to increase my score to Accept.
questions: I would like to ask the authors about the issues mentioned in the "weaknesses" section above. In particular, I'd like to highlight he first one:
do you have theoretical or empirical proof of how well your approach approximates the results obtained from the Importance Function?
limitations: I don't think a discussion on potential negative societal impact is needed. 
The authors do not list the limitations of their method and in my opinion, a section like that should be added. Just to name a few: 
- The method relies on experience replay, so it cannot be applied in settings where data privacy is crucial.
- The proposed approach is rather slow (2x slower than ER even with the "use MetaSP only in the last 5-epoch" trick), so it might be an issue in some settings.
- The method was implemented only for classification tasks. How hard would it be to extend it to other problems?
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors investigate the stability dilemma through the lens of the training data and the selection for the replay buffer.
They evaluate the influence of data points on stability and plasticity using small perturbation to data instead of computing hessian (for computation sake).
The authors use the stability and plasticity influence to optimize the gradient update and the sample selection.
The algorithm is evaluated on CIFAR10, CIFAR100, and Mini-Imagenet and compared with several state-of-the-art algorithms.

Overall I like the paper and the idea. I think, however, that the authors could realize some improvements to make the paper clearer and better.
strengths_and_weaknesses: Strengths:
The paper is well written and organized, with a detailed description of the method.
Sample selection is a crucial problem to study to improve replay methods.
the method is interesting and looks well-performing.

Weaknesses:
The definition of the metrics is not very clear. "Finished Accuracy" seems to be the classical "Final Accuracy". "First Accuracy" and its relationship with plasticity is not very clear.
Is it the average best performance realized on each task?
Maybe, "backward transfer" from "Gradient Episodic Memory for Continual Learning" Lopez-Paz et al would be more precise in measuring stability.
And "Final performance" would measure the mixture between S and P. I am not sure about what "Mean Average Accuracy" brings here.

questions: Table 1: "ours" method use random sampling selection? Maybe comparing the sampling selection to another example from the literature would make sense to assess the method better. 
(cf. eg "An Investigation of Replay-based Approaches for Continual Learning" Bagus et al)

What does that mean that a sample increases plasticity or stability? Beyond the definition provided in section 3, I miss the high-level idea....

l266 "We divide all example influence equally to 5 groups from the minimum to the maximum." how does this division is realized exactly?

l269 "The two observations suggest we should consider the example difference to improve model training." but you do not do it, right? So what is the point of this figure in the paper?

Fig.3: left is the evaluation of the data stored in the memory, and the left is the new data? I think the legend could be more detailed to make it clear from the beginning.

Tab 1: code color is not defined.

Is the data used for validation in the memory buffer used at some point for training?

Table 2: how many seeds lead to those results?

l.69 "4) By considering the example influence, in our experiments on both task- and class-incremental, better S and more stable P can be observed" I do not understand what this contribution is...


********
Some Comments:
**************

"KKT" is never defined.

I would move the task-incremental results in the appendix (task incremental is not very classical to study when using replay, but this is not problematic) and add std of results in the main body.

It would be better to group figures 3 and 4 or to modify figure 4, such as the legend is not in figure 3.
limitations: It would be nice to see in the table which results are statistically significant.

The legend of several figures could be improved.
review_title: null

Meta-review: metareview: There was a consensus among reviewers that this paper should be accepted. The paper investigates an interesting direction of combining research on Example Influence with Continual Learning. The methods they introduce was considered to be novel and well-motivated by the reviewers and the experiments show good improvement over the many rehearsal-based baselines.
confidence: Certain
Decision: Accept