Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: **Summary of paper**

This paper draws links between three common regularisation methods for continual learning: EWC, MAS, and SI. It shows that MAS and SI approximate the Absolute Fisher matrix. The authors provide many experiments to test their claims and assumptions. Finally, the authors also propose a cheaper way to run EWC.

**Review summary**

I really like the majority of this paper. Unifying these regularisation methods is great, and not obvious (particularly in the case of SI). The accompanying experiments are crucial and well-conducted. The paper is also written well, with an emphasis on good research practices. However, I have an issue with the proposed quicker/cheaper update for calculating the (diagonal empirical) Fisher Information Matrix for Online EWC ("Batch-EF"), as I detail later. If it were not for this, this paper would be a clear accept for me. I hope to resolve this issue with the authors during the discussion period, depending on which I can raise (or lower) my score.

**Pros of paper** (mostly already written in the "Review summary")

1. The paper is written well, with good detail and very good experiments.
2. The work is of significance for continual learning, with interesting conclusions.

**Cons of paper**

3. I am not convinced that the minibatching that the authors suggest (both for SI as an approximation to the AF and for EWC in the last paragraph of Section 5) is correct ("Batch-EF"). It appears to me that by minibatching instead of squaring each gradient element, one should obtain much worse approximations to the empirical Fisher information matrix ("EF"). 
Intuition: By calculating the gradient over minibatches and then squaring, one is reducing the noise that is being squared. Intuitively, this must affect the EF calculation in a bad way. For example, consider a full-batch calculation. In this case, Batch-EF will just be $g^2$. At the end of training, when we have converged to a low loss, this will be very small. In Appendix D.2, the authors argue that when gradient noise >> gradient, then Batch-EF $\approx$ EF, and derive that Batch-EF has larger values than EF. However, I expect Batch-EF to have smaller values than EF because of the reduced noise on average.
Additionally, I have myself experimented in the past with Batch-EF. I did not find that Batch-EF gave the same results as EF for EWC on similar benchmarks. I do not know why, in this paper, the authors found that the two gave same results (Table 1); perhaps it only works for specific hyperparameters.
Finally, a small note that may be of interest to the authors: the HAT codebase (github.com/joansj/hat) implements EWC as a baseline, however, my collaborators and I found that they implement EWC differently/incorrectly. One of the ways they are different is to do Batch-EF (along with other differences).
4. I am also not convinced that OnAF ("Online Absolute Fisher") and AF are / should be the same. After training, individual gradients should be relatively small (as we have converged to a solution), meaning that I would expect the AF to have small values in general. However, during training (especially near the beginning), gradients can be large, meaning that OnAF can end up having large values. Empirically, the Pearson correlations in Figure 2 (mid) show differences between the OnAF and AF versions.

**Additional suggestions to authors**

- The authors could consider adding a reference for the Absolute Fisher (Section 4.1). Can they say anything about the links between the Absolute Fisher and the empirical Fisher?
- Typo Section 4.2 second para: "Max-likeilhood"
- I felt that Section 5 got complicated, with many algorithms that need to be compared. I strongly recommend splitting the experiments part (Section 5.3) into experiments relevant for the two preceding sections (5.1 and 5.2) to reduce the complexity of writing.

**Update to review**

I am increasing my score from 5 to 6. I believe this paper is a good paper. However, an extremely extensive discussion with other reviewers has left some questions / concerns. Although I disagree with some of these, I agree with others:
- Some claims are overstated in the paper. The authors already changed these claims somewhat in the updated paper. Some reviewers are arguing for further changes. I think some claims can still be reduced, particularly, the link between AF and EF (and hence the link to EWC).
- One of the biggest reason I find this paper is interesting is not mentioned (enough) by the authors. In my opinion, this is a big reason why the work is significant, and if I were writing the paper, I would put it as one of the biggest motivations:
    -  There have been works recently looking at the Generalised Gauss-Newton approximation (= EF for classification), and trying to view optimisation algorithms as approximating the Hessian matrix. For example, see Khan et al., 2018 ("vAdam"), Kessler et al., 2020 ("BAdam"), Zhang et al., 2018 ("Noisy Adam"), Osawa et al., 2019 ("VOGN"). Such works provide evidence that different approximations of the EF can work well. Although I am not aware of previous works using the absolute value of gradients (as in AF), this paper provides evidence that such an approximation might be worth considering. Should we try and approximate the Fisher matrix in more ways in CL? 
- Finally, it is my personal opinion (although others disagree) that the current paper is significant enough / provides enough insight already to be a good paper. However, performing a further state-of-the-art experiment or similar would undoubtedly improve the quality significantly.

I very much look forward to an updated version of this paper.
review_title: AnonReviewer2 Review

final_score: rating: 5: Marginally below acceptance threshold
review_content: ******************************************************************
********************** POST DISCUSSION UPDATE **********************
******************************************************************
Thank you to the authors for the discussion. Given that the relationship between AF and F has now been addressed, I will increase my score. However, since the connection mainly hinges on the empirical correlation between the two, I still don't think that the paper quite establishes the theoretical connection between the three continual learning methods that it aims for. Finally, I'm not quite convinced by the potential impact of the insights, which seem fairly specific to choosing the batch size of SI, so overall I think the paper still is below the acceptance threshold. 
******************************************************************
************************** END OF UPDATE***************************
******************************************************************

The paper unifies two regularisation-based continual learning methods from the literature (Synaptic Intelligence and Memory Aware Synapses) by arguing that they both approximate the "Absolute Fisher", a variant of the Fisher Information that averages absolute instead of squared gradients, to determine the regularisation weights. By this the authors claim to establish a relationship between these two approaches and Elastic Weight Consolidation, which approximates the diagonal of the Fisher.

While the paper is well-written and -structured, and SI and MAS are popular methods in the literature, I'm not convinced by the link through the "Absolute Fisher". The Fisher Information is a well-studied quantity with many appealing statistical properties and even a subtle change to its definition such as replacing the expectation over the labels with the empirical data breaks many of these (see the referenced Kunstner et al. paper). So simply taking absolute instead of squared gradients and stating that this is "a natural variant" is not a sufficient basis for a paper, especially considering that the term has not appeared anywhere in the literature before as far as I could tell. I think the authors really need to either provide some references to existing work or establish themselves that this variant makes sense theoretically. So overall my recommendation is to **reject** the paper.

Further, I'm not really sure what the takeaway from the proposed unification of these methods is even assuming that it can be put on a more solid foundation. Is it that through the relationship to EWC they are all approximately Bayesian? How would this inform future work? I feel like the paper pokes in that direction through empirically comparing some variants of SI and MAS, but most variants perform almost identically and are highly correlated, so again I am not sure what exactly to take away from this.  The part on SI relying on its 'bias' seems potentially interesting, but since -- if I understand things correctly -- this is an unexpected empirical result from the theoretical point the paper is trying to establish, it would be necessary to go a bit more in depth. To summarise, I think the authors need to more clearly explain and establish the impact of their work.

As a minor final point, I do not find the proposed efficient version of EWC convincing at all. First, estimating the Fisher as the square of the averaged gradients has been done before, for example it is implemented in the [tensorflow KFAC codebase](https://github.com/tensorflow/kfac/blob/3ee1bec8dcd851d50618cd542a8d1aff92512f7c/kfac/python/ops/fisher_factors.py#L945-L950). Second, the empirical comparison does not make much sense, since Jacobians in Pytorch can be calculated efficiently for linear and convolutional layers without too much effort through backward hooks (see e.g. [the backpack library](https://backpack.pt/) for an implementation). In more recent versions, Pytorch also provides autograd functions for computing jacobians natively -- I'm not sure how efficiently they are implemented, but in any case the empirical comparison here needs to use an efficient, batched implementation for calculating the Fisher in order for it to be meaningful. Calculating gradients for individual data points is neither sensible nor necessary.
review_title: Not convinced by connection through the "absolute Fisher"

final_score: rating: 3: Clear rejection
review_content: ## Summary

This paper attempts to unify the three most prominent regularization-based continual learning methods: EWC, MAS, and SI.
While EWC has a solid theoretical justification under certain assumptions, the other two are based on intuition and heuristics.
The authors show that the importance weights of MAS and SI are similar to the diagonal absolute Fisher.

---

## Pros

- Connecting the path integral of SI to the Absolute Fisher is interesting.
- The paper is well-organized and easy to follow.

---

## Cons

### Lack of justification for the Absolute Fisher

The Absolute Fisher seemingly appears from nowhere. In contrast to the diagonal Fisher, it does not seem to have any other interpretation. The authors merely show that MAS and SI are similar to using Absolute Fisher as importance weights. The similarity alone is not enough to be a theoretical explanation for effectiveness. I think the authors should justify that the Absolute Fisher is an optimal regularization under certain assumptions. 

### Weak connection between the diagonal Fisher Information and the Absolute Fisher

Despite searching the web, I could not find any proper material on the Absolute Fisher that explains its connection to the diagonal Fisher.
The only similarity that I find between the Absolute Fisher and the original Fisher is that they can be computed with the gradient. I do not see any reason to call it the Absolute *Fisher*.
Therefore, I cannot agree with the claim that this paper presents a unified framework, just because it fits several methods into distinct concepts that have "Fisher" common in their names.

### Doubts about the value of a unified framework

I ask the authors a more fundamental question: do we really need a unified framework? Although the derivation of EWC is mathematically grounded, it heavily relies on certain assumptions about the shape of the loss surface. The assumptions make the use of the diagonal Fisher information an optimal choice. However, considering its poor CL performance, those assumptions seemingly do not hold in deep neural networks.

Similarly, I think other methods, such as MAS and SI, can also be optimal under different assumptions. Therefore, I argue that it is more meaningful to investigate which condition/assumption makes a certain algorithm optimal, rather than framing all algorithms into one unified framework. I suspect that the latter case is not even possible.

The experiments also support my claim: AF does not necessarily outperform MAS or SI. Instead, the focus of this paper is the correlation among methods. I think the best regularization method varies depending on the specific model architecture and task design.

### Minor issues
- Assumption 1 in Section 5.2 depends heavily on the batch size. The gradient noise will quickly diminish as the batch size grows.

---

## Overall evaluation

It is hard for me to agree with this paper's fundamental motivation: a unified framework for regularization methods. Also, the overall logic of this paper is too weak. Since I could not see other utility in this paper, I recommend rejection.

---

## Post rebuttal

In response to my doubts about a unified framework, the authors claimed that their *theory* could *predict* an algorithm or hyperparameter's performance.
Since there was no description of the theory, I assumed that the theory is:
> If an algorithm is different from AF, its performance is expected to be poor.

And the authors refuted my interpretation:
> We claim that SI and MAS work because they are similar to AF.

The authors claim that the theory somehow applies to SI and MAS but not others. However, I could not find any description of why the applicability is restricted and to what extent it is applicable. I think these are vital parts of a proper theory. Without them, a theory is useless since we cannot decide whether it applies to a new CL algorithm until we actually run some tests.

Also, I want to emphasize that association is not causation. The authors should have claimed, "SI and MAS work, **and** they are similar to AF," instead of "SI and MAS work **because** they are similar to AF."

Even after the discussion with other reviewers, my concerns are not resolved.
Therefore, I retain my initial rating.
review_title: Three Methods Awkwardly Unified with Weak Logic

final_score: rating: 5: Marginally below acceptance threshold
review_content: Disclaimer: I am not an expert in continual learning even if I have already experimented with EWC, but rather my main expertise is related to FIM in other contexts.

This works aims at unifying 3 popular regularisation type continual learning methods, namely EWC, SI and MAS, by showing that under some assumptions they all relate to the Fisher Information Matrix. These assumptions are shown to hold on 2 different tasks trained using Adam.

I however found several limitations while reading your paper:

### Limitations regarding MAS

In the github repo of MAS (it's not entirely clear from their paper), they seem to consider the function $F$ to be the score for each class (i.e. the output of the last linear transformation), while in your paper your $\mathbf q_X$ is the class probability (i.e. the softmax). So their $\omega\left(\text{MAS}\right)$ is different from yours.

### Limitations regarding SI

Your argument regarding SI (sec 5.2) seems to be valid only when using Adam, while SGD+momentum is the standard for image classification nowadays. In my opinion you can come up with a more general argument even using standard stochastic gradient algorithm, e.g. the relationship between the FIM and the covariance of the minibatch gradients as already been studied e.g. in [1] and [2]. If your empirical analysis and conclusion that it is the bias that explains SI's performance still holds with SG (without Adam), then the argument is straightforward.

Moreover the denominator $\theta\left(T\right) - \theta\left(0\right)$ of eq. 4 is not discussed later in the text, or in other words, if $\tilde\omega\left(\text{SI}\right)$ is similar to $\omega\left(\text{EWC}\right)$, what about $\omega\left(\text{SI}\right)$?

### Conclusion

In conclusion, I really appreciate the effort of trying to unify which is certainly more useful than inventing countless slightly different variants of the very same technique. I would however like to see these points addressed before publication.


[1] Le Roux, N., Manzagol, P. A., & Bengio, Y., Topmoumoute online natural gradient algorithm, NeurIPS 2008.

[2] Thomas, V., Pedregosa, F., Merriënboer, B., Manzagol, P. A., Bengio, Y., & Le Roux, N., On the interplay between noise and curvature and its effect on optimization and generalization. In International Conference on Artificial Intelligence and Statistics, AISTATS 2020
review_title: Interesting but some limitations

Meta-review: The paper proposes a unification of three popular baseline regularizers in continual learning. The unification is realized through a claim that they all regularize (surprisingly) related objectives.

The key strengths of the paper highlighted by the reviewers were:
1. The established connection is valuable and interesting, even if weaker than suggested originally
2. Good motivation (unifying different regularization methods is useful for the community)
3. Clear writing

The key weakness of the paper is a weak empirical validation of the claim that these three regularizers work *because* they regularize the norm of the gradient (as mentioned in the discussion by R3). Rather, the key claims are correlational. The authors correctly say that (1) the three regularizers all regularize related objects (namely different norms of the gradient) and (2) they reduce forgetting. However, it is not sufficiently well demonstrated that (1) => (2). Relatedly, given that the paper does not have a very clear theoretical contribution, it would be really helpful to demonstrate utility. It would be useful to extend experiments that apply these insights to developing novel regularizers or improving/simplifying hyperparameter tuning.

Additionally, in the review process, the link was discovered to be weaker than originally suggested. The paper casts the relation in terms of the Fisher Information Matrix, which suggests it is theoretical and sound. After the discussion, it seems that viewing this relationship in terms of the Fisher Information Matrix is somewhat misleading. The three different regularization methods all regularize different norms of the gradient (L1 or L2), which are empirically, and under some assumption theoretically, related. More precisely, EWC regularizes the trace of the *Empirical* Fisher, which is equivalent to the L2 norm of the gradient of the loss function. SI regularizes a term similar to the L1 norm of the gradient. These effects were seen by the reviewers to be somewhat loosely related to the Fisher Information Matrix.

Based on the above, I have to recommend rejection. I would like to thank the Authors for submitting the work for consideration to ICLR. I hope the feedback will be useful for improving the work.
Decision: Reject