An Empirical Investigation of Catastrophic Forgeting in Gradient-Based Neural Networks
Authors: Yoshua Bengio, Mehdi Mirza, Ian Goodfellow, Aaron Courville, Xia Da
ICLR 2014

Abstract: Catastrophic forgetting is a problem faced by many machine learning models and algorithms. When trained on one task, then trained on a second task, many machine learning models 'forget'' how to perform the first task. This is widely believed to be a serious problem for neural networks. Here, we investigate the extent to which the catastrophic forgetting problem occurs for modern neural networks, comparing both established and recent gradient-based training algorithms and activation functions. We also examine the effect of the relationship between the first task and the second task on catastrophic forgetting.


Peer review:
[Review]:
The paper discusses the problem of catastrophic forgetting, in the context of deep neural networks. While this is the purported goal of it, an equally interesting outcome of this analysis is the in-depth comparison between a variety of common activation functions used with or without dropout. This analysis reveals that training with dropout is always beneficial in terms of adapting to the new task, but that the various activation functions studied can perform quite differently depending on the task in question. The authors’ analysis does suggests the maxout activation function is consistently a good choice.

I am mystified about the comment on dropout having a “known” range of values that works well (0.5 or 0.2, depending on whether it’s a hidden or visible unit). This may very well be true in the context of obtaining a good performance on a given set of tasks that the authors consider. But since dropout can be seen as having a regularization effect: this effect can manifest itself by less (or more) catastrophic forgetting. I encourage the authors to come up with a single number than can represent well the entirety of the curves shown in Figs. 1,3,5 and then plot out the influence of dropout vs. catastrophic forgetting, one curve per activation function (and per dataset, of course). One good single number may be the area under the curve, perhaps discounting for the left-most and right-most parts? Such an analysis might elucidate the aspects related to the increased number of parameters that dropout seems to favor in the “similar” tasks scenario.

Other than that, it would be nice if the authors proposed hypotheses for why maxout seems to perform better -- it is not obvious from their analysis that this should be the case.

A general comment: it seems rather odd that catastrophic forgetting seems to be somehow tied to the choice of an activation function in this paper -- this (plus dropout) cannot possibly be the only aspect that influences the magnitude of the catastrophic forgetting problem. Arguably, things like the optimization and, generally speaking, the regularization strategies employed, are equally as important. This paper touches very little upon those issues, glossing over anything other than SGD + dropout.

Small comment: there’s a dangling “McClelland, James L” reference.