An Empirical Investigation of Catastrophic Forgeting in Gradient-Based Neural Networks
Authors: Yoshua Bengio, Mehdi Mirza, Ian Goodfellow, Aaron Courville, Xia Da
ICLR 2014

Abstract: Catastrophic forgetting is a problem faced by many machine learning models and algorithms. When trained on one task, then trained on a second task, many machine learning models 'forget'' how to perform the first task. This is widely believed to be a serious problem for neural networks. Here, we investigate the extent to which the catastrophic forgetting problem occurs for modern neural networks, comparing both established and recent gradient-based training algorithms and activation functions. We also examine the effect of the relationship between the first task and the second task on catastrophic forgetting.


Peer review:
[Review]:
In response to David's comments:

-The version of the paper you saw was just an incomplete placeholder we uploaded to be able to enter the openreview submission system. We just barely uploaded a new version to ArXiv that is complete and ready to be reviewed. The new version addresses your comments, and should be visible as soon as ArXiv approves it.
-In particular, we added some more explanation of how the possibilities frontiers curves are generated. To be clear, they don't show performance on the new task worsening over time, as you thought. What they show is performance on the old task task worsening as performance on the new task improves. We do in fact use early stopping, so any overfitting is intrinsic to methods themselves. The points that are most overfit are probably bad at both tasks so they lie above and to the right of the frontier and have no effect on the plot.
-It sounds to me like you assumed one of the axes of the plots was a time axis, but I'm not sure if that was your mistake. If you read the axis labels you can see there is no time axis, both are error. You can't tell for sure which points came from what point in time, but presumably the ones on the upper left are from later in time, and the ones on the lower right are from earlier in time.
-We don't have any control over the author order display bug but we've asked the ICLR organizers to fix it.