An Empirical Investigation of Catastrophic Forgeting in Gradient-Based Neural Networks
Authors: Yoshua Bengio, Mehdi Mirza, Ian Goodfellow, Aaron Courville, Xia Da
ICLR 2014

Abstract: Catastrophic forgetting is a problem faced by many machine learning models and algorithms. When trained on one task, then trained on a second task, many machine learning models 'forget'' how to perform the first task. This is widely believed to be a serious problem for neural networks. Here, we investigate the extent to which the catastrophic forgetting problem occurs for modern neural networks, comparing both established and recent gradient-based training algorithms and activation functions. We also examine the effect of the relationship between the first task and the second task on catastrophic forgetting.


Peer review:
[Review]:
Main comment: I would add 'dropout' to the abstract and even the title.  It seems like the main result is that dropout helps ameliorate catastrophic forgetting.

I also don't understand why the production possibility frontier curves don't decrease monotonically.  

My impression is that you create these curves by training the network (initialized via training for the old task) on the new task and then at various points during training, examining test performance on both tasks.  So then why does test error go up on the new task when you are training the network to perform this task? Is this just overfitting? If so, why not do early stopping?  Or am I incorrect about how these plots are generated?


Little things:
The authors are listed in a different order on this site vs. 

on the paper.
Hyperparameters is misspelled a few times.  
3. I would add citation for hard LWTA (at least), I think 

this stands for local winner takes all, but it took me a bit 

to figure that out.  
3. 'and being training on the 'new task'' -> 'and BEGIN 

training on the new task'

3.3 '...two cases that are NOT semantically similar' 

(missing not)

For the graphs, I would color match the dropout vs. SGD 

models with the same conditions.

Also, did you train the dropout nets using something other 

than SGD?  If not, I would clarify this and rename the 'SGD' 

models as 'no dropout' or something like that.