Venue: ICLR 2017 workshop

Reviews:

final_score: rating: 7: Good paper, accept
review_content: The authors introduce a penalty term that encourages neuron stability.
The idea is clear and the experiments illustrate the point.

Though not groundbreaking, I do like the paper and would recommend it for publication.

Given the following will be done:
- Please discuss the exact computational cost of the method in absolute numbers.
- An analysis of the fraction of weights receiving almost no updates on the given tasks. It would be interesting to see, if any surprising observations can be made here.
review_title: Nice idea, suitable for workshop track

final_score: rating: 6: Marginally above acceptance threshold
review_content: This submission presents a technique which allows to train neural networks subsequently on a series of task without loosing too much performance on the tasks which have already been trained. The methods estimates the importance of synapses in training by keeping track of the gradient of the loss with respect to the synapse: large gradients indicate that the synapse is important for the task and therefore this synapse will be prevented from changing too much when training for subsequent tasks.

I think the method provides a very nice approach to multi-task training and is worth presenting at ICLR. I am however missing an analysis on more state-of-the-art datasets like ImageNet or at least TinyImageNet since a lot of interesting methods fail to work (or get too expensive) on large datasets.

On first reading the paper I was surprised that the gradient would serve as a good estimate for the importance of the synapse since it should be approximately zero at the end of the optimization (for comparison, fisher information is related to the hessian matrix). But since the authors are integrating the gradient over the course of optimization, a large value should mean that the gradient used to be large in the optimization and is small at the end, which one might take as a second order statement.
review_title: Nice work, what about larger datasets?

Decision: Accept