Winning the Lottery with Continuous Sparsification
The Lottery Ticket Hypothesis from Frankle & Carbin conjectures that, for typically-sized neural networks, it is possible to find small sub-networks which train faster and yield superior performance than their original counterparts.The proposed algorithm to search for such sub-networks, Iterative Magnitude Pruning, consistently finds sub-networks with 90-95% less parameters which indeed train faster and better than the overparameterized models they were extracted from, creating potential applications to problems such as transfer learning.In this paper, we propose a new algorithm to search for winning tickets, Continuous Sparsification, which continuously removes parameters from a network during training, and learns the sub-networks structure with gradient-based methods instead of relying on pruning strategies.We show empirically that our method is capable of finding tickets that outperforms the ones learned by Iterative Magnitude Pruning, and at the same time providing up to 5 times faster search, when measured in number of training epochs.
We propose a new algorithm that quickly finds winning tickets in neural networks.This paper proposes a novel objective function that can be used to jointly optimize a classification objective while encouraging sparsification in a network that performs with high accuracy.This work propose a new iterative pruning methods named Continuous Sparsification, which continuously prunes the current weight until it reaches the target ratio.