text: 359,139,1271709221206151168,1037195648636989442,Hidenori Tanaka,"Q. Can we find winning lottery tickets, or sparse trainable deep networks at initialization without ever looking at data? A. Yes, by conserving ""Synaptic Flow"" via our new SynFlow algorithm. co-led with Daniel Kunin & @dyamins, @SuryaGanguli paper: <LINK> 1/ <LINK> We can potentially reduce the cost of training if we can prune neural networks at initialization. The key challenge is ""layer-collapse,"" the premature pruning of an entire layer making a network untrainable. 2/ <LINK> To better understand the phenomena, we first mathematically formulate and experimentally verify a conservation law. This conservation law explains why existing gradient-based pruning algorithms at initialization suffer from layer-collapse. 3/ <LINK> We then hypothesize that the conservative scoring combined with ""iterative"" re-evaluation can avoid layer collapse.  This insight also explains how iterative magnitude pruning avoids layer-collapse to identify ""winning-lottery ticket ""subnetworks at initialization. 4/ <LINK> We prove that layer-collapse can be entirely avoided by designing an algorithm with iterative, positive, conservative scoring. We design SynFlow satisfying the key requirements and show that it reaches the theoretical limit of max compression without collapsing a network. 5/ <LINK> Notably, SynFlow makes no reference to the training data and consistently outperforms existing state-of-the-art pruning algorithms at initialization on 12 distinct combinations of models and datasets. 6/ <LINK> Overall, our data-agnostic pruning algorithm challenges the existing paradigm that data must be used to quantify which synapses are important. Please check out the paper for more details <LINK> 7/ @tingwuc Yes, we are working to incorporate them into our codebase. In the meantime, this paper <LINK> did very careful work on how pruning at initialization methods (SNIP, GraSP) compare with ""train-prune"" methods, including IMP and others. @xaqlab Thank you for the question. SynFlow naturally avoids layer-bottlenecking that starts well before the eventual collapse. This is why we see a significant gain in performance compared to other methods that don't reach max compression.",http://arxiv.org/abs/2006.05467,"Pruning the parameters of deep neural networks has generated intense interest due to potential savings in time, memory and energy both during training and at test time. Recent works have identified, through an expensive sequence of training and pruning cycles, the existence of winning lottery tickets or sparse trainable subnetworks at initialization. This raises a foundational question: can we identify highly sparse trainable subnetworks at initialization, without ever training, or indeed without ever looking at the data? We provide an affirmative answer to this question through theory driven algorithm design. We first mathematically formulate and experimentally verify a conservation law that explains why existing gradient-based pruning algorithms at initialization suffer from layer-collapse, the premature pruning of an entire layer rendering a network untrainable. This theory also elucidates how layer-collapse can be entirely avoided, motivating a novel pruning algorithm Iterative Synaptic Flow Pruning (SynFlow). This algorithm can be interpreted as preserving the total flow of synaptic strengths through the network at initialization subject to a sparsity constraint. Notably, this algorithm makes no reference to the training data and consistently competes with or outperforms existing state-of-the-art pruning algorithms at initialization over a range of models (VGG and ResNet), datasets (CIFAR-10/100 and Tiny ImageNet), and sparsity constraints (up to 99.99 percent). Thus our data-agnostic pruning algorithm challenges the existing paradigm that, at initialization, data must be used to quantify which synapses are important. ","Pruning neural networks without any data by iteratively conserving