text:   synaptic flow",9,"['Q. Can we find winning lottery tickets, or sparse trainable deep networks at initialization without ever looking at data?\n\nA. Yes, by conserving ""Synaptic Flow"" via our new SynFlow algorithm.\n\nco-led with Daniel Kunin\n&amp; @dyamins, @SuryaGanguli\n\npaper: <LINK>\n1/ <LINK>', 'We can potentially reduce the cost of training if we can prune neural networks at initialization.\n\nThe key challenge is ""layer-collapse,"" the premature pruning of an entire layer making a network untrainable.\n2/ https://t.co/hTB5jEdeuD', 'To better understand the phenomena, we first mathematically formulate and experimentally verify a conservation law.\n\nThis conservation law explains why existing gradient-based pruning algorithms at initialization suffer from layer-collapse.\n3/ https://t.co/a8jBLZ5Uoh', 'We then hypothesize that the conservative scoring combined with ""iterative"" re-evaluation can avoid layer collapse. \n\nThis insight also explains how iterative magnitude pruning avoids layer-collapse to identify ""winning-lottery ticket ""subnetworks at initialization.\n4/ https://t.co/kl7sXpTHN2', 'We prove that layer-collapse can be entirely avoided by designing an algorithm with iterative, positive, conservative scoring.\n\nWe design SynFlow satisfying the key requirements and show that it reaches the theoretical limit of max compression without collapsing a network.\n5/ https://t.co/3eLdvdifXB', 'Notably, SynFlow makes no reference to the training data and consistently outperforms existing state-of-the-art\npruning algorithms at initialization on 12 distinct combinations of models and datasets.\n\n6/ https://t.co/q7KjKMK3pd', 'Overall, our data-agnostic pruning algorithm challenges the existing paradigm that data must be used to quantify which synapses are important.\n\nPlease check out the paper for more details\nhttps://t.co/AAHQchcRKC\n7/', '@tingwuc Yes, we are working to incorporate them into our codebase.\n\nIn the meantime, this paper https://t.co/b5KoA2vPnk did very careful work on how pruning at initialization methods (SNIP, GraSP) compare with ""train-prune"" methods, including IMP and others.', ""@xaqlab Thank you for the question.\nSynFlow naturally avoids layer-bottlenecking that starts well before the eventual collapse.\nThis is why we see a significant gain in performance compared to other methods that don't reach max compression.""]",20,06,2183