NeuroFabric: Identifying Ideal Topologies for Training A Priori Sparse Networks

1 INTRODUCTION . Training deep neural networks requires both powerful hardware and a significant amount of time . Long training times are a significant bottleneck to deep learning research , as researchers typically iteratively design and test new architectures for a specific problem . While a lot of research has been dedicated to accelerating inference , we investigate training as ( 1 ) accelerating training can speed up research iteration , ( 2 ) evolutionary algorithms for DNN architecture exploration are increasingly being used as an alternative to domain expertise ( Jaderberg et al. , 2017 ) , and network training is moving to edge devices ( Pirk et al. , 2019 ) . Unfortunatelly , the memory requirements of dense , convolutional and recurrent layers grow quadratically with layer information bandwidth 1 . In other words , doubling the size of layer inputs and outputs quadruples the size of the layer . This causes majority of the networks to be memory-bound , making DNN training impractical without batching , a method where training is performed on multiple inputs at a time and updates are aggregated per batch . While batching alleviates the pressure on DRAM bandwidth , it can decrease model accuracy ( Masters & Luschi , 2018 ) especially when scaling training on large clusters ( Akiba et al. , 2017 ) . Furthermore , larger models in off-chip memory become dominant energy cost ( Han et al. , 2015a ) , complicating on-line training on battery-power devices . Conventional dense and convolutional layers do not offer the user to individually tune layer size and the number of layer inputs and outputs . In this work , we seek a method to decouple the information bandwidth from layer expressivity . Such a method would allow us to ( 1 ) speed up training networks by storing them in on-chip memory , ( 2 ) remove the memory bottleneck and the need for batching , ( 3 ) allow more efficient training on distributed systems , and ( 4 ) reduce the energy consumption due to the excessive compute and storage requirements of modern DNNs , potentially allowing us to move training to edge devices . Several works have proposed a priori structured sparsity ( Prabhu et al. , 2017 ; Isakov et al. , 2018 ) or weight sharing ( Ding et al. , 2017 ) to allow training simpler but ‘ wider ’ models . A priori sparsity , where the sparse network topology is selected before training has started , is a promising approach that allows the user to finely and transparently tune the ratio of information bandwidth to memory requirements . If the topology is structured , efficient software 1We define a layer ’ s information bandwidth as the number of independent signals passing through that layer . or hardware implementations can be built to accelerate processing with dense network performance . However , before custom architectures or low-level kernels can be built , a general theory of why certain topologies perform – or underperform – is needed . To the best of our knowledge , no work yet tackles the question of the existence of a ‘ best ’ topology for sparse neural network training . This paper provides an answer on how a topology should be selected . Our contributions are as following : • We propose a sparse cascade architecture that can replace dense or convolutional layers without affecting the rest of the network architecture . • We develop a sparse neural network initialization scheme that allows us to train very deep sparse networks without suffering from the vanishing gradient effect . • We evaluate sevaral topologies on a matrix reconstruction task and show that the choice of topology has a strong effect on attainable network accuracy . • In order to evaluate topologies independently of a dataset , we develop a data-free heuristic for predicting the expressiveness of a given sparse network . • From the heuristic , we derive requirements that make a good topology , and settle on a single family of sparse networks . 2 RELATED WORK . We classify methods that arrive at a sparse network into those that enforce sparsity before , during , or after training . The following is a brief description of each class . Enforcing sparsity after training : In this class of methods , certain weights are zero-ed out after training has finished . This approach has the benefit of first discovering the baseline model accuracy , allowing the training mechanism to evaluate the accuracy to size trade-off . Since training is performed using the dense model , only inference can benefit from these post-training pruning methods . One of the early pruning methods are Optimal Brain Damage ( LeCun et al. , 1990 ) and Optimal Brain Surgeon ( Hassibi & Stork , 1993 ) , where authors remove weights based on the second derivative of the loss w.r.t . to each weight . The insight here is that removing a weight causes some perturbation in the network , and by picking weights with the smallest second derivative of the loss , the effect of the perturbation on the network functionality is be minimized . DeepCompression ( Han et al. , 2015a ; b ) uses a similar approach , but replaces the Hessian-based metric with weight magnitudes . Authors show that high ( > 95 % ) sparsity can be achieved as long as networks are finetuned after pruning to restore performance . Alternatively , in ( Liu et al. , 2015 ) authors decompose convolutional layers into a set of per-channel basis kernels , which are applied to input feature maps , and a sparse kernel matrix that mixes outputs of the basis kernels into the output feature maps . However , all of these methods lead to unstructured sparsity that is difficult to take advantage of . Structured sparsity , where some assumptions can be made on the structure of sparse kernels , has been explored as a way to improve the execution efficiency of sparse structures on GPUs and CPUs . In ( Anwar et al. , 2015 ) , authors use particle filters to prune whole channels or kernels . Similarly , Kadetotad et al . explore structured intra-layer sparsity , where instead of individual weights , small blocks of weights are pruned . Enforcing sparsity during training : Instead of pruning after training , pruning can also be applied during training . This has the benefit of potentially reducing the computational load of the training phase , however , the device performing training must still be able to store the whole dense model at the beginning of training . L1 regularization or L1 weight decay is known to cause sparsity during training , as unlike in the case of L2 regularization , all weights will equally be incentivized to approach zero . However , L1 weight decay often causes a decrease in accuracy , and the sparsity is unstructured . In ( Wen et al. , 2016 ) , authors use Group Lasso ( Yuan & Lin , 2006 ) regularization to enforce the sparsity of more coarse-grained structures instead of individual weights . Enforcing sparsity before training : Model size can also be reduced before training has started . We focus on layer-level methods and not architecture level approaches , as they are orthogonal . Majority of works reducing the size of layers before training have focused either on a priori sparsity or weight reuse . On the weight reuse side , HashedNets ( Chen et al. , 2015 ) use a hash function to group multiple weights and have them share and train a single value . CirCNN ( Ding et al. , 2017 ) uses block-circulant matrices for storing weights , where elements are shared in a predictable manner and Fourier transforms are used for inference , reducing the computational complexity of both inference and training . On the a priori sparsity side , several topologies have been proposed in literature . Deep Expander Networks ( X-Nets ) ( Prabhu et al. , 2017 ) replace dense layers with sparse layers with the expander graph topology . Authors give guarantees of each input neuron being connected to each output neuron within a logarithmic number of layers . Similarly , RadiX-Nets ( Robinett & Kepner , 2019 ) build on X-nets but use the radix topology instead of graph expanders . Alternatively , ClosNets ( Isakov et al. , 2018 ) replace a single dense layer with a cascade of three sparse layers with the Clos topology . Clos topology guarantees full connectivity , and has a tunable parameter for the path diversity between all inputs and outputs . While deep expander networks grow in depth with the number of neurons per layer , ClosNets grow in width . None of the above a priori sparse network works give a definitive answer to which topology maximizes performance per weight . In this work we aim to answer that question . 3 APPROACH . The number of parameters in a neural network layer is decided by the number of input and output neurons ( in case of fully-connected networks ) , or the number of input and output channels ( in the case of convolutional networks ) . This prevents decoupling the network bandwidth ( i.e . number of inputs or outputs of a certain layer ) and the parameter count . We propose that sparsifying layers can allow users to train wider networks without the quadratic growth in network size . Our approach is to replace each fully-connected layer with a cascade of sparsely-connected layers , as shown in Figure 1 . The cascade topology and depth are selected at design time so that the number of parameters in the cascade is lower than in the original dense layer . Hidden layer neurons in the cascade have a linear activation function , while the output neurons use the activation of the original network . The cascade needs to have certain properties such as connectivity between all cascade input-output pairs , a small parameter count , and hardware efficiency . In Section 7 we explore the topology requirements further . Similarly , a priori pruning can be applied to convolutional networks . In conventional CNN layers , each input channel is connected to each output channel by a convolution . For a filter of size f × f , c input and k output channels , the convolutional layer has f2ck parameters . Since the number of input and output channels c and k directly control both the information bandwidth and the size of the network , we propose to disentangle the number of input/output features and the number of convolutional filters . We adopt the architecture of MobileNets ( Howard et al. , 2017 ) and break up convolutional layers into depthwise and pointwise convolutions , as seen in Figure 2a . In the original MobileNets , the majority of parameters belong to the pointwise convolutions , which are simply dense neural networks applied to each ‘ pixel ’ individually . We propose to prune only the pointwise convolutions in the same manner we prune dense layers . 4 INITIALIZING A PRIORI SPARSE NEURAL NETWORKS . Initializing deep neural networks highly affects the training accuracy of the models , but is often not given as much attention as network architecture . If a network is suboptimally initialized , it may train slower than a correctly initialized one or may even not train at all . This problem has been extensively researched in the context of deep neural networks ( Glorot & Bengio , 2010 ; He et al. , 2015a ) and recurrent neural networks ( Henaff et al. , 2016 ) . Our proposed approach replaces fully-connected layers with cascades of sparse layers . As such , the new network may be several times deeper than the original one , and may not train as efficiently . This problem is further compounded by the fact that our networks are a priori sparse . Our tests show that deep sparse networks initialized with common initialization schemes like Xavier initalization ( Glorot & Bengio , 2010 ) completely fail to learn . By observing the activation and error values of deep sparse networks , we deduce that these networks suffer from the vanishing gradient problem , i.e. , with each successive layer , the variance of both the activations and the errors drops exponentially . In this section , we develop a new initialization scheme for a priori sparse networks that alleviates the vanishing gradient problem by taking layer sparsity into account .

Summary: This paper proposes to replace dense layers with multiple sparse linear layers.  The idea is that the product ABC (for A, B, C all sparse matrices)  can accurately approximate a dense matrix D, but A(B(Cx))) requires much less work than Dx.  The paper then continues with the assumption that topology of the sparse matrices should be fixed before training, and that given this assumption we would like to find the "best" fixed topology to pick.  The paper introduces a new task to determine the "best" topology - that of reconstructing a random dense matrix.  On somewhat of a tangent the paper also introduces a minor modification to the Xavier initialization scheme that works better for deep stacks of sparse layers.