Dynamic Sparse Graph for Efficient Deep Learning

1 INTRODUCTION . Deep Neural Networks ( DNNs ) ( LeCun et al. , 2015 ) have been achieving impressive progress in a wide spectrum of domains ( Simonyan & Zisserman , 2014 ; He et al. , 2016 ; Abdel-Hamid et al. , 2014 ; Redmon & Farhadi , 2016 ; Wu et al. , 2016 ) , while the models are extremely memory- and compute-intensive . The high representational and computational costs motivate many researchers to investigate approaches on improving the execution performance , including matrix or tensor decomposition ( Xue et al. , 2014 ; Novikov et al. , 2015 ; Garipov et al. , 2016 ; Yang et al. , 2017 ; Alvarez & Salzmann , 2017 ) , data quantization ( Courbariaux et al. , 2016 ; Zhou et al. , 2016 ; Deng et al. , 2018 ; Leng et al. , 2017 ; Wen et al. , 2017 ; Wu et al. , 2018 ; McKinstry et al. , 2018 ) , and network pruning ( Ardakani et al. , 2016 ; Han et al. , 2015b ; a ; Liu et al. , 2017 ; Li et al. , 2016 ; He et al. , 2017 ; Luo et al. , 2017 ; Wen et al. , 2016 ; Molchanov et al. , 2016 ; Sun et al. , 2017 ; Spring & Shrivastava , 2017 ; Lin et al. , 2017a ; Zhang et al. , 2018 ; He et al. , 2018a ; Chin et al. , 2018 ; Ye et al. , 2018 ; Luo & Wu , 2018 ; Hu et al. , 2018 ; He et al. , 2018b ) . However , most of the previous work aim at inference while the challenges for reducing the representational and computational costs of training are not well-studied . Although some works demonstrate acceleration in the distributed training ( Lin et al. , 2017b ; Goyal et al. , 2017 ; You et al. , 2017 ) , we target at the single-node optimization , and our method can also boost training in a distributed fashion . DNN training , which demands much more hardware resources in terms of both memory capacity and computation volume , is far more challenging than inference . Firstly , activation data in training will be stored for backpropagation , significantly increasing the memory consumption . Secondly , training iteratively updates model parameters using mini-batched stochastic gradient descent . We almost always expect larger mini-batches for higher throughput ( Figure 1 ( a ) ) , faster convergence , and better accuracy ( Smith et al. , 2017 ) . However , memory capacity is often the limitation factor ( Figure 1 ( b ) ) that may cause performance degradation or even make large models with deep structures or targeting high-resolution vision tasks hard to train ( He et al. , 2016 ; Wu & He , 2018 ) . It is difficult to apply existing sparsity techniques towards inference phase to training phase because of the following reasons : 1 ) Prior arts mainly compress the pre-trained and fixed weight parameters to reduce the off-chip memory access in inference ( Han et al. , 2016 ; 2017 ) , while instead , the dynamic neuronal activations turn out to be the crucial bottleneck in training ( Jain et al. , 2018 ) , making the prior inference-oriented methods inefficient . Besides , during training we need to stash a vast batched activation space for the backward gradient calculation . Therefore , neuron activations creates a new memory bottleneck ( Figure 1 ( c ) ) . In this paper , we will sparsify the neuron activations for training compression . 2 ) The existing inference accelerations usually add extra optimization problems onto the critical path ( Wen et al. , 2016 ; Molchanov et al. , 2016 ; Liu et al. , 2017 ; Luo et al. , 2017 ; Liang et al. , 2018 ; Zhang et al. , 2018 ; Hu et al. , 2018 ; Luo & Wu , 2018 ; Ye et al. , 2018 ) , i.e. , ‘ complicated training⇒ simplified inference ’ , which embarrassingly complicates the training phase . 3 ) Moreover , previous studies reveal that batch normalization ( BN ) is crucial for improving accuracy and robustness ( Figure 1 ( d ) ) through activation fusion across different samples within one mini-batch for better representation ( Morcos et al. , 2018 ; Ioffe & Szegedy , 2015 ) . BN almost becomes a standard training configuration ; however , inference-oriented methods seldom discuss BN and treat BN parameters as scaling and shift factors in the forward pass . We further find that BN will damage the sparsity due to the activation reorganization ( Figure 1 ( e ) ) . Since this work targets both training and inference , the BN compatibility problem should be addressed . From the view of information representation , the activation of each neuron reflects its selectivity to the current stimulus sample ( Morcos et al. , 2018 ) , and this selectivity dataflow propagates layer by layer forming different representation levels . Fortunately , there is much representational redundancy , for example , lots of neuron activations for each stimulus sample are so small and can be removed ( Figure 1 ( f ) ) . Motivated by above comprehensive analysis regarding memory and compute , we propose to search critical neurons for constructing a sparse graph at every iteration . By activating only a small amount of neurons with a high selectivity , we can significantly save memory and simplify computation with tolerable accuracy degradation . Because the neuron response dynamically changes under different stimulus samples , the sparse graph is variable . The neuronaware dynamic and sparse graph ( DSG ) is fundamentally distinct from the static one in previous work on permanent weight pruning since we never prune the graph but activate part of them each time . Therefore , we maintain the model expressive power as much as possible . A graph selection method , dimension-reduction search , is designed for both compressible activations with elementwise unstructured sparsity and accelerative vector-matrix multiplication ( VMM ) with vector-wise structured sparsity . Through double-mask selection design , it is also compatible with BN . We can use the same selection pattern and extend our method to inference . In a nutshell , we propose a compressible and accelerative DSG approach supported by dimension-reduction search and doublemask selection . It can achieve 1.7-4.5x memory compression and 2.3-4.4x computation reduction with minimal accuracy loss . This work simultaneously pioneers the approach towards efficient online training and offline inference , which can benefit the deep learning in both the cloud and the edge . 2 APPROACH . Our method forms DSGs for different inputs , which are accelerative and compressive , as shown in Figure2 ( a ) . On the one hand , choosing a small number of critical neurons to participate in computation , DSG can reduce the computational cost by eliminating calculations of non-critical neurons . On the other hand , it can further reduce the representational cost via compression on sparsified activations . Different from previous methods using permanent pruning , our approach does not prune any neuron and the associated weights ; instead , it activates a sparse graph according to the input sample at each iteration . Therefore , DSG does not compromise the expressive power of the model . Constructing DSG needs to determine which neurons are critical . A naive approach is to select critical neurons according to the output activations . If the output neurons have a small or negative activation value , i.e. , not selective to current input sample , they can be removed for saving representational cost . Because these activations will be small or absolute zero after the following ReLU non-linear function ( i.e. , ReLU ( x ) = max ( 0 , x ) ) , it ’ s reasonable to set all of them to be zero . However , this naive approach requires computations of all VMM operations within each layer before the selection of critical neurons , which is very costly . 2.1 DIMENSION-REDUCTION SEARCH . To avoid the costly VMM operations in the mentioned naive selection , we propose an efficient method , i.e. , dimension reduction search , to estimate the importance of output neurons . As shown in Figure2 ( b ) , we first reduce the dimensions of X and W , and then execute the lightweight VMM operations in a low-dimensional space with minimal cost . After that , we estimate the neuron importance according to the virtual output activations . Then , a binary selection mask can be produced in which the zeros represent the non-critical neurons with small activations that are removable . We use a top-k search method that only keeps largest k neurons , where an inter-sample threshold sharing mechanism is leveraged to greatly reduce the search cost 1 . Note that k is determined by the output size and a pre-configured sparsity parameter γ . Then we can just compute the accurate activations of the critical neurons in the original high-dimensional space and avoid the calculation of the noncritical neurons . Thus , besides the compressive sparse activations , the dimension-reduction search can further save a significant amount of expensive operations in the high-dimensional space . 1Implementation details are shown in Appendix B . In this way , a vector-wise structured sparsity can be achieved , as shown in Figure 3 ( b ) . The ones in the selection mask ( marked as colored blocks ) denote the critical neurons , and the non-critical ones can bypass the memory access and computation of their corresponding columns in the weight matrix . Furthermore , the generated sparse activations can be compressed via the zero-value compression ( Zhang et al. , 2000 ; Vijaykumar et al. , 2015 ; Rhu et al. , 2018 ) ( Figure 3 ( c ) ) . Consequently , it is critical to reduce the vector dimension but keep the activations calculated in the low-dimensional space as accurate as possible , compared to the ones in the original high-dimensional space . 2.2 SPARSE RANDOM PROJECTION FOR EFFICIENT DIMENSION-REDUCTION SEARCH . Notations : Each CONV layer has a four dimensional weight tensor ( nK , nC , nR , nS ) , where nK is the number of filters , i.e. , the number of output feature maps ( FMs ) ; nC is the number of input FMs ; ( nR , nS ) represents the kernel size . Thus , the CONV layer in Figure 3 ( a ) can be converted to many VMM operations , as shown in Figure 3 ( b ) . Each row in the matrix of input FMs is the activations from a sliding window across all input FMs ( nCRS = nC×nR×nS ) , and after the VMM operation with the weight matrix ( nCRS×nK ) it can generate nK points at the same location across all output FMs . Further considering the nPQ = nP × nQ size of each output FM and the mini-batch size of m , the whole nPQ×m rows of VMM operations has a computational complexity of O ( m×nPQ× nCRS × nK ) . For the FC layer with nC input neurons and nK output neurons , this complexity is O ( m × nC × nK ) . Note that here we switch the order of BN and ReLU layer from ‘ CONV/FCBN-ReLU ’ to ‘ CONV/FC-ReLU-BN ’ , because it ’ s hard to determine the activation value of the non-critical neurons if the following layer is BN ( this value is zero for ReLU ) . As shown in previous work , this reorganization could bring better accuracy ( Mishkin & Matas , 2015 ) . For the sake of simplicity , we just consider the operation for each sliding window in the CONV layer or the whole FC layer under one single input sample as a basic optimization problem . The generation of each output activation yj requires an inner product operation , as follows : yj = ϕ ( 〈Xi , Wj〉 ) ( 1 ) where Xi is the i-th row in the matrix of input FMs ( for the FC layer , there is only one X vector ) , Wj is the j-th column of the weight matrix W , and ϕ ( · ) is the neuronal transformation ( e.g. , ReLU function , here we abandon bias ) . Now , according to equation ( 1 ) , the preservation of the activation is equivalent to preserve the inner product . We introduce a dimension-reduction lemma , named Johnson-Lindenstrauss Lemma ( JLL ) ( Johnson & Lindenstrauss , 1984 ) , to implement the dimension-reduction search with inner product preservation . This lemma states that a set of points in a high-dimensional space can be embedded into a low-dimensional space in such a way that the Euclidean distances between these points are nearly preserved . Specifically , given 0 < < 1 , a set of N points in Rd ( i.e. , all Xi and Wj ) , and a number of k > O ( log ( N ) 2 ) , there exists a linear map f : R d ⇒ Rk such that ( 1− ) ‖Xi −Wj‖2 ≤ ‖f ( Xi ) − f ( Wj ) ‖2 ≤ ( 1 + ) ‖Xi −Wj‖2 ( 2 ) for any given Xi and Wj pair , where is a hyper-parameter to control the approximation error , i.e. , larger ⇒ larger error . When is sufficiently small , one corollary from JLL is the following norm preservation ( Vu , 2016 ; Kakade & Shakhnarovich , 2009 ) : P [ ( 1− ) ‖Z‖2 ≤ ‖f ( Z ) ‖2 ≤ ( 1 + ) ‖Z‖2 ] ≥ 1−O ( 2 ) ( 3 ) where Z could be any Xi or Wj , and P denotes a probability . It means the vector norm can be preserved with a high probability controlled by . Given these basics , we can further get the inner product preservation : P [ |〈f ( Xi ) , f ( Wj ) 〉 − 〈Xi , Wj〉| ≤ ] ≥ 1−O ( 2 ) . ( 4 ) The detailed proof can be found in Appendix A . Random projection ( Vu , 2016 ; Ailon & Chazelle , 2009 ; Achlioptas , 2001 ) is widely used to construct the linear map f ( · ) . Specifically , the original d-dimensional vector is projected to a kdimensional ( k d ) one , using a random k × d matrix R. Then we can reduce the dimension of all Xi and Wj by f ( Xi ) = 1√ k RXi ∈ Rk , f ( Wj ) = 1√ k RWj ∈ Rk . ( 5 ) The random projection matrix R can be generated from Gaussian distribution ( Ailon & Chazelle , 2009 ) . In this paper , we adopt a simplified version , termed as sparse random projection ( Achlioptas , 2001 ; Bingham & Mannila , 2001 ; Li et al. , 2006 ) with P ( Rpq = √ s ) = 1 2s ; P ( Rpq = 0 ) = 1− 1 s ; P ( Rpq = − √ s ) = 1 2s ( 6 ) for all elements in R. This R only has ternary values that can remove the multiplications during projection , and the remained additions are very sparse . Therefore , the projection overhead is negligible compared to other high-precision operations involving multiplication . Here we set s = 3 with 67 % sparsity in statistics . Equation ( 4 ) indicates the low-dimensional inner product 〈f ( Xi ) , f ( Wj ) 〉 can still approximate the original high-dimensional one 〈Xi , Wj〉 in equation ( 1 ) if the reduced dimension is sufficiently high . Therefore , it is possible to calculate equation ( 1 ) in a low-dimensional space for activation estimation , and select the important neurons . As shown in Figure 3 ( b ) , each sliding window dynamically selects its own important neurons for the calculation in high-dimensional space , marked in red and blue as two examples . Figure 4 visualizes two sliding windows in a real network to help understand the dynamic process of dimension-reduction search . Here the neuronal activation vector ( nK length ) is reshaped to a matrix for clarity . Now For the CONV layer , the computational complexity is only O [ m × nPQ × nK × ( k + ( 1 − γ ) × nCRS ) ] , which is less than the original high-dimensional computation with O ( m × nPQ × nCRS × nK ) complexity because we usually have [ k + ( 1 − γ ) × nCRS ] nCRS . For the FC layer , we also have O [ m× nK × ( k + ( 1− γ ) × nC ) ] O ( m× nC × nK ) .

Summary: In this paper, the authors proposed to use dynamic sparse computation graph for reducing the computation memory and time cost in deep neural network (DNN). This method is applicable in both DNN training and inference. Unlike most of previous work that focusing on the reduction of computation during the inference time, this new method propose a dynamic computation graph by pruning the activations on the fly during the training of inference, which is an interesting and novel exploration. In the experiments, the authors performed extensive experiments to demonstrate the effectiveness of the proposed method compared with several baseline methods and original models. It is clear to me that this method helps to reduce the memory cost and computation cost for both DNN training and inference.