Differentiable Sparsification for Deep Neural Networks

Deep neural networks have relieved the feature engineering burden on human1 experts . However , comparable efforts are instead required to determine an effective2 architecture . In addition , as the sizes of networks have over-grown , a considerable3 amount of resources is also invested in reducing the sizes . The sparsification of an4 over-complete model addresses these problems as it removes redundant parameters5 or connections . In this study , we propose a fully differentiable sparsification6 method for deep neural networks , which allows parameters to be zero during7 training with the stochastic gradient descent . Thus , the proposed method can8 simultaneously learn the sparsified structure and weights of networks in an end-9 to-end manner , which can be directly applies to modern deep neural networks10 and imposes minimum overhead to the training process . To the authors ’ best11 knowledge , it is the first fully [ sub- ] differentiable sparsification method that zeroes12 out components , and it provides a foundation for future structure learning and13 model compression methods.14 1 Introduction15 The success of deep neural networks has changed the paradigm of machine learning and pattern16 recognition from feature engineering to architecture engineering [ 16 , 14 , 22 , 7 , 32 ] . Although deep17 neural networks have relieved the burden of feature engineering , comparable human efforts are instead18 required to determine an effective architecture , such as the number of neurons or layers and the19 connections between nodes . In addition , as deep neural networks have over-grown ( even up to 10–6820 million parameters ) [ 8 , 10 , 15 , 32 ] , considerable effort is also being invested in reducing existing21 model sizes and in meeting the demands of deploying such networks on constrained platforms at22 inference time [ 26 , 25 ] .23 These problems can be addressed by the sparsification of an over-complete model [ 20 ] . A network24 structure can be carved out of an over-complete model by removing redundant blocks [ 3 , 29 ] or25 deleting unnecessary connections between nodes or blocks [ 2 , 18 , 30 ] , which also reduces the network26 size . Among several approaches , pruning has long been adapted [ 17 , 6 , 26 , 19 , 5 ] . It typically requires27 a pre-trained model and several steps ( select unimportant parameters of a pre-trained model , delete28 the parameters , and retrain the pruned model ) and may repeat the process multiple times . Another29 approach is a sparsity regularizer with the proximal gradient [ 21 , 3 , 29 , 33 ] which shrinks redundant30 parameters to zero during training and requires no pre-trained model . Among the most popular ones31 is l1-regularizer [ 28 ] . However , as it acts on an individual parameter , it often produces unstructured32 irregular models ; thus , it diminishes the benefit of parallel hardware computation , such as GPUs [ 29 ] .33 In order to obtain regular sparse structures , a sparse regularization with l2-norm [ 3 , 29 ] was adopted34 on a group of parameters so that all parameters under the same group are either retained or zeroed-35 out together . By zeroing-out parameters at a group level , the number of neurons or layers can be36 automatically determined as a part of training . However , the optimization of a regularization term is37 performed as a separate step separately from the gradient descent-based optimization for prediction38 Submitted to 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Do not distribute . loss . The update rules should be implemented manually and the approach is limited to the cases39 where closed form solutions for the proximal operation are known.40 In this work , we propose a fully [ sub- ] differentiable sparsification method for deep neural networks,41 which directly optimizes a regularized objective function and allows parameters to be exactly zero42 during training with the stochastic gradient descent . Thus , it can simultaneously learn the sparsified43 structure and weights of deep neural networks in an end-to-end manner . It leads to simpler implemen-44 tation and it does not require to manually code a pruning step or an update rule like a soft-thresholding45 operator . It can adopt various norms as a regularizer regardless of whether their closed form solutions46 for the proximal operator are known or not . Another advantage of the proposed method is that it47 can be easily applied on a group of parameters or a building block ; thus , it can produce a structured48 model and maximize the benefits of parallel hardware computation ( e.g. , GPUs ) and it well suits the49 trend of a modularized design in deep learning [ 22 , 27 , 7 , 32 ] .50 2 Related Work51 2.1 Proximal Gradient52 Our proposed method is related to sparsity regularization with the proximal gradient [ 21 ] . A regular-53 ized objective function is written as54 L ( D , W ) + λR ( W ) , ( 1 ) where L denotes a prediction loss , R is a regularization term , D is a set of training data , W is a set55 of model parameters , and λ controls the trade-off between prediction loss and model complexity . The56 most popular regularizer is l1-norm,57 R ( W ) = ∑ i |wi| , ( 2 ) where wi is an individual element of W . To optimize the regularization term , parameter updating is58 performed with a proximal operator,59 wi ← sign ( wi ) ( |wi| − ηλ ) + , ( 3 ) where← denotes an assignment operator , η is a learning rate , and ( · ) + represents max ( · , 0 ) . As this60 approach acts on an individual parameter , it often produces unstructured irregular models . In order61 to obtain regular sparse structures , the sparse regularization with l2,1-norm can be adopted [ 3 , 29 ] .62 All parameters in the same group are either retained or zeroed-out together . The regularization with63 l2,1-norm is written as64 R ( W ) = ∑ g ∥∥wg∥∥2 =∑ g √∑ i w2g , i , ( 4 ) where W = { wg } and wg represents a group of model parameters . The regularization term is65 optimized with a proximal operator,66 wg , i ← ( ∥∥wg∥∥2 − ηλ∥∥wg∥∥2 ) + wg , i . ( 5 ) When a group has only one single parameter , it degenerates to the l1-norm regularization.67 Another group regularization is exclusive lasso with l1,2-norm [ 34 , 33 ] . Rather than either retaining68 or removing an entire group altogether , it promotes the competition or sparsity within a group . The69 regularization term is written as70 R ( W ) = 1 2 ∑ g ∥∥wg∥∥21 = 12∑ g ∑ i ∣∣wg , i∣∣ 2 , ( 6 ) and its updating rule is derived as71 wg , i ← sign ( wg , i ) ( ∣∣wg , i∣∣− ηλ∥∥wg∥∥1 ) + . ( 7 ) These proximal operators consist of weight decaying and thresholding steps , and they are performed72 at every mini-batch or epoch in a sperate step after the optimization of a prediction loss . It requires73 some extra efforts to implement the update rules manually and applications are limited to the cases74 where closed form solutions for the proximal operator are known . In contrast , our approach optimizes75 a regularized objective function directly and allows parameters to be zero during training with the76 stochastic gradient descent . It leads to simpler implementations and it can employ various norms77 regardless of whether closed form solutions for proximal operators are known or not , for example78 p-norm with p < 1.79 2.2 Differentiable Approach80 Similar to our work , previous differentiable approaches [ 2 , 18 ] learn the structure of a neural81 network by optimizing architecture parameters in a relaxed continuous domain , where architecture82 parameters represent the importance scores of building blocks or the connection strengths between83 them . However , as the architecture parameter magnitudes can not be zero during training , the84 top k connections or components are stochastically or deterministically selected according to the85 architecture parameter values to derive a discretized architecture . Therefore , the approach may suffer86 from the discrepancy between a learned architecture and a final discretized one . Moreover , the87 value of k should be pre-specified manually ; thus , the same value is set for all blocks or modules,88 which may be sub-optimal . Our approach drives the architecture parameters to zero by optimizing a89 regularized objective function . This can minimize the model discrepancy , and a network can choose90 the different numbers of components or connections in each module through training.91 3 Proposed Approach92 3.1 Base Model93 We assume that there are n components in a module . A component can be any building block for a94 deep neural network or its output . For example , it can be a channel or a layer of convolutional neural95 networks such as ResNet [ 7 , 8 ] and DenseNet layer [ 10 ] . It can also represent a node in a neural96 graph [ 30 ] or a convolutional neural network [ 11 ] . A module represents a composite of components,97 such as a group of channels or nodes . For illustration purposes , we assume that a module y can be98 written as the linear combination of components fi:99 y ( x ) = n∑ i=1 aifi ( x ; wi ) , ( 8 ) where x denotes a module input , wi model parameters for component fi , and ai an architecture100 parameter . Model parameters wi denote ordinary parameters , such as a filter in a convolutional layer101 or weight in a fully connected layer . The value of ai represents the importance of component i , and it102 represents the connection strength between nodes in another context . Enforcing ai to be zero amounts103 to removing component fi or zeroing-out wi . Thus , by creating the competition between elements of104 a and driving them to be zero , we can eliminate unnecessary components or connections . The sample105 model is simple , but we will show that it can be applied to various cases.106 3.2 Differentiable Sparse Parameterization107 First , we show how to parameterize architecture parameters with non-negative constraints , which is108 useful for the attention mechanism with the softmax . To set up the competition between the elements109 of a and allow them to be zero , we parameterize the architecture parameters as follows:110 γi = exp ( αi ) ( 9 ) 111 γ̃i = ( γi − σ ( β ) ·‖γ‖1 ) + ( 10 ) 112 ai = γ̃i∑n j=1 γ̃j , ( 11 ) where αi and β are unconstrained free parameters , σ ( · ) denotes a sigmoid function , and ( · ) + rep-113 resents relu ( · ) = max ( · , 0 ) . When a parameter is non-negative , the proximal operator of Eq . ( 7 ) 114 is reduced to Eq . ( 10 ) . Although the forms are similar , they have completely different meanings.115 The proximal operator is a learning rule , whereas Eq . ( 10 ) is the parameterized form of architecture116 parameters , which is part of a neural network.117 We can easily verify that ai is allowed to be zero and is also differentiable from a modern deep118 learning perspective . The free parameters αi and β are real-valued and they do not restrict a training119 process with the stochastic gradient descent . Thus , we can train ai through αi and β . The exponential120 function in Eq . ( 9 ) ensures that the architecture parameters are non-negative . Typically , ai cannot121 be zero due to the exponential function of Eq . ( 9 ) . However , γ̃i in Eq . ( 10 ) can be zero by the122 thresholding operation ; hence , ai can be zero as well . The term σ ( β ) ·‖γ‖1 plays the role of a123 threshold , and the thresholding operation is interpreted as follows : if the strength of component i124 in a competition group is small compared to the total strength , it is dropped from the competition.125 Note that the scalar parameter β in Eq . ( 10 ) , which determines the magnitude of a threshold , is126 not a hyper-parameter , but its value is automatically determined through training . Mathematically,127 the thresholding operator is not differentiable , but this should not pose an issue considering the128 support of relu as a built-in differentiable function in a modern deep learning tool . Additionally , γ is129 non-negative ; thus , its l1-norm is simply the sum of γi ( i.e. , ‖γ‖1 = ∑ γi ) . The softmax of Eq . ( 11 ) 130 is also differentiable . The softmax is optional but useful when we need to promote the competition131 between components as in the attention mechanism.132 By relaxing the differentiability , singed architecture parameter can be similarly formed as133 ai = sign ( αi ) ( |αi| − σ ( β ) ‖α‖1 ) + , ( 12 ) where α and β are free parameters . The gradient of the sign function is zero almost everywhere , but134 it does not cause a problem for a modern deep learning tool . The equation can be rewritten as135 ai = ( αi + σ ( β ) ‖α‖1 ) − if αi < 0 ( αi − σ ( β ) ‖α‖1 ) + otherwise , where ( · ) − = min ( · , 0 ) . The gradient can be computed separately according to whether its value136 is negative or not . To the authors ’ understanding , sign function is already taken care in the ex-137 plained manner by TensorFlow [ 1 ] , and thus tf.math.sign ( ) can be simply used without the manual138 implementation of the conditional statement.139 3.3 Sparsity Regularizer140 In the proposed approach , an objective function is written as141 L ( D , W , a ) + λR ( a ) , ( 13 ) where a denotes the vector of architecture parameters . Sparsifying a is equivalent to sparsifying a142 deep neural network . Therefore , we can use the regularization term on a to encourage the sparsity of143 a . The proposed method is not limited to a particular norm and we can drive different sparsity patterns144 depending on the types of norms . For example , the most popular choice for parameter selection is145 l1-norm , but it is unsuitable on a in Eq . ( 11 ) as it is normalized using the softmax . Its l1-norm is146 always 1 , i.e. , ‖a‖1 = ∑n i=1|ai| = 1 . Therefore , we should employ p-norm with p < 1:147 R ( a ) = n∑ i=1 |ai|p 1p = n∑ i=1 api 1p , ( 14 ) where the second equality holds as ai is always non-negative . To the authors ’ best knowledge , a closed148 form solution for the proximal operator of p-norm with p < 1 is unknown , but the regularization term149 is differentiable almost everywhere as relu is . Thus , the proposed approach can directly optimize the150 regularized objective function and zero-out components with the stochastic gradient descent.151 By simply switching one norm to another one for a regularizer , different sparsity patterns can be152 derived . For example , an individual component can be removed with l1-norm ( Eq . 2 ) and a group of153 components or an entire module can be zeroed-out with l2,1-norm ( Eq . 4 ) . Note that we do not need154 to manually implement different updating rules as in the proximal gradient approach . We just need to155 rewrite a regularization term in the objective function . Examples codes and experiment results are156 shown in the supplementary material.157 3.4 Rectified Gradient Flow158 If γi Eq . ( 10 ) or αi in Eq . ( 12 ) is less than the threshold , the gradient will be zero , and it will159 not receive learning signals . However , note that it does not necessarily mean that a component160 dies permanently once its importance score is less than the threshold . The component still has a161 chance to recover because the threshold is adjustable and the importance scores of other components162 may decrease . Nevertheless , to ensure that the architecture parameters of dropped components163 continuously receive learning signals , we propose approximating the gradient of the thresholding164 function . As in [ 31 ] where the gradient of a step function was approximated using that of leaky relu165 or soft plus , we suggest employing elu [ 4 ] as a variant of the proposed method : relu is used in the166 forward pass , but elu is used in the backward pass.167 This heuristic approach leads to a similar learning mechanism proposed in [ 30 ] , where the gradient168 flows to dropped ( or zeroed out ) edges but does not flow through these dropped edges . The architecture169 parameters in our proposed method correspond to the edges in [ 30 ] . Note that our approach can170 be easily implemented , and it does not require additional codes to control the gradient flow . The171 implementation codes are shown in the appendix.172 4 Application and Experiment173 In this section , we show that the proposed approach can be applied to reduce the size of a network as174 well as to learn the structure . Our aim is not to achieve state-of-the-art performance but to validate the175 idea and the broad applicability of the proposed approach . In order to show the broad applicability176 with limited computing resources , we perform experiments with relative small datasets such as177 CIFAR-10/100 [ 13 ] . Our implementations closely follow those of baseline models , including model178 structures and hyper-parameter settings.179 4.1 Channel Pruning in Convolutional Network180 Network-slimming ( NS ) [ 20 ] prunes unimportant channels in convolutional layers by leveraging the181 scaling factors in batch normalization . Let xi and yi be the input and output of batch normalization182 for channel i and then the operation can be written as183 x̃i = xi − µi√ σ2i + ; yi = aix̃i + bi , where µi and σi denotes the mean and standard deviation of input activations , a and b are scale184 and shift parameters , is a small constant for numerical stability . The scaling parameter a can be185 considered as an importance score or an architecture parameter , and the affine transformation can be186 re-written as187 yi = a ( x̃i + bi ) . By pushing ai to be zero , a corresponding channel can be removed . Network-slimming trains an188 initial network with l1-regularization on a to identify insignificant channels . After the training,189 channels with small values of a are pruned . To compensate the damage caused by pruning , a pruned190 network is fine-tuned . In our approach , we parameterize the scaling parameter using Eq . ( 12 ) and train191 a network with l1-norm on a using the stochastic gradient descent without pruning and fine-tuning.192 We perform comparison experiments on CIFAR-10/100 [ 13 ] . The training and test sets contain193 50 , 000 and 10 , 000 samples respectively , and the final test error is reported at the end of training or194 fine-tuning on all training images . We adopt a standard data augmentation scheme ( random shifting195 and flipping ) as in [ 7 , 8 ] . In network-slimming , λ in Eq . 13 is fixed to 10−5 , but in our approach196 we vary its value to induce different level of sparsity . DenseNet-BC-K12 with 100 layers [ 10 ] and197 ResNet with 164 layers [ 7 , 8 ] are employed as base networks . In network-slimming , pre-trained198 models are obtained by training networks for 160 epochs with the initial learning of 0.1 . The learning199 rate is divided by 10 at 50 % and 75 % of the total number of training epochs . After the training,200 channels with small values of a are pruned and a slimmed network is fine-turned for another 160201 epochs with the same setting as in the initial training , but learned weights are not re-initialized . In our202 approach , we train networks for 320 epochs without fine-tuning or re-training . Network-slimming203 initializes the scaling factor to be 0.5 and we set α = 0.5 ( n+ 1 ) /n and β = log ( n2 + n− 1 ) in204 Eq . 12 so that a starts with 0.5.205 Table 1 shows experiment results on CIFAR-10 with DenseNet . More experiments , including ResNet206 and CIFAR-100 , are given in the supplementary materials due to page limitation . The authors strongly207 urge readers to see the supplementary materials . We ran each experiments 5 times and showed the208 average and the standard deviation . Our proposed method is denoted by Differentiable Sparsification209 ( DS ) . We controlled the value of λ such that similar pruning rate with that of the network-slimming210 approach . In the tables , sparsity denotes the number of removed channels in hidden layers . The211 experiments show that the proposed differentiable approach more effectively learns slimed models.212 4.2 Discovering Neural Wirings213 Discovering Neural Wirings ( DNW ) [ 30 ] relaxes the notion of layers and treats channels as nodes214 in a neural graph . By allowing channels to learn connections between them , it jointly discovers the215 structure and learns the parameters of a neural network . An input to node v , xv , is expressed216 yv = ∑ ( u , v ) ∈E wu , vxu , where xu denotes the state of a proceeding node , E represents a edge set and wu , v is a connection217 weight of an edge . The structure of a neural graph can be determined by choosing a subset of edges.218 At each iteration of training , DNW chooses the top k edges with the highest magnitude , which is219 called a real edge set , and refers to the remaining edges as a hallucinated edge set . On the forwards220 pass or at inference time , real edges are only used . As DNW allows the magnitude of the weights221 in both sets to change throughout training , a hallucinated edge may replace a real edge when it222 strengthens enough . The weights of real edges are updated in an ordinary manner with the stochastic223 gradient descent , but those of hallucinated edges are updated by a specialized leaning rule : the224 gradient flows to hallucinated edges but does not flow through them.225 The architecture parameters in our proposed method correspond to the edges in DNW . We parame-226 terize the edges using Eq . ( 12 ) and train a network with l1-norm on edges to induce sparsity . The227 rectified gradient leads to an update rule which is similar to that of DNW : the rectified gradient228 ensures that dropped architecture parameters continuously receive learning signals by approximating229 the gradient of the thresholding function . However , we do not need to keep track of the real and230 hallucinated edge sets . We simply optimize the objective function with approximated gradients . The231 rectified gradient can be implemented in a couple of lines using modern deep learning tools and the232 code is shown in the supplementary material.233 We perform experiments on CIFAR-10/100 [ 13 ] . The final test error is reported at the end of training234 without using separate validation data set . MobileNetV1 ( ×0.25 ) [ 9 ] is employed as a base model235 and our implementation closely follows that of DNW . We train for 160 epochs with initial learning236 rate 0.1 . The learning rate is scheduled using Cosine Annealing . DNW chose the value of k such that237 a final learned model has similar Mult-Adds with the base model , and we also set the value of λ in238 the same manner.239 Table 2 shows experiment results on CIFAR-10 . We ran each experiments 5 times and showed the240 average and the standard deviation . Our proposed method is denoted by DS . DNW without the update241 rule corresponds to DS without the rectified gradient method . Even without the rectified gradient , the242 performance of the proposed method is close to that of DNW with the update rule . It validates the243 effectiveness of our approach . We also performed experiments with the proximal gradients of Eq . ( 3 ) 244 and Eq . ( 7 ) , which is denoted by PG in Table 2 . PG-l1-norm also uses the l1-norm as a regularizer,245 but the learning is not as effective as ours . Similarly , PG-l1,2-norm uses the update rule of Eq . ( 7 ) 246 whose shape is similar to our sparse parameterization Eq . ( 12 ) , but the performance is worse than247 ours . More experiments , including CIFAR-100 , are given in the supplementary materials due to page248 limitation.249 DNW determines the size and the structure of a network by choosing k edges . However , there is no250 clear notion how to choose k for different stages ( or blocks ) and thus it uses the same pruning rate for251 all stages , which may be restrict because each stage may play a different role and need a different252 amount of resources . In contrast , our approach controls the model complexity by adjusting the value253 of λ in the objective function , and a different amount of resources is allocated for each stage through254 training . As shown in Table 2 , the proposed method uses model parameters more efficiently.255 4.3 Learning relationship between Nodes in Graph256 In this section , we applied the proposed approach to learn the structure of an adjacency matrix in a257 graph convolutional network ( GCN ) . The purpose of this case study is to test whether our approach258 can learn semantic structure from data rather than reducing the size of a neural network.259 We adopted the model of [ 11 ] , one of the most successful GCN models . A GCN block or a layer is260 defined ( see Fig . 1 ) as261 H l+1 = F ( AH lW l ) , where A is an adjacency matrix ; H l and W l are an input feature and a weight matrix for layer l,262 respectively ; and F is a nonlinear activation function . In general , A is non-negative and shared263 across GCN blocks . It is obtained by normalization . For example , A = D̃−1Ã or A = D̃− 1 2 ÃD̃− 1 2 ,264 where Ã is an unnormalized adjacency matrix ; and D̃ is a diagonal matrix , where D̃i = ∑ j Ãi , j .265 The adjacency matrix represents the connections or relationships between nodes on a graph and is266 usually given by prior knowledge . Learning the value of Ai , j amounts to determining the relationship267 between nodes i and j . If the value of Ai , j is zero , it can be considered that the two nodes are268 unrelated.269 As shown in Fig . 1 , we defined each row and column as a group . Row grouping creates the competition270 between in-coming nodes , whereas column grouping creates the competition between out-going271 nodes . Each row and column of unnormalized adjacency matrix Ã can be parameterized similarly as272 in γ̃ of in Eq . ( 10 ) :273 γi , j = exp ( αi , j ) 274 Ãi , j = ( γi , j − σ ( βri ) · ∥∥γi , :∥∥1 − σ ( βcj ) ·∥∥γ : ,j∥∥1 ) + . The softmax normalization of Eq . ( 11 ) is replaced with Sinkhorn normalization [ 23 , 24 , 12 ] to make275 A is doubly-stochastic : each row and column sum up to 1 . Initializing A with Ã , we can convert Ã276 into a doubly stochastic matrix by iteratively applying the following equations:277 A = D−1r A and A = AD −1 c , where Dr and Dc are diagonal matrices ; [ Dr ] i = ∑ j Ai , j ; [ Dc ] j = ∑ iAi , j . Note that although the278 normalization is iterative , it is differentiable . Balanced normalization is also possible by iteratively279 applying280 A = D − 12 r AD − 12 c . We verified through numerical experiments that iteratively applying the above equation also makes Ã281 to doubly stochastic , but we could not find a theoretical justification . We leave the mathematical proof282 as an open question for a future work . As competition groups are created in row- and column-wise283 approaches , a regularized objective function can be written as284 L ( D , W , A ) + λ 2 N∑ i=1 { R ( Ai , : ) +R ( A : ,i ) } , where W = { W l } , N is the size of square matrix A , and Ai , : and A : ,i denote ith row and column285 vector of A , respectively . We employ lp-norm of Eq . ( 14 ) with p = 0.5 for a regularizer.286 To validate our purposed method , we applied a GCN to estimate future traffic speeds in a road network.287 The traffic speed data were collected from 170 road segments.Thus , the sizes of an adjacent matrix is288 170× 170 . The map of the data area collected is shown in 2 . One-step ahead observation is estimated289 from eight past observations and an output layer generates 170 estimates , one for each road segment.290 A prediction loss is measured using the mean relative error ( MRE ) . More detailed specifications of291 the experimental data and our GCN model can be found in the supplementary material . A prediction292 loss is measured using the mean relative error ( MRE ) .293 Three baseline models were used : two were given the road connectivity and the other was not . For294 the first baseline , we set the value of Ai , j as a constant such that Ai , j = 1ni if node ( or road ) i and j295 are adjacent to each other ( ni is the number of neighbors of node i ) , and Ai , j = 0 otherwise . The296 second baseline was taken in a similar approach , but we set Ãi , j = exp ( αi , j ) if node i and j were297 adjacent to each other to ensure that the strengths of the connections were learned . For the third298 baseline , the connectivity was not given . However , we set Ãi , j = exp ( αi , j ) for all i , j regardless of299 the actual connections . For the proposed method , we parameterized the adjacency matrix as in the300 third baseline but applied the sparsification technique . The balanced normalization was applied to all301 cases except the first baseline , for which the row sum is 1.302 To measure the learned relationship between nodes , we propose the following scoring function:303 1 2N N∑ i=1 N∑ j=1 [ ( Ar +Ac ) Mk ] i , j , where Ar = D−1r A , A c = AD−1c , denotes the element-wise product , and [ Mk ] i , j = 1 if the304 geodesic distance between node i and j is less than or equal to k , whereas [ Mk ] i , j = 0 otherwise.305 The maximum value is 1 , and the minimum is 0 . For example , the first and second baselines always306 have the maximum value because their adjacency matrices have exactly the same structure of M1.307 We calculated the scores for k = 1 and 2 . Note that we used Ar and Ac instead of A because a308 sparsified matrix is not guaranteed to be doubly stochastic even if the original Sinkhorn normalization309 is adopted.310 The performance of the baseline models is shown in Table 4a . We ran each experiment five times and311 selected the median among the five lowest validation errors . For the first and second baselines , the312 road connectivity is given , and the number of non-zero elements of the adjacent matrices is 878 . Note313 that the value of the learned relationship for Baselines I and II is constant , but we show it for reference.314 The road connectivity is not given for the third baseline , and the number of non-zero elements of315 the adjacent matrix is 28 , 900 ( = 170× 170 ) . The performance of the proposed model is shown in316 Table 4b . The experiment of Baseline III shows that a GCN finds a nonlinear mapping between input317 and target values simply in the way of reducing the prediction loss without learning the semantic318 relationships between nodes , but the proposed approach finds actual relationships between nodes . We319 further compared the proposed method with the proximal gradient method . The experimental results320 are reported in the supplementary material due to page limitation.321 5 Scope and Limitation322 Our aim is not to achieve state-of-the-art performance but to validate the idea and the broad applicabil-323 ity of the proposed approach . To the authors ’ best knowledge , it is the first fully [ sub- ] differentiable324 sparsification method that zeroes out components , and we wish our work would provide a foundation325 for future structure learning and model compression methods . The limitation of our approach is that326 a sparsity rate can not be explicitly specified before training as in conventional pruning approaches . If327 a specific sparsity rate is required , it should be obtained by a try-and-error . The rectified gradient is328 effective as shown in the experiments of discovering neural wirings , but it is not clear in which cases329 it is effective or not . We need more theoretical analysis and leave it for a future work.330 6 Conclusion331 In this study , we proposed a fully differentiable sparsification method that can simultaneously learn332 the sparsified structure and weights of deep neural networks . Our proposed method is versatile in that333 it can be seamlessly integrated into different types of neural networks and various problems.334 Checklist335 1 . For all authors ... 336 ( a ) Do the main claims made in the abstract and introduction accurately reflect the paper ’ s337 contributions and scope ? [ Yes ] 338 ( b ) Did you describe the limitations of your work ? [ Yes ] See the scope and limitation339 section.340 ( c ) Did you discuss any potential negative societal impacts of your work ? [ N/A ] 341 ( d ) Have you read the ethics review guidelines and ensured that your paper conforms to342 them ? [ N/A ] 343 2 . If you are including theoretical results ... 344 ( a ) Did you state the full set of assumptions of all theoretical results ? [ N/A ] 345 ( b ) Did you include complete proofs of all theoretical results ? [ N/A ] 346 3 . If you ran experiments ... 347 ( a ) Did you include the code , data , and instructions needed to reproduce the main experi-348 mental results ( either in the supplemental material or as a URL ) ? [ Yes ] Main Codes are349 included in the supplemental material and a snippet is also given in the appendix ( pdf350 supplemental material ) 351 ( b ) Did you specify all the training details ( e.g. , data splits , hyperparameters , how they352 were chosen ) ? [ Yes ] 353 ( c ) Did you report error bars ( e.g. , with respect to the random seed after running experi-354 ments multiple times ) ? [ Yes ] We ran each experiment five times and gave the mean355 and the standard deviation.356 ( d ) Did you include the total amount of compute and the type of resources used ( e.g.,357 type of GPUs , internal cluster , or cloud provider ) ? [ No ] However , readers may easily358 estimate required resources because we used well-known data and models in most359 experiments.360 4 . If you are using existing assets ( e.g. , code , data , models ) or curating/releasing new assets ... 361 ( a ) If your work uses existing assets , did you cite the creators ? [ N/A ] 362 ( b ) Did you mention the license of the assets ? [ N/A ] 363 ( c ) Did you include any new assets either in the supplemental material or as a URL ? [ N/A ] 364 365 ( d ) Did you discuss whether and how consent was obtained from people whose data you ’ re366 using/curating ? [ N/A ] 367 ( e ) Did you discuss whether the data you are using/curating contains personally identifiable368 information or offensive content ? [ N/A ] 369 5 . If you used crowdsourcing or conducted research with human subjects ... 370 ( a ) Did you include the full text of instructions given to participants and screenshots , if371 applicable ? [ N/A ] 372 ( b ) Did you describe any potential participant risks , with links to Institutional Review373 Board ( IRB ) approvals , if applicable ? [ N/A ] 374 ( c ) Did you include the estimated hourly wage paid to participants and the total amount375 spent on participant compensation ? [ N/A ] 376 References377 [ 1 ] Martín Abadi , Ashish Agarwal , Paul Barham , Eugene Brevdo , Zhifeng Chen , Craig Citro,378 Greg S. Corrado , Andy Davis , Jeffrey Dean , Matthieu Devin , Sanjay Ghemawat , Ian Goodfellow,379 Andrew Harp , Geoffrey Irving , Michael Isard , Yangqing Jia , Rafal Jozefowicz , Lukasz Kaiser,380 Manjunath Kudlur , Josh Levenberg , Dandelion Mané , Rajat Monga , Sherry Moore , Derek381 Murray , Chris Olah , Mike Schuster , Jonathon Shlens , Benoit Steiner , Ilya Sutskever , Kunal382 Talwar , Paul Tucker , Vincent Vanhoucke , Vijay Vasudevan , Fernanda Viégas , Oriol Vinyals , Pete383 Warden , Martin Wattenberg , Martin Wicke , Yuan Yu , and Xiaoqiang Zheng . TensorFlow : Large-384 scale machine learning on heterogeneous systems , 2015 . URL https : //www.tensorflow.385 org/ . Software available from tensorflow.org.386 [ 2 ] Karim Ahmed and Lorenzo Torresani . Connectivity learning in multi-branch networks . CoRR,387 abs/1709.09582 , 2017 . URL http : //arxiv.org/abs/1709.09582.388 [ 3 ] Jose M. Alvarez and Mathieu Salzmann . Learning the number of neurons in deep networks . In389 NIPS , 2016.390 [ 4 ] Djork-Arné Clevert , Thomas Unterthiner , and Sepp Hochreiter . Fast and accurate deep network391 learning by exponential linear units ( elus ) . In ICLR , 2016.392 [ 5 ] Jonathan Frankle and Michael Carbin . The lottery ticket hypothesis : Finding sparse , trainable393 neural networks . In ICLR , 2019.394 [ 6 ] Babak Hassibi , David G. Stork , and Gregory J . Wolff . Optimal brain surgeon and general395 network pruning . In ICNN , 1993.396 [ 7 ] Kaiming He , Xiangyu Zhang , Shaoqing Ren , and Jian Sun . Deep residual learning for image397 recognition . In CVPR , 2016.398 [ 8 ] Kaiming He , Xiangyu Zhang , Shaoqing Ren , and Jian Sun . Identity mappings in deep residual399 networks . In ECCV , 2016.400 [ 9 ] Andrew G. Howard , Menglong Zhu , Bo Chen , Dmitry Kalenichenko , Weijun Wang , Tobias401 Weyand , Marco Andreetto , and Hartwig Adam . Mobilenets : Efficient convolutional neural402 networks for mobile vision applications . CoRR , abs/1704.04861 , 2017 . URL https : //arxiv.403 org/abs/1704.04861.404 [ 10 ] Gao Huang , Zhuang Liu , and Laurens van der Maaten . Densely connected convolutional405 networks . In CVPR , 2017.406 [ 11 ] Thomas N. Kipf and Max Welling . Semi-supervised classification with graph convolutional407 networks . In ICLR , 2017.408 [ 12 ] P. A. Knight . The sinkhorn-knopp algorithm : convergence and applications . SIAM Journal on409 Matrix Analysis and Applications , 30 ( 1 ) , 2008.410 [ 13 ] Alex Krizhevsky . Learning multiple layers of features from tiny images . Technical Report,411 2009.412 [ 14 ] Alex Krizhevsky , Ilya Sutskever , and Geoffrey E. Hinton . Imagenet classification with deep413 convolutional neural networks . In NIPS , 2012.414 [ 15 ] Gustav Larsson , Michael Maire , and Gregory Shakhnarovich . Fractalnet : Ultra-deep neural415 networks without residuals . In ICLR , 2017.416 [ 16 ] Y. LeCun , B. Boser , J. S. Denker , D. Henderson , R. E. Howard , W. Hubbard , and L. D. Jackel.417 Backpropagation applied to handwritten zip code recognition . Neural Computation , 1 ( 4 ) :418 541–551 , Dec 1989 . ISSN 0899-7667. doi : 10.1162/neco.1989.1.4.541.419 [ 17 ] Yann LeCun , John S. Denker , and Sara A. Solla . Optimal brain damage . In NIPS , 1989.420 [ 18 ] Hanxiao Liu , Karen Simonyan , and Yiming Yang . Darts : Differentiable architecture search . In421 ICLR , 2019.422 [ 19 ] Zhuang Liu , Jianguo Li , Zhiqiang Shen , Gao Huang , Shoumeng Yan , and Changshui Zhang.423 Learning efficient convolutional networks through network slimming . In ICCV , 2017.424 [ 20 ] Zhuang Liu , Mingjie Sun , Tinghui Zhou , Gao Huang , and Trevor Darrell . Rethinking the value425 of network pruning . In ICLR , 2019.426 [ 21 ] Neal Parikh and Stephen Boyd . Proximal algorithms . Found . Trends Optim. , 1 ( 3 ) :127–239,427 January 2014 . ISSN 2167-3888. doi : 10.1561/2400000003 . URL http : //dx.doi.org/10.428 1561/2400000003.429 [ 22 ] Karen Simonyan and Andrew Zisserman . Very deep convolutional networks for large-scale430 image recognition . In ICLR , 2015.431 [ 23 ] R. Sinkhorn . A relationship between arbitrary positive matrices and doubly stochastic matrices.432 The annals of mathematicalstatistics , 35 ( 2 ) :876–879 , 1964.433 [ 24 ] R. Sinkhorn and P. Knopp . Concerning nonnegative matrices and doubly stochastic matrices.434 Pacific Journal of Mathematics , 21 ( 2 ) , 1967.435 [ 25 ] Huizi Mao Song Han and William J. Dally . Deep compression : Compressing deep neural436 network with pruning , trained quantization and huffman coding . In ICLR , 2016.437 [ 26 ] John Tran Song Han , Jeff Pool and William Dally . Learning both weights and connections for438 efficient neural network . In NIPS , 2015.439 [ 27 ] Christian Szegedy , Wei Liu , Yangqing Jia , Pierre Sermanet , Scott Reed , Dragomir Anguelov,440 Dumitru Erhan , Vincent Vanhoucke , and Andrew Rabinovich . Going deeper with convolutions.441 In CVPR , 2016.442 [ 28 ] Robert Tibshirani . Regression shrinkage and selection via the lasso . Journal of the Royal443 Statistical Society . Series B ( Methodological ) , 58 ( 1 ) :267–288 , 1996.444 [ 29 ] Wei Wen , Chunpeng Wu , Yandan Wang , Yiran Chen , and Hai Li . Learning structured sparsity445 in deep neural networks . In NIPS , 2016.446 [ 30 ] Mitchell Wortsman , Ali Farhadi , and Mohammad Rastegari . Discovering neural wirings . In447 NeurIPS , 2019.448 [ 31 ] Xia Xiao , Zigeng Wang , and Sanguthevar Rajasekaran . Autoprune : Automatic network pruning449 by regularizing auxiliary parameters . In NeurIPS , 2019.450 [ 32 ] Saining Xie , Ross Girshick , Piotr Dollar , Zhuowen Tu , and Kaiming He . Aggregated residual451 transformations for deep neural networks . In CVPR , 2017.452 [ 33 ] Jaehong Yoon and Sung Ju Hwang . Combined group and exclusive sparsity for deep neural453 networks . In ICML , 2017.454 [ 34 ] Yang Zhou , Rong Jin , and Steven Chu–Hong Hoi . Exclusive lasso for multi-task feature455 selection . In AISTATS , 2010.456

Summary: This paper presents a network sparsification method. It represents a network as a linear combination of different components, and the weights for different components are parameterized by some unconstrained parameters. By enforcing the Lp norm on the weights of different components, all parameters can be learnt in an end-to-end manner. Then the proposed method is applied to three applications: channel pruning in CNN, Discovering Neural Wirings, and Learning relationship between nodes in Graph. Experiments validate the effectiveness of the proposed solution.