Differentiable Sparsification for Deep Neural Networks

Deep neural networks have relieved the feature engineering burden on human1 experts . However , comparable efforts are instead required to determine an effective2 architecture . In addition , as the sizes of networks have over-grown , a considerable3 amount of resources is also invested in reducing the sizes . The sparsification of an4 over-complete model addresses these problems as it removes redundant parameters5 or connections . In this study , we propose a fully differentiable sparsification6 method for deep neural networks , which allows parameters to be zero during7 training with the stochastic gradient descent . Thus , the proposed method can8 simultaneously learn the sparsified structure and weights of networks in an end-9 to-end manner , which can be directly applies to modern deep neural networks10 and imposes minimum overhead to the training process . To the authors ’ best11 knowledge , it is the first fully [ sub- ] differentiable sparsification method that zeroes12 out components , and it provides a foundation for future structure learning and13 model compression methods.14 1 Introduction15 The success of deep neural networks has changed the paradigm of machine learning and pattern16 recognition from feature engineering to architecture engineering [ 16 , 14 , 22 , 7 , 32 ] . Although deep17 neural networks have relieved the burden of feature engineering , comparable human efforts are instead18 required to determine an effective architecture , such as the number of neurons or layers and the19 connections between nodes . In addition , as deep neural networks have over-grown ( even up to 10–6820 million parameters ) [ 8 , 10 , 15 , 32 ] , considerable effort is also being invested in reducing existing21 model sizes and in meeting the demands of deploying such networks on constrained platforms at22 inference time [ 26 , 25 ] .23 These problems can be addressed by the sparsification of an over-complete model [ 20 ] . A network24 structure can be carved out of an over-complete model by removing redundant blocks [ 3 , 29 ] or25 deleting unnecessary connections between nodes or blocks [ 2 , 18 , 30 ] , which also reduces the network26 size . Among several approaches , pruning has long been adapted [ 17 , 6 , 26 , 19 , 5 ] . It typically requires27 a pre-trained model and several steps ( select unimportant parameters of a pre-trained model , delete28 the parameters , and retrain the pruned model ) and may repeat the process multiple times . Another29 approach is a sparsity regularizer with the proximal gradient [ 21 , 3 , 29 , 33 ] which shrinks redundant30 parameters to zero during training and requires no pre-trained model . Among the most popular ones31 is l1-regularizer [ 28 ] . However , as it acts on an individual parameter , it often produces unstructured32 irregular models ; thus , it diminishes the benefit of parallel hardware computation , such as GPUs [ 29 ] .33 In order to obtain regular sparse structures , a sparse regularization with l2-norm [ 3 , 29 ] was adopted34 on a group of parameters so that all parameters under the same group are either retained or zeroed-35 out together . By zeroing-out parameters at a group level , the number of neurons or layers can be36 automatically determined as a part of training . However , the optimization of a regularization term is37 performed as a separate step separately from the gradient descent-based optimization for prediction38 Submitted to 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Do not distribute . loss . The update rules should be implemented manually and the approach is limited to the cases39 where closed form solutions for the proximal operation are known.40 In this work , we propose a fully [ sub- ] differentiable sparsification method for deep neural networks,41 which directly optimizes a regularized objective function and allows parameters to be exactly zero42 during training with the stochastic gradient descent . Thus , it can simultaneously learn the sparsified43 structure and weights of deep neural networks in an end-to-end manner . It leads to simpler implemen-44 tation and it does not require to manually code a pruning step or an update rule like a soft-thresholding45 operator . It can adopt various norms as a regularizer regardless of whether their closed form solutions46 for the proximal operator are known or not . Another advantage of the proposed method is that it47 can be easily applied on a group of parameters or a building block ; thus , it can produce a structured48 model and maximize the benefits of parallel hardware computation ( e.g. , GPUs ) and it well suits the49 trend of a modularized design in deep learning [ 22 , 27 , 7 , 32 ] .50 2 Related Work51 2.1 Proximal Gradient52 Our proposed method is related to sparsity regularization with the proximal gradient [ 21 ] . A regular-53 ized objective function is written as54 L ( D , W ) + λR ( W ) , ( 1 ) where L denotes a prediction loss , R is a regularization term , D is a set of training data , W is a set55 of model parameters , and λ controls the trade-off between prediction loss and model complexity . The56 most popular regularizer is l1-norm,57 R ( W ) = ∑ i |wi| , ( 2 ) where wi is an individual element of W . To optimize the regularization term , parameter updating is58 performed with a proximal operator,59 wi ← sign ( wi ) ( |wi| − ηλ ) + , ( 3 ) where← denotes an assignment operator , η is a learning rate , and ( · ) + represents max ( · , 0 ) . As this60 approach acts on an individual parameter , it often produces unstructured irregular models . In order61 to obtain regular sparse structures , the sparse regularization with l2,1-norm can be adopted [ 3 , 29 ] .62 All parameters in the same group are either retained or zeroed-out together . The regularization with63 l2,1-norm is written as64 R ( W ) = ∑ g ∥∥wg∥∥2 =∑ g √∑ i w2g , i , ( 4 ) where W = { wg } and wg represents a group of model parameters . The regularization term is65 optimized with a proximal operator,66 wg , i ← ( ∥∥wg∥∥2 − ηλ∥∥wg∥∥2 ) + wg , i . ( 5 ) When a group has only one single parameter , it degenerates to the l1-norm regularization.67 Another group regularization is exclusive lasso with l1,2-norm [ 34 , 33 ] . Rather than either retaining68 or removing an entire group altogether , it promotes the competition or sparsity within a group . The69 regularization term is written as70 R ( W ) = 1 2 ∑ g ∥∥wg∥∥21 = 12∑ g ∑ i ∣∣wg , i∣∣ 2 , ( 6 ) and its updating rule is derived as71 wg , i ← sign ( wg , i ) ( ∣∣wg , i∣∣− ηλ∥∥wg∥∥1 ) + . ( 7 ) These proximal operators consist of weight decaying and thresholding steps , and they are performed72 at every mini-batch or epoch in a sperate step after the optimization of a prediction loss . It requires73 some extra efforts to implement the update rules manually and applications are limited to the cases74 where closed form solutions for the proximal operator are known . In contrast , our approach optimizes75 a regularized objective function directly and allows parameters to be zero during training with the76 stochastic gradient descent . It leads to simpler implementations and it can employ various norms77 regardless of whether closed form solutions for proximal operators are known or not , for example78 p-norm with p < 1.79 2.2 Differentiable Approach80 Similar to our work , previous differentiable approaches [ 2 , 18 ] learn the structure of a neural81 network by optimizing architecture parameters in a relaxed continuous domain , where architecture82 parameters represent the importance scores of building blocks or the connection strengths between83 them . However , as the architecture parameter magnitudes can not be zero during training , the84 top k connections or components are stochastically or deterministically selected according to the85 architecture parameter values to derive a discretized architecture . Therefore , the approach may suffer86 from the discrepancy between a learned architecture and a final discretized one . Moreover , the87 value of k should be pre-specified manually ; thus , the same value is set for all blocks or modules,88 which may be sub-optimal . Our approach drives the architecture parameters to zero by optimizing a89 regularized objective function . This can minimize the model discrepancy , and a network can choose90 the different numbers of components or connections in each module through training.91 3 Proposed Approach92 3.1 Base Model93 We assume that there are n components in a module . A component can be any building block for a94 deep neural network or its output . For example , it can be a channel or a layer of convolutional neural95 networks such as ResNet [ 7 , 8 ] and DenseNet layer [ 10 ] . It can also represent a node in a neural96 graph [ 30 ] or a convolutional neural network [ 11 ] . A module represents a composite of components,97 such as a group of channels or nodes . For illustration purposes , we assume that a module y can be98 written as the linear combination of components fi:99 y ( x ) = n∑ i=1 aifi ( x ; wi ) , ( 8 ) where x denotes a module input , wi model parameters for component fi , and ai an architecture100 parameter . Model parameters wi denote ordinary parameters , such as a filter in a convolutional layer101 or weight in a fully connected layer . The value of ai represents the importance of component i , and it102 represents the connection strength between nodes in another context . Enforcing ai to be zero amounts103 to removing component fi or zeroing-out wi . Thus , by creating the competition between elements of104 a and driving them to be zero , we can eliminate unnecessary components or connections . The sample105 model is simple , but we will show that it can be applied to various cases.106 3.2 Differentiable Sparse Parameterization107 First , we show how to parameterize architecture parameters with non-negative constraints , which is108 useful for the attention mechanism with the softmax . To set up the competition between the elements109 of a and allow them to be zero , we parameterize the architecture parameters as follows:110 γi = exp ( αi ) ( 9 ) 111 γ̃i = ( γi − σ ( β ) ·‖γ‖1 ) + ( 10 ) 112 ai = γ̃i∑n j=1 γ̃j , ( 11 ) where αi and β are unconstrained free parameters , σ ( · ) denotes a sigmoid function , and ( · ) + rep-113 resents relu ( · ) = max ( · , 0 ) . When a parameter is non-negative , the proximal operator of Eq . ( 7 ) 114 is reduced to Eq . ( 10 ) . Although the forms are similar , they have completely different meanings.115 The proximal operator is a learning rule , whereas Eq . ( 10 ) is the parameterized form of architecture116 parameters , which is part of a neural network.117 We can easily verify that ai is allowed to be zero and is also differentiable from a modern deep118 learning perspective . The free parameters αi and β are real-valued and they do not restrict a training119 process with the stochastic gradient descent . Thus , we can train ai through αi and β . The exponential120 function in Eq . ( 9 ) ensures that the architecture parameters are non-negative . Typically , ai cannot121 be zero due to the exponential function of Eq . ( 9 ) . However , γ̃i in Eq . ( 10 ) can be zero by the122 thresholding operation ; hence , ai can be zero as well . The term σ ( β ) ·‖γ‖1 plays the role of a123 threshold , and the thresholding operation is interpreted as follows : if the strength of component i124 in a competition group is small compared to the total strength , it is dropped from the competition.125 Note that the scalar parameter β in Eq . ( 10 ) , which determines the magnitude of a threshold , is126 not a hyper-parameter , but its value is automatically determined through training . Mathematically,127 the thresholding operator is not differentiable , but this should not pose an issue considering the128 support of relu as a built-in differentiable function in a modern deep learning tool . Additionally , γ is129 non-negative ; thus , its l1-norm is simply the sum of γi ( i.e. , ‖γ‖1 = ∑ γi ) . The softmax of Eq . ( 11 ) 130 is also differentiable . The softmax is optional but useful when we need to promote the competition131 between components as in the attention mechanism.132 By relaxing the differentiability , singed architecture parameter can be similarly formed as133 ai = sign ( αi ) ( |αi| − σ ( β ) ‖α‖1 ) + , ( 12 ) where α and β are free parameters . The gradient of the sign function is zero almost everywhere , but134 it does not cause a problem for a modern deep learning tool . The equation can be rewritten as135 ai = ( αi + σ ( β ) ‖α‖1 ) − if αi < 0 ( αi − σ ( β ) ‖α‖1 ) + otherwise , where ( · ) − = min ( · , 0 ) . The gradient can be computed separately according to whether its value136 is negative or not . To the authors ’ understanding , sign function is already taken care in the ex-137 plained manner by TensorFlow [ 1 ] , and thus tf.math.sign ( ) can be simply used without the manual138 implementation of the conditional statement.139 3.3 Sparsity Regularizer140 In the proposed approach , an objective function is written as141 L ( D , W , a ) + λR ( a ) , ( 13 ) where a denotes the vector of architecture parameters . Sparsifying a is equivalent to sparsifying a142 deep neural network . Therefore , we can use the regularization term on a to encourage the sparsity of143 a . The proposed method is not limited to a particular norm and we can drive different sparsity patterns144 depending on the types of norms . For example , the most popular choice for parameter selection is145 l1-norm , but it is unsuitable on a in Eq . ( 11 ) as it is normalized using the softmax . Its l1-norm is146 always 1 , i.e. , ‖a‖1 = ∑n i=1|ai| = 1 . Therefore , we should employ p-norm with p < 1:147 R ( a ) = n∑ i=1 |ai|p 1p = n∑ i=1 api 1p , ( 14 ) where the second equality holds as ai is always non-negative . To the authors ’ best knowledge , a closed148 form solution for the proximal operator of p-norm with p < 1 is unknown , but the regularization term149 is differentiable almost everywhere as relu is . Thus , the proposed approach can directly optimize the150 regularized objective function and zero-out components with the stochastic gradient descent.151 By simply switching one norm to another one for a regularizer , different sparsity patterns can be152 derived . For example , an individual component can be removed with l1-norm ( Eq . 2 ) and a group of153 components or an entire module can be zeroed-out with l2,1-norm ( Eq . 4 ) . Note that we do not need154 to manually implement different updating rules as in the proximal gradient approach . We just need to155 rewrite a regularization term in the objective function . Examples codes and experiment results are156 shown in the supplementary material.157 3.4 Rectified Gradient Flow158 If γi Eq . ( 10 ) or αi in Eq . ( 12 ) is less than the threshold , the gradient will be zero , and it will159 not receive learning signals . However , note that it does not necessarily mean that a component160 dies permanently once its importance score is less than the threshold . The component still has a161 chance to recover because the threshold is adjustable and the importance scores of other components162 may decrease . Nevertheless , to ensure that the architecture parameters of dropped components163 continuously receive learning signals , we propose approximating the gradient of the thresholding164 function . As in [ 31 ] where the gradient of a step function was approximated using that of leaky relu165 or soft plus , we suggest employing elu [ 4 ] as a variant of the proposed method : relu is used in the166 forward pass , but elu is used in the backward pass.167 This heuristic approach leads to a similar learning mechanism proposed in [ 30 ] , where the gradient168 flows to dropped ( or zeroed out ) edges but does not flow through these dropped edges . The architecture169 parameters in our proposed method correspond to the edges in [ 30 ] . Note that our approach can170 be easily implemented , and it does not require additional codes to control the gradient flow . The171 implementation codes are shown in the appendix.172 4 Application and Experiment173 In this section , we show that the proposed approach can be applied to reduce the size of a network as174 well as to learn the structure . Our aim is not to achieve state-of-the-art performance but to validate the175 idea and the broad applicability of the proposed approach . In order to show the broad applicability176 with limited computing resources , we perform experiments with relative small datasets such as177 CIFAR-10/100 [ 13 ] . Our implementations closely follow those of baseline models , including model178 structures and hyper-parameter settings.179 4.1 Channel Pruning in Convolutional Network180 Network-slimming ( NS ) [ 20 ] prunes unimportant channels in convolutional layers by leveraging the181 scaling factors in batch normalization . Let xi and yi be the input and output of batch normalization182 for channel i and then the operation can be written as183 x̃i = xi − µi√ σ2i + ; yi = aix̃i + bi , where µi and σi denotes the mean and standard deviation of input activations , a and b are scale184 and shift parameters , is a small constant for numerical stability . The scaling parameter a can be185 considered as an importance score or an architecture parameter , and the affine transformation can be186 re-written as187 yi = a ( x̃i + bi ) . By pushing ai to be zero , a corresponding channel can be removed . Network-slimming trains an188 initial network with l1-regularization on a to identify insignificant channels . After the training,189 channels with small values of a are pruned . To compensate the damage caused by pruning , a pruned190 network is fine-tuned . In our approach , we parameterize the scaling parameter using Eq . ( 12 ) and train191 a network with l1-norm on a using the stochastic gradient descent without pruning and fine-tuning.192 We perform comparison experiments on CIFAR-10/100 [ 13 ] . The training and test sets contain193 50 , 000 and 10 , 000 samples respectively , and the final test error is reported at the end of training or194 fine-tuning on all training images . We adopt a standard data augmentation scheme ( random shifting195 and flipping ) as in [ 7 , 8 ] . In network-slimming , λ in Eq . 13 is fixed to 10−5 , but in our approach196 we vary its value to induce different level of sparsity . DenseNet-BC-K12 with 100 layers [ 10 ] and197 ResNet with 164 layers [ 7 , 8 ] are employed as base networks . In network-slimming , pre-trained198 models are obtained by training networks for 160 epochs with the initial learning of 0.1 . The learning199 rate is divided by 10 at 50 % and 75 % of the total number of training epochs . After the training,200 channels with small values of a are pruned and a slimmed network is fine-turned for another 160201 epochs with the same setting as in the initial training , but learned weights are not re-initialized . In our202 approach , we train networks for 320 epochs without fine-tuning or re-training . Network-slimming203 initializes the scaling factor to be 0.5 and we set α = 0.5 ( n+ 1 ) /n and β = log ( n2 + n− 1 ) in204 Eq . 12 so that a starts with 0.5.205 Table 1 shows experiment results on CIFAR-10 with DenseNet . More experiments , including ResNet206 and CIFAR-100 , are given in the supplementary materials due to page limitation . The authors strongly207 urge readers to see the supplementary materials . We ran each experiments 5 times and showed the208 average and the standard deviation . Our proposed method is denoted by Differentiable Sparsification209 ( DS ) . We controlled the value of λ such that similar pruning rate with that of the network-slimming210 approach . In the tables , sparsity denotes the number of removed channels in hidden layers . The211 experiments show that the proposed differentiable approach more effectively learns slimed models.212 4.2 Discovering Neural Wirings213 Discovering Neural Wirings ( DNW ) [ 30 ] relaxes the notion of layers and treats channels as nodes214 in a neural graph . By allowing channels to learn connections between them , it jointly discovers the215 structure and learns the parameters of a neural network . An input to node v , xv , is expressed216 yv = ∑ ( u , v ) ∈E wu , vxu , where xu denotes the state of a proceeding node , E represents a edge set and wu , v is a connection217 weight of an edge . The structure of a neural graph can be determined by choosing a subset of edges.218 At each iteration of training , DNW chooses the top k edges with the highest magnitude , which is219 called a real edge set , and refers to the remaining edges as a hallucinated edge set . On the forwards220 pass or at inference time , real edges are only used . As DNW allows the magnitude of the weights221 in both sets to change throughout training , a hallucinated edge may replace a real edge when it222 strengthens enough . The weights of real edges are updated in an ordinary manner with the stochastic223 gradient descent , but those of hallucinated edges are updated by a specialized leaning rule : the224 gradient flows to hallucinated edges but does not flow through them.225 The architecture parameters in our proposed method correspond to the edges in DNW . We parame-226 terize the edges using Eq . ( 12 ) and train a network with l1-norm on edges to induce sparsity . The227 rectified gradient leads to an update rule which is similar to that of DNW : the rectified gradient228 ensures that dropped architecture parameters continuously receive learning signals by approximating229 the gradient of the thresholding function . However , we do not need to keep track of the real and230 hallucinated edge sets . We simply optimize the objective function with approximated gradients . The231 rectified gradient can be implemented in a couple of lines using modern deep learning tools and the232 code is shown in the supplementary material.233 We perform experiments on CIFAR-10/100 [ 13 ] . The final test error is reported at the end of training234 without using separate validation data set . MobileNetV1 ( ×0.25 ) [ 9 ] is employed as a base model235 and our implementation closely follows that of DNW . We train for 160 epochs with initial learning236 rate 0.1 . The learning rate is scheduled using Cosine Annealing . DNW chose the value of k such that237 a final learned model has similar Mult-Adds with the base model , and we also set the value of λ in238 the same manner.239 Table 2 shows experiment results on CIFAR-10 . We ran each experiments 5 times and showed the240 average and the standard deviation . Our proposed method is denoted by DS . DNW without the update241 rule corresponds to DS without the rectified gradient method . Even without the rectified gradient , the242 performance of the proposed method is close to that of DNW with the update rule . It validates the243 effectiveness of our approach . We also performed experiments with the proximal gradients of Eq . ( 3 ) 244 and Eq . ( 7 ) , which is denoted by PG in Table 2 . PG-l1-norm also uses the l1-norm as a regularizer,245 but the learning is not as effective as ours . Similarly , PG-l1,2-norm uses the update rule of Eq . ( 7 ) 246 whose shape is similar to our sparse parameterization Eq . ( 12 ) , but the performance is worse than247 ours . More experiments , including CIFAR-100 , are given in the supplementary materials due to page248 limitation.249 DNW determines the size and the structure of a network by choosing k edges . However , there is no250 clear notion how to choose k for different stages ( or blocks ) and thus it uses the same pruning rate for251 all stages , which may be restrict because each stage may play a different role and need a different252 amount of resources . In contrast , our approach controls the model complexity by adjusting the value253 of λ in the objective function , and a different amount of resources is allocated for each stage through254 training . As shown in Table 2 , the proposed method uses model parameters more efficiently.255 4.3 Learning relationship between Nodes in Graph256 In this section , we applied the proposed approach to learn the structure of an adjacency matrix in a257 graph convolutional network ( GCN ) . The purpose of this case study is to test whether our approach258 can learn semantic structure from data rather than reducing the size of a neural network.259 We adopted the model of [ 11 ] , one of the most successful GCN models . A GCN block or a layer is260 defined ( see Fig . 1 ) as261 H l+1 = F ( AH lW l ) , where A is an adjacency matrix ; H l and W l are an input feature and a weight matrix for layer l,262 respectively ; and F is a nonlinear activation function . In general , A is non-negative and shared263 across GCN blocks . It is obtained by normalization . For example , A = D̃−1Ã or A = D̃− 1 2 ÃD̃− 1 2 ,264 where Ã is an unnormalized adjacency matrix ; and D̃ is a diagonal matrix , where D̃i = ∑ j Ãi , j .265 The adjacency matrix represents the connections or relationships between nodes on a graph and is266 usually given by prior knowledge . Learning the value of Ai , j amounts to determining the relationship267 between nodes i and j . If the value of Ai , j is zero , it can be considered that the two nodes are268 unrelated.269 As shown in Fig . 1 , we defined each row and column as a group . Row grouping creates the competition270 between in-coming nodes , whereas column grouping creates the competition between out-going271 nodes . Each row and column of unnormalized adjacency matrix Ã can be parameterized similarly as272 in γ̃ of in Eq . ( 10 ) :273 γi , j = exp ( αi , j ) 274 Ãi , j = ( γi , j − σ ( βri ) · ∥∥γi , :∥∥1 − σ ( βcj ) ·∥∥γ : ,j∥∥1 ) + . The softmax normalization of Eq . ( 11 ) is replaced with Sinkhorn normalization [ 23 , 24 , 12 ] to make275 A is doubly-stochastic : each row and column sum up to 1 . Initializing A with Ã , we can convert Ã276 into a doubly stochastic matrix by iteratively applying the following equations:277 A = D−1r A and A = AD −1 c , where Dr and Dc are diagonal matrices ; [ Dr ] i = ∑ j Ai , j ; [ Dc ] j = ∑ iAi , j . Note that although the278 normalization is iterative , it is differentiable . Balanced normalization is also possible by iteratively279 applying280 A = D − 12 r AD − 12 c . We verified through numerical experiments that iteratively applying the above equation also makes Ã281 to doubly stochastic , but we could not find a theoretical justification . We leave the mathematical proof282 as an open question for a future work . As competition groups are created in row- and column-wise283 approaches , a regularized objective function can be written as284 L ( D , W , A ) + λ 2 N∑ i=1 { R ( Ai , : ) +R ( A : ,i ) } , where W = { W l } , N is the size of square matrix A , and Ai , : and A : ,i denote ith row and column285 vector of A , respectively . We employ lp-norm of Eq . ( 14 ) with p = 0.5 for a regularizer.286 To validate our purposed method , we applied a GCN to estimate future traffic speeds in a road network.287 The traffic speed data were collected from 170 road segments.Thus , the sizes of an adjacent matrix is288 170× 170 . The map of the data area collected is shown in 2 . One-step ahead observation is estimated289 from eight past observations and an output layer generates 170 estimates , one for each road segment.290 A prediction loss is measured using the mean relative error ( MRE ) . More detailed specifications of291 the experimental data and our GCN model can be found in the supplementary material . A prediction292 loss is measured using the mean relative error ( MRE ) .293 Three baseline models were used : two were given the road connectivity and the other was not . For294 the first baseline , we set the value of Ai , j as a constant such that Ai , j = 1ni if node ( or road ) i and j295 are adjacent to each other ( ni is the number of neighbors of node i ) , and Ai , j = 0 otherwise . The296 second baseline was taken in a similar approach , but we set Ãi , j = exp ( αi , j ) if node i and j were297 adjacent to each other to ensure that the strengths of the connections were learned . For the third298 baseline , the connectivity was not given . However , we set Ãi , j = exp ( αi , j ) for all i , j regardless of299 the actual connections . For the proposed method , we parameterized the adjacency matrix as in the300 third baseline but applied the sparsification technique . The balanced normalization was applied to all301 cases except the first baseline , for which the row sum is 1.302 To measure the learned relationship between nodes , we propose the following scoring function:303 1 2N N∑ i=1 N∑ j=1 [ ( Ar +Ac ) Mk ] i , j , where Ar = D−1r A , A c = AD−1c , denotes the element-wise product , and [ Mk ] i , j = 1 if the304 geodesic distance between node i and j is less than or equal to k , whereas [ Mk ] i , j = 0 otherwise.305 The maximum value is 1 , and the minimum is 0 . For example , the first and second baselines always306 have the maximum value because their adjacency matrices have exactly the same structure of M1.307 We calculated the scores for k = 1 and 2 . Note that we used Ar and Ac instead of A because a308 sparsified matrix is not guaranteed to be doubly stochastic even if the original Sinkhorn normalization309 is adopted.310 The performance of the baseline models is shown in Table 4a . We ran each experiment five times and311 selected the median among the five lowest validation errors . For the first and second baselines , the312 road connectivity is given , and the number of non-zero elements of the adjacent matrices is 878 . Note313 that the value of the learned relationship for Baselines I and II is constant , but we show it for reference.314 The road connectivity is not given for the third baseline , and the number of non-zero elements of315 the adjacent matrix is 28 , 900 ( = 170× 170 ) . The performance of the proposed model is shown in316 Table 4b . The experiment of Baseline III shows that a GCN finds a nonlinear mapping between input317 and target values simply in the way of reducing the prediction loss without learning the semantic318 relationships between nodes , but the proposed approach finds actual relationships between nodes . We319 further compared the proposed method with the proximal gradient method . The experimental results320 are reported in the supplementary material due to page limitation.321 5 Scope and Limitation322 Our aim is not to achieve state-of-the-art performance but to validate the idea and the broad applicabil-323 ity of the proposed approach . To the authors ’ best knowledge , it is the first fully [ sub- ] differentiable324 sparsification method that zeroes out components , and we wish our work would provide a foundation325 for future structure learning and model compression methods . The limitation of our approach is that326 a sparsity rate can not be explicitly specified before training as in conventional pruning approaches . If327 a specific sparsity rate is required , it should be obtained by a try-and-error . The rectified gradient is328 effective as shown in the experiments of discovering neural wirings , but it is not clear in which cases329 it is effective or not . We need more theoretical analysis and leave it for a future work.330 6 Conclusion331 In this study , we proposed a fully differentiable sparsification method that can simultaneously learn332 the sparsified structure and weights of deep neural networks . Our proposed method is versatile in that333 it can be seamlessly integrated into different types of neural networks and various problems.334 Checklist335 1 . For all authors ... 336 ( a ) Do the main claims made in the abstract and introduction accurately reflect the paper ’ s337 contributions and scope ? [ Yes ] 338 ( b ) Did you describe the limitations of your work ? [ Yes ] See the scope and limitation339 section.340 ( c ) Did you discuss any potential negative societal impacts of your work ? [ N/A ] 341 ( d ) Have you read the ethics review guidelines and ensured that your paper conforms to342 them ? [ N/A ] 343 2 . If you are including theoretical results ... 344 ( a ) Did you state the full set of assumptions of all theoretical results ? [ N/A ] 345 ( b ) Did you include complete proofs of all theoretical results ? [ N/A ] 346 3 . If you ran experiments ... 347 ( a ) Did you include the code , data , and instructions needed to reproduce the main experi-348 mental results ( either in the supplemental material or as a URL ) ? [ Yes ] Main Codes are349 included in the supplemental material and a snippet is also given in the appendix ( pdf350 supplemental material ) 351 ( b ) Did you specify all the training details ( e.g. , data splits , hyperparameters , how they352 were chosen ) ? [ Yes ] 353 ( c ) Did you report error bars ( e.g. , with respect to the random seed after running experi-354 ments multiple times ) ? [ Yes ] We ran each experiment five times and gave the mean355 and the standard deviation.356 ( d ) Did you include the total amount of compute and the type of resources used ( e.g.,357 type of GPUs , internal cluster , or cloud provider ) ? [ No ] However , readers may easily358 estimate required resources because we used well-known data and models in most359 experiments.360 4 . If you are using existing assets ( e.g. , code , data , models ) or curating/releasing new assets ... 361 ( a ) If your work uses existing assets , did you cite the creators ? [ N/A ] 362 ( b ) Did you mention the license of the assets ? [ N/A ] 363 ( c ) Did you include any new assets either in the supplemental material or as a URL ? [ N/A ] 364 365 ( d ) Did you discuss whether and how consent was obtained from people whose data you ’ re366 using/curating ? [ N/A ] 367 ( e ) Did you discuss whether the data you are using/curating contains personally identifiable368 information or offensive content ? [ N/A ] 369 5 . If you used crowdsourcing or conducted research with human subjects ... 370 ( a ) Did you include the full text of instructions given to participants and screenshots , if371 applicable ? [ N/A ] 372 ( b ) Did you describe any potential participant risks , with links to Institutional Review373 Board ( IRB ) approvals , if applicable ? [ N/A ] 374 ( c ) Did you include the estimated hourly wage paid to participants and the total amount375 spent on participant compensation ? [ N/A ] 376 References377 [ 1 ] Martín Abadi , Ashish Agarwal , Paul Barham , Eugene Brevdo , Zhifeng Chen , Craig Citro,378 Greg S. Corrado , Andy Davis , Jeffrey Dean , Matthieu Devin , Sanjay Ghemawat , Ian Goodfellow,379 Andrew Harp , Geoffrey Irving , Michael Isard , Yangqing Jia , Rafal Jozefowicz , Lukasz Kaiser,380 Manjunath Kudlur , Josh Levenberg , Dandelion Mané , Rajat Monga , Sherry Moore , Derek381 Murray , Chris Olah , Mike Schuster , Jonathon Shlens , Benoit Steiner , Ilya Sutskever , Kunal382 Talwar , Paul Tucker , Vincent Vanhoucke , Vijay Vasudevan , Fernanda Viégas , Oriol Vinyals , Pete383 Warden , Martin Wattenberg , Martin Wicke , Yuan Yu , and Xiaoqiang Zheng . TensorFlow : Large-384 scale machine learning on heterogeneous systems , 2015 . URL https : //www.tensorflow.385 org/ . Software available from tensorflow.org.386 [ 2 ] Karim Ahmed and Lorenzo Torresani . Connectivity learning in multi-branch networks . CoRR,387 abs/1709.09582 , 2017 . URL http : //arxiv.org/abs/1709.09582.388 [ 3 ] Jose M. Alvarez and Mathieu Salzmann . Learning the number of neurons in deep networks . In389 NIPS , 2016.390 [ 4 ] Djork-Arné Clevert , Thomas Unterthiner , and Sepp Hochreiter . Fast and accurate deep network391 learning by exponential linear units ( elus ) . In ICLR , 2016.392 [ 5 ] Jonathan Frankle and Michael Carbin . The lottery ticket hypothesis : Finding sparse , trainable393 neural networks . In ICLR , 2019.394 [ 6 ] Babak Hassibi , David G. Stork , and Gregory J . Wolff . Optimal brain surgeon and general395 network pruning . In ICNN , 1993.396 [ 7 ] Kaiming He , Xiangyu Zhang , Shaoqing Ren , and Jian Sun . Deep residual learning for image397 recognition . In CVPR , 2016.398 [ 8 ] Kaiming He , Xiangyu Zhang , Shaoqing Ren , and Jian Sun . Identity mappings in deep residual399 networks . In ECCV , 2016.400 [ 9 ] Andrew G. Howard , Menglong Zhu , Bo Chen , Dmitry Kalenichenko , Weijun Wang , Tobias401 Weyand , Marco Andreetto , and Hartwig Adam . Mobilenets : Efficient convolutional neural402 networks for mobile vision applications . CoRR , abs/1704.04861 , 2017 . URL https : //arxiv.403 org/abs/1704.04861.404 [ 10 ] Gao Huang , Zhuang Liu , and Laurens van der Maaten . Densely connected convolutional405 networks . In CVPR , 2017.406 [ 11 ] Thomas N. Kipf and Max Welling . Semi-supervised classification with graph convolutional407 networks . In ICLR , 2017.408 [ 12 ] P. A. Knight . The sinkhorn-knopp algorithm : convergence and applications . SIAM Journal on409 Matrix Analysis and Applications , 30 ( 1 ) , 2008.410 [ 13 ] Alex Krizhevsky . Learning multiple layers of features from tiny images . Technical Report,411 2009.412 [ 14 ] Alex Krizhevsky , Ilya Sutskever , and Geoffrey E. Hinton . Imagenet classification with deep413 convolutional neural networks . In NIPS , 2012.414 [ 15 ] Gustav Larsson , Michael Maire , and Gregory Shakhnarovich . Fractalnet : Ultra-deep neural415 networks without residuals . In ICLR , 2017.416 [ 16 ] Y. LeCun , B. Boser , J. S. Denker , D. Henderson , R. E. Howard , W. Hubbard , and L. D. Jackel.417 Backpropagation applied to handwritten zip code recognition . Neural Computation , 1 ( 4 ) :418 541–551 , Dec 1989 . ISSN 0899-7667. doi : 10.1162/neco.1989.1.4.541.419 [ 17 ] Yann LeCun , John S. Denker , and Sara A. Solla . Optimal brain damage . In NIPS , 1989.420 [ 18 ] Hanxiao Liu , Karen Simonyan , and Yiming Yang . Darts : Differentiable architecture search . In421 ICLR , 2019.422 [ 19 ] Zhuang Liu , Jianguo Li , Zhiqiang Shen , Gao Huang , Shoumeng Yan , and Changshui Zhang.423 Learning efficient convolutional networks through network slimming . In ICCV , 2017.424 [ 20 ] Zhuang Liu , Mingjie Sun , Tinghui Zhou , Gao Huang , and Trevor Darrell . Rethinking the value425 of network pruning . In ICLR , 2019.426 [ 21 ] Neal Parikh and Stephen Boyd . Proximal algorithms . Found . Trends Optim. , 1 ( 3 ) :127–239,427 January 2014 . ISSN 2167-3888. doi : 10.1561/2400000003 . URL http : //dx.doi.org/10.428 1561/2400000003.429 [ 22 ] Karen Simonyan and Andrew Zisserman . Very deep convolutional networks for large-scale430 image recognition . In ICLR , 2015.431 [ 23 ] R. Sinkhorn . A relationship between arbitrary positive matrices and doubly stochastic matrices.432 The annals of mathematicalstatistics , 35 ( 2 ) :876–879 , 1964.433 [ 24 ] R. Sinkhorn and P. Knopp . Concerning nonnegative matrices and doubly stochastic matrices.434 Pacific Journal of Mathematics , 21 ( 2 ) , 1967.435 [ 25 ] Huizi Mao Song Han and William J. Dally . Deep compression : Compressing deep neural436 network with pruning , trained quantization and huffman coding . In ICLR , 2016.437 [ 26 ] John Tran Song Han , Jeff Pool and William Dally . Learning both weights and connections for438 efficient neural network . In NIPS , 2015.439 [ 27 ] Christian Szegedy , Wei Liu , Yangqing Jia , Pierre Sermanet , Scott Reed , Dragomir Anguelov,440 Dumitru Erhan , Vincent Vanhoucke , and Andrew Rabinovich . Going deeper with convolutions.441 In CVPR , 2016.442 [ 28 ] Robert Tibshirani . Regression shrinkage and selection via the lasso . Journal of the Royal443 Statistical Society . Series B ( Methodological ) , 58 ( 1 ) :267–288 , 1996.444 [ 29 ] Wei Wen , Chunpeng Wu , Yandan Wang , Yiran Chen , and Hai Li . Learning structured sparsity445 in deep neural networks . In NIPS , 2016.446 [ 30 ] Mitchell Wortsman , Ali Farhadi , and Mohammad Rastegari . Discovering neural wirings . In447 NeurIPS , 2019.448 [ 31 ] Xia Xiao , Zigeng Wang , and Sanguthevar Rajasekaran . Autoprune : Automatic network pruning449 by regularizing auxiliary parameters . In NeurIPS , 2019.450 [ 32 ] Saining Xie , Ross Girshick , Piotr Dollar , Zhuowen Tu , and Kaiming He . Aggregated residual451 transformations for deep neural networks . In CVPR , 2017.452 [ 33 ] Jaehong Yoon and Sung Ju Hwang . Combined group and exclusive sparsity for deep neural453 networks . In ICML , 2017.454 [ 34 ] Yang Zhou , Rong Jin , and Steven Chu–Hong Hoi . Exclusive lasso for multi-task feature455 selection . In AISTATS , 2010.456

Summary: This paper proposes a new differentiable sparsification method to learn sparse structure in neural networks in an end-to-end manner. This objective is straightforward and well-motivated. However, the execution is not novel, comparisons to the SOTA is lacking, reasons for design choices were unsubstantiated with experiments and would thus require additional ablation analysis.