Learning Compact Representations of Neural Networks using DiscriminAtive Masking (DAM)
Authors: Jie Bu
Arka Daw
M. Maruf
Anuj Karpatne
Venue: NeurIPS 2021 Poster (2021)
Keywords: Deep Learning
Network Pruning
Representation Learning
Feature Compression
Dimensionality Reduction
Graph Representation Learning
Recommendation System

Abstract: A central goal in deep learning is to learn  compact representations of features at every layer of a neural network, which is useful for both unsupervised representation learning  and structured network pruning. While there is a growing body of work in structured pruning, current state-of-the-art methods  suffer from two key limitations: (i) instability during training, and (ii) need for an additional step of fine-tuning, which is resource-intensive. At the core of these limitations is the lack of a systematic approach that jointly prunes and refines weights during training in a single stage, and does not require any fine-tuning upon convergence to achieve state-of-the-art performance. We present a novel single-stage structured pruning method termed DiscriminAtive Masking (DAM). The key intuition behind DAM is to discriminatively prefer some of the neurons to be refined during the training process, while gradually masking out other neurons. 
We show that our proposed DAM approach has remarkably good performance over a diverse range of applications in representation learning and structured pruning, including dimensionality reduction, recommendation system, graph representation learning, and structured pruning for image classification. We also theoretically show that the learning objective of DAM is directly related to minimizing the L_0 norm of the masking layer. All of our codes and datasets are available https://github.com/jayroxis/dam-pytorch.

Learning Compact Representations of Neural Networks using DiscriminAtive Masking (DAM)
Jie Bu" Virginia Tech j jayroxis@vt.edu
Arka Daw" Virginia Tech darka@vt.. edu
M. Maruf Virginia Tech marufm@vt. edu
Anuj Karpaine Virginia Tech
karpatnešvt.edd
Abstract
A central goal in deep learning is to learn compact representations of features at every layer of neural network, which is useful for both unsupervised representa- tion learning and structured network pruning. While there a growing body of work in structured pruning, current state-of-the-art methods suffer from two key limitations: (i) instability during training, and (ii) need for an additional step fine-tuning, which is resource-intensive. At the core of these limitations is the lack of a systematic approach that jointly prunes and refines weights during training in single stage, and does not require any fine-tuning upon convergence achieve state-of-the-arr performance. We present a novel single-stage structured pruning method termed DiscriminAtive Masking (DAM). The key intuition behind DAM is to discriminatively prefer some of the neurons to be refined during the training pro- cess, while gradually masking out other neurons. We show that our proposed DAM approach has remarkably good performance over diverse range of applications in representation learning and structured pruning, including dimensionality reduction, recommendation system, graph representation learning, and structured pruning for image classification. We also theoretically show that the learning objective DAM is directly related to minimizing the Lo norm of the masking layer. All of our codes and datasets are available http:::///gttuuucoooo.o cam-pytorch
Introduction
central goal in deep learning is to learn compact (or sparse) representations of features every layer of a neural network that are useful in variety of machine learning tasks. For example, in unsupervised representation learning problems [3), there is long-standing goal to learn low- dimensional embeddings of input features that are capable of reconstructing the original data [44]
35 24 33). Similarly, in supervised learning problems, there is a growing body of work in the area of nefwork praning [4], where the goal is to reduce the size of modern-day neural networks (that are known to be heavily perrpeeraeetrrized 7 51 45) so that they can be deployed in resource-constrained environments (e.g., over mobile devices) without compromising on their accuracy. From this unified view of representation learning and network pruning, the generic problem of "learning compact representations" has applications in several machine learning use-cases such as dimensionality reduction, graph representation learning, matrix factorization, and image classification.
'These authors contributed equally to this work.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

A theoretically appealing approach for learning compact representations is to introduce regularization penalties in the learning objective of deep learning that enforce Lg sparsity of the network parameters, O. However, directly minimizing the Lo norm requires performing a combinatorial search over all possible subsets of weights in 0, which computationally intractable. In practice, a common approach for enforcing sparsity is to use a continuous approximation of the Lo penalty in the learning objective, e.g., -based regularization (or Lasso [42]) and its variants [39 47). While such techniques are capable of pruning individual weights and thus reducing storage requirements, they do not offer any direct gains in inference speed since the number of features generated at the hidden layers can still be large even though the network connectivity is sparse 4]. Instead, we interested in the area of structured network pruning for learning compact representations, where the sparsity is induced at the level of neurons by pruning features (or channels) instead of individual weights.
While there is a growing body of work in structured network pruning 126 28 43 14], the basic structure of most state-of-the- art (SOTA) methods in this area (e.g., ChipNet [4] and NetSlim 28)) can be described as training a learnable vector of mask parameters, g E IR"; g = [g1. 92, .... 92......99)) go]". which when multiplied with the features extracted at a hidden layer, h E R", results in the pruned outputs of this layer, o = @ h. Sparsity in the mask parameters is generally enforced using different approximations of the Lo norm of g (c.g. use of Lasso in NetSlim and use of "crispnees" loss in ChipNet). Despite recent progress in this area, current SOTA in structured pruning suffer from two key limitations. First, since most methods do not explicitly minimize the Lo norm of g during pruning, they often suffer from training instabilities. In particular, most SOTA methods 28 involve thresholding techniques to set small non-zero weights to zero leading to large drops in accuracy during the training process, evidenced by our results in this paper. Second, once the pruning process is complete and we have converged a compact network, most SOTA methods still need an additional step of fine-tuming the network in order to achieve reasonable accuracy. This is not only resource-intensiv but there is also an on on-going discussion on whether and how we should fine-tune (36) or rewind initial weights 26 or train from scratch [29]. making it difficult to prefer one approach over another.
Gose Punttio
Training
Epoch
Epoch T
9,
gx
9,
9x
0
0g
0
0
hr
Figure Illustration of Discrimi- native Masking. The gate function shifts to the "right" during training resulting in more zeros in the mask convergence.
At the core of these limitations is the lack of systematic approach for structured network pruning that jointly prunes and refines weights during training in a single stage, and does not require any fine-tuning upon convergence to achieve SOTA performance. Notice that in existing methods for structured pruning, allowing every neuron j to be pruned differently using an independent mask parameter 9j only increases the number of learnable (or free) parameters in the learning objective, increasing the complexity of the problem. Instead, we ask the question: "Can we leverage the intrinsic symmetry of neural network initialization to design a pruning mask using the least number of free parameters?"
We present simple solution to this question by proposing new single-stage structured pruning method that learns compact representations while training and does not require fine-tuning, termed DiscriminAtive Masking (DAM). The basic idea of DAM is use monotonically increasing gate function G for masking every neuron in a layer that only depends on the index j = 1, 1,2, ..., n (or position) of the neuron in the layer and scalar parameter to be learned during training (see Figure At the start of training, the gate function admits non-zero values for all neurons in the layer, allowing all neurons to be unmasked (or active). As the training progresses, the gate function gradually shifts from "left" to "right" as a result of updating / such that upon convergence, only a subset of neurons (on the extreme right) are active while all others are masked out. The key intuition behind DAM is to discriminatively prefer some of the neurons (on the right) to be refined (or re-adapted) during the training process for capturing useful features, while gradually masking out (or pruning) neurons on the left. This preferential pruning of neurons using a very simple gate function helps in regulating the number of features transmitted to the next layer]
Contributions: We show that our proposed DAM approach has remarkably good performance over various applications, including dimensionality reduction, recommendation system, graph represen-
This is somewhat analogous to how physical dam regulates the flow of water by shifting a movable gate.

---

tation learning, and structured pruning for image classification, achieving SOTA performance for structured pruning. This shows the versatility of DAM in learning compact representations on diverse problems and network choices, in contrast to baseline methods for structured pruning that are only Our approach single-stage, does not require fine-tuning (and hence has lower training time), and does not suffer from training instabilities. We also theoretically show that the learning objective of DAM is directly related to minimizing the Lo norm of the discriminative mask, providing new differentiable approximation for enforcing Lo sparsity. Our approach is also easy to implement and can be applied to any layer in network architecture. The simplicity and effectiveness of DAM provides unique insights into the problem of learning compact representations and its relationship with preferential treatment of neurons for pruning and refining, opening novel avenues of systematic research in this rapidly growing area.
developed and tested for the problem of image classification [28 43
2 Related Work
Learning with Sparsity: There is a long history of methods for enforcing compactness (or sparsity) in learned parameters (2), where it is understood that compact hypotheses can not only result in more efficient data representations but also promote better generalizability [37.. This connection between compactness and generalization has also been recently explored in the context of deep motivating our problem of learning compact representations.
learning [49
Unstructured Pruning: There are several methods that have been developed for unstructured network pruning (30 IO 38), where the goal is to prune individual weights of the network. A widely used technique in this field is referred to as the Lottery Ticket Hypothesis (LTH) [TO], which suggests that certain subset of weights in a network may be initialized in such way that they are likely "winning tickets", and the goal of network pruning is then simply uncover such winning tickets from the initialization of over-parameterized network using magnitude-based pruning. Similar to our work, there exists a line of work in the area of dynamic sparse training for unstructured pruning 500 13 32 9 27 that gradually prunes the model to the target sparsity during training. However, while unstructured pruming methods can show very large gains in terms of pruning ratio, they are not directly useful for learning compact representations, since the pruned weights, even though sparse, may not be arranged in a fashion conducive to the goal of reducing the number of features.
The goal of structured pruning is to enforce sparsity over the network substructures (e.g., neurons or convolutional channels) that is directly relevant to the goal of learning compact representations. Most structured pruning SOTA methods need a three-stage process [6], i.e., training, pruning and finetuning, in order to get highly compressed models. Early exploration on the Ln norm based filter pruning can be traced back to (66. followed by Network Slimming (NetSlim) (28). which uses sparsity- -inducig regularization based L1 norm. Following this line of work. a recent development in the area of network pruning is ChipNet [43], which introduces additional crispness loss to the L1 regularization, and achieves SOTA performance on benchmark tasks. However, network slimming can only work if the underlying architecture has batch normalization layers. It also has training instabilities due to soft- noft-thresholding inherent to Lasso. ChipNet also iterates between soft- and hard- hard-pruning, which is done after the training phase. Further, both these approaches require finetuning or retraining, which can be resource intensive.
Structured Network Pruning:
Faster pruning schemes have also been explored by previous works. Bai structured pruning algorithm utilizing filter cross distillation. The SNIP 25 took one step ahead by performing single-stage pruning. However, SNIP falls in the category of unstructured pruning. To the best of our knowledge, no existing structured pruning method has been demonstrated to achieve SOTA performance without finetuning. For example, while the method proposed in [30] can be potentially used as a single-stage structured pruning method with the help of group sparsity, the empirical analysis of the paper only focused on unstructured pruning.
proposed a few-shot
Deep Representation Learning: Leveraging the power of deep neural networks, high- figh--dmensional data can be encoded into a low-dimensional representation [8. Such process known as deep representation learning. A good representation extracts useful information by capturing the underlying explanatory factors of the observed input. The existence of noises, spurious patterns and complicated correlations among features make it challenging problem. Previous work [4] show that an
3

---

Autoencoder architecture can be used for denoising images. To disentangle correlated features, VAE 19 and B- 3-VAE [17 encourages the posterior distribution over the generative factors g(2(2) to be closer to the isotropic Gaussian (0, T) that explicitly promotes disentanglement of the latent generative factors. Driven by the idea of the autoencoder, the deep representation learning has been found remarkably successful in various of applications, e.g., graph representation learning and recommendation system [48].
21 6
31
46, 34]
However, most existing works in representation learning treat the embedding dimension as a hy- perparameter, which can be crucial and difficult to choose properly. Small embedding dimensions cannot sufficiently represent the important information in the data that leads to bad representations. On the other hand, large embedding dimensions allow some level of redundancy in the learned representations, even picking up spurious patterns that can degrade model performances.
3 Proposed Approach
Let us represent the general architecture of a neural network with I layers as F(x) = V V o ...V/2 (z),, where æ are the input features to the network, V is the learnable function at layer that generates hidden features h, = v (,(h;-1), and ff o g represents the composite function (')).. Let us also consider the "masked" version of this network where the features learned at every hidden layer, h; are multiplied with a vector of mask values g. using a mask function M, (h,)) 9: @ h,. The masked neural network can then be described as F (a) = V M1 o 1 o ...My o 12 o M1 a V (x:). The goal of learning compact representations can be framed as minimizing the following learning objective:
Problem Statement:
A
min F FarDDrr (Fmaskedd (z:)
where O are the learnable parameters of the network, Dir is the training set, E() is the loss function, and À is the trade-off parameter for enforcing Lo sparsity in the mask values gi- Note that this formulation provides a unified view of both representation learning and structured network pruning, which differ in the choice of loss function (typically, reconstruction error for representation learning and cross-entropy for structured pruning), and the number of layers that are pruned (in representation learning, we only aim to prune the bottleneck layer while in structured pruning, we aim to prune all of the layers in the network, Figure 2).
Our proposed approach presents a novel perspective of structured pruning by introducing new strategy of aeddering-andmaskiiie " To understand the basic idea of this strategy, let us assume that we are given the ideal "ordering" of neurons in a layer that reflect their importance (i.e., neurons with higher order number more useful for the learning task). Given this ordering, we can perform structured pruning by incrementally "masking" out neurons with lower order (i.e., setting their mask values to 0) before going neurons with higher order during the training process. In other words, we discriminatively (or preferentially) focus on pruning neurons lower in the order, while retaining and refining the features learned neurons higher in the order. This approach for masking can be realized using a monotonic gate function G that maps neuron order mask values, i.e., lower order neurons are assigned smaller mask values than higher order neurons. By dynamically regulating the gate function during the course of training, we can control the percentage of neurons pruned at every training epoch.
DiscriminAtio Ative Masking (DAM):
Specifically, let us assign every neuron j at layer with an "order number," Pky. Before we describe how we arrive at these neuron orders, we first present the formulation of our proposed gate function g, whose value at neuron j monotonically increases with the order number of the neuron, May. as follows:
gij = G(ug, a,, Bi) = ReLUUtann (au(ug + +B))).
(2)
where a; is a constant scalar parameter (termed as the steepness parameter) that is not optimized during training, while 33 is 'learnable' scalar parameter (termed the offset parameter) that is the only parameter optimized during training to control (or regulate) the process of masking in DAM Figures c and (d) show examples of gate function values (in red) at two different values of [r, Note that using a single learnable parameter 8; to implement the masking process introduces significant simplicity in the design of the learnable mask, in contrast to SOTA methods in structured

---

(a) Reposseatation Learning
Networks Pruning
(0) 9 -2 (30% praned)
[4) P 46 (60% prinoc)
Figure 2: Illustration of the problems of representation learning (a) and structured network pruning (b). Blue blocks are layers in neural networks while red blocks show non-zero values in the mask layers. Figure (c) and (d) show two states of the gate function for is = 10. As an example, 3= in (c) prunes 20% of the neurons while 8=-6 in (d) prunes 60% of the neurons.
where a different masking parameter is trained for every neuron in the network. We hypothesize this simplicity of the gate function, in conjunction with the preferential pruning property of DAM, to result remarkable gains learning compact representations, as evidenced later in experiments. Further, note that while implement our gates, we found ReLU-tanh to perform best in practice.
pruning 28 43
other monotonically increasing function to
Assigning Neuron Orders Pay: We now discuss our approach for assigning neuron order numbers Pig. Note that the order numbers only have be assigned during the initialization step, and do not change during the training process. Further, note that every neuron is initialized with weights that are independent and identically distributed (because of the i.i.d nature commonly used initialization methods in deep learning [4l).. Hence, all neurons initialization and any neuron has an equally good chance of refining themselves to capture useful features later in the training process as any other neuron. Hence, any random ordering of the neurons the DAM approach would result a similar pruning process (we empirically demonstrate the permutation invariance of P4j in Section 5). For simplicity, consider trivial choice of Hij = kjjnng n,, where & is a constant parameter that determines the domain size of μij. and n, is the total number of neurons at layer
symmetrically invariant to each other at
How Does Reducing 3, Enforce Lo Sparsity? Let us first understand how mask values gi respond to the only learnable parameter 33. [ shows an example of gi when & = 2. At this stage, we can see that only small number of neurons have zero gate values (white). As we reduce 3 to -6 in Figure 2 (d), we can see that the gate moves towards the 'right', resulting in more excessive pruning of neurons. In general, the number of non-zero values the gate function (and hence, its Lo norm) is directly related to 3, follows:
1gg: llu = Σ
19g > 0)
Σ
1(p6 S B.)) [n,(1 + B,/K)], for B -
(3)
where 1 indicates the identity operator and [-] is the ceiling operator. order to back proppgate gradients, regularization as llg.lo & n1,1 + B,/k). Hence, minimizing Lo norm of gi is directly proportional to minimizing 8 with scaling factor of ni/k.
continuous approximation of Equation (3) (by dropping the ceiling operator) for
Learning Objective of DAM: While minimizing 6, with scaling factors proportional n, can explicitly minimize Lu norm of the gate values, we found in practice that layers with smaller number of neurons nı still require adequate pruning of features for learning compact representations proportional to layers with large nı, Hence, in our learning objective, we drop the scaling factor and directly minimize the sum of 3, across layers the following objective function.
A
min EawDer 0,3
( [(Fmasted(()))
Σ
8
(4)
where, ,3 is the set of all B3's in the network, {31, B2, 82.,,,,.., 31-1)
Theoretical Analysis of DAM: Analysis of the dynamics of the gate functions and additional specifications for choosing a; and the gate function are provided in Appendix A.
Parameter Specification and Implementation Details: In all of our experiments, we used a; = 1 and k = 5. In our structured network pruning experiments, we used cold-start of epochs (i.e., the 3, 's were frozen for the duration of cold-start), as allow the leftmost neurons undergo
5

---

some epochs of refinement before beginning the pruning process. We also set the initial value of 3g to which can also be thought of another form of -old-stariing (since pruning layer only starts when 33 becomes less than zero).
4 Experiments on Representation Learning Problems
4.1 Results on Dimensionality Reduction (DR) Problems
We evaluate the effectiveness of DAM in recovering the embeddings of synthetically generated dimensionality reduction problems. The general form of the problem is expressed by (5). Suppose S2 E R" and all its r dimensions are independent from each other. A d-dimensional data X can be expressed using transformation v IR" Rd defined on Hilbert space, and X (S), d > r. Let M be the learnable mask, we formulated the dimensionality reduction problem as:
Problem Setup:
min @
| M FE(X) XI| + + θ = 10 0D.3)
(5)
The FE. FD.D are trained end-to-end using gradient descent. In our experiments, 12 sampled from isotropic normal distribution, i.e., Ω P~~ N,, I.). We let v to use the same structure as the decoder FD with randomly generated parameters 0g. i.e., V = Fpl8g=0er so that the minimum number of dimensions needed in the encoded representation for the decoder to reconstruct X is r.
Linear DR: We DAM for removing linear correlations case, V is a matrix representing linear projection from R" to R We use two real matrices as the encoder and decoder and train the DAM to find the full-rank representation of X.
given rank-deficient matrix. In this
Nonlinear DR: To test the ability of DAM for disentangling nonlinearly correlated dimensions, we present cases: (i) T is polynomial kernel of degree 2 (we QRes layer from (5); (ii) v is a nonlinear transformation expressed by neural network. We use a deep neural network as the encoder with sufficient complexity.
Bettenekk Dimenstonn
Betllewekk Dimeraito
Bettlneec Dimersion
60-
au
40
30
20-
dim
20-
dim
20
dim
dim
10
dim
10
dim 10
10
dim
Epochs
Epochs
Epochs
(a) Linear V
(b) Polynomial V
(c) Neural Network V
Figure 3: Bottleneck dimension (of nonzero entries) vs number of epochs during training.
Observation: Figure 3 shows the convergence of bottleneck dimensions over the training epochs for different synthetic data with varying sizes of underlying factors (r). The curves and shades show the mean and standard deviation of five different runs. We can see that DAM consistently uncovers the exact dimension of the underlying factors We provides details of in-depth sensitivity analysis of DAM results based on its hyper-parameters (e.g., learning rate and X) in Appendix C.
Theoretical Analysis of DAM for linear DR: We theoretically show that DAM converges to the optimal solution for the linear DR case (see Appendix B for details).
4.2 Recommendation System Results
Problem Setup: We consider the state-of-the-ar method for recommendation system problems (IGMC [48), which transforms the rating matrix completion task to a link prediction problem, where the ratings interpreted links between users and items.
In particular, IGMC generates 128- 128-dimensional embeddings to represent the enclosing subgraph and further avoids overfitting using dropout layer with dropout rate of 0.5. We replaced the dropout

---

Table 1: Results of IGMC with and w.o. DAM on recommendation system tasks.
IGMC
KGMC-DAM
RMSE
Dimension
RMSE
Dimension
Flixter
0,8715 = 0.0005
128
0.8706
0.0003
32
Douban
0.7189 0.0002
128
0.7183
0.0004
17
Yahoo-Musk
19.2488 + 0.0123
128
19.0166 0.0444
83
layer with a DAM layer in the IGMC to reduce the dimension the learned representations. We train our IGMC-DAM model for 100 epochs under the same training configurations.
Observation: Table D shows that DAM successfully reduces the dimensions of the learned represen- tations without any increase in errors, demonstrating that DAM is able to boost the performance of IGMC by learning compact representations over the target graph.
4.3 Graph Representation Learning Results
Following the previous experiment, we further explore the effectiveness of DAM in learning compact graph representations. Our goal learn low-dimensional embeddings for each node in graph that captures the structure of interaction among nodes. A simple graph autoencoder, e.g., GAE [21]. uses graph convolutional network (GCN) [20] as an encoder and an inner product as a decoder. The encoder calculates the embedding matrix Z from the node feature matrix X with the adjacency matrix A, and the decoder reconstructs adjacency matrix such that: Á = oZZ ) with Z = GCN(X, A). The reconstruction loss, BCE (A, A) is backpropagated train the GAE model. To reduce the dimension the learned representation, we add DAM layer in GAE after the encoder (GAE-DAM).
Problem Sctup:
0.92
0.89
0.97
AUC
AUC
0.85
AUC
0196
0.95
0.88
GAE
GAE-DAM
GAE-DAM
0.77
0.94
20
Bottleneck Dimension
Bottleneck Dimension
Bottleneck Dimension
(a) Cora
(b) CiteSeer
(c) PubMed
Figure 4: Link prediction performance for Cora, CiteSeer and PubMed Dataset. Competing models
GAE, GAE-DAM.
Observation: Figure shows that DAM based GAE method able to learn meaningful structural information in compact latent embeddings. DAM improves the link prediction performance of the simple GAE for Cora, CiteSeer and PubMed dataset by learning compact representations.
4.4 Representation Learning Results for MNIST Dataset
Problem Setup: We further evaluate DAM on the problem of dimensionality reduction using simple auto-encoders on the MNIST dataset, and also compare the effectiveness DAM (that directly enforces Lo sparsity) with ablation of our approach that instead minimizes the L1 norm (similar to Lasso). We vary the trade-off parameter to compare the sparsity the learned representations between DAM and L, -nor based ablation method.
Figure shows that DAM is able to achieve lower reconstruction error as well as higher F1 scores over the same bottleneck dimensions than L1 based method. In Figure (c).. (c). we can see that DAM shows near-linear descending trend bottleneck dimensions as increases (further evidence presented in Appendix E1). In contrast, the Ln based method has saturating effect at large values since the weights come close to zero but require some thresholding to pruned. This shows that DAM is amenable to learning highly compact representations in contrast to L₃ based methods.
Observation:

---

Rec. Loss
0.125
DAM
FI-Macro
0.98
Bottleneck Dim.
20
DAM
0.100
Ly morm
0.92
Li -orm
DAM
10-
0.073
0.86
I 1nnorrm
0.050
0.80
12
19
40
12
19
26
33
40
10
Bottleneck Dimension
Bottleneck Dimension
Figure 5: Performance comparison plots of DAM with I1-norm method on MNIST dataset.
Experiments on Structured Network Pruning
Evaluation Setup Here we demonstrate the effectiveness of DAM on the task of structured network pruning. We compared the performance of our proposed DAM with two SOTA structured pruning algorithms, Network Slimming (Net-Slim) (28] and ChipNet 4
Net-Slim is well-established SOTA that is widely used strong baseline, while ChipNet is the latest SOTA representing very recent development in the field. ChipNet and Net-Slim are both pretrained for 200 epochs, while the DAM performs single-stage training and pruning in the does not require additional fine-tuning, we imposed limited budget of 50 epochs for fine-tuning the SOTA methods. Note that ChipNet also has a pruning stage which involves 20 additional epochs. We evaluate the performance of these methods on the PreResnet- 164 architecture on benchmark computer vision datasets, CIFAR- 10 and CIFAR- 100. Additional evaluation setup details are in Appendix E2.
number of epochs. Since, our DAM approach
Performance and Running time comparison: Figures 6a and 6b demonstrate the performance of different network pruning methods for various pruning ratios. We observe that for both datasets, DAM is able to outperform Net-Slim by a significant margin specially when the models are sparse. Additionally, DAM is able achieve similar slightly better performance than ChipNet with no additional fine-tuning. We also compare the total running time for pruning across the different models (divided into the three categories) in Figure 6c We are almost the same, with ChipNet being slightly small. However, the running time for the pruning and ane-tuning stages for DAM are both zero. Net-Slim also does not involve additional pruning after training. However, ChipNet involves 20 epochs of pruning which is significant and is almost comparable to its pretraining time. Finally, comparing the total running time taken by each of the structured pruning methods, we observe that DAM is significantly faster than the current SOTA counterparts owing to its stage pruning approach.
that the training time of the three methods
Test Accuracy (%)
95
Test Accuracy (%)
Itain
Finelune
90
Prune
Total
DAM
65
DAM ChipNet
ChipNet
Net-Stim
55
Net-Slim
25
50
75
100
20
40
60
80
100
Parameters Remaining (%)
Parameters Remaining (%)
DAM
ChipNet
Net-Stim
(a) CIFAR-10
(b) CIFAR-100
(c) Training Time
Figure 6: Performance and running time comparison of DAM with state-of-the-art structured pruning methods on 1re for various parameter pruning ratios.
We further analyse the stability of our proposed DAM approach for network pruning through visualization of the training dynamics. We observe that the training cross- cross-entropy (CE) loss and the validation CE loss are very similar to what we expect from a Preresnet model trained on benchmark vision datasets with learning rate changes at 100 and 150 epochs, respectively. We further notice that the total training loss for the DAM also very stable and does not show any sudden variations, leading to steady convergence. Finally, from the convergence plot for E[8.] (green-dashed line), we can see that our pruning does not involve any sudden steps and happens in continuous and steady rate throughout the training stage.
Stability Analysis:

---

The training dynamics of the ChipNet (blue shaded region in the middle panel) is exactly the same as training vanilla vision models since does not involve sparsity constraints. However, as move to the pruning stage, we notice sharp rise in both training and validation losses (white region). This is due to the use of a different optimizer AdamW as opposed SGD which was used in the training stage. Further, very interesting phenomenon happens in the fine-tuning stage (red shaded region) where the training and validation losses increases for the first 25 epochs and then takes a sharp decrease once the learning rate is adjusted. This suggests that after the pruning stage, finetuning the model with large learning rate forces the model to make large updates the initial epochs (akin to training from scratch), which further drives the model out of the local minima. Once the learning rate adjusted, the model is able to slowly progress towards a new local optima and ultimately converges to the fine-tuned minima.
Finally, both the training and fine-tuning dynamics the Net-Slim (rightmost panel) appears to be highly unstable. This suggests that the pre-training with sparsity loss is not stable and can lead to large fluctuations in the training loss. Also, note that Net-Slim does not have a validation set as instead uses the test set to choose the best model. We have refrained from using the best model by looking at the performance on the test set, and instead evaluate the model from the last epoch.
2
Vil CE Lows -.. Mean &
3
Val Lose
Tine CE Loss
2-
2
-2
0
100
200
100
200
Epoch
Epoch
Epoch
Figure 7: Stability analysis for DAM (left), ChipNet (middle) and Network Slimming (right) through visualization of the loss curves. Blue, white and red shades denote the training, pruning and fine- tuning stages respectively.
Permutation Invariance: To test the effect of permuting the neuron order on the performance of DAM, Table 2 provides results across five randomly permuted values of juy across every neuron in the network, while using the same initialization previous experiments. We can see that DAM invariant to permutations of neuron order during initialization, validating our simple choice of assigning neuron ordering based their index values.
Table 2: Permutation Invariance results PreResNet-164 on CIFAR-10. RSD denotes the relative standard deviation.
Run
Run 2
Run 3
Run 4
Run
RSD
Test Accuracy
94.1 11.11%
93.86%
93.71%
93.93%
93.52%
0.0024
Channels Pruned
43.31 %
43.14%
43.31%
43.14%
43.02%
0.0029
Parameters Pruned
64.01%
63.40%
63.97%
63.64%
63.37%
0.0048
To illustrate generalizibility we provide additional network pruning results for more networks such as VGG-19, PreResnet-20, PreResNet-56 and PreResnet- 110 on CIFAR datasets and LeNet-5 MNIST in Appendix E3. To demonstrate decorrelated kernels obtained after pruning using DAMs we use CKA similarity (22]. The results can be found Appendix E3. For extreme values of X, we demonstrate that DAM is able to pruning entire bottleneck layers for ResNet architectures by visualizing the pruned architectures in Appendix E4.
Additional Results:
Discussion, Open Questions, and Future Works
The effectiveness of DAM learning compact representations provides novel perspective into structured network pruning and raises several open questions about its relationships with other concepts this area. the following, we list some of these connections and describe the limitations and future extensions of our work.

---

Connections with Lottery Ticket Hypothesis: Our work builds on the intuition of discriminative masking, i.e. we preferentially favor some neurons be refined while favoring some other neurons to be pruned. This has interesting connections with the lottery ticket hypothesis (LTHH) [10] where some subsets of initialized weights in the network are hypothesized to contain "winning tickets" that are uncovered after few epochs of training. While there is some open discussion on whether the winning tickets exist even before initialization [ there are several studies supporting this hypothesis Our results attempts to throw light on the question: "can we find winning tickets if we discriminatively search for it in a certain subregion the network?" Further, we are able to show that the results of DAM are invariant random permutation of the neuron indices initialization, since all neurons receive identically distributed weights. Along these lines, we can also explore there certain ordering of neurons (e.g., in accordance with their likelihood of containing winning tickets revealed through LTH) that can perform better than random ordering in DAM.
38 [2.
Connections with Dropout: Another interesting connection of DAM is with the popular regular- ization technique of Dropout 40 that randomly drops out some neurons during training to break "co-adaptation" patterns in the learned features so to avoid overfitting Essentially, by randomly dropping a neuron at some epoch of training, the other neurons are forced to pick up the learned features of the dropped neuron, thus resulting in the learning of robust features. This is similar to the "re-adaptation" of weights at the active neurons at some epoch of DAM training, while the mask value of the neurons in the transitioning zone (when 0 9j 1) are gradually dropped by the shifting of the gate function. This motivates us to postulate the weight re- re-adaptation hypothesis as a potential reason behind the effectiveness of DAM in learning compact representations, which requires further theoretical justifications.
Budget-aware Variants of DAM: One promising future extension of our current DAM formulation would be to make budget-aware, i.e., to stop the training process once we arrive at a target level of sparsity. Note that there is direct correspondence between the value of the learnable gate offset parameter 3; (that is minimized at every epoch) and the resulting Lo sparsity (see Equation 33. Hence, simple budget-aware extension of our current DAM formulation would be to start from a large value of lambda (to provide sufficient scope for aggressive pruning) and keep monitoring the level of sparsity at every layer during the training process. As soon the target sparsity mark is achieved, we can freeze 3 's at every layer thus essentially halting the pruning process (note that the network architecture becomes immutable if 3,' is fixed).
One of the fundamental assumptions of our current DAM formulation is that all neurons are symmetrically invariant to each at initialization, and hence we can simply use random neuron ordering for discriminative masking. While this assumption valid for "training from scratch", this may not hold for neurons in pre-trained network. Future extensions of DAM can include advanced ways of ordering neurons that do not rely on the above assumption and hence can even be used with pre- are-trained networks. For example, we can order neurons based on the magnitudes of weights of every neuron or the value of mutual information between neuron activations and outputs. This would open novel avenues of research in structured network pruning by finding effective neuron orderings for different initializations of network weights.
DAM Variants for Pre-trained Networks:
7 Acknowledgement
This work was supported by the NSF Eager Grant #2026710.
References
[1] Sanjeev Arora, Rong Ge, Behnam Neyshabur, and Yi Zhang. Stronger generalization bounds for deep nets via a compression approach. In International Conference on Machine Learning, pages 254-263, PMLR, 2018.
[2] Haoli Bai, Jiaxiang Wu, Irwin King, and Michael Lyu. Few shot network compression via cross distillation. In AAAI, volume 34, pages 3203-3210, 2020.
[3] Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 35)) 1828, 2013.
10

---

[4] Davis Blalock, Jose Javier Gonzalez Ortiz, Jonathan Frankle, and John Guttag, What is the state neural network pruning? arXiv preprint arXiv:2003.03033. 2020.
[5] Jie Bu and Anuj Karpatne. Quadratic Residual Networks: A New Class of Neural Networks for Solving Forward and Inverse Problems in Physics Involving PDEs, pages
SIAM, URL https://epubs. siam. wtg/
2021. doi: 10. 103711 97199988161111 19766700.....
1137/1 97816119767000 76
[6] Tim R Davidson, Luca Falorsi, Nicola De Cao, Thomas Kipf, and Jakub M Tomezak. Hyper- spherical variational auto-encoders. arXiv preprint 1801v01804.0081, 2018.
[7] Misha Denil, Babak Shakibi, Laurent Dinh, Marc' Aurelio Ranzato, and Nando de Freitas. Predicting parameters in deep learning. In Proceedings of the 26th International Conference on Neural Information Processing Systems- Jystems-Volume Volume 2, pages 2148-2156, 2013.
[8] Xinhan Di, Penggian Yu, Rui Bu, and Mingchao Sun. Mutual information maximization in graph neural networks. In 2020 International Joint Conference on Neural Networks (IJCNN). pages 1-7. IEEE, 2020.
[9] Utku Evci, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning. volume 119 of Proceedings of Machine Learning Research, pages 2943-2952, PMLR, 13-18 Jul 2020. URt.httpss/prroeediins mlr. www.ppees/eee v1 19/ evei html
[10] Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In ICLR, 2018.
[11] Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M Roy, and Michael Carbin. Stabilizing the lottery ticket hypothesis. arXiv preprint (rXiv:1903.0161/ 2019.
[12] Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M. Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis, 2020.
[13] Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity in deep neural networks, 2019.
[14] Ariel Gordon, Elad Eban, Ofir Nachum. Bo Chen, Hao Wu, Tien-Ju Yang, and Edward Choi. Morphnet: Fast & simple cecuurre-constrinnee structure learning of deep networks. In Proceed- ings of the IEEE conference on computer vision and pattern recognition, pages 1586-1595, 2018.
[15] Kathrin Grosse and Michael Backes. Do winning tickets exist before DNN training?, pages 549 559- SIAM, 2021. doi: 10.1 9781611976700666 URL https:://pubbs.. siam org/doi/abs 10. 1137/1 97811119767000.c 62
[16] Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connec- tions for efficient neural network. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 28. Cur- ran Associates, Inc., 2015. URL httpss://ppoooedingg/ wewrooeedngg..nuisss..ee../////e nwwrnne.uc//ppppoeeeee pow.rppp.cco/ppppp//0////111 as00bec332berre2222999466633 Paper pdf
[17] I. Higgins, Loic Matthey, A. Pal, Christopher P. Burgess, Xavier Glorot, M. Botvinick, S. Mo- hamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. In ICLR, 2017.
[18] G. E. Hinton and R. R. Salakhutdinov. Reducing the dimensionality of data with neural networks. Science, 313(5786):504-500 2006. ISSN 0036-8075. doi: 10. 10.1126/science. 1276447. URL https://science..set www.:ennccc.scccccmmm.... org/ http://ntn////////////5555
[19] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint 131266 14, 2013.
[20] Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:160.002907, 2016.

---

[21] Thomas N Kipf and Max Welling. Variational graph auto-encoders. arXiv preprint arXiv:1611.07308, 2016.
[22] Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International Conference on Machine Learning, pages 2519-3529. PMLR, 2019.
[23] Duong Hoang Le and Binh-Son Hua. Network pruning that matters: A case study retraining variants. In International Conference on Learning Representations, 2021. URL https:
[24] Quoc Le, Jiquan Ngiam, Adam Coates, Ahbik Lahiri, Bobby Prochnow, and Andrew Y Ng. On optimization methods for deep learning. In ICML, 2011.
[25] Namhoon Lee, Thalaiyasingam Ajanthan, and Philip Torr. Snip: Single-shot network pruning based on connection sensitivity. In International Conference on Learning Representations, 2018.
[26] Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. arXiv preprint arXiv:1608.087: 2016.
[27] Shiwei Liu, Lu Yin, Decebal Constantin Mocanu, and Mykola Pechenizkiy, Do we actually need dense over-paramettriiation?? in-time vve-ppramettrizztioon in sparse training, 2021.
[28] Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learning efficient convolutional networks through network slimming. In ICCV, 2017.
[29] Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. Rethinking the value of network pruning. In ICLR, 2018.
[30] Christos Louizos, Max Welling, and Diederik P Kingma. Learning sparse neural networks through 1_0 regularization. In ICLR, 2018.
[31] Costas Mavromatis and George Karypis. Graph infoclust: Leveraging cluster-level node information for unsupervised graph representation learning. arXiv preprint urXiv:2009.06944 2020.
[32] Decebal Constantin Mocanu, Elena Mocanu, Peter Stone, Phuong H. Nguyen, Madeleine Gibeseu, and Antonio Liotta. Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science. Nature Communications, 9, 2018.
[33] Bruno A Olshausen and David Field. Emergence of simple-cell receptive field properties by learning sparse code for natural images. Nature, 381 (6583))607-609, 1996.
[34] Shirui Pan, Ruiqi Hu, Guodong Long, Jing Jiang. Lina Yao, and Chengqi Zhang. Adversarially regularized graph autoencoder for graph embedding. arXiv preprint arXiv:1802.04400) 2018.
[35] Marc' Aurelio Ranzato, Y-Lan Boureau, Yann LeCun, et al. Sparse feature learning for deep belief networks. Advances in neural information processing systems, 20:1 10:1185-1199 1192, 2007.
[36] Alex Renda, Jonathan Frankle, and Michael Carbin. Comparing rewinding and fine-tuning in neural network pruning. International Conference on Learning Representations, 2020. URL https://openrevies http:///ppaaeeeeeeeee///oooaooap
[37] Jorma Rissanen. Stochastic complexity and modeling. The annals of statistics, pages 1080-1100, 1986.
[38] Pedro Savarese, Hugo Silva, and Michael Maire. Winning the lottery with continuous sparsifi- cation. NIPS, 33, 2020.
[39] Skyler Seto, Martin T Wells, and Wenyu Zhang. Halo: Learning to prune neural networks with shrinkage. In Proceedings of the 2021 SIAM International Conference on Data Mining (SDM), pages 558-566. SIAM, 2021.
12

---

[40] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, llya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929-1958, 2014.
[41] Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initialization and momentum in deep learning. In International conference on machine learning.
pages 1139- 1147. PMLR, 2013.
[42] Robert Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society: Series B (Methodological), 58(1):267-288, 1996.
[43] Rishabh Tiwari, Udbhav Bamba, Arnav Chavan, and Deepak Gupta. Chipnet: Budget- aware pruning with heaviside continuous approximations. ICLR, 2021. URL https
//oparoview..et/ooruufff wtwp://oorrm///ddxxxxxxxxx
[44] Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. Proceedings of the 25th International Conference Machine Learning, ICML '08, page 1096-1103, New York, NY, USA, 2008. Association for Computing Machinery. ISBN 978 97816055820 doi: 10. 14./119999955 390156. 13902999 URL https://doi....... orgooog//// 10. 1111/1111/111666.1 1390294
[45] Jayyun Wang. Yubei Chen, Rudrasis Chakraborty, and Stella X Yu. Orthogonal convolutional neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11505-115155 2020.
[46] Hong Yang, Shirui Pan, Peng Zhang, Ling Chen, Defu Lian, and Chengqi Zhang. Binarized attributed network embedding. In 2018 IEEE International Conference on Data Mining (ICDM), pages 1476-1481. IEEE, 2018
[47] Jihun Yun, Peng Zheng, Funho Yang, Aurelie Lozano, and Aleksandr Aravkin. Trimming the regularizer: Statistical analysis, optimization, and applications to deep learning. In International Conference on Machine Learning, pages 7242-7251. PMLR, 2019.
[48] Muhan Zhang and Yixin Chen. Inductive matrix completion based on graph neural networks. In ICLR, 2019.
[49] Wenda Zhou, Victor Veitch, Morgane Austern, Ryan Adams, and Peter Orbanz. Compress- ibility and generalization in large-scale deep learning. CoRR, Habs 1804-05862 http://dblp... uni-turiieete de/db/ http://oonaassco///o/o//////////
2018. URL
[50] Michael Zhu and Suyog Gupta. To prune, or not to prune: exploring the efficacy of pruning for model compression, 2017.
[51] Xiaotian Zhu, Wengang Zhou, and Houqiang Li. Improving deep neural network sparsity through decorrelation regularization. In IJCAI, pages 3264-3270, 2018
13


Reviews:
[{"review_id": "ykyhFIs1FUN", "reviewer": "Reviewer_69J5", "summary": "This paper proposes a gradual structured pruning method, DAM, that can achieve good performance with various applications. The paper conducts extensive experiments to support the effectiveness of the proposed methods. The method part is reasonable and makes sense.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I generally like the simplicity of the proposed method and the empirical results are promising. \n\nWhat I don't like is the method is proposed heuristically and suddenly. I want to read more reasoning parts of the proposed method. For instance, is this method stemmed from other disciplines or just heuristically? Can we use sigmoid function rather than tanh in equation 2? \n\nMoreover, I encourage the authors to add more unstructured sparsity methods in the related work. First,  gradual magnitude pruning [1,2] also gradually prunes the model to the target sparsity during training, which should be added to the related work. An important branch of unstructured sparsity methods, dynamic sparse training [3,4,5], is ignored. Even though it is an unstructured method, the sparse connectivity is allowed to be gradually optimized during training.  \n\n[1] Zhu, Michael, and Suyog Gupta. \"To prune, or not to prune: exploring the efficacy of pruning for model compression.\" arXiv preprint arXiv:1710.01878 (2017).\n[2]Gale, Trevor, Erich Elsen, and Sara Hooker. \"The state of sparsity in deep neural networks.\" arXiv preprint arXiv:1902.09574 (2019).\n[3] Mocanu, Decebal Constantin, et al. \"Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science.\" Nature communications 9.1 (2018): 1-12.\n[4] Evci, Utku, et al. \"Rigging the lottery: Making all tickets winners.\" International Conference on Machine Learning. PMLR, 2020.\n[5] Liu, Shiwei, et al. \"Do we actually need dense over-parameterization? in-time over-parameterization in sparse training.\" arXiv preprint arXiv:2102.02887 (2021).", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No. I agree with the authors that the limitations can be good future work directions.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5 hours"}, "extra_text": {}}, {"review_id": "aiRsMl_6Rs", "reviewer": "Reviewer_pwT3", "summary": "This paper presents a simple and novel in-training model pruning algorithm to learn sparse feature representation with reduced computational load w/o compromising accuracy. To do this, authors introduce a single learnable parameter to simplify the optimization involved in enforcing L0 sparsity. Authors further provide theoretic analysis on how reducing the single learnable parameter would enforce L0 sparsity. Finally, authors demonstrate the remarkable performance over various applications comparing with the established strong baselines.    ", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "While authors have theoretically and empirically justified the using a single learnable parameter to achieve L0 sparsity, they only empirically demonstrate the attractive property of permutation invariance via sensitivity analysis.  In addition to the claim “all neurons are symmetrically invariant to each other at initialization and any neuron has an equally good chance of refining themselves to capture useful features later in the training process as any other neuron”, can authors provide analysis why a trivial choice of \\mu_{ij} in line 187 would work? For empirical analysis reported in Table 2, what are the corresponding values of the learnable parameters? Any guidance on the “optimal” sparsity for a given task and data set that would balance the trade-off between accuracy and network size? \n\nIn addition to reducing resource consumption, network pruning is also to improve generalization to new examples, either In-Distribution (ID) or Out-Of-Distribution (OOD), it seems that all the evaluations are based on ID validation sets in a relatively smaller scale (up to CIFAR-100), additional experiments using new larger data sets (e.g., mini-ImageNet or ImageNet) would further demonstrate the advantages of your network pruning approach.     \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4 hours"}, "extra_text": {}}, {"review_id": "_4yMUmILnRd", "reviewer": "Reviewer_5AFY", "summary": "The authors propose a single stage pruning method (DAM) that jointly prunes and refines weights during training. The method uses a monotonically increasing gate function for the neurons/channels in each layer with one trainable parameter. The gate function only discriminates neurons based on the position of them in the layer. The proposed method achieves reasonable results without any fine-tuning.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed method is rather simple, in a positive way, and tries to address an important problem. It does not need fine-tuning which can result in lower training time. The performance of it is evaluated on several applications, like graph representation learning and structured network pruning for image classification, where it shows reasonable performance compared with the baselines. In addition, studies on sensitivity to hyperparameters are provided which are helpful, and detailed experimental setups are included that is a plus.\n\nIn the related work, [24] can be applied as a structured pruning through group sparsity, it also considers the L0 problem and can be single stage and versatile. So, the discussion there can be adjusted as the proposed method is not the first single-stage pruning method considering the L0 problem. [24] can also be used as a baseline.\n\nIn the contributions, it is stated that the proposed method can be applied to both training from scratch and pre-trained setups. Currently, no experiment is done to support the pre-trained setup. It appears that the success of the method is claimed to be based on neurons being symmetrically invariant to each other at initialization. For a pre-trained network, that symmetry may not hold so it is unclear whether the method would work in that case. \n\nIn the recommendation system (where the results are very close) and graph representation examples, it will be helpful to tune the dropout rate or replace the layer with a learnable version like concrete dropout to have a better baseline for comparison to support the paper’s claims. \n\n---After author response---\nI thank the authors for their responses. I still think that for the mentioned experiments having a more comparable and better baseline would be helpful. After reading other reviews and responses, I remain positive about the paper.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5"}, "extra_text": {}}, {"review_id": "LY-E7RkgpS4", "reviewer": "Reviewer_uBCs", "summary": "The paper proposes a single-stage structured pruning method, DiscriminAtive Masking (DAM), which learns a mask to discriminatively identify neurons that needs to be pruned out.", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, the contribution of the paper is not that clear to me. My first impression is that this paper proposes a structured pruning method (as indicated in the abstract) and expect that it will conduct extensive experiments on pruning/compressing various networks on various benchmark datasets and compare the performance (generalization/FLOPS/runtime/size) with other state-of-the-art methods. However, it spends 2-3 pages in describing its applications in dimensionality reduction, recommendation system and graph representation learning, which confused me a lot. Most importantly, due to the limited length of the paper and so many topics it intends to cover, for each of these applications as well as the structured pruning in Section 5, the demonstrated experiments are rather superficial: either only synthetic data is involved in the experiments or the algorithm is only evaluated on toy dataset (MNIST, CIFAR10...). In Section 5 of structured pruning, the competitors are only restricted to [15,16], which is not that convincing to me. \n\nI would recommend concentrating on one direction, namely the structured pruning, and conducting serious experiments on this particular part. For example, it would be better to test different backbones, e.g., VGG/ResNet/DenseNet, on ImageNet, and compare the FLOPS reduction, generalization performance, inference speed-up of the resulting compressed model with many other state-of-the-art methods. This would highlight the main contribution to a much larger extent than the current practice.\n\nIn the same time, I think the paper lacks a basic proofreading since there are so many typos or mistakes:\n\n- line 73, function \"g\" should be bold to be consistent with the notation later?\n- equation 1, the subscript g in \"F_g\" should also be bold; the \"x\" in \"x\\sim ...\" should be bold\n- line 152, \\theta is not only the learnable parameters of the network, since it also optimizes over \\beta. The current formulation is a little bit wired since there is no \\theta appears in the objective function \n- Figure 2, in the caption \"k=... \\beta=...\" should be in math mode when typesetting; in (c), (d), what is the setting for \\alpha and \\mu?\n- line 171-172, it says the learnable \\beta introduces \"a tremendous amount of simplicity\", which is a negative description, however, what it actually wants to convey is that this is a positive aspect of the proposed method\n- line 207, after the bold phrase, there should be a period at the end\n- line 268, after \"Lasso)\" there are two periods\n- line 304, the \"E[\\beta]\" should be in math mode; more importantly, I think think the plot is for the expectation of \\beta, since it is impossible to collect infinitely many of running samples for calculating such expectation, what you actually do here is only the average of \\beta?\n\n\n\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "9kBi_ks6KvU", "reviewer": "Reviewer_VSsj", "summary": "This paper proposes a novel structured pruning method called DiscriminAtive Masking (DAM) for neuron-level sparsity. A relu-tanh gating function is proposed to apply on the neuron activations, and gradually zeroing out the neurons at lower order by introducing a L0 norm. The proposed method is a single stage method, and does not require further after-prune finetuning. Extensive experiments demonstrate the effectiveness of the proposed method", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed DAM method for learning compact representations is novel. Unlike previous method in applying L1 norm as sparsity-inducing regularization followed by further pruning and finetuning, the proposed method applies the L0 norm and does not need additional step of finetuning with the proposed gating function and the neuron ordering. And it does not require combinatorial search, and can be trained with simple gradient descent.\n\nExtensive experiments have been conducted to demonstrate the effectiveness of the proposed method. Specifically, the authors conduct experiments for representation learning problems, including dimensionality reduction, recommendation, graph representation, etc., in addition to the structure network pruning. The results show that even better performance can be achieved with more compact representations with the proposed DAM for recommendation problem. For structure pruning, it is shown to perform better than the state-of-the-art ChipNet. Stability analysis and permutation invariance experiments have also been conducted.\n\nOverall, I think the paper is of high quality, and contain significant novelty.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "- It is understandable that lambda controls the compactness of tthe representations or the pruning sparsity. But it seems that what is the exact relationship between lambda and the resulting compactness is not clear and could depend on the specific dataset. For example, if we want to achieve 20% sparsity, in one dataset lambda needs to be 10.512, but in another dataset it could be 5.123, etc. Is it a right understanding? In practise, how should be find the right lambda if we have an sparsity in mind?", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The submission proposes a method for learning a network with sparse parameters.  The key idea is to optimize an L0 penalized objective (1).  This is non-differentiable, so instead a ReLU is applied to a parametrized tanh scaling (2) which will selectively zero out some fraction of the weights, where the fraction is monotonic in the beta parameter.  This leads to the main result of the paper (3), which explicitly demonstrates this relationship.  I was initially confused by the comment of Reviewer VSsj who stated that an L0 constraint (non-differentiable) could be trained by gradient descent.  One clue is that the "equality" in (3) includes a ceiling function, implicitly showing that there is indeed a relaxation of the L0 constraint, through the tanh function, albeit a different relaxation than is given by e.g. the L1 relaxation.

Overall, a majority of the reviewers feel that this paper is ready to be accepted at NeurIPS.  It is an interesting and clearly presented contribution to learned network sparsity.  Comparison to baseline methods is on the weak side, but overall this paper is acceptable at NeurIPS.
Decision: Accept (Poster)