Only Train Once: A One-Shot Neural Network Training And Pruning Framework
Authors: Tianyi Chen
Bo Ji
Tianyu DING
Biyi Fang
Guanyi Wang
Zhihui Zhu
Luming Liang
Yixin Shi
Sheng Yi
Xiao Tu
Venue: NeurIPS 2021 Poster (2021)
Keywords: model compression
structured pruning
group sparsity

Abstract: Structured pruning is a commonly used technique in deploying deep neural networks (DNNs) onto resource-constrained devices. However, the existing pruning methods are usually heuristic, task-specified, and require an extra fine-tuning procedure. To overcome these limitations, we propose a framework that compresses DNNs into slimmer architectures with competitive performances and significant FLOPs reductions by Only-Train-Once (OTO). OTO contains two key steps: (i) we partition the parameters of DNNs into zero-invariant groups, enabling us to prune zero groups without affecting the output; and (ii) to promote zero groups, we then formulate a structured-sparsity optimization problem, and propose a novel optimization algorithm, Half-Space Stochastic Projected Gradient (HSPG), to solve it, which outperforms the standard proximal methods on group sparsity exploration, and maintains comparable convergence. To demonstrate the effectiveness of OTO, we 
train and compress full models simultaneously from scratch without fine-tuning for inference speedup and parameter reduction, and achieve state-of-the-art results on VGG16 for CIFAR10, ResNet50 for CIFAR10 and Bert for SQuAD and competitive result on ResNet50 for ImageNet. The source code is available at https://github.com/tianyic/only_train_once.

Only Train Once: A One-Shot Neural Network Training And Pruning Framework
Tianyi Chen'
Bo Ji
Microsoft
National University of Singapore jibolcomp. nus edu. sg
Blaccen@@iicrooof..com
Tianyu Ding
Biyi Fang
Guanyi Wang Georgia Institute of Technology
Johns Hopkins University tdingle @ jhu.ce. edu
Microsoft
biffmmicrosoft.com
gwang93@gatech.co edu
Zhihui Zhu
Luming Liang
Yixin Shi Microsoft
University of Denver zh zhihui. wet.hi..uuuuaeee www@du.eeeeoooooooooo edu
Microsoft
luli anêmi crosoft. www.ianmicrooot.ccmmmm com
y rixshi @mi crosoft.com com
Sheng Yi
Xiao Tu Microsoft
Microsoft
shengyi@microsoft.com
xiaotu@microsoft.ccm com
Abstract
Structured pruning is commonly used technique in deploying deep neural net- works (DNNs) onto resource-cosstrained devices. However, the existing pruning methods are usually heuristic, task-specified, and require extra fine-tuning pro- cedure. To overcome these limitations, we propose framework that compresses DNNs into slimmer architectures with competitive performances and significant FLOPs reductions by Only- Only-Train-O Train-Once (OTO). OTO contains two keys: (i) we partition the parameters of DNNs into zero-invariant groups, enabling us prune zero groups without affecting the output; and (it) to promote zero groups, we then formulate structured- structuredsparrsity optimization problem and propose novel opti- mization algorithm, Half-Space Stochastic Projected Gradient (HSPG), to solve it, which outperforms the standard proximal methods group sparsity explo- ration and maintains comparable convergence. To demonstrate the effectiveness of OTO, we train and compress full models simultaneously from scratch without fine- tuning for inference speedup and parameter reduction, and achieve state-of-the-art results on VGG16 for CIFAR10, ResNet50 for CIFAR10 and Bert for SQuAD and competitive result on ResNet50 for ImageNet. The source code is available https:://// / com/ t aaaanic/oolly train oncee.
Introduction
Deep neural networks (DNNs) have been shown be effective in various real applications (51; 28). is widely acknowledged that large-scale DNN models not only learn faster but also outperform their slimmer counterparts. However, such heavy models pose great challenge to the deployment stage due to their resource-consuming nature. In addressing this issue, many model compression
Cooresspodiing author.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Partition Full
Structured- Sparsity Inducing
Zero Group
Filter,
bias
Model Into Zero- Invariant
ZIG,
ZIG;
Pruning Without
ZIG
Layer,
Optimization (HSPG)
Retrain
Groups (
Filter,
bias.ii
ZIG
ZIG
Compressed Model
Figure I: Overview of OTO. Without loss of generality, illustrate OTO on a model with only vanilla convolutional layers, and for simplicity we only show Layer; with m 3D filters and their biases. The key its success is twofold: (i) identify and partition the parameters of the model into zero-invariant groups (ZIGs); and (ti) solve the structured-ppariie regularization problem using HSPG. Finally, we obtain the compressed model by directly pruning the zero groups, i.e., ZIG
techniques (5: 11) are proposed in the past decade that aim at compressing those large and complex models into slimmer and simpler ones while suffering negligible loss in performance.
Pruning methods as one of the main categories of model compression, focus on identifying and pruning redundant structures via various mechanisms to achieve a slimmer architecture, and thus improve the interpretability of a DNN model (26; 11; 65). For example, (32; 33) adopt fine-grained pruning via f or l regularization, which prune the small-weight connections based on some hard threshold. (36; 57; 60) measure the importance of filters to accelerate the networks by removing insignificant feature maps. (37; 7) utilize reinforcement learning agent to predict compression action.
Nevertheless, many of the existing pruning methods (i) often rely on criteria based on heuristics or empirical cues, e-g., magnitude of connection weight and importance score of filter, to identify redundant parameters, which may cause divergence during optimization; (iz) thus require complex multi-stage pipelines that involve either a retraining or fine-tuning procedure regain the accuracy during constructing a slimmer model, which is time-consuming: and (zii are specific to certain architectures or applications, and are consequently less applicable various downstream scenarios. Recently, there have been a few efforts (14; 58; 8) to directly train the network with sparsity inducing regularizers, which provide generality and convergence guarantee. However, these approaches focus on either merely the individual sparsity of the parameters or the group sparsity of the filters, and thus cannot directly remove those zero components (still require subsequent fine-tuning) since the zeros are entangled with other commonly used components, e.g., bias, batch normalization or skip connection. Furthermore, the optimization algorithms used in (14; 58) lack sufficient capability to explore (group) sparsity in DNNs effectively and require a post-processing step to yield exact zeros.
In this paper, we overcome the above limitations of existing pruning methods by proposing a one-shot neural network pruning framework, with which we are able to train a full heavy model from scratch only once, and obtain a slim architecture without fine-tuning while maintain high performance. As shown in Figure 1, the key to its success is twofold: (i) we identify and partition the parameters of DNNs into zero-invariant groups (ZIGs), enabling us to prune redundant structures according to zero groups without affecting the output of the network; and (ii) to promote zero groups, we formulate the pruning task as structured-spant -spasssty optimization problem and propose a novel optimization method, Half-Space Stochastic Projected Gradient (HSPG), to solve it, which outperforms the standard proximal methods on sparsity exploration and maintains comparable convergence. We highlight that both zero-invariant group partition and the novel optimization algorithm in promoting zero group lead to achieve one-shot neural network training and pruning regardless of its architecture.
Our main contributions are summarized as follows.
One-Shot Training and Pruning. We propose OTO, training and pruning framework that compresses a full neural network with competitive performance by Only-Train-Omen thereby one-shot. OTO dramatically simplifies the complex multi-stage training pipelines of the existing pruning approaches, fits various architectures and applications, and hence is generic and efficient.
Group. We define vero-invarian groups for neural networks. If network is partitioned into ZIGs, it allows us to prune the zero groups without affecting the output, which results in one-shot pruning. Such property is applicable to various popular structures from plain fully connected layers to sophisticated ones such as residual blocks and multi-head attention.
Novel Seneturrd-SSaariin Optimization Algorithm. We propose Half-Space Stochastic Pro- jected Gradient (HSPG), a method that solves structured-sparsity inducing regularization problem. We show and analyze the superiority of HSPG in promoting zero groups of networks than the

---

standard proximal methods and the competitive objective convergence in practice. The fact that ZIG and HSPG are designed agnostic networks makes OTO generic to various applications.
Experimental Results. We train and compress full models simultaneously from scratch without fine-tuning for inference speedup and parameter reduction, and achieve state-of-the-art results on compression benchmark VGG for CIFARIO, ResNet50 for CIFAR Io/mmeeNet,, Bert for SQuAD.
2
Related Work
Structured pruning focuses on identifying and pruning the redundant structures full model to achieve slimmer architectures for efficient model inference and storage (26; 32), where there have been numerous efforts dedicated. For CNN compression, the general procedure can be largely summarized as: (i) train full model; (ii) identify and prune the redundant structures to build slimmer model based on various criteria, including (structured) sparsity (58; 85; 14; 56; 102; 27; 102; 62; 91), Bayesian pruning (101; 65; 59; 81), ranking importance (54; 60; 41; 36; 57; 100), reinforcement learning (37; 7), adversarial robustness (76), scientific control (79), lottery ticket (23; 24; 72), joint quantization learning (80; 90), etc.; (iii) retrain iteratively fine-tune the slimmer model to regain the accuracy regression during pruning. These methods cannot avoid the extra and usually time- consuming fine-tuning step because the identified redundant structures, even parametrized with zeros, actually contribute to the model output, thereby additional fine-tuning step is an absolute necessity.
For pruning Bert (82), knowledge distillation (40) and LayerDropout (21) shorten Bert by reducing the number of layers directly. Other methods (29; 75; 30) build slimmer Berts in the manner of individual sparsity, but require specially designed data structure for storage and computing library to take advantage of sparse data (31; 10), and typically cannot achieve inference speedup against the highly optimized library (16) for dense model due to the discontiguous memory allocation (9).
The structured sparsity for weight pruning the most relevant to the algorithm described this paper. The existing structure learning works (58; 85; 14; 56; 102) have the respective disadvantages: (i) multiple trainings during the whole procedure since their group partition cannot isolate the impact of pruned structures to the model output; and (ii) heuristic post-processing to generate zero groups as the standard proximal methods (19; 87; 88; 12) and ADMM (100; 58; 4) defective on the sparsity exploration for deep learning (8), which may deteriorate the performance of the model significantly.
Avoiding fine-tuning step during the whole pruning procedure receiving more and more attentions because of its efficiency. In particular, SNIP (52) and GraSP (83) identify redundancy via salience scores at the initialization stage to construct pruned structures, then train the pruned models by the standard optimizers. SCP (48) isolates the impact of batch normalization, while lacks the consideration of more general DNN architectures.
3
OTO
In essence, OTO frames the network training and pruning as a structure learning problem. Given a full model M, OTO trains and compresses it simultaneously from scratch without fine-tuning, and achieves significant reduction in both FLOPPs and parameters. Particularly, as stated in Algorithm 1, the trainable parameters of M are firstly partitioned into a ZIG set & (Section 3. We then construct and solve a diructured-sparsit inducing optimization problem (Section 3.2) by proposed stochastic optimizer (HSPG) seek a highly group-sparse solution æiseo (Section 3.3). Lastly, we obtain compressed model M by directly pruning these zero groups (Section 3.4).
Algorithm Outline of OTO.
1: Input: Full model M (no need to be pretrained).
2: Construct G: Partition the trainable parameters of M into ZIG set G.
3: Train: Train the model M using HSPG (Algorithm. 2) to obtain a group-sparse solution øjisG
4: Prune: Construct slimmer model architecture M by directly pruning groups of æfssG:
5: Output: Compressed model M
3

---

3.1 Zero-Invariant Group
The root cause of the existing methods having multi-stage training pipeline that despite the pruned structure (e.g., 3D filter) being zeros, its associated structures (e.g., non-zerr con- bias) still contribute its corresponding output the next layer (e.g., feature map). As a result, the model aceuracy regresses, hence an extra step of fine-tuning is necessary. OTO avoids the necessity by partitioning the parameters DNNs into a set so-called zero-invariant groups (ZIGs) & defined as follows.
Definition (Zero-Invarian Invariant Groups (ZIGs)). For layer-wise DNN, partition its entire trainable parameters into disjoint groups = {0}- Then we call G zero-invariant groups (ZIGs) if each group g 9 is zero-invariant in the sense that all of the parameters in g being zeros results in its corresponding output to the next layer to be zeros as well.
In effect, and only if DNN model partitioned into ZIG set and one or more of its element are parameterized by zeros, the entire corresponding structures contribute none to the model outputs and hence can be pruned directly. Such partition applicable to various structures of DNN models. Without loss of generality, we define and describe ZIG partition for three most popular structures: (i) Conv-BN. (ii) Residual Block, and (iii) Fully Connected and Multi Multi-Head Attention Layer
b
9
Activation
9/.
Pool BN
I
g
(ot 22.
Conv-BN. denotes the number of channels in O².
6
72 3
.3a
Conv2
BN2
91
92
Conv1
Conv3
Om
G
(g1, 92.
". Im
(b) Residual block. denotes the number of output channels of the residual block.
W
W
b
Uie
91
9a
9m
Agg
G (01- 92. gs}
G = (gr
A. the 7th head
(c) Fully connected layer (Left), Multi-head attention layer (Right). m denotes the length of output vector.
Figure 2: Zero-invariant group partition for three popular structures.
ZIG of Conv-BN. Convolutional layer (Conv) followed by batch-normalization layer (BN) is exten- sively used DNN models. Figure 2a shows the ZIG partition for Conv-BN. The 4D filter tensor is flattened into filter matrix Æ During the forward pass, the input tensor I' is transformed into the output tensor O of Conv and then into the input tensor of the (1 (1+ 1)th layer I++1 by
a(0') o¹
μ 3 o +
O
+ I
(1)
where denoted by X the convolutional operation, O the deement-wise multiplication and a(-) the activation function. BN is parameterized by mean pt, standard deviation σ¹ weight y and bias 3 respectively. The activation function needs to be zero-invariant, i.e., a00) = O. where most instances satisfy, e.g., ReLU (25), PReLU (34), GELU (39) and LeakyReLU (89). Hence, each row of flattened filter matrix K and its bias b' belong to one ZIG because they being zeros results in their corresponding channel of O (i.e., feature map) to be zeros as well. Subsequently, A and B of this corresponding channel in BN are also included into this ZIG to avoid the value shift (zero to

---

non-zero) during normalization. Note that grouping these four sets of parameters channel -wisely makes Conv-BN vero-invariant regardless of the value of 2e and o', and hence they are excluded from the ZIG. For illustration, each ZIG is highlighted in the same color (e.g- gi in blue).
ZIG of Residual Block. The residual block adds another layer of challenge because its output tensor is the summation of the outputs of two Conv-BNs. Figure 2b shows the ZIG partition for the residual block. As illustrated, before propagated Conv3, the outputs of I-BVI and Conv2-BN2 are summarized and hence share the same dimension. As such, to make residual block zero-invariant, we group the four sets of parameters i.e., each row of R², b. b. x', 3' of Conv I-BNI and each row of R², b² 7², 3 of Conv2-BN2, In Appendix A. we describe the zero-invariant group partition of ResNet50 in greater detail.
-eaanel-wisely of both C 1-BN1 and Conv2-BN2 into ZIGs,
ZIG of Fully Connected and Multi- Attention Layer. Figure 2c shows the ZIG partition for fully connected and multi-head attention layer. Partieularly, we partition each row of weight matrix and its associated bias into a ZIG, and therefore any input element is turned to zero if that ZIG is parameterized with zeros, making the fully connected layer zero-invariant. Multi-head attention layer is the key building block of the transformer architectures (82). Its trainable parameters contain a weight matrix and bias vector, consisting of the sub-matrix and sub-vector of each head (we use two heads as an example). We form ZIG by grouping each row of every sub-matrix and sub-vector, i.e., each row of Wk1 bh, Wna and o of h2 and hy, respectively.
Automatic ZIG Partition. Based on the above illustrating examples, we provide prescribed ZIG partition for the tested DNNs in Section 4. Furthermore, given an arbitrary DNN architecture, the procedure of partitioning variables into ZIGs could be automatically proceeded, wherein the key would be identifying the connections among various layers, then performing corresponding group partition. We will leave the automatic ZIG partition for arbitrary DNNs as future work.
3.2 Structured-Sparsity Regularization
We now formulate a structured-spariity regularization problem over the ZIG set & for the trainable parameters of the full model M as follows
minimize (xx) = f(x) + Ar(x), r(x)= = Σ || *ER"
(2)
gEG
where À >> 0 a weighting coefficient, (ax) is a tpsk-specific loss function, and r(x:) is an augmented structured-sparsity inducing regularization term encoding the topological structure of M over g. larger À typically results in a higher group sparsity while sacrifices more on the bias of model estimation. We aim at computing a local optimum to achieve both low loss and high group sparsity.
To induce group sparsity onto the solution of (2), there exist several candidates for r(æ), including mixed fi/Ep norm (p >>> 1) (1; 20) and group Minmax Concave Penalty (MCP) (96). Among these candidates, the mixed l norm as defined in (2) is arguably the most popular choice in classical machine learning applications (1; 92), where ||-I| is the fg-norm, and each component g E G indexes a group of variables. In this paper, we will demonstrate the effectiveness of OTO by selecting r(x) as the mixed / norm. We highlight OTO is applicable for other group sparsity regularizers as well.
3.3 Half-Space Stochastic Projected Gradient (HSPG)
To solve the non-smooth regularization problem as (2) in deep learning applications, the standard proximal method and the ADMM lack capability to effectively identify group sparsity; see the discussions later in this Section. Therefore, we propose a novel stochastic optimization algorithm so-called Half-Space Stochastic Projected Gradient (HSPG) to enhance the group sparsity exploration more effectively than the classical methods while maintain a similar convergence property.
Outline. We state the outline of HSPG in Algorithm 2. It contains two stages: Initialization Stage and Group-Sparsity Stage. The first Initialization Stage employs Stochastic Gradient Descent (SGD) step to search for good but usually non-sparse solution estimate. Then the second stage proceeds Half-Space step started with the non-sparse iterate to effectively exploit the group sparsity within a sequence of reduced spaces and converges to the group-sparse solutions. Half-Space step performs SGD update on free non-zero variables along with a novel projection operator so-called Half-Space Projection, which significantly outperforms the standard proximal operators sparsity exploration.
5

---

Initialization Stage. The Initialization Stage performs the vanilla SGD to find good initial point for the subsequent Croup--Sprsiiy Stage. At km iteration, a stochastic gradient of e.g., based on mini-batch generated denoted as V.. Since the group sparsity inducing regularizer r(x:) in the form as (2) is non-smooth, we select subgradient ((xx) from its subdifferential Ör 0r(zx) to form stochastic subgradient of (xx)) as v((xs)) == fa + Aς(xx). We then compute the next iterate as Øk+1 1- == zy (xv(2x) ) by subgradient descent update.
2
S
Pross SG Prow-SV Spide SAGA
[a]
HSPG e (0,
90
RDA
Tetl
c3c
O
[x]]
Ax
Øx+1
[
(a) Half-Space Projection
Projection Region For Mixed f Regularization
Figure Illustration of Half Half-Space Step with projection in (6), where G {{1, 2)}-
Group-Sparsity Stage. The Group-Sparsity Stage is designed to effectively determine the groups of zero variables and capitalize convergence characteristic, which is in sharp contrast to other heuristic aggressive weight pruning methods that typically lack theoretical guarantees (55; 60). The intuition of Half-Space Step is to project [æ to zero only if - serves
descent step to (xxx), i.e.,
- [ [vb((x)))
(xx)){ ))g < O, hence updating [1g+11g + [2*]] [ = 0 still results in some progress the optimality. particular, we first define the following index sets for any x: E R":
(x) = (g: g G, [ = 0} and I (x) = (g E 9, [2]g * 0),
(3)
where (x: represents the indices of groups of zero variables at at, and (=) indexes the groups of nonzero variables æ. To proceed, we further define an artificial set that lies
S(x) = (0) U(=E I [2]g 0 if E (xx), and [=]] [a]g ||(>]]|| if (x)), (4)
which consists of half-spaces and the origin. Here the parameter 2 0 controls how aggressively we promote group sparsity, and typically fixed as zero in practice. Hence, S& S(æx) only if: [o, 1) shown in
(i) [zly lies in the upper half-space for all g E Ixx) for some prescribed Figure 3a; and (ii) (x) equals to zero for all y E I (mxx) Intuitively, S₁ establishes the region where important structures inhabit, thereby redundant structures vanish if falling outside.
Ideally, the Initialization Stage has produced reasonably well but typi- cally non-sparse iterate 24 nearby a group-sparse solution x' of prob- lem (2), i.e., the optimal distance
Algorithm Outline of HSPG for solving (2).
1: Input: 3g E R", ay > 0, c E [0, 1), and E Z
2: Output: group-sparse solution ZHseG from {xx).
3: for l = 0, 1, do
||e - x=1 is sufficiently small. As seen in Appendix B, it further indi- cates that the group-sparse optimal solution x" inhabits Sg. and S₁ has already covered the group-support of x², Le.. I#7(x") C I (Ix). (**).. Our goal now becomes minimizing (xx) over S₃ to identify the remaining zero groups, ie.,, (x ) / (æx). which is formulated the following problem:
4: Compute stochastic subgradient v(xx)) of ((x)).
5:
if < N then
6:
Subgradient Descent Update:
7:
Set 0&+1 + 26. (x)(
8:
else
9:
Half-Space Update:
10:
Set trial iterate T as
| 12 [ (eLu((xx)l))))) (
0.
minimize (xx) = (=) + (æ)).. (5) zES
11:
for each group in 9 do
12:
[#x+1lg + [Proj HS
(#x+1))g
The next iterate 24+1 is computed as an solution estimate of problem (5).
13:
Update 0'++1

---

Particularly, in Algorithm 2, (#++1l77(mz)) O will not be updated, and only the entries in (xxa) are free to move. Hence (x) is smooth on Sj. and (5) is a reduced space optimization problem. A standard way to solve problem (5) would be the stochastic gradient descent equipped with Euclidean projection (68). However, such a projected method rarely produces zero (group) variables, as the dense Euclidean projected point &g A illustrated in Figure 3a. To address, we introduce a novel half-space projection operator to effectively project an entire group of variables to zeros.
As line 4 and 9-12 in Algorithm 2, first approximate the (sub)gradient of 1b on the free variables by
then employ gradient descent over Z=*(ax to compute a trial point 2k41 which is passed into fresh half-space projection operator Projs,
H.8 (-) defined
HS (=)] { O if [ [ < c [< Proi]
(6)
[
otherwise.
The above projector of form (6) is not the standard one in Euclidean sense", and it has two advantages:
(i) the actual search direction dg == (Projš. H. (#x+1) z*)/ag performs as a descent direction to (xx)),, i.e., [dxlg y [)( < 0 as 0 < 90 in Figure 3a, hence the progress to the optimum is made via the sufficient decrease property drawn Lemma 1 in Appendix B; then (ii) it effectively projects entire groups of variables to zero if the inner product of corresponding entries is sufficiently small. In contrast, the Euclidean projection operator is far away effective to promote group sparsity.
Superiority of HSPG on Group Sparsity Identification. We now intuitively illustrate the strength of HSPG on group sparsity exploration. In fact, the half-space projection (6) is more effective sparsity promotion mechanism compared to the standard proximal methods. Particularly, it benefits from a much larger projection region to map a reference point Φx+1 := 3& ag Vf())) or its variants zero. As the 2D case described in Figure 3b, the projection regions of the state-of-the-an art Prox-SG (19), Prox-SVRE (88), Prox-Spider (97) and SAGA (12) for (2) are fz-balls with radius as ag In deep learning applications, the step size Or is usually selected around 10- to 10-4 or even smaller for convergence. Together with the common setting of X < 1, their projection regions would vanish rapidly, resulting in the difficulties to produce group sparsity. As sharp contrast, even though 0g.A is near zero, the projection region of HSPG {ar x x < (ax.) + e ||axe]] I) || | (seen in Appendix B) is still an open half-space which contains those E2 balls as well RDA (87)'s if is large enough. Conversely, vanilla ADMM alone lacks the mechanism to project a group of variables to zero, unless equips with extra post-procsssing step (100; 58). In Appendix B, we further reveal that HSPG still maintains the convergence to the optimality as drawn in Theorem I. Moreover, we numerically demonstrate the superiority of HSPG in the sense of optimization in Appendix C.
3.4 Pruning Without Fine-Tuning
The group-sparse solution THSPG over ZIGs to the full model M is leveraged to construct the slimmer model M Particularly, we prune the redundant structures identified as zero groups Io and retain non-zero groups I in T Because the parameters of full model are partitioned into ZIGs, the pruned structures contribute none the model output. Therefore, given the same input, the slimmer model M computes the identical output as the full model M parameterized with #HSPG-
4 Experiment
In this section, we numerically demonstrate the effectiveness of OTO by one-shot training and pruning without fine-tuning on several benchmark compression tasks for CNNs, i.e., VGG16 (77) for CIFAR1O (49) and ResNet50 (35) for CIFARIO (49) and ImagetNet (ILSVRC2012) (15). We also verify the scalibility of OTO onto Bert (82) evaluated on SQuAD (69). All datasets are free to academic usage and do not contain personally identifiable information or offensive content. CIFARIO is under the MIT license, consisting of 50,000 training and 10,000 test images from 10 classes. ImagetNet is large-scale dataset without license and contains about 1.2 million and 50,000 images in training and validation sets from 1,000 classes. SQuAD is under the CC BY-SA 4.0 license with about 100,000 question/answer pairs splitted into train/dev/test sets as (80/10/10%) We conduct all experiments Nvidia RTX8000 GPU and provide implementation details in Appendix A.
Note that when r(z) = ||0||| where each g E 9 is singleton, then St becomes an orthant face (8).
7

---

Table 1: VGG16 and VGG16-BN for CIFARIO. Convolutional layers are in bold.
Method
BN
Architecture
FLOPs
of Params
Top- Acc.
Baseline
X
100%
100%
91.6%
SBP (65)
77510911152221110066622122222222
31.1%
5.9%
91.0%
BC (59)
X
38.5%
5.4%
91.0%
RBC (101)
X
32.3%
3.9%
90.5%
RBP (101)
28.6%
2.6%
91.0%
OTO
16.3%
2.5%
91.0%
Baseline
6446--2-4122226-2222222211111111
100%
100%
93.2%
EC (55)
65.8%
37.0%%
93. 1%
Hinge (56)
60.9%
20.0%
93.6%
SCP (48)
33.8%
7.0%
93.8%
OTO
22-56-93-1123-18--222228
26.8%
5,5%
93.3%
4.1 Deep Convolutional Neural Network
The results on CNN experiments are summarized in Table 1, 2 and 4. In particular, we compare OTO to its state-of-the- -ortte--rttttttt-- counterparts by Top-1/5 accuracy, remaining FLOPs and parameters against the corresponding baseline (full model). We report the numbers of other methods based on the corresponding literature and leave as --' if not reported. The best pruning results are marked bold.
VGG16 for CIFAR1O. We consider the standard VGG16 and the version with batch normalization layer after each convolutional layer, referred to as VGGI6-BN. OTO partitions the parameters into ZIGs following Section 3.1, then trains and prunes the model via HSPG, and finally constructs the slimmer model without fine-tuning, For VGG16, as shown in Table I, the pruned architecture of OTO indicates that OTO identifies similar redundancy of the intermediate and late convolutional layers compared to other methods, but significantly more of the early convolutional layers. As result, ОTO achieves 83.7% (1 16.3%) FLOPs reduction and 97.5% (1 2.5%) parameter reduction with the best Top-1 accuracy, which outperforms other state-of-the-art significantly. For VGGI6-BN, among all, OTO reduces FLOPs and parameters to the lowest 26.8% and 5.5%, respectively. EC (55) and Hinge (56) achieve the same level of Top-1 accuracy OTO, but are substantially outperformed when it comes to FLOPs and parameter reduction. We further present the FLOPs reductions per layer of OTO Table 7 of Appendix A.4.
ResNet50 for CIFAR1O. Since OTO is able to au- tomatically learn slimmer model of high perfor-
Table 2: ResNet50 for CIFARIO.
Method
FLOPs 100%
# of Params
Top-1 Acc.
mance, we compare it with two state-of-the-ae au-
Baseline
100%
93.5%
tomatic neural network compression frameworks,
AMC (37)
60.0%
93.6%
ie., AMC (37) and ANNC (90). AMC trains a re-
ANNC (90)
50.0%
95.0%
inforcement learning agent predict compression action for each layer environment. ANNC jointly proceeds pruning and quantization within energy constraint. We conduct OTO on their shared experiment, i.e., ResNet50 on CIFARIO. ResNet50 includes both the standard convolutional layers and the layers with residual connections, which are partitioned into ZIGs following Section 3.1. We report the results in Table 2 along with other competi- tors from (61; 78). Based on the results, all methods achieve competitive validation accuracies, where most of them are even higher than the baseline reported in (37). OTO outperforms AMC, ANNC without quantization, PruneTrain and N2NSkip by using only 12.8% FLOPs and 8.8% parameters. Note that no FLOPs reduction is reported in (37) and (90). Finally, we highlight that OTO is flexible to incorporate quantization as the two techniques are complementary and will leave to future work.
Prunc (61)
30.0%
93.1
N2NSkip (78)
10.0%
94.4%
OTO 12.8%
8.8%
94.4%
Ablation Study on Switching Pa- rameter N. We provide ablation study regarding the impact the switch (parameterized as N) between the initialization stage and the group- sparsity stage in Algorithm 1. In the- ory, as shown in Theorem 1 of Ap- pendix B.4, the projection stage should start when the iterate falls nearby a group sparse local minimizer. In practice, we relax it to start the group sparsity stage once the iterate falling into some stationary status regarding the validation accuracy. As described Appendix A.2, throughout all experiments, periodically decay the learning rate per fixed number of epochs parameterized as T.
Table 3: OTO Under Different Switchings (N = T, T,2T,33 2T) for VGG16, VGG1 6-BN and ResNet50 on CIFARI0
Backend VGG16 VGG16-BN
FLOPs
# of Params
Top- 1 Acc.
17.0% + 1.4%
2.6% + 0.4%
90.9% 1 0.3%
25.4% + 1.1%
5.0% + 0.5%
93.3% 1 0.2%
ResNet50
12.9% + 1.5%
8.5% + 1.0%
94.2% + 0.2%
8

---

At the end of each I epochs, we then proceed a statistical test similar to (98) but on the validation accuracy and find that the validation accuracy falls into stationarity near the late epochs of each period. Therefore, in our pruning experiments, we switch to the group-sparsity stage right after the first 7 epochs. Table describes the performance of OTO under varying switching parameters, from which we observe that OTO is not largely sensitive to the switching parameter if the group-sparsity stage starts after some stationary condition has been numerically satisfied.
ResNet50 for ImageNet. We now evaluate OTO on ResNet50 for ImageNet. As shown in Ta- ble 4, OTO prunes 64.5%(( 35.5%) parameters to achieve 65.5%(1-33 (1-4.5%)) FLOPs reduc- tion with only 1.4%/0.8% Top- 1/5 accuracy regression com- pared to the baseline. OTO con- sistently outperforms the major- ity of counterparts especially the FLOPs reduction and the parameter reduction. We note that Hinge (56) prunes CNNs via structured-sparsity, optimization by employing standard stochas- tie proximal gradient method. It requires several trainings including fine-tuning the pruned model, because it partitions the parameters into non-ZIGs and relies on an empirical truncation mechanism generate zero groups due the weakness of proximal operator in deep learning applications (8). In contrast, OTO only trains and prunes the full model from scratch once and obtains better pruning results. The comparison between OTO and Hinge stand as evidence of the superiority of OTO due to ZIGs and HSPG. Furthermore, if with more training efforts, OTO reaches higher Top- 1/5 accuracy marked as in Table and becomes more competitive to stronger competitors, such as GBN (94), Group-HS (91) and ResRep (48).
Table ResNer50 for ImageNet.
Method
PLOPs
# of Params
Top-I Acc.
Top-5 Acc.
Baseline
100%
100%
76. 1%
92.9%
DDS-26 (43)
57.0%
61.2%
71.8%
91.9%
CP (38)
66.7%
72.3%
90.8%
ThiNet-50 (45)
44.2%
48.3%
71.0%
90.0%
RBP (101)
43.5%
48.0%
71.1%
90.0%
RRBP (101)
45.4%
73.0%
91.0%
SFP (36)
41.8%
74.6%
92.1%
Hinge (56)
46.6%
74.7%
GBN-50 (94)
44.9%
46.6%
75.2%
92,4%
GBN-60 (94)
59.5%
68.2%
76.2%
92.8%
Group-HS (2e-5) (91)
32.4%
75.2%
92.5%
Group-HS (le-5) (91)
52.9%
76.4%
93.1%
ResRep (18)
45,5%
76.2%
92.9%
SCP (48)
45.7%
74.2%
92.0%
ОТO
34.5%
35.5%
74.7%
92.1%
OTO* 34,5%
35.5%
75.1%
92.5%
Representation of Deep Features of ImageNet. It is widely acknowledged that deep neural ar- chitectures could be treated as non-linear feature representation extractors. Therefore, we further study the feature representation extracted by OTO to demonstrate its generalizability to other visual applications besides image classification. Figure 4 shows the clustering results of ImageNet validation images using the deep feature extracted by both the baseline ResNet50 and the pruned ResNet50 by OTO. Specifically, we extract the deep features over the validation samples in ImageNet, i.e. the tensors fed into the fully connected layer, and project them onto a 2-dimensional space via PCA (47). For illustration, following the hierarchy of ImageNet (3), two sets of five classes are randomly selected We observe that the deep features of the pruned ResNet50 by OTO remain structured the sense that distinct classes are well separated from each other. Over all 1000-class ImageNet validation images, OTO achieves 48.2% clustering accuracy compared to 42.5% of the baseline ResNet50 using k-means. Both observations indicate that with only 35.5% parameters and 34.5% FLOPs, the pruned ResNet50 still able to extract highly discriminative deep features. We argue that during model compression, OTO not only achieves parameter and FLOPs reduction, but also preserves the ability of capturing perceptual properties (99). This is especially important in training and compressing models for many vision tasks, e.g., object detection (70; 71), frame interpolation (2; 17; 67) and video synthesis (84; 50). We leave the application of OTO to broader tasks to future work.
4.2 Large-Scale Transformer
We show the scalability of OTO by pruning the large-scale transformer Bert (82), evaluated on SQuAD, question-answe answering benchmark (69). Bert mainly includes embedding layers, fully connected layers and multi-head attention layers. The fully connected layers and the multi-head attention layers are partitioned into ZIGs following Section 1.1. For fair comparisons, we follow the prior Bert compression works (14; 75) and do not prune the embedding layers.
Each selected class belongs disjoint upper category.

---

gar
brambling
(h)
Figure 4: Clustering results of ImageNet validation images using deep features extracted by full ResNet50 (left of and b) and pruned ResetNet50 by OTO (right of a and b). The points are visualized by projecting deep features onto two-dimensional space via PCA.
To the best of our knowledge, OTO the first work that compresses Bert by exploring group sparsity on individual layers and achieves sig- nificant parameter reduction and in- ference speedup". In contrast, the existing works (29; 75; 30) prune individual parameters instead, i.e,, the generated sparsity is not struc- tured. Hence, the computed mod- els typically do not have inference speedup (75), unless are executed by specialized hardware and sparse computing library (31; 10). As shown in Table 5, under different group sparsity upper bound constraints, OTO reduces 9% to 60% parameters and achieves up to 1.8x inference speedup based on the average model execution time 5. In comparison, despite that the pruned model contains 10% parameters, MaP and MvP (75) do not have any inference speedup. On the other hand, the structured sparsity on Bert is studied in (14) (referred as ProxSSI), where an adaptive proximal method is proposed to yield group-sparse solution. Nonetheless, ProxSSI optimizes over non-ZIGs and relies on proximal operator to identify group sparsity. Therefore, the groups even parameterized with zeros have be retained in the model rather than pruned. As consequence, ProxSSI is not competitive to OTO on parameter reduction. and there no reported inference speedup. Note that all the pruning methods achieve comparable exact match rate and Fl Fl-score.
Table 5: Pruning Bert on SQuAD
Method
of Params
Exact
Fl Fc-sc
SpeedUp
Baseline
100%
81.0%
88.3%
MaP (75)
10.0% %
67.7%
78.5%
1x*
MvP (75)
10.0 %
71.9%
81.7%
1x*
ProxSSI (14)
83.4%
72.3%
82.0%
1x
OTO
91.0%
75.0%
84.1 %
1.1x
OTO
76.2%
72.3%
82.1 1%
1.2x
OTO
66.7%
71.9%
82.0%
1.3x
OTO
53.3%
71.4%
81.5%
1.5x
OTO
40.0%
70.9%
81.1%
1.8x
Based on statement in official git repository of (75).
Approximate value based on the group sparsity reported in (14).
5 Conclusion And Future Work
We propose OTO, a one-shot deep neural networks (DNNs) training and pruning framework, that compresses full DNNs into slimmer architectures with competitive performances and significant FLOPs and parameter reduction without fine-tuning. OTO contains two fundamentals: (i) partitions the trainable parameters of DNNs into zero-invariant groups (ZIGs), thereby pruning zero groups does not affect the model output, and (ii) trains by a novel optimizer, Half-Space Stochastic Projected Gradient (HSPG), which outperforms proximal methods group sparsity exploration and maintains comparable convergence. We numerically demonstrate OTO on benchmark experiments, i.e., VGG16 for 10, ResNet50 for CIFAR art pruning results. We leave automatically generating ZIGs for arbitrary DNNs, incorporating quantization and applying OTO to other tasks to future work.
and Bert for SQuAD, and achieve state-of-the-
'Knowledge distillation (40) and LayerDropout (21) compresses Bert by pruning layers in their entirety. Runn by OnnxRuntime (16)
10

---

References
[1] Francis Bach, Rodolphe Jenatton, Julien Mairal, Guillaume Obozinski, et al. Structured sparsity through convex optimization. Statistical Science, 27(4) 468, 2012.
[2] Wenbo Bao, Wei-Sheng Lai, Chao Ma, Xiaoyun Zhang, Zhiyong Gao, and Ming-Hsuan Yang. Depth-aware video frame interpolation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3703-3712, 2019.
[3] Mike Bostock. Imagenet hierarchy.
com/ @mbostock/
imagenet- wwweettthierrcchhh
[4] Stephen Boyd. Neal Parikh, and Eric Chu. Distributed optimization and statistical learning via the alternating direction method of multipliers. Now Publishers Inc, 2011.
[5] Cristian Buciluă, Rich Caruana, and Alexandru Niculescu- Mizil. Model compression. In Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 535-541, 2006.
[6] Chih-Chung Chang and Chih-Jen Lin. Libsvm: A library for support vector machines. ACM transactions on intelligent systems and technology (TIST), 2(3): 2011.
[7] Jianda Chen, Shangyu Chen, and Sinno Jialin Pan. Storage efficient and dynamic flexible runtime channel pruning via deep reinforcement learning. 2019.
[8] Tianyi Chen, Tianyu Ding. Bo Ji, Guanyi Wang, Yixin Shi, Sheng Yi, Xiao Tu, and Zhihui Zhu. Orthant based proximal stochastic gradient method for ell_I-regularized optimization. arXiv preprint arXiv:2004. 03639,, 2020.
[9] Tianyi Chen, Yixin Shi, and Sheng Yi. Spatially sparse convolutional neural networks for inking applications, Sept. 17 2020. US Patent App. 16/355,702.
[10] Xuhao Chen. Escoin: Efficient sparse convolutional neural network inference on gpus. arXiv preprint arxiv:1802.11288 10280. 2018.
[11] Yu Cheng, Duo Wang. Pan Zhou, and Tao Zhang. survey of model compression and 2017.
acceleration for deep neural networks. arXiv preprint
[12] Aaron Defazio, Francis Bach, and Simon Lacoste-Julien. Saga: A fast incremental gradient method with support for non-strongly convex composite objectives. In Advances in neural
information processing systems, pages 1646-1654, 2014.
[13] Aaron Defazio and Léon Bottou. On the ineffectiveness of variance reduced optimization for deep learning. In Advances in Neural Information Processing Systems, 2019.
[14] Tristan Deleu and Yoshua Bengi Structured sparsity inducing adaptive optimizers for deep learning. arXiv preprint 2rrxiv2102.000
2021.
[15] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255, Ieee, 2009.
[16] ONNX Runtime developers. Onnx runtime. https: /ttp:///nnxtttt/////o/oo ime 202,
[17]
Tianyu Ding, Luming Liang. Zhihui Zhu, and Ilya Zharkov. Cdfi: Compression- network design for frame interpolation. arXiv preprint arXiv:2103. 10555599 2021.
[18] Xiaohan Ding, Tianxiang Hao, Jianchao Tan, Ji Liu, Jungong Han, Yuchen Guo, and Guiguang Ding. Lossless enn channel pruning via decoupling remembering and forgetting. Proceedings of the IEFE International Conference on Computer Vision, 2021.
[19] John Duchi and Yoram Singer. Efficient online and batch learning using forward backward splitting. Journal of Machine Learning Research. 10(Dec): 28899-2488992294, 2009.
[20] Marwa El Halabi, Francis Bach, and Volkan Cevher. Combinatorial penalties: Which structures are preserved by convex relaxations? In International Conference on Artificial Intelligence and Statistics, pages 1551-1560. PMLR, 2018
[21] Angela Fan, Edouard Grave, and Armand Joulin. Reducing transformer depth on demand with structured dropout. arXiv preprint arXiv:1909 1ee.115 1156,, 2019.
[22] Cong Fang. Chris Junchi Li, Zhouchen Lin, and Tong Zhang. Spider: Near-optimal non- convex optimization via stochastic
differential estimator. In Advances in Neural Information Processing Systems, pages 689-699, 2018
[23] Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable
neural networks. arXiv preprint ArXiv:1803. 03635, 2018
[24] Jonathan Frankle. Gintare Karolina Dziugaite, Daniel M Roy, and Michael Carbin. Stabilizing
the lottery ticket hypothesis. arXiv preprint arXiv: 1903.01611, 2019.
[25] Kunihiko Fukushima. Visual feature extraction by multilayered network of analog threshold elements. IEEE Transactions on Systems Science and Cybernetics, 5(4):322-333, 1969. [26] Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity deep neural networks.
arXiv preprint 1902:0900.00974, 2019.

---

[27] Shang-Hus Gao, Yong- -iinn Tan, Ming-Ming Cheng, Chengze Lu, Yunpeng Chen, and Shuicheng Yan. Highly efficient salient object detection with 100k parameters. In European Conference on Computer Vision, pages 702-721. Springer, 2020.
[28] Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning.
volume 1. MIT press Cambridge, 2016.
[29] Mitchell A Gordon, Kevin Duh, and Nicholas Andrews. Compressing bert: Studying the
effects of weight pruning on transfer learning. arXiv preprint arXiv:2002 083007 2020.
[30] Fu-Ming Guo, Sijia Liu, Finlay S Mungall, Xue Lin, and Yanzhi Wang. Reweighted proximal pruning for large-scale language representation. arXiv preprint arXiv:1909 I 12486, 2019.
[31] Song Han, Xingyu Liu, Huizi Mao. Jing Pu, Ardavan Pedram, Mark A Horowitz, and William J Dally. Eic: Efficient inference engine on compressed deep neural network. ACM SIGARCH Computer Architecture News, 44(3):24-225, 2016.
[32] Song Han, Huizi Mao, and William Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffmman coding. arXiv preprint vrXiv:/510.00145 2015.
[33] Song Han, Jeff Pool, John Tran, and William J Dally, Learning both weights and connections for efficient neural networks. arXiv preprint arXiv:150k 02626, 2015.
[34] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance imagenet classification. In Proceedings of the IEEE international conference on computer vision, pages 1034,, 2015.
[35] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition,
2016.
[36] Yang He, Guoliang Kang, Xuanyi Dong, Yanwei Fu, and Yi Yang. Soft filter pruning for accelerating deep convolutional neural networks. arXiv preprint IrXiv:1808 06866, 2018.
[37] Yihui He, Ji Lin, Zhijian Liu, Hanrui Wang, Li-Jia Li, and Song Han. Ame: Automl for model compression and acceleration on mobile devices. In Proceedings of the European Conference on Computer Vision (ECCV). pages 784-800, 2018
[38] Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural networks. In The IEEE International Conference on Computer Vision (ICCV), Oct 2017. [39] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint
arXiv:1606.0844
2016.
[40] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network.
arXiv preprint arXivv/5503.. 1503.02533 2015.
[41] Hengyuan Hu, Rui Peng, Yu-Wing Tai, and Chi-Keung Tang. Network trimming: A
data-driven neuron pruning approach towards efficient deep architectures. arXiv preprint
arXiv:/607.00 00050,0300,000 2016.
[42] Junzhou Huang, Tong Zhang, and Dimitris Metaxas. Learning with structured sparsity. Journal
of Machine Learning Research, ((Nov):3371-3413 2011.
[43] Zehao Huang and Naiyan Wang. Data-driven sparse structure selection for deep neural networks. In Proceedings of the European conference on computer vision (ECCV), pages 304-320, 2018
[44] Rodolphe Jenatton, Guillaume Obozinski, and Francis Bach. Structured sparse principal component analysis. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statisties, pages 366-373, 2010.
[45] Jianxin Wu Jian-Hao Luo and Weiyao Lin. Thinet: A filter level pruning method for deep
neural network compression. In ICCV, pages 5058-5066, 2017.
[46] Rie Johnson and Tong Zhang. Accelerating stochastic gradient descent using predictive variance reduction. In Advances in neural information processing systems, pages 315-323,
2013.
[47] Ian Jolliffe. Principal Component Analysis, pages 1094-1096. Springer Berlin Heidelberg. Berlin, Heidelberg, 2011.
[48] Minsoo Kang and Bohyung Han. Operation-aware soft channel pruning using differentiable masks. In International Conference on Machine Learning, pages 1122-113.. PMLR, 2020. [49] A. Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images. Master's thesis, Department of Computer Science University of Toronto, 2009.
[50] Vivek Kwatra, Arno Schödl, Irfan Essa, Greg Turk, and Aaron Bobick. Graphcut textures: Image and video synthesis using graph cuts. Acm transactions on graphies (tog), 22(3):277-
286, 2003.
[51] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, 521((753)) 444. 2015.
12

---

[52] Namhoon Lee, Thalaiyasingam Ajanthan, and Philip HS ToMr. Snip: Single-shot network pruning based on connection sensitivity. arXiv preprint arXiv:1810.023440 2018.
[53] Sangkyun Lee and Stephen J Wright. Manifold identification in dual averaging for regularized stochastic online learning. The Journal of Machine Learning Research, ((1):1705-1744
2012.
[54] Bailin Li, Bowen Wu, Jiang Su, and Guangrun Wang. Eagleeye: Fast sub-net evaluation for efficient neural network pruning. In European Conference on Computer Vision, pages 339.654. Springer. 2020.
[55] Hao Li, Asim Kadav, Igor Durdanovie, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. arXiv preprint arXiv: I 6608, 0870007710 2016.
[56] Yawei Li, Shuhang Gu, Christoph Mayer. Luc Van Gool, and Radu Timofte. Group sparsity: The hinge between filter pruning and decomposition for network compression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8018 8 8-8027, 2020.
[57] Yuchao Li, Shaohui Lin, Baochang Zhang, Jianzhuang Liu, David Doermann, Yongjian Wu, Feiyue Huang, and Rongrong Ji. Exploiting kernel sparsity and entropy for interpretable cnn compression. In Proceedings of the IEÉE Conference on Computer Vision and Pattern Recognition, pages 2800-2809 2019.
[58] Shaohui Lin, Rongrong Ji, Yuchao Li, Cheng Deng, and Xuelong Li. Toward compact convnets via structure-sparsity regularized filter pruning. IEEE transactions on neural networks and learning systems, (1(2):574-588 2019.
[59] Christos Louizos, Karen Ullrich, and Max Welling. Bayesian compression for deep learning. In Advances in neural information processing systems, pages 3288- (888-329,, 2017.
[60] Jian-Hao Luo. Jianxin Wu, and Weiyao Lin. Thinet: A filter level pruning method for deep neural network compression. In Proceedings of the IEEE international conference on computer vision, pages 0058-5066, 2017.
[61] Sangkug Lym, Esha Choukse, Siavash Zangench, Wei Wen, Sujay Sanghavi, and Mattan Erez. Prunetrain: fast neural network training by dynamic sparse model reconfiguration. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1-13, 2019.
[62] Fanxu Meng. Hao Cheng. Ke Li, Huixiang Luo, Xiaowei Guo, Guangming Lu, and Xing Sun. Pruning filter in filter. arXiv preprint arXiv:2009. 14410, 2020.
[63] Michael Mitzenmacher. Probability and rannuttin-randdmized algorithums and probabilistic analysis. NOURNAL-OPERATIONS
RESEARCH SOCIETY, 16512))454, 2005.
[64] Eugene Ndiaye, Olivier Fercoq, Alexandre Gramfort, and Joseph Salmon. Gap safe screening rules for sparsity enforcing penalties. The Journal of Machine Learning Research, 18(1):4671-
4703, 2017.
[65] Kirill Neklyudov, Dmitry Molchanov, Arsenii Ashukha, and Dmitry P Vetrov. Structured bayesian pruning via log-normal multiplicative noise. In Advances in Neural Information Processing Systems, pages
2017.
[66] Yurii Nesterov. -rimal-dual subgradient methods for convex problems. Mathematical pro- gramming. 2009,
[67] Simon Niklaus, Long Mai, and Feng Liu. Video frame interpolation via adaptive separable convolution. In Proceedings of the IEEE International Conference on Computer Vision, pages 261-270, 2017.
[68] Jorge Nocedal and Stephen Wright. Numerical optimization. Springer Science & Business Media, 2006.
[69] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint 1oX.::6606.02250 2016.
[70] Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 779-788. 2016.
[71] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in neural information processing systems, pages 91-99, 2015.
[72] Alex Renda, Jonathan Frankle, and Michael Carbin. Comparing rewinding and fine-tuning in neural network pruning. arXiv preprint arXiv:2003.02::3 2020.
[73] Volker Roth and Bernd Fischer. The group-lasso for generalized linear models: uniqueness
of solutions and efficient algorithms. In Proceedings of the 25th international conference on Machine learning, pages 848-855 855, 2008
13

---

[74] Nicolas L Roux, Mark Schmidt, and Francis R Bach. A stochastic gradient method with an exponential convergence rate for finite training sets. In Advances neural information processing systems, pages 2663-2266 1671, 2012.
[75] Victor Sanh, Thomas Wolf, and Alexander M Rush. Movement pruning: Adaptive sparsity by 2020.
fine-tuning. arXiv preprint arXiv:2005.0766
[76] Vikash Sehwag, Shiqi Wang, Prateck Mittal, and Suman Jana. Hydra: Pruning adversarially
robust neural networks. Advances in Neural Information Processing Systems (NeurIPS), 7, 2020.
[77] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: / 1556, 2014.
[78] Arvind Subramaniam and Avinash Sharma. N2nskip: Learning highly sparse networks using neuron-to-neuron skip connections. In BMVC, 2020.
[79] Yehui Tang, Yunhe Wang, Yixing Xu, Dacheng Tao, Chunjing Xu, Chao Xu, and Chang Xu. Scop: Scientific control for reliable neural network pruning. arXiv preprint arXiv:2010.10732. 2020.
[80] Frederick Tung and Greg Mori. Clip-q:
Deep network compression learning by in-parallel In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). June 2018.
[81] Mart van Baalen, Christos Louizos, Markus Nagel, Rana Ali Amjad, Ying Wang, Tijmen Blankevoort, and Max Welling. Bayesian bits: Unifying quantization and pruning. arXiv 2020.
preprint www.ix2005.0000000000
[82] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[83] Chaoqi Wang. Guodong Zhang, and Roger Grosse. Picking winning tickets before training by preserving gradient flow. arXiv preprint
(tt::2002.073772 2020.
[84] Ting-Chin Chun-Chun Wang. Ming- Yu Liu, Jun- Yan Zhu, Guilin Liu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. vodeo-to-video synthesis. arXiv preprint arxiv: 1808 808.06601, 2018.
[85] Wei Wen, Chumpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured sparsity in deep neural networks. arXiv preprint arXiv:1608 0306800366,..0 2016.
[86]
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: novel image dataset for benchmarking machine learning algorithms, 2017.
[87] Lin Xiao. Dual averaging methods for regularized stochastic learning and online optimization.
Journal of Machine Learning Research, 1 1(0cct):2533-2555 2010.
[88] Lin Xiao and Tong Zhang. A proximal stochastic gradient method with progressive variance
reduction. SIAM Journal on Optimization, 24(4):2057-2075, 2014.
[89] Bing Xu, Naiyan Wang, Tianqi Chen, and Mu Li. Empirical evaluation of rectified activations in convolutional network. arXiv preprin! arXiv: / /505000833 2015.
[90] Haichuan Yang. Shupeng Gui, Yuhao Zhu, and Ji Liu. Automatic neural network compression by sparsity-quantization joint learning: A constrained optimization- approach. In Pro- ceedings of the IEFF/CVE Conference on Computer Vision and Pattern Recognition, pages 1188-2188. 2020.
[91] Huanrui Yang, Wei Wen, and Hai Li. Deephoyer: Learning sparser neural network with differentiable scale-invariant sparsity measures. arXiv preprint arXiv: /908. 09999, 2019.
[92] Haiqin Yang, Zenglin Xu, Irwin King, and Michael R Lyu. Online learning for group lasso. In Proceedings of the 27th International Conference Machine Learning (ICML-10), pages
1 1191-1198, 2010.
[93] Minghan Yang, Andre Milzarek, Zaiwen Wen, and Tong Zhang. A stochastic extra-step quasi- newton method for nonsmooth nonconvex optimization. arXiv preprint arXiv: 190.0933, 2019.
[94] Zhonghui You, Kun Yan, Jinmian Ye, Meng Ma, and Ping Wang. Gate decorator: Global filter pruning method for accelerating deep convolutional neural networks. arXiv preprint
arXiv:/909.00 080008744 2019.
[95] Ming Yuan and Yi Lin. Model selection and estimation in regression with grouped variables. Journal of the Royal Statistical Suciety; Series B (Statistical Methodooogy). 68(1) 1949667, 2006.
[96] Cun-Hui Zhang et al. Nearly unbiased variable selection under minimax concave penalty. The
Annals of statistics, 38(2)8894994,, 2010.
[97] Junyu Zhang and Lin Xiao. Multi-level composite stochastic optimization via nested variance reduction, arxiv preprint arXiv: /908. 1468, 2019.
14

---

[98] Pengchuan Zhang, Hunter Lang, Qiang Liu, and Lin Xiao. Statistical adaptive stochastic gradient methods. arXiv preprint arXiv:2002. 10597. 2020.
[99] Richard Zhang, Phillip Isola, Alexei Efros, Eli Shechtman, and Oliver Wang. The unrea- sonable effectiveness of deep features perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586-595, 2018.
[100] Tianyun Zhang, Shaokai Ye, Kaiqi Zhang. Jian Tang, Wujie Wen, Makan Fardad, and Yanzhi Wang. A systematic dnn weight pruning framework using alternating direction method of multipliers. In Proceedings of the European Conference on Computer Vision (ECCV), pages 184-199, 2018.
[101] Yuefu Zhou, Ya Zhang. Yanfeng Wang, and Qi Tian. Accelerate cnn via recursive bayesian pruning. In Proceedings of the IEEE International Conference Computer Vision, pages 3306-3315, 2019.
[102] Tao Zhuang, Zhixuan Zhang, Yuheng Huang, Xiaoyi Zeng, Kai Shuang, and Xiang Li. Neuron- level structured pruning using polarization regularizer. Advances in Neural Information Processing Systems, 33, 2020.
15


Reviews:
[{"review_id": "zNxJOjLP3Pt", "reviewer": "Reviewer_N3gq", "summary": "This paper first proposes the zero-invariant group for neural network. Zero-invariant group is the disjoint group of trainable parameters which results in corresponding output to be zero. Then, this paper formulates a optimization problem with group sparsity regularizer on the zero-invariant groups. Then, authors alternate SGD and Half-space projection to optimize the problem. This optimization method is called Half-Space Stochastic Projected Gradient (HSPG) which benefits from the larger projection region. This method does not require fine-tuning  and pruned model is directly used on the inference. On various dataset (Cifar-10 and ImageNet), this paper achieves state-of-the-art performance. \n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Weakness\n1.  I believe the novelty of the group sparsity regularizer on the zero-invariant group is not enough. The idea of a group sparsity regularizer has received attention for a long time. Please refer to [1], [2]. The only extension of this paper is to leverage the group sparsity regularizer on the zero-invariant group which includes the parameter in the batch normalization. Therefore, authors need to compare these related works and this paper, then show the superiority of the proposed method with the experiments.\n\n2. The paper insists on the superiority of the HSPG method. However, there is no intuitive explanation for this. The only explanation is the larger projection space. But, readers might not agree that this point directly contributes to the performance of the pruning method. The authors need to provide the intuitive explanation that most of the readers and reviewers agree with. I suggest the authors reduce the ZIG section and add an ablation study and explanation from the appendix. \n\n3. Table1 is quite weird. It seems that the most of FLOPs reduction comes from the first channel reduction (64->3). There should be sufficient analysis and explanation for this.\n\n4. The baselines on the experiment results are outdated and there are not enough baselines in Tables 2 and 3. Authors need to compare the proposed method with recent pruning papers. For example, GBN [3] achieves 75.18 with 50% FLOPS. \n\nTherefore, I conclude that this paper is not ready for submission. The novelty of the proposed method is hard to find and the superiority of the proposed method is hard to believe since there is no intuitive explanation and not enough experiment results.\n\n[1] Learning Structured Sparsity in Deep Neural Networks, NeurIPS 16\n[2] DeepHoyer: Learning Sparser Neural Network with Differentiable Scale-Invariant Sparsity Measures, ICLR 20\n[3] Gate Decorator: Global Filter Pruning Method for Accelerating Deep Convolutional Neural Networks, NeurIPS 20.\n\n\nPost : \n\nI have read all the feedback. Some issues are resolved. However, still other critical issues are not fully resolved. Also, I could not agree with the novelty from ZIG and do not fully agree with importance of ZIG. Furthermore, I still could not fully understand the correlation between the large projection area and group sparsity. Furthermore, I could not see the updated experiment results of important baselines in the main table. Therefore, I keep my score.\n\n\nPost Post: \n\nI have read authors' responses. Most of critical issues which I pointed are resolved. Please update the experiment results (after fine-tuning) + other baselines. I wish to see them in the final version. Some parts are still hard to understand (HSPG) and I'm not sure that claims on ZIG are impressive. However, clear experiment results support the novelty and importance of the proposed method. There is no reason to reject this paper. Therefore, I raise my score.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "6 hours"}, "extra_text": {}}, {"review_id": "w7JOpWFsG5t", "reviewer": "Reviewer_Echy", "summary": "This paper proposes a structured pruning method called Only-Train-Once (OTO). OTO first partitions the trainable parameters into zero-invariant groups called ZIGs. This way allows one to prune without affecting the output and therefore it requires no additional training steps after pruning. Once they define ZIGs, they perform Half-space Stochastic Projected Gradient (HSPG), starting from a pre-trained estimate using SGD, it performs a form of projected gradient descent while gradually increasing group sparsity. The authors demonstrate the effectiveness of OTO on CIFAR, ImageNet, SQuAD by showing an impressive gain in terms of FLOPs while maintaining the performance of resulting slim network to the original dense counterparts.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This work is motivated to remove the subsequent fine-tuning process after pruning (i.e., making some parameters zero-valued) that is often performed for various pruning approaches, in particular those in a similar fashion such as Group Lasso or any group sparsity inducing regularizer based methods. The idea of first creating zero invariant groups is interesting and new in the context of pruning as far as I know, and designing these groups for major components in modern neural networks is done in an easy and nice way. This can actually be very pragmatic in practice. Their actual pruning and training algorithm, HPGS, differs from rather aggressive filter pruning methods that work based on heuristics and lack theoretical guarantees on convergence, and combined with ZIGs, it seems that the idea is very effective as supported by experimental evidence. Overall, the paper is written with clarity and organized well.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "- Can the process of creating ZIGs be made automatic? I feel there should be a generalized way to perform the process. Discussing this may make the manuscript stronger. \n\n- The idea of finishing the pruning step in one shot (and training only once) seems to be shared with other “single-shot” pruning methods either before such as SNIP by Lee et al. 2019, GraSP Wang et al. 2020, or any potential pruning method relying on a saliency criterion. While OTO is based on an optimization method with some convergence guarantee (I haven’t checked this in detail; is it applied for non-convex objectives?) and the above methods seem to rely on heuristics, crediting the former for algorithmic simplicity (L94: “OTO is simple”) over the others seems a bit stretched, considering that OTO requires additional step of creating ZIGs, performing a projected gradient descent, especially without any analysis on the computational cost of HSPG in comparison to others.\n\n- Related to this, the paper defines pruning approaches to be those that mostly require fine-tuning (or re-training) and claims their contribution for removing it, and yet, this doesn’t seem to be a fair scope setting. Any sparsity inducing regularization based method (including those cited in this work such as [50]) can theoretically remove fine-tuning step.\n\n- Discussing these properly in the paper seems necessary. Having said that, the title as well as the name of the method are not quite representative of the main idea.\n\n- Any idea how to perform HSPG without ZIGs? Would it basically reduce to [58]?\n\n- How do you control the level of sparsity (set hyperparameters in general) found by HSPG?\n\n- Any analysis on the cost of HSPG and its comparison to other methods?\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "F8u5r5nyUG", "reviewer": "Reviewer_DjRg", "summary": "This paper proposes a one-shot pruning strategy, in which the pruned subnetwork achieves good performance on several benchmarks without post procedures or iterations. They make use of the ZIG and propose HSPG to enhance the group sparsity exploration. Good performance across several benchmarks supports  the validity of OTO.\n\n", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength\n\n1. The pruning method no longer requires fine-tuning and achieves impressive results on several benchmarks, little performance drop with high compression ratio. \n\n2.  The compression experiments are designed for both vision  and NLP tasks. \n\nWeakness\n1.  Missing one important baseline ResRep[1],  which can make this work weaker if included. Note that [1] is a lossless pruning method.\n\n2.  The paper claims that it achieves SOTAs compression benchmark such as R50. However, [1] also prunes without fine-tuning. And the latter slims down a standard ResNet-50 with 76.15% accuracy on ImageNet to a narrower one with only 45% FLOPs.  Morever, ResRep[1] achives 75.3% under similar FLOPS reduction (OTO 74.7%). \n \n\nOthers\n\n1. It's interesting to see the performance of OTO on vision transformers such as ViT.\n\nReference\n\n[1]Lossless CNN Channel Pruning via Decoupling Remembering and Forgetting, https://arxiv.org/abs/2007.03260 （ICCV21）\n\nPost:\n\nThanks for the reply. My concern is not well addressed. And I have carefully read the reviews of other reviewers. \n\nI still feel that ignoring some strong baselines  can be misleading and greatly weaken the contribution of this paper. Therefore, I lower my score to below the acceptance threshold.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "See the weakness.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "6sVPeNFqCpk", "reviewer": "Reviewer_7J6K", "summary": "This paper proposes a group sparsity optimisation approach for training pruned models. This is done by splitting the optimisation into an initial phase where the non-differentiable regularisation objective is optimised using subgradients, followed by a second stage where channels/units (plus their corresponding bias/BN/residual path) are mapped to exact zero by augmenting the sub-gradients with an indicator function.  For each parameter group, this function compares the direction of subgradient with the direction you'd get as a result of pruning that group, and if the angle between the two is larger than 90 it prunes them away.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I think this is an interesting work and the results seem significant but I have a few concerns and suggestions before I can increase my score.\n\n* I feel the way OTO and ZIG are presented as novel ideas is a bit misleading. In the list of contributions OTO is presented as a separate thing in addition to ZIG and HSPG, but it's essentially just the combination of two? The idea of pruning corresponding parameters attached to a channel/weight/unit is also not exactly novel and in some cases (for instance with residuals) is necessary to get shapes right. Similarly, framing this work as \"one-shot\" pruning is a little bit misleading. The sparsity-inducing objective is present in both initialisation and the projection steps for the entirety of the training so I'm not sure what's \"one-shot\" about this approach.\n* At the beginning of Section 3.1, the authors claim that the _root cause_ that other pruning methods fail is the fact that saliency scores are computed for units/channels in isolation, and their corresponding bias/BN/residual terms are not included. I think this is a bold claim and is not supported by any evidence in the paper. Many pruning objectives are trivial to expand to include the corresponding parameters so if authors want to make this claim this claim I suggest they support it with some ablation experiments with some baselines where this change can be easily tested.\n* Section 3.3:  \n    * I found this section difficult to read and follow. Unless I missed something I felt this section explains the same idea three times with slightly different notations. The beginning of this section includes an \"intuition\" for doing HSPG but after reading that I was expecting to see the half-space _perpendicular_ to the direction of the gradient that takes you from $x_k$ to $x_{k+1}$. But instead, the half-spaced is based on the origin. Could the authors pehaps make this bit clearer? \n    * I was also a bit confused by Fig3a as I was expecting to see projections of $x_{k+1}$ on [x]1 and [x]2 axes?\n    * Am I right in thinking that the optimization using subgradient stage doesn't actually stop, but is simply augmented with this additional projection? If yes I think the writing could be improved to include this fact early on.\n    * if epsilon is non-zero in all of the experiments why is mentioned at all?\n* Section 3.4\n    * I'm not sure what's the point of this section. Surely by the end of the optimisation the zero groups have either emerged or not. So I'm not sure what's left to do and what authors mean by \"full model\" vs \"pruned model\". Are you talking about physically removing those rows/columns from weight/bias tensors?\n\nOther questions and suggestions\n* I'd be curious to see an ablation study to analyse the effect of initialization stage vs the projection stage. What's the split (in terms of the number of epochs) between the two and how did the authors choose these numbers? what is the performance of the neural network _before_ the projection phase?\n* I don't find the ResNet-50 experiments with CIFAR-10 convincing. Why have you decided on two baselines (AMC and ANNC) that achieve such low pruning rates (and report no FLOPS numbers)? The fact that these two methods also use quantisation only muddies the waters furthermore. I suggest including some good old pruning baselines with similar pruning ratios.\n* how do you control the sparsity target?\n* using $x$ to denote parameters is kind of confusing. Perhaps use $\\theta$?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "NA", "review_text": "", "extra_scores": {"ethical_concerns": "NA", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose a fine-tuning free structured pruning method (OTO). The idea is to first partition the parameters into zero-invariant groups, pruning the zero groups, and solving for a structured sparsity optimization problem with projections. The experiment results on CIFAR10, ImageNet, and SQuAD show the competitive performance on FLOPs and number of parameters reduction.

The paper initially received a split review. I would like to thank the time and effort the authors and reviewers spent engaging in the active discussion during the rebuttal phase.

I strongly recommend the authors to revise the submission and include the ablation study and the baseline comparisons the reviewers requested.

Specifically, please include

1. The ablation study of initialization stage vs projection stage

2. Baseline comparison with related methods [1,2,3,4,5].

[1] Learning Structured Sparsity in Deep Neural Networks, NeurIPS 16 
[2] DeepHoyer: Learning Sparser Neural Network with Differentiable Scale-Invariant Sparsity Measures, ICLR 20
[3] Gate Decorator: Global Filter Pruning Method for Accelerating Deep Convolutional Neural Networks, NeurIPS 20.
[4] Operation-Aware Soft Channel Pruning using Differentiable Masks, ICML2020
[5] Lossless CNN Channel Pruning via Decoupling Remembering and Forgetting. ICCV 2021.


Decision: Accept (Poster)