Feature Flow Regularization: Improving Structured  Sparsity in Deep Neural Networks
Authors: Yue Wu
Yuan Lan
Luchan Zhang
Yang Xiang
Venue: ICLR 2022 Submitted (2022)
Keywords: structured pruning

Abstract: Pruning is a model compression method that removes redundant parameters and accelerates the inference speed of deep neural networks (DNNs) while maintaining accuracy. Most available pruning methods impose various conditions on parameters or features directly. In this paper, we propose a simple and effective regularization strategy to improve the structured sparsity and structured pruning in DNNs from a new perspective of evolution of features. In particular, we consider the trajectories connecting features of adjacent hidden layers, namely feature flow. We propose feature flow regularization (FFR) to penalize the length and the total absolute curvature of the trajectories, which implicitly increases the structured sparsity of the parameters. The principle behind FFR is that short and straight trajectories will lead to an efficient network that avoids redundant parameters. Experiments on CIFAR-10 and ImageNet datasets show that FFR improves structured sparsity and achieves pruning results comparable to or even better than those state-of-the-art methods.

Under review as a conference paper at ICLR 2022
FEATURE FLOW REGULARIZATION: IMPROVING STRUCTURED SPARSITY IN DEEP NEURAL NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
Pruning is model compression method that removes redundant parameters and accelerates the inference speed of deep neural networks (DNNs) while maintain- ing accuracy. Most available pruning methods impose various conditions on pa- rameters or features directly. In this paper, we propose a simple and effective regularization strategy improve the structured sparsity and structured pruning in DNNs from new perspective of evolution of features. In particular, we con- sider the trajectories connecting features of adjacent hidden layers, namely fea- ture flow. We propose feature flow regularization (FFR) to penalize the length and the total absolute curvature of the trajectories, which implicitly increases the structured sparsity of the parameters. The principle behind FFR is that short and straight trajectories will lead to an efficient network that avoids redundant param- eters. Experiments CIFAR-10 10 and ImageNet datasets show that FFR improves structured sparsity and achieves pruning results comparable to or even better than those sfate-of-the-ar methods.
INTRODUCTION
Deep neural networks (DNNs) have achieved huge success in a wide range of applications. Mean- while, DNNs require considerably more computational cost and storage space as they become deeper in order achieve higher accuracy. Denil al. (2013) demonstrated that there is significant redun- dancy in the parameterization of DNNs. The Lottery Ticket Hypothesis (Frankle & Carbin, 2019) conjectures that there exist sparse sub-networks that can obtain a comparable accuracy with the original network when trained in isolation.
Model compression methods have been proposed to balance accuracy and model complexity, e.g. weight pruning (Drucker & Le Cun 1992; Hassibi & Stork 1993] Han et al, 2015; Guo et al.) 2016 Hu et al.
2016 Li et al. 2016)) and quantization (Gong et al. 2014). low-rank approximation (Denton et al. 2014 Jaderberg et al. 2014 Liu et al., 2015). and sparsity structure learning (Wen
ct al. al.
2016 Alvarez & Salzmann] 2016|/Zon et al. 2016 Liu et al. 2017] Louizos et al. 2018 Gao
2019 Yuan al.) 20200). Weight pruning removes less important parameters in the network. In particular, filter pruning (Li et al. 2016) removes entire filters in the network together with their related channels, which can compress and accelerate DNNs efficiently.
Existing structured pruning methods can be divided into two categories: parameter-based methods (Li et al. 2016, Molchanov et al. 2016 Liu al. [0117] He al., 2018 Lin al. 2019: He al.] 2019 Zhuang et al.) 20200 Liebenwein et al. (020) that use some criteria to identify unimportant filters and remove them, and feature- based methods (Luo al. 2017 He et al.
2017: Zhuang al., 2018 Ye et al. 2018 Li et al. 2020 Lin et al.) 2020 Tang et al. 2020) that select unimportant feature maps and then remove related filters and channels. For example, et al. (2020) incorporated two feature map selections: discovering features with low diversity and removing features that have high similarities with others.
In this paper, we propose new regularization method the trajectory connecting features of adja- cent hidden layers, namely feature flow regularization (FFR). FER smooths the trajectory of features, which implicitly improves the structured sparsity in DNN. Our motivation is that the trajectory of data along the network reflects the DNN structure. Shorter and straighter trajectory corresponds to an efficient and sparse structure of DNN. An illustration is given in Figure Tb.

---

Under review as a conference paper at ICLR 2022
Inout
Features
Outout
Filters
structured sparsity improved
Input
Output
Feature FFR
Feature flow
Input
Output
(
Baseline
Filters
Features
20
Layer
0
L
Data space
FFR:
>
I8i4 all
2zz +m--ll
(c)
Length Curvature
(b)
Figure 1: (a) Feature flow demonstration: Each curve represents feature flow, i.e., the trajectory connecting the features of hidden layers, where nodes are input, features and output. (b) Feature flow regularization demonstration: Feature flow under FFR shorter and straighter due to the length and curvature penalty. As result, FFR improves the structured sparsity and leads to effective filter pruning. (c) An illustration example in two dimensional space showing the smooth effect of FFR, which is five-block ResNet trained with FFR and without FFR (the baseline). The input data, features and targets are all points in two dimensions, and the feature flows are curves in two dimensions. The green cluster and red cluster contain input data points and the targets, respectively.
Our main contributions are: (1) We propose new regularization (FFR) the trajectory connect- ing the features of hidden layers, to improve the structured sparsity DNN from perspective of the trajectory of data along the network. This method is different from the existing sparsity struc- ture learning methods, which directly impose regularization or constraints on the parameters. Our method is also different from those pruning methods based feature maps, which use the infor- mation of the feature map individually or in pairs (for similarity) without global relationship. (2) We analyze the effect of FFR applied to convolutional layer and residual layer, and show that FFR encourages DNN to learn sparse structure during training by penalizing the sparsity of both pa- rameters and features, (3) Experimental results show that FFR achieves comparable even better pruning ratio in terms of parameters and FLOPs than recent state-of of-toe--a-tt-e---- the-ar art pruning methods.
2 RELATED WORK
Filter pruning. Various criteria for filter selection in pruning have been proposed. Li et al. (2016) used L1 norm select unimportant filters and removed the filters whose norm lower than the given threshold together with their connecting feature maps. Molchanov et al. (2016) measured the importance filters based on the change in the cost function induced by pruning. Luo et al. (2077): He et al. (017) formulated pruning as constraint optimization problem and selected most representative neurons based on minimizing the reconstitution error. Lin et al. (2019) pruned filters as well as other structures by generative adversarial learning. He et al. (2019) pruned redundant filters utilizing geometric correlation among filters in the same layer. Hu et al. (2016); Zhuang et al. 2018); et al. (2020): Lin al. (2020) removed filters based on the information. e.g. sparsity, rank or diversity, of feature maps that are generated by the filters. Our method learns sparse DNN during training and adopts magnitude-based pruning scheme after training.
2

---

Under review as a conference paper at ICLR 2022
Sparsity regularization. Some studies introduced sparsity regularization to find sparse structure of DNN. A commonly used strategy is to impose group Lasso and relaxed Lo regularization (Zhou
et al., 2016, Wen et al. 2016 Alvarez & Salzmann, 2016 Louizos al. 2018). Liu et al. (2017) 7); Huang & Wang 2018) associated a scaling factor with feature maps and imposed regularization on these scaling factors during training to automatically identify unimportant channels. The feature maps with small scaling factor values will be pruned. [aao et al. (2019) imposed Cross-Layer grouping and Variance Aware regularization on the parameters to improve the structured sparsity for residual models. Zhuang et al. (2200) used polarization regularizer on scaling factors. Yuan et al.) (2020) proposed method to dynamically grow deep networks by continuously sparsifying struc- tured parameter sets. Different from these available methods that directly introduce regularization on the parameters or scaling factors, our FFR imposes regularization on the trajectory connecting features of hidden layers to control the parameters and enforce structured sparsity implicitly.
3 METHOD
3.1 FEATURE FLOW REGULARIZZTTIN
We define feature flow as the trajectory formed by connecting the output features of adjacent hidden layers. For DNN, the collection of trajectories with different input data from the training dataset reflect the network structure. We control these trajectories to obtain a sparse network.
Consider the forward propagation of DNN with L layers { i.0......
(1) where 12+11 is the output feature of the 1-th layer, by is the mapping in the 1-th layer, and to is the
21-1 = h4(21, wt),
collection of trainable parameters. Introducing a temporal partition: {4 = 1/L) with time inter-
val At 1/ L, and regarding æ) the value of function sxt) at time step 4, without considering
dimensional consistency, Eq. [ can be rewritten as (He et al. 2016 2017 Lu et al. 2017 Chen
al. 2018
a(t++1) =tt) + At (x(2((t), wt),
where hy = (hu - / / At. This can be interpreted as a discretization of evolution along a trajectory of the network described by an ordinary differential equation (E) 2017 Lu et al. 2017 Chen al. 2018
dr(() (((((())))))))))))))))))) w()), dt
The feature flow is the trajectory formed by connecting features of {m)). and is denoted by l'(z1). See Figure a for an illustration of the feature flow.
We regard the trajectory ar(() as well as the feature flow trajectory I [(ar] a "curve". Recall that for a curve (t) [0, 1] + Rd with are length parameter s, its length C(r) and total absolute curvature K(r) are
((5) C(7) = L. In'O) K(>)= = (000)) In(s)lds,
(4)
where x(s) = (s)] is the curvature of the curve.
We introduce feature flow regularization (FFR) to improve the structured sparsity of DNN, bor- rowing the definitions of length and total absolute curvature of a curve to the feature flow, i.c., the trajectory formed by connecting features of hidden layers. For feature flow associated with hidden layer features {2:)1=0,1,... the FFR
R(x) kn C(x) + k2K(x),
(5)
where
C(x) = Σ | mill. K(xx))
| 2n + -m-1l|
(6)
1=0
with || being the L1 norm, and ki K2 > 0 the hyperparameters. Here up to some constant factors, C(x) Σt-1 [[)((t:)] approximation of the total length of the trajectory sr(), and
3

---

Under review as a conference paper at ICLR 2022
K(x) Elsi [x(tt] is an approximation of its total absolute curvature, where {ar x=(4)] is discretization of trajectory xxt) with time partition { = 1//
FFR smooths the feature flow by controlling the length and curvature. Intuitively, the length term in Eq. (6) makes the feature flow short, and the curvature term in Eq. (6) keeps the feature flow from bending too much. As a result, DNN trained under FFR has a more sparse structure; See demonstration in Figure Пpp. More quantitative analysis is given in Sec. + We further give an illustration example in Figure [.. which is a two-dimensional visualization showing the smooth effect of FFR on the feature flow of a ResNet. In this example, the input, features and output all points in two dimensions, and the feature flows are actual curves in two dimensions. We can see that the feature flow under FFR is shorter and straighter. More detail of this illustration example given in Appendix Sec. A.I. The curvature term also encourages more uniform distribution of hidden features on the trajectory from the input to the output, which makes it easier and more accurate to learn with sparse parameters; see the discussion in appendix A.4 and Figure 7
3.2
FFR APPLIED TO DNN AND PRUNING
For a DNN and training dataset {x((1, g00 y0) 1==1" using Eq. (). the FFR is:
1
L-1
R(X()) = 11 1 - 1 11 12
00)
| 2x + x1-111
(7)
i=1
denotes the set of output features of hidden layers for the j-th input data. The loss function with FFR in training is:
where x(O (=,,1= 0....,, (5)
I Σ J(xb), (6) w) + R(X()))
(8)
where [xx y, W) is the loss function before applying FFR.
Note that two hidden states Fı, I14 may have different dimensions. To fix the dimension mismatch problem, we adopt the same strategy as He et al.] (2016), i.c, using a linear projection P. by the shortcut connections match the dimensions. We replace z; in Eq. 7 by Pizı, where P, is the learnt projection matrix and will be treated as learnable parameters in training. In implementation, first group the features according the stage (features dimensions):
x = U1 1 J9.2 ,
L
9
gm
Here G is the number of stages in ( and lg is the number of hidden layers in stage g. Secondly, use the projection matrix to link different groups since the dimensional mismatch only occurs the first feature of each stage. Using this method, the FFR of x66 becomes:
G
4--1
R(X)))
= K1
|
P g-14k
||
| x (5)
()
g,
9.6
g=1
(10)
G
1
20(5) g.1 + Pa U)
.) || - -2x g.2 + x g. 1--111 Σ
+ K2
+
9=1
1=2
In the meantime, the hyperparameters kj, ko may vary with the feature dimensions so that FFR can uniformly control the features at different stages. In our experiments, we adjust k k2 to be inversely proportional to the scale of feature maps. Moreover, the FFR process can be generalized to the case where we choose features every several layers. We can denote the selected hidden layers as L = (4, ì 0, 1. m. #LJ. Then we apply FFR to the feature flow that connects the features in the set x (
After training, we conduct one-shot filter pruning: removing filters with small magnitude and re- moving channels in the next layer that convolve with the feature maps generated by the pruned filters. Finally, we fine tune the pruned network for a few epochs.
Our FFR training and pruning method is summarized in Algorithm
4

---

Under review as a conference paper at ICLR 2022
Algorithm FFR Training and One-shot Pruning
Require: training dataset (x(() (005) // a neural network and hyperparameters k1, kg
Pre-step 1: group the features in X = { 1 } according to the stage as in Eq. (9)
Pre-step 2: write down FFR R(x in Eq. (10) for each paired data (xx() ))))
Training step: train the network under loss function with FFR given Eq. (8).
Pruning step: remove filters and the corresponding channels in the trained model, fine tune the pruned model for a few epochs.
return compact neural network.
3.3 FEATURE FLOW OF VGGNET AND RESNET
In this subsection, we demonstrate how to construct feature flows of VGGNet (Simonyan & Zis- serman 2015) and ResNet (He et al., 2016), which are two commonly used network architectures. Similar construction can apply to other neural networks.
Conv1
Conv1
f BN NKU
Conv
Conv2
AN,
Conv2
Fia,
* TRG
(Manop
F(a, Wi]
Conv3
(b)
Figure 2: (a) A convolutional block in VGGNet. (b) Residual blocks in ResNet.
VGGNet. VGGNet is a convolutional neural network with a plain structure. Each convolutional layer is followed by some activation functions: batch normalization, rectified linear activation func- tion (ReLU), and sometimes a maxpooling layer. We regard the convolutional layer with all its activations as a block; see Figure 2a We collect the output of every convolutional block in VGGNet to form the feature flow [(z)), Let xy be the output feature of the 1-th block, then the feature of the I + 1-th convolutional block is
31+1 (Max pooling) aBBN @ (Wi @ sz)),
(11)
where Wi is the convolutional weights, BN is batch normalization, and a(.) is ReLU. Max pool- ing only appears once every several layers. @ denotes convolution operation and denotes batch normalization operation.
ResNet. ResNet takes residual blocks as building blocks, each which contains two or three convolutional layers; see Figure 2b We collect the output of every residual block to form the feature flow [( The feature of + 1-th residual block is
24 = oFF(z, Wi) W 21)
(12)
where F(a,, Wi) is the residual function, W, (Wi,lk 1, ,....,K K} is the set of convolutional weights, = 2 or 3 is the number of convolutional layers in each residual unit. W., is the identity matrix if xy and F(zi, Wi) are in the same dimension, and the learnt projection matrix otherwise.
4 SPARSITY ANALYSIS
In this section, we demonstrate how FFR improves structured sparsity. As explained in the previous section, the idea of FFR to shorten and straighten the trajectory the input along the network. We will show that such effect of FFR encourages feature sparsity in addition penalizing the sparsity of
5

---

Under review as a conference paper at ICLR 2022
the parameters. Sparse features contribute structured sparsity: for zero-valle value feature map, remove the filter which produces the feature mäp and the channel which convolves with the feature map. Therefore. FFR improves the structured sparsity in the network.
4.1 FEATURE AND PARAMETER SPARSITY
We analyze the effect of FFR when it is applied to the output features of the convolutional block and the residual block, which are commonly used in DNNs. We focus on the length term in Eq. (6) in FFR.
Convolutional block. Ignoring the activation functions in Eq. reduced to
the length term in FFR is
|| - x, [W, & x, - xyه1 = ([ - @ xzll.
(13)
where denotes the 'identity" convolution kernels: for the i-th filter, only the i-th channel a nonzero matrix and all other channels are zero-matrices. This length term pushes the parameter Wi to be close to the identity kernel I, which is highly structured sparse. In the meantime, the length term also pushes the feature Ii to be sparse. Figure 3a shows the L1 norm of 512 feature maps generated by the last convolutional layer in VGG16 trained with and without FFR on CIFAR-10. 10. As shown in the figure, the baseline network (VGG16 trained without FFR) has few zero-valued feature maps. In contrast, the network trained under FFR learns much more sparse features: most feature maps have zero-valued norm that can be removed. Such significant improvement of feature sparsity under FFR is also observed in the comparison for other layers of VGG16; see Appendix section A.2 for more plots.
Residual block. The length term in FFR enhances sparsity of the residual block by pushing the residual function F(x,, Wi) and features zι to zeros. We ignore the projection matrix W, in the shortcut connection since it only appears when the stage changes. From Eq. 112 the length term in FFR is reduced to
(14) Under the hypothesis that the optimal function closer to an identity mapping than to a zero map- ping (He et al. 2016), the first case F(z), Wi) x1 2 0 holds most of the time, and in this case, FFR penalizes the norm of the residual function. For a residual block with two or three convolutional layers, and ignoring the activations, the residual function
|| m| [F Wi) + 21) z1 element-wise wise
[ Will. ||21||,
F(m, 2 F(æ,, Wi) + æ
be reduced to
F(m,, Wi) a (BN, @ W12 @ oBN1 @ W7,1 @ mx)) ~ W1,2 @ W11 ( 21,
(15)
F(a, Wi) = a (BN3 C W (BN O (BN O Wi, @z)))) ~ W1, @ W2 8 Wi,1
@
(16)
Hence imposing penalty on the residual function can make both parameters and features sparse. In the second case, where F(z,, Wi) -1, FFR encourages the feature z; be sparse.
Further details of how FFR improves the structured sparsity are given in appendix A.3
Figure Shaure3bJshower shows the L1 norm of 64 feature maps of the first residual block ResNet56 trained with and without FFR on CIFAR-IO. 10. As shown in the figure, the network trained under FFR outputs more zero value feature maps than the baseline (trained without FFR). In Figure we visualize the parameters of shape 16 x 16 x 3 x 3 from the first convolutional layer in the first residual block of ResNet56 trained with FFR on CIFAR- 10. We display the filter row according to the magnitude: from top to bottom, the norm of the filter is increasing. It can be seen from the figure that the parameters of the network trained with FFR are of high structured sparsity. In the figure, small value filters are circled with a red horizontal square, and small value channels convolved with the same feature map are respectively circled with orange vertical squares.
In short, Figures 3a and 3b empirically prove that FFR improves structured sparsity of features, and Figure 3e shows its effect on the structured sparsity of the parameters.

---

Under review as a conference paper at ICLR 2022
Fitt
(a) VGGI6
16
(b) ResNet56
(c) Filter visualization: 16 x 16 3
Figure 3: (a) VGGI6 feature maps trained with and without FFR: L1 norm plot, with 512 feature maps in the feature of the last convolutional layer. (b) ResNet56 feature maps trained with and without FFR: L1 norm plot, with 64 feature maps in the feature of the nineteenth residual bock. (c) Visualization of the filters the first residual block in ResNet56 trained with FFR. From top to bottom, the filters are displayed according to their norm from small to large. Square denotes the filters (red, horizontal) and channels (orange, vertical) that have small magnitude and can be removed. Both VGG16 and ResNet56 are trained CIFAR-10.
4.2 STRUCTURED SPARSITY IMPROVEMENT
To further show the ability of FFR in improving structured sparsity, examine the relation between accuracy and structured sparsity. We compare VGG16 and ResNet56 with and without FFR trained on CIFAR-10. 10. After training. under the threshold and the corresponding channels. Then we calculate the accuracy and structured sparsity that is defined the percent of the parameters zeroed. Figure andand4b show the accuracy- sparsity trade-off curves of VGG16 and ResNet56 on CIFAR-10, 10, respectively. It can be seen that the network trained with FFR will not suffer accuracy degradation until has significantly large structured sparsity compared with the baseline case.
increasing threshold to zero the filters whose magnitude is
90
Baseline FFR(5e-7)
70
Accuracy
60
80
50
30
Baseline
20
FFR(2e-7)
0.0
0.1 02
0.3
04
0.6
0.7
08 0.9
1.0
0.0
0.1
0.2
0.3
0.4
0.
0.6
0.
0.8
0.9 1.0
Structured sparsity
Structured sparsity
(a) VGGI6
(b) ResNet56
Figure 4: Accuracy-sparsity trade off curve of (a) VGG16 and (b) ResNet56 trained with FFR (blue) and without FFR (orange, Baseline) on CIFAR-10.
5 EXPERIMENTS
5.1 IMPLEMENTATION
Models and datasets. To demonstrate the effectiveness of FFR in pruning, we perform exper- iments with VGGNet and ResNet on the CIFAR-10 (Krizhevsky & Hinton] 2009) and ImageNet Deng et al, 2009) datasets. The VGG16 a modified VGGNet in Simonyan Zisserman (015)): cullecconeeeted layers and only one collecconneeted layer for classification. We compare the network trained with FFR and without FFR as a baseline. We adopt weights initializa- tion in He et al. (015)).
remove
7

---

Under review as a conference paper at ICLR 2022
Training setting. FFR and baseline have the same training setting. For CIFAR- 10, the mini-bate size is 128. We train VGGI6 and ResNet56 200 epochs and set initial learning rate as 0.1 and divide it by 10 the 80th, 120th, and 160th epoch. For ImageNet, the mini-batch size is 256. We train ResNet50 90 epochs and divide initial learning rate 0.1 by 10 at the 30th and 60th epoch. We SGD optimizer with weight decay 5e-1 1 for CIFAR-10, 1 for ImageNet, and momentum 0.9. Note that Le regularization is implemented by applying weight decay in SGD. We ran our experiments with pytorch.
Pruning setting. We perform one-shot structured pruning: remove filters based on the magnitude and their corresponding channels in the next layer. For experiments on CIFAR-I0 and ImageNet, only fine tune the pruned model 30 epochs with learning rate Le-4 on CIFAR-I0 and Le-3 on ImageNet. For VGGNet, we prune the network directly, and for ResNet, we use zeros padding in the pruned dimension. The outputs of the shortcut and the last convolutional layer in the residual block in ResNet must have the same dimension because of the addition operation between them. Using zero padding.
remove the filters flexibly. Moreover, if all filters of the first convolutional layer are pruned, then the residual block can be pruned.
5.2 STRUCTURED PRUNING
We compare pruning results FFR with recent state-of-the-aa pruning methods in Figure 5lnd and Ta- We present ablation study on the hyperparameters k,, k2 and a rule on hyperparameter selection in Appendix A.5 A.5
bles I and 2
Results on CIFAR-10. Table shows the pruning results comparisons of VGGI6 and ResNet56 on CIFAR-10. For better comparison, in Figure 5 we plot error-paranne parameter reduction and error- FLOPs reduction trade-off curves of VGG16 trained with FFR together with the results of other methods shown in [ We can see that FFR is able to achieve larger pruning ratio in terms of parameters and FLOPs. Specifically, we prune VGG16 with 90.2% parameters reduction and 61.2% FLOPS reduction with a slight drop (0.66%) in accuracy, and ResNet56 at large FLOPSs reduction 60.6% with accuracy drop 1.05%. From the comparisons of VGG16 results in Figure 5 the FFR method has lower eгrоr reductions in parameter and FLOPs than most of the available methods, and achieves larger pruning ratio. Note that the DCP method (Zhuang al. 2018) conducts channel pruning and fine tunes the network stage by stage achieve lower error, whereas in FFR, we perform only one-shot structured pruning with few epochs of fine tune. We also visualize the feature maps of the reserved and pruned filters of the first convolution layer in VGG16 using FFR in Appendix
A.7
Figure
10
FFR
S5S
DCP
FPGM
Hrank
PR
baseline
8:0
1.50
7.50
Erratie
1.75
7.25
2.00
7.00
6.25
6.25
5.00
6,00
50
80
70
Param. Reduction(%)
FLOPS [educcionn96)
Figure 5: Ernor-parameter reduction and error-FLOPS reduction trade-off curves of VGG16 trained with FFR on CIFAR-10, and comparison with the results SSS (Huang & Wang 2018). DCP Zhuang et al. 2018), FPGM (He et al. 2019), Hrank (Lin et al. 2020), PR (Zhuang al. 2020
Results on ImageNet. Table 2alle2pa shows the pruning results comparisons of ResNet50 on ImageNet, which is large-sacle dataset. Although there is only one-shot pruning and few epochs fine-tuning in FFR, our FFR pruned ResNet50 achieves comparable parameters and FLOPs reductions with the recently proposed pruning methods.
Methodology comparison. Methodologically, our FER method manipulates the trajectory con- necting features of hidden layers, which is different from pruning methods based on feature maps
8

---

Under review as a conference paper at ICLR 2022
information, e.g. Hu al. (2016): Li al. 20200: Lin al. (2020) and is different from meth- ods that impose regularization the parameters, e.g. Huang & Wang (2018); Zhuang et al. (2020).
In terms of implementation simplicity, use SGD during training and adopt one-shot pruning
strategy instead of additional optimization steps (e.g., He et al. (019)) or iterative pruning Zhuang
al. 2018).
Table I: Pruning results on CIFAR-10. '_' represents result reported. 'M' represents lee.
Model
Method
Baseline Error (%)
Pruned Error (%)
Param. Reduction
FLOPS
Reduction
VGG16
Baseline (Ours)
6.10
6.10
0% (14.72M)
0% (313M)
SSS (Huang & Wang
2018
6.10
6.98
73.8%
41.6%
DCP Zhuang cf al. 2018
6.01
5.84
47.9%
50.0%
FPGM He et al. 2019
6,42
6.46
34.2%
Hrank (Lin et al. 2020
6.04
7.66
82.1%
65.3%
PR ZZuuagg al.
2020
6.12
6.08
54.0%
FFR (A1, K2 2e
6.10
6.76
90.2%
61.2%
ResNet56
Baseline (Ours)
6.60
6.60
0% (0.86M)
0% (126M)
CP (He al. 2017)
7.20
8.20
50.0%
DCP (Zhuagg et al.
2018
6.20
6.51
49.2%
49.7%
SFP (He et al.
2018
6.41
6.65
52.6%
FPGM (He et al.
2019
6.41
6.51
52.6%
Hrank (Lin
2020
6.74
6.83
42.4%
50.0%
PR ÇZhuang et al.
2020
6.20
6.17
47.0%
FFR (k1, K2
6.60
7.65
49.7%
60.6%
Table 2: Pruning results on ImageNet.
'represents result not reported. 'M' represents 1e
Model
Method
Top-1 Err. Increase (%)
Top-5 Err. Increase (%)
Param. Reduction
FLOPs Reduction
ResNet50
Baseline (Ours)
(28.44)
0 (9.72)
0% (25.56M)
0% (4089M)
SSS (Huang& Wang
2018)
4.30
2.07
38.8%
43.0%
DCP Zhuang ct al. 2018)
1.06
0.61
51.5%
55.6%
SFP (He et al. 2018)
14.01
8.27
41.8%
FPGM (He et al.
2019
1.32
0.55
53.5%
Hrank (Lin et al.
2020
4.17
1.86
62.1%
46.0%
PR (Zhuang al.
2020
0.52
54.0%
FFR (Æ1 K2 5e
2.68
1.40
52.3%
47.2%
CONCLUSIONS AND DISCUSSION
In this paper, propose simple and effective regularization method (FFR) from perspective of the data trajectory along the network. FFR smoothes the trajectory by imposing controls on the length and total absolute curvature of the feature flow, leading to significant increase of structured sparsity DNNs. We perform a sparsity amalysis FFR for VGGNet and ResNet to validate this method. Experimental results show that FFR can significantly enhance structured sparsity, which enables us to prune filters efficiently in one pass. Future work may include: (1) perform rigorous sparsity analysis for FFR, and (2) integrate FFR with other pruning methods further improve prun- ing results. The FFR method is a one-shot pruning method, in which the regularization is imposed on features instead of predetemmining the sparsity. (See more discussion appendix (A.9) Abla- tion study has been performed and rule has been summarized on selecting the hyperparameters. The framework feature flow proposed in this paper can be extended to other applications beyond pruning, e.g., network generalization (Drucker & Le Cun 1992 Raghu et al. 2017).

---

Under review as a conference paper at ICLR 2022
REPRODUCIBILITY STATEMENT
The details of experiments are given in section 5.1
REFERENCES
Jose M Alvarez and Mathicu Salzmann. Learning the number of neurons in deep networks. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and Garnett (eds.), Advances in Neural Information Processing Systems, volume 29. Curran Asso- ciates, Inc., 2016. URL
wew.eeeeeddiins..e.eeeeeeeee.e.e ips. wttp:aapppppppppppppppeeeaee cc/paperr2216//ile/ Paper. pdf
Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary dif- ferential equations. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa Cesa-Biande Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran As- sociates, Inc., 2018. URL htLhttss: http::::sp/pppppeeeeeeeeooee www.nngg....uureee neur ips. wtt.:/rrppp.....ppereeee///. wwtpppppe/211//1111111 6$986f6bbacec6666222266663333333
Paper.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei. Imagenet: A large- hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. leee, 2009.
Misha Denil, Babak Shakibi, Laurent Dinh, Marc Aurelio Ranzato, and Nando de Freitas. Predict- ing parameters in deep learning. In C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger (eds.). Advances in Neural Information Processing Systems, volume 26. Cur- ran Associates, Inc., 2013. URL https: /ttp:e/ei///ccceeeegeeooo neurips. cc/paper/2013/ htwpeeedddiinnnnnnnnnreeee
Emily Denton, Wojciech Zaremba, Joan Bruna, Yann LeCun, and Rob Fergus. Exploiting linear structure within convolutional networks for efficient evaluation. CoRR, abs/1404.0036, 2014.
H. Drucker and Y. Le Cun. Improving generalization performance using double backpropagation. IEEE Transactions Neural Networks, 36);9911997, 1992. doi: 10. 10.1109/72.165600
Weinan E. A proposal on machine learning via dynamical systems. Communications in Mathematics and Statistics, 5(1), 2017. ISSN 2194-6701. doi: 10.100775-00304007--000--
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neu- ral networks. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL httpt::t///ccc... //pp///e/ee/r.......
nnt//Eoumm? iidiiid-- D1337
Susan Gao, Xin Liu, Lung-Sheng Chien, William Zhang, and Jose M. Alvarez. VACL: variance- aware cross-layer regularization for pruning deep residual networks. In 2019 IEEE/CVF Interna- tional Conference on Computer Vision Workshops, ICCV Workshops 2019, Seoul, Korea (South), October 27-28, 2019, pp. 2980-2988 IEEE, 2019, doi: 10. 10.110//CCCW.2219900066 URL https: / //do: .r.oorg/ll wtt./////......//...... 1109/1
00360
Yunchao Gong, Liu Liu, Ming Yang. and Lubomir D. Bourdev. Compressing deep convolutional networks using vector quantization. CoRR, 4bs/1412.6115,, 2014. URL/attp://arxiv...
abs/1412, 0tt//412..1111 6115.
Yiwen Guo, Anbang Yao, and Yurong Chen. Dynamic network surgery for effi- cient dnns. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Gamett (eds.), Advances in Neural Information Processing Systems. volume 29. Curran Asso- ciates, Inc., 2016. URL. https: http::/:e///pcceeeenreeee 282314797102cen (5 poae
cc/ pttpc//pee//011//1eeeeee
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 28. Curran Asso- ciates, Inc., 2015. URL htt.atpp.s.pppoeee..sseexxx..... lec
10

---

Under review as a conference paper at ICLR 2022
Babak Hassibi and David Stork. Second order derivatives for network pruning: Optimal brain
surgeon. In S. Hanson, J. Cowan, and C. Giles (eds.), Advances in Neural Information Processing
Systems, volume 5. Morgan-Kaufmann Kaufmann, 1993. URL https::/proceeddiitt Cc/ /19ee//r/////11/ hot.:eeee000000ee536666662000200
6ab
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. 2015 IEEE International Conference on Computer Vision (ICCV), pp. 1026-1034, 2015. doi: 10.1 109/1CCV.2015.223.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In 2016 IEEE Conference on Computer Vision and Puttern Recognition (CVPR), pp- 770-778, 2016.
doi: 10.1 1119///PR.2016.999
Yang He, Guoliang Kang, Xuanyi Dong, Yanwei Fu, and Yi Yang. Soft filter pruning for accelerating deep convolutional neural networks, 2018.
Yang He, Ping Liu, Ziwei Wang. Zhilan Hu, and Yi Yang. Filter pruning via geometric median for deep convolutional neural networks acceleration. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4335-4344, 2019, doi: 10. 109/C 9,00477.
Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural net- works. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), Oct 2017.
Hengyuan Hu, Rui Peng, Yu-Wing Tai, and Chi-Keung Tang. Network trimming: A data-driven neuron pruning approach towards efficient deep architectures. CoRR, abs/1607.03250, 2016. URL http:htttt//.....oo/p
abs/1607 03250
Zehao Huang and Naiyan Wang. Data-driven sparse structure selection for deep neural networks. In Proceedings of the European Conference on Computer Vision (ECCV), September 2018.
Max Jaderberg, Andrea Vedaldi, and Andrew Zisserman. Speeding up convolutional neural networks with low rank expansions. In British Machine Vision Conference, 2014.
A. Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images. Master's thesis, Department of Computer Science, University of Toronto, 2009.
Hang Li, Chen Ma, Wei Xu, and Xue Liu. Feature statistics guided efficient filter pruning. CoRR, abs/2005. 12193, 2020. URL
12091
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for abs/1608.
efficient convnets. CORR, Abs//608.087100 2016. URL
08710
Lucas Liebenwein, Cenk Baykal, Harry Lang, Dan Feldman, and Daniela Rus. Provable filter pruning for efficient neural networks. In 8th International Conference on Learning Represen- tations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL
opw.reveevivveeee https::/oppenrevie
https:/
net for.mmid--Bxx00sss..
Mingbao Lin, Rongrong Ji, Yan Wang, Yichen Zhang, Baochang Zhang, Yonghong Tian, and Ling Shao. Hrank: Filter pruning using high-rank feature map, 2020.
Shaohui Lin, Rongrong Ji, Chenqian Yan, Baochang Zhang, Liujuan Cao, Qixiang Ye, Feiyue Huang, and David S. Doermann. Towards optimal structured CNN pruning via generative adversarial learning. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019. Long Beach, CA, USA, June 16-20, 2019, pp. 2790-2799. Computer Vision Foundation / IEEE, 2019. doi: 10.1109/CVPP 20 002900
thecvf. com/ content CVPR 20w9//ttllenaeee Towards Optimal Structured CNN Pruning Gee.Geeeeeieeeeee wwwpaaivvvvvvvvevveeeeeeeee Advervvrvvaa..... al Learning_CCoe 2019 www.pp.ppr...m
URL http::/openaceee
Baoyuan Liu, Min Wang, H. Foroosh. M. Tappen, and M. Penksy. Sparse convolutional neural networks. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp.
806-814, 2015. doi: 10.

---

Under review as a conference paper at ICLR 2022
Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang. Shoumeng Yan, and Changshui Zhang. Learn- ing efficient convolutional networks through network slimming. In ICCV, 2017.
Christos Louizos, Max Welling, and Diederik P. Kingma. Learning sparse neural networks through to regularization, 2018.
Yiping Lu, Aoxiao Zhong, Quanzheng Li, and Bin Dong. Beyond finite layer neural networks: Bridging deep architectures and numerical differential equations. CoRR, abs/ 2017.
URL
org/ abs/aa///// 1710.10121
Jian-Hao Luo, Jianxin Wu, and Weiyao Lin. Thinet: A filter level pruning method for deep neural network compression. In IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017, pp. 10068-5076 IEEE Computer Society, 2017. doi: 10.1 109/1V9/
2017.541. URL htt.::::://o..ogg//// 11t.: 2017. 2017.541
Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning convolutional neural networks for resource efficient transfer learning. CoRR, abs/1611.06440, 2016. URL
http://arxiv. abs http://www..orrraa///11.........
Maithra Raghu, Ben Poole, Jon Kleinberg, Surya Ganguli, and Jascha Sohl-Dickstein. On the expressive power of deep neural networks. In Doina Precup and Yee Whye Teh (eds.). Pro- ceedings of the 34th International Conference on Machine Learning, volume 70 of Proceed- ings of Machine Learning Research, pp. 2847-2854. PMLR, 06-11 Aug 2017. URL https: //proceeding ngs. ml wwtpppaassseeeeeeepeeeeep
ht.m
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogmition. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceed- ings, 2015. URL httt::://axxiv.or//eeee aa//aa/ss.19...///// 1556
Yehui Tang, Yunhe Wang. Yixing Xu, Dacheng Tao, Chunjing Xu, Chao Xu, and Chang Xu. SCOP: scientific control for reliable neural network pruning. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12. 2020, virtual, 2020. URL ptttps::/pprceediin neurips......p.... 7bcdf 7b8e02e301 F4091 fb6bc8 Abstr html
htc/paper/222//haas
Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured spar- sity in deep neural networks. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 29. Curran Asso- ciates, Inc., 2016. URL httpt:::///// http::s:s////ppedddsise hww.rroeeeednnn.....r.eeeeeeeeee nwwrips...p.... bottee/e/acc000555500--------0-0 http://sse...sspeppaaaeee/////// pctpp//aeee/221///leeee
pdf
Jianbo Ye, Xin Lu, Zhe L. Lin, and James Z. Wang. Rethinking the smaller-norm-less: heoommlles-informatiee assumption in channel pruning of convolution layers. CoRR, abs/1802.00022 2018. URLhttps
//arxiv..rr//abs//3322ooo
Xin Yuan, Pedro Savarese, and Michael Maire. Growing efficient deep networks by structured continuous sparsification. CORR, abs/2007. 15353,, 2020. URL htt.tpss///xxii..rgg///////
2007.15353
Hao Zhou, Jose M. Alvarez, and Fatih Porikli. Less is more: Towards compact cnns. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling (eds.), Computer Vision ECCV 2016 14th European Conference, Amslerdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV, volume 9908 of Lecture Notes in Computer Science, pp. 6762-67..00 Springer, 2016. doi: 10. 978- 978-3-319-46499-0)_e URL https: /tts://doo.. http:://ooooooggggg/oo 10001 3-----3-311----- 37177797778--331---4------ 40
Tao Zhuang, Zhixuan Zhang, Yuheng Huang, Xiaoyi Zeng, Kai Shuang, and Xiang Li. Neuron-level structured pruning using polarization regularizer. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 9875-9877. Curran Associates, Inc., 2020. URL ht http::/sp////ppppeppeccceoe .tw.:eeeeeddnng.......s.ee. www.ins..pp... cct
12

---

Under review as a conference paper at ICLR 2022
Zhuangwei Zhuang, Mingkui Tan, Bohan Zhuang, Jing Liu, Yong Guo, Qingyao Wu, Junzhou Huang. and Jin-Hui Zhu. Discrimination-awve aware channel pruning for deep neural networks. CoRR, abs/1810.11809, 2018. URL/http: /ttp:v///axivvvv/a wwt.://.rgggggabbbgee.eeee
abs / 11809
A APPENDIX
A.1 SETTING OF THE ILLUSTRATION EXAMPLE IN SECTION 3.1
To visualize the smoothing effect of FFR, we use a two-dimensional example. Figure Dc. the green and red clusters contain 50 paired data points in two dimensional space. The green cluster is evenly distributed a circle with center (2, 6) and radius 0.5. The red cluster obtained by exactly shifting the green cluster four units to the right and four units down, that is in a circle with center (6, 2). we use a ResNet to learn a translation mapping from the green cluster to the red cluster. The ResNet has five fully-com collectdd residual blocks. Here each residual block contains two linear layers and outputs a two-dimensional feature. We train ResNet without and with FFR to learn the mapping under the same setting for comparison. trajectory is obtained by starting from input data point, connecting five output features and ending with the output. From Figure Te. the trajectories of the network trained with FFR, represented by the orange curves, are shorter and more straight than those of the network trained without FFR, represented by the blue curves. We plot the feature flows of three test data for comparison. shows that FFR indeed effectively smooths the feature flow.
A.2 FEATURE MAPS NORM PLOT SECTION
We show the L1 norm plot of 256 feature maps generated by the fifth convolutional layer and 512 feature maps generated by the eighth convolutional layer VGG16 trained with and without FFR on CIFAR-10 in Figure 6a| and (6b respectively. Consistent with the claim in the main text, the net- work trained under FFR learns much more sparse features with zero-valued norm than the baseline network (VGGI6 trained without FFR). FFR improves the feature sparsity which contributes the structures sparsity in DNN.
Baseline
FFR
L1 norm
32
62
93
124
135
186
217
24B
Feature map index
(a) 256 feature maps of the fifth convolutional layer.
Baseline
FFR
L1 norm
Feature map index
(b) 512 feature maps of the eighth convolutional layer.
Figure 6: VGG16 feature maps trained with and without FFR on CIFAR-10: L norm plot.
13

---

Under review as a conference paper at ICLR 2022
A.3 FURTHER SPARSITY ANALYSIS ON THE LENGTH TERM FFR
In Secl4 we show that the proposed FFR method encourages structured sparsity of features in addi- tion penalizing the structured sparsity of the parameters. Here we give more details of how FFR improves the structured sparsity based on Eqs. 133 (16) in Sec
For W 1=*xewxk, X Rexaxb, assume that W & X R=xxxo, where @ denotes the convo- lutional operation, and n. P, are the dimensions of the output. We have
IW & X||
Σ n.P-q
@ X (ap
Σ T.P-4
Σ
W Wnee. & X Nolg
(17)
where ||||| denotes elementwise Li-norm and denotes the absolute value, and X Xo, by Rexwxh represents the subset in features X that produces the (p, 9)-th entry (W @ x) in each cannel of the output.
We first show that controlling | & X|| encourages both W and X to be small. Expanding tensor W into matrix and tensor X into vector, we rewrite the convolutional operation as matrix- multiplication:
[ & X|| = |
(18)
where || || is the elementwise norm. Thus we have
V V
(WX),
(19)
where sign(-) is the sign function elementwise applied to the matrix. It can be calculated that
x - WT Wig x) = || - s O.
(20)
Noticing that the direction of X is the increasing direction of the function 1| 11 or equivalently, the increasing direction the norm ||X||| = ||_]]. the above equation implies that minimizing I & X|| = || leads to decrease in || Similarly, we can have the conclusion that minimizing ||W < X|| leads to decrease in |W]].
In the meantime, under the regularization effects, in order to maintain the performance of the neural network as much as possible, the feature maps in X tend to have only few large norm feature maps (which capture the most representative information) and many zero norm feature maps, rather than feature maps with norms distributed evenly. One extreme example without expressiveness is noise which is uniform distribution. Therefore, the tradeoff between performance and sparsity of the network leads to high channelwise sparsity in features.
As result, the convolutional parameters W are trained to have high filterwise sparsity, since the filter Wn..:: that generates the zero feature map will become zero training.
That is why controlling || & X|| encourages W W....r and X to be small, which correspond to the parameters W wise sparsity and the feature X channel-wise sparsity, respectively.
Using the claim above, the length term in FFR penalizes I[ - I) & zı for convolutional block and encourages high channel-wist sparsity of xj and high filter-wise sparsity of W; - I, which leads high structured sparsity of Wi. For the residual block, the length term in FFR penalizes [|W,2 @ W.2 8 æ21l or ||al]] so improves the structured sparsity of the convolutional parameters and the features. See Eqs. 13 (6).
A.4 DISCUSSION ON THE SPARSITY EFFECT OF THE CURVATURE TERM IN FFR
In fact, in addition the effect that the curvature term keeps the trajectory from bending too much as shown in See 3.1 and [[ the curvature term also encourages more uniform partition of the trajectory from the input to the output, which makes it easier and more accurate learn with sparse parameters. This effect is associated with the "discretized" trajectory. This can be seen from the following illustrative example.
Consider two caputfcaaure-outppe trajectories as in Fig. 7 where dots represent the input, the hid- den features the output, and the dashed lines denote segments the trajectory whose length is
14

---

Under review as a conference paper at ICLR 2022
proportional the L1 distance. Here trajectories (a) and (b) have the same total length, the length term cannot distinguish them. Whereas trajectory (a) has smaller "curvature term" than that (b), thus it is preferred by the curvature term. In general, the longer L distance between features of adjacent layers, the less possible it is for that layer to learn with sparse parameters.
(a)
(b)
Figure 7:
Illustration of the sparsity effect of the curvature term in FFR. Trajectory (a) preferred by the curvature term over that in (b). The length is proportional to the L1 distance between features different layers.
A.5 ABLATION STUDY ON HYPERPARAMETERS k2
tune the hyperparameters k 22 with experiments. The choice of hyperparameters depends on the architecture of the DNN and the scale of the input: for the DNN with more layers and higher resolution input. we use smaller k1, 12.
We perform ablation study hyperparameters with different values. In Table B report results of ablation study based on the pruning experiments of VGG16 on CIFAR-10. We use the same threshold prune the network trained under FFR with different values of hyperparameters, and fine tune 30 epochs. We report error, parameters reduction and FLOPs reduction. We have also plotted accuracy-sparsity trade-off curves in Figure g for clearer comparison.
As shown in the table, the baseline (VGG16 trained without FFR) cannot preserve the accuracy un- der the same pruning and fine-tuning setting. From the table and Figures 8aj and [8b] the curvature term has a stronger impact on the sparsity improvement, and in the mean time, higher error increase, than the length term. Compared with applying the length term the curvature term alone, com- bining the length term and the curvature term can achieve higher accuracy and sparsity at the same time (see Figure Scp. Moreover, network trained under FFR with appropriate k,, h2 achieves large pruning ratio while maintaining the accuracy.
03
Accuracy
Easzline
Baseline
ITRIZe -21
#RR12e-1)
k1-22-6
K2-2e 5
k2=1e-8
k1+50:7
PRR126-71
k1=2e-7
k2-2e
K1-2e7
k1-18-
k2-1e-7
K2-207
02.
da 63 04 25 at
67
Structured spersity
Structured spersity
Structured
(a)
(b)
Figure 8: Accuracy-sparsity trade off curves of VGG16 trained under FFR with different hyperpa- rameters on CIFAR-I0: (a) l22 different k,, (b) k different K2, (c) k k2 work separately or together.
15

---

Under review as a conference paper at ICLR 2022
Table 3: Alation study on k 82. Pruning results of VGG16 on CIFARIO. 'M' represents 1e
Hyperparamerer
Error(%)
Param. Reduction
FLOPs Reduction
Baseline
6.10
0%(14.72M)
0%(311M)
11, lg 0
34.57
74.1%
35.0%
1 lee 7, 22 0
6.21
75.4%
34.4%
1ı 2e 7, K2 O
6.18
77.0%
37.2%
11 5e 7, 22
O
6.41
84.0%
44.3%
k1 = 1c
- 6, 22 O
7.08
91.8%
62.6%
2e 6, A-2 O
8.59
95.7%
70.6%
1 = 0, 1/ = 1e 7
6.21
77.1%
37.9%
k1 0, 12
2e 7
6.06
79.2%
38.9%
k1 = O, 12
5e
7
6.98
92.4%
67.2%
k O, 122
1e
6
8.01
96.3%
80.0%
K1 = O, 1/
2c
6
9.31
98.1%
90.0%
k1, l2 5e 8
6.36
76.5%
36.4%
ki, 22 le
6.45
77.5%
38.6%
k1, l2 20
6.32
85.0%
47.5%
A.6 MORE EXPERIMENTS STRUCTURED SPARSITY IMPROVEMENT
We further verify the effect of FFR method in terms of structured sparsity improvement using ResNet18, ResNet34 and ResNe150 on TinyImageNet. In Fig. g we plotted the accuracy-sparsity trade-off curves of FFR compared with the baseline similar those Fig. and these figures also show that FFR improves the structured sparsity and maintains the accuracy in the experiments.
Base
Base
Accuracy
Accuracy
Accuracy
93
12
Base
FFR
Filter sparsity
Filter sparsity
Filter sparsity
(a) ResNet18
(b) ResNet34
(c) ResNet50
Figure 9: Accuracy-sparsity trade off curves of (a) ResNet18 (b) ResNer34 (c) ResNet50 trained with FFR and without FFR (Base) on TinylmageNet.
A.7 FEATURE MAPS VISUALIZATION
(a) Input image
(b) Feature maps of preserved filters
(c) Feature maps of pruned filters
Figure 10: Feature maps visualization of the first convolutional layers VGG16 using FFR.
16

---

Under review as a conference paper at ICLR 2022
A.8 FEATURE MAPS VISUALIZATION OF DIFFERENT LAYERS
We visualize the feature maps of different hidden layers in VGG16 with and without FER on CIFAR- 10. These figures show that in the baseline, the features of each layer contain various less informative feature maps that can be pruned, and FFR selects the most representative feature maps and lets other feature maps be zeros.
Therefore, FFR is indeed able to learn highly sparse structure while maintaining the representation power of extracting high-level features from low-level features.
A.9 COMPARISONS WITH METHODS WITH PREDETERMINED PRUNING RATIO
Our FFR method performs one-shot pruning after training with regularization, and different prun- ing or sparsity ratios can be achieved by adjusting the hyperparameters instead of direct select- ing the pruning ratio. Such indirect control of pruning ratio is common in many other one-shot regularization-baat pruning methods such as SSS (Huang & Wang 2018) and PR (Zhuang et al.] 2020). In addition, parameters in different layers in our method.
also obtain the desired sparsity using different thresholds prune the
On the other hand, the pruning methods that can predetermine the pruning ratio may also have the limitation of empirically adjusted parameters or other limitations. For example, the ratio of channels to prune in each layer may be user-specified such as SFP (He et al. (018)) and Hrank (Lin et al. 2020), and the compression ratio in each layer is usually selected by empirical studies these methods. Another type of ratio- cetoo--reeeemmeed-meeeeee iteratively until the predefined pruning ratio is achieved, e.g. ThiNet (Luo et al. (017), which is more complicated than the one-shot pruning.
is to train and prune the model
17

---

Under review as a conference paper at ICLR 2022
Figure 11: Feature maps from the 3rd layer in VGG16 trained with FFR (a) and without FFR (b) on
CIFAR-10. 10.
18

---

Under review as a conference paper at ICLR 2022
Figure 12: Feature maps from the 8th layer in VGG16 trained with FFR (a) and without FFR (b) on
CIFAR-10 10.
19


Reviews:
[{"review_id": "uVqXN8xxs3h", "reviewer": "Reviewer_wyu7", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I like the basic premise that encouraging simple mappings in networks 1) doesn't seem to hurt and 2) seems to lead to highly prunable models. I'm not super familiar with the latest developments in the crowded pruning literature and will have to rely on other reviewers on that. \n\nI want to underline a connection to some seemingly unrelated topics. The present idea seems to extend an earlier, frequently-reinvented idea that a network's output should change only a little when the input changes a little. This was probably first coined by Drucker and LeCun as \"double backpropagation\" [1], and has been heavily used in adversarial learning (input regularization [2]) and GAN training (R1 regularization [3]) in recent years. This was furthermore explicitly relied on in StyleGAN2 [4] for trying to guarantee a simple, well-behaved generator mapping between a latent space and output images (path length regularization). Now, I think your method goes further than any of these by regularizing entire paths through the network instead of just input --> output, thus potentially offering further upside in many applications. That seems like an exciting prospect.\n\nThat said, I think the authors need to contrast the method with \"trajectory regularization\" [5], which was put forward in yet another context, but would seem to do a closely related thing by encouraging simpler representations by regularizing transitions inside a network. It will be important to draw the connection to assess how far beyond pruning the novelty extends.\n\nIn my opinion Fig 1 should come sooner than page 4. It is useful mainly as a high-level intuition builder. Figures 3 & 4 are good. Typo: \"Strucuted sparsity\" (Fig 4).\n\nI did not check all the equations.\n\n\n[1] Improving generalization performance using double backpropagation, IEEE Trans. Neural Networks, 3(6):991–997, 1992.\n\n[2] Improving the adversarial robustness and interpretability of deep neural networks by regularizing their input gradients, https://arxiv.org/abs/1711.09404\n\n[3] Which training methods for GANs do actually converge?, https://arxiv.org/abs/1801.04406\n\n[4] Analyzing and Improving the Image Quality of StyleGAN, https://arxiv.org/abs/1912.04958\n\n[5] On the Expressive Power of Deep Neural Networks, https://arxiv.org/abs/1606.05336", "summary_of_the_paper": "This paper deals with network pruning from a fresh perspective. It essentially argues that networks that implement simple mappings are more amenable to pruning. The authors support this claim with experimental evidence. Personally I think this idea is intellectually appealing, and may have uses beyond the network pruning setting. Some concerns related to prior art linger.", "summary_of_the_review": "An intellectually stimulating take on network pruning, with possible applications elsewhere.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "epryQkJegIG", "reviewer": "Reviewer_D39H", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:  \n1. The feature flow regularization method is inspiring.  The feature flow regularization method (FFR) is simple and effective, and create smooth  consecutive features.   \n2. Authors provides detailed analysis in Sec.4 to show the method is valid.   \n  \nWeaknesses:   \n1. The curvature of two-order regularization needs more illustration. Why we need this item to create sparisty? Besides, ablation studies on the role of length and curvature are absent.    \n2. As we all know, Convolutional networks were inspired by biological processes. Individual cortical neurons respond to stimuli only in a restricted region of the visual field known as the receptive field. At the bottom level of CNN, the features are basically similar,  such as the edges. The higher, the more features of objects can be extracted (wheels, eyes, etc.), the different advanced features are finally combined into corresponding features, allowing CNNs to accurately distinguish between different objects.   \nHowever, with constraints on contingous features and two-order features, the representation power of intermediate features will be reduced significantly. The feature maps of CNNs are used to explore the high-level distinctiveness of inputs. So, more explantations about the FFR will make this paper clear.  \n3. The results of experiments in Table1,2 have no advantage in contrast to other methods. Expecially in Table.2, the baseline of resnet-50 on Imagenet is 71.56\\%, which is significantly lower than 76.1\\% officially.  \"SCOP: Scientific Control for Reliable Neural Network Pruning\" in Neurips 2020 had competitive results, and constastiation should be provided.\n4. Table.1 and Table.2 is not consistency. Table.1 reports the error rate, but Table.2 presents the accuracy.", "summary_of_the_paper": "This paper proposes a regularization strategy for learning sparse deep CNN models. The Feature Flow Regularization (FFR), penalizes first and second order changes in the intermediate features between consecutive layers of the network. The intention of this regularization is to  smooth the evolution of features and make sparse weights. ", "summary_of_the_review": "The paper proposes a regularization loss that penalizes L1 distances between consecutive feature maps that are projected to the same dimension. This loss leads to smooth features and this characteristic is unclear to the target of classification.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "R7uVNYJ5y5b", "reviewer": "Reviewer_Ac8a", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The method seems to be original for structured sparsity and pruning and is rather simple. The experimental comparison is with SOTA methods, and the results show that its performance is reasonable.\n\nThe paper provides some analysis for the connection of the proposed regularization and sparsity but unfortunately the analysis is not really rigorous. Moreover, there is no discussion and analysis for the curvature term. Even in the experiments, there is no ablation study to dissect the effect of the two regularization terms. Although the presentation of the method and Eqs. 5 and 7 include separate hyperparameters for the two terms, there is no experiment in the paper to investigate unequal values for these hyperparameters.\n\nThe experiments are a bit limited, as they are only on two datasets and two architectures. While the results on CIFAR-10 are somewhat encouraging, overall, the results do not seem strongly in favor of the proposed method, especially on ImageNet.\n\n Also, one disadvantage of the method is that a user cannot select a desired pruning or sparsity ratio, and the sparsity is indirectly controlled by hyperparameters which are not adequately investigated in the paper. \n", "summary_of_the_paper": "The paper proposes a new method, called feature flow regularization, for structured sparsity and pruning in deep neural networks. In the proposed method, the length and curvature of trajectories connecting the features of adjacent hidden layers are penalized which is claimed to implicitly result in structured sparsity. Experiments compare the method with several state-of-the-art methods on two datasets and architectures.", "summary_of_the_review": "The proposed method seems original, but in my view the theoretical analysis and experimental support are not strong enough and need improvement. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The submission proposes "feature flow regularization" during training to enforce (approximately) sparse network weights which can then be post-hoc pruned.  The form of the regularizer is reasonably well motivated, and the method seems interesting.  The reviewers were split on this, with two recommendations for "marginally above" and one for "marginally below" the threshold of acceptance.  I therefore read the paper in detail, in addition to reading the reviews, rebuttal, and private reviewer comments.  The appendix on the sparsity-accuracy tradeoff and its relationship to the hyperparameters k1 and k2 is an interesting experiment, and overall the authors were very engaged in the reviewing process.

In an initial reading, the term trajectory is indeed vague, although it is presented as a definition.  This ambiguity is reflected in the reviewer discussion where in response to Reviewer wyu7, the authors indicate that there are two different meanings of the word in different papers that are being confused.  In a mature presentation, these definitions should probably be given mathematically early on in a formal definition box, but this would require significantly tightening up the mathematical notation early on.

The results table does not show that the proposed method Pareto dominates other methods (accuracy, sparsity, and latency), which themselves are necessarily limited due to the very high number of published papers on network pruning.  Furthermore, some of the selected comparisons appear to be optimizing for different metrics rather than network sparsity, e.g. DCP reports better accuracy for VGG-16 after pruning is applied.  This indicates that the proposed method is somehow in the crowd, but does not seem to show a clear consistent improvement over SOTA.

Analysis in Appendix A.3 does not really depend on which kind of norm is used - the same conclusion will be reached that ||X|| decreases, while structured sparsity, e.g. with expected sparsity rates, is dependent on the kind of norm.  As such, it's OK, but not a particularly specific result.

On the whole, this indicates that the paper is interesting, but borderline with room for concrete improvements that go beyond the scope of a simple refinement for a camera ready presentation.
Decision: Reject