LAYER SPARSITY IN NEURAL NETWORKS
Authors: Mohamed Hebiri
Johannes Lederer
Venue:  (2021)

Abstract: Sparsity has become popular in machine learning, because it can save computational resources, facilitate interpretations, and prevent overfitting. In this paper, we discuss sparsity in the framework of neural networks. In particular, we formulate a new notion of sparsity that concerns the networks’ layers and, therefore, aligns particularly well with the current trend toward deep networks. We call this notion layer sparsity. We then introduce corresponding regularization and refitting schemes that can complement standard deep-learning pipelines to generate more compact and accurate networks.

Under review as a conference paper at ICLR 2021
LAYER SPARSITY IN NEURAL NETWORKS
Anonymous authors
Paper under double-blind review
ABSTRACT
Sparsity has become popular in machine learning, because it can save computational resources, facilitate interpretations, and prevent overfitting. In this paper, we discuss sparsity in the framework of neural networks. In particular, we formulate a new notion of sparsity that concerns the networks' layers and, therefore, aligns particularly well with the current trend toward deep networks. We call this notion layer sparsity. We then introduce corresponding regularization and refitting schemes that can complement standard deep-learning pipelines to generate more compact and accurate networks.
INTRODUCTION
The number of layers and the number of nodes in each layer are arguably among the most fundamental parameters of neural networks. But specifying these parameters can be challenging: deep and wide networks, that is, networks with many layers and nodes, can describe data in astounding detail, but they are also prone to overfitting and require large memory, CPU, energy, and so forth. The resource requirements can be particularly problematic for real-time applications or applications on fitness trackers and other wearables, whose popularity has surged in recent years. A promising approach to meet these challenges is to fit networks sizes adaptively, that is, to allow for many layers and nodes in principle, but to ensure that the final network is "simple" in that it has a small number of connections, nodes, or layers (Changpinyo et al., 2017; Han et al., 2016; Kim et al., 2016; Liu al., 2015; Wen et al., 2016).
Popular ways to fit such simple and compact networks include successively augmenting small networks (Ash, 1989; Bello, 1992), pruning large networks (Simonyan & Zisserman, 2015), or explicit sparsity-iduccing regularization of the weight matrices, which we focus on here. An example is the l fo-rr,, which can reduce the number of connections. Another example is the formrr grouped over the rows of the weight matrices, which can reduce the number of nodes. It has been shown that such regularizers can indeed produce networks that are both accurate and yet have a small number of nodes and connections either in the first layer (Feng & Simon, 2017) or overall (Alvarez & Salzmann, 2016; Liu al., 2015: Scardapane et al., 2017). Such paarstt--nnucing regularizers also have a long-standing tradition and thorough theoretical underpinning in statistics (Hastic et al., 2015).
But while sparsity on the level of connections and nodes has been studied in some detail, sparsity on the level of layers is much less understood. This lack of understanding contrasts the current trend deep network architectures. which is supported by state-of-the-us performances of deep networks (LeCun et al., 2015; Schmidhuber, 2015), recent approximation theory for ReLUU acti- vation networks (Liang Srikant, 2016; Telgarsky, 2016; Yarotsky, 2017), and recent statistical theory (Golowich et al., 2017; Kohler & Langer, 2019; Taheri al., 2020). Hence, better under- standing of sparsity on the level of layers seems to be in order.
Therefore, we discuss in this paper sparsity with special emphasis on the networks' layers. Our key observation is that for typical activation functions such as ReLU, a layer can be removed all its parameter values are non-negative. We leverage this observation in the development of new regularizer that specifically targets sparsity on the level of layers, and we show that this regularizer can lead to compact and more accurate networks.

---

Under review as a conference paper at ICLR 2021
Our three main contributions are:
1. We introduce new notion of sparsity that we call layer sparsity.
2. We introduce a corresponding regularizer that can reduce network sizes.
3. We introduce an additional refitting step that can further improve prediction accuracies.
In Section 2. speciff our framework, discuss different notions of sparsity, and introduce refitting scheme. In Section 3, we establish a numerical proof of concept. Section 4, we conclude with discussion.
2
SPARSITY IN NEURAL NETWORKS
We first state our framework, then discuss different notions of sparsity, and finally introduce a refitting scheme.
2.1 MATHEMATICAA FRAMEWORK
To fix ideas, first consider full fully-connected neural networks that model data according to
y: f W [" mx]]] + u,
where i E (l..... n indexes the n different samples, E R is the output, o; E Rd is the corresponding input with d the input dimension, is the number of layers, W E RP xP5+1 for
[I.....{} are the weight matrices with p1 = and Pi+1 = d, f² R + R for j E {1......} are the activation functions, and u, IR is the random noise. Extensions beyond fully-connected networks are straightforward-ss see Section 2.5.
We summarize the parameters in W == (W W E V = (V (VL, ,..., V4 V RPj [ppx*p;11} }, and we write for ease of notation
= =
(2)
for V E V.
Neural networks are usually fitted based on regularized estimators Lagrange
W E
argmin DDtaFit(yy Un. 1...... Fx, + h []}
(3)
Vev
or constraint form
W E argmin DataFit(yi...... Un. 2 Vev A
x,,}}
(4)
where DataFit : Rn x Raxd is a data-fitting function such as least-squares Σt= (yi [a and h V - [0, oo) regularizer such as the elementwise & fi-norm EES,/ (zl) We are particularly interested in regularizers that induce sparsity.
2.2 STANDARD NOTIONS OF SPARSITY
We first state two regularizers that are known in deep learning and the corresponding notions of sparsity.
Connection sparsity
Consider the vanilla & regeulrrre
P541 Σ
h v
Σ
I(V))uwl
where C E [0, 0o)' is vector of tuning parameters. This regularizer is the deep learning equivalent of the lasso regularizer in linear regression (Tibshirani, 1996) and has received considerable attention
2

---

Under review as a conference paper at ICLR 2021
sparsity
connection sparsity
node sparsity
layer sparsity
combined sparsity
output
output
output
output
output
input
input
input
input
input
Figure I: feedforward neural networks with different types of sparsity
recently (Barron & Klusowski, 2018; 2019; Kim al., 2016). The regularizer acts each individual connection, pruning a full network (first network from the left in Figure 1) to a more sparsely connected network (second network Figure 1). We, therefore, propose speak of connection sparsity.
Node sparsity
Consider grouped version of the above regularizer
A I
(V))
[0, oo)' is again a vector of tuning parameters. This regularizer the deep learning equivalent of the group lasso regularizer in linear regression (Bakin, 1999) and has received some attention recently (Alvarez & Salzmann, 2016; Feng & Simon, 2017: Scardapane al., 2017). The regularizer acts on all connections that go into node simultaneously, rendering entire nodes inactive (third network in Figure 1). We, therefore, propose to speak node sparsity.
where N
2.3
LAYER SPARSITY
We now complement the two existing regularizers and notions of sparsity with new, third notion.
Layer sparsity
Consider the regularizer
Pj Ps+1
h M'[ = Σ 66 | 12,- 1=1
Σ
(neg((V))
(5)
where r E [0, 00) is vector of tuning parameters, and neg[a] = min(a, 0} the negative part of real value R. This regularizers does not have equivalent in linear regression, and it is also new in deep learning.
We argue that the regularizer give rise gled along the layers according to
type of sparsity. The regularizer be disentan-
h M Σ ah
with
Ps Ps+1
negl (V)) )vw
for E (1,...,-- 1)
3

---

Under review as a conference paper at ICLR 2021
We then focus an individual inner layer, that is, layer that corresponds to index E {2.....
1}. To fix ideas, we consider the popular ReLU activation (Glorot et al.. 2011; 1; Hahnloser, 1998; Hahnloser et al., 2000; Salinas & Abbott, 1996); in other words,
consider (f)),(t) := fReLU := Pi}. J. and & E R (the activation of the output layer can be arbitrary). is now easy to show that the regularizer h indeed induces sparsity on the level of layers.
max 0} for all j E (2...., 1).
E (1.... .P.P
Theorem (Layer Sparsity). Consider {2..... - 1}, and define merged weight matrix as Vi-1d = Vi-1ys E RP: 1XP512, It holds that
A - =0
f fー [V* f 1- [1)-111 =] for all 2 E 00)
Proof of Theorem 1. If [EAl"i[']] 0, then (V)) gm 2 0 for all q E (I.....,pp). E {1,..., P5+1) }.
Hence, it holds for all q {1...., } that (V'z), 2 0 and, therefore, that ('V 2] Viz. The theorem follows then by the fact that Vi-1J = V
key property of the ReLU function is positive homogeneity. That positive homogeneity can allow for moving weights between layers had been observed in Barron & Klusowski (2018); here, the positive homogeneity to merge layers. The idea is as follows: h 0 means in view of the stated theorem that we redefine the network of depth I network of depth / by removing the function f', replacing the weights Vj-1 by ソソートツ, and then removing the jth layer altogether.
Theorem can be applied sequentially to neighboring layers; hence, the regularization can merge not only one but many layers into one. In conclusion, our new regularizer ht acts on all nodes and connections of each layer simultaneously, rendering entire layers inactive (fourth network in Figure 1). We, therefore, propose to speak of layer sparsity.
The concept of layer sparsity and Theorem in particular do not hinge on the exact choice of the regularizer in (5): one can take any function h that can be disentangled along the layers as described and that ensures the fact that h h (V3)] implies (Vn),((V)).. 2 O.
We illustrate layer sparsity with two examples.
Example (Identity Activation). We first highlight the meaning of layer sparsity in a simplistic setting that does not rely on Theorem We consider identity activation, that is, (f) ) for all j 12....,1J. E (1,....P5h and I. E IR. The networks in (2) can then be written as
fv[z,]= [Vy].
In other words, the initial l-layer network can be compressed into a one-layer network with activation function f² and parameter matrix v'...V' E glxd, This setting with identity activation is, of course, purely academic, but it motivates an important question: can parts of networks be compressed similarfy in the case of ReLU?
Theorem I gives an answer to this question: if [V3] = 0, then the jth and (j 1)th layers and non-negative imput, the
can combined. In the extreme case hv=2( [ network can be condensed into a one-layer network just in the linear case. In this sense, one can understand our layer regularizer is as measure for the networks' "distance to linearity." We detail this further in the following example.
Example 2 (ReLU Activation). We now illustrate how layer sparsity compresses and, therefore, simplifies networks in the case of ReLU activation. We fix an initial network IN parameterized by N E V. We identify the active layers of the network by
S S=S = (2..2...,111) NWI(N (1,1)
Thus, S and {l.....(} .0) \ always consider the input and output layers active.) The level of sparsity, that is, the number of active layers, s := [S] <
contain the indexes of the relevant and irrelevant layers, respectively. (We
Observe first that the theorem's restriction z's that have non-negative elements makes sense: by the definition of fneLU. the outputs of every ReLU layer are non non-negative. We now denote the indexes in S in orderly fashion: Ji:-..-js is E S such that jr < < js l. We then define scaled
4

---

Under review as a conference paper at ICLR 2021
versions of the corresponding merged matrices: j- S or j. E (1,1), do the "trivial merge" M = Noi RPh 12;; otherwise. we do the "non-trivial merge"
M := N .... 1BP
In other words, we merge all irrelevant layers between the Ji- and jath layers into the jath layer. We can then compress the data-g aata--enerating model in (1) into
yi f M f [Mex:]] +
with M == (M M E Vs == (V = (V v*): Vi E 1R 1183,, 123,, (.). Formulated differently, can condense the original network according
[N(2:] - [x.] f | f (.. [M
[M
that we can formulate the initial ReLU activation network with layers as a new ReLU activation network with layers.
The new network is still ReLU activation network but has smaller number of layers if l and, consequently, smaller number of parameters in total: the total number of parameters in the initial network is (p=( x Py+1), while the total number of parameters in the transformed network is only Ei (P X Pj+t1).
Our concept for regularizing layers is substantially different from existing ones: layer-wise regularizer induces weights to be non-negative, whereas existing layer-wise regularizers induce weights be zero (Wen et al., 2016, Section 3.3). The two main advantages of our approach are that it (i) does not require shortcuts to avoid trivial networks and (ii) does not implicitly enforce connection or node sparsity. We thus argue that our layer sparsity a much more natural and appropriate way to capture and regularize network depths.
Layer sparsity more closely relates to ResNets (He et al., 2016). The recent popularity ResNets is motivated by two observations: 1. Solvers seem to struggle with finding good minima of deep networks; even training accuracies can deteriorate when increasing the number of layers. Allowing for linear mappings that short-circuit parts of the network seem to help solvers in finding better minima. From our viewpoint here, one can argue that ResNets use these linear mappings to regulate network depths adaptively and, therefore, related to layer sparsity. But importantly, ResNets even more complex than the networks they are based on, while our notion simplifies networks.
verify again readily, all three regularizers convex, any combination of them is also convex. Such combinations can be used to obtain networks that are sparse in two or all three aspects (last network in Figure 1). In this sense, the different notions of sparsity are not competing but rather complementing each other.
Since,
2.4 REFITTING
Sparse networks can be used directly, but they can also be a basis for further optimization: one can adjust the network architecture according the non-zero pattern of the sparse network and then re-estimate the parameters of this smaller network. Such strategies are well-known in statistics and machine learning under the name refitting (Lederer, 2013: Chzhen et al., 2019). The theoretical underpinning of these strategies is the insight that regularization creates a bias that in certain cases-cal be alleviated by subsequent "unbiasing" step. In deep learning, refitting has been studied recently under the name lottery-ticket hypothesis (Frankle & Carbin, 2019).
We now formulate version of refitting for layer sparsity. We stay the framework Example to keep the notation light.
Example 3 (ReLU Activation Cont.). Consider the model in (1) with the specifications of Example 2, and consider corresponding layer- sparse estimator W of the parameters such (3) with h ht. In line with (6), we denote the set of the active layers by
S = S(W) {i E (1, (1....,L 1 13 1 # u
5

---

Under review as a conference paper at ICLR 2021
and the corresponding parameter space of the condensed network by Vs Vs(W (V
(V'...., V*) V E [Po--++>Xh+tt,,, where == |S). The least-squares refitted estimator for the parameters in the condensed network then
- sv(m
Ws E argmin VEVs
Hence, the estimator W complemented with least-squares refitting yields the network
fw [ f"
W f [.. [W3s xi]]
(8)
This strategy corresponds the "one-shot approach" in Frankle & Carbin (2019); could extend it along the lines of their iterative approach, but the numerical results indicate that this is not necessary. Also, in contrast the results in Frankle Carbin (2019), our results indicate that keeping the initialization is not necessary either.
2.5 EXTENSIONS BEYOND FULLY-CONNECTED NETWORKS
We have illustrated our ideas with feedforward networks that have fully connected layers, but the principles of layer sparsity apply much more generally. Consider fixed hidden layer with index j . 1}. In the fully-connected networks (2), this layer corresponds to function [] with weights V RPsxP5+ We now generalize these functions to
f
v k
b'
with weights V (Vil Vim Mi Mam and bias b' E B', and with arbitrary nonempty subsets M x x M C xp,+1 and Bi c RRs, The corresponding layer-sparse regularizer is then
Pi Ps+1 112
P5 (neg[(b) (b²) (bb) )w.)
h V
ΣΣΣ
(neg((VVk)) 1) +
We can confirm immediately that this regularizer has the same effect as its analog in the fully- connected case. Indeed, under the assumptions of Theorem 1, it holds that
f--1
v--1g f
V
jikk R + b¹
=f 1 1a b for all x 10, P:11
where V/-1d- := Σm= Vi-i 1X2=11 and b--1j = Vi-1カ E RP--1, In other words, if the value of the regularizer is zero, the jth layer can be merged into the (j 1)th layer as before.
These observations highlight the fact that layer sparsity applies very generally: it only requires the properties of ReLU-type activations and the linearities that exist within most types of layers. As concrete application, layer sparsity can compress networks that have convolutional layers, where specifies the number of feature maps and Mim the non-zero patterns of the filters.
EMPIRICAL STUDY
We now confirm in a brief empirical study the fact that layer regularization can 1. improve prediction accuracies and 2. reduce the number active layers.
3.1 ARTIFICIA DATA
We start with artificial data.

---

Under review as a conference paper at ICLR 2021
2. 12 100
10
25
Fw
Rw
Rw
0.9
SW
Sw
0.3
Method
inse
insé
Tuse
insé
Ls
1.123 (1.396)
10 (0) 1.112 (1. 148)
1010] 1 10.0061113330 (1.31))
(0(0) 0.989 (1 263)
26 (0 0.958 (1 260)
25 (0)
0.018 (0.023)
(0) 0.035 (0.616)
(0) 0.203 (0.830)
(0) 0. (1.020)
(0)0..1 0.147 (0.985)
25 (0)
IL.
0.006 (0.011)
1 (2) 0.008 (0.023)
3 0.97 (1.272)
9 (10) 0.003 (0.024)
2 (4) (1.012)
(
FLS
0.007 (0.013)
(1)) 0.008 (0.128)
2 0.36 (1.137)
(9) (0.009 (0.985)
2 (5) (1.120)
(13)
Table 1: encouraging layer sparsity can reduce the prediction error and the model complexity
3.1.1 SIMULATION FRAMEWORK
We generate data according to the model in (2). The most outside activation function f the identity function, and the coordinates of all other activation functions f f are ReLU functions. The input vectors .1....;. In are jointly independent and standard normally distributed in dimensions; the noise random variables u1:....: M., are independent of the input, jointly independent, and standard normally distributed in one dimension. For a given sparsity level sw E [0, 1]. a vector 8 E {0. 10-1 with independent Bernoulli distributed entries that have success parameter sw is generated. The entries of the parameter matrix W are sampled independently from the uniform distribution on (-2, 2). and the entries of the parameter matrices W ...., W are sampled independently from the uniform distribution on (0, 2) if sj = 0 and on (-2,2) otherwise. Hence, the parameter sw controls the level of the layer sparsity: the smaller sw, the higher the network's 's layer sparsity.
In concrete numbers, the input dimension is d = 2, the network widths are P2 .... the number of hidden layers I E 25.. and the sparsity level is s E 0.3, Our settings and values represent, of course, only a very small part of possible networks in practice, but given the generality of our concepts, any attempt of an exhaustive simulation study must fail, and the simulations at least allow us (i) to corroborate our theoretical insights and (ii) to indicate that our concepts can be very useful in practice.
5,
Datasets of 150 samples are generated; 100 of the samples are assigned to training and the rest to testing. The relevant measures for an estimate W of the network's parameters are the empirical mean squared error
mse
msee
W
Σ (u - (w(w>)) [a=))2
(U)+)T
over the test set T with cardinality
[T|
50 and the level of sparsity among the hidden layers
$
[[
{ (L....,/--11 13 : h [W' + #0}
Reported are the medians (and third quantiles in paranthesis) over 30 simulation runs for each setting.
3.1.2 METHODS
Our first method (SLS) is a standard least-squares complemented with the layer regularizer (5) in Lagrange form W. The baseline for this estimator is vanilla least-squares (LS), Since our estimator in contrast to squarrs allows for merging layers, we can also complement it with our refitting scheme of Section 2.4 (FI.S). The baseline for our refitted estimator is the least-squares estimator that "knows" the relevant layers beforehand (IT.S), that is, a least-squares on the relevant layers Vs W] with W the true parameter- see Example 2. The latter estimator cannot be used in practice, but can serve as benchmark here in the simulations.
The objective functions are optimized by using mini-batch gradient descent with batch size 10, learning rate 10-2 ? and number of epochs 200 (for = 10, Sw = 0.1, 0.3), 300 (for = 10. Sw 0.9), 400 (for I 25, sw 0.1), and 500 (otherwise). The tuning parameters (r"); are 0.2
(for = 10. sw = 0.1), 0.12 (for I = 10, sw = 0.3). 0.07 (for I = 10, Sw = 0.9), and 0.05 (for 1 25).
7

---

Under review as a conference paper at ICLR 2021
3.1.3 RESULTS
The numerical results show that our layer-regularized version SLS can improve on the prediction accuracy of the standard least-squares LS considerably (inse-columns of the first and second rows in Table 1). The results also show that the refitting in FLS can improve the prediction aceuracy further, and that refitted estimator can rival the infeasible ILS in terms of prediction (mse-columns of the third and fourth rows). The results finally show that the layer regularization can detect the correct number of layers (5-columns of the third and fourth rows). In summary, our layer-regularized estimator outmatches the standard least-squares, and the refitted version of our estimator rivals the infeasible least-squares that knows which are the relevant layers beforchand- both in terms of prediction accuracy and sparsity. Hence, layer regularization can condense networks effectively,
The results also reveal that the prediction accuracies of our estimators increase with By decreasing (mse-columns across different sw). This trend is expected: the higher the layer sparsity, the more reyur-regularizattio condense the networks. This behavior is confirmed in the sparsities (s- columns across different sw). In other words, the layer regularization is adaptive to the true layer sparsity.
The tuning parameters (rh) have been calibrated very roughly by hand. We expect that a more careful calibration r based on pross-validatio,, for example, accentuates the positive effects of layer sparsity even further. But since our goal in this section general proof of concept for layer sparsity rather than the optimization of a specific deep learning pipeline, we do not pursue this further here.
The tuning parameters of the descent algorithm, such as the batch size, number of epochs, learning rate, and forth, have also been calibrated very roughly by hand. An observation is the fact that all methods can sometimes provide accurate prediction if the number of epochs is extremely large, but our layer-regularized methods SLS and FLS generally lead to accurate prediction after much less epochs than their unregularized counterpart LS. This observation indicates that layer regularization also impacts the algorithmic aspects of deep learning beneficially.
3.2 REAL DATA
We now turn to real data. Specifically, we consider subsamples of MNIST (LeCun et al., 1998), FashionMNIST (Xiao et al., 2017), and KMNIST (Clanuwat et al., 2018).
3.2.1 SETUP
For each of the three data examples, the training data consists of 10 000 images sampled uniformly at random; the test data also consists of 10 000 images. The network consists of = 10 fully- connected layers; the hidden layers have width P2 have been outperformed image classification by other, more intricate pipelines, they still provide decent results (even on the subsetted data) and are perfectly suited for illustrating our ideas.
pt 50. While fully-connected networks
The baseline is cross-entropy (CE), which is LS but with the least-squares loss replaced by the cross-entropy loss. Our layer-sparse method refitted cross-entropy (FCE), which is the pendant of FLS, that is, cross-entropy with additional layer-sparse regularization and refitting.
The objective functions are optimized by using mini-batch gradient descent with batch size 100, learning rate 10-3, and number of epochs 100. In line with theoretical considerations (Lederer & Vogt, 2020), the tuning parameters are set to (r²)); = Infp]/ / with p the total number of network parameters. The performances are measured in terms of average classification accuracies (denoted those by AC) and level of sparsity among the hidden layers
3.2.2 RESULTS
The results summarized in Table 2. Similarly as before, find that layer-sparse regularization can reduce the number of layers while retaining the classification accuracy or even improving it.
To highlight the features of layer sparsity more, also look at the training losses and testing accuracies over the course of the optimization; Figure 2 contains these data averaged over 20 draws of the MN data. We find that both the initial estimator well the refitted version eventually
8

---

Under review as a conference paper at ICLR 2021
MN1ST
F'shiioMNis7
KMNIST
Method
AAC
8
AC
8
AC
8
CE FCE
0.92
10
0.84
10
0.76
10
0.92
0.84
0.78
Table 2: layer sparsity can reduce model complexity without sacrificing classification accuracy
achieve the same accuracies, but the refitted version can be trained much faster. These observations are commensurate with the observations in (Frankle & Carbin, 2019), who study refitting of connection- sparse networks.
In contrast to (Frankle Carbin, 2019), however, we could not find any benefits of keeping the initializations of the original network: both the training and the accuracy curves remain virtually the same. This observation might indicate that layer sparsity might be particularly robust under refitting.
reltied
0.20
6.55
6.55
inreguat arguu
reftted
Figure 2: compressed layer-sparse networks can be trained faster than the original networks
4 DISCUSSION
We have shown that layer sparsity can compress layered networks effectively both in theory (Section 2) and practice (Section 3).
We have also shown that refitting with respect to layer sparsity can facilitate the optimization of the network. Refitting has mingsstandig tradition in high-dimension statistics see Lederer (2013), for example- but has been applied in deep learning only recently (Frankle & Carbin, 2019). Our research supports the usefulness of refitting in general, and it demonstrates that layer sparsity is particularly suited for refitting.
Related concepts such as ResNets add complexity to the network descriptions. This makes these networks not only unfit for refitting but also very hard to analyze statistically. Layer sparsity, in contrast, simplify network architectures and seem amenable to statistical analyses via recent techniques for regularized deep learning (Taheri et al., 2020). Statistical theory for layer sparsity, therefore, seems a feasible goal for further research.
In summary, layer sparsity complements other notions of sparsity that concern individual connections or nodes. All of these concepts can help to fit networks that are efficient in terms of memory and computations and easy to interpret.
REFERENCES
J. Alvarez and M. Salzmann. Learning the number of neurons in deep networks. In Adv. Neural Inf. Process Syst., pp. 2270-2278, 2016.
T. Ash. Dynamic node creation in backpropagation networks. Connect. Sci., 1(4):365-375 1989.
Bakin. Adaptive regression and model selection in data mining problems. PhD thesis, The Australian National University, 1999.

---

Under review as a conference paper at ICLR 2021
A. Barron and J. Klusowski. Approximation and estimation for high-dimensional deep learning networks. arXiv:1809.030900 2018.
A. Barron and Klusowski. Complexity, statistical risk, and metric entropy of deep nets using total path variation. arXiv: 2019.
M. Bello. Enhanced training algorithms, and integrated training/architecturee selection for multilayer perceptron networks. IEEE Trans. Neural Netw., 3(6):864-875, 1992.
J. Bien, I. Gaynanova, J. Lederer, and C. Müller. Prediction error bounds for linear regression with the TREX. Test, 28(2):451-4)) 4744, 2019.
Changpinyo, M. Sandler, and A. Zhmoginov. The power of sparsity in convolutional neural networks. arXiv: http:::72..25555 2017.
M. Chichignoud, J. Lederer, and M. Wainwright. A practical scheme and fast algorithm to tune the lasso with optimality guarantees. J. Mach. Learn. Res., 17(1):1-20, 2016.
E. Chzhen, M. Hebiri, and Salmon. On Lasso refitting strategies. Bernoulli, 25(4A):3155-3200, 2019.
T. Clanuwat, M. Bober-Irizar, A. Kitamoto, A. Lamb, K. Yamamoto, and D. Ha. Deep learning for classical Japanese literature. arXiv:1812.017718 2018.
J. Feng and N. Simon. Sparse-input neural networks for high -igh-dimensional nonparametric regression and classification. arXiv:1711.07592 2017.
Frankle and M. Carbin. The lottery ticket hypothesis: finding sparse, trainable neural networks. In International Conference Learning Representations, 2019.
X. Glorot, A. Bordes, and Y. Bengio. Deep sparse rectifier neural networks. In International Conference on Artificial Intelligence and Statistics, volume Proc. Mach. Learn. Res., pp. 315-323, 2011.
N. Golowich, A. Rakhlin, and O. Shamir. Size-independent sample complexity of neural networks. 172:0: 2017.
R. Hahnloser. On the piecewise analysis of networks of linear threshold neurons. Neural Networks, 11(14):691-697, 1998.
R. Hahnloser, R. Sarpeshkar, M. Mahowald, R. Douglas, and Seung. Digital selection and analogue amplification coexist in cortex-inspired silicon circuit. Nature, 405:947-951 2000. 2000.
S. Han, H. Mao, and W. Dally. Deep compression: Compressing deep neural networks with pruning. trained quantization and huffman coding. In International Conference on Learning Representations, 2016.
T. Hastie, R. Tibshirani, and M. Wainwright. Statistical learning with sparsity: The lasso and generalizations. CRC press, 2015.
K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In IEEE Int. Conf. Comput. Vis. Pallern Recognit., pp. 770-778, 2016.
Kim, V. Calhoun, E. Shim, and J.-H. Lee. Deep neural network with weight sparsity control and tre-training extracts hierarchical features and enhances classification performance; Evidence from whole-brain resting- state
state functional connectivity patterns of schizophrenia. Neuroimage, 124: 127-146, 2016
M. Kohler and S. Langer. On the rate of convergence of fully connected very deep neural network regression estimates. arXiv:1908. 133, 2019.
Y. LeCun, L. Bottou, Y. Bengio, and P. Haffine. Gradient-based learning applied to document recognition. Proc. IEEE, 86(11):227--3222 1998.
Y. LeCun, Y. Bengio, and G. Hinton. Deep learning. Nature, 521:436 444, 2015.
10

---

Under review as a conference paper at ICLR 2021
J. Lederer. Trust, but verify: benefits and pitfalls of least-squares refitting in high dimensions. arXiv:1306.0113, 2013.
J. Lederer and M. Vogt. Estimating the lasso's effective noise. arXiv:2004.11554, 2020.
S. Liang and R. Srikant. Why deep neural networks for function approximation? arXiv:1610004111, 2016.
B. Liu, M. Wang, H. Foroosh, Tappen, and M. Pensky. Sparse convolutional neural networks. In IEEE Int. Conf. Comput. Vis. Pattern Recognit. pp. 806-814, 2015.
E. Salinas and L. Abbott. A model of multiplicative neural responses in parietal cortex. Proc. Natl. Acad. Sci. USA, 93(21))11956-11999 1961, 1996.
S. Scardapane, D. Comminiello, A. Hussain, and A. Uncini. Group sparse regularization for deep neural networks. Neurocomputing, 241:81-89, 2017.
Schmidhuber. Deep learning neural networks: An overview. Neural Networks, 61:85-117, 2015.
K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations, 2015.
M. Taheri, N. Lim, and Lederer. Balancing statistical and computational precision and applications to penalized linear regression with group spansity. arXiv:1609.071955, 2016.
M. Taheri, F. Xie, and J. Lederer. Statistical guarantees for regularized networks. arXiv:2006.00294, 2020.
M. Telgarsky. Benefits of depth in neural networks. Annual Conference on Learning Theory, volume 49 of Proc. Mach. Learn. Res., pp. 1517-1539, 2016.
R. Tibshirani. Regression shrinkage and selection via the lasso. J. R. Stat. Soc. Ser. Stat. Methodol., 58(1):267-288, 1996.
W. Wen, C. Wu, Y. Wang, Y. Chen, and H. Li. Learning structured sparsity in deep neural networks. In Adv. Neural Inf. Process Syst., pp. 2082-2090, 2016.
H. Xiao, K. Rasul, and R. Vollgraf. Fashion-MNIST: a novel image dataset for benchmarking machine learning algorithms. arXiv:170..07747, 2017
D. Yarotsky. Error bounds for approximations with deep ReLU networks. Neural Networks, 94: 103-114, 2017.

---

Under review as a conference paper at ICLR 2021
A TUNING-PARAMETER CALIBRATION
Regularizing layer sparsity involves the tuning parameters (r²)), Such tuning parameters are integral regularization in deep learning and in machine learning more generally. In sparse linear regression, there has been some progress for developing theories for calibrating these parameters (Bien et al., 2019; Chichignoud al., 2016; Taheri et al., 2016). sparse deep learning, however, theories for tuning-parament parameter calibration are missing completely.
In the real-data analysis of Section 3.2, we have used a theory- inssiied tuning parameter. The goal of this section here is to give more insights into the calibration of the tuning parameter. Figure 3 shows the accuracies of refitting with different number of hidden layers and locates the tuning parameters selected by approach and by cross-validation, that is, training/validation based on 10 10000 training and validation samples. The results are averaged over 20 draws of MNIST data as described earlier but only over epochs each for illustration. The plot shows the expected upside-down- Saeenn---hhppee of the accuracies, which reflects the trade-off between variance (many layers/small tuning parameters) and bias (few layers/large tuning parameters). The plot shows that cross-validation can even improve the impact of layer sparsity further for a small number of epochs. (As illustrated in the right panel of Figure 2, tuning parameters become as far as accuracy concerned less important for large number of epochs.)
Tuning
funing-parameter parameter calibration remains a challenge not only here but in deep learning much more generally. But our observations in this section and the main body of the paper demonstrate that layer-sparse regularization can improve deep-learning pipelines substantially with data-adaptive schemes such as cross-validation as well with simple, parameter.
choice of the tuning
0040 Syynr
Figure 3: the accuracy function of the number of hidden layers has typical upside-down- U- shape: the cross-validated tuning parameters (bluc) improve the theory-inspired tuning parameters (red) further
12


Reviews:
[{"review_id": "zsaL5iKfYdN", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##########################################################################\n\nSummary:\n \nThe paper proposes a regularizer enforcing a novel form of sparsity that authors call \"layer sparsity\". Under certain conditions on layer weights, two consecutive layers in a deep neural network (with certain nonlinear activation functions) can be represented exactly as a single layer. The authors proposed a regularizer that can lead to such layer collapse thus resulting in shallower and more compact models.\n\n\n##########################################################################\n\nReasons for score: \n\nOverall, I vote for rejecting this paper. In my opinion, the proposed regularizer is quite simple and intuitive, but is at the same time novel and could potentially be useful in practice. However, the proposed empirical studies shed very little light on how effective this method can be in practical and even remotely realistic circumstances. I am afraid that the current empirical evaluation is not sufficient.\n\n\n##########################################################################\n\nPros: \n\n1. The paper is clearly written. The core idea of the layer sparsity regularizer is introduced and explained with a sufficient level of mathematical rigor.\n\n2. The proposed technique could become a useful addition to the deep learning practitioner's toolbox.\n\n \n##########################################################################\n\nCons: \n\n1. While the experiments described in the paper are a reasonable first step in exploring the effect of adding the proposed regularizer, more empirical exploration (particularly with realistic and practical models) might be necessary for a sufficiently thorough analysis. Currently there is just not enough information to judge the effectiveness of this method in any practical circumstances.\n\n##########################################################################\n\nQuestions during rebuttal period: \n\nPlease address and clarify the cons above. (I will update the score depending on the authors reply.)\n\nI also have a question (and doubts) about Theorem 1 and its proof in Appendix A.\n\nTo illustrate my confusion consider a simple model with ReLU activations $f(x)$.\nThen $o_i = f\\left(\\sum_j w_{ij} f\\left(\\sum_k w_{jk} x_k\\right)\\right)$ can be rewritten as $o_i = \\sum_j f\\left(\\sum_k w_{ij} w_{jk} x_k\\right)$ assuming that all $w_{ij}\\ge 0$.\nEven after reading Appendix A, I do not understand how dependence on index $j$ inside $f(\\cdot)$ can be factored out.\nIn other words, I do not understand how this expression can be written as $f^{j,j+1}(V^{j,j+1}z)$ with $V^{j,j+1}\\in \\mathbb{R}^{p_j \\times p_{j+2}}$ and $f^{j,j+1}:\\mathbb{R}^{p_j}\\to \\mathbb{R}^{p_j}$.\nI think I can even come up with simple numerical examples illustrating this point (all we need to do is make sure that the argument of $f$ is negative for one particular $j$ thus making this term vanish completely in the original expression, but still be present in the result of Theorem 1).\n\nIn a long derivation in Appendix A, in a transition from line 2 to line 3, the expression with an index $m$ fixed externally is seemingly replaced with a scalar product involving a summation over this index (at which point it disappears altogether from inside the activation function). It is entirely possible that I do not understand something trivial, but I would ask the authors to explain this transition.\n\n#########################################################################\n\nOther minor typos:\n\n1. A. Barron and J. Klusowski 2018 reference is repeated twice (there must be two records for it in the bibliography file).\n\n#########################################################################\n\nPost-rebuttal.\n\nI would like to thank the authors for their reply, which addressed some of my questions. While I now agree with the main theoretical result when applied to a ReLU nonlinearity, this of course also reduces the area of applicability of the proposed technique. I am also happy to see additional empirical results, which I believe will be the key to making this paper much stronger (since the current theoretical result is actually quite straightforward when applied to the ReLU nonlinearity). But I think the results are still a bit insufficient to make this submission sufficiently strong. One of my concerns is the final accuracy in Figure 2. We can see that the unregularized model surpasses the accuracy of the refitted model and could potentially get higher (or even much higher) should it not have been cut at <100 epochs. I will be excited to see an updated and improved version of this paper in the future, but in my opinion, the current version still needs a bit of work and is not entirely convincing.", "extra_scores": {}, "extra_text": {}}, {"review_id": "ZTMPByIu3Gp", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper introduces a new, interesting definition of \"sparsity\" in a deep network. It penalizes as a single group the positive values in a layer. When combined with an activation function such as ReLU, they show that this allows to remove entire layers by merging two adjacent weight matrices.\n\nThe paper is well written and relatively easy to follow. The mathematical notation could be simplified (especially the difference between W and V).\n\nHowever, the experimental results are definitely below what one expects in a deep learning paper today. The paper only considers a very simple, artificial setup with \"Datasets of 150 samples\" and tuning parameters \"calibrated very roughly by hand\". In my opinion this is not sufficient to show that a method is useful in a realistic context.\n\nThe paper should provide some discussion of the relation with recent literature on the lottery ticket hypothesis, especially when introducing its retraining procedure.", "extra_scores": {}, "extra_text": {}}, {"review_id": "Ee3N8a-A63K", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new notion of layer sparsity for neural networks that aims at simplifying network architectures and reducing the number of parameters. A type of regularizers has been introduced to encourage this specific structure.\n\nOverall the paper is well written and the idea is interesting. It pushes the network towards less nonlinear layers if not needed. Does the method apply to scenarios where there are bias terms between the layers?\n\nThe numerical results are not sound. Using a fixed number of epochs in each setting, we have no idea of the convergence status for different methods. The authors claim in the end of the results section that all methods can sometimes provide accurate prediction if the number of epochs is extremely large. Does this happen to the numerical examples presented here? Plotting some training curves might help readers understand more of the behavior.\n\nIt is fairly standard practice to tune the regularization parameters based on cross-validation or an independent validation set. The authors should include this for the completion of the manuscript, even if the numerical experiment might serve as a proof of concept.\n\nIt would be better to include comparison with other sparsity-inducing methods such as connection sparsity and node sparsity. Also, it would be interesting to see the performance in other settings than the most favorable one, for example on some real data sets.", "extra_scores": {}, "extra_text": {}}, {"review_id": "4FG9-eZxxkV", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "__how I would summarize the paper.__\nThe paper gives an interesting new paradigm of the neural network compression called the *layer sparsity*. The paper builds on the (somewhat underappreciated) observation of Barron&Klusowski that positive weight parameters of multiple layers can be aggregated and reparametrized through the positively homogeneous activation functions. Based on the observation, the paper designs a regularizer that enhances the possibility of such aggregation by regularizing the negative parts only. Whenever successfully regularized, the number of layers in the model can be reduced without sacrificing too much performance, as empirically verified; actually, using the regularizer per se helps reducing the test mean-squared error.\n\n__review tl;dr.__\nWhile the idea itself is quite interesting, I believe that there should be more practical evidence and algorithmic extension of the framework, for the idea to fully bloom.\n\n__what I like.__\nThe underlying idea itself is quite interesting (somewhat reminiscent of the [lookahead pruning](https://openreview.net/forum?id=ryl3ygHYDB)), and the design of the regularizer makes a perfect sense. Also, potential practical impact of the notion of layer sparsity seems to be big, as reducing the number of layers has a more straightforward practical benefit, in terms of reducing the inference flops/time.\n\n__what I think is missing.__\nModel compression became an important research area after deep, convolutional networks gained popularity. Perhaps this is why most of the works on network pruning focus on evaluating their algorithms on compressing deep convolutional networks trained on real (as opposed to synthetic) datasets. To fully persuade the benefit of the proposed algorithm, I think there should be (1) an extension of the algorithm to the aggregation of convolutional layers, which may lead to a larger kernel widths, which is okay, (2) an empirical validation on larger models trained on non-synthetic datasets.\n\nA minor concern is that the refitting step does not seem particularly novel, as the retraining step is already typical in model compression literature.\n\n__a question.__\nWould it be possible to compress the layer which does not perfectly satisfy the criterion (with a slight abuse of terminology) at \"page 6, penultimate line,\" but only approximately 0?", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewers agree that the idea of layer wise regularization is interesting and is in line with many efforts in the optimization realm to specialize in the training procedure and the learning rate to each layer.  Given the depth of some state of the art neural networks, efficiency is at stake and the idea brought up in this paper naturally falls into that.  While the theoretical result in Theorem1 is sound and clear, an extended result on the impact of such « merge » and « layer skipping » on the overall predictions of the algorithm can be well appreciated. The overall goal of network compression should remain to reduce drastically the network size, and thus the training time (energy consumption etc...), while keeping a relatively good prediction accuracy (at least of the same order). Being able to back this with theory (and of course experiments) is crucial.   Reviewers also pointed out that the empirical evaluations were not sufficient for ICLR. For example, there are no enough comparisons with existing algorithms and there should be more experimental results based on real datasets. Although the rebuttals did help clarify some of the issues raised by the reviewers, overall this paper does not seem to meet the bar to be accepted. 


Decision: Reject