Learning $k$-Level Structured Sparse Neural Networks Using Group Envelope Regularization
Authors: Yehonathan Refael
Iftach Arbel
Wasim Huleihel
Venue: Accepted by TMLR ()

Abstract: The extensive need for computational resources poses a significant obstacle to deploying large-scale Deep Neural Networks (DNN) on devices with constrained resources. At the same time, studies have demonstrated that a significant number of these DNN parameters are redundant and extraneous. In this paper, we introduce a novel approach for learning structured sparse neural networks, aimed at bridging the DNN hardware deployment challenges. We develop a novel regularization technique, termed Weighted Group Sparse Envelope Function (WGSEF), generalizing the Sparse Envelop Function (SEF), to select (or nullify) neuron groups, thereby reducing redundancy and enhancing computational efficiency. The method speeds up inference time and aims to reduce memory demand and power consumption, thanks to its adaptability which lets any hardware specify group definitions, such as filters, channels, filter shapes, layer depths, a single parameter (unstructured), etc. The properties of the WGSEF enable the pre-definition of a desired sparsity level to be achieved at the training convergence. In the case of redundant parameters, this approach maintains negligible network accuracy degradation or can even lead to improvements in accuracy. Our method efficiently computes the WGSEF regularizer and its proximal operator, in a worst-case linear complexity relative to the number of group variables. Employing a proximal-gradient-based optimization technique, to train the model, it tackles the non-convex minimization problem incorporating the neural network loss and the WGSEF.  Finally, we experiment and illustrate the efficiency of our proposed method in terms of the compression ratio, accuracy, and inference latency.

Published in Transactions on Machine Learning Research (08/2024)
Learning Le-el Structured Sparse Neural Networks Using Group Envelope Regularization
Yehonathan Refael
@efoelkollim@mail.tau.....
Deportment of Electrical BeniineeringSysteewe
Tel Aviv Universily
Iftach Arbel
L.arbel84 @gmaii com
Independent Reseorcher
Wasim Huleihel
wasimb Stauex fau. ac...ucc.......
Deportment of Electrical Engineering--ysteems
Tel Aviv University
Reviewed on OpenReview: https: // openreview. net/ forum? id= XPLXY>711
Abstract
The extensive need for computational resources poses significant obstacle to deploying large-scale Deep Neural Networks (DNN) on devices with constrained resources. At the same time, studies have demonstrated that significant number of these DNN parameters are redundant and extraneous. In this paper, we introduce a novel approach for learn- ing structured sparse neural networks, aimed at bridging the DNN hardware deployment challenges. We develop a novel regularization technique, termed Weighted Group Sparse Envelope Function (WGSEF), generalizing the Sparse Envelop Function (SEF), to select (or mullify) neuron groups, thereby reducing redundancy and enhancing computational ef- ficiency. The method speeds up inference time and aims to reduce memory demand and power consumption, thanks to its adaptability which lets any hardware specify group defini- lions, such as filters, channels, filter shapes, layer depths, a single parameter (unstructured), etc. The properties of the WGSEF enable the pre-definition a desired sparsity level to be achieved at the training convergence. the case redundant parameters, this approach maintains negligible network accuracy degradation or can even lead to improvements in accuracy. Our method efficiently computes the WGSEF regularizer and its proximal op- erator, in a worst-case linear complexity relative to the number of group variables. Em- ploying groxinal-graddiet--eseed oplimization technique, to train the model, it tackles the non-convex minimization problem incorporating the neural network loss and the WGSEF. Finally, we experiment and illustrate the efficiency of our proposed method in terms of the compression ratio, accuracy, and inference latency.
1 Introduction
In the past decade, significant progress has characterized the study of Deep Neural Networks (DNNs), which consistently demonstrate superior performance across the entire spectrum of machine learning tasks. As modern neural networks increase in size and complexity, with parameters often surpassing the number of available training samples, their deployment on resource-limited edge devices becomes incrcasingly challeng- ing. This difficulty stems from the higher computational demands that lead to greater power consumption, longer inference times, and the need for substantial memory space for storage, which edge devices typically lack Deng al. (2020); Cheng et al. (2017). Notwithstanding, many studies have revcaled that modern neural networks tend to be excessively over-parametrize Han et al. (2015b); Ullrich et al. (2017). This over-parametriiation implies the existence of redundant. parameters that could be pruned (or, nullified)

---

Published in Transactions on Machine Learning Research (08/2024)
without compromising network accuracy Molchanov et al. (2017) which are also responsible for issues such as overfitting Allen-Zhu et al. (2019), memorization of random patterns in the data Zhang et al. (2021) and a potential degradation in generalization. The realization that numerous redundant parameters exist has prompted a quest for neural network architectures that are both sparse and efficient, which emorges as a
prominent challenge in the field.
To mitigate the challenges associated with the deployment of modern large DNNs, numerous studies have suggested compressing their scale. Various approaches have been explored, including (unstructured) sparsity including regularization Liu et al. 2015),, pruming Han et al. (2015), low-rank approximation [Han et al.
: Wu et al. (2021), and even sparse neural architecture search (NAS) Yang et al. (020bb:; Wu et al. (2021). In the case of the unstructured the most natural regularizer would be the so-called to fo-pseudo-non norm function that counts the number of nonzero elements the input vector, i.c., ||2|lo _ [{:: # |. These sparse regularized siiimization//tranniii problems are of the form mingeR" {f(z) + Al|z[lo), or, alternatively, one can explicitly constrain the number of parameters used for regression and solve minzeR= (f(z) ||2 S k). Unfortunately, the norm is difficult function to handle being non-convex and even non-continuous. Indeed, these types of regression models are known to be NP-hard problems, in general, Natarajan 1995 (global optimal solution can not be computed in a reasonable time, even for a very small number of parameters). As a remedy for the inherent. problem above, Beck & Refael (202 proposed a highly efficient tractable convex relaxation technique, termed sparse envelope function (SEF), for the sum of both &o and & norms. Specifically, Beck & Refael (202) suggested using this relaxation as a regularizer term for a convex loss objective, particularly for linear regression model, to achieve feature selection while explicitly limiting the number of features to be fixed parameter k. It was shown that the performance of this sparse inducing regularization method in both reconstruction of a sparse noisy signal and recovering its support, surpass the performance of stale-of-the-art techniques, such as, the Elastic-net Zou & Hastie (2005), k-support norm Argyriou et al. (2012), etc. Also, it was shown that the computational complexity of the SEF approach is linear in the number of paramoters, while all others require at least quadratic in the number of features, thus SEF was found very attractive.
(2015b); Denton et al. (2014a). quantization Gholami ct al. (2022);
Not long ago, the idea of structured sparsification was used in Wen et al. (2016); Bui et al. (2021) to learn sparse neural networks that leverage tensor arithmetic in dedicated neural processing units (NPUs). In nutshell, structured sparsity learning amounts to inducing sparsity onto structured components (e.g.. channels, filters, or layers) in the neural network during the optimization procedure. This leads, in practice, to both low latency and lower power consumption, which can not be obtained by deploying unstructured sparse models on such modern hardware.
With the goal of enabling structured sparsification learning that can be customized for different NPU devices, in this paper we propose a novel generalized notion of the SEF regularizer to handle group structured sparsification in neural network training. Our new generalized regularization term selects the most essential k S m predefined groups of neurons (which could be convolutional filters, channels, individual neurons, any other user-defined/NPP definition, where m is the total number of groups) and prunes all others, while maintaining minimal network accuracy degradation. We define the new regularization term mathematically, propose an efficient method to calculate its value and proximal operator, and suggest a new algorithm to solve the complete optimization problem involving the non-convex term, which is the composition of the loss function and the neural network output.
Related work. The topic of regularization-bass/ pruning received a lot of attention in recent years. Generally speaking, these studies can be divided into unstructured and structured pruning. Most prominent regularizers are the convex & and &a norms Liu et al. (2017): Ye et al. (2018): Han et al. (2015b), as well as the non-convex to "norm" Louizos et al. (2017b); Han et al. 2015); Louizos et al. (2017a), where Bayesian methods and additional regularization terms for practicality, were used to deal with the non-convexity of the tn norm. Additional works of Donoho & Elad (2003); Delmer et al. (2021) suggest methods for norm to relaxation by employing 4 minimization in general (nonorthogonal) dictionaries and leading to an error surface with fewer local minima than the lo norm. The motivation for these regularizers their "sparsity- inducing" property which can be harnessod to learn sparse neural networks. While these fundamental papers significantly reduce the storage needed to store the networks on hardware, there were no benefits in reducing
2

---

Published in Transactions on Machine Learning Research (08/2024)
the inference latency time or either in cutting down power consumption. That is, the sparse neural networks, learned by the aforementioned methods, were not adapted to the tensor arithmetic of the hardware they aimed to run on.
The practical inefficiency of unstructured sparsity-inducing methods has led researchers to propose regularization--usse structured pruning in favor of accelerating the running time. For example, Lebedev & Lempitsky (2015); Wen al. (2016); Yuan & Lin (2006) proposed the use of the Group Lasso regulariza- tion technique to learn sparse structures, and [Scaddapane et al. (2017) uses Sparse Group Lasso, summing Group Lasso with the standard Lasso penalty. Other convex regularizers include the Combined Group and Exclusive Sparsity (CGES) Yoon & Hwang (2017), which extends Exclusive Lasso (in essence, squared & over groups) Zhou et al. (2010) using Group Lasso. Recently, Bui et al. (2021) suggested a family of non- convex regularizers that blend Group Lasso with nonconvex terms (lo, l Eg Lou et al. (2015), and SCAD Fan & Li (200T)) Since Bui et al. (202T) introduces non-convexity term into the penalty, it also requires an appropriate optimization scheme, for which the authors propose an Augmented Lagrangian type method. However, this optimization algorithm has an inner optimization loop with a high computational cost. More- over, their extensive experiments do not show accuracy sparsity advantages over convex penalties, suggesting that it might be still desirable to use a convex regularizer. Other methods, such as Chen et al.) (2 is Li et al. (2019), focus on a group structure that captures the relations between parameters, neurons, and layers, in order to construct groups that can maximize network compression while minimizing accuracy loss. However, these methods still apply Group Lasso regularization. Specifically in Chen et al. (2021)
the authors introduce the concept of Zero- Gero-Invariant Groups (ZIGs), which includes all input and output connections between layers. In the context of CNNs, it extends the channel-wise grouping Wen et al. (2016) to include corrosponding batch normalization parameters. By using this group structure, entire blocks of pa- rameters can be removed while keeping dimensions aligned between layers, and ultimately allowing network compression. Moreover, their optimization scheme utilizes a two-phase algorithm to include a half-space projection step, which they name HSPG. Lately, a novel methodology that applics adaptive optimization via weighted proximal operators to induce structured sparsity was suggested in Deleu Bengio (2023) in seamn lessly integrating numerical solvers to preserve convergence guarantees, albeit with computational efficiency concerns due approximation requirements.
Finally, we mention that there exist other techniques for neural net compression, such as, quantization, low-rank decomposition, to name few. In quantization, Courbariaux et al. (2016) Rastegari et al. (2016):
Gong et al. (2014), ; the precision of the weights is reduced, by representing weights using a low number of bits (i.c.. 8-bit) instead of higher one (i.c., 32-bit floating point values). The low-rank decomposition approach Denton el al. 22144b); Jaderberg et al.] (2014); Lebedev et al. (2014); Li el al. (2020b) is based on the observation that many weight matrices neural networks are highly correlated and can be well approximated by matrices with a lower rank. By decomposing weight matrix into lower-rank matrices, one can reduce the total number of parameters in the network.
Notation. We denote e for the vector of all ones. For positive integer m, we denote [m] = 2,, 2,., m... We denote by zQy the component of x with the the largest absolute value, meaning in particular that |aas 2 | 2 ... 2 |tn) |- [S] refers to the number of elements in the set S. Let f 18" + R. be extended real-valued function, then, the conjugate function of S, denoted by ' R R is defined as (z) maXxeR" {(x, y) - f()),, , for any y E R". The bi-conjugate function is defined as the conjugate of the conjugate function, i.c., f*(xx = max ((x. - f")),, for any x E R". Fi- nally, the proximal operator of a proper, lower semiicontinuous convex function f R" + R defined as prox (v) = arg minzer {f(xx) + 3 | - -m]l2), for any E RR... The sets B and R++ denote all non-negative and positive real numbers, respectively.
2 Structured sparsity via WGSEF
2.1 Problem formulation
In this subsection, we formulate the problem and introduce the weighted group sparse envelop function (WGSEF). Without. loss of generality, our method is formulated weights sparsity, but it can be directly
3

---

Published in Transactions on Machine Learning Research (08/2024)
extended to neuron sparsity (i.e., both weights and bias). Let D be dataset consisting of N iidd. imput output pairs VI). (.N,.,,N,,,,,,,,,, (eN,UN)]. The general neural network training problem is formalized as the following regularized empirical risk minimization procedure on the parameters 0 E θ of a given neural network architecture f(::6),
1
argmin
Σ +x- BEe N
+
(1)
where, f(;0) is the hypothesis, that is, given neural network architecture, L(-) 2 0 corresponds to a loss
function, c.g., cross-entropy loss for classification, mean-squared error for regression, etc, (2(-) θ + R4 is
the parameters regularization term, À E R4, is the regularization maguitude. Below, n |+] denotes the number of parameters.
The most prodominant regularizer used for DNNs is the weight decay, also known the & norm regular- ization. It is known to prevent overfitting and to improve generalization since it enforces the weights to decrease proportionally to their magnitudes. The most natural way to force predefined k-level sparsity would be to constrain the number of non-zeros parameters (e.g., the model weights), which can be done by adding the constraint that ||0]|o < k, where S n is the required predefined level of sparsity. In this case, the training problem is formalized as follows,
N Σ
1
2 2 1018
argmin 6Ee
C( (a., .))
v.)
N
(2)
2=1
s.t.
1101|o < k.
We refer to the above training problem as unstructured sparsification. In the case of structured spersity, the parameters 0 are divided into predefined disjoint sub-groups. These subgroups could define the building blocks architecture of DNNs, i.e., filters, channels, filter shapes, and layer depth. Consider the following definition.
Definition 1 (Group projection). Let be subset of inderes S (1,2, n of size |s| S n. Then, given some vector E IR", the projection M, : IR" + R² preserves only the entries of 0 that belong to the set 8. Furthermore, let A, be an n x n diagonal matriz, where [Ar]i = if E s, and sero, otherwise. Note that M,( AgO.
Example 1. Let n 3,0 (3,6,9) [1,3) S [n]. and accordingly |s| then M, M, (3,6,9) (3,0,9) with [A.]u [A.]33 = 1, and zero otherwise.
Following the above definition, let [n], namely, n 8, = 0, for all + J. and U 8; [n]. Without loss of generality, assune that n mod m = 0; otherwise, the groups would have different coordinates. Every group is associated with some weight d2 E R++, whore j E [m]) c.g., dj = [sj] namely, we normalize by the group size. For simplicity of notation, let 0s, Ms, (0), for i 1,2,... m. Then, our structured training problem is,
subsets 51, 82 , 8m be given (non-overlapping) partition of
1
N Σ
A
argmin d
(z (z,, ().0).
d 10s
(3)
N
2
1=1
1=1
s.t.
|||0011
2 | ll,...., 110mm
Ello s k.
To wit, we constrain the number of groups which has at least one non-zero coordinate, to be at most k. Let C₁ denote the set of all l sparse groups, i.e.,
C e (0: || 11 ||0 12 ... 1s 11 < kk]. and define 8c, as the following extended real-valued function,
0, || ||0s2|12 ||0sm111|| < k,
gc, (8)
00, else.
4

---

Published in Transactions on Machine Learning Research (08/2024)
Then, the optimization problem in equation can be reformulated as,
N
argmin BEE
1
Σ 1=1
(c),0))) + ggx((0)
(4)
N
where (0) A
2 Σ d,110=ı11 dc, (0). Equivalently, (0) can be rewritten as goa(0) A
E di
03 + dc, (0), where d, = dj for every i 8j.
The fu-norm that appears in 4 is difficult function handle being nonconvex and non-continuous, making the problem an untraceable combinatorial NP hard problem Natarajan (1995) Following the work in Book & Refacl 2022). one approach to deal with this inherent difficulty is to consider the best convex underestimator of (-)(-). The later is its bi-conjugate function, namely, GS (0) = gst (0), which we refer to as the weighted group sparse envelope function (WGSEF).
Remark (Generalization of SEF). Consider the case m n, namely, every subset 8,, i E [m]] is singleton and vj E [m], dj 1. Here, g8x(f) sy sx) where su (0) ; 1101 if [|0]|o S k and sk (6) = oo, otherwise. Accordingly, in this case, (0) = Sx (0) = st" (0), and st (0) is exactly the classical SEF, namely, S.(-)) Therefore, GS is indeed a new generalization of SEF to handle group sparsity.
Thus, the path laken in this paper is to consider the following relaxed learning problem (training),
N Σ ((f(),)) () + (O) N
1
argmin O
(5)
In the following subsoction, we develop an efficient algorithm to calculate the value and the prox-operator Beck 2017) of WGSEF; these will play an essential ingredient when solving 5
2.2
Convex relaxation
Let us start by introducing some notation. For any RR" and m subgroups of indexes 81, 82, ......, Sm C [n]. we denote by (0) the corresponding subgroup of coordinates in 0 with the ith largest 4--nomm, i.e.
Vi * [m].
( (0)l 2 [0)1)))00112= ... |
We next show that the conjugate of the group sparse envelopes is the & weighted group hard thresholding function. In the sequel, we let D be the diagonal positive-defnite weights matrix, such that D,, = Vd,, Vi E Sj: and j E [m].
Lemma 1 (The k weighted group sparse envelop conjugate). Let subsets $1, $2 ....... Sm be set of m S n disjoint indexes that partition [n]. Then, for any E TR"
4 Σ ==1
got(0)
(001)
(6)
2
Next, we obtain the bi-conjugate function of the & weighted group sparse envelope. To express the following results explicitly, will deliberately utilize the group projection definition exprressed M.(0) A,O, for some set of indices
Lemma 2 (The variational bi-conjugate weighted group sparse envelop). Let 81, S2...., Sm a set of m S n disjoint subsets that partition [n]. Then, for any 0 RR,, the bi-conjugate of the k group sparse envelop is given by
Σ oeه (A,, 0, 0,,))
(S) = min
(7)
2 ne
5

---

Published in Transactions on Machine Learning Research (08/2024)
where,
0 0
Ug > 0,
"
(A,,0,ug) &
0,
U 0 As, 8 0,
(8)
else.
2 The following is a Mtraightforwan corollary of Lemma Corollary 2.1. The following holds:
GSx(0)=
S((vdilAA,)1o v 0l.2.... Ao 0)
where S(0) (0) is the slandard SEF.
The above corollary implies that in order to calculate GS, (0) we only need to apply an algorithm that calculates the SEF at (ydillA,, 0ll2, Al Vdmilll..... 012)
Remark 2. Noting l.dd,A., 0112 = Eie,, dj0e we observe that since 8j is given, the number of operations required to colculate / A,, | is linear w.r.l. 18yl- Thus, the computational complexity of calculating the vector A,, f112, /2 Al, fll...... dd As 0112) is linear in n.
2.3 Proximal mapping of the WGSEF
In this subsection, we will show how efficiently compute the proximal operator of positive scalar multiples of GSt. The ability to perform such operation implies that it is possible to employ fast proximal gradient methods to solve equation 5 We begin with the following lemma that shows that the proximal operator can
be determined in terms the optimal solution of a convex problem that resembles the optimization problem defined in Lemma equation 7 for computing GS
Lemma 3. Let > 0, t R", and $1. $2,..., be a set of m S n disjoint subsets that partition [n]. Then, proxAgs, (f) given by
ut t Au₂V = Ad2 + Utj
j E [m],
where (u1, u2...., un)) is the minimizer of
m
min Σ Φ (Vd,Ao,t, + +w) D j=1
(9)
Next, we show that the proximal operator of GS₃ roduces to an efficient one-dimensional search.
Corollary 3.1 (The proximal operator of gSx). The solution =u((²)) of equation with uz(.) defined a
|b1
1,
V 1b,1
<
astl'
u(p*)
V
1b11
aj
a;+l
p
(10)
111
for b, = Al and a, >dj(> 0), and ij
1
is root of the function
(e(n) = Σ u9(1)) k
(11)
ITF As, = then equation 10 implies that ( for all p

---

Published in Transactions on Machine Learning Research (08/2024)
which is nondecreasing and satisfies
X- minje/m) {dp}
9t
Vd || 21l2. vd2 | 2/22 ... vmm || t1|2
m
Σ 3 00
and,
> [11di,d2..... d2,.................. dm dmallo +1 1 1 11-k> k> 0.
M[Mem)) (Vdgi) 112 +1)
Σ
9t
In addition, 9t can be reformulated the sum of pairs of the functions
Us (n)) =17/b,| agl1,es(n) 11116611 (ag + 1)1. [m].
such that,
m
m
(e) = (n) +
Σ 2 j=1
wy wy(n)
The following important remarks are in order.
Remark (Root search application for function
Employing the randomized root search method in [Beck & Refael 2022 Algorilim 1) with the 2m one break point piece-wise lineur functions Vy Wjy as an inpul to the algorithm, the root of 9t can be found in (m) time.
Remark 4 (Computational complexity of prox xrox) The computation of prox boils down to root search problem (see, Remark EV. which requires O(m) operations. In addition, before employing the root search, the assembly of Wy,; requires the calculations of the m values of by defined in Covollary 77, Note that for any j [m] calculating by is equivalent to t A t.. Since sj 's are given, the computational complenity of calculating all m of by is linear in n, which is the dimension of Thus, the total computational operations of calculating prox summarizes to n with is the dimension of all groups porameters logether.
3 Optimization procedure
The general training problem we are solving is of the form
min
F(x) = f(x) + M(x),
(12)
xER"
where f= E function, but nonsmooth. We adopt the ProxGen Yun et al. (2021), which can accommodate momentum, and also has a proven convergence rate with a fixed and reasonable minibatch size order @(vN) (a comprehensive discussion on the selection of the optimization method provided in the appendix A.44. Next, we provide a convergence guarantee for Algorithm [ as given in Yang et al. (2020a). This result holds under several regularity assumptions which can be found in Appendix A.4.1
is contimuuusly differentiable, but possibly nonconvex, and h is convex
Corollary 3.2. Under Assumptions is guaranteed to yield E dist(0, OF()))
(C-1
Algorithm with constont minibatch size b, = O 0 (T 1) where OF is the Fréchet sub-differential function of F.
Forthwith, we propose Algorithm| 2 as an implementation of Algorithm 1 to solve equation 5 where f C and h & GS,. Calculating E.) commonly approached using a propagation algorithm, is least linear in the number of parameters and is obviously getting more complex as the number of layers increases. Therefore, the calculation of the prox of the WGSEF is not bottleneck of the update step complexity (since
7

---

Published in Transactions on Machine Learning Research (08/2024)
Algorithm 1: General Stochastic Proximal Gradient Method
Input: Stepsize a₁, {pt
1t=7 E [0, 1), regularization parameter X. Initialization: 0, E R2 and mg = 0 E R".
for iteration t = I:
Draw a minibatch sample & 8:
m, P + (1 pt)ge
0.+1 <- A (0, a.m))
return 0
Algorithm 2: Learning structured k-level sparse neural-network by Prox SGD with WGSEF regularization
Input:
Stepsize at {or E [0, 1), regularization parameters {Ar] E (0,
Initialization: Randomly initialize the weights θ== RR" my = 0 E R
for iteration t t= I:
Draw minibatch sample & B:
(,,;
m P=m;- + (1 -1-p8 pt)g:
01+1 6 AGS, (0; - a.))
Prune (Optional): all # (m k) smallest & group norm values of O.
return 0
it is linear in the number of parameters). Notice that in our setting, one option is that the regularization may be separable by layer, as indicated by the regularization function's definition h(x)= = E/= (x1). Therefore, in this case, prox is applied to each layer separately (Beck) 2017 Theorem 6.6) with different Ar, ki parameters per layer. Another option that the regularization is applied to all groups' layers collectively, namely, not in a per-layer fashion, to allow more flexibility in group selection. Formally, in this latter option, only a single pair of X, l values needs be selected, so VI [4] X = k. For example, regularization is applied simultancously levels in each layer, but overall adhering to the pre-defined filter sparsity level k. The only condition for both options is that groups are not overlapping. We note that since Assumptions (C-1) C-3 are met,
filters across all convolutional layers once, resulting in different filter sparsity
Algorithm 2
converges to an e-stationary point.
Another technique for solving equation 12 using the HSPG family of algorithms in Chen et al. 2020 Dai
et al. (2023). These algorithms utilize a two-step procedure which optimization carried by standard first-order methods (i.e., subgradient or proximal) to find an approximation that "sufficiently close" to a solution. This step is followed by a half-space step that freezes the sparse groups and applies tentative gradient step the dense groups. Over these dense groups, parameters are zeroed-out if a sulficient decrease condition is met, otherwise, a standard gradient step is executed. Notice that the half-space step, as variant of a gradient method, requires the regularizer term h to have Lipschitz continuous gradient, which is not satisfied in our setting. However, this property is required only for the dense groups as there is no use of the gradient groups that are already sparse. Since the continuity is violated only for sparse groups, the condition is satisfied in the required region. Finally, while the "sufficiently close" condition mentioned above cannot be verified in practice, simple heuristics to switch between steps still work well. We can either the first-order step for fixed number of iterations before switching to the half-space step, or, alternatively, run the first-order step until the sparsity level stabilizes, and then switch to the half-space step. The dense groups are defined as (x(x)=
== {> := I : {r > e G. |x*| * 0. The HSPG pseudo-code (proximal-gradian the appendix A.6 We mention the enhancod variant of the standard HSPG algorithm named AdaHSPG+ Dai et al. (2023) improves upon that implementing adaptive strategies that optimize performance, focusing better handling of complex dynamic problem scenarios where standard HSPG may be less efficient.
E G, ||*]|| = 0}, and sparse groups are defined as variant) is given in Algorithm 3 (in
8

---

Published in Transactions on Machine Learning Research (08/2024)
4
Experiments
In this section, present comprehensive benchmark of structured sparse-inducing regularization tech- niques. Our evaluation covers wide range of model architeclures, datasets, regularizers, oplimizers, and pruning techmiques. This extensive benchmarking demonstrates that WGSEF achieves state-of-the-art the-art per- formance across all these dimensions.
4.1 Evaluation of iparsity-inducing optimization methods
To demonstrate the performance of Algorithm 2 in terms of compression and accuracy, compared to tate-of-ttheart Frox-SGD-based optimization methods, architectures: VGG16 Simonyan & Zisserman (2014), ResNet18 He et al. (2015), and MobileNet Howard et al. (0177). These architectures were tested on the datasets CIFAR-10 Krizhevsky et al. (2009) and Fashion MNIST Xiao et al. (2017). While WGSEF is a regularizer rather than an optimization algorithm, we benchmark versus the Group Lasso regularizer using various optimization algorithms that are well- known for their superior performance with group sparsity inducing rogularization. All experiments were conducted over 300 epochs. For the first 150 epochs, employed Algorithm 2 and for the leftover epochs, regularizer (i.e., Algorithm
the following well-known DNNs benchmark
used the HSPG with the WGSEF acting using maini-batcl, size of = 128 on an A100 GPU. The cocfficient for the WGSE regularizer was set to == 10-2. In Table 1 we compare our results with those reported in Dai et al. of interest are the neural network group sparsity ratio, and the prediction accuracy (Top=1). Notably, the WGSE achieves a markedly higher group sparsity compared to all other methods except AdaHSPG++ for which we obtained slightly better results. It should be mentioned that all techniques achieved comparable generalization error rates the validation datasets.
3 Experiments were conducted
2023 The primary metrics
Table 1: Comparison of statc-of-tho-art techniques to our WGSEF regularization technique, in terms of group-sparsity--rtioo hatili/aaaaiiiiiivaaioooo---ooe validattion-aaccaaay (Top-1), both in percentage, for various models and datasets. Our method provides the highest sparsity level with comparable accuracy.
Model
Dataset
Prox-SG
Prox-SVRG
HSPG
AdaHSPG+
WGSEF
CIFAR-10 10
54.0 90.6
14.7 89.4
74.6/ 74.6 91.1
76.1/90 /91.0
76.8 / 91.5
VGG16
F-MNIST
19.1 93.00
0.5/ 0.5/ 92.7
39.7 / 93.0
51.2 61.2/92.9 92.9
51.9 / 92.8
CIFAR-10
26.5 / 94.1
2.8/0 94.2
41.6 94.4
42.1 94.5
42.6
94.5
R8sNet18
F-MNIST
0.0/94.8
0.0/94.6 94.6
10.4 94.9
43.9 / 94.99
44.2
94.9
CIFAR-10
58.1 /91.7
29.2 / 90.7
65.4 92.0
71.5 19.8/118 19.88
71.8 / 91.9
MobileNet
F-MNIST
62.6 / /91.2
42.0 94.2
74.3 94.5
78.9 94.6
79.1 / 94.5
Ratio Sparsed Fibers Rasret18-Cifar10
D4b0 Spared thtre
0.
Conaoaatinn
Figure 1: Ratio of the sparse filters in the convolutional layers, according to the order of the layers in the resnet18 18 model, obtained by Algorithm 2 and corresponds to the experiment in row 3 of table 4

---

Published in Transactions on Machine Learning Research (08/2024)
4.2 Evaluation of different sparsity-inducing regularizers
In this experiment, we Irained deep residual networks ResNet440 He el al. (0015) CIPAR-10, while applying Algorithm 2 with a predefined sparsity level of 55% for all filters in all convolutional layers of the network, over runs. Again, to have a fair comparison, the baseline model was trained using SGD, both with an initial learning rate ag = 0.01, regularization magnitude = 0.03, a batch size 128, and a cosine annealing learning rate scheduler. Our results Table 2 structured sparsity-inducing regularizers. These results are similar those in [ui al. (2021), and are obtained through a grid search process that varied the magnitude of regularization w.r.t. the sparsity level. Note that the model trained using our method achieved 2.14x in speedup and 47.3% reduction in FLOPs.
demonstrate our method's superiority over satle-of-the-art
Table 2: Comparison of state-of-the-arr structured-spars inducing regularization methods to WGSEF technique, in terms of filter sparsity compression and accuracy in percentage, for Resnet40 and CIFAR-10. The WGSEF regularization provides the highest sparsity level with even better accuracy.
Method
Error
Sparsed Filters
Baseline (SGD)
6.854%
0%
SGL1 Scardapane et al. (0177)
7. 7.760%
50.7%
SGL.g Bui et al. 2021
8.146%
53.4%
SGSCAD Lv & Fan 2009
8.026%
52.2%
SGTL, Bui el al. 20021
8.096%
53.7%
SGTL Tran Webster 2019
7.968%
53.7%
WGSEF
7.264%
54.3%
4.3 Evaluation of different state-of-the-art pruning techniques on ImageNet
In this subsection, we compare our method to the tf pruning techniques, which are often used as an alternative for model compression during (or, post.) training. We train Resent50 with ImageNet dataset, using À 0.05, with an initial learning rate of ag = 0.01, sparsity level k = 0.34, and use cosine anncaling learning rate scheduler. In table 3 we compare our results to those obtained by the methods in Chen et al.) (2021). We emphasize that, mentioned later, some of the methods require several stages of training, fino-tuning, etc. Our method trains the model from scratch once, well as OTO, and thus, this is the most fair comparison. Additionally, it should be noted that all techniques achieved comparable generalization error on the validation datasets, while our method achieved better compression performance as compared to the other techniques.
Table 3: Comparison different pruning methods with ResNet50 for ImageNet. Our method provides the highest sparsity level, lowest total number of parameters, with comparable accuracy in both Top-1 and Top-5 metries.
Number of Params
Top-1
Top-5
Method
FLOPS
Acc.
Acc.
Baseline
100%
100%
76.1%
92.9%
DDS-26 Huang et al.
(018b)
57.0 %
61.2 %
71.8 % 91.9 %
CP He ct al. (2017)
66.7 %
72.3 % 90.8 %
RRBP Zhou all 2019
45.4 %
73.0 %
91.0 %
SFP
al. (2018)
41.8 %
74.6 %
92.1 %
Hinge T
(2020s)
46.6 %
74.7
GBN-6
al.
(2019)
59.5
%
76.2
92.8
ResRop Ding al.
[022
45.5
76.2
92.9
DDS-26 Hu et al.
(0166)
57.0%
61.2%
71.8%
91.9%
ThiNet-50 500--o0LLooo L.uo al.
2017
44.2%
48.3%
71.0%
90.0%
RBP Zhou el. al.
2011
43.5%
48.0%
71.1%
90.0%
GHS Yang et al.
2019
52.9%
76.4%
93.1%
SCP Kang & Han
(2020)
45.7%
74.2%
92.0%
OTO Chen et al. 2021
34.5%
35.5%
74.7%
92.1%
WGSEF
34.2%
35.1%
74.2%
92.0%
10

---

Published in Transactions on Machine Learning Research (08/2024)
4.4 Evaluation of different group structures
In this experiment, we remonstrate WGSEF's performance across various architectures, datasets, and group structures. We specifically compare WGSEF to standard training using SGD without regularization. Our aim is to demonstrate WGSEF's flexibility by showing its ability to accommodate different group structures, allowing for consideration of the input data, model architeclure, hardware properties, elc. We examine the effectiveness of the WGSEF in the LeNet-5 convolutional neural network LeCun et al. (1998) (the architecture is Pytorch and not Caffe and is given in Appendix [A.5), on the MNIST dataset LeCun & Cortes (2010). The networks were trained without any data augmentation. We apply the WGSEF regularization on filters in convolutional layers using a predefined value for the sparsity level k. Table 4 summarizes the number of remaining filters at convergence, FLOPs, and the speedups. We evaluate these metrics both for LeNet-5 baseline (i.e., without sparsity learning), and our WGSEF sparsification technique. To ensure fair and accurate comparison, the baseline model was trained using SGD. It can be seen that WGSEF reduces the number of filters in the convolution layers by a factor of half, as dictated by k = 8, while the accuracy level did not decrease. Furthermore, since the sparsification is structural, there is a significant improvement in FLOPs, as well as in the latency time of inference. Repeating the same experiment, but now constraining the number of non-pruned filters in the second convolutional layer to be most 4 (i.e., most quarter of the bascline), the accuracy slightly deteriorates; however, significant improvements can be observed in both the FLOPs number and the speed up, as expected. The networks were trained with learning rate of 0.001, regularization magnitude À = 10-5, and a batch size of 32 for 150 epochs across 5 runs.
Table 4: Results of running Algorithm 2 onlo redundant filters in LeNet (in the order of conv convl-conv2).
LeNet-5 (MNIST)
Error
Filter (sparsity-level)
FLOPs
Speedup
Baseline (SGD)
0.84 %
6-16
100 %--00 %
1.00 x-1.00 X
WGSEF
0.78 %
3-8
48.7 2-2116 %
2.06x-4.53 x
WGSEF
0.89 %
3-4
48.7 %- %
2.06x-7.31
LeNet-5 (MNIST)
Error
Parameters
FLOPs
Speodup
Unstructured WGSEF
0.76 %
75(/ (700110))1200(/22000
68.7 %6-9.2 %
1x-1x
In Table 5 we present the results when training both VGG16 and DenseNet40 Huang et al. (2018a) CIFAR- 100 Krizhevsky et all while applying WGSEF regularization with predefined sparsity level with half the number of channels for VGG16, and 60% of those for the DenseNet40. The baseline model was trained using SGD, both with an initial learning rate of a = 0.01 and regularization magnitude X = 0.01.
Table 5: Results of running Algorithm 2 onto redundant Channels on CIFAR-100, over 250 epochs.
DCNN
Pruned
Overall Density
Model
Error (%)
CIFAR-100
Channels
Baseline
26.28
~ 0%
~ 0%
VGG16
WGSEF
26.46
50%
41.3%
Baseline
25.36
~ 0%
~ 0%
DenseNet40
WGSEF
25.6
60%
42.8%
4.5
Impact of sparsification levels on model error and training dynamics
Here, we examine the effectiveness of WGSEF in training the LeNet-5, the FasionMNIST dataset. The networks were trained without any data augmentation. We apply the WGSEF regularization on filters in convolutional layers using a predefined value for the sparsity level k. Table t summarizes the number of remaining filters at convergence, FLOPs, and the speedups. We evaluate these metrics both for a LeNet bascline (i.c., without sparsity learning), and our WGSEF sparsification technique. To be accurate and fair in comparison, the baseline model was trained using SGD. We use a learning rate equal to le 4, with a batch size of 32, momentum 0.95, and 15 epochs.
11

---

Published in Transactions on Machine Learning Research (08/2024)
Table 6: Results of training on FasionMNIST while applying WGSEF sparsification (with = 0.05), onto redundant filters in LeNet-5, (in the order of convl-conv2), and neurons in Linear layers. The baseline model was trained using SGD.
LeNet-5 (F-MNIST)
Filter
FC-layers sparsity
Error
Spoedup
(non-sparse)
Baseline
11.1 %
6-16
%
1.0 x-1.0x
WGSEF
11 %
3-8
5%
2x-4.5x
WGSEF
14 %
4-6
62%
1.7x-6.1x
WGSEF
12.3 %
2-3
1%
2x-7.12x
The second row of Table G shows that our method exhibits a significant decrease in the number of non-zero filters, specifically, half of the filters (groups of parameters) were mullified and at the same time the model performance is improved. The rest of the experiments show that there was higher sparsification in the number of non-zero filters (groups of parameters), with only negligible degradation in the model's Is accuracy. Finally, in Figure 2 we illustrate the sparsity level as a function of the epoch number, during the training of the model which corresponds to the last row of Table 6 The desired (predefined) sparsity level was rapidly attained within the first three epochs, while the model continued to improve its accuracy throughout the remaining epochs without compromising the achieved sparsity.
Sparsity Level Epoch Number
Accuracy us Epoch Number
80
87-
75
56-
ACCUrDCy
71-
52
16 15
Spock Mambe
Facch
Figure 2: The graph on the left shows the level of sparseness as a function epoch number, while the graph on the right shows the model's accuracy as a function of epoch number.
5 Discussion
In this study, we introduce a novel method for structured sparsification in neural network training, aiming to accelerate neural network inference and compress the neural network memory size, while minimizing the accuracy degradation (or improving accuracy). Our method utilizes new novel regularizer, termed weighted group sparse envelope function (WGSEF), which is adaptable for pruning different specified neuron groups (e.g convolutional filter, channels), according to the unique requirements of NPU's tensor arithmetic. Mathematically, the Weighted Group Sparse Envelope Function (WGSEF) represents the optimal convex underestimator of the combined sum of weighted f2 norms and lo norms. In this context, the to norm is applicd to the squared norms of cach group. During the neural network training, the WGSE regularizer selects the & most essential predefined neuron groups, where k that controls the compression of the network is configurable to the trainer. Consequently, the trained neural network benefits from reduced inference latency, more compact size, and decrcased power consumption. Additionally, we show that the computational complexity of the prox operator of WGSEF, a key component in the training phase, is linear relative to the number of group parameters. This ensures that it is highly efficient and which does not constitute a bottleneck in the calculation complexity of the training process.
The experimental results show the effectiveness of WGSEF in achieving high compression ratios (reduced memory demand), and speed up in inference, with negligible compromising in accuracy. Compared to the
12

---

Published in Transactions on Machine Learning Research (08/2024)
previous approaches, the proposed method stood out in its compression capabilities while maintaining similar network performance.
Along with the method's ability to predetermine the extent of network compression to be obtained at the training convergence, it essential to have a prior understanding of the maximum compression level that can be applied without compromising the network's performance, and accordingly set the parameter. Naturally, it is also necessary define the groups to which the pruming will be encouraged.
Future research could extend this work by delving into more intricate group definitions such as what could be defined in large language models. Moreover, we suggest studying a different mechanism for assessing the importance of each group being regulraized, as an alternative to the current approach which is based on the group's squared norm magnitude.
References
Zeyuan Allen-Zhu, Yuanzhi Li, and Yingyu Liang. Learning and generalization in overparameterized neural networks, going beyond two layers. Advances in neural information processing systems, 32, 2019.
Andreas Argyriou, Rina. Foygel, and Nathan Srebro. Sparse prediction with the k-support norm. Advances in Neural Information Processing Systems, 25, 2012.
Amir Beck. First-order methods in optimázation. SIAM, 2017.
Amir Beck and Yehonathan Refael. Sparse regularization via bidualization. Journal of Global Optimization, 82(3):463 482, 2022.
Kevin Bui, Fredrick Park, Shuai Zhang, Yingyong Qi, and Jack Xin. Structured sparsity of convolutional neural networks via nonconvex sparse group regularization. Frontiers in applied mathematics and statistics, pp. 62, 2021.
Tianyi Chen, Guanyi Wang, Tianyu Ding, Bo Ji, Sheng Yi, and Zhihui Zhu. Half-space proximal stochastic gradient method for group-sparsity regularized problem. arXiv preprint arXiv:2009. 120788 2020.
Tianyi Chen, Bo Ji, Tianyu Ding, Biyi Fang, Guanyi Wang, Zhihui Zhu, Luming Liang, Yixin Shi, Sheng Yi, and Xiao Tu. Only train once: A one-shot neural network training and pruning framework. Advances in Neural Information Processing Systems, 34:19637-19651, 2021.
Yu Cheng, Duo Wang, Pan Zhou, and Tao Zhang. A survey of model compression and acceleration for deep neural networks. arXie preprint arXiv: 710. 092809 2017.
Matthieu Courbariaux, Itay Hubara, Daniel Soudry, Ran Yaniv,, and Yoshua Bengio. Binarized neural networks: Training deep neural networks with weights and activations constrained to+ 1 or-1, arXiv preprinl Xix:::1602. 02830, 2016.
Yutong Dai, Tianyi Chen, Guanyi Wang, and Daniel Robinson. An adaptive half-space projection mothod for stochastic optimization problems with group sparse regularization. Transactions on Machine Learning Research, 2023.
Aaron Defazio, Francis Bach, and Simon Lacoste-Julien. Saga: fast incremental gradient method with support for non-strongly convex composite objectives. Advances in neural information processing systems, 27, 2014.
Tristan Deleu and Yoshua Bengio. Structured sparsity inducing adaptive optimizers for deep lcarning, 2023.
Alice Delmer, Anne Ferréol, and Pascal Larzabal. On the complementaritty of sparse 10 and cel0 regularized loss landscapes for doa estimation. Sensors, 21(18), 2021. ISSN 1424-8220. doi: 10.3390/s211860081 URL
https://www www.mdppiccoommm wtt.//ooo///// 14tp:/11444--- 421144444222222///666999
Lei Deng, Guoqi Li, Song Han, Luping Shi, and Yuan Xic. Model compression and hardware acceleration for neural networks: A comprehensive survey. Proceedings of the IEFF, 108(4):485 532, 2020.
13

---

Published in Transactions on Machine Learning Research (08/2024)
Emily L Denton, Wojeiech Zaremba, Joan Bruna, Yann LeCun, and Rob Fergus. Exploiting linear structure within convolutional networks for efficient evaluation. or Xrxiv preprinl. or Xre: 145.:105..888 3866, pp. 1277, 2014a.
Emily L Denton, Wojeiech Zaremba, Joan Bruna, Yann LeCun, and Rob Fergus. Exploiting linear structure within convolutional networks for efficient evaluation. Advances in neural information processing systems. 27, 2014b.
Xiaohan Ding, Tianxiang Hao, Jianchao Tan, Ji Liu, Jungong Han, Yuchen Guo, and Guiguang Ding. Resrep: Lossless cnn pruning via decoupling remembering and forgetting, 2021.
David I, Donoho and Michael Flad. Optimally sparse representation in general (nonorthogonal) dictionaries via 4 minimization. Proceedings of the National Academy of Sciences, 100(5):2197 2202, 2003.
Jianqing Fan and Runze Li. Variable selection via nonconcave penalized likelihood and its oracle properties. Journal of the Americon statistical Association, 96(456))1348-1335) 2001.
Saced Ghadimi, Guanghui Lan, and Hongchao Zhang. Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Progronmming, 155(1):267 305, 2016.
Amir Gholami, Sehoon Kim, Zhen Dong, Zhewei Yao, Michael W Mahoney, and Kurt Keutzer. A survey of quantization methods for efficient neural network inference. In Low- mow-Power Compuler Vision, pp. 291-326. Chapman and Hall/CRC, 2022.
Yunchao Gong, Liu Liu, Ming Yang, and Lubomir Bourdev. Compressing deep convolutional networks using vector quantization. arXiv preprint arXi::1412.60 2.151, 2014.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. Advances in neural information processing systems, 28, 2015.
Song Han, Huizi Mao, and William J. Dally. Deep compression: Compressing deep neural network with pruning, trained quantization and huffman coding, arXie preprinl xrXvv:1110.00144 2015b.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. 2015. URL http://arxiv, org/ abs//b12. 03385
CoRR, abs/1512.00
Yang He, Guoliang Kang, Xuanyi Dong, Yanwei Fu, and Yi Yang. Soft filter pruning for accelerating deep convolutional neural networks, 2018.
Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural networks, 2017.
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang. Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. Xixiv preprint orXiv:1704, 04861, 2017.
Hengyuan Hu, Rui Peng, Yu-Wing Tai, and Chi-Keung Tang, Network trimming: A data-driven neuron pruning approach towards efficient deep architectures. arXie preprint w:Xiw:6607003550 2016.
Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger. Densely connected convolu- tional networks, 2018a.
Zehao Huang, Naiyan Wang, and Naiyan Wang. Data-driven sparse structure selection for deep neural networks, 2018b.
Sashank J Reddi, Suvrit Sra, Barnabas Poczos, and Alexander Smola. Proximal stochastic methods for nonsmooth nonconvex finito-sum optimization. Advances in neural information processing systems, 29, 2016.
Max Jaderberg, Andrea Vedaldi, and Andrew Zisserman. Speeding up convolutional neural networks with low rank expansions. Xix preprint 405.::4405.3866, 2014.
14

---

Published in Transactions on Machine Learning Research (08/2024)
Ric Johnson and Tong Zhang. Accelerating stochastic gradient descent using predictive variance reduction. Aduances in neural informalion processing systems, 26, 2013.
Minsoo Kang and Bohyung Han. Operation-aware soft channel pruning using differentiable masks. In International Conference on Machine Learning, pp. 5122 5131. PMLR, 2020.
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar-100 (canadian institute for advanced research). URL htttp://ww..c. cs. www.ooooooooo.oooooooooooooo...t toroonto..du/kkrri/cchii.i html
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Vadim Lebedev and Victor Lempitsky. Fast convnets using group-wise brain damage, 2015. URL https:
//arxiv waa.ivv.org/bb///00000 0211111166111555
Vadim Lebedev, Yaroslav Ganin, Maksim Rakhuba, Ivan Oscledets, and Victor Lempitsky. Speeding-up convolutional neural networks using fine-tuned cp-decompasition. arXiv preprinl arXin:1 2014.
Yann LeCun and Corinna Cortes. Muist handwritten digit database, 2010. URL http://yann: lecun com/ exdb/nnist/
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to docu- ment recognition. Proceedings of the IEFE, 86(11):2278 2222222222,, 1998. doi: 10.1109 10.1109/5.726791.
Jiashi Li, Qi Qi, Jingyu Wang, Ce Ge, Yujian Li, Zhangzhang Yue, and Haifeng Sun. Oiesr: Out-in-channel sparsity regularization for compact deep neural networks. In Proceedings of the IEEE/CVF Conference on Compuler Vision and Pallern Recognition, pp. 7046 7055, 2019.
Yawei Li, Shuhang Gu, Christoph Mayer, Lue Van Gool, and Radu Timofte. Group sparsity: The hinge between filter pruning and decomposition for network compression. 2020 IEEE/CVE Confer- ence on Computer Vision and Pattern Recognition (CVPR), pp. 8015-8024, 2020a. URI. https:
//api. wemmaniicschollrr...oorororoooos wwwp:o/rrps.::21300000000 2130051111
Yawei Li, Shuhang Gu, Christoph Mayer, Lue Van Gool, and Radu Timofte. Group sparsity: The hinge between filter pruning and decomposition for network compression, 2020b.
Baoyuan Liu, Min Wang, Hassan Foroosh, Marshall Tappen, and Marianna Pensky. Sparse convolutional neural networks. The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang, Learning efficient convolutional networks through network slimming. In Proceedings of the TEEE international conference on computer vision, pp. 2736 2744, 2017.
Yifei Lou, Penghang Yin, Qi He, and Jack Xin. Computing sparse representation in a highly coherent dictionary based on difference of 1
1111 and 1212. Journal of Scientific Computing, 64:178-196, 2015.
Christos Louizos, Karen Ulirich, and Max Welling, Bayesian compression for deep learning. Advances in neural information processing systems, 30, 2017a.
Christos Louizos, Max Welling, and Diederik P Kingma. Learning sparse neural networks through 1_ regularization. Xr preprint arXiv: 712001311, 2017b.
Jian-Hao Luo, Jianxin Wu, and Weiyao Lin. Thinet: A filter level pruming method for deep neural network compression. In Proceedings of the IEEE international conference on computer vision, pp. 5058 5066, 2017.
Jinchi Lv and Yingying Fan. A unified approach to model selection and sparse recovery using regularized least squares. 2009.
Dmitry Molchanov, Arsenii Ashukha, and Dmitry Vetrov. Variational dropout sparsifies deep neural net- works. In International Conference on Machine Learning, pp. 2498 2507. PMI.R, 2017.
15

---

Published in Transactions on Machine Learning Research (08/2024)
Balas Kausik Natarajan. Sparse approximate solutions to linear systems. SIAM journal computing, 24 (2):227-234) 22222,234,, 1995.
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. In Computer Vision ECCV 2016, pp. 525 542. Springer, 2016.
Simone Scardapane, Danilo Comminiello, Amir Hussain, and Aurelio Uncini. Group sparse regularization for deep neural networks. Newrocomputing 241:81-800 89, 2017.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint 1rXix:1409. 1556, 2014.
Hoang Tran and Clayton Webster. A class of null space conditions for sparse recovery via nonconvex, non-separable minimizations. Results in Applied Mathematics, 3:100011, 2019.
Karen Ullrich, Edward Meeds, and Max Welling, Soft weight-sharing for neural network compression. arXiv preprint Xiv:: 1702. 04008, 2017.
J v. Neumann. Zur theoric der gesellschafssspicle. Mathematische annalen, 100(1):295-320) 1928.
Wei Wen, Chumpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured sparsity in deep neural networks. Advances in newral information processing systems, 29, 2016.
Yan Wu, Aoming Liu, Zhiwu Huang, Siwei Zhang, and Luc Van Gool. Neural architecture search as sparse supernet. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 10379 10387, 2021.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Pashion-mnist novel image dataset for benchmarking machine learning algorithms. Xiviv preprint 1 701.07787777777 2017.
Huanrui Yang, Wei Wen, and Hai Li. Deephoyer: Learning sparser neural network with diffcrentiable scale- invariant sparsity measures. Xrx preprint arXiv:1908.0900 09979, 2019.
Yang Yang, Yaxiong Yuan, Avraam Chatzimichailidis, Ruud JG van Sloun, Lei Lei, and Symeon Chatzinotas. Proxsgd: Training structured neural networks under regularization and constraints. In International Conference on Learning Representations (ICLR) 2020, 2020a.
Yibo Yang, Hongyang Li, Shan You, Fei Wang, Chen Qian, and Zhouchen Lin. Ista-nas: Efficient and consistent neural architecture search by sparse coding. Advances in Newral Information Processing Systems, 63:10503-1055 2020b.
Jianbo Ye, Xin Lu, Zhe Lin, and James Z Wang. Rethinking the maaallernoomm-les--inooonne assumption in channel pruning of convolution layers. arXiv preprint arXiv:1802.001 2018.
Jachong Yoon and Sung Ju Hwang. Combined group and exclusive sparsity for deep neural networks. In International Conference on Machine Learning, pp. 1058-3966. PMTR, 2017.
Zhonghui You, Kun Yan, Jinmian Ye, Meng Ma, and Ping Wang. Gate decorator: Global filter pruning method for accelerating deep convolutional neural networks, 2019.
Ming Yuan and Yi Lin. Model selection and estimation in regression with grouped variables. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 68(1):49-6) 67, 2006.
Jihun Yun, Aurélie C Lozano, and Eunho Yang. Adaptive proximal gradient methods for structured neural networks. Advances Neural Information Processing Systems, 34:24365-24978, 2021.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Rocht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107-115, 2021.
16

---

Published in Transactions on Machine Learning Research (08/2024)
Yang Zhou, Rong Jin, and Steven Chu-Hong Hoi. Exclusive lasso for multi-task feature selection. In Proceedings of the thirleenth international conference artificial intelligence and stalistics, pp. 998- JMLR Workshop and Conference Proceedings, 2010.
Yuefu Zhou, Ya Zhang, Yanfeng Wang, and Qi Tian. Accelerate cnn via recursive bayesian pruning. In
Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3306-3315, 2019.
Hui Zou and Trevor Hastie. Regularization and variable selection via the elastic net. Journal of the royal statistical society: series B (statistical methodology), (2):3330 320, 2005.
A Appendix
A.1 Proofs of results in Subsection 2.2
A.1.1 Proof of Lemma 1
Proof. Let us define the axillary diagonal positive definite matrix D"x", where the DA, entry holds D vd,, Vi E &g. = dz. Now, consider the following chain of equalities:
gs max (0, gsx BER"
T
d 1110,1122 - dc, (0)
max BER"
2 1=1
max AEC₁ ViEs :i=dd
T Σ
d 110s;112
j=1
TT 1 T D T DO
max BECC
2
di==)
1
max rEC.
0 DDD 4
ttt
2
ViEsj,d;=d,
max tEC, ViEsy,di=d,
b1lt D 0-10
D
01
{
1
D 10112 +
2 ||t D 0-1013
max AEC,
2
ViEa,, mdy
1ID-1111
(4 - (D-1), ss 0,)2
max LEC2
2
di=
i=1
(6)
2 Σ
Maj (1 --100
max TECx
diE
m
D
10112 2 j=m-
Σ
M Miss) (D DD-10)11
2
(
| M(ee) (D 11-10)11
1=1
1
|| | (1)0
2
17

---

Published in Transactions on Machine Learning Research (08/2024)
where, (a) the set Ck is given by
Ck = {0: ||| (DO) 81 s1l12,11 (D0) s21l?.... (D0) 8m 12 s kk)
(b) follows by the fact that the sum of the squares of the coordinates the imput vector in the support of the disjoint subset $1, , Sm that completes the index space U 8, = [n| equal the sum of squares of the coordinates of the original vector, and (c) follows by the fact that -j=m= (M) is the sum of square f--norm of the m k disjoint subsets of D D with the smallest f-rnm,, while is the sum of squared fo-morm of all disjoint subsets of D-16
A.1.2 Proof of Lemma
Proof. We first note that
max ac B,
U4 ||, (D-10)11
(13)
where
Bg - f 10
(14)
Now, Consider the following chain of inequalities:
(0) max 8ER
{00, 03 - gss (0)}
k
TT
Σ
M (D- (D-10)11
max BER"
2 t=1
max PER" 1ñ
T DI
Mon. 10011
2 1=1
max tER"
0 Dt
1
max
m
Ue M| (0)1
2
DE
8 Dt
Uj (t (TTAT, Ao,t
max tER"
max
2 DE
1
max YER"
T Dt
min 2 DED,
Σ (-w)) (t A0,
1
max
min uED
20 DI.
Σ Ugl AngL
2
FE
(b) 1
min DED,
20 DI
uzt' Asst
2
j=1
1
min
max 20 As, Dt - ugt.Ast EER
2 De
==1
min 2 D
Σ
max 2 2 T uuttAst tER*
A,,t
18

---

Published in Transactions on Machine Learning Research (08/2024)
Σ (Vd.Aa,0.u))
m
min 2 wr Die
(15)
1 min
m
d3 (10 0,4,0,,)))
D
where (a) follows by the fact that As, is self-adjoint matrix, and (b) follows from the fact that the objective function is concave w.r.t. and convex w.r.t u, and the MinMax Theorem v. Neumann 1928
A.1.3 Proof of Corollary 2.1
Proof. Directly by expression (34).
A.2 Proofs of results in Subsection 2.3
A.2.1 Proof of Lemma
Proof. Recall that
prox (t) argmin (GS)(0) + 2=10 e13 dER"
prox xgs,
Using Lemma 2
the above minimization problem can be written
m
1 5110 0 - 11/ S 0,
min min ueD, BER"
$(0, u, 1)
j=1
m
min min nED, lERn
((0, t, = Σ
d 0, + Mo, (0 -t)3
j=l
X m
min min ue DER
Φ u1,
Σ dip (A, 0,5) =0 1) A., -1)
(16)
Solving for 0, get that for any [m], if d, A,;
0 then,
d dyAA,,0 0
As, (0 =
Uj
Y,p
A
+ 1
Ag,t
Uj
djA
As,
0
+1
-t) = 0,
Uj
meaning that,
tug
=
Ady Uj
j E [m]. 83.
(17)
or, equivalently,
Uy.9,, [m].
A,, ddl + uj
(18)
Next, show that is the minimizer of the problem minnED, (00 u, t). Equation equation 18 also holds when uj 0, since in that case, is = 0: 0, for all i E sy. Plugging equation 18 in Q yields,
m
(x AA,
(0,,
u, Σd dj
1 t
(19)
Ui
19

---

Published in Transactions on Machine Learning Research (08/2024)
77
+ ||,ng (0 (0-)) 112 θT AA,,
Ad;
==1
ustT Asst
Adz Ui 133
Ad2 As, 12
Adj Ug (Ad; + u,}²
2
Ust Ac,t Adj (Xdz + uz)
(t As,t (Ad, uz}²
+
X m
As²
dj Adj + u4j
2 j=1
Σ Φ (dd +
(20)
2
/=1
which concludes the proof.
A.2.2 Proof of Corollary 3.1
Proof. Assigning Lagrange multiplier for the inequality constraint e² S. l: in problem 16 we obtain the Lagrangian function
[(u, E ( (V Ad, tug) uwa))) kjp.
Therefore, the dual objective function is given by
a(n) min u:OSuee
L(u (4) = Σ Pbyyas (1) kjj,
(21)
for, bj = Vd, A., 11 and a₁ Adj(> 0), where for any R and a 2 O, the function P6,ax defined
Beck & Refael (2022) by
Pb (1)
min (o(b, a + u) + pu), 1 0. 05w51
(22)
Thus, the dual of problem is the maximization problem
max(g(1) : p 2 0}
(23)
Lemma 2.4) is that if j > 0, the function u u I(u, p1) has a unique minimizer over {u E R" :0 < u <e e) given by uy = P, (ū), where it was shown that
A direct projection of Lemma (Beck & Refael 2022
62 a,tl + H4
VF S
1b,1
a,t1
ab (14)
2|by V agp,
0 V
1b11
bi1
0g
6a
/p a,
1b1
a,
for > 0, otherwise 0, and the minimizer is given by
Vp S bb,1
1,
ayt1'
u(x²)) =
bb
aj.
a
1b:1 < v
vp 2 1b,
0,
aj
Problem 23 is concave differentiable and thus the minimizer jú holds o' (ūt 0, meaning
((()= Σ (g(4) k =.
20

---

Published in Transactions on Machine Learning Research (08/2024)
We observe that for any E [m] the functions (z(4) are monotonically continuous nonincresing, and therefore
utilizing Lemma (Beck & Refael 2022 Lemma 3.1) nr x7z is the root of the nondecreasing function,
9:(n) = uy(n) - k.
Note that for
A -mnggg/m {dp}
9t
[[ ||A,, 41 vdl 1 ..... .v den A|A,,, c1le
0 k <<00 0,
==1
while
> Ald1, d2,
dm loc +1
m
Σ 11
gt
(mm)
dgt) 11
Now, applying Lemma (Beck & Refael
2022
Lemma 3.2), we deduce that uz (p), can be divided into the
sum of the two following functions,
(n) In/,, - ayI. wn()) = 1 1n-,,1,,ll-- (ay + 1)1.3 E [m]]
and thus Ir can be reformulated as follows
(c(n) mn(n) + ト Σ wn(T) k.
j=1
A.3 Proof of Corollary 3.2
Proof. The proof follows from
Yun et al. 2021 Corollary 1), by laking C. = 0 and d = 1.
A.4 Discussion on the selection of the optimization method
The general training problem we are solving is of the form
f(x) M(x), xER
min
where f f= + f: X R is continuously differentiable, but possibly nonconvex, and h is convex function, but possibly nonsmooth. For practical reasons, cannot store the full gradient ff(x). Hence, we would like to use stochastic gradient type algorithm. However, such structure posses several difficulties from an optimization perspective, as most research of stochastic first-order algorithms does not account for both nonconvex smooth term and a nonsmooth convex regularize. In Ghadimi et al. (2016) the authors provide an analysis of simple stochastic proximal gradient. algorithm, where at each iteration a minibatch of weights updatod using a gradient step followed by a proximal step. This algorithm is proved to converge, however, the rate of convergence depends heavily on the minibatch size, and, in fact, for reasonably sized variance-reduucionn minibatches it will not converge. J Reddi et al. (2016) proposes variance-rediee reduction type algorithms, but since these extend SAGA Defazio et al. (2014) and SVRG JJhnssn & Zhang (2013) to the nonconvex and nonsmooth setting, they require storing the gradient for each sample (SAGA) which requires O(Nn) storage, or recomputing the full gradient every > N iterations (SVRG), which is undesirable for training neural networks.
The ProxSGD algorithm appears appealing to our problem as it allows for momentum. While the algorithm has convergence guarantee, the authors do not provide the rate, making it less appealing, given the known
21

---

Published in Transactions on Machine Learning Research (08/2024)
issue with the minibatch size. We have found the most suitable optimization algorithm to be ProxGen Yun et al. (2021), it can accommodate momentum, and also has proven convergence rate with a fixed and reasonable minibatch size of order @(vN). Next, we provide a convergence guarantee for Algorithm as given in Yang et al. (2020a). The convergence is in terms of the subdifferential defined as follows.
Definition 2 (Fréchet Subdifferential). Let &p be a real-valued function. The Préchet subdifferential of al with (p(0) < oo is defined by
0Φ(σ) a for EΩ lim inf ((()) ((0)
(8", 0 8
0
8-+0
|0
A.4.1 Assumptions
The following assumptions in terms the objective function f and the algorithm parameters are required:
(C-1) (T-smoothness) The loss function f is differentiable, T-smooth, and lower-bounded:
[Vf(2) vf(ii - L||x vll and f(=")>
(C-2) (Bounded variance) The stochastic gradicnt 9r (f(:(E) is unbiased with bounded variance:
Be [V[ vO())), Eg [lls: ((0))) oo
(C-3) (i) Final step-vector is finite, (ii) the stochastic gradient is bounded, and (iii) the momentum pa- rameter exponentially decaying, namely,
ii) 10e+1 0ell S D, (ii) 1 |l S G,
(iii
Pok
with D,G> 0 and Po. / E 1).
Based on these assumptions we can state the following general convergence guarantee.
A.5
LeNet convolutional neural network architecture
# filters neurons
Size
Activation function
Layer
Filter size
Stride
feature map
Input
32 x 32 1
Conv
6
5 x
28 x 28
Relu
MaxPool2d
2 x
14 14
Conv
16
5
x
10 x 10 x 16
Relu
MaxPool2d
2 x 2
5 x x 16
Fully Connected
120
Relu
Fully Connected
84
Relu
Fully Counected
10
Softmax
A.6
Additional algorithm
22

---

Published in Transactions on Machine Learning Research (08/2024)
Algorithm 3: General Stochastic Proximal Gradient Method
Input: Stepsize a,, {P:)E= E [0, 1), regularization parameter X, switch condition S, projection threshold c.
Initialization: 01 E R" and my = 0 E R".
for iteration t I:
if condition is not satisfied:
Apply Algorithm 2
else:
Draw a minibatch sample &
g: 00 i + (h :()
07 0:-1 - 0ggr. 0
for each group 7 E 170,
if (07 1 6 0- 11:
0
0++1 07
return 0
23


Reviews:
[{"review_id": "SCWIwf2Tfu", "reviewer": "Reviewer_ah6q", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strength\n\nThe focus on training sparse neural networks addresses a critical challenge in the field of machine learning. Modern neural networks are computationally intensive and inefficient in terms of parameter usage. The proposed method of this paper trains the model from scratch only once to yield structurally sparse networks. Compared to post-training pruning techniques and unstructured methods, this technique can substantially reduce both training and inference costs. This suggests a high potential for practical applications.\n\n## Weakness\n1. The experiments are not conducted in a unified framework. Various models were trained and evaluated under different conditions. For instance,  train ResNet40 on CIFAR-10 compared with other regularizers, train Resent50 on ImageNet compared with pruning techniques, and train LeNet-5 on MNIST/F-MNIST compared with SGD). This may affect the consistency and comparability of the results.\n\n2. The presentation of this paper can be improved. There are various typos and unclear specifications, for examples:\n- Page 2: \"selects the most essential $k \\leq m$ predefined groups\", but $m$ is not previously defined.\n- Page 4: It is not clear why $\\delta_{C_k}(\\Theta)$ takes the whole parameter space $\\Theta$ as input.\n- Page 5, Remark 1: $s_k(\\theta)$ is defined to be a function depending on the scalar $k$, not related to the index set $s_j$. This creates great confusion.\n- Page 10, Table 3: Why the *numbers of parameters* for half of the methods are not presented?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a novel approach for training sparse neural networks. Specifically, the approach starts with a dense network and employs a new regularization technique called the Weighted Group Sparse Envelope Function (WGSEF). This technique dynamically nullifies groups of neurons throughout the training process. The authors also develop new proximal-gradient-based optimization algorithms to handle the WGSEF regularization effectively. By the end of the training process, this method successfully achieves a structurally sparse network that meets a predefined sparsity level. Experimental results demonstrate that the proposed method outperforms traditional sparsity-inducing techniques, such as regularization or pruning, by achieving lower computational costs and higher sparsity levels while maintaining comparable accuracy.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. It is recommended to conduct the experiments within a unified framework. While it may be challenging to accommodate all the settings described in this paper into a single framework, any effort would be beneficial. \n\n2. The paper should undergo thorough proofreading to correct typos and unclear specifications.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "BjGZQ4Hqbd", "reviewer": "Reviewer_kBUo", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The method is well motivated\n2. The paper in fairly well written\n3. Empirical results seem good\n\nWeakness:\n1. It is a relatively straightforward combination of existing methods and thus not terribly creative", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The reviewed work is concerned with the sparsity inducing training of neural networks.\n\nArguably the most common approach is $L^1$ regularization \"Lasso.\" In some circumstances, this regularization can be improved upon by using the \"elastic net regularizer\" that consists of a sum of $L^1$ and $L^2$ regularization. Recently, an alternative approach was introduced by Beck and Refael that instead computes the bidual of the regularizer $\\|\\cdot\\|_{L^2} + \\|\\cdot\\|_{L^2}$. \n\nA limitation of all the above regularization schemes is that the resulting unstructured sparsity can not efficiently leverage the tensor-level computational primitives that enable the efficiency of modern computing hardware. This has motivated previous authors to propose \"group versions\" that enforce sparsity not on the level of individual entreis, but on the level of groups of variables, thus enabling the use of efficient batched operations while benefitting from sparsification.\n\nThe reviewed work proposes a group version of the bidualization approach to be used in the sparse training of neural networks. The authors prove a theoretical result on the convergence of the resulting proximal gradient descent algorithm and show its practical performance in the sparsity inducing training of neural networks", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "No request for changes, although I recommend some more spell/typo checking", "broader_impact_concerns": "No concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "YzHa2TD02P", "reviewer": "Reviewer_smJw", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength: \n1) The paper is well-written and easy to follow, with extensive review of related works. The key ideas are articulated well. \n2) The proposed method shows strong performance in numerical tests.  \n\nWeakness: \n1) The paper appears to build on the previous work Beck & Refael (2022), extending the idea of sparse envelope function (SEF) to group sparse envelop function (WGSEF) to enforce group sparsity and applying to neural network compression problem. While this extension may not be a trivial task, I am not convinced that the paper introduces significant technical novelty. There seems a notable similarity between the development of convex relaxation and proximal mapping for WGSEF (Section 2) and SEF in the aforementioned paper.   \n2) The authors mentioned that the proposed approach aims to reduce memory demand, power consumption and bridge the DNN hardware deployment challenges. Please elaborate (and quantify if possible) how the results in the numerical tests relate to this goal? \n3) It is not clear how the neurons are grouped in most of the numerical tests. Are they all according to convolutional filter? Is the performance of WGSEF sensitive to the definition of groups? An ablation study would be helpful. \n4) The authors presented multiple tests in the Experiments section, but there is only one test comparing the method with the SOTA pruning techniques. Here, the performance of WGSEF and the existing method OTO were shown very close. Additional evaluations would be needed to firmly establish the advantage of WGSEF over other baselines.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper develops a new group sparse regularization to compress neural network models without compromising its accuracy.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See comments in the Weakness part.", "broader_impact_concerns": "No concerns on the ethical implications of the work", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers are in agreement that the paper can be accepted for TMLR, with the paper's main claims being sufficiently supported by both theoretical and empirical evidence, and the results clearly being of interest to some ML researchers.  At least two reviewers had some concerns regarding the novelty, but the authors gave reasonable responses regarding this, and in any case, novelty is not a main factor for TMLR.  No concerns were raised regarding correctness.

The reviewers didn't list any required changes, but the authors may want to look over the reviews and/or latest changes once more just in case.

I would also encourage the authors to consider the wording of the main claims, e.g.:
"The properties of the WGSEF allow to pre-define the desired sparsity level that would be achieved at the training convergence while maintaining negligible network accuracy degradation..."
This wording sounds strange, because it must be impossible to pre-define an arbitrary sparsity level without degradation (because if it's too low, there will be unavoidable degradations).  I suggest some re-phrasing to preclude potential concerns of this kind, and similarly double-checking the other main claims made.

As a minor style point, in the Appendix A headings (and possibly elswhere), replace lemma -> Lemma, section -> Section, etc.
Decision: Accept (poster)