A UNIFIED VIEW OF FINDING AND TRANSFORMING WINNING LOTTERY TICKETS
Authors: Kun Wang
Yuxuan Liang
Pengkun Wang
Pengfei Gu
Zhengyang Zhou
Chao Huang
Yang Wang
Venue: Submitted to ICLR 2023 (2023)
Keywords: Lottery Tickets Hypothesis
Dual Lottery Tickets Hypothesis
Non-linear increased regularization
early stopping

Abstract: While over-parameterized deep neural networks obtain prominent results on various machine learning tasks, their superfluous parameters usually make model training and inference notoriously inefficient. Lottery Ticket Hypothesis (LTH) addresses this issue from a novel perspective: it articulates that there always exist sparse and admirable subnetworks in a randomly initialized dense network, which can be realized by an iterative pruning strategy. Dual Lottery Ticket Hypothesis (DLTH) further investigates sparse network training from a complementary view. Concretely, it introduces a gradually increased regularization term to transform a dense network to an ultra-light subnetwork without sacrificing learning capacity. After revisiting the success of LTH and DLTH, we unify these two research lines by coupling the stability of iterative pruning and the excellent performance of increased regularization, resulting in two new algorithms (UniLTH and UniDLTH) for finding and transforming winning tickets, respectively. Unlike either LTH without regularization or DLTH which applies regularization across the training, our methods first train the network without any regularization force until the model reaches a certain point (i.e., the validation loss does not decrease for several epochs), and then employ increased regularization for information extrusion and iteratively perform magnitude pruning till the end. We theoretically prove that the early stopping mechanism acts analogously as regularization and can help the optimization trajectory stop at a particularly better point in space than regularization. This not only prevent the parameters from being excessively skewed to the training distribution (over-fitting), but also better stimulate the network potential to obtain more powerful subnetworks. Extensive experiments are conducted to show the superiority of our methods in terms of accuracy and sparsity. 

Under review as a conference paper at ICLR 2023
A UNIFIED VIEW OF FINDING AND TRANSFORMING WINNING LOTTERY TICKETS
Anonymous authors Paper under double-blind review
ABSTRACT
While werrparamettrized deep neural networks obtain prominent results on var- ious machine learning tasks, their superfluous parameters usually make model training and inference notoriously inefficient. Lottery Ticket Hypothesis (LTH) addresses this issue from a novel perspective: articulates that there always ex- ist sparse and admirable subnetworks in randomly initialized dense network, which can be realized by an iterative pruning strategy. Dual Lottery Ticket Hy- pothesis (DLTH) further investigates sparse network training from complemen- tary view. Concretely, it introduces a gradually increased regularization term to transform dense network
ultra-light subnetwork without sacrificing learn- ing capacity. After revisiting the success of LTH and DLTH, we unify these two research lines by coupling the stability of iterative pruning and the excellent per- formance of increased regularization, resulting in two new algorithms (UniLTH and UniDLTH) for finding and transforming winning tickets, respectively. Unlike either LTH without regularization or DLTH which applies regularization across the training, our methods first train the network without any regularization force until the model reaches a certain point (i.e., the validation loss does not decrease for several epochs), and then employ increased regularization for information ex- trusion and iteratively perform magnitude pruning till the end. We theoretically prove that the early stopping mechanism acts analogously as regularization and can help the optimization trajectory stop at particularly better point in space than regularization. This does not only prevent the parameters from being excessively skewed to the training distribution (over-fitting) but also better stimulate the net- work potential to obtain more powerful subnetworks. Extensive experiments show the superiority of our methods in terms of accuracy and sparsity.
INTRODUCTION
Exactly as saying goes: you can't have your cake and eat it though over-parameterized deep neural networks achieve encouraging performance over widespread machine learning tasks Zagoruyko & Komodakis 2016);; Arora et al. (2019): Devlin et al.] (2018); Brown et al. (2020), they usually suffer notoriously high computational costs and necessitate unaffordable storage resources [Cheng et al. (017); Deng al. (2020); Wang et al. (2019a). To alleviate this issue, stream of pruning approaches Han et al. (2015) Liu et al. [201 7); He et al. (2017): Gale et al. (2019); Ding et al. (2019) tries to uncover a sparse subnetwork that can retain the learning capacity of the original dense network as much as possible. While these algorithms seek to reach a preferable trade-off between performance and sparsity, they fall short of satisfying the joint optimization of both.
Recently, Lottery Ticket Hypothesis (LTH) has provided a novel perspective to investigate sparse network training Frankle & Carbin 2018). It articulates that there consistently exist sparse and high- performance subnetworks in a randomly initialized dense network, like winning tickets in lottery pool. To identify such admirable sparse subnetworks (i.e., winning tickets), LTH trains an over- parameterized neural network from scratch and prunes its smallest-magnitude weights iteratively, which is so called iterative pruning. This repeated pruning method, as opposed to one-shot pruning, allows us to learn faster and achieve higher test accuracy at smaller network size. LTH innovatively exposes the internal relationships between a randomly initialized network and its corresponding subnetworks, inspiring series of follow-ups to explore various iterative pruning and rewind criteria for training light-weight networks [Morcos et al. (2019); Maenee et al.] (202T); Chen et al. (202T);
Frankle et al. (2019 2020); Ding et al. (2021): Ma al. (2021); Chen al. (2022).

---

Under review as a conference paper at ICLR 2023
Pre-training
Pruning
Randomly initialized
Iterative
Train
Randomly initialized
Winning ticket
Training without Reg
Rewind
Checking validation loss
Early stopping with patience d
Rei
A Reg: regularization
Rewinding epochs
Lottery ticket hypothesis (LTH)
Stage
(c) Graphical Illustration
Dual lottery ticket hypothesis (DLTH)
Stoge 2
Increased Reg for all weights
Random selection
Increased Reg for discarded weights
t
Iterative
Iterative
Randomly initialized
Winning
ticket
Pruning
Selected
Pruning
structure
Increased
Winning
Discarded
Winning ticket
Random
Reg
Transform
ticket
weights
selection
& Pruning
UniLTH
UniDLTH
(a) Pipeline of LTH and DLTH
(b) Pipeline of our UniLTH and UniDLTH
Figure I: Illustration of LTH/DLTH and our UniLTH/UniDLTH In (c), the blue/green solid contour lines denote the contours of the training/validation negative log-likelihood. Our goal is to drew the weights closer to u. The black line indicates the training trajectory taken by SGD. Our algorithm rewinds the training procedure (the yellow line) and add increased regularization (the purple line) to move towards the validation set distribution when training reaches an early stopping threshold.
Though promising, LTH concentrates solely on identifying one sparse subnetwork by iterative prun- ing, which is not universal to both practical usages and investigating the relationship between dense networks and subnetworks Bai et, al. al. (2022). Hence, [Bai et al. (2022) go from a complementary di- rection to propose Dual I Lottery Hypothesis (DLTH) which studies randomly selected subnetwork rather than particular one. As dual problem of L.TH, it hypothesizes that randomly selected sub- network in randomly initialized dense network can be turned into appropriate condition with excellent performance, analogy to transforming a random lottery ticket to a winning ticket. To val- idate this, DLTH trains a dense network and conducts one-shot pruning with simple yet effective strategy it identifies the sparse subnetwork by utilizing a gradually increased regularization term throughout the training phase, which extrudes information from unimportant weights (which will be pruned afterward) to target sparse neural structure. Although this hypothesis does not provide any theoretical proof how much information extrusion we can achieve, it does provide novel view on harnessing regularization terms to link the dense network with hidden winning tickets.
As the key element to DLTH's success, the regularization term realizes information extrusion from the unimportant weights which will be masked (i.e., discarded), but may also become its undoing. In a training process, the equilibrium of all network weights is usually determined by two forces: loss gradient force and regularization gradient force. The latter one is generally maintained in small regime, as the excessive weight penalty will cause the network to collapse into a suboptimal local minimum, corresponding
the ill ill-conditioned small weights LeCun et al. (2015). Using a regu- larization term at the early training phase as DLTH dlees, may cripple the model performance since complicates the network optimization and misleads the finding reliable equilibrium. Mean- while, regularization-base pruning approaches (e.g., DLTH) typically perform one-shot pruning, which exacerbates the instability of sparse network training. Considering the efficacy of iterative pruning in LTH, transforming random tickets into winning tickets iteratively is appealing as well.
In this paper, we aim at presenting resilient and unified paradigm for searching winning tickets in dense network (LTH) or transforming random tickets to winning tickets (DLTH), leading to two new pruning algorithms termed UniLTH and UniDLTH. As illustrated in Fig. [b). both UniLTH and UniDLTH decouple the pruning task into two separate stages. At the first stage, the two algorithms share identical procedure they do not set up any obstacle force (regularization) when training randomly initialized network. Once the validation loss does not decrease for several training cycles, cut off the training and rewind the network parameters to several epochs earlier. We demonstrate that utilizing such an early stopping strategy can defeat the instability caused by regularization, thus achieving similar or even better performance without compromising the learning potential.
2

---

Under review as a conference paper at ICLR 2023
At the second stage, we integrate the iterative pruning with the increased regularization for searching or transforming winning tickets. To be more specific, we altemnaely train the network with increased regularization and perform pruning to tilt the network distribution towards the validation distribu- tion until the network reaches the corresponding sparsity. The major difference between UniLTH and UniDLTH is the weights to which we apply regularization (see Fig. [b)) UniLTH differen- tiates the magnitudes of the weights by applying progressively increasing regularization on all the weights, while UniDLTH applies L2 regularization only on the unimportant weights for information extrusion. The contributions of this paper can be summarized as follows:
We introduce unified view of searching and transforming winning lottery tickets. We find that the removal of regularization force of the early training phase more helpful for preserving network expressivity. This simple yet efficient winning ticket search/transfom paradigm can seamlessly translate to arbitrary networks without being subject to specific network structure.
We provide theoretical proof of the early stopping strategy's 's ability to substitute L2 regular- ization, as well as an intuitive explanation of its benefits. This new training paradigm show great promise in retrieving winning tickets from large dense network. We also verify that networks can perform better under lower regularization pressures with a novel nonlinear regularization scheme.
We conduct extensive experiments to evaluate our algorithms mance. In particular, UniL.TH outperforms L.TH by 0. 0.19%~11. .35% and UniDLTH surpasses DLTTH 0.29%~0.566% on accuracy over four representative backbones on the CIFAR- 10 dataset. Remark- ably, we can even obtain 90% sparse winning tickets without performance dropping, especially on the large-scale dataset ImageNet, which verifies the superiority of our algorithms.
terms of sparsity and perfor-
2 RELATED WORK
Winning Lottery Tickets. LTT elaborately draw an analogy between uncovering admirable subnet- works in a dense network and finding winning tickets in lottery pool. It articulates that randomly initialized dense network contains a fiigh-performacc subnetwork which can be trained in isolation for at most the same number of iterations the original network Frankle & Carbin (2018). In light of LTH, some follow-ups have explored the prospect of training sparse subnetworks in place of the entire models without sacrificing performance Malach et al. (2020). LTH is also adopted to discover the presence of "supermasks", which can transform an untrained, randomly initialized network to far higher-performance model /Zhou et al. (2019). Furthermore, stream of research has been dedicated to the findings of early-bird (EB) tickets You et al. (2019) (the tickets which emerge at the very early training phase) to reduce the computational overheads, c.g., in graph neural networks You iet al. (2021): Chen et al. (2021). in natural language processing Chen et al. (2020). To generalize the lottery tickets across tasks, Zhou et al. (019); Morcos et al. (2019) verify the performance of the winning ticket initializations generated by sufficienlly large datasets, and discover these sub- networks contain inductive biases generic to neural networks more broadly which improve training across many tasks. In addition to the above work, DLTH considers more general and challenging case to find the relationship between a dense network and its sparse counterparts Bai et al. (2022) it argues that randomly selected subnetwork from dense network can be transformed into trainable condition and achieve admirable performance compared with the winning lottery pool Bai et al.) ) which suggests a more adjustable way of investigating sparse neural networks.
Regularization-basee Pruning. Regularization has long been exploited for pruning deep neural networks by enforcing part of parameters in the original network to zeros. The most popular ap- proaches are using Lo-norm or -norm Louizos et al. 20: Liu et al. (2017): |): Ye et al. (2018). For instance. He et al. (0177) LASSO to achieve channel pruning for accelerating very deep neural networks. following this trend, the Group LASSO algorithm is further introduced to obtain a regular sparse subnetwork Lebedev & Lempitsky (2166; Wen et al. (016). Ding et al. (2018) proposes to employ various penalty factors for different weights. Among these works, the regularization force" maintained in small regime to avoid crippling the model performance. To take advantage of the model sparseness brought by the large penalty strength. Waan et al. (2019b) 2020) present the first attempt to utilize gradually increased regularization terms to achieve high sparsity and preserve the admirable performance of the original model. However, the magnitude of the regularization term is extremely important and needs to be carefully controlled, since an excessive weight penalty will may cause the model weights ill-conditioned. In this paper, we focus the regularization term to discover more reliable subnetworks, i.c., winning tickets, while ensuring that the regularization term does not have catastrophic influence the training process.
simple and efficient usage of
3

---

Under review as a conference paper at ICLR 2023
3 METHODOLOGY
3.1 A UNIFIED APPROACH FOR SOLVING LTH AND DLTH
Here, we delineate our algorithm which unifies the line of LTH and DLTH to obtain winning lottery tickets. As depicted in Algo. [ it can be flexibly adopted to the settings of LTH and DLTH with few modifications. We denote in blue the procedures for uncovering the winning tickets (marked as UniLTH), and in red the procedures for transforming random selected subnetwork to the winning tickets (i.e., UniDLTH). The other lines (in black) are shared by both.
UniDLTH. We first describe UniDLTH which aims to transform randomly-selected tickets to the winning tickets. Unlike DLTH, we do not introduce any regularization force at the early training phase to ensure that the model accurately learns the training data distribution. Instead, we perform training, early stopping, and rewind at the first stage (Line 1-6) when the validation loss does not drop in epochs, we stop training and rewind the parameters to epochs earlier. Then produce the random tickets by randomly selecting a set of important parameters (the rest are unimportant) in Line 7. After rewinding all weights, we integrate the iterative pruning strategy with the gradually increased regularization (Line 8-16) to extrude information from unimportant weights O.n to target the sparse structure, i.e., the winning tickets 4m.
UniLTH. For UniLTH, our target is to find the winning tickets in a dense, random initialized neural network. The whole pipeline is shown in Algo. and almost similar UniDLTH, thus we mainly highlight how UniLTH differs from UniDLTH. Firstly, UniLTH does not need to specify a random set of weights to realize information extrusion as UniDLTH does the second stage (in Line 7). Secondly, UniLTH applies regularization to the universe of the network while UniDLTH merely decays the unimportant weights (see Line 9). Lastly, UniDLITH prunes the non-important weights at each iteration, whereas UniL.TH has no such restriction (see Line 13).
As described, our pruning algorithm is iterative-based Suppose a dense network has been pruned for i iterations and each round cut off p% of the weights that survive in the previous iteration, we target sparse structure to be r% of the original network size. In this case, p can be expressed as p% = 1 //V and decreased over iterations. Notably, such pruning rate becomes very low near the end of the pruning, which analogy to the fine-tuning technique. The rest of this section are organized as follows. In Sec. [3.2] we will theoretically prove the equivalence between early stopping and regularization, and explain why we should employ early stopping and rewinding rather than regularization at the early stage. In Sec. 3.3, we will elaborate on the second stage, i.e., iterative pruning with three variants of nonlinear increased regularization.
Algorithm UniLTH and UniDLTH Algorithms (aligned with Fig.
Require: Network f(X, θ) with data X and parameters Q; sparsity level Sy; mask matrix me with initial- ization mg 1; pruning rate p%; times vi: step size y/ patience for early stopping.
1: while |mmella Sy do
Jm
Forward f(X, θ) to compute loss L. (X,0)
3: Update @0+1) +- ( V C.
4: the validation loss is not descending for & epoch then
5: Rewind O to several training epochs ago (Note weights as (())
6: Load weight (E)
7: Select unimportant weights O... and important weights Oim
8: for iteration i = 2, ..... do
9:
Forward f(X; mn; θ) to compute the loss C. = L. (X, me O)) + a 1|01 (or C. C. (X, me @ θ) + a [|0mnl12). where a hyper-parameter to control sparsity
10:
Update ((++)) ( vVoe
11:
Adopt nonlinear increasing regularization (i.c., LogLaw, LanLaw and ExpLaw.
12:
if ==
= 1,2... then
13:
Prune 5% parameters with the lowest magnitude values in θ (or @..)
14:
Update me by zeroing the elements in me which corresponding the p% pruned parameters
15:
if
limelle Sy (or Own 0) then Ing
16:
Stopping training and obtain (M) return ma @ (M) or Om
17:
4

---

Under review as a conference paper at ICLR 2023
3.2 EARLY STOPPING VERSUS REGULARIZATION
As the key operation of DLTH, regularization borrows learning capacity and perform information extrusion from the pruned weights from beginning to end. However, both early and excessive reg- ularization considerably impedes the model expressivity: 1) At the early stage, the model should focus more on fitting the data distribution without restricting its capacity. 2) Excessive regulariza- tion force makes the training process hard to control, c.g., slight excess may cause irreversible reactions or ill-conditioned weights. To this end, we figure out that the early stopping mechanism is simple yet effective alternative to regularization at the early training phase with a theoretical proof. Compared to regularization, early stopping not only has no negative effect on the early-stage behav- ior of fitting to the data distribution, but is also more controllable due to its non-parametric nature. In next part, we will mathematically analyze why regularization is equivalent to early stopping.
L2 Regularization. We first revisit the theory of regularization. Let J(w) denote an unregularized objective function, w² = argmin, J(w) is the weight vector when (ww achieves the minimum training error. Assuming the existence of second-order partial derivatives, we perform a quadratic approximation to the unregularized objective function in small neighborhood of w" as
j(w) = J [ww) +
2 (w ) H (w - (w-w)),
(1)
where w in the neighborhood of wt and H is the Hessian matrix LeCun et al. (9990) at point w".
Given w is a local optimal point, the first-order term (Jacobian matrix) in Eq. I has been eliminated
and H is positive semi-definite. When ((x) get minimum, we have v (w) H (w ==) = 0.
L2 regularization has long been the most popular form of regularization, also known as weight decay Cortes et al. 2012). which ensures the weight constrained within a small range by adding penalty
term (2w))= al 112 to [ww). Under this circumstance, the optimal point of w will be perturbed by the new force, causing the network to reach a new balance point ró as
orú + H (1é w²) = 0 =
vé (H + aT) Hw"
(2)
With the increase of a, we decompose H (H is real symmetric) into the diagonal matrix A and the standard orthonormal basis Q of the eigenvectors, H QAQ":
l = (QAQ" + al) QAQ -- Q(A -'"-"
(3)
As seen above, the weight decay is essentially scaling w" along the axis defined by the eigenvectors of H. This scaling effect has less effect the direction with larger eigenvalues and more the direction with smaller eigenvalues, indicating that the weights of various curvatures in the network behave differently when performing such regularization.
Early Stopping. A fundamental principle is recognized in deep neural networks - when the valida- tion set loss does not decrease in several training cycles, we can intuitively find that the model has undergone slight over-fitring We usually stop training as soon as the error on the validation set is higher than it was the last time it was checked, which is called early stopping. Compared with L2 regularization, early stopping is more inconspicuous since it barely affects the training process, the objective function, or any set of allowable parameter values.
Suppose ((w) reaches its minimum (see Eq. D. we obtain Vw.j(w) H (w- w²) = O. We study the process of parameter updates during training, assuming we optimize 7 steps with the learning rate e, by analyzing the gradient descent on I to approximately study the gradient descent on J. The parameters of the 7-th step can be derived from the (7 - 1)-th step as follows:
w!() (r-) eH (w(r-1) w²)
( w² (I ([-
( (w
- w²) (4)
Taking H = QAQT into the above equation, we can easily obtain:
QT (wlr) - w²) = (I - =A) (w(r-1)- w²) Q" (w( - -w" = - (A) Q (vo(t-2) - ww)
QF = =(1 -II- -eA)² (1
[] -I - +A) ( n²
Q" (w(1) - -w*)= = (I - -cA (w(()) - -w*)
(5)
5

---

Under review as a conference paper at ICLR 2023
For simplicity, we first prove the equivalence under w 0. As result. Eq. can be simplified into Q² w((r) = [T (T Q"=".. For I/2 regularization (see Eq. 3 A can be written as = diag(N1, A1,..., : Xn), then we can get an expression similar to the early stopping:
A1 \1 +
Q' = (A + aa) AQ"'" wt diag(
Xn QQ" 2t==== II al) a'
(6)
Now are able to link the formula of L2 regularization (Eq. 6 with the early stopping (Eq. 5
Qº v [1 - (A + al) Q w'
Q² w(( [I (I -I- - Q"" "
L2 Regularization
early stopping
As shown in Eq. 7 the early stopping is equivalent to L2 regularization if (A +al) -aaalaaaaa- (I- satisfies. In this case, we further take the logarithm form on both sides and derive:
Tlog (1 - eA) log (1 A/a)
(8)
A (i.e., a R3 4)) L2 regularization equivalent to the early stopping mechanism. More generally, when (o))) o O, we can draw similar conclusions (The above two proofs will be shown in detail in Appendix A and eT is the product of the learning rate and the number of steps, which can be intuitively regarded as the capacity of the network. For some changing learning rates, eT can be viewed as the distance that the optimization curve moves in the nigh-dimensional space.
When TeA Ri a
Why Early Stopping is Preferable? After demonstrating their equivalence, we articulate the major advantages of our early stopping strategy against regularization. In the case of early termination, this actually means in the direction of the large curvature parameters than smaller curvature direction ear- lier to learn, at the same time can terminate automatically determine the result of the regularization (only need to change the observation test loss decline times), and the weight decay require different parameter values of weights training experiment. We believe that excessive reliance on regulariza- tion at the early stage of training is not conducive to the network moving the training distribution. At the early stage training, we only use the carly stopping strategy, which can not only learn the data distribution better, but also will not disrupt the dynamic learning process of the network.
3.3 ITERATIVE PRUNING WITH INCREASED REGULARIZATION
We perform our pruning procedure iteratively after rewinding the weights to several epochs earlier, i.e., we repeatedly train, prune, and reset the network across M rounds (i.e., iterations). Each round eliminates p% of the weights that survive the pre- vious round, indicating that the remaining weights in the final round has been constantly verified for M times, which en- sures the reliability of the retained weights. In comparison to one-shot pruning, this iterative strategy empowers us to learn faster and achieve higher test accuracy with small-scale net- work (see our experiments in Sec.
Loglay
Tanlay
Tanear
Explaaw
E./2
E
Figure 2: Different nonlinear in- creased regularization methods. For simplicity, we draw these nonlinear growth curves in continuous forms.
During the pruning process, we perform increased regulariza- tion to tilt the network distribution towards the validation dis- tribution. Existing regularization-bant bassd pruning approaches (e.g., DLTH) mostly utilize linear increased regularization, where the coefficient a grows by constant for each round. As a result, the regularization will increase continuously at equal intervals to mine out the diversity of parameters Bai et al. (2022) and enable us to obtain a more expressive subnetwork (see the proof in Appendix C).
However, linear increased regularization may not be conducive to the search of optimal parameters, since a large magnitude of regularization may destroy the network dynamics at the late training phase. Instead, we expect that the regularization term can be controlled in relatively small range for a long time while continuously increasing. As opposed to the linear increased regularization, we argue that using nonlinear increased regularization term is more beneficial to the expressiveness of the winning tickets. As shown in Fig. 3.3 we propose several nonlinear regularization schemes: (1) Logarithnic-Lav LLogLaw): R
Inz Read In (g 1); (2) Exponential-Laan (ExpLaw):

---

Under review as a conference paper at ICLR 2023
R= -- R----.Reet Reest (e - 1); (3) Tangent-Law (TanLaw): R = Reeit tan 2
+
Resi is the ceiling value of the regularization term. yp is the current epoch and E is the total epochs. Different growth strategies can increase the degrees of freedom in such regularization methods, thereby enabling us to obtain more diverse optimal subnetworks.
EXPERIMENTS
In this section, we evaluate our pruning algorithms (UniLTH and UniDLTH) on several widely- used datasets. Our goal is to answer the following research questions. RQ1: How effective do algorithms find and transform winning lottery tickets (Sec 4.1)? RQ2: How does the subnetwork perform under different pruning rates (Sec 4.1)? RQ3: Is nonlinear increased regularization more powerful than its linear counterpart (Sec 4.2)? RQ4: How should we specify the detection the patience (people typically define patience, i.e. the number of epochs to wait before early stop no progress on the validation set.) Φ of early stopping and the rewind epochs & (Sec 4.3)? RQ5: Can our algorithms generalize to more backbones and larger-scale datasets (Appendix E
Datasets and Backbones. We first evaluate our algorithms on CIFARIO/100 datasets Krizhevsky et al. (2009) using VGG-19 SSionnynn & Zisserman (2014) or ResNer-50 He et al. (2016) as the backbone. To further verify its effectiveness on lcrge-scale datasets, we conduct experiments on ImageNet Simonyan & Zisserman (2014) using the backbone ResNet-50. Meanwhile, we consider the following methods for comparisons: 1) L1 is the T,1 regularization pruning based on a pre- trained network [i et al. (2016). 2) LTH/LTH--Iter is the lottery ticket hypothesis based on one- shot/iterative pruning strategies Frankle Carbin (2018). 3) EB is the Farly-Bird ticket (i.e., LT which emerge at the very early training phase) for LTH with one-shot pruning strategy Frannlle & Carbin (2018). 8). 4) DLTH/DLTH-Itr is the dual lottery ticket hypothesis based on one-shoV/teraaive pruning strategies Bai et al. (2022). Our experiments run on two NVIDIA Tesla V100 GPUs.
Due to the page limit, we describe the experimental settings in Appendix D
4.1 PRUNING ALGORITIMMS COMPARISON
In this part, we explore the efficacy of our proposed algorithms. For LTH, different iterative pruning values proposed for comparison. For EB, we follow its original paper to set the early stopping point at 37 epochs (1/8 of the total epochs). For DLTH, we set weight decay starting from 0, and ceiling bound is 2e-3. Here we just adopt linear growth regularization for comparison and nonlinear regularization will be discussed in Sec. 4.2. In Tab. I standard deviation of threc-run experiments and we have placed notations in Appendix D
we report the mean top-1 accuracy with its
Performance Comparison (RQ1). From Tab. we have the following observations: (1) In most cases, our algorithms outperform other approaches on the four datasets in terms of both accuracy and standard deviation; (2) Using our training paradiam on the training process, the parameters of a large model (e.g., ResNet50, MobileNet) are heavily slashed (even by factor of 10) while obtaining an excellent subnetwork under the same sparse condition (e.g.. 03.09% and 73.88% 338%%7...6. 73.68% of Vgg-19+CIFARI 10 and ResNet-50+CIIAN 100 settings); (3) Iterative pruning can help us get better lottery tickets. As seen in Tab. [ UniLTH (Iter- 10) mostly outperforms UniLTH (one-shot) among various backbones counterpart with datasets experimental settings. Similarly, we can draw the same conclusions in UniDLTH algorithm. These results experimentally demonstrate that iterative pruning allows our model to gain benefits to get better sub-networks. algorithm can considerably improve the inference speed without significant performance drop.
(4) Similar to standard LTH and DLTH,
Results of Different Pruning Rate (RQ2). Iterative pruning, early stopping with rewind, and grad- ually increased regularization three key components of our algorithms. To answer RQ2,
Proning tille
Porring
Prusiza ratie:70%
70%
Renand poies
poice
Revind point
Revind point
70.
60-
50
E22E
DETH
DL1B
22
Unl
TH
UnDDTH
10
53
100
253
300
200
300
52
100
350
200
25g
302
Epoch
100
150
Epoch
Figure 3: ResNet-50 on CIFAR-10 using 50%/70% sparsity with UniLTH/UniDLTH algorithms.
7

---

Under review as a conference paper at ICLR 2023
Table 1:
Performance comparison of V9g-19/ResNet50 on CIFARIOICIFARION datasets using 30%, 50%, 70% and 90% sparsity ratios. Except for the L1 row, the highest/second highest per- formances are emphasized with red/blue fonts
Vgg19+CIFARII; Baseline accurucy; 92.19%
Vgg19+CIEARR00; Baseline accuracy: 56.33%
Pruning aatoo(%//Sppdii
30%/
50%6/2.0x
70%//333xx
90%/10.0x
40%
50%/220x
70%/3.33x
90%/10.xx
Lr(pee-tranned model)
90.9910.09
89.9210.17
87.7420.26
84.27:00.35
56.2140.21
54.3910.37
50.47=1.233
49.77=1.47
LTHoone-shot
90,28+0.09
93.01+0.21
92.37+0.39
92.08-0.54
56,3440.22
5637+0032
56.09-0.43
55.98-0.56
LTH(her-10)
95.2940.16
93.15+0.38
92.77+0.23
92,34+0.25
56.51+0177
56,33+0.23
56332:0.27
56 56.38-0.28
EB
92.89+0.14
91,44+0.27
90.41+0.26
90:33+0:477
56.32+0.11
55.89+0.29
55.77=0.33
55.43-0.42
DLTH(one-shot)
92.9410.18
92.8710.44
92.1610.51
92.07.00449
56.2310.19
56.21:0.211
55.87-00388
55.71=0.53
DLTH((eer-10)
92.95:0.12
92.7811.222
92.8411.311
92.40.0.37
56.3710.13
56.4320.114
56.2990.25
06.04
Unil/Honn--shot
93.34+0.06
93.21+0.38
93.08+0.25
92. 08.88-0028
56.4740.09
56.42+0022
55 98.98-00.33
56.01-0.34
Unie/H(tte-10)
93:40+0.44
93,45+0.27
93.28+0018
93.09-0.14
56.50+0.08
56,73+0.15
5627=0.18
56.44-0.24
UniDLTHoone-suuu
93.27.0.13
93.2011..26
92.8910.34
92.72.00.35
56.4810.14
56.4310.31
56.2120.35
05.78
UnDDLHH(ker-00
93.52+0.11
93.48+0.17
93.19 19:0088
92.96-0.21
56,77+0.13
56.68+00.24
5647=0027
56.41-0:3
ResNeE00-CIAAIO: Baseline accuracy; 93.45%
ResNe500-IFAAIOO: Baseline accuracy: 73.87%
Pruning ratio SaSpechh
30%/1.43x
50%//2.x
70%/3.33x
90%/10.0
50%/1.43x
50%/22.x
70%/3.33x
90%/100xx
(pre-traided model)
93.28:0.17
92.17:0.24
90.3420.31
87.25:0:42
72.9810.16
70.2620.33
69.30=0.47
68.43=0.42
L.THone-sho)
92.68+0.17
92.78+0.36
92.38+00.29
92.29-+0.43
72.38+0.14
72.66+00.27
72.45-0.33
02.37-0.36
LTH(lie-10)
93.5622..15
93.3810.21
93.40=0.25
93.21:00.77
73.27:0.09
73.66:20.11
73.42=0.23
73.49=0.28
92.9+-0.1
91.56++0.38
89.73+-0.32
89.69-00.35
73.1340.18
72.88+0.24
02.04-0.35
72.16-0.63
DLTH(one-hhot)
93.42::0.25
93.37210.36
93.1720.43
92.87220:46
72.88:0.17
72.67110229
72.8420.11
72.71=0:47
DLTH(Bcr-10)
93.5720.11
93.6910.23
93.50:0.155
93.29:0028
73.6710.04
73.23=0.14
73.41=0.25
73.22=0.30
Unil/HHone-shot)
93.63+40.3
93.58+0.22
93.26++0.38
93.18-40.36
73.7740.18
73.60+0.33
73.19-0.43
73.41-0.52
UniLTH((ter-00)
93.71+0.27
03.6610.15
93.5820.19
93.69:20.22
73.9120.07
73.89:0.20
73.75=0022
73.68=0.18
UnDDTH(onesssee))
928740038
93.46+0.29
93.5240034
93.28-00.40
73.1040.19
73.12+0.21
72.89-0.38
73.11=0.37
UnDDLTH(her-10)
94.15+0.21
93.58+0.20
93.62400117
93.48-00117
74.05+0.12
73.98+0.14
73.60-0.12
73.88-0.26
control the number of pruning iterations v to 10, the monitoring threshold & (i.e., patience) for the descent of validation loss be 2. and the rewind epoch be 2 to keep these variables consistent. As shown in Fig. 3 we validate our methods under 50%/70% pruning ratio on combination settings of ResNet-50 and CIFAR-10. can be seen that our unified algorithms (i.e., UniLTT and UniDI.TH) clearly surpass the traditional algorithms on both pruning rates, which demonstrates their effective- ness. Additionally, we observe in Fig. 3 that at the carly stage of training, the abandonment of regularization can help the network to learn data distribution faster.
4.2 LINEAR VERSUS NONLINEAR INCREASED REGULARIZATION
In this paper, propose three forms of nonlinear increased regularization (including LogLaw, TanLaw and ExpLaw) with different regularization intensities at the early, middle and late stage, re- spectively. To answer RQ3, compare them with the linear counterpart and meanwhile investigate the effect of different regularization force ceiling Reed on the model performance.
In Tab. [ when using Recit with the same magnitude, ExpLaw significantly outperforms the other three strategies under all pruning ratios. As mentioned before, increased regularization stimu- lates the diversity of parameters. Dif- ferent growth strategies (linear or non- linear) further increase the degrees of freedom in such regularization meth- ods, i.e., enabling us to obtain more di- verse optimal subnetworks. For exam- ple. LogLaw keeps the network under intense regularization during the whole training process (see Fig.2), which might lead to ill-conditioned weights. Conversely, ExpLaw utilizes smaller penalty for a long train- ing time and simultaneously ensure that the weights magnitude discrepancy will be magnified.
ExeLow UnLLTH/UmDD
ResNerSoi (Priiig miloo5005)
90
25
60
pulicnce-1
patience-2
40
3
30
putience-d
UniDLTH
patience
80
10
200
250
300
Rell
Epoch
Figure 4: (Left) ExpLaw on ResNet-50 + CIFAR-10
50% sparsity experimental settings under different Reeit- (Right) Network performance under different validation loss monitor threshold ş (i.e., patience).
8

---

Under review as a conference paper at ICLR 2023
Table 2: Performance comparison of different increased regularization strategies on Vgg- 19/CIFAR10 and GesNet-50/CIFARI00 experimental settings. For fairesss, we set all Reeit to be consistent and keep le-3 (UniLTH/LTH) and le-2 (UniDLTH/DLLH),
Settings
Pruning ratio S USpeeddup
Datasets/ Balaneet/Backbness
Algorithass
Regularization
30%//1.43x
5018/2.xx
70%/3,33x
W/S/10.0x
Linear
93.40a0.16
93,4520,27
93.28%0.18
93.0920.1 14
UniLTH (Itr-10)
LogLaw
91.89=0. 12
91.77=0.17
90.79=0.31
91.04:0.43
TanLaw
92.4420.18
92.35=0.23
91.4820.25
91.21+0.39
ExpLaw
93,9840.24
93.79=0.26
93,48=0.27
93.2220.31
CIFAR 10V Vigg
Linear
93.52+0.11
93.48=0.17 17
93.19:0.28
92.9620.21
UnDLLTH (Iter-
LogLaw
91.5740.18
91.3240.44 08
91. 01.03=0.44
89.9920.3
TanLaw
92.48a0.32
92.36,0,39
91.4920.35
91.15
ExpLaw
93.80sa0.27
93.65+0.34
93.77=0.29
93.29:00.43
Linear
73.91=0.07
73.89+0.20
73.75=0.22
73.68+0.1
UniLTH (Iter-10)
Logl
71.1740.34
0..2
70.66=0.47
70.83+0.41
TanLaw
72.17=0.20
72.05=0.34
72.15=0.44
71662+0.53
Expl.aw
74.14=0.36
74.29=0.32
74.77=0.28
74.47=0030
CIFAR 100/ ResNet-50
Linear
74.05+0.12
73.98+0.14 14
73.60=0.12
73.88+0.26
UniDLTH (Iter-10)
LogLaw
71.12a0.15
71.29:1022
70.37=0.35
70.27400466
TanLaw
72.21=0.17
72.24+0.18
71.79=0.41
71.62+0.53
Expl.aw
74.1840..15
74.09:0.24
74.64=0.35
74.38+0.37
Moreover, the ill-conditioned weight issue can also be prevented by means of small penalty, which is more conducive to the search/ranssoormation of excellent subnetworks. We further explore the impacts of different Recit on the model results. As depicted in the left hand side of Fig. 4 find that UniL.TH requires less penalty force than UniDLTH. This may be attributed to the uncertainty of the subnetwork structure at the heginning of LTH and the need to slowly drop weights for reliable subnetwork. Excessive regularization force will destroy the learning process and complicate the search of winning tickets. In contrast, the discarded parameters in UniLTH are given at the begin- ning, which requires larger gradually increasing regularization force to extrude the information as much as possible, so Reeal large allowed.
4.3 EFFECTS OF PATIENCE FOR EARLY STOPPING AND REWIND VALUES
To answer RQ4, we first set patience 2 and mon- itor the top-1 accuracy under various rewind epochs ç.
Table 3: Top-1 accuracy of UniLTH vs. rewind epochs & and early-stopping pa- tience p. We adopt CIFAR (0+Vgg-19 for evaluation and set Φ 2.
After that, fix & as 2 and test network performance under different early stopping patience & under 30%, 50%, 70% and 90% pruning ratio (denoted by p). Re-
lated results are reported in Tab. 3 and Fig. 44 (Right), from which we have the following findings. The model exhibits strong expressiveness when & is controlled in a small regime such as or 2. In contrast, when enlarg- ing the rewind epochs to some extent, e.g., < 4,
Pruning ralio
{
1
{
5=2
2
{
=
3
4
[= 5
p = 30%
93.44
93.05
92.98
91.34
89.45
50%
93.48
93.87
92.68
92.01
90.35
70%
93.45
93.64
93.35
92.17
90.11
90%
93.65
93.66
93.44
92.45
90.24
the model performance will be degenerated by a considerable margin. Based on such observations, smaller are recommended to avoid crippling the model capacity. Meanwhile, find that different é does not have much effect on final model performance. Although the early train performance of the model was different in various patience, the model always reached the optimum nearly 200 epochs with similar accuracy (Seen Fig. (Right)).
5 CONCLUSION
In this work, we present unified paradigm for searching and transforming winning lottery tick- ets. At the early training phase, we replace regularization with early stopping to mitigate overfitting and ensure the stability training. When parameter distribution reaches certain point, we rewind weights and alternately adopt increased regularization and pruning to search/transform subnetworks to achieve network capacity and sparsity coint-optimization. In addition, to compensate for the ill- conditioned small-weight problem caused by linearly increased regularization, a variety of nonlinear regularization methods are proposed, among which ExpLaw method improves the network expres- sion power by limiting the network role to small penalty force for a long time. We have bench- marked our algorithms on extensive datasets and backbones, achieving comparable performance on ultra-lightweit subnetworks.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Sanjeev Arora, Simon Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. Fine-grained analysis of op- timization and generalization for overparameterized two-layer neural networks. In International Conference on Machine Learning, pp. 322-332. PMLR, 2019.
Yue Bai, Huan Wang. Zhiqiang Tao, Kumpeng Li, and Yun Fu. Dual lottery ticket hypothesis. arXiv preprint arXiv:2203.04222 2022.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Tianlong Chen, Yongduo Sui, Xuxi Chen, Aston Zhang, and Zhangyang Wang. A unified lottery ticket hypothesis for graph neural networks. In International Conference on Machine Learning, pp. 1695-1706. PMLR, 2021.
Tianlong Chen, Xuxi Chen, Xiaolong Ma, Yanzhi Wang, and Zhangyang Wang. Coarsening the granularity: Towards structurally sparse lottery tickets. arXiv preprint vrXiv:2202.04736, 2022.
Xiaohan Chen, Yu Cheng, Shuohang Wang, Zhe Gan, Zhangyang Wang, and Jingjing Liu. Earlybert: Efficient bert training via early-bird lottery tickets. arXiv preprint arXiv:2101.00063 2020.
Yu Cheng, Duo Wang, Pan Zhou, and Tao Zhang. A survey of model compression and acceleration for deep neural networks. arXiv preprint urXiv: 2017.
Corinna Cortes, Mehryar Mohri, and Afshin Rostamizadeh. L2 regularization for learning kernels. arXiv preprint arXiv: /205. 2653, 2012.
Lei Deng, Guoqi Li, Song Han, Luping Shi, and Yuan Xie. Model compression and hardware acceleration for neural networks: A comprehensive survey. Proceedings of the IEEE, 108(4): 485-532, 2020.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 1110.8800000 2018.
Shaojin Ding, Tianlong Chen, and Zhangyang Wang. Audio lottery: Speech recognition made ultra-lightweight, noise-robust, and transferable. In International Conference on Learning Repre- sentations, 2021.
Xiaohan Ding, Guiguang Ding, Jungong Han, and Sheng Tang. Auto-balanced filter pruning for efficient convolutional neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Xiaohan Ding, Guiguang Ding, Yuchen Guo, Jungong Han, and Chenggang Yan. Approximated acle filter pruning for destructive enn width optimization. In International Conference on Machine Learning. pp. 1607-1616. PMLR, 2019.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv:1803.03333 2018.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M Roy, and Michael Carbin. Stabilizing the lottery ticket hypothesis. arXiv preprint arXiv: 1903.0161 2019.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear mode con- nectivity and the lottery ticket hypothesis. In International Conference on Machine Learning, pp. 2269. PMLR, 2020.
Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity in deep neural networks. arXiv preprint arXiv: arXiv:1902.09577 2019.
Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint 150.00 2015.
10

---

Under review as a conference paper at ICLR 2023
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural net- works. In Proceedings of the IEEE international conference on computer vision, pp. 1389-1397, 2017.
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint (eXxv:/704.048000 2017.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Vadim Lebedev and Victor Lempitsky. Fast convnets using group-wiss wise brain damage. Proceed- ings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 2554-2564, 2016.
Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. Advances in neural information processing systems, 2, 1989.
Yann LeCun, John S Denker, and Sara Solla. Optimal brain damage. In Advances in neural information processing systems, pp. 598-605, 1990.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, 521(7553):436 444, 2015.
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. arXiv preprint 1rXiv:1608.08710 2016.
Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang. Shoumeng Yan, and Changshui Zhang. Learn- ing efficient convolutional networks through network slimming. Proceedings of the IEEE international conference computer vision, pp. 2736-2744, 2017.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv: 03983, 2016.
Christos Louizos, Max Welling, and Diederik P Kingma. Learning sparse neural networks through 1.0 regularization. arXiv preprint arXiv:
2017.
Xiaolong Ma. Geng Yuan, Xuan Shen, Tianlong Chen, Xuxi Chen, Xiaohan Chen, Ning Liu, Ming- hai Qin, Sijia Liu, Zhangyang Wang, et al. Sanity checks for lottery tickets: Does your winning ticket really win the jackpot? Advances in Neural Information Processing Systems, 34:12749- 12760, 2021.
Jaron Maene, Mingxiao Li, and Maric-Francine Moens. Towards understanding iterative magnitude pruning: Why lottery tickets win. arXiv preprint arXiv:2106.06955 2021.
Eran Malach, Gilad Yehudai, Shai Shalev-Schwarte and Ohad Shamir. Proving the lottery ticket hypothesis: Pruning is all you need. In International Conference on Machine Learning, pp. 6682- 6691. PMLR, 2020.
Ari Morcos, Haonan Yu, Michela Paganini, and Yuandong Tian. One ticket to win them all: general- izing lottery ticket initializations across datasets and optimizers. Advances in neural information processing systems, 32, 2019.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409. 1556, 2014.
Mingxing Tan and Quoc Lc. Efficientnet: Rethinking model scaling for convolutional neural net- works. In International conference on machine learning, pp. 6105-6114, 14. PMLR, 2019.
Chaoqi Wang, Roger Grosse, Sanja Fidler, and Guodong Zhang. Eigendamage: Structured pruning in the kronecker-face factored eigenbasis. International Conference on Machine Learning, pp- 6566-6575. PMLR, 2019a.
11

---

Under review as a conference paper at ICLR 2023
Huan Wang, Qiming Zhang, Yuehai Wang, Lu Yu, and Haoji Hu. Structured pruming for efficient convnets via incremental regularization. In 2019 International Joint Conference on Neural Net- works (IJCNN), pp. 1-8. IEEE, 2019b.
Huan Wang, Can Qin, Yulun Zhang, and Yun Fu. Neural pruning via growing regularization. arXiv preprint arXiv:2012.09243, 2020.
Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured sparsity in deep neural networks. Advances neural information processing systems, 29, 2016.
Jianbo Ye, Xin Lu, Zhe Lin, and James Z Wang. Rethinking the -ma informative assumption in channel pruning convolution layers. arXiv preprint arXiv: 1802. 0224 2018.
Haoran You, Chaojian Li, Pengfei Xu, Yonggan Fu, Yue Wang. Xiaohan Chen, Richard G Baraniuk, Zhangyang Wang, and Yingyan Lin. Drawing early-bird tickets: Towards more efficient training of deep networks. arXiv preprint
11957 2019.
Haoran You, Zhihan Lu, Zijian Zhou, and Yingyan Lin. Gebt: Drawing early-bird tickets in graph convolutional network training. arXiv preprint arXiv:2 007.40 2021.
Sergey Zagoruyko and Nikos Komodakis.
Wide
residual networks. arXiv preprint
arXiv: 1605.07600 2016.
Hattie Zhou, Janice Lan, Rosanne Liu, and Jason Yosinski. Deconstructing lottery tickets: Zeros, signs, and the supermask. arXiv preprint arXiv:1905.0106,, 2019.
A PROOF OF EQ. 8
In Eq. 7 the early stopping is equivalent to L2 regularization when (A + +a))=++ = -A- We take the logarithm on both sides as:
logi(A + al)"'aa a] log (1 eA)T
log(diagl + A +
1] T log(! - -ed) (9)
Substituting 3,ta (1+ )
into Eq. 9 we can obtain:
log 1 A
rlog - -
(10)
According the Taylor's Expansion, when approaches zero, log(1 x) can be approximated as r. Likewise, when the eigenvalues in A approach zeros, it can be seen that - log (7 + A/a) x A/a and rlog (I eA) x TEA. Therefore, the equivalence holds if a
B PROOF OF L2 REGULARIZATION IS EQUIVALENT TO EARLY STOPPING WHEN (0) +
When (0) # 0, we have:
QT (0) + +11 (1 (1- =T)
(1I)
This equation can also be written as:
@uu((? (T -A)
(!) w + I -T- @Tu= - 1- (T =AA² )
QTuu-
(12)
In practice (") usually much smaller than w*. which means that approaches (i.e., 1 (0)
( a
T). Compare Eq. 12 with Eq. the equivalence still holds if a
12

---

Under review as a conference paper at ICLR 2023
C PROOF OF INCREASED REGULARIZATION
Increased Regularization. As the formula vù = (H + - shown, after the regularization force increases (increase da on the original basis), the parameters in the network will exercise to new position. When the network converges again, it still exists vi = (H + da1)- - Hw" Due to (H + óal) is very complicated parameters in the network after the regularization force increases, we simply explore two H forms of H forms (we investigate two simplify cases help move forward) Wang et al. (2020).
difficult to solve, to better analyze the changes of the
(1) is diagonal matrix, which is a simplest form for Hessian information LeCun et al. 1989). We assume H = diag(h, 1, ... diag n), then:
h1,1
nn.n ha.n + 5a
v (H + 6al) HHw diag
w=
(13)
h1,1 + òa
We can find that for i-th wi w", s h difficult to find larger curvature corresponding to the larger hessian element. The closer the ratio of dy is to 1, the less the weight moves.
E (0, since h4s > and da > 0. It is not
wy 5o
w w::
h1,1 h2
h1.2 h22
(2) H is not diagonal. Here we analysis 2d matrix, w*
and H
then:
1 + h1,10a h2 w + sah12w?
(14)
riby
H
da7
122,2 + h222 10 m
) + Sah1,w
Due to da is small, we can obtain:
1
h2,2 + h1,1da - h w
uti2
~
H dall h2, h2.2 + 22220a - 1 W
(15)
is not difficult to find that at h1,1 > 12,2, we can still get sn To summarize, due to different local second-order partial derivative structures, different weight sponses are different response to increased regularization forces. Larger curvature results in the weights being relatively less moved towards the original points. The magnitude of the difference among the weights will increase as regularization grow.
w2/ww/ similar conclusion.
D EXPERIMENTAL SETTINGS AND NOTATIONS
The experiments are conducted on two NVIDIA Tesla V100 (16GB GPU) and all selected backbones follow the same experimental settings for faimess. Specifically, experiments on CI- FARIO/CIFARI00 are optimized by Stochastic Gradient Descent (SGD), and we control the learn- ing rate to 0.1 with 0.9 momentum using batch size 128. We use the Cosine Annealing Warm Restarts Loshchiilv & Hutter (2 scheduler as our learning rate scheduler and set the maximum number iterations T Mmaz~ 200. Those models for ImageNet classification are optimized by SGD and 1r=0.1 with 0.9 momentum and we follow the same setting in learning rate scheduler.
Table 4: The notations commonly reported in this work placed here.
Notation
Definition
X
Input data
e
Network parameters
me
Mask matrix
p%
Pruning rate
Pruning times
Validation patience
E
Rewind epochs
Roeil
The ceiling regularization term
13

---

Under review as a conference paper at ICLR 2023
E EXPERIMENTS ON MORE BACKBONES AND LARGE-SCALE DATASETS
To answer RQ5, we here evaluate MobileNets (v1) Howard et al. (2017) and EffficienNNetB(Tan & Le (019) two backbones on CIFAR-IO datasets, and add additional ImageNet dataset to verify the performance of our proposed unified lottery ticket search/transform algorithm. The results are given the table below.
Table 5: Performance comparasion of different pruning algorithm counterpart with MobileNets(v1) and EfficientNetBO backbones on CIFAR 10 dataset using 30%, 50%, 70% and 90% sparsity ratios. For convenience, we highlight show the highest/second highest performances with red/blue fonts.
MohileNets+IIFAR 10; Baseline accuracy: 91.11%
pruning ratio (% 3//ppeddu
30/1.43x
50/2.0x
70/3.33x
90/10.0x
L. (pre-tradnee model)
90.21=0.09
89.37=0.25
88.77=0.41
87.21=0.52
(TH((oe--hho))
90.23=0.38
90.162+0.47
89.91=0.52
88.73+0.60
LTH((ter-O))
91.37=0.25
90.67+0.29
91.11+0.32
89.1640.37
EB
91.4720.23
90.1640.34
89.9620.41
89.25+0.39
DLTH(one-shol)
90,98+0.34
90.16+0.33
89.65=0.48
89.31
DLTH(Iter-10)
91.27=0. 18
91.10=0.26
89.49=0.34
89 89.99+0.40
UniLTH(one-shot)
91.17=0.24
90.89+0.17
90.04+0.25
89.90=0.33
UniLTH((ter-10
92.14+0.07
91.18=0.15
91.12+0.19
90.24+0.26
OniDLTH(one-shot)
91.24+0.21
90.4440.12
90.01#0.32
89.52+0.27
UniDLTH((ter-OO)
92.07=0.11
91.20+0.14
90.10#0.21
89.97=0.25
EfficienNNetD+CCTAAA TO; Baseline accuracy: 90.64%
pruning ratio(%% (/speeddu
30/1.43x
50/2.0x
70/3.33x
90/10.0x
(prre-traince model)
10.72#0.19
89.88=0.21
89.02=0.33
87.16=0.59
LTH(one-shot)
90.29+0.32
90.16+004|
89.87+0.50
89.72+0.68
LTH((ter-10)
90.6620.25
90.21
89.98+0.37
89.9640.34
EB
90.27+0.37
89.98+0.46
89.45+0.62
88.9340.73
DLTHoone-hhot)
90.4420.31
90.27+0.45
90.10=0.39
89.65+0.47
DLTH(Iter-10)
90.79=0.22
90.66*00.38
90.09=0.25
90.11=0.35
UniLTH(one-shot)
90.37=0.27
90.23+0.33
90.16200.2
89.26=+037
UniLTH(Iter-00)
90.77=0.19
90.79a0.25
90.23+0.26
90.34+0.22
UniDLLHH(one-stttt
90.44:0.34
90.21+0.41
89.40#0.39
89.18+0.58
UniDLTH((ter-10)
91.26*0.37
90.1440.24
90.18+0.25
90.42+0.31
The results of comparative evaluation experiments on MobileNets(v!) and EfficientNetBO back- bones are summarized Table Our proposed pruning algorithm achieves almost the best perfor- mance under the same pruning ratios, which demonstrates the effectiveness of our unified winning lottery tickets search/transform algorithms in a more general scenario.
Table Performance on ImageNet dataset counterpart with different backbones and UniLTH prun- ing algorithm using 30%, 50%, 70% and 90% sparsity ratios. Acc@ @ 1: If the highest probability is the correct answer, it is considered correct. Acc@5: correct answer considered correct the top five probabilities contain the correct answer.
Different backbones UniLTH: ImageNet
Pruning ratio %e Vspeedup
MV/.43x
50/2.0x
70/3,33x
91V10.Ux
Metries
Acce9
Accers
Acc
Accis
Accol 1 Acc
Acc681 I Accor5
Vgg19 (buseline Acc@ kcc@1=71.9. Acc 5=90.18)
72.66
90.47
72.35
90.35
72.17
90.24
71.63
89.96
ResNet50 (buseline Acc@ 74.72, @cc@5m91.
74.98
92.03
74.86
92.19
73.96
92.07
73.88
91
MobileNets (haseline Acc 1= Acc
71.34
90.17
70.86
89.52
71.05
89.11
70.69
88.26
EfficientNet (baseline Acc@ www.662288..000-22..20
76.96
93.14
76.77
93.17
76.23
92.45
75.89
92.34
From Table 6 and Table 7 we can observe that when using the larger-scale dataset ImageNet, our unified lottery pruning algorithm is still effective, and when the model is pruned to one tenth of the original network, the backbones performance can still be comparable to the original backbones excellent performance. To summarize, We conclude our UniLTH/UniDLIT are also adaptable and robust for general backbones and large-scale datasets and even obtains better performances using the same parameter setting compared with LTH/DLTH, wheih can further search/transform the ultra- lightweight subnetworks.
14

---

Under review as a conference paper at ICLR 2023
Table 7: Performance on ImageNet dataset counterpart with different backbones and UniDLTH pruning algorithm using 30%, 50%, 70% and 90% sparsity ratios. Acc@ If the highest probability is the correct answer, it is considered correct. Aco@5: A correct answer is considered correct if the top five probabilities contain the correct answer.
Differeat backbomes + UniDLTH: ImageNet
Pruning ratio 96 Isppeuuu
30/1.43x
50/2.0x
70/3.33x
90/10.0x
Metries
Ace@:
Acc
Acc
AccG
Acc
Acc
Acc
Acc
Vgg19 (buscline Acci 1.99, S20.188
12.25
90.28
71.98
90.66
12.13
90.33
71.41
89.87
ResNet50 (baseline Acc@ 74,22, Acc05=9915) 87)
74.96
92.14
74.75
92.23
73.84
92.11
73.81
91.5
MobileNets (baseline Acc 7-770.77..c kco@5=88.43)
71.11
89.13
70.72
89.24
71.47
89.23
70.48
88.22
EfficientNet (baseline Acc@ 7c., 9992966
76.97
93.07
76.76
92.98
76.45
92.08
76.07
92.41
15


Reviews:
[{"review_id": "yOwiTk29td", "reviewer": "Reviewer_vpf2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The proposed UniLTH and UniDLTH pruning methods are well-motivated. The authors provide a new understanding of the regularization in the early stage. The paper is well-written and provides rigorous mathematical proof, which is a good contribution to the community. In the main paper and supplementary materials, extensive experiments with different sparsity levels on diverse tasks also verify the effectiveness of the proposed method.", "summary_of_the_review": "The proposed method is well-motivated. The authors provide a new understanding of regularization along with rigorous proof. Thus, I tend to vote accept.", "strength_and_weaknesses": "Strengths:\n\n1. This paper conducts experiments on diverse datasets and architectures. The experiments are thorough and convincing.\n2. The paper provides clear motivation and complete theoretical proof of the early stopping strategy.\n\nWeaknesses:\n\n1. As mentioned in Section 1, regularization-based pruning approaches (e.g., DLTH) typically perform one-shot pruning, which exacerbates the instability of sparse network training. Nevertheless, it seems no results to show how unstable the DLTH method is.\n\n2. This paper mainly compares the accuracy on several benchmarks and shows that the resultant winning tickets often outperform the full network. It would be interesting to see whether the winning ticket is also more robust than the full model against common corruptions (e.g., on CIFAR-10-C or CIFAR-100-C). \n\n3. There are some missing related work [A,B] that also focus on finding and exploiting winning/losing tickets. It would be better to discuss the differences between the proposed approach and these methods from the perspective of how to find subnetworks of interest.\n\n\n4. This paper mainly compares the proposed method with LTH and DLTH. It is interesting to see how large the improvement is compared with some popular one-shot pruning methods, such as ThiNet [C] and DCP [D].\n\n5. Apart from the improved pruning results, training cost is another important factor to evaluate a method. Will the proposed method introduce additional training cost compared with LTH and DLTH? If so, how much cost is introduced?\n\n[A] A winning hand: Compressing deep networks can improve out-of-distribution robustness, NeurIPS 2021.\n[B] Improving robustness by enhancing weak subnets, ECCV 2022.\n[C] Thinet: A filter level pruning method for deep neural network compression, ICCV 2017.\n[D] Discrimination-aware channel pruning for deep neural networks, NeurIPS 2018.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "s1w6neXyeC", "reviewer": "Reviewer_4vCa", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper revisits the LTH and DLTH by relaxing the weight regularization in the early training stage, which tries to adopt early stopping instead to realize a better pre-trained weight initialization. Two sparse network training methods, termed as UniLTH and UniDLTH, were designed and developed through a nonlinear increasing weight regularization method. Extensive experimental results were provided on two public datasets with several backbones. ", "summary_of_the_review": "Please refer to my comments in \"Strength And Weaknesses\" for more details. ", "strength_and_weaknesses": "Pros:\n- It is interesting and well-demonstrated to find a connection between the $\\ell_2$ weight regularization and early stopping by providing an approximate condition between the hyperparameters governing weight decay and SGD update.\n- The study on different nonlinear increasing regularization functions seems like a novel direction to the community. It will be interesting to learn nonlinear functions to dynamically control the whole regularization growing process –– which may avoid the early stopping yet unify the two-stage in the proposed method as an adaptive nonlinear regularization function.\n- The experiment was well-designed and conducted. Extensive experimental results and analyses were provided to validate the proposed methods in several aspects. \n\nCons:\n- It is not clear to me how the proposed method is called a unified approach/view over LTH and DLTH. I may miss something; yet I did not find a unified framework/formulation for these two complementary methods. It seems like the proposed methods simply adopt early stopping in these two methods, respectively. By using a linear regularization function, is UniLTH (UniDLTH) w/o early stopping the same to the LTH (UniLTH)? \n- While the connection between weight decay and early stopping is interesting, it is not well-motivated why using early stopping can benefit the training process, especially when they could be equivalent by tuning the learning rate and pre-training steps (given by Eq (8)). This is also demonstrated by the limited improvement of testing accuracy on CIFAR 10 and CIFAR100.\n- Some necessary experimental results are missing -- 1) the ablation of the proposed method on early stopping and nonlinear regularization functions; 2) the comparison results with other baselines on the ImagNet dataset; 3) the proposed method under 5 iterative pruning and a more aggressive pruning ratio. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "cFoo6m7PIPM", "reviewer": "Reviewer_3BTa", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a unified method to combine both iterative magnitude pruning and the Dual Lottery Ticket Hypothesis (Bai et al.) by combining early stopping and regularization in addition to traditional magnitude pruning. They also prove that early stopping is equivalent to $L_2$ regularization and also suggest some novel nonlinear regularization schemes. They conduct experiments on CIFAR-10 and Imagenet to back their claims.", "summary_of_the_review": "I have several concerns about the writing and the algorithm itself. If those are clarified, then I would reconsider but in its current form, I do not recommend acceptance. I am also not convinced of the value of combining these two algorithms into a single framework.", "strength_and_weaknesses": "**Strenghts:**\n- Their method combines two complementary views of finding lottery tickets: (i) IMP, the traditional mechanism of pruning a neural network to find subnetworks that can be trained to full accuracy, (ii) Dual LTH (Bai et al.), which \"transforms\" a random subnetwork so that it can be trained to full accuracy through regularization.\n- They conduct experiments on multiple datasets including CIFAR-10, CIFAR-100 and Imagenet.\n\n**Weaknesses:**\n1. I'm not sure it is important to combine the LTH and Dual LTH into a single framework. That said, UniLTH and UniDLTH don't seem to perform particularly well against their counterparts and the comparisons seem to be incomplete.\n    - Consider Table 1: Is baseline dense training? If so, how is it worse than sparse training? If this is true, then the hyperparameters of the different algorithms have not been tuned properly. Also, the accuracy seems to typically increases with sparsity, which again does not make sense.\n    - Since the DLTH is not a pruning at init method, you should be comparing against pruning after training schemes as well. In which case, you should compare against Renda et al. (Comparing Rewinding and Fine-tuning in Neural Network Pruning) which performs significantly better than IMP even with warmup.\n    - Table 2: LogLaw and TanLaw regularization is always worse than Linear and ExpLaw is only marginally better. This makes me wonder if hyperparameters were tuned for all of them. Moreover, it does not really show that nonlinear regularization is important which is the claim of the work.\n    - Figure 3: I think it would be beneficial to include other algorithms in this plot. This plot merely tells me that LTH and UniLTH are almost the same. Also, Figure 3 is not mentioned in the text, so I am not sure what inference I am supposed to make from it.\n2. The writing needs to be improved. Several paragraphs are convoluted and hard to read.\n    - Please use \\citep{} when including multiple citations which are not part of the sentence. For example, in the 3rd line of the introduction: \"... though over-parameterized deep neural networks achieve encouraging performance over widespread machine learning tasks Zagoruyko & Komodakis (2016); Arora et al. (2019); Devlin et al. (2018); Brown et al. (2020), they usually suffer notoriously ...\". The citations should be in parenthesis otherwise it is difficult to read.\n    - Page 2: analogy -> analogous\n    - LTH is used synonymously with IMP as an algorithm in this paper. This is incorrect. LTH is a claim regarding the existence of sparse subnetworks. Iterative Magnitude Pruning (IMP) is the algorithm you are referring to which is commonly used to find Lottery Tickets. There are several other algorithms: Refer SNIP (Tanaka et al.), GraSP (Wang et al.), Continuous Sparsification (Savarese et al.) to mention a few.\n    - In Section 2, I believe Malach et al. is cited incorrectly. Their result does not include any mention of training. It is purely an approximation result known as the Strong Lottery Ticket Hypothesis.\n    - Section 2 is also missing several references: Ramanujan et al., Renda et al., Su et al., Sreenivasan et al.\n    - Section 3.2 is confusing. You claim to prove that early stopping is equivalent to regularization but then say that early stopping is better. Isn't this a contradiction? I aalso do not agree with the claim that early stopping is \"non-parametric\". You still need to choose some thresholds on what constitutes convergence of validation loss as well as choice of the validation set.\n    - Section 4.1: paradiam -> paradigm.\n3. Questions about the algorithm:\n    - In Section 3, it is stated that training is performed until validation loss stops dropping. And then the parameters are rewound to $\\xi$ epochs earlier. I don't follow this step. Why train if you are going to rewind all of the weights? Are these trained weights used in some way before rewinding? This is not clear in the algorithm.\n    - What is $\\psi$ in Section 2? It is not defined.\n    -  In Algorithm 1, line 5: what does \"several epochs ago\" mean? Please be precise.\n    -  Is the nesting in Algorithm 1 correct? If *if* condition in line 15 is False, then the *while* loop in line 1 will also be false. Both these things do not need to exist. Please clarify/rewrite Algorithm 1. I'm not sure I follow it.\n4. Result about early stopping is equivalent to regularization:\n    - You assume that $J(w)$ reaches its minimum claiming that this is the condition for early stopping. However, this is the true objective, not validation loss. If you assume that these two are equivalent, then early stopping is essentially stopping at the minimizer of the true function, which is incorrect.\n    - Analyzing gradient descent on $\\hat{J}$ to approximately study gradient descent on $J$ needs to be justified more. Firstly, the approximation only holds in a small neighborhood and it is not clear that GD will stay in that neighborhood. Moreover, this is a quadratic function and it is well known that GD concerges exponentially fast for quadratics. This is surely not true for the function you are considering which might be highly non-convex.\n    - Typo: satisfies -> satisfied.\n    - Please rephrase the paragraph on \"Why Early Stopping is preferable\": It is quite convoluted.\n    - Appendix B: What does $\\frac{w^{0}}{w^*}$ mean? If they are both vectors, then I am not sure what dividing them even means. Are you conisdering norms? Furthermore, I don't agree that $w^{0}$ is usually much smaller than $w^*$. If you are training with regularization, the expectation is that the norm of $w$ will decrease as training proceeds.\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "WDOXoQu-i7A", "reviewer": "Reviewer_FGiK", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper extends the dual lottery ticket hypothesis (DLTH), which consists in randomly selecting a subnetwork and transforming it in a winning ticket through training with regularization. In this paper, the authors propose s slightly modified procedure, called UniDLTH, where they introduce a preliminary training phase without regularization. The authors also propose a novel procedure for the lottery tickets hypothesis where they introduce a gradually increased regularization term.", "summary_of_the_review": "I think that this paper presents some serious issues. In particular, the novelty and the presentation quality are below the level expected for ICLR.", "strength_and_weaknesses": "I have several concerns regarding this paper. First of all, I think that the technical contribution of the proposed method is limited: the increased regularization term was already presented in the DLTH, instead a pretraining phase is already present in the procedure of the lottery ticket hypothesis. I have also some doubts regarding the regularization term: it is well known that the l1 norm is more effective than the l2 norm in terms of sparsification, therefore it is not clear to me why the authors choose to use the l2 norm as regularization term. Another concern is related to the English usage: the paper is not well written, there are many grammar and syntax errors making the paper hard to follow.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject