Split LBI for Deep Learning: Structural Sparsity via Differential Inclusion Paths
Authors: Yanwei Fu
Chen Liu
Donghao Li
Xinwei Sun
Jinshan ZENG
Yuan Yao
Venue:  (2020)

Abstract: Over-parameterization is ubiquitous nowadays in training neural networks to benefit both optimization in seeking global optima and generalization in reducing prediction error. However, compressive networks are desired in many real world applications and direct training of small networks may be trapped in local optima. In this paper, instead of pruning or distilling over-parameterized models to compressive ones, we propose a new approach based on \emph{differential inclusions of inverse scale spaces}, that generates a family of models from simple to complex ones by coupling gradient descent and mirror descent to explore model structural sparsity. It has a simple discretization, called the Split Linearized Bregman Iteration (SplitLBI), whose global convergence analysis in deep learning is established that from any initializations, algorithmic iterations converge to a critical point of empirical risks. Experimental evidence shows that\ SplitLBI may achieve state-of-the-art performance in large scale training on ImageNet-2012 dataset etc., while with \emph{early stopping} it unveils effective subnet architecture with comparable test accuracies to dense models after retraining instead of pruning well-trained ones.

Under review as a conference paper at ICLR 2020
SPLIT LBI FOR DEEP LEARNING: STRUCTURAL SPAR- SITY VIA DIFFERENTIAL INCLUSION PATHS
Anonymous authors Paper under double-blind review
ABSTRACT
Over-pan parameterization is ubiquitous nowadays in training neural networks to ben- efit both optimization in seeking global optima and generalization in reducing prediction error. However, compressive networks are desired in many real world applications and direct training of small networks may be trapped in local op- tima. In this paper, instead of pruning distilling over-parameterrize models compressive ones, we propose a new approach based on differential inclusions of inverse scale spaces, that generates family of models from simple complex ones by coupling gradient descent and mirror descent to explore model structural sparsity. It has simple discretization, called the Split Linearized Bregman Iteration (BplitLBI), whose global convergence analysis in deep learning is established that from any initializations, algorithmic iterations converge ical risks. Experimental evidence shows that SplitLBI may achieve comparable and even better performance than other training algorithms on ResNet- 18 in large scale training on maa dataset etc., while with early stopping it unveils effective subnet architecture with comparable test accuracy dense models after retraining instead of pruning well-trained ones.
critical point of empir-
INTRODUCTION
The expressive power of deep neural networks comes from the millions of parameters, which are optimized by Stochastic Gradient Descent (SGD) (Bottou, 2010) and variants like Adam (Kingma Ba, 2015). Remarkably, model prrnettrttttiiiooonnn helps both optimization and generalization. For optimization, over- peee-parameterization may simplify the landscape of empirical risks toward locating global optima efficiently by gradient descent method (Mei et al., 2018; 2019; Venturi et al., 2018; Allen-Zhu et al., 2018; Du et al., 2018). On the other hand, over-parameterizzaion does not necessarily result in bad generalization overfitting (Zhang et al., 2017), especially when some weight-size dependent complexities are controlled (Bartlett, 1997; Bartlett et al., 2017; Golowich et al., 2018; Neyshabur et al., 2019).
However, compressive networks are desired in many real world applications, e.g. robotics, self- driving cars, and augmented reality. Despite that & regularization has been applied to deep learning to enforce the sparsity on weights toward compact, memory efficient networks, it sacrifices some prediction performance (Collins & Kohli, 2014). This is because that the weights learned in neural networks are highly correlated, and é regularization on such weights violates the incoherence or irrepresentable conditions needed for sparse model selection (Donoho & Huo, 2001; Tropp, 2004; Zhao & Yu, 2006), leading to spurious selections with poor generalization. On the other hand, &2 regularization often utilized for correlated weights as some low tow-pass filtering, sometimes in the form of weight decay (Loshchilov & Hutter, 2019) or early stopping (Yao et al., 2007: Wei et al., 2017). Furthermore, group sparsity regularization (Yuan Lin, 2006) has also been applied neural networks, such finding optimal number of neuron groups (Alvarez & Salzmann, 2016) and exerting good data locality with structured sparsity (Wen et al., 2016; Yoon & Hwang. 2017).
Yet, without the aid of over-parameterization directly training a compressive model architecture may meet the obstacle of being trapped in local optima in contemporary experience. Alternatively, researchers in practice typically start from training big model using common task datasets like ImageNet, and then prune or distill such big models to small ones without sacrificing too much of the performance (Jaderberg al., 2014; Han et al., 2015; Zhu al., 2017: 7; Zhou al., 2017; Zhang et al.,

---

Under review as a conference paper at ICLR 2020
2016; Li et al., 2017; Abbasi-Asl & Yu, 2017; Yang et al., 2018; Arora al., 2018). In particular, a recent study (Frankle & Carbin, 2019) created the lottery ticket hypothesis based on empirical observations: "dense, randomly- -nniiailiiee,, feed-forward networks contain subnetworks (winning tickets) that - when trained in isolation - reach test accuracy comparable to the original network in a similar number of iterations". How effectively reduce the key to compressive deep learning. Yet. Liu et al. (2019) raised question, is it necessary to fully train a dense, over-parameterize model before finding important structural sparsity?
eereparameterized model thus becomes
In this paper, we provide a novel answer by exploiting a dynamic approach to deep learning with structural sparsity. We are able to establish a family of neural networks, from simple complex, by following regularization paths as solutions of differential inclusions of inverse scale spaces. Our key idea to design some dynamics that simultaneously exploit werrparameterrzed models and structural sparsity. To achieve this goal, the original network parameters are lifted to coupled pair, with one weight set W of parameters following the standard gradient descend to explore the wver-parameterized model space, while the other set of parameters learning structure sparsity in an inverse scale space, i.e., structural sparsity set r. The large-scale important parameters are learned at a fast speed while the small unimportant ones are learned at slow speed. The two sets of parameters are coupled in an E2 regularization. The dynamics enjoys a simple discretization, i.e. the Split L Linearized Bregman Iteration (SplitL.BI) with provable global convergence guarantee shown in this paper. Here, SplitIBI is natural extension of SGD) with structural sparsity exploration: SplitLBB reduces the standard gradient descent method when the coupling regularization is weak, while leads to a sparse mirror descent when the coupling is strong.
Critically, SplitLBI enjoys a nice property that important subnet architecture can be rapidly learned via the structural sparsity parameter I' following the iterative regularization path, without fully training a dense network first. Particularly, the support set of structural sparsity parameter I' learned the early stage of this inverse scale space discloses important sparse subnet architectures. Such architectures can be fine-tuned or retrained to achieve comparable test accuracy as the dense, rree-parameterieed networks. As a result, the structural sparsity parameter N may enable us to rapidly find "winning tickets" in early training epochs for the "lottery" of identifying successful subnetworks that bear comparable test accuracy to the dense ones. This point is empirically validated in our experiments.
Historically, the Linearized Bregman Iteration (LBI) was firstly proposed in applied mathematics as iterative regularization paths for image reconstruction and compressed sensing (Osher et al., 2005; Yin et al., 2008), later applied to logistic regression (Shi et al., 2013). The convergence analysis was given for convex problems (Yin et al., 2008; Cai et al., 2009). yet remaining open for non-convex problems met in deep learning. Osher et al. () established statistical model selection consistency for high dimensional linear regression under the same irrepresentable condition as Lasso, later extended to generalized linear models (Huang & Yao. 2018). To relax such conditions, SplitLBI was proposed by Huang et al. (2016) to learn structural sparsity in linear models under weaker conditions than generalized Lasso, that was successfully applied in medical image analysis (Sun et al., 2017) and computer vision (Zhao et al., 2018). In this paper, it the first time that SplitLBI is exploited to train highly non-convex neural networks with structural sparsity, together with global convergence analysis based on the
framework Eojasiewicz (1963).
Contributions. (1) SBI, as an extension of SGD, is applied to deep learning by exploring both werr-paraeetrrized models and structural sparsity in the inverse scale space. (2) Global convergence of BplitLBI in such nonconvex optimization is established based on the Kurdyka- Lojasiewicz framework, that the whole iterative sequence converges to critical point of the empirical loss function from arbitrary initializations. (3) Stochastic variants of SplitLBI demonstrate the comparable and even better performance than other training algorithms on ResNet- 18 in large seale training such ImageNet-2012. among other datasets, together with additional structural sparsity in successful models for interpretability. (4) Structural sparsity parameters in SplitI.BI provide important information about subnetwork architecture with comparable or even better accuracies than dense models before and after retraining - SplitLBI with early stopping can provide fast "winning tickets" without fully training dense, over-parameterized models.
2

---

Under review as a conference paper at ICLR 2020
Fandom
Weight
Figure 1: Visualization solution path and filter patterns third convolutional layer (i.e., conv.c5) of LetNet-S, trained on MNIST. The left figure shows the magnitude changes for each filter of the models trained by SplitLBI and SGD, where z-axis and y-axis indicate the training epochs, and filter magnitudes (f2-norm), respectively. The SplitI.BI path of filters selected in the support of I are drawn in blue color, while the red color curves represent the filters that are not important and outside support r. We visualize the corresponding learned filters by Erhan al. (2009) at 20 (blue), 40 (green), and 80 (black) epochs, which shown in the right figure with the corresponding color bounding boxes, i.e., blue, green, and black, respectively. It shows that our SplitLBI enjoys sparse selection of filters without sacrificing accuracy (see Table
2
METHODOLOGY
The supervised learning task learns mapping Φw X' + Y, from input space X' to output space ), with a parameter W such as weights in neural networks, by minimizing certain loss functions on training samples E. (W)) = E E(y,, Pw (xx)). For example, a neural network of I-layer is defined as (x) =
o.W" where W = (W*) Ti= is the nonlinear
activation function of the i-th layer.
Differential Inclusion of Inverse Scale Space. Consider the following dynamics,
W. (W,,))
(la)
V/. = (W,,))
(Ib)
Ve ON(()))
(1c)
where is a sub-gradient of (TT) 22 (T) + 2n ||] for some sparsity-enforced regularization
(T) = Xl2 (T') (A E R4) such Lasso group Lasso penalties for 92 (1'), k > is damping
parameter such that the solution path is continuous, and the augmented loss function is
1 _|W -Il. (W,r) E., (W) + 2v
with >> 0 controlling the gap admitted between W and I. Compared to the original loss function En, (W), the E (W, additionally adopt the variable splitting strategy, by lifting the original neural network parameter W to (W, I') with I' modeling the structural sparsity of W. For simplicity, we assumed is differentiable with respect to W here, otherwise the gradient in Eq. (la) is understood as subgradient and the equation becomes an inclusion.
The differential inclusion system (1), called Split Inverse Scale Space (SplitISS), can be understood as gradient descent flow of W; in the proximity of I, and mirror descent flow (Nemirovski & Yudin, 1983) of T2 associated with a sparsity enforcement penalty ②2. In mirror descent flow, gradient descent goes on the dual space consisting of sub-gradients Vi, driving the flow in sparse primal space of T,- For large enough v, it reduces to the gradient descent method for W.. Yet the solution path of I exhibits the following property in the separation of scales: starting at the zero, important parameters of large scale will be learned fast, popping up to be nonzeros early, while unimportant parameters of small scale will be learned slowly, appearing to be nonzeros late. In fact, taking (T) [[[]] and h - 2o for simplicity, V, as the subgradient of Ω, undergoes a gradient descent flow before reaching the unit box, which implies that T, 0 in this stage. The earlier component in V, reaches
3

---

Under review as a conference paper at ICLR 2020
the -unit box, the earlier a corresponding component in H2 becomes nonzero and rapidly evolves toward a critical point of L under gradient flow. On the other hand, the W, follows the gradient descent with a standard &2 fg-regularizatonn Therefore, W, closely follows dynamics of I2 whose important parameters are selected. Such a property is called as the inverse scale space in applied mathematies (Burger al., 2006) and recently was shown to achieve statistical model selection consistency in high dimensional linear regression (Osher et al., 2016) and general linear models (Huang & Yao, 2018), with reduction of bias as K increases. In this paper, we shall see that the inverse scale space property still holds empirically for the highly nonconvex neural network training via Eq. (1), For example, Fig. shows a LeNet trained on MNIST by the discretized dynamics, where important sparse filters are selected in early epochs while the popular SGD returns dense filters.
Compared with directly enforcing a penalty function such as l or l2 regularization
min (W) == E., (W) +22 (W), X R W
SplitSS avoids the parameter correlation problem in over- over-parameterized models. In fact, a necessary and sufficient condition for Lasso or tt-yee sparse model selection is the incoherence or irrepre- sentable conditions (Tropp (2004); Zhao Yu (2006)) that are violated for highly correlated weight parameters, leading to spurious discoveries. In contrast, Huang et al. (2018) showed that equipped with such a variable splitting where enjoys an orthogonal design where the restricted Hessian of the augmented loss on I is orthogonal, the SplitSS can achieve model selection consistency under weaker conditions than generalized Lasso, relaxing the incoherence or irrepresentable conditions when parameters are highly correlated. For weight parameter W. instead of directly being imposed with & fs-sparyyy, it adopts fz-regularization in the proximity of the sparse path of I that admits simultaneously exploring highly correlated parameters in over-parameterized models and sparsity regularization.
Split Linearized Bregman Iterations. SplitIS admits an extremely simple discrete approximation, using the Euler forward discretization of dynamics (I):
W WR kaa Vw E (W,. ls).
(4a)
Vk+1 = V22 - -ak . Vr (WW,W,
(4b)
F'+11 = K- Proxaa (V++1),
(4c)
where Vo r, = 0, Wo can be small random numbers such as Gaussian distribution in neural networks, for some complex networks it can be initialized as common setting. The proximal map in Eq. (4c) that controls the sparsity of is given by
11 Sx (
Proxo, (V) arg min
We shall call such an iterative procedure as Split Linearized Bregman Iteration (SplitLBI), that was firstly coined in Huang al. (2016) an iterative regularization path for sparse modeling in high dimensional statistics. In the application to neural networks, the loss becomes highly non-convex, the SplitLBI returns sequence of sparse models from simple to complex ones whose global convergence condition to be shown below, while solving Eq. (3) at various levels of X might not be tractable except for wver-parameterize models.
The enfrrstt-enforemmentt penalty used in convolutional neural networks can be chosen as follows. Our sparsity framework aims at regularizing the groups of weight parameters using group Lasso penalty
[2 (T9)² and [ is the number of weights in 1 Thus Eq. (4c) has a closed form solution r9 = max (0, 1/1 V for the g-th filter. We treat convolutional and fully connected layers in different ways.
(Yuan & Lin, 2006), 22 (T) = Ey | |2, where ||T0|| = E
(1) For convolutional layer, 1 (cin, Cout, size) denote the convolutional filters where size denotes the kernel size and cin and Cout denote the numbers of input channels and output channels, respectively. When regard each group as each convolutional filter, g = Cout' otherwise for weight sparsity, g can be every element in the filter that reduces to the Lasso.
(2) For fully connected layer, I I'Can Cout) where Cin and Cout denote the numbers of inputs and outputs of the fully connected layer. Each group g corresponds to each element (i, (). and the group Lasso penalty degenerates the Lasso penalty.
4

---

Under review as a conference paper at ICLR 2020
3 GLOBAL CONVERGENCE OF SPLITLBI FOR NEURAL NETWORKS
We present a theorem that guarantees the global convergence of Sp i.e. from any intialization, the SplitLBI sequence converges to a critical point of L. Our treatment extends the block coordinate descent (BCD) studied in Zeng et al. (2019), with crucial difference being the mirror descent involved in SplitLBI. Instead of the splitting loss in BCD (Zeng et al., 2019), a new Lyapunov function is developed here to meet the Eojjssewwiii property Lojasiewicz (1963). Xue & Xin (2018) studied convergence of variable splitting method for single hidden layer networks with Gaussian inputs.
Let P == (W,T) Following Huang & Yao (2018), the SplitLBI algorithm in Eq. (4a-4c) can be rewritten as the following standard Linearized Bregman Iteration,
Pk+1 arg min (IP Pk. + (P.Pk))
(6)
where
(PP)= = -x(T)) | 111 (2)
1
W
2x
2x
2x
Pk E (P)),, and B% is the Bregman divergence associated with convex function v, defined by
B (P.Q) == (PP) (Q) - (q, Q), for some E (V)
(8)
Without loss of generality, consider À = 1 in the sequel. One can establish the global convergence of SplitIBI under the following assumptions.
Ei = Elyi, Dw (I:)) is continuous differentiable and Vnn is Lipschitz continuous with positive constant Tip; (b)Ln(W) (W) has bounded level sets; (c) E. (W) is lower bounded (without loss of generality, we assume that the lower bound is 0); (d) S2 is a proper lower semi-continuous convex function and has locally bounded subgradients, that is, for every compact set S C R", there exists a constant C >> O such that for all I E S and all E (N(T), there holds |gg|| S C; and (e) the Lyapunov function
Assumption 1. Suppose that: (a) (W) =
F(P,G) == aL(W,r) (T, (T,
(9)
is a Banddla-Lojjsseewiccffree
any bounded set, where (1 ((T) -()))())--r--
I 00" (ğ), and O is the conjugate of 5t defined as
(g) = sup UER=
Remark 1. Assumption (a)(Cc are regular in the analysis of nonconvex algorithm (see, Attouch al. (2013) for instance), while Assumption (d) is also mild including all Lipschitz continuous convex function over compact set. Some typical examples satisfying Assumption I(d) are the fı norm, group norm, and every continuously differentiable penalties. By Eq. (9) and the definition of conjugate, the Lyapunov function F can be rewritten as follows.
F(W, gg)) aC(W, + Q(T) + +2" ((T)))))))))
(10)
Now are ready to present the main theorem.
Theorem 1. /Global Convergence of SplitLBI] Suppose that Assumption I holds. Let (Wk, I'x) be the sequence generated by SplitLBI (Eq. (4a-4c)) with a finite initialization. If
2
x<a==00 n(Lip+u
then (Wr, I'x) converges to a critical point of E defined in Eq. (2), and {Wk} converges to critical point of (W).
Applying the neural networks, typical examples are summarized in the following corollary.
5

---

Under review as a conference paper at ICLR 2020
Corollary 1. Let {Wk. k. 9k be a sequence generated by SLBI (16a-16c) for neural network training where (a) is any smooth definable loss function, such as the square loss (f?), exponential loss (e'), logistic loss log(1 + t and cross- loss; (b) 0; is any smooth definable activation, such as linear activation (t), sigmoid (1te-.), hyperbolic tangent (e and softplus (a log(1 + + for some c > 0) as a smooth approximation of ReLU; (c) is the group Lasso. Then the sequence } converges to a stationary point of (W) under the conditions of Theorem 1.
Proofs of Theorem and Corollary are given in Appendix A.
EXPERIMENTS WITH STOCHASTIC SPLITLBI
We begin with some stochastic variants of SplitLBI and implementations, followed by four groups of experiments demonstrating the utilities weight parameter W, and structural sparsity parameter I, in prediction, interpretability, and capturing effective sparse subnetworks.
Batch Split LBI. For neural network training with large datasets, stochastic approximation of the gradients in Split LBI over the mini-batch (X, Y bbttch, is adopted to update the parameter W,
V W Vw En (W) (X.Y)
(11)
BplitLBI with momentum (Mom). Inspired by the variants of SGD, the momentum term can be also incorporated the standard Split LBI that leads to the following updates of W by replacing Eq (4a) with,
U++1 = TU, +
(12a)
W41 W, K00044
(12b)
where r the momentum factor, empirically setting as 0.9 default. One immediate application of such stochastic algorithmss of SplitLBI is to "boost networks", f.e. growing network from the null to a complex one by sequentially applying our algorithm on subnets with increasing complexities.
SplitLBI with momentum and weight decay (Mom-Wd), The update formulation is,
TU, + Vw 2 T.) W4+1 = W, - K000++ - 3W,
U441
(13)
(14)
where 3 is set as 14-4.
Implementation. Various algorithms are evaluated over the various backbones LeNet (LeCun et al.. 2015), AlexNet (Krizhevsky et al.. 2012), VGG (Simonyan & Zisserman, 2014), and ResNet (He et al., 2016) etc., respectively. For MNIST and Cifar-10. 10, the default hyper-paramettrss of Split LBI are K 1, 10 and og is set as 0.1, decreased by 1/10 every 30 epochs. In ImageNet-2011 the Split LBI utilizes & = 1, v/ = 1000, and aş is initially set as 0. decays 1/10 every 30 epochs. We set À = 1 in Eq. (5) by default, unless otherwise specified. On MNIST and Cifar-10, the batch size is set as 128; and for all methods, the batch size of ImageNet 2012 is 256. The standard data augmentation implemented in pytorch is applied to Cifar-10 10 and ImageNet2012 datasets, as He et al. (2016). The weights of all models are initialized as He et al. (2015). In the following experiments, we define sparsity percentage of non-zero parameters, i.e. the number of non-zero weights dividing the total number of weights in consideration, that equals to one minus the pruning rate of the network. We also have the reprodueible source codes 1.
4.1 IMAGE CLASSIFICATION
In BpIitLBI, the weight parameter W, explores over-parameterizee models that can achieve the state-of-the-art performance large scale training such as ImageNet-2012 classification.
Experimental Design. We compare different variants of SGD and Adam in the experiments. By default, the learning rate competitors set as 0.1 for SGD and its variant and 0.001 for Adam and its variants, and gradually decreased by 1/10 every 30 epochs. In particular, we have,
'ttpps: /atpas:://nnn/mm//// Sp1 pp LB____BL_oo code
5005--00-----0-- 4e60- b2es 4ded4e93db63/

---

Under review as a conference paper at ICLR 2020
Dataset
MNIST
Cifar-10
ImageNet-2012
Models
Variants
LeNet
ResNet-20
AlexNet
ResNet-11
Naive
98.87
86.46
+
60.76/79.18
4
98.52
67.60
+/-
SGD
Mom
99.16
89.44
55.14/78.09
66.9% 66.98/86,97
Mom-Wa
99.23
90.31
56.55/79.09
69.76/89.11
Nesterov
99.23
90.18
-/-
70.19/89.30
Naive
99.19
89.14
+
59.66/83.28
Adabound
99.15
87.89
+
Adam
Adagrad
99.02
88.17
+
Amsgrad
99.14
88.68
+
Radam
99.08
88.44
-/-
-/-
Naive
99.02
89.26
55.06/77.60
65. 655.2686.57
Sp
Mom
99.19
89.72
56.23/78.48
68.55/87.88
Mom- Wd
99.20
89.95
57.09/79.86
70.55/89.56
Table 1:
Top- Top-1/Top-5 accuracy(%) on ImageNet-2012 and test accuracy on MNIST/Cifar- 10. results from the official pytorch website. We use the official pytorch codes to run the competitors. All models are trained by 100 epochs. In this table, we run the experiment by ourselves except for SGD Mom- Nom-WWdon ImageNet which is reported in http:///ww...ooooo/aa/a/////oote
SGD: (1) Naive SGD: the standard SGD with batch input. (2) SGD with 4 penalty (Lasso). The norm applied penalize the weights of SGD by encouraging the sparsity of learned model, with the regularization parameter of the 4 penalty term being set as le (3) SGD with momentum (Mom): we utilize momentum 0.9 in SGD, (4) SGD with momentum and weight decay (Mom-Wd): we set the momentum 0.9 and the standard la weight decay with the coefficient weight 1e -4 (5) SGD with Nesterov (Nesterov): the SGD uses nesterov momentum 0.9.
Adam: (1) Naive Adam: it refers to the standard version of Adam. We report the results of several recent variants of Adam, including (2) Adabound, (3) Adagrad, (4) Amsgrad, and (5) Radam.
SplitLBI achieves the state-of-the-art performance on ImageNNet-2012, etc. Tab. shows the experimental results on ImageNet-2012,, Cifar-10, and MNIST of some classical networks LeNet, AlexNet and ResNet. Our SplitLBI variants may achieve comparable or even better performance than SGD variants in 100 epochs, indicating the efficacy learning dense, werrpparaeeterized models.
4.2 SPLITLBI LEARNS SPARSE FILTERS FOR IMPROVED INTERPRETATION
01-878
Input Image
Intiaiizaioon
086
Figure 2:
Visualization of the first convolutional layer filters of ResNet- 18 trained on ImageNet-
2012. Given the input image and initial weights visualized in the middle, filter response gradients at 20 (purple), 40 (green), and 60 (black) epochs are visualized by Springenberg et al. (2014). The "SLBI-I0" ("SLBI-1") the right figure refers to SplitLBI with K = and k = 1. respectively.
7

---

Under review as a conference paper at ICLR 2020
In SplitLBI, the structural sparsity parameter r explores important sub-network architectures that contributes significantly to the loss or error reduction in early training stages. Through the f2-coupling, structural sparsity parameter I, may guide the weight parameter to explore those sparse models in favour of improved interpretabiity. For example, Fig. visualizes some sparse filters learned by SplitI.BI of LeNet-5 trained MNIST (with comparison with dense filters learned by SGD. The activation pattern of such sparse filters favours high order global correlations between pixels of input images. To further reveal the insights of learned patterns of SplitLBI, we visualize the first convolutional layer of ResNet-18 18 on ImageNet-2012 along the training path of our SplitLBI as in Fig. 2. The left figure compares the training and validation accuracy of SplitLBI and SGD. The right figure compares visualizations of the filters learned by SplitLBI and SGD using Springenberg et al. (2014).
10 and weight decay every 40 epochs), in
Implementation. To be specific, denote the weights of an 1-layer network as (W1, W W For the i-th layer weights W', denote the j-th channel Wi- Then we compute the gradient of the sum of the feature map computed from each filter W; with respect to the input image (here a snake image). We further conduct the min-max normalization the gradient image, and generate the final visualization map. The right figure compares the visualized gradient images of first convolutional layer of 64 filters with (green), and 60 (black) epochs, respectively, which corresponds to the bounding boxes in the right figure annotated by the corresponding colors, i.e., purple, green, and black. We order the gradient images produced from 64 filters by the descending order of the magnitude (42-norm) of filters, i.e., images are ordered from the upper left to the bottom right. For comparison, we also provide the visualized gradient from random initialized weights.
receptive fields. We visualize the models parameters at 20 (purple), 40
Filters learned by ImageNet prefer to non-semantic texture rather than shape and color. The filters of high norms mostly focus on the texture and shape information, while color information is with the filters of small magnitudes. This phenomenon is in accordance with observation of Abbasi- Asl & Yu (2017) that filters mainly of color information can be pruned for saving computational cost. Moreover, among the filters of high magnitudes, most of them capture non-semantic textures while few pursue shapes. This shows that the first convolutional layer of R trained on ImageNet learned non-semantic textures rather than shape to do image classification tasks, in accordance with recent studies (Geirhos et al., 2019). How to enhance the semantic shape invariance learning, is arguably a key to improve the robustness of convolutional neural networks.
4.3 GLOBAL CONVERGENCE AND STRUCTURAL SPARSITY OF SPLITLBI
We conduct ablation studies based on Cifar- dataset with VGG-16 and ResNet-56 to evaluate (i) global convergence of L; and (ii) the structural sparsity learned by H3 via exploring test accuracies of sparse models obtained by projecting W, onto the support set of T (mask), by varying two key hyper-parameters and V/-
Implementation. We choose SplitLBI with momentum and weight decay, since it achieves very good performance large-scale experiments. Specifically, we have two set of experiments, where each experiment is repeated for 5 times: (1) we fix v = 100 and vary K = 1, 2, 5, 10, where sparsity of T, and validation accuracies of sparse models are shown in top row of Fig. 4. Note that we keep k -. = 0.1 in Eq (1a), to make comparable learning rate of each variant, and also consistent with SGD. Thus the learning rate Or will be adjusted by different x values. (2) we fix
1, and validate the results of SplitLBI with v/ = 10, 20, 50, 100, 200, 500, 1000, 2000 in the second row of Fig. 4 with the learning rate ag 0.1. Moreover, rather than using sparse models associated with r,,, Fig. 6 in Appendix shows the validation accuracies of full models learned by W,.
BBlitLBI converges to Critical Point. Figure 3 shows the curves of training loss (Zn) and accuracies, with each point representing the average and variance bar over 5 times. As shown, both training loss and training accuracy will converge, which validates our theoretical result in Theorem 1. Besides, larger k brings in slower convergence, which agrees with the analysis the convergence rate is inversely scale in Lemma A.5.
Sparse subnetworks achieve comparable performance to dense models without fine-tuning or retraining. From the experiments above, the sparsity of I grows as x and v increase. While large may cause a small number of important parameters growing rapidly, large v will decouple W, and I such that the growth of W, does not affect I, that may over-sparsify and deteriorate model accuracies.
8

---

Under review as a conference paper at ICLR 2020
Train Accuracy for Difireett VOG16
Train Accuracy for Offfeent for VGC16
Train Accuracy for Diffeeent for RectietS
Tran Accurscy for Difirent for Rechettd
0.00
0.00
6.70
0.40
0:50
0:3
0.60
0.50
Porafomatt00
Ranaforntt000
Aansterettoo
Train Loss for Oifferent for VGG16
Trait Loss for Different VGG16
Train Loss for Oifferntt for RechettS
Lass for Difierene Reateed2
1.5
1.2
3.0
0.0
0.0
43
80
100
150
140
160
20 20
230 140
150
Renaleens0001
Figure Loss and training accuracy by different k and v validates the global convergence of SplitLBI. The X-axis and Y-axis indicate the training epochs, and sparsity/accuracy. The results are repeated for 5 rounds, by keeping the exactly same initialization for each model. In each round, we use the same initialization for every hyperparameter. For all models, we train for 160 epochs with initial learning rate (Ir) of 0.1 and decrease by 0.1 at epoch 80 and 120.
Spares yaalee Acaracy Liftrerte fastect
4.
Spariiy Fer
Spartly Diferent Profect0
Scams Pibel vdddi Acuuagyl Ulhrret Redlece
120
Figure 4: Sparsity and validation accuracy by different k and / show that moderate sparse models may achieve comparable test accuracies to dense models without fine-tuning. Sparsity is obtained as the percentage of nonzeros in I, and sparse model at epoch is obtained by projection of We onto the support of Tr, i.e. pruning the weights corresponding to zeros in r,, The best accuracies achieved are recorded in Tab. and 4 of Appendix for different k and v, respectively. X-axis and Y-axis indicate the training epochs, and sparsity/accuracy. The results are repeated for times. Shaded area indicates the variance; and in each round, we keep the exactly same initialization for each model. In each round, we use the same initialization for every hyperparameter. For all the model, we train for 160 epochs with initial learning rate (Ir) of 0.1 and decrease by 0.1 at epoch 80 and 120.
Thus moderate choice of 6 and v is preferred in practice. In all cases, one can see that moderate sparse models can achieve comparable predictive power to dense models, even without tne-ttnnig or retraining. This shows that the structural sparsity parameter I' can indeed capture important weight parameter W, through their coupling.
4.4 SPLITLBI WITH EARLY STOPPING AND RETRAIN FINDS EFFECTIVE SUB-NETWORKS
Equipped with early stopping, H2 in early epochs may leam effective subnetworks (i.e. "winning tickets" (Frankle & Carbin, 2019; 90 Liu et al., 2019)) that achieve comparable or even better performance after retraining than existing pruning strategies by SGD.

---

Under review as a conference paper at ICLR 2020
VoS
550
MAXE
04.
$4.0
15
67
048
8.6
0.50
0.15 030
0.3
044
(a) VGG-16
(b) ResNet-55
(c) VGG-16 (Lasso)
(d) ResNer-50 (Lasso)
Figure 5: SplitLBI with early stopping finds sparse subnets whose test accuracies (stars) after retrain are comparable or even better than the baselines (Network Slimming (reproduced by the released codes from Liu ct al. (2019)), ) Soft-Filter Pruning(Tai) (Tab. 10), Scratch-B(Tab. 10), Scratch-E(Tab 10), and "Rethinking-Lotter
(Tab. 9a)) as reported in Liu et al. (2019). Sparse filters of VGG-16 and ResNet-56 are show in (a) and (b), while sparse weights of VGG-16 and ResNet-50 are shown in (c) and (d).
Experimental Design. We adopt a comparison baseline as the one-shot pruning strategy in Frankle & Carbin (2019), which firstly trains a dense over-parameterized model by SGD for 7 = 160 epochs and find the sparse structure by pruning weights or filters (Liu et al., 2019), then secondly retrains the structure from the scratch with T epochs from the same initialization as the first step. For SplitLBI, instead of pruning weights/filters from dense models, directly utilize the structural sparsity I, different training epochs to define the subnet architecture, followed by retrain-fromm cooe-ffrro----ttch (Fine-tune is shown in Appendix Sec. D with preliminary results). Experiments dataset where we still use VGG-16, ResNet-50, and ResNet-56 as the networks to make direct comparisons to previous works. SplitLBI uses momentum and weight decay with hyperparameters shown in Tab. 10 in Appendix. In particular, we set À = 0.1, and 0.05 for VGG- 16, and ResNet-56 respectively, since ResNet-56 has less parameters than VGG-16. Furthermore, we introduce another variant of our SplitLBI by using Lasso rather than group lasso penalty for I2 to sparsify the weights of convolutional filters; and the corresponding models are denoted as VGG-16 (Lasso) and ResNet-50 (Lasso). Every experiment is repeated for five times and the results are shown in Fig. 5. Note that in different runs of SplitLBI, the sparsity of I' slightly varies.
conducted on Cifar- 10
Sparse subnets found by early stopping of SplitLBI achieve remarkably good accuracy after retrain from scratch. In Fig.5 (a-b), sparse filters discovered by I' at different epochs are compared against the methods of Network Slimming (Liu et al., 2017), Soft Filter Pruning (Yang et al., 2018), Scratch-B, and Scratch-E, whose results are reported from Liu et al. (2019). At similar sparsity levels, SplitLBI can achieve comparable even better accuracy than competitors, even with sparse architecture learned from very early epochs (e.g. ( = 20 or 10). Moreover in Fig.5 (c-d), we can draw the same conclusion for the sparse weights of VGG-16 (Lasso) and ResNet-50 (Lasso), against the results reported in Liu et al. (2019). These results shows that the structural sparsity parameter I' found by early stopping of SplitLBI already discloses important subnetwork architecture that may achieve remarkably good accuracy after retrain from scratch. Therefore, it is not necessary to fully train a dense model to find successful sparse subnet architecture with comparable performance to the dense ones one can early stop SplitLBI properly where the structural parameter I, unveils "winning tickets" (Frankle & Carbin, 2019).
5 CONCLUSION
In this paper, a parsimonious deep learning method is proposed based on differential inclusions of inverse scale spaces. Implemented by a variable splitting scheme, such a dynamics system can exploit over-parameterized models and structural sparsity simultaneously. Besides, its simple discretization, i.e. the SplitLBI, has a proven global convergence and hence can be employed to train deep networks. We have experimentally shown that it can achieve the state-of-the-art performance on many datasets including ImageNet-2011 2012 with better interpretability than SGD. What's 's more, equipped with early stopping, such a structural sparsity can unveil the "winning tickets" the architecture of sub-networks which after re- re-training can achieve comparable and even better accuracy than original dense networks.
10

---

Under review as a conference paper at ICLR 2020
REFERENCES
Reza Abbasi-Asi and Bin Yu. Structural compression of convolutional neural networks based on greedy filter pruning. arXiv preprint arxiiv:/05.07335.. 2017. 1, 4.2
Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. A convergence theory for deep learning via over- parameterization. 2018. 1Xiv:1811.03962
Jose M Alvarez and Mathieu Salzmann. Learning the number of neurons in deep networks. In NIPS, 2016. 1
Sanjeev Arora, Rong Ge, Behnam Neyshabur, and Yi Zhang. Stronger generalization bounds for deep nets via a compression approach. arXiv preprint arXiv:1802.05296, 2018. 1
Hédy Attouch, Jérôme Bolte, and Benar Fux Svaiter. Convergence of descent methods for semi- algebraic and tame problems: proximal algorithms, forward-backward splitting, and regularized Gauss-Seidel methods. Mathematical Programming, 137:91-129, 2013. 1, A.3
Peter Bartlett, Dylan J. Foster, and Matus Telgarsky. Spectrally-normaaizee margin bounds for neural networks. In The s1 Conference on Neural Information Processing Systems (NIPS), Long Beach, CA, USA. 2017. 1
Peter L. Bartlett. For valid generalization the size of the weights is more important than the size of the network. In M. C. Mozer, M. I. Jordan, and T. Petsche (eds.), Advances in Neural Information Processing Systems 9, pp. 130-140. MIT Press, 1997.
Martin Benning. Marta M. Betcke, Matthias J. Ehrhardt, and Carola-Bibiane SchönlieB. Choose your path wisely: gradient descent in bregman distance framework. arXiv preprint arXiv:1712.04045., 2017. A.3
Jacek Bochnak, Michel Coste, and Marie-Francoise Roy. Real algebraic geometry, volume Ergeb. Math. Grenzgeb. Springer- Verlag, Berlin, 1998. 3, A.1
Jérôme Bolte, Aris Daniliddis, and Adrian Lewis. The Lojasiewicz inequality for nonsmooth subana- 1ytic functions with applications to subgradient dynamical systems. SIAM Journal on Optimization, 17:1205-122, 1223, 2007a. A.1, A.1
Jérôme Bolte, Aris Daniliddss, Adrian Lewis, and Masahiro Shiota. Clark subgradients of stratifiable functions. SIAM Journal on Optimization, 18:556-572, 2007b. A.I, A.1, A.2, A.2
Léon Bottou. Large-scale machine learning with stochastic gradient descent. In COMPSTAT, 2010. 1
Martin Burger, Guy Gilboa, Stanley Osher, and Jinjun Xu. Nonlinear inverse scale space methods. Communications Mathematical Sciences, 4(1):179-212, 2006. 2
Jian-Feng Cai, Stanley Osher, and Zuowei Shen. Convergence of the linearized bregman iteration for 11-norm minimization. Mathematics of Computation, 2009.
Maxwell Collins and Pushmeet Kohli. Memory bounded deep convolutional networks. In arXiv preprint 1412. 1442, 2014, 2014.
M. Coste. An introduction to o-minimal geometry. RAAG Notes, 81 pages, Institut de Recherche Mathematiques de Rennes, 1999. A.2
David L. Donoho and Xiaoming Huo. Uncertainty principles and ideal atomic decomposition. IEEE Transactions on Information Theory, 47(7):2845-262.. 2001. 1
Simon S. Du, Jason D. Lee, Haochuan Li, Liwei Wang, and Xiyu Zhai. Gradient descent finds global minima of deep neural networks. 2018. N0V.:181100804. 1
Dumitru Erhan, Yoshua Bengio, Aaron Courville, and Pascal Vincent. Visualizing higher-layer features of deep network. University of Montreal, Technical Report, 1341, 2009.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. International Conference on Learning Representations (ICLR), 2019. arXiv preprint w0w.11803.0333. 1, 4.4, 4.4,
11

---

Under review as a conference paper at ICLR 2020
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M Roy, and Michael Carbin. The lottery ticket hypothesis at scale. arXiv preprint arXiv:1903.01611 2019.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. Imagenet-trained enns are biased towards texture; increasing shape bias improves accuracy and robustness. International Conference on Leurning Representations (ICLR), 2019. arXiv preprint arXiv:1811.1222. 4.2
N. Golowich, A. Rakhlin, and O. Shamir. Size- independent sample complexity of neural networks. Conference on Learning Theory (COLT), 2018. arXiv preprint arXiv:1712.06541. 1
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In NIPS, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In ICCV, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.
Chendi Huang and Yuan Yao. A unified dynamic approach to sparse model selection. In The 2 International Conference on Artificial Intelligence and Statisties (AISTATS), Lanzarote, Spain, 2018. 1, 2,
Chendi Huang, Xinwei Sun, Jiechao Xiong, and Yuan Yao. Split lbi: An iterative regularization path with structural sparsity. In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems (NIPS) 29, pp. 3369-3377. 2016. 1,
Chendi Huang, Xinwei Sun, Jiechao Xiong, and Yuan Yao. Boosting with structural sparsity: A differential inclusion approach. Applied and Computational Harmonic Analysis, 2018. arXiv preprint arXiv: 704.04833. 2
Max Jaderberg, Andrea Vedaldi, and Andrew Zisserman. Speeding up convolutional neural networks with low rank expansions. BMVC, 2014.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015. 1
Steven Krantz and Harold R. Parks. A primer of real analytic functions. Birkhäuser, second edition, 2002. 2, A.2
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolu- tional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012. 4
Krzysztof Kurdyka. gradients of functions definable in o-minimal structures. Annales de l'institut Fourier, 48:769-783, 1998. A.1, A. A.2, A.2
Yann LeCun al. Lenet-5, convolutional neural networks. URL: http://yann. lecun. com/exdb/lenet. 20, 2015. 4
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. In ICLR, 2017.
Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learning efficient convolutional networks through network slimming. In ICCV, 2017. 4.4
Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. Rethinking the value of network pruning. In ICLR, 2019. 1, 4.4, 4.4
Stanistaw Lojasiewicz. Une propriété topologique des sous-ensembles analytiques réels. In: Les Équations aux dérivées partielles, Éditions du centre National de la Recherche Scientifique, Paris, pp. 87-89, 1963. 1, 3, A.1
12

---

Under review as a conference paper at ICLR 2020
Stanistaw Eojasiewiez. Ensembles semi-analytiques. Institut des Hautes Etudes Scientifiques, 1965. A.1
Stanislaw Lojasiewicz. Sur la geometrie semi-et sous-analytique. Annales de I' l'institul Fourier, 43: 1575-1595, 1993. A.1
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. International Conference on Learning Representations (ICLR), 2019. arXiv preprint arXiv:1711.05I0
Song Mei, Andrea Montanari, and Phan-Minh Nguyen. A mean field view of the landscape of two-layers neural network. Proceedings of the National Academy of Sciences (PNAS), 2018.
Song Mei, Theodor Misiakiewicz, and Andrea Montanari. Mean-field theory of two-layers neural networks: dimension-fit bounds and kernel limit. Conference on Learning Theory (COLT), 2019.
Boris S. Mordukhovich. Variational analysis and generalized disserentiation Basic Theory. Springer, 2006. A.1
Arkadi Nemirovski and David Yudin. Problem complexity and Method Efficiency in Optimization. New York: Wiley, 1983. Nauka Publishers, Moscow (in Russian), 1978.
Behnam Neyshabur, Zhiyuan Li, Srinadh Bhojanapalli, Yann LeCun, and Nathan Srebro. The role of over-parametriiaaion in generalization of neural networks. In International Conference on Learning Representations (ICLR), New Orleans, Louisiana, USA. 2019. 1
Stanley Osher, Martin Burger, Donald Goldfarb, Jinjun Xu, and Wotao Yin. An iterative regularization method for total variation-base image restoration. Multiscale Modeling & Simulation, 4(2):460- 489, 2005.
Stanley Osher, Feng Ruan, Jiechao Xiong, Yuan Yao, and Wotao Yin. Sparse recovery via differential inclusions. Applied and Computational Harmonic Analysis, 2016. 1,2
R. Tyrrell Rockafellar and Roger J-B Wets. Variational analysis. Grundlehren Math. Wiss. 317, Springer-Verlag, New York, 1998. A.I
Jianing V Shi, Wotao Yin, and Stanley Osher. Linearized bregman for f--regularized logistic regression. In Proceedings of the 30th International Conference on Machine Learning (ICML), 2013. 1
Masahiro Shiota. Geometry of subanalytic and semialgebruic sets, volume 150 of Progress in Mathematics. Birkhäuser, Boston, 1997. A.1
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1556, 2014. 4
Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller. Striving for simplicity: The all convolutional net. arXiv preprint arXiv: 1:Xiv:1412.6806 2014. 2, 4.2
Xinwei Sun, Lingjing Hu, Yuan Yao, and Yizhou Wang. Gsplit lbi: Taming the procedural bias in neuroimaging for disease prediction. In International Conference on Medical Image Computing and Computer-Assissed Intervention (MICCAI), pp. 107-115. Springer, 2017.
Joel A. Tropp. Greed good: Algorithmic results for sparse approximation. IEEE Trans. Inform. Theory, 50(10):2231-2242 2004. 1.2
L. den Dries. A generalization of the tarski-seidenberg theorem and some nondefinability results. Bull. Amer. Math. Soc. (N.S.), 15:189-193, 1986. A.2
I. van den Dries and C. Miller. Geometric categories and o-minimal structures. Duke Mathematical Journal, 84:497-540, 1996. A.2
Luca Venturi, Afonso S. Bandeira, and Joan Bruna. Spurious valleys in two-layer neural network optimization landscapes. 2018. 8.
13

---

Under review as a conference paper at ICLR 2020
Yu Wang, Wotao Yin, and Jinshan Zeng. Global convergence of admm in nonconvex nonsmooth optimization. Journal of Scientific Computing, 78(1):29-6)) 63, 2019. A.1
Yuting Wei, Fanny Yang, and Martin J. Wainwright. Early stopping for kernel boosting algorithms: A general analysis with localized complexities. The 31st Conference on Neural Information Processing Systems (NIPS), Long Beach, CA, USA, 2017.
Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning the number of neurons in deep networks. NIPS, 2016.
Fanghui Xue and Jack Xin. Convergence of a relaxed variable splitting method for learning sparse neural networks via (,, fo. and transformed-ff penalties. arXivv//12200571922, 2018. URL
http://arxiv. org/ abs 11t./1111.0051..
He Yang. Guoliang Kang, Xuanyi Dong, Yanwei Fu, and Y. Yang. Soft filter pruning for accelerating deep convolutional neural networks. In IJCAI 2018, 2018. 1, 4.4
Yuan Yao, Lorenzo Rosasco, and Andrea Caponnetto. early stopping in gradient descent learning. Constructive Approximation, 26(2):289-315, 2007.
Wotao Yin, Stanley Osher, Jerome Darbon, and Donald Goldfarb. Bregman iterative algorithms for compressed sensing and related problems. SIAM Journal on Imaging sciences, 1(1):143-168, 2008.
Jachong Yoon and Sung Ju Hwang. Combined group and exclusive sparsity for deep neural networks. In ICML, 2017.
Ming Yuan and Yi Lin. Model selection and estimation in regression with grouped variables. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 68(1))49-67, 2006. 1,
Jinshan Zeng, Tim Tsz- Kit Lau, Shao-Bo Lin, and Yuan Yao. Global convergence of block coordinate descent in deep learning. In Proceedings of the 36th International Conference Machine Learning, Long Beach, California, 2019. URL https::/. : a rxxiv. org/ abs /tt//18033.. 00225. 3, A.2
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. International Conference on Learning Representations (ICLR), 2017. arXiv: 1. .03530. 1
Xiangyu Zhang, Jianhua Zou, Kaiming He, and Jian Sun. Accelerating very deep convolutional networks for classification and detection. IEEE Transactions on Puttern Analysis and Machine Intelligence, 38(10): 2016. 1
Bo Zhao, Xinwei Sun, Yanwei Fu, Yuan Yao, and Yizhou Wang. Msplit lbi: Realizing feature selection and dense estimation simultaneously in few-shot and zero-shot learning. In International Conference Machine Learning (ICML), 2018.
Peng Zhao and Bin Yu. On model selection consistency of lasso. J. Machine Learning Research, 2541-2567, 2006. 1,
Aojun Zhou, Anbang Yao, Yiwen Guo, Lin Xu, and Yurong Chen. Incremental network quantization: Towards lossless cnns with low-precision weights. ICLR, 2017.
Chenzhuo Zhu, Song Han, Huizi Mao, and William J Dally. Trained ternary quantization. ICLR,
2017.1 1
14

---

Under review as a conference paper at ICLR 2020
APPENDIX TO SplitLBI deep learning: structural sparsity via differential inclusion paths
A PROOF OF THEOREM
First of all, we reformulate Eq. (6) into an equivalent form. Without loss of generality, consider Ω = 22 in the sequel.
Denote R(P) (TT), then Eq. (6) can be rewritten as,
Pk+1 Prox, Ps + n(pp - aVL(P)))).
(15a)
Pk+1 = Pk k (PP+1 - Pk + kaVL(P)))
(15b)
where Pk= [0. gul" (R(P)) and Ok (9((T)) Thus SplitLBI is equivalent the following iterations,
Wk+1 W4 K C(W,, L'k);
(16a)
Tk+1 Prox.n + k(g) - a [VV 9k+1 = - (T N4
(16b)
ka VrC(W, [x)).
(16c)
Exploiting the equivalent reformulation (16a-16c), one can establish the global convergence of (Wi, kk gs) based on the Laodykk--oojsicwiizz framework. In this section, the following extended version of Theorem is actually proved.
Theorem 2. /Global Convergence of SplitLBIJ Suppose that Assumption I holds. Let (Wkr L'k, gs) be the sequence generated by SplitLBI (Eq. (16a-16c)) with a finite initialization. If
2
0 < ag =a x(Lip+u-1)
then (W&, Tu, gk) converges to a critical point of F. Moreover, {(W,, [&)) converges to a stationary point of L defined in Eg. 2, and {W*} converges to stationary point of Ln).
A.1 SORDDYK--LIISIEWWEE PROPERTY
To introduce the definition of the Handyka-Looasiewiice (KL) property. need some notions and notations from variational analysis, which can be found in Rockafellar & Wets (1998).
The notion of subdifferential plays central role in the following definitions. For foreachxx dom(h) {x E IP :{(x) < +oo), the Fréchel subdifferential of at x, written @x(x), is the set of vectors E RP which satisfy
h(y) 1((x)= xll
(v.y x 0.
lim y
inf
When x & dom(h), we set Th(x) = Ø. The suntrinngsubdfffeeenttal (or simply subdifferential) of introduced in Mordukhovich (2006), written 0h(x) at x E dom(h), is defined by
@h(x) = e R": 3x
1x(*) h(x), * E 0h(x*)) + v
(17)
A necessary (but not sufficient) condition for x Rp to be minimizer of h is 0 ah(x). A point that satisfies this inclusion is called limiting-critical simply critical. The distance between point to subset S of RP, written dist(x, S), is defined by dist(x, = inf s|| E S}, where || represents the Euclidean norm.
Let h RRP + R u {+o0} be an extended-real socteed-eeallvaae real-valued function (respectively, h RP = R be point-to-set mapping), its graph is defined by
Graph(h) : {(x, v) E R x R =
(resp. Graph(h) {(x,y) R R y h())))
15

---

Under review as a conference paper at ICLR 2020
and its domain by (om := {x E R' h(x) +oo} (resp. dom(h) := fx R2 h(x) & 2)). When h is a proper function, i.e,, when dom(h) * 2, the set of its global minimizers (possibly
empty) is denoted by
arg min h == (x E RP: l(x) = inf h).
The KL property (Lojasiewicz, 1963; 1993; Kurdyka, 1998; Bolte et al., 2007a;b) plays central role in the convergence analysis of nonconvex algorithms (Attouch et al., 2013; Wang et al., 2019). The following definition is adopted from Bolte et al. (2007b).
Definition 1.
property] A function h is said to have the
(KL) property at u (ōm == {u E |0 [h(()) * 0}, if there exists a constant 1j E (0, neighborhood N of u and a function o ().m) 7 R., which is a concave function that is continuous at and satisfies ((0) Φ E C ((0, n)), Le., à continuous differentiable on (0, n),
and ø' (s) > 0 for all s E (0,n), n). such that for all E EN {u Rn (h(ū) < hhlu) < h((ū) + n), the following inequality holds
& (1((u)) - h(a)) dist(O, @h(u)) 1
(18)
If h satisfies the KL property at each point of dom(@)), h is called a KL functiioon.
KL functions include real analytic functions, semialgebraic functions, tame functions defined in some o-minimal structures (Kurdyka, 1998; Bolte et al., 2007b), continuous subanalytic functions (Bolte et al., 2007a) and locally strongly convex functions. In the following, we provide some important examples that satisfy the
property.
Definition 2. (Real analytic] A, function h with domain an open set U C R and range the set of either all real or complex numbers, said to be real analytic at u if the function h may be reprexented by convergent power series on some interval of positive radius centered at u. h(x) = EE== ag (2 -u}, for some {az} C R.. The function is said to be real analytic on V UU if is real analytic at each u E V (Kruntz & Purks, 2002, Definition 1.1.5). The real analytic function f over IPP for some positive integer p >> 1 can be defined similarly.
According to Krantz & Parks (2002), typical real analytic functions include polynomials, exponential functions, and the logarithm, trigonometric and power functions on any open set of their domains. One can verify whether a multivariable real function h(x) on IPP is analytic by checking the analyticity of g(t) h(x + ty) for any x,y E TR?
Definition 3. [Semialgebraic]
(a) A set D C TR& is called semialgebraic (Bochnak al., 1998) if can be represented
D D-ÜN EXER: : = 0, 0, 0},
i=1j=1
where Pij; Qij are real polynomial functions for ssis 8, sist.
(b) A function TRP + IR U (+xo) (resp. a point-to-set mapping h Rp = RR) is called semialgebraic if its graph Graph(h) is semialgebraic.
According to (Lojasiewicz, 1965; Bochnak et al., 1998) and (Shiota, 1997, I.2.9, page 52), the class of semialgebraic sets are stable under the operation of finite union, finite intersection, Cartesian product or complementation. Some typical examples include polynomial functions, the indicator function of semialgebraic set, and the Euclidean norm (Bochnak al., 1998, page 26).
A.2 KI. PROPERTY IN DEEP LEARNING AND PROOF OF COROLLARY 1
In the following, we consider the deep neural network training problem. Consider a l-layer feedfor- ward neural network including 1 hidden layers of the neural network. Particularly, let d, be the number of hidden units in the i-th hidden layer for i == 1,....7 1. Let da and dj be the number of units of input and output layers, respectively. Let W E xd be the weight matrix between the (i 1) layer and the i-th layer for any = I....??
2To simplify notations, we regard the input and output layers as the 0-th and the I-th layers, respectively, and absorb the bias of each layer into W
16

---

Under review as a conference paper at ICLR 2020
According Theorem 2, one major condition is to verify the introduced Lyapunov function F defined in (9) satisfies the aurdyka-Lojasiewvee of semialgebraic set, called the o-minimal structure (see, for instance Coste (1999), van den Dries (1986), Kurdyka (1998), Bolte et al. (2007b)). The following definition is from Bolte et al. (2007b).
property. For this purpose, we need an extension
Definition 4. fa-minimal structure] An o-minimal struciure (R. +,,) is a sequence of boolean algebras On of "definable" subsets of R", such that for each vt E M
(i) if A belongs to On then R and A belong Ont1i
(ii) if 11 [R=+1 _ is the canonical projection onto R", then for any A in On+1, the set (A) belongs to On:
(iii) On contains the family of algebraic subsets of 12, that is, every set of the form
{x RR p(x)= = 0),
where p: R T is a polynomial function.
(iv) the elements of On are exactly finite unions of intervals and points.
Based on the definition of o-minimal structure, we can show the definition of the definable function.
Definition 5. [Definable function] Given o-minimal structure O (over +,))))), function
f: RR" + R said be definable O if its graph belongs to On+1-
According to van den Dries & Miller (1996); Bolte et al. (2007b), there are some important facts of the o-minimal structure, shown as follows.
ii) The collection of semialgebraic sets is an o-minimal structure. Recall the semialgebraic sets are Bollean combinations of sets of the form
{x E R" p(z) = 0, (z) < 4m (x) < 0},
where p and q 's are polynomial functions in R""
(ii) There exists o-minimal structure that contains the sets of the form
{xx, t) E 1|" R =]
where is real-analytic around [|1,
(iii) There exists an o-minimal structure that contains simultaneously the graph of the exponential
function IR. 3 æ +> exp(x) and all semialgebraic sets.
(iv) The o-minimal structure is stable under the sum, composition, the inf-convolution and several other classical operations of analysis.
The Kurdyka--.ojasiewiez property for the smooth definable function and non-smooth definable function were established (Kurdyka, 1998, Theorem 1) and (Bolte et al., 2007b, Theorem 11), respectively. Now are ready to present the proof of Corollary 1.
Proof. [Proof of Corollary 1] To justify this corollary, we only need to verify the associated Lyapunov function F satisfies Kundyka-Lojasieve inequality. this case and by (10), F can be rewritten as follows
F(W,T,9==o -a (E., ((..
1 20 IW [ rIp))
+(()) + (2) - (W.g).
Because L and a,'s are definable by assumptions, then Σ., (W.T) are definable as compositions of definable functions. Moreover, according to Krantz & Parks (2002), |W - r||2 and (W. g) are semi-algebras and thus definable. Since the group Lasso Q(T) Eg [['|2 is the composition of l2 and & norms, and the conjugate of group Lasso penalty is the maximum of group norm,, í.e. Ω (T) = max, ||T 11l2, where the lg, f1, and foo norms are definable, hence the group Lasso and its conjugate are definable as compositions of definable functions. Therefore, F is definable and hence satisfies Sanddyka-oojsiiewaa
inequality by (Kurdyka, 1998, Theorem 1).
The verifications of other cases listed in assumptions can found in the proof of (Zeng et al., 2019, Proposition 1). This finishes the proof of this corollary.
17

---

Under review as a conference paper at ICLR 2020
A.3 PROOF OF THEOREM 2
Our analysis is mainly motivated by a recent paper (Benning et al., 2017), as well as the influential work (Attouch al., 2013). According (Attouch al., 2013, Lemma 2.6), there mainly four ingredients in the analysis, that is, the sufficient descent property, relative error property, continuity property of the generated sequence and the Saaddkaa-Ljjsiewiic. property of the function. More specifically, we first establish the sufficient descent property of the generated sequence via exploiting the Lyapunov function F (see, (9)) in Lemma A.4 Section A.4, and then show the relative enror property of the sequence in Lemma A.5 in Section A.5. The continuity property is guaranteed by the continuity of Z(W, I') and the relation limss-oc Bh (T 1 = established in Lemma I(i) in Section A.4. Thus, together with the Kurdyka-Loojsiewiczz assumption of F,, we establish the global convergence of SLBI following by (Attouch et al., 2013, Lemma 2.6).
Let (W, g) be critical point of F, then the following holds
dw F(W, g) a: CV (W) + "( - [)) = 0,
F. r, g az W) + +Q((T 9 0,
(19)
FW (, = (0) 3.
By the final inclusion and the convexity S2, implies g E an(T). Plugging this inclusion into the second inclusion yields ov (T- W) = 0. Together with the first equality imples
VC(W, I1) = 0, V(x(W)= =.
This finishes the proof this thcorem.
A.4 SUFFICIENT DESCENT PROPERTY ALONG LYAPUNOV FUNCTION
Let Ps (Wk,Tw), and Qs (Pk. 9k-) e N. In the following, we present the sufficient descent property of Qs along the Lyapunov function F.
Lemma. Suppose that En is continuously differentiable and VŻ, is Lipschitz continuous with constant Lip O. Let {Qx} be sequence generated by SLBI with finite initialization. If 0 <<a x(Lipto Tj. then
((0++)) Qx -u
where p "
Proof. By the optimality condition of (15a) and also the inclusion pk 10, 9x E @R((P)), there holds
<(aVL(PP) + P&+1 - PP)) + Pa+1 - Ps =0
which implies
(V PA+1 - Pk) k PA+1 Pxll2 R(T
(20)
where
D k+1, ak) = (gh+1 9k. Tk+1 Tx)- -T))
Noting that Z(P)= E., (W) + 1 IW - T and by the Lipschitz continuity V E,, (W) with
20
constant Lip > 0 implies VL Lipschitz continuous with constant Lip + -1, This implies
[(Px+1) S ((P) + (VC(P)) Pa+1 - Pb) Lip + v
|P - Ps1l2
2
Substituting the above inequality into (20) yields
aC((et)) ((T Tx) + p|| - Px < af(p))
(21)
Adding some terms in both sides of the above inequality and after some reformulations implies aL(Pk+1) + B [Rk+
(22)
S <aC(Pi) + B (Tx, Tx-1) - Pa+1 P&ll3 (D(T K+1, TR) + Bat- [x-1) (T*+1,
aC(P)) + Ba TTT, Pk+1 Pall
B 'TT,, TR-1) B3 (T-x;
18

---

Under review as a conference paper at ICLR 2020
where the final equality holds for D(T,+ [k) B (T*+1,
B + (Tk, (k1)). That is,
F(Qu+1) & F(Q2) Pl|Pt+1 -Pxl2
B gu+1 (xx, -
(23)
< F(Q2) pl|Px+1 Px
(24)
where the final inequality holds for Bh (TTK, R- 2 0 and B "TTK, (k-1) 2 0. Thus, we finish the proof of this lemma.
Based on Lemma A.4, we directly obtain the following lemma.
Lemma 1. Suppose that assumptions of Lemma A.4 hold. Suppose further that Assumption (b)-(d)
hold. Then
(i) both a(C(Pk) and {F(Q+)) converge to the same finite value, and limg Bh (I [x) = O.
(i) the sequence {(Wk. k:, 9hh) is bounded,
(ii))
lim |P Ps 11 0 and limgg O(TR41,TR)
(iv) Ek=o |P Ps 2 a rate of OL//K).
Proof. By (21), [(Ps) is monotonically decreasing due ([T)+1,TA) 2 0. Similarly, by (24), F(Q*) also monotonically decreasing. By the lower boundedness assumption of (W). both C(P) and F(Q) are lower bounded by their definitions, i.e., (2) and (9), respectively. Therefore, both (C(P)) and (F(Qx)) converge, and it is obvious that limg-soe F(Qx) 2 limg-so aL(P)). By (23),
Bh B (k-)) & F(Qx) F(QR+1), .....
By the convergence of F(Qu) and the nonegativeness of 1 (I'k, 1R)-)), there holds
lim
B
(T,, =.
10000
By the definition of F(Qu) aL(pp) + B (T'- I R11 and the above equality, it yields
lim F(Qx) lim a.((P)) k000
k-000
Since E,, (W) has bounded level sets, then Wt is bounded. By the definition of Σ(W, I) and the finiteness C(W,, w. Is), I' is also bounded due to W. is bounded. The boundedness of ga due to ga @Q((T.). condition (d), and the boundedness of lk.
By (24), summing up (24) over 0,1...., K yields
(pllPat1 Pall D()) < aC(P)) oc.
(25)
Letting K + 2o and noting that both |P 12 and D Px) are nonnegative, thus
lim |P - Pall = 0, lim D(T Fx) 0. k
t00
Again by (25),
1 K
(pll Pa + ((T)+1.PD) < <-
K
k=0
which implies & E PP psl12 rate of O(1KK).
19

---

Under review as a conference paper at ICLR 2020
A.5 RELATIVE ERROR PROPERTY
In this subsection, we provide the bound of subgradient by the discrepancy of two successive iterates. By the definition of F (9).
a w WWWCWWWe (W441, TR+1)
HK+1
a aVrL(WW I,, TRt1) 1) + 9k+1 -
9k
@F((Q++),, EeN.
(26)
Tx Tk+1
Lemma.
Under assumptions of Lemma 1, then
||+++1| S pi pil|Q4+1 Qull, for HR41 =0F(QR+)), k KEN,
where P1 : -2 + aLLip + 2v-1), Moreover, * E [Hx 1 + a rate of (11/K).
Proof. Note that
Fett) (e (wr((W++1THH)
(x41) +(VW Vw [b)) + Vw.((W,
(27)
By the definition of (see (2)),
||V [(W++1, V L( [w)ll -
[Ww Vw V rx)]l 1IVV,, (W (Wi)) + (W
W&)l
(Tip + W|WW++ Well.
where the last inequality holds for the Lipschitz continuity of VLn with constant Lip 0, and by (16a),
| w C(W,, [x)]l =
Wall-
Substituting the above (in)equalities into (27) yields
(WW+, < [(Ra)-1+ + Lip + v v1 || Well + + - - -Tll
Thus,
||o CW k+1, 11 [i-1 + (Tip + |W - Wall + +ov -all-- (28)
By (16c), it yields
gh++ - 9k = x (T
Noting that Vr T&) = v- Ws), and after some simplifications yields
(WVT + gh+1 gkl = ||(x - av -v-1) (I's - l'k+1) + av I(W, Wx+1)
av av - Wk+1ll ll + (x av "'][T= - -r++ill
(29)
where the last inequality holds for the triangle inequality and x > av-1 by the assumption.
By (28), (29), and the definition of Hk+1 (26), there holds
| < - (Lip 1) || Will + (N T) Fkl
< 2x + +1+ a( 2y } || PPll
(30)
< 2x + +1+ a( + 2n -1]] 1[Qu+1 - QuIl.
By (30) and Lemma 1(iv), * > || 0 at rate of O(IKK).
This finishes the proof of this lemma.
20

---

Under review as a conference paper at ICLR 2020
valdeto asocner the tener Wagis
Ve deter Acomno bes
Valceror accuncc Newerr Tereeee
Figure 6: Validation curves of dense models W, for different k and v. For SLBI we find that the model accuracy is robust to the hyperparameters both in terms of convergence rate and generalization ability. Here validation accuracy means the accuracy on test set of Cifarl0. The first one is the result for VGG16 ablation study on K, the second one is the result for ResNet56 ablation study on K, the third one is the result for VGG16 ablation study on and the forth one is the result for ResNet56 ablation study on
Type
Model
K = 1
==
2
5
K
10
SGD
Vgg 16
93.46
93.27
92.77
92.03
93.57
Full
ResNet56
92.71
92.18
91.50
90.92
93.08
1gg16
93.31
93.00
92.36
76.25
Sparse
ResNet56
92.37
91.85
89.48
87.02
Table 2: This table shows results for different K, the results all the best test accuracy. Here test two widely-used models: VGG16 and ResNet56 on Cifar10. For results in this table, we keep 100. Full means that use the trained model weights directly, Sparse means the model weights are combined with mask generated by I' support. Sparse result has no finetuning process, the result is comparable its Full counterpart. For this experiment, we propose that K 1 good choice. For all the model, we train for 160 epochs with initial learning rate (Ir) of O. and decrease by 0.1 at epoch 80 and 120.
B SUPPLEMENTARY EXPERIMENTS
B. ABLATION STUDY OF VGGI6 AND RESNET56 ON CIFAR10
To further study the influence hyperparameters, we record performance of W, for each epoch t with different combinations of hyperparameters. The experiments is conducted 5 times each, we show the mean the table, the standard emror can be found in the corresponding figure. We perform experiments Cifar10 and two commonly used network VGG16 and ResNet56.
On K, we keep // = 100 and try A = 1, 2, 5, 10, the validation curves of models W. are shown in Fig. 6 and Table 2 summarizes the mean accuracies. Table summarizes best validation accuracies achieved at some epochs, together with their sparsity rates. These results show that larger kappa leads to slightly lower validation accuracies, where the numerical results are shown in Table 2, We can find that x = 1 achieves the best test accuracy.
and try 10, 20, 50, 100, 200, 500, 1000, 2000 the validation curve and mean accuracies are show in Fig. and Table Table summarizes best validation accuracies achieved at some epochs, together with their sparsity rates. By carefully tuning we can achieve similar even better results compared to SGD. Different from K, # has less effect on the generalization performance. By tuning it carefully, we can even get sparse model with slightly better performance than SGD trained model.
On keep
C COMPUTATIONAL COST OF SPLITLBI
We further compare the computational cost of different optimizers: SGD (Mom), SplitLBI (Mom) and Adam (Naive). We test each optimizer on one GPU, and all the experiments are done on one GTX2080. For computational cost, judge them from two aspects : GPU memory usage and time needed for one batch. The batch size here is 64, experiment is performed on VGG- 16 as shown in Table 6.
21

---

Under review as a conference paper at ICLR 2020
Model
Ey20
Ep40
Eys0
Egl60
Term
Spansay
Aec
Spasity
Acc
Spasity
Ace
Spasity
Ace
96.62
71.51
96.62
76.92
96.63
77.48
96.63
93.31
Va216
K
51.86
72.98
71.99
73.64
75.69
74.54
75.72
93.00
8.19
10.00
17.64
34.25
29.76
69.92
30.03
92.36
10
0.85
10.00
6.62
10.00
12.95
38.38
13.26
76.25
Term
Spansay
Spasity
Ade
Spanity
Spasity
Ace
96.79
73.50
96.87
75.27
96.69
77.47
99.68
92.37
ResNe156
x
76.21
72.85
81.41
74.72
84.17
75.64
84.30
91.85
36.58
60.43
53.07
76.00
57.48
75.67
57.74
89.48
10
3.12
10.20
29.43
53.36
41.18
74.56
41.14
87.02
Table 3: Sparsity rate and validation accuracy for different at different epochs. Here we pick the test accuracy for specific epoch. In this experiment, we keep v/ = 100. We pick epoch 20, 40, 80 and 160 to show the growth sparsity and sparse model accuracy. Here Sparsity is defined Sec. and Acc means the test accuracy for sparse model. A sparse model is a model at designated epoch t combined with the mask as the support of r,-
Type
Model
10
20
V 50
v= 100
200
500
1000
2000
SGD
1gg16
93.66
93.59
93.57
93.39
93.38
93.35
93.43
93.46
93.57
ResNet56
93.12
92.68
92.78
92.45
92.95
93.11
93.16
93.31
93.08
V2g16
93.39
93.42
93.39
93.23
93.21
93.01
92.68
10
Spanse
ResNe156
92.81
92.19
92.40
92.10
92.68
92.81
92.84
88.96
Table 4: Results for different v, the results are all the best test accuracy. Here test two widely-used
model : VGG16 and ResNet56 on Cifar10. For results in this table, we keep k = 1. Full means that we use the trained model weights directly, Sparse means the model weights mask generated by I support. Sparse result has no finetuning process, the result is comparable to its Full counterpart. For all the model, we train for 160 epochs with initial learning rate (Ir) of 0.1 and decrease by 0.1 at epoch 80 and 120.
combined with
Fg20
Eg40
Fp80
Ep160
Sparityy
Sposity
Spasity
Sposity
10
96.64
71.07
96.64
77.70
96.65
79.46
96.65
93.34
96.64
69.11
96.64
77.63
96.65
77.08
96.65
93.42
50
96.64
74.91
96.65
74.21
96.65
79.15
96.65
93.38
Veg16
96.64
74.82
96.64
73.22
96.64
78.09
96.64
93.23
200
91.69
73.67
94.06
74.67
94.15
75.20
94.15
93.21
18.20
10.00
59.94
67.88
82.03
78.69
82.32
93.01
1000
6.43
10.00
17.88
10.00
49.75
61.31
31.21
92.68
0.22
10.00
6.89
10.00
18.15
10.00
19.00
10.00
Spasint
Spesiy
Ace
Spasity
Spasit
10
99.97
73.37
99.95
71.64
99.74
76.46
99.74
92.81
99.97
72.58
99.84
74.16
99.69
72.37
99.72
92.19
50
99.96
70.72
99.89
73.96
99.79
74.93
99.77
92.40
ResNec56
96.31
73.63
96.63
75.79
96.55
72.94
96.57
92.10
# 200
91.98
75.30
94.38
72.13
94.87
73.75
94.88
92.68
74.44
65.58
90.00
74.12
92.96
71.91
92.99
92.81
1000
24.32
10.85
75.68
70.23
88.56
79.67
88.80
92.48
0.65
10.00
26.66
13.30
74.98
70.38
75.92
88,95
Table 5: Sparsity rate and validation accuracy for different at different epochs. Here we pick the test accuracy for specific epoch. In this experiment, we keep x = 1. We pick epoch 20, 40, 80 and 160 to show the growth of sparsity and sparse model accuracy. Here Sparsity is defined in Sec. 4 as the percentage of nonzero parameters, and Acc means the test accuracy for sparse model. A sparse model is a model designated epoch combined with mask the support of Lr.
22

---

Under review as a conference paper at ICLR 2020
optimizer
SGD
SLBI
Adam
Mean Batch Time
0.0197
0.0221
0.0210
GPU Memory
1161MB
1459MB
1267MB
Table 6: Computational and Memory Costs. ( (Ad GPU memory means the whole memory model weights and the activation cache.
Layer
FC1
FC2
FC3
Sparsity Number of Weights
0.049
0.087
0.398
235200
30000
1000
Table This table shows the sparsity for every layer of Lenet-3. Here sparsity is defined in Sec. 4, number of weights denotes the total number of parameters in the designated layer. It is interesting that the I' tends to put lower sparsity on layer with more parameters.
D FINE-TUNING OF SPARSE SUBNETWORKS
We design the experiment on MNIST, inspired by Frankle Carbin (2019). Here, we explore the subnet obtained by I after T 100 epochs of training. As in Frankle et al. (2019), we adopt the "rewind" trick: re-loading the subnet mask of l 100 at different epochs, followed by fine-tuning. In particular, along the training paths, we reload the subnet models Epoch O, Epoch 30, 60, 90, and 100, and further fine-tune these models by SplitLBI (Mom-Wod Wd). All the models use the same initialization and hence the subnet model Epoch 0 gives the retraining with the same random initialization proposed to find winning tickets of lottery in Frankle & Carbin (2019). We will denote the rewinded fine-tuned model epoch 0 as (Lottery), and those at epoch 30, 60, 90, and 100, as F-epoch30, F-epoch60. F-epoch900, and F-epoch respectively. Three networks are studied here LeNet-,, Conv-2, and Conv-4, LeNet-3 removes one convolutional layer of LeNet-5; and thus less wee-parrmeterize than the other two networks. Conv-2 and Conv-4, as the scaled-down variants of VGG family as done in Frankle & Carbin (2019), have two and four fully-connected layers, respectively, followed by max max-pooling after every two convolutional layer.
The whole sparsity for Lenet-3 is 0.055, Conv-2 is 0.0185, and Conv-4 is 1378. Detailed sparsity for every layer of the model is shown in Table 7, 8, 9. We find that fc-layers are sparser than conv-layers.
We compare SplitLBI variants to the SGD (Mom- dd) and SGD (Lottery) (Frankle & Carbin, 2019) in the same structural sparsity and the results are shown in Fig. 7. In this exploratory experiment, one can see that for overparameterized networks - Conv-2 and Conv-4, fine-tuned rewinding subnets F-epoch30, F-epoch60, F-epoch90,, and F-epoch100, can produce better results than the full models; while for the less eeer-parameterized model LeNet-3, fine-tuned subnets may achieve less yet still comparable performance to the dense models and remarkably better than the retrained sparse subnets from beginning (i.e. AplitLBI/SGD (Lottery)). These phenomena suggest that the subnet architecture disclosed by structural sparsity parameter Fr valuable, for fine- fine-tuning sparse models with comparable or even better performance than the dense models of WT.
E RETRAINING OF SPARSE SUBNETS FOUND BY SPLITLBI (LOTTERY)
Here we provide more details on the experiments in Fig. 5. Table 10 gives the details on hyper- parameter setting. Moreover, Figure provides the sparsity variations during SplitI.I training in Fig. 5.
Layer
Convi
Conv2 1
FC1
FC2
FC3
Sparsity 0.9375
0.0067
0.0284
0.1551
Number of Weights 576
36864
3211264
65536
2560
Table This table shows the sparsity for every layer of Conv-2. Here sparsity is defined in Sec. number of weights denotes the total number of parameters in the designated layer. The sparsity is more significant fully connected (FC) layers than convolutional layers.
23

---

Under review as a conference paper at ICLR 2020
Layer Conv1
Conv2
Conv3
Conv4
FC1
FC2
FC3
Sparsity
0.921875
0.0040
0.0094
0.1004
Number of Weights
576
36864
73728
147456
1605632
65536
2560
Table 9: This table shows the sparsity for every layer of Conv-4. Here sparsity is defined in Sec.
number of weights denotes the total number of parameters in the designated layer. Most of the convolutional layers are kept while the FC layers are very sparse.
The important Subnet Learned by SpilL Bi
5C31Mum Wall
59
SCDiLitary)
Sp We)
35
Fepechas
Pesecho)
75
90.00
Networks
Figure 7: Fine-tuning of sparse subnets learned by SplitLBI may achieve comparable or better performance than dense models. F-epochk indicates the fine-tuned model comes from the Epoch k. SplitLBI (Lottery) and SGD (Lottery) use the same sparsity rate for each layer and the same initialization for retrain.
Network
Penalty
Optimizer
X
Momentum
Nesterov
VGG-16
Group Lasso
SLBI
0.1
100
0.1
0.9
Yes
ResNet-56
Group Lasso
SLBI
0.1
100
0.05
0.9
Yes
VGG-I6(LLasso
Lasso
SLBI
0.1
500
0.05
0.9
Yes
ResNet-50(Lasso)
Lasso
SLBI
0.1
200
0.03
0.9
Yes
Table 10: Hyperparameter setting for the experiments in Figure 5.
Buster
10
(a) VGG-16
(b) 5e
(c) VGG-16 (Lasso)
(d) ResNet-50 (Lasso)
Figure 8: Sparsity changing during training process of Sp (Lottery) for VGG and ResNets (corresponding to Fig. 5). We calculate the sparsity in every epoch and repeat five times. The black curve represents the mean of the sparsity and shaded area shows the standard deviation of sparsity. The vertical blue line shows the epochs that we choose to early stop. We choose the log-scale epochs for achieve larger range of sparsity.
24


Reviews:
[{"review_id": "HJePlf46YH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies the problem of finding important structural sparsity of over-parameterized neural networks. Compared to the traditional pruning and distillation based compression methods, the author propose a novel optimization based algorithm called SplitLBI, which can efficiently and effectively find the important subnetwork architecture. In addition, the authors provide the convergence guarantee of the proposed method for solving nonconvex optimization problems under certain conditions. Thorough experiments and ablation studies validate the advantage of the proposed method.\n\nThe contributions of this paper are as follows:\n1. A novel optimization based method for finding important sparse structure of large-scale neural networks using the idea of coupling the learning of weight matrix and sparsity constraints.\n2. The convergence guarantee of the proposed method for solving nonconvex optimization is established based on a novel Lyapunov function.\n3.Extensive experiments show that the proposed algorithm can: (a) train over-parameterized neural networks to achieve the state-of-the-art performances on image classification tasks; (b) find sparse networks with competitive networks; and (c) early stopping plus retrain from scratch can achieve similar or better  performance than existing compression method.\n\nI believe that this work is very interesting and will be of interest to the ICLR community. The presentation of this paper is clear and the idea of coupling the gradient descent and mirror descent is very interesting. In addition, the empirical evaluations, including the sensitivity of the parameters and computation and memory costs, of the proposed method are solid.\n\nMinor comments:\n1.There is no definition of KL function in the main content.\n2.Before presenting Corollary 1, please specify the g_k as well as the updates for training neural networks.\n3.Is it possible to extend the convergence guarantee to the ReLU activation function since the architectures considered in experiments using such activation. \n4.In experiments it is better to present the convergence of the training loss to validate the convergence results.\n5.In Figure 2, what is SLBI-10 and SLBI-1?\n\nUpdate: Thanks for your clarification, and I recommend acceptance of this work.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BJxzGS6vjB", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n=======\nThis paper aims to train sparse neural networks efficiently, by jointly optimizing the weights and sparsity structure of the network. It applies the Split Linear Bregman Iteration (Split LBI) method from [1] in a large-scale setting, to train deep neural networks.\n\nThe approach works by considering optimization in a joint space (W, \\Gamma) consisting of the network weights W and a new set of parameters \\Gamma that model the structural sparsity of the network.\n\nThe problem of learning sparse networks efficiently is important for modern applications that run on embedded devices, as well as for fast training on specialized hardware. I think the approach is interesting as a potential alternative to more expensive methods for finding sparse networks, such as NAS and the successive pruning & re-training approach of [2].\n\nOverall, the paper pursues a promising direction to induce network sparsity, and presents some interesting results. However, there are several issues with the experiments and the structure/presentation of the paper that should be addressed.\n\nPros\n====\n* As far as I am aware, this is the first application of Split LBI to train deep neural networks.\n* It shows that joint optimization of the weights and sparsity structure performs on par with baselines that only optimize weights, on MNIST, CIFAR-10, and ImageNet.\n* It provides a global convergence analysis that shows that the weights optimized with Split LBI converge to a critical point of the training loss, regardless of initialization.\n* It provides an ablation study for the two hyperparameters \\kappa and \\nu of Split LBI.\n* I think the most interesting parts of the paper are those that examine the structural sparsity learned by Split LBI (Sections 4.3 and 4.4). In particular, the fact that Split LBI was able to match or outperform the test accuracy of several baselines (network slimming, soft filter pruning, and the method in Rethinking the Lottery Ticket Hypothesis) in a single training run (without re-training) is a nice result.\n\nIssues\n======\n* The Split LBI method is presented as a novel contribution (in the abstract: \"we propose a new approach based on differential inclusions of inverse scale spaces ...\"), but it was already described in detail in a paper that they cite ([1]) published at NeurIPS 2016. The only theoretical contribution of this paper is section 3: Global convergence of Split LBI.\n\n* The claim that \"Split LBI demonstrates SOTA performance in large scale training on ImageNet\" (from the abstract) is not correct, and needs to be qualified. This paper reports 70.55/89.56% (top 1 / top 5) accuracy; as far as I am aware, the current SOTA on ImageNet is [3], which achieves 86.4/98.0% (top 1 / top 5) accuracy. I think it would be better for this paper to argue that it achieves comparable performance to baselines with a particular architecture and training regime.\n\n* The structure and presentation of the paper could be improved in several ways, outlined as follows:\n    - Most of the Methodology section discusses the Split LBI method from prior work. I would encourage the authors to split the Methodology section into a separate Background section for Split LBI, followed by a new section specifically about applying Split-LBI to convolutional and fully-connected layers in neural networks.\n\n    - The writing is missing some details and explanations that would be very helpful for readers. For example, it should clearly state that the dimension of \\Gamma is the same as the dimension of the weights W.\n\n    - It would also be good to expand the explanation about why SplitISS avoids the parameter correlation problem, and what it means for \\Gamma to have an orthogonal design?\n\n    - The figures are too small to be readable without a lot of zooming.\n\n    - The paper ends abruptly, with no conclusion.\n\n    - The appendix contains some useful material, but much of it is not referenced from the main paper. I think some parts of the appendix could be moved to the main paper, for example the comparison of computational and memory costs between Split LBI and SGD.\n\n* I am not sure what the purpose of the comparisons between optimizers in Table 1 is. The motivation given in the abstract and introduction is to learn sparse networks online during optimization; it does not propose Split LBI as a new optimizer to compete with Adam. Couldn't one use the Adam update rule to optimize the weights in Split-LBI? I think it makes sense to compare Split LBI to standard training setups that do not enforce any sparsity, as well as to setups that use L1 and L2 regularization, but I do not think that Table 1 is set up correctly for this. Different optimizers are paired with different regularizers, and crucially the choice of hyperparameters is not discussed---how did you choose the coefficient of L1 regularization to be 1e-3? Additionally, many rows in Table 1 are missing data (e.g., the variants of Adam are only run for CIFAR-10).\n\n* Regarding experiments, it is not clear which experiments the authors actually performed, for which they took the results from previously published papers, and sometimes where the results come from at all.\n\n    - In Table 1, there is an asterix next to SGD-Mom-Wd that the authors say indicates \"results from the official pytorch website.\" That would imply that the rest of the experiments were done by the authors (and the authors say in the caption \"we use the official pytorch codes to run the competitors\"). However, Table 2 (found in the Appendix, page 20), contains identical numbers and a sign # that, according to the authors, indicates results of their own experiments. That would mean that of all the SGD and Adam experiments shown in the table, the authors only performed SGD-naive and Adam-naive. Where do the other numbers come from? What does \"official pytorch code to run the competitors\" mean? Where is that code from?\n\n    - Figure 4 contains baselines and SplitLBI results. Where do the numbers for the baselines come from? The caption mentions another paper, [5], but I did not find the source of the numbers in that paper. The caption seems to point to Table 9a in [5], but that table does not deal with Network Slimming, Soft-Filter Pruning, Scratch B, or Scratch-E. Additionally, Table 9a of [5] only contains results for VGG-16 and ResNet-50. Where do the baselines for ResNet-56 (Figure 4b) come from?\n\n* How is the proximal objective in Eq. 5 optimized? That is, how do you compute the argmin?\n\n* Figure 2 shows results for SLBI-1 and SLBI-10, but no discussion of what SLBI-1 and SLBI-10 mean. Also regarding Fig.2, the authors claim that \"Filters learned by ImageNet prefer non-semantic texture rather than shape and color.\" How did the authors come to this conclusion? I looked carefully at the filter visualizations, and I cannot see a clear difference between the filters learned by Split LBI and SGD.\n\n* The computation time comparison in Table 11 (Appendix E) is a bit strange, because it shows that Adam takes 2x as long as SGD, which does not align with my experience; in practice, the wall-clock time is nearly identical between Adam and SGD. It would be good to provide more details about how the time was measured. Also, does the memory comparison measure only the memory used for model parameters (W and \\Gamma), or also activation memory? Shouldn't Split LBI use 2x the memory of SGD (if measuring only the weights)?\n\n* In Figure 1, it looks like the initial magnitude of the filters is larger for SGD compared to Split LBI. Are the weights initialized in the same way? Also, why is the setup of the MNIST experiment in Fig.1 different from the setup in Table 1 (e.g., learning rate decay every 40 epochs vs every 30 epochs)? In addition, it looks like the first learning rate decay causes the filter weight magnitudes to flatten out and stay constant.\n\n* What is the learning rate schedule used for the runs in Figure 3? It looks like the lr decays at epoch 80 and 120, but this is only mentioned in table captions in the appendix. This should be stated in the main paper. Also, why is this a different training setup from that used for Table 1?\n\nI also noted that the authors do not intend to make the code public upon publication of the paper. On page 6, they state that \"source codes will be released upon requests.\" At present, the preferred path is to make the code public upon publication of the paper.\n\nMinor points\n============\n* In the caption of Figure 3, it says \"The results are repeated for 5 times. Shaded area indicates the variance; and in each round, we keep the exactly same initialization for each model.\" What is different between the 5 runs if the initialization is the same?\n\n* There are too many different colors used in Figures 1 and 2. Since the purple, green, and black boxes are important to see for figures 1 and 2, it is confusing to have to deal with additional blue, pink, and yellow boxes around every three.\n\n\n[1] Huang et al., Split LBI: An iterative regularization path with structural sparsity. NeurIPS 2016.\n[2] Frankle & Carbin, The Lottery Ticket Hypothesis: Finding sparse, trainable neural networks. ICLR 2019.\n[3] Touvron et al., Fixing the train-test resolution discrepancy. https://arxiv.org/abs/1906.06423.\n[4] He et al., Deep residual learning for image recognition. https://arxiv.org/abs/1512.03385.\n[5] Liu et al., Rethinking the value of network pruning. ICLR 2019.\n\n\n\nPost-rebuttal Update\n====================\n\nI thank the authors for their rebuttal, and for clarifying some details in the paper.\n\n* I think the experiments on sparsity are interesting. More efficient ways to find good sparse networks are certainly of interest to the community.\n\n* I appreciate that the authors released the source code.\n\n* In summary, this paper applies Split-LBI to neural network training, and provides a global convergence result as one of the main contributions. Operationally, compared to the original Split LBI approach, it changes the loss function from squared error to cross entropy, and uses mini-batches for training, which are fairly straightforward.\n\n* One important issue with the paper is that it blurs the distinction between prior work and the new contribution. For example, the subsection on Split Linearized Bregman Iteration in the \"Methodology\" section does not contain anything new compared to [1], and this is not clear enough to the reader. Also, not enough credit is given to [2] for the \"Differential of Inclusion of Inverse Scale Space\" subsection. I maintain that there needs to be a separate \"Background\" section for this, and that it should be made absolutely clear what is new in the \"Methodology\" section. It feels like this distinction is obfuscated in the writing.\n\n* Given that the authors propose Split-LBI as a new optimizer that can be compared to others (e.g., Adam), one issue is that there doesn't seem to be any search done over the hyperparameters of each optimizer, including the learning rate and amount of weight decay. For example, Adam is only run with learning rate 1e-3 and SGD is run with 0.1; in addition, the weight decay (where used) is only set to 1e-4. Thus, it is not clear how meaningful these comparisons are. Also, in Table 1, the CIFAR-10 test accuracies are fairly low at ~90%, while modern models such as Wide ResNets can achieve ~95%.\n\n* The newly-written conclusion is still incorrect, stating again that Split LBI achieves SOTA performance on ImageNet. Also, if  \"with better interpretability than SGD\" refers to the qualitative comparison of the learned filters, I think this conclusion is a bit too strong, because I don't think the difference is visible enough to aid interpretability.\n\n* Minor point: On further inspection, the legend in the left-side plots in Figure 2 does not match the labels of the visualizations on the right. There is no yellow training curve in Figure 2, despite the assertion in the rebuttal.\n\n\n[1] Huang et al., Split LBI: An iterative regularization path with structural sparsity. NeurIPS 2016.\n[2] Osher et al., \"Sparse recovery via differential inclusions.\" Applied and Computational Harmonic Analysis, 2016.\n\n\nI maintain my score of weak reject, but am not totally opposed to it being accepted, because it provides a way to find sparse networks more efficiently.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BJxS7wJZcr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nEdit after rebuttals: I have read all other reviews and rebuttals and maintain my assessment.\n----\n\nI believe this paper should have been desk-rejected, as it is 9p long in the main text (above the suggested limit of 8), plus 4p of bibliography, for a total of 13p, above the hard limit of 10p. It is concerning that all figures are so squeezed as to be unreadable in print version. In the remainder of my review, as well as my rating, I will ignore the length issue, and leave it to the AC to include this fact.\n\nThis paper is concerned with methods to enforce sparsity in neural networks (specially CNN for image classification) at training time. It is motivated by the issue that L1 regularisation violates incoherence and irrepresentability conditions (cf sec1 and sec2 below eq3). The approach taken to encourage structural sparsity is to formulate the training dynamics in terms not only of the neural network weights $W$, but also of their sparse version $V$.\n\nI recommend (other than for the length issue) this paper for publication. My review needs to be relativised as I am not familiar with the literature on inverse scale spaces. Factors for my recommendation are: (1) the paper is clearly written (apart from an important language issue, cf below) and laid out, (2) the problem is well-motivated and contextualised (although I cannot judge novelty), (3) experiments, including analytic and ablation experiments, seem well designed and conducted, (4) experimental evidence shows that the proposed method is efficient, (5) the formal treatment is strong and thorough, with relevant literature cited.\n\nSuggestions for improvement\n- faulty language definitely stands in the way of understanding, in several places. There is about one syntax error in every paragraph of the paper. Sentences are too long as they span several lines, and should be broken up for clarity. In particular, the use of \"that\" and gerunds vs present tense is mostly erroneous. Eg: \"while the other set of parameters learning\", \"structural sparsity set $\\Gamma$ that\", \"comparable test accuracy as\", \"that the whole iterative sequence\", \"which drives $W_t$ closely following\", \"whose global convergence condition to be shown\" etc.\n- to allow non-specialists to read this paper, you should more clearly explain and illustrate the spaces in which different variables are, starting with $V$ and $\\Gamma$.\n- the paper lacks a conclusion which should bring together several strands of argumentation\n- I strongly suggest publishing the source code, not just promising to release them upon request. The methods described here require some skill to be coded, especially I cannot see how to easily re-use existing library components; this makes it hard for the ideas in this paper to be applied and diffused.\n- it is insufficient to state that \"official source code\" was used, as in table 1; this should definitely state the (Github or other) URL where the code is available", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper investigates an existing method for fitting sparse neural networks, and provides a novel proof of global convergence.  Overall, this seems like a sensible, if marginal, contribution.  However, there were serious red flags regarding the care which which the scholarship was done which make me deem the current submission unsuitable for publication.  In particular, two points raised by R4, which were not addressed even after the rebuttal:

1) "One important issue with the paper is that it blurs the distinction between prior work and the new contribution. For example, the subsection on Split Linearized Bregman Iteration in the "Methodology" section does not contain anything new compared to [1], and this is not clear enough to the reader."

2) "The newly-written conclusion is still incorrect, stating again that Split LBI achieves SOTA performance on ImageNet."

I believe that R3's high score is due to not noticing these unsupported or misleading claims.


Decision: Reject