Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: Summary
=======
This paper aims to train sparse neural networks efficiently, by jointly optimizing the weights and sparsity structure of the network. It applies the Split Linear Bregman Iteration (Split LBI) method from [1] in a large-scale setting, to train deep neural networks.

The approach works by considering optimization in a joint space (W, \Gamma) consisting of the network weights W and a new set of parameters \Gamma that model the structural sparsity of the network.

The problem of learning sparse networks efficiently is important for modern applications that run on embedded devices, as well as for fast training on specialized hardware. I think the approach is interesting as a potential alternative to more expensive methods for finding sparse networks, such as NAS and the successive pruning & re-training approach of [2].

Overall, the paper pursues a promising direction to induce network sparsity, and presents some interesting results. However, there are several issues with the experiments and the structure/presentation of the paper that should be addressed.

Pros
====
* As far as I am aware, this is the first application of Split LBI to train deep neural networks.
* It shows that joint optimization of the weights and sparsity structure performs on par with baselines that only optimize weights, on MNIST, CIFAR-10, and ImageNet.
* It provides a global convergence analysis that shows that the weights optimized with Split LBI converge to a critical point of the training loss, regardless of initialization.
* It provides an ablation study for the two hyperparameters \kappa and \nu of Split LBI.
* I think the most interesting parts of the paper are those that examine the structural sparsity learned by Split LBI (Sections 4.3 and 4.4). In particular, the fact that Split LBI was able to match or outperform the test accuracy of several baselines (network slimming, soft filter pruning, and the method in Rethinking the Lottery Ticket Hypothesis) in a single training run (without re-training) is a nice result.

Issues
======
* The Split LBI method is presented as a novel contribution (in the abstract: "we propose a new approach based on differential inclusions of inverse scale spaces ..."), but it was already described in detail in a paper that they cite ([1]) published at NeurIPS 2016. The only theoretical contribution of this paper is section 3: Global convergence of Split LBI.

* The claim that "Split LBI demonstrates SOTA performance in large scale training on ImageNet" (from the abstract) is not correct, and needs to be qualified. This paper reports 70.55/89.56% (top 1 / top 5) accuracy; as far as I am aware, the current SOTA on ImageNet is [3], which achieves 86.4/98.0% (top 1 / top 5) accuracy. I think it would be better for this paper to argue that it achieves comparable performance to baselines with a particular architecture and training regime.

* The structure and presentation of the paper could be improved in several ways, outlined as follows:
    - Most of the Methodology section discusses the Split LBI method from prior work. I would encourage the authors to split the Methodology section into a separate Background section for Split LBI, followed by a new section specifically about applying Split-LBI to convolutional and fully-connected layers in neural networks.

    - The writing is missing some details and explanations that would be very helpful for readers. For example, it should clearly state that the dimension of \Gamma is the same as the dimension of the weights W.

    - It would also be good to expand the explanation about why SplitISS avoids the parameter correlation problem, and what it means for \Gamma to have an orthogonal design?

    - The figures are too small to be readable without a lot of zooming.

    - The paper ends abruptly, with no conclusion.

    - The appendix contains some useful material, but much of it is not referenced from the main paper. I think some parts of the appendix could be moved to the main paper, for example the comparison of computational and memory costs between Split LBI and SGD.

* I am not sure what the purpose of the comparisons between optimizers in Table 1 is. The motivation given in the abstract and introduction is to learn sparse networks online during optimization; it does not propose Split LBI as a new optimizer to compete with Adam. Couldn't one use the Adam update rule to optimize the weights in Split-LBI? I think it makes sense to compare Split LBI to standard training setups that do not enforce any sparsity, as well as to setups that use L1 and L2 regularization, but I do not think that Table 1 is set up correctly for this. Different optimizers are paired with different regularizers, and crucially the choice of hyperparameters is not discussed---how did you choose the coefficient of L1 regularization to be 1e-3? Additionally, many rows in Table 1 are missing data (e.g., the variants of Adam are only run for CIFAR-10).

* Regarding experiments, it is not clear which experiments the authors actually performed, for which they took the results from previously published papers, and sometimes where the results come from at all.

    - In Table 1, there is an asterix next to SGD-Mom-Wd that the authors say indicates "results from the official pytorch website." That would imply that the rest of the experiments were done by the authors (and the authors say in the caption "we use the official pytorch codes to run the competitors"). However, Table 2 (found in the Appendix, page 20), contains identical numbers and a sign # that, according to the authors, indicates results of their own experiments. That would mean that of all the SGD and Adam experiments shown in the table, the authors only performed SGD-naive and Adam-naive. Where do the other numbers come from? What does "official pytorch code to run the competitors" mean? Where is that code from?

    - Figure 4 contains baselines and SplitLBI results. Where do the numbers for the baselines come from? The caption mentions another paper, [5], but I did not find the source of the numbers in that paper. The caption seems to point to Table 9a in [5], but that table does not deal with Network Slimming, Soft-Filter Pruning, Scratch B, or Scratch-E. Additionally, Table 9a of [5] only contains results for VGG-16 and ResNet-50. Where do the baselines for ResNet-56 (Figure 4b) come from?

* How is the proximal objective in Eq. 5 optimized? That is, how do you compute the argmin?

* Figure 2 shows results for SLBI-1 and SLBI-10, but no discussion of what SLBI-1 and SLBI-10 mean. Also regarding Fig.2, the authors claim that "Filters learned by ImageNet prefer non-semantic texture rather than shape and color." How did the authors come to this conclusion? I looked carefully at the filter visualizations, and I cannot see a clear difference between the filters learned by Split LBI and SGD.

* The computation time comparison in Table 11 (Appendix E) is a bit strange, because it shows that Adam takes 2x as long as SGD, which does not align with my experience; in practice, the wall-clock time is nearly identical between Adam and SGD. It would be good to provide more details about how the time was measured. Also, does the memory comparison measure only the memory used for model parameters (W and \Gamma), or also activation memory? Shouldn't Split LBI use 2x the memory of SGD (if measuring only the weights)?

* In Figure 1, it looks like the initial magnitude of the filters is larger for SGD compared to Split LBI. Are the weights initialized in the same way? Also, why is the setup of the MNIST experiment in Fig.1 different from the setup in Table 1 (e.g., learning rate decay every 40 epochs vs every 30 epochs)? In addition, it looks like the first learning rate decay causes the filter weight magnitudes to flatten out and stay constant.

* What is the learning rate schedule used for the runs in Figure 3? It looks like the lr decays at epoch 80 and 120, but this is only mentioned in table captions in the appendix. This should be stated in the main paper. Also, why is this a different training setup from that used for Table 1?

I also noted that the authors do not intend to make the code public upon publication of the paper. On page 6, they state that "source codes will be released upon requests." At present, the preferred path is to make the code public upon publication of the paper.

Minor points
============
* In the caption of Figure 3, it says "The results are repeated for 5 times. Shaded area indicates the variance; and in each round, we keep the exactly same initialization for each model." What is different between the 5 runs if the initialization is the same?

* There are too many different colors used in Figures 1 and 2. Since the purple, green, and black boxes are important to see for figures 1 and 2, it is confusing to have to deal with additional blue, pink, and yellow boxes around every three.


[1] Huang et al., Split LBI: An iterative regularization path with structural sparsity. NeurIPS 2016.
[2] Frankle & Carbin, The Lottery Ticket Hypothesis: Finding sparse, trainable neural networks. ICLR 2019.
[3] Touvron et al., Fixing the train-test resolution discrepancy. https://arxiv.org/abs/1906.06423.
[4] He et al., Deep residual learning for image recognition. https://arxiv.org/abs/1512.03385.
[5] Liu et al., Rethinking the value of network pruning. ICLR 2019.



Post-rebuttal Update
====================

I thank the authors for their rebuttal, and for clarifying some details in the paper.

* I think the experiments on sparsity are interesting. More efficient ways to find good sparse networks are certainly of interest to the community.

* I appreciate that the authors released the source code.

* In summary, this paper applies Split-LBI to neural network training, and provides a global convergence result as one of the main contributions. Operationally, compared to the original Split LBI approach, it changes the loss function from squared error to cross entropy, and uses mini-batches for training, which are fairly straightforward.

* One important issue with the paper is that it blurs the distinction between prior work and the new contribution. For example, the subsection on Split Linearized Bregman Iteration in the "Methodology" section does not contain anything new compared to [1], and this is not clear enough to the reader. Also, not enough credit is given to [2] for the "Differential of Inclusion of Inverse Scale Space" subsection. I maintain that there needs to be a separate "Background" section for this, and that it should be made absolutely clear what is new in the "Methodology" section. It feels like this distinction is obfuscated in the writing.

* Given that the authors propose Split-LBI as a new optimizer that can be compared to others (e.g., Adam), one issue is that there doesn't seem to be any search done over the hyperparameters of each optimizer, including the learning rate and amount of weight decay. For example, Adam is only run with learning rate 1e-3 and SGD is run with 0.1; in addition, the weight decay (where used) is only set to 1e-4. Thus, it is not clear how meaningful these comparisons are. Also, in Table 1, the CIFAR-10 test accuracies are fairly low at ~90%, while modern models such as Wide ResNets can achieve ~95%.

* The newly-written conclusion is still incorrect, stating again that Split LBI achieves SOTA performance on ImageNet. Also, if  "with better interpretability than SGD" refers to the qualitative comparison of the learned filters, I think this conclusion is a bit too strong, because I don't think the difference is visible enough to aid interpretability.

* Minor point: On further inspection, the legend in the left-side plots in Figure 2 does not match the labels of the visualizations on the right. There is no yellow training curve in Figure 2, despite the assertion in the rebuttal.


[1] Huang et al., Split LBI: An iterative regularization path with structural sparsity. NeurIPS 2016.
[2] Osher et al., "Sparse recovery via differential inclusions." Applied and Computational Harmonic Analysis, 2016.


I maintain my score of weak reject, but am not totally opposed to it being accepted, because it provides a way to find sparse networks more efficiently.
review_title: Official Blind Review #4

final_score: rating: 6: Weak Accept
review_content: 
Edit after rebuttals: I have read all other reviews and rebuttals and maintain my assessment.
----

I believe this paper should have been desk-rejected, as it is 9p long in the main text (above the suggested limit of 8), plus 4p of bibliography, for a total of 13p, above the hard limit of 10p. It is concerning that all figures are so squeezed as to be unreadable in print version. In the remainder of my review, as well as my rating, I will ignore the length issue, and leave it to the AC to include this fact.

This paper is concerned with methods to enforce sparsity in neural networks (specially CNN for image classification) at training time. It is motivated by the issue that L1 regularisation violates incoherence and irrepresentability conditions (cf sec1 and sec2 below eq3). The approach taken to encourage structural sparsity is to formulate the training dynamics in terms not only of the neural network weights $W$, but also of their sparse version $V$.

I recommend (other than for the length issue) this paper for publication. My review needs to be relativised as I am not familiar with the literature on inverse scale spaces. Factors for my recommendation are: (1) the paper is clearly written (apart from an important language issue, cf below) and laid out, (2) the problem is well-motivated and contextualised (although I cannot judge novelty), (3) experiments, including analytic and ablation experiments, seem well designed and conducted, (4) experimental evidence shows that the proposed method is efficient, (5) the formal treatment is strong and thorough, with relevant literature cited.

Suggestions for improvement
- faulty language definitely stands in the way of understanding, in several places. There is about one syntax error in every paragraph of the paper. Sentences are too long as they span several lines, and should be broken up for clarity. In particular, the use of "that" and gerunds vs present tense is mostly erroneous. Eg: "while the other set of parameters learning", "structural sparsity set $\Gamma$ that", "comparable test accuracy as", "that the whole iterative sequence", "which drives $W_t$ closely following", "whose global convergence condition to be shown" etc.
- to allow non-specialists to read this paper, you should more clearly explain and illustrate the spaces in which different variables are, starting with $V$ and $\Gamma$.
- the paper lacks a conclusion which should bring together several strands of argumentation
- I strongly suggest publishing the source code, not just promising to release them upon request. The methods described here require some skill to be coded, especially I cannot see how to easily re-use existing library components; this makes it hard for the ideas in this paper to be applied and diffused.
- it is insufficient to state that "official source code" was used, as in table 1; this should definitely state the (Github or other) URL where the code is available
review_title: Official Blind Review #2

final_score: rating: 8: Accept
review_content: This paper studies the problem of finding important structural sparsity of over-parameterized neural networks. Compared to the traditional pruning and distillation based compression methods, the author propose a novel optimization based algorithm called SplitLBI, which can efficiently and effectively find the important subnetwork architecture. In addition, the authors provide the convergence guarantee of the proposed method for solving nonconvex optimization problems under certain conditions. Thorough experiments and ablation studies validate the advantage of the proposed method.

The contributions of this paper are as follows:
1. A novel optimization based method for finding important sparse structure of large-scale neural networks using the idea of coupling the learning of weight matrix and sparsity constraints.
2. The convergence guarantee of the proposed method for solving nonconvex optimization is established based on a novel Lyapunov function.
3.Extensive experiments show that the proposed algorithm can: (a) train over-parameterized neural networks to achieve the state-of-the-art performances on image classification tasks; (b) find sparse networks with competitive networks; and (c) early stopping plus retrain from scratch can achieve similar or better  performance than existing compression method.

I believe that this work is very interesting and will be of interest to the ICLR community. The presentation of this paper is clear and the idea of coupling the gradient descent and mirror descent is very interesting. In addition, the empirical evaluations, including the sensitivity of the parameters and computation and memory costs, of the proposed method are solid.

Minor comments:
1.There is no definition of KL function in the main content.
2.Before presenting Corollary 1, please specify the g_k as well as the updates for training neural networks.
3.Is it possible to extend the convergence guarantee to the ReLU activation function since the architectures considered in experiments using such activation. 
4.In experiments it is better to present the convergence of the training loss to validate the convergence results.
5.In Figure 2, what is SLBI-10 and SLBI-1?

Update: Thanks for your clarification, and I recommend acceptance of this work.
review_title: Official Blind Review #3

Meta-review: This paper investigates an existing method for fitting sparse neural networks, and provides a novel proof of global convergence.  Overall, this seems like a sensible, if marginal, contribution.  However, there were serious red flags regarding the care which which the scholarship was done which make me deem the current submission unsuitable for publication.  In particular, two points raised by R4, which were not addressed even after the rebuttal:

1) "One important issue with the paper is that it blurs the distinction between prior work and the new contribution. For example, the subsection on Split Linearized Bregman Iteration in the "Methodology" section does not contain anything new compared to [1], and this is not clear enough to the reader."

2) "The newly-written conclusion is still incorrect, stating again that Split LBI achieves SOTA performance on ImageNet."

I believe that R3's high score is due to not noticing these unsupported or misleading claims.


Decision: Reject