Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: In this paper, the authors propose a sparse momentum algorithm for doing efficient sparse training. The technique relies on identifying weights in a layer that do not have an effect on the error, pruning them, and redistributing and growing them across layers. The technique is compared against other recent algorithms on a range of models.

The paper is well written, and very easy to read. The proposed algorithm looks interesting and seems to empirically work well. 

I am, however, a bit confused with how the sparse momentum algorithm is discussed in this paper. In the paper, momentum is intuitively presented as an algorithm that reduces the variance of the noise in the gradients. However, there are a number of papers that show that this is not the case (and if anything it is the opposite). Further, a few recent papers show that momentum works better than SGD only for learning rates that are not too small, and this is because it averages out the gradients in the high-curvature directions (these are the directions where the gradients switch signs) and makes them stable in these directions, thus allowing larger steps in the low curvature directions. See for example the following two papers:
Momentum Enables Large Batch Training. Samuel L Smith, Erich Elsen, Soham De. ICML Workshop on Physics for Deep Learning, 2019.
Which Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic Model. Guodong Zhang, Lala Li, Zachary Nado, James Martens, Sushant Sachdeva, George E Dahl, Christopher J Shallue, Roger Grosse. NeurIPS 2019.

Given these papers, can the authors comment on what they think is the reason for the effectiveness of their sparse momentum algorithm? These papers seem to indicate to me that an interesting (and important) ablation study would be to compare using just the gradients vs using momentum for the sparse training algorithm for both a small batch and a large batch. Without this ablation study, it is a bit unclear to me why/when this algorithm is working well, and this primarily explains my current score.

The experimental results look impressive, although I am not very aware of other work in sparse training, so unfortunately it is harder for me to properly evaluate the significance of the empirical results in this paper, and whether the numbers reported are indeed a significant improvement over current state-of-the-art algorithms. I do have a few questions about the design and the results of the experiments presented:

1. Because the momentum of zero-valued weights are used, does that mean that the gradient over all weights are required to be taken at each step? How much does this affect training time in your experiments?

2. How were the learning rates decided, and why are they kept fixed across methods? It seems feasible that the optimal learning rate could vary highly between methods?

==========================================

Edit after rebuttal:
I thank the authors for the very detailed response and for doing the additional experiments requested. Due to the thoroughness of the response, I am increasing my score to a weak accept. I do however have a couple of comments regarding the author response:

1. I do not agree that keeping the learning rate fixed across methods is the right approach. Many different things might interact with each other to change the optimal learning rate, and it is always more interesting to see the optimal performance attained by a method through a grid search. As long as the tuning budget across all methods are somewhat similar, I think the experiments would be more informative when doing a learning rate sweep. Further, while doing the sweep, the authors should be careful that the optimal learning rate does not lie on the edge of the grid search. This is what happens in the experiments reported in the authors' response titled "Results from learning rate grid search" where the optimal learning rate for all models lie at one of the extremes of the grid search (0.09).

2. I would request the authors to slightly rewrite certain parts of their paper so as not to imply that momentum decreases the variance of the gradients in general. There are many papers (including the ones I mentioned in my review) that specifically show momentum does not have this effect. All of these papers however consider a different setting from the one considered in this paper (where there is a parameter redistribution step), so it is not immediately clear whether momentum has the same effect in this case. However, unless there is very specific evidence to show a variance reduction effect, I think putting in that intuition might be misleading. 
It is interesting to see that momentum helps for small batches in this setting, and I agree that further investigation into this would be an interesting future direction. Note that there is a concurrent submission on sparse training (https://openreview.net/forum?id=ryg7vA4tPB&noteId=ryg7vA4tPB) that seems to show that higher momentum values (0.99) does better when considering large batches. So perhaps additional and more careful experiments need to be done about this before these additional experimental results can be put in the paper.

review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: This paper proposes an algorithm called Sparse Momentum for learning sparse neural networks. They claim to maintain sparse weights throughout training while achieving dense network performance levels. They also show their method improves training speed up to 5.61x faster training. The provides a decent motivation for why sparse networks can be helpful. The related work section is well summarized and they emphasize that the current work's primary motivation is to reduce training time while maintaining performance. They compare their method with other methods that also maintain sparse neural networks throughout training and involve single training phase, which is fair. Their method consists of primarily 3 phases - i) pruning weights ii) redistribution of weights iii) regrowing of weights based on the exponentially smoothed momentum term for each layer. The method is well explained and motivation is clear. Edge case was also explained for more clarity. However, there is something that needs more clarification in Pg-3, 3rd line, they say the 3 components of the algorithm i) ii) and iii) can be tackled independently with a divide and conquer strategy to gain some computational benefits. In my understanding, the method performs these 3 steps sequentially after each epoch. Not sure how divide and conquer strategy can be used here ?

The results were shown for MNIST & CIFAR-10 using AlexNet, VGG16 and LeNet-5 models. They show that the current algorithm is better than other proposed methods in the literature in most of the cases. The claims made in some cases that this method reaches dense network performance is not completely true though. For example, in Table 1, the only case where the proposed method reaches dense network's performance is for VGG16-D. In all the rest of the cases, the current method and dense network error differ by at least 5%. 

The authors compare speed up results for training in two ways: theoretical speedups which are proportional to reduction in number of FLOPS and practical speedups using dense convolutional algorithms corresponding to completely empty channels. It's good that the authors have mentioned due to the current lack of optimal sparse matrix multiplication implementations these speedups cannot be garnered practically yet. The estimates based on FLOPs reduction or Empty Channel based look promising. They also show ablation study of how the redistribution and weight regrowth based on momentum is better than doing in a random fashion.

Overall, I think the paper proposes an interesting idea of using momentum with promising results to learn sparse neural networks. 
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: 
The paper proposes a method to train a sparse network and achieve "dense-level" performance. The method redistributes the sparsity according to momentum contribution of each layer during training after each epoch. Experiments on multiple datasets and architectures are conducted. 

While the method itself seems interesting, I do have have several important concerns.

1. My biggest concern is about the experiments. Some results sometime seem questionable.

1) Looking at the results for ImageNet ResNet50, the 20%-weights model achieve 0.7% test error loss compared with dense baseline. According to Table 6 in [1], using the simple pruning and fine-tuning method (Han et al. 2015), the 40%-weights model suffers from 0.06% error loss compared with dense baseline. I don't see a clear advantage of sparse momentum here. It seems possible that under the same sparsity level just pruning using Han et al. is better. Can the authors compare with Han et al. 2015 fairly at the same sparsity level, on multiple datasets? An apple-to-apple comparison is extremely necessary. 

Also the baseline result differ: in [1] dense ResNet-50 achieve 76.15% but in this paper baseline result is 74.9%. Both papers use Pytorch. And Pytorch official number is actually 76.15% (see https://pytorch.org/docs/stable/torchvision/models.html ). In my experience that is reproduceble if you follow the official pytorch code of training ImageNet. What's the difference here? I think the authors should follow the standard way of training ImageNet to make the results more convincing.  Finally [1] is a relevant paper and should be discussed.

2) I suggest that MNIST results be moved to Appendix since accuracy on MNIST is too easy to reach a high level, and the interpretation over it might not be convincing. Yet a lot of the analysis of the method's effectiveness is on MNIST.

3) For CIFAR, the results of SNIP on VGG in Table 1 also seems inconsistent with their original paper. In their paper's Table 2, the VGG-like model achieve ~0.3% error reduction at 5% weights while in this paper's Table 1 it's a 0.5% error increase. Again, what is the difference? Is the SNIP method reimplemented? Putting that potentially flawed result aside, the performance on CIFAR is not better than other methods as well according to Figure 3.

2. I don't think the general performance of the trained sparse models can be said to "rival" the dense model. It's a non-negligible margin in most times, for CIFAR and ImageNet. It's a little exaggerating to say that in the title, abstract and introduction.

3. The benefit of the method seems unclear. It does not speedup training compared with the conventional training a dense model and then pruning pipeline (Han et al.). The test-time real speedup is also limited according to Table 3. The real benefit is the compressed model but that is also achievable by traditional pruning (Han et al.). 

4. The ablation study should compare with smarter baselines than random regrowth/no redistribution. Clearly each layer needs different level of sparsity so no redistribution is of course not a competitive one. But there might exist other criterion (e.g, weight magnitudes) than momentum which gives good results. The ablation study did not demonstrate why using momentum is justified.

5. "For dense convolution algorithms, we estimate speedups as follows: If a convolutional channel consists entirely of zero-valued weights we can remove these channels from the computation without changing the outputs and obtain speedups." How is it possible that all weights associated with a channel are all pruned, in a sparse pruning setting? A channel typically has at least hundreds or even thousands of weights connected with it, so even with 95% sparsity it's extremely unlikely (consider 0.95^1000). The estimate of speedup might be flawed.

In summary, those serious issues with experiments make me vote a rejection for the paper.

[1] Rethinking the Value of Network Pruning, ICLR 2019.

------------------------------------------------------------------------------------
After author rebuttal:

Thank you for the response.

The authors view that their method is not comparable with Han et al. 15. because this work started training a sparse model while Han et al. 15 trains a dense model and then do pruning. But I'm not sure whether the rather limited actual (1.2-1.3x, and lower on Wide ResNet) speedup justify this argument. What's more, these speedup were said to be "estimated" so I'm not sure whether it's actual speedup. 

The pruning ratio with dense performance is not remarkable: 50% sparsity ratio should be easily achievable with Han et al.

Experiments on ResNet (not wide ResNet) were not provided and I assume ResNet would be harder to sparsify than AlexNet, VGG, Wide ResNet which may have more redundancy. I think thus the "faster" training part in the title is not entirely justified. 

I'm ok with other parts of the rebuttal and I will raise my score to weak reject. But I still wonder why we should use sparse training in the first place if it brings so little "estimated" speedup.

review_title: Official Blind Review #3

Meta-review: This paper presents a method for training sparse neural networks that also provides a speedup during training, in contrast to methods for training sparse networks which train dense networks (at normal speed) and then prune weights.

The method provides modest theoretical speedups during training, never measured in wallclock time.   The authors improved their paper considerably in response to the reviews.  I would be inclined to accept this paper despite not being a big win empirically, however a couple points of sloppiness pointed out (and maintained post-rebuttal) by R1 tip the balance to reject, in my opinion.  Specifically: 

1) "I do not agree that keeping the learning rate fixed across methods is the right approach."  This seems like a major problem with the experiments to me.

2) "I would request the authors to slightly rewrite certain parts of their paper so as not to imply that momentum decreases the variance of the gradients in general."  I agree.
Decision: Reject