Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper proposes a pruning technique called RigL, which performs sparse initialization of the network weights, and allows the network to grow weights during training. The sparse initialization makes the pruning algorithm to be memory- and computation- efficient, unlike existing models that starts with dense network weights. The authors train RigL with three different sparsity distributions that considers the number of input and output nodes, and validate them on various deep convnets for training on ImageNet, on which the model outperforms several existing sparsification methods and even dense counterparts. 

Pros
- The proposed model, RigL, is memory- and computation- efficient, and thus allows to train a large network in an efficient manner.
 
- RigL obtains impressive sparsification performance, even yielding sparse networks that outperform their dense counterparts.

Cons
- The idea of starting from a small, sparse network and expanding it is not novel. DEN [Yoon et al. 18] proposed the same bottom-up approach with sparsely initialized networks, while they allowed to increase the number of neurons at each layer and focused more on continual learning. The authors should compare the two methods both conceptually and experimentally. 

- The method is more like a set of heuristics rather than a principled approach, which makes it less appealing. This is not really an issue if the paper includes extensive experimental validation and in-depth analysis, but this is not the case. 
 
- The experimental validation is largely lacking, as the authors only perform experiments on ImageNet and do not compare against recent state-of-the-art Bayesian sparsification methods (SBP, VIB, L0-regularization). Without such extensive experimental validation, it is uncertain whether the result will generalize, given the highly empirical nature of the work. 

In sum, although I believe that the paper proposes a very practical method that is easy to implement and is promising, due to lack of experimental validation against a similar approach, state-of-the-art sparsification methods, and results on more datasets, I temporarily provide the rating of weak reject. I may change my opinion if the authors provide those results during the rebuttal period.

[Yoon et al. 18] Lifelong learning with dynamically expandable networks, ICLR 2018
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: This paper proposes a method for training sparse network without first training a dense network (e.g. the lottery ticket hypothesis or distillation). The method involves a combination of dynamic pruning of weights coupled with a dynamic "growing" of new weights given a novel criterion based on the magnitude of the gradient of the loss. As a result, networks can stay sparse throughout training and testing, leading to a large reduction in computational cost.

The paper's approach of dynamically changing the topology of networks is an interesting and motivated idea that seems to work rather well. I also appreciate the experiments on MobileNet, a setting where one expects investigations into sparse network architectures to have significant application. Relatedly, I appreciate the importance of the fact that the computational cost of training and evaluating the network is proportional to the sparse model size, which is not normally true for masked dense models. Overall, I found the paper to be very clear and of high quality, and thus I find this to be an interest addition to investigations into the lottery ticket hypothesis.

As the authors state, the novelty of their method is that they use the gradients with the highest magnitudes to grow connections. This is somewhat intuitive given the role gradients play in gradient descent based optimization, but I was wondering if they had any further intuition as to why this is the right criterion?

In section 4.3, I was a bit confused by Figure 5. My understanding is that many paths between loss landscape minima follow nonlinear paths -- why is it at all significant that there's a linear barrier? Why are only quadric and cubic Bezier curves used, rather than a more general path finding algorithm?

Overall, this is a nice paper that should be accepted to ICLR.
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: Overview:

The paper is dedicated to developing a more efficient and powerful dense-to-sparse training method. In order to break the limits of the size of the largest trainable sparse model to that of the largest trainable dense model, the author proposes a dynamic method that updates the network topology via parameter magnitudes and infrequent gradient calculation. In the experiments parts, they conduct extensive studies to show the proposed approach can surpass the previous sota with ResNet-50, MobileNet v1 and v2 on the imagenet 2012. What's more, the author also provides some intuitive explanation about why allowing topology change during the optimization is beneficial. 

Strength Bullets:

1. Due to the dynamic network topology, the paper's methods exactly achieve the memory and computation efficient. i) Required memory is only proportional to the size of the sparse model. ii) The amount of computation is proportional to the number of nonzero parameters in the model.
2. The author performs detailed comparison experiments among different sparsity distribution and different pruning methods. And the results overcome the previous state-of-the-art results.
3. Fig 5 shows some interesting insight. It suggests that static sparse training may be stuck at some local minima which are isolated from improved solutions. However, the dynamic update has a big chance to avoid this problem.

Weakness Bullets:

1. The author claims that the ticket in the paper does not rely on a "lucky" initialization. But it doesn't exclude the possibility that starting from the original initial conditions may give a better performance. Even if the connection is dynamic, we can still record the initial point for each weight. It will be better the author can provide related analysis.
2. In my opinion, in order to prove dynamic pruning is better than static methods, the author needs to provide a comparison with the previous sota method in it's own setting. i.e. The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks

Recommendation:

I think this paper is a novel work. Although it has some flaws in the experiment design, the motivation and experiment results are conniving enough. So, this is a weak accept.
review_title: Official Blind Review #3

Decision: Reject