Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: The paper focuses on sparse neural networks. Typically, l1 regularization is the go-to strategy, however, it is not scale invariant. That is, all weights are affected by the regularization, not only those that are being driven to 0. l0 regularization is theoretically optimal, however, it is not smooth and has no gradients almost everywhere, so it cannot be used for training. As a compromise the paper proposes Hoyer regularization, that is the l1/l2 ratio. The Hoyer regularization has the same minima structure and leads to sparse solutions while being scale invariant, that is it does not affect all weights in the process. Additionally, the paper proposes structured Hoyer regularization. Last, it employs the said regularizations in deep networks: LeNet, AlexNet and ResNet on several datasets: MNIST, CIFAR, ImageNet.

Strengths:
+ The described method is simple, intuitive and straightforward. By applying the said regularization (~ Σ |w_i|/sqrt(Σ w_i^2)), one arrives at seemingly sparser solutions, which is verified in practice.

+ The experiments are extensive and convincing. I particularly like that the authors have used their method with complex and deep models like ResNets, on large scale datasets like ImageNet.

+ The presentation is generally clear and one can understand the paper straightaway.

Weaknesses:
+ The contributions of the paper are rather on the thin side. At the end of the day, Hoyer regularization is taken from another field (compressed sensing) and applied on deep networks. This is also witnessed by some moderate repetition in the writing, e.g., between the introduction and the related work.

+ There are some points where the paper becomes unclear. For instance, in Figure 3 what are the "other methods"?

+ In Figure 1 it is explained that the Hoyer regularization leads to minima along the axis. The gradients then push the models "rotationally". Could this lead to bad multiple local optimal problems? Is there any guarantee that any particular axis will generate better solutions than the other?

All in all, I would recommend for now weak accept. I find the work interesting and solid, although not that exciting.
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: To enforce sparsity in neural networks, the paper proposes a scale-invariant regularizer (DeepHoyer) inspired by the Hoyer measure. It is simply the ratio between l1 and l2 norm, which is almost everywhere differentiable, and enforces element-wise sparsity. It further proposes the Hoyer measure to quantify sparsity and applies the DeepHoyer in DNN training to train pruned models. The extension of Hoyer-Square is also straightforward. 

I generally enjoy simple yet effective ideas. The idea is very straightforward and well intuitive. The paper is well written and easy the follow. The discussion on the Hoyer measure is inspiring and the empirical studies on various different network architecture/datasets compared to several competitive baselines verify the effectiveness of the DeepHoyer model. 

Therefore I'm leaning to accept it. 
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: The paper is written very nicely and the experiments are convincing (though you can always show more)

In terms of novelty, I shamelessly can say the idea is very simple and the basic form, the Hoyer regularization, was known. That said, I am all in for simple and efficient solutions so I am giving this paper a weak accept for now.

There is not much to ask here (unless I say I want to see how this would work on other backbones and problems). nevertheless, to improve this work, I think the authors need to compare which solution (referring to algorithms in Table1, 2 etc.) is faster/more well-behaved given the combo explained at the bottom of page 5 . This is basically my question/request for the rebuttal.  
review_title: Official Blind Review #2

Meta-review: The authors propose a scale-invariant sparsity measure for deep networks. The experiments are extensive and convincing, according to reviewers. I recommend acceptance.
Decision: Accept (Poster)