Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: In this paper the authors propose a new model compression method based on subdifferential inclusion. The key idea is to make the outputs of the neurons in the sparse and dense networks at the same input close enough. They rewrite the activation function as the proximity operator of a proper convex function and finally formulate the compression problem into a constraint minimization problem using the technique of subdifferential inclusion. They conduct a series of experiments to evaluate the performance of their proposed methods.

Positive aspects:
1.	The idea of this paper makes sense. 
2.	The experiment results show that the proposed method can achieve better performance than the baselines under this paper’s experimental setting. 
3.	This paper is well written and easy to follow. 

My concerns are:

1. In the proposed model, we need to choose proper value of $\eta$ for each layer, which is the required accuracy of the neuron’ output after compression. I understand that as reported by the authors in this paper, only in few experiments, they need to search for a good $\eta$. However, I think it is non-trivial to find proper value for $\eta$. I mean that different layers could have different tolerances on accuracy.  Since our goal is to achieve high test accuracy in the compressed network instead of the accuracy of the neuron’ output after compression, if we can find better values for $\eta$ we could achieve higher test accuracy. In other words, as it is challenging to find  near optimal $\eta$ for each layer, we could not reduce the network into very small size. 

2. Because of my above concern, I recommend the authors to give more results of compressing networks into much smaller sizes. For example, in RigL, the size of ResNet50 on ImageNet is compressed by more than 97%.

review_title: In this paper, the authors propose a new model compression method based on subdifferential inclusion. 

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper propose a network compression algorithm by exploiting a reformulation of activation function as proximity operator. The latter is an optimization problem whose optimality condition reveals constraints on the weight matrix W of the neural net. The main idea is then to "biasedly" select W as a minimizer of a sparsity inducing penalties under a relaxation of the previous optimality conditions. The authors provide details on solving such problem as well as numerical experiments that leads to similar results than competitors.

- The proposed algorithm does not significantly improves the accuracy of estimators (eg convNet in Cifar) when compared with actual methods.

- The claim in the conclusion "SIS is reliable in term of convergence guarantee" is not supported by clear evidence. I did not find any such convergence proof in the paper. Once the SIS compression is used, it unclear that the same accuracy than the non-compressed NN is preserved.

- The core optimization problem (7) is solved approximately with Douglas-Rachford iterations. Neither the effect of the optimization error nor the selection of eta is clearly discussed.

- Should be nice if the authors can provide a pseudo-code of the overall SIS strategy in a practical deep neural net (not only one layer). As stated, the main idea is lost in the technical details for solving (7) (where there is few or no new contribution).
review_title: The paper propose a compression scheme for NN that does not achieve SOTA but is an appealing competitors according the presented numerical experiments. 

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: ## Summary
The authors pose sparsification as a subdifferential inclusion problem, a novel formulation that results in quite meaningful results on established benchmarks/tasks. The paper overall is very well-written with a detailed overview of current sparsification techniques and how the proposed method differs.

## Pros
* Very comprehensive analysis and proofs (which seem correct, although not thoroughly verified)
* Empirical results justify this novel approach across the board

## Suggestions
The computational characteristics of using SIS has not been characterized in the manuscript; it is no very clear what the complexity of training a large model is using the proposed approach. The authors suggest their training approach is efficient, but do not provide any empirical results or further justification. For example, all of the results in Table 3 and Table 4 can have an additional column that characterizes the time to train. 
review_title: Highly impactful contribution with end-to-end results and analysis

final_score: rating: 7: Good paper, accept
review_content: Summary:

The authors propose a new algorithm for inducing sparsity in the weights of neural networks after training. The proposed algorithm exploits the properties of commonly used activation functions to cast the sparsification problem as the minimization of a sparsity measure subject to approximation accuracy constraints. The proposed problem can be solved using convex optimization.

Pros:

The authors’ insights about popular activation functions and the approach used to cast the sparsification problem as a convex optimization problem are clever and interesting. The authors presented experiments on a wide range of deep learning models and tasks.

Cons:

Some of the details of the authors experiments are not clear or potentially misleading:
1. Some results presented for existing techniques are from those techniques’ original papers, but some results were re-run by the authors. For example, consider the ResNet-50 results on ImageNet (table 3, left). The RigL authors did not present results at 60% sparsity, and Appendix D does not include details on how this number was generated beyond the authors using the released code with RigL. The numbers at 80% and 90% sparsity are taken from the RigL paper. However, these numbers were achieved with 5x the number of training steps which was enabled by the reduced number of FLOPs used by RigL during training because it maintains a constant level of sparsity throughout the training process. They also use non-uniform distributions of sparsity across the layers of the network which affects the number of FLOPs in the resultant network. The author’s of this paper report a lower top-1 accuracy at 60% sparsity than the RigL paper reports at 80% sparsity, which leads me to believe that the training conditions (time, sparsity distribution) are not the same. Similarly, all of the RigL results for MobileNet family models (table 3, right) appear to have been run by the authors and the training setup details are not clear. For these results generated by the authors of this paper, they should also detail the amount of hyperparameter tuning performed for these baseline, as this can make a large difference in accuracy. I focused here on RigL because it appears to be the most commonly used baseline by the authors of this paper, but it seems likely that these observations apply to other techniques as well.
2. Using RigL as the “state-of-the-art” baseline for most comparisons is not entirely fair given it has additional capabilities (i.e., the ability to enable sparse training by maintaining a constant number of parameters across training) compared to the authors’ proposed post-training sparsification algorithm. Sparse training (i.e., sparse-to-sparse training) is known to be a more difficult problem than dense-to-sparse training [1] or post-training sparsification. It is good to include RigL for comparison, but this distinction should be made clear and other techniques that have comparable ability to the proposed technique should be included as well.

My suggestion to the authors are the following:
1. All results should be reported as accuracy with a given parameter count and accuracy with a given FLOP count. Ideally, these tradeoff curves should be plotted across a range of accuracies and FLOP counts. This helps to avoid many of the pitfalls in the comparisons of model compression approaches details by [2] and [3].
2. Make it clear that some algorithms under comparison have additional capabilities compared to the proposed approach (e.g., RigL with sparse training).
3. Add comparisons with other algorithms of similar capability to the proposed approach. The magnitude pruning approach of Zhu & Gupta [1] would be ideal for this I believe.

Comments:

The brackets are backwards in the last paragraph on page 4. I would encourage the authors to explain more of the background of their approach (proximal operators, convex optimization, etc.) in sections 3 and 4. Many of those working in model compression who would be interested in this work will not be familiar with these topics.

References:

1. https://arxiv.org/abs/1710.01878
2. https://arxiv.org/abs/1902.09574
3. https://arxiv.org/abs/2003.03033 

[original score: 3 (clear rejection)]
11/24: Updated score based on updates from the authors. The addition of FLOP counts and more baselines in the experiments section greatly improved the paper. The proposed approach appears to achieve excellent FLOP-accuracy tradeoffs relative to existing approaches.

[2nd score: 6 (marginal acceptance)]
11/30: Updated score based on updates from the authors. The discrepancy with some baseline numbers has been resolved and the authors added clarifying information to the paper regarding the counting of FLOPs.
review_title: Interesting technique, but experiments need significant improvement

Meta-review: I have serious concerns about how experiments are reported in this paper. Most methods tried to compare at an iteration complexity of roughly 100 epochs because it is known more computation improves performance very significantly but the computational resources are limited for many researchers, especially in academia. While this convention may not be the ideal way to compare different methods, for fairness, this practice has been followed in most of previous papers. 

Unfortunately this paper disregarded this practice, and on Imagenet the reported results from previous works were mixed at 100 epochs (e.g. STR) and at 500 epochs (rigL — which was explicitly marked to be 5x in the original paper) without any clarification, and the only other method in the table showing comparable performance to the proposed method, LRR,  also requires many more than 100 epochs. Moreover, the authors did not explicitly disclose the equivalent epochs of their algorithms in the Imagenet experiments, and this is not acceptable. Based on the information inferred from the current writing, it is extremely likely that significant unfair advantages were given to the proposed algorithms. 

Since the authors did not report experiments appropriately,  this paper cannot be accepted in its current form regardless of other potential merits of the proposed methods. I hope the authors view this outcome positively, and proactively fix the problem. If in revised versions, the experiments are reported according to the common practice, I am sure the work would become publishable.

Decision: Reject