Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: The authors propose a hardware-agnostic metric called effective signal norm (ESN) to measure the computational cost of convolutional neural networks. This metric aims to fairly measure the effects of pruning and quantization. What’s more, based on the metric, the authors demonstrate that models with fewer parameters achieve far better accuracy after quantization. A large number of experiments are carried out to prove the effects of the metric and related conclusions, however, several experiments and arguments are confusing. 

Please see my detailed comments below.

Positive Points:
1. The authors propose a hardware-agnostic metric named effective signal norm (ESN), aiming to measure the computational cost of the pruned and quantized models.

2. Based on the proposed metric, the authors conclude that a moderately quantized slim model with fewer weight parameters achieves better performance rather than an extremely quantized fat model with huge number of parameters.

3. This paper presents many interesting assumptions and possibilities, which can be further researched and explored in the future.


Negative Points:
1. The authors argue that ESN is a hardware-agnostic metric. However, ESN is based on ideal hardware, where the energy consumption is linearly proportional to the number of non-zero weight parameters and monotonically depends on the bit-width of weight parameters. Therefore, ESN is not suitable for existing hardware.

2. Both of ESN_a and ESN_d are based on the assumptions instead of the fact, which is not convincing. What’s more, the assumptions are hard to be proved.

3. The experiments are not strong enough to support the author's conclusions. For example, the authors argue that “models with fewer parameters achieve far better accuracy in low computational cost region after quantization”. However, this conclusion is only based on the ESN metric. Since ESN is based on some assumptions, this conclusion is not convincing.

4. Please give more details about the quantitative relation between the quantization noise and the perturbation of weight parameters during SGD training.

5. In terms of Figure 2, the author's description is not objective. The similarity between the ESN_a curves and the validation accuracy curves is not very high. Besides, the authors mention that “after steep rise at 80 epoch, both ESN_a and validation accuracy decrease gradually”. However, the performance degradation in Figure 2 is not obvious. Therefore, the conclusions drew by the observation are also unreliable.

6. The ESN can not make an accurate evaluation of a model. The trade-off between accuracy, speed, model size is often required in model compression. However, the ESN can not make the trade-off between them. Moreover, the ESN does not have a clear application scenario.

Minor issues:
1.	There are lots of spelling and grammar mistakes in the paper, such as “an ideal hardware ”, “a extremely quantized fat model” and so on.

review_title: Official Blind Review #4

final_score: rating: 3: Weak Reject
review_content: This paper proposes an effective signal norm metric that measures the cost of the neural networks under both compute(ESN_a) or memory(ESN_d) in an ideal hardware setting. The authors then show that the slimmer models with fewer parameters are better than fatter models.

Strength

First of all, I like the discussion about different hardware(CPU) setups and corresponding cost. I also like the usage of pareto frontier to compare experimental methods. The proposed metric was simple but the author justifies it with respect to possible hardware setups in an ideal setting.

Weakness

The main drawback of the paper is the lack of novelty in proposed method. The metric itself appeared to be the main contribution, but as the author said, the metric was based on an ideal hardware setup that ignores the memory hierarchy and data transfer cost, which could be the bottleneck in reality.

More importantly, it is hard to use the empirical evaluation to justify the conclusion “fatter models are worse”. As we know that extremely low bit quantized models are very hard to train, and it is not surprising -- by setting the number of bits to extremely low bits, the models cannot recover in a few rounds of re-training. 

This being said, I cannot think of a better alternative. While I do not think the experimental results offers new insights(other than it is hard to re-train lower bits models). How to quickly train low bits models remains an open question.

Finally given that there are already quite a few methods that prunes model based on real hardware evaluations, it would be great to compare to these methods as well.

Because the above weakness of the paper, I think this paper should not be accepted to the program.


review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: The paper proposes a new metric to evaluate both the amount of pruning and quantization. This metric is agnostic to the hardware architecture and is simply obtained by computing the Frobenius norm of some point-wise transformation of the quantized weights. They first show empirically that this Evaluation metric is correlated with the validation accuracy. Then use this metric to provide some general rules for pruning/quantizing to preserve the highest validation accuracy. Finally, they derive a strategy to perform pruning by monitory the signal to noise ratio during training and show experimentally that such method performs better than competing ones.

Pros: - Extensive experiments were performed to test the methods, and the results seem promising.

Cons: - The paper is not very clear, and the structure is somehow confusing.
- It is not easy at first to understand the experimental setup and requires to make a lot of guesses in my opinion. 
- The paper didn't motivate properly the use of a hardware-agnostic metric in the context of the quantization and pruning. Isn't the ultimate goal of pruning/quantization is to optimize the run time/energy consumption of the specific device with the least compromise on the accuracy?

I feel that the paper currently jumps between very different ideas: 
	- Evolution of the proposed metric during training: 2.3 and 2.5. While the 2.3, the take-home message is relatively clear:  the ESN is correlated with the validation accuracy, I don't fully get the point of section 2.5: It suggests that the optimizer does some sort of pruning just by choosing a higher learning rate.   
	- Finding an optimal strategy for pruning/quantizing a network: 2.4 and 2.6. Those two sections are relatively clear, although I have some questions about the experiments.
	- Developing a new strategy based on the proposed metric to quantize and prune a network in a Pareto optimal sense: This is briefly and not very well explained in section 2.7, which sends back to 2.3, but it is hard to understand how it is exactly done. It seems that section 3 provides some empirical evidence supporting this strategy, but the description of the method is hidden in the experimental details.


Some questions: 
- In figure 3, the blue dots represent validation vs ESN at each training iteration? What about the red plot, is it obtained by quantization of the parameters at different stages of training, or is it using the final parameters? Which equation was used to compute the red curve (2) or (4)?  How much quantization was performed? If the quantization was chosen to match the level of noise then it seems natural to expect such behavior in figure 3.  

- In figure 4, how much pruning was performed for each network and was it the same quantization? In other words how each point in the plot was obtained?  The authors come to the conclusion that one should 'prune until the limit of the desired accuracy and then quantize', but it is hard for me to reach the same conclusions as I don't see the separate effect of pruning and quantization in those figures. Or maybe pruning is implicitly done by choosing a small network? In this case, it makes more sense, but still, some clarifications are needed.

- Which equation for the ESN was used to produce figure 5? Equation (2) or (4)? 

- What is the Pareto frontier? I think it is worth first introducing this concept and describing more precisely how those curves are obtained. For someone who is not very familiar with these concepts, which is my case, it makes the reading very hard. 

- How was the number of pruned filters computed in figure 5 (right)? I don't expect the solutions to be sparse during training, especially that no sparsity constraint was imposed, or was it?
	


-------------------------------------------------------------------------------
Revision:

Thank you for all the clarifications and the effort to make provide a clearer version of the paper. 

Regarding section 2.7: ESNa FOR QUANTIZATION: Would it make sense to include the paragraph 2.7 at the end of 2.3, since it related to it and doesn't seem to require any of the intermediate subsections.

response to Comment 3: Unfortunately, I'm not convinced by the explanation about the effect of the lr on sparsity. The decay coefficient controls the saparsity indeed, but not the lr. That is because unlike the lr, the decay coefficient defines the cost functions to be optimized:  L+dc ||W||^2, while the lr corresponds simply to the discretization of some gradient flow.   For instance, in a deterministic and convex setting, the solution that is obtained would be the same, regardless of the chosen lr  ( provided the lr is smal enough so that the algorithm converges) see for instance [1].  In a non-convex and stochastic setting do the authors have a particular reference in mind? I'm not aware of such behavior. I would expect a similar sparsity if dc is kept fixed and only lr changed. Is it likely that with smaller lr, the algorithm just didn't have time to converge? This would explain why the obtained solutions were less sparse.
response to answer 5:  it is indeed well known that L1 norm induces sparsity, however l2 doesn't, it just encourages the weights to be smaller. In the optimization litterature sparsity of x% means x% of the parameters  are exactly 0. This is achieved with l1 norm, however l2 norm would only enforce that the coefficients are small but not necessarily 0 (see [1])
[1]:  ROBERT TIBSHIRANI, Regression Shrinkage and Selection via the Lasso.

Although the paper improved in terms of clarifications and experimental details, I still think it will benefit from additional work on careful interpretation of the results.











review_title: Official Blind Review #2

Meta-review: The authors propose a hardware-agnostic metric called effective signal norm (ESN) to measure the computational cost of convolutional neural networks. They then demonstrate that models with fewer parameters achieve far better accuracy after quantization. The main novelty is on the metric ESN. However, ESN is based on ideal hardware, and thus not suitable for existing hardware. Assumptions made in the paper are hard to be proved. Experimental results are not convincing, and related pruning methods are not compared. Finally, the paper is not written clearly, and the structure and some arguments are confusing.
Decision: Reject