Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: Overall an interesting paper, though I wished a more detailed presentation of the reasoning behind the algorithm would have been provided. As it stands it feels a bit heuristic. 

In particular I don't understand the motivation between the switching mechanism. Basically it says if the gradients are co-aligned between epochs it means there is not much to learn anymore!? Why? Intuitively if the gradients would go to 0 or become very small maybe you would want to increase precision. Or if you have high variance you could argue that the expected gradient would be 0 and hence you are not really making progress, i.e. you are just moving left-right. But if all gradients agree on a moving direction, why is that a bad thing? I know the heuristic is borrowed from a different work, but since it feels as such an integral part of MuPPET I think you should explain it better. 

I guess a few details about the algorithm as well. When you say you look at the diversity of the gradients over the epochs, is this the batch gradient !? 

There are some small typos (e.g. FP23 instead FP32). 

I find the justification for AlexNet to be adhoc (it switched at the wrong time, but that allowed to take more advantage of computation in the low precision hence it was faster). The switching mechanism should only care of when the gradients are not informative anymore, not how much compute you are wasting .
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: Summary:
This paper proposes a training strategy called Multi-Precision Policy Enforced Training(MUPPET). This strategy aims to reduce training time by low-precision data representation and computations during the training stage. According to the gradient diversity, the authors introduce a precision-switching mechanism which chooses the best epoch to increase the precision. The validation accuracy and training time across several networks and datasets are shown in the experiments. However, the results are not superior enough compared with the state-of-the-art.

My detailed comments are as follows.


Positive points: 

1. This paper proposes a new reduced-precision training scheme to speed up training by progressively increasing the precision of computations from 8-bit fixed-point to 32-bit floating-point. This scheme moves to reduced-precision fixed-point computations while updating an FP32 model in order to push the boundaries of reduced-precision training. 

2. The authors propose a metric to decide when to switch the precision inspired by gradient diversity introduced by [1]. In this paper, the gradient diversity is enhanced by considering gradients across epochs instead of mini-batches. The proposed metric can be seen as a proxy for the amount of new information gained in each training step. Therefore, the metric can decide the most appropriate epoch at run time to increase the precision.
 
3. The proposed low-precision CNN training scheme is orthogonal and complementary to existing low-precision training techniques.




Negative points:

1. The proposed approach does not match the description in this paper. The authors describe “This approach enables the design of a policy that can decide at run time the most appropriate quantization level for the training process”. In fact, this approach just decides which epoch to increase the quantization level while the levels of quantized precisions are fixed, rather than deciding the most appropriate quantization level. 

2. The setting of quantized precision levels (8-, 12-, 14- and 16-bit precisions) is confusing. Please illustrate how to choose the number of quantized bit and the number of quantized precision levels.

3. The presentation of the precision switching policy is confusing and the notations are unclear. For example, in section 3.3, the ratio “p” needs more description because it is a key value in the policy, but lacks an explanation in this section. So please explain more about the motivation of ratio “p” in this section. 	In section 3.3, in step 5 of the proposed precision switching policy, the authors do not explain the meaning of “y”.

4. In figure 2, the precision switch is not triggered even though the value of p violates the threshold more than 2 times, which mismatches the description in section 3.3.

5. The proposed strategy has no obvious advantages. There are some scenes that the proposed strategy does not perform well. For example, the Top-1 validation accuracy on ImageNet of AlexNet and ResNet with MuPPET strategy is much lower than FP32 baseline. Compared with [2], the proposed method is more complex but not superior enough.

6. The authors do not show the training and validation curves. However, the training and validation curves are common used to show more details of the training process, such as in [2] and [3]. Please show and analyze the training and validation curves of the proposed scheme and the baseline.


Minor issues:
Some spelling and grammar mistakes.


Reference：
[1] Dong Yin, Ashwin Pananjady, Max Lam, Dimitris Papailiopoulos, Kannan Ramchandran, and Peter Bartlett. Gradient Diversity: a Key Ingredient for Scalable Distributed Learning. In 21st International Conference on Artificial Intelligence and StatiZZstics (AISTATS), pp. 1998–2007, 2018.
[2] Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. Mixed Precision Training. In International Conference on Learning Representations (ICLR), 2018.
[3]  Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, and Pritish Narayanan. Deep Learning with Limited Numerical Precision. In 32nd International Conference on Machine Learning (ICML), pp. 1737–1746, 2015.                                                                                                                                                                                                                                                                                                        

review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: The article presents an approach to reduce the precision of weights, activations and gradients to speed up the training of deep neural networks. The precision of these values is increased according to a dynamic schedule such that the original classification accuracy is reached after training.

The manuscript is in most parts well written and the addressed topic is of general interest for the research community represented at ICRL. Still, I recommend a weak reject, since the core idea of the manuscript, i.e. the dynamic switching between precision levels, is not shown to be a necessary condition for good classification results.


Major points:
•	The introduction does not give a clear statement about the novel contribution of the paper. Only the very last paragraph is specific about the paper.
•	Your results support that step-wise increasing the resolution speeds up training without significant losses in accuracy. However, the impact of the gradient diversity, choice of p and threshold parameters on the performance of the trained networks are unclear. What is the isolated impact of every of these choices? According to Figure 2, pre-defined switching points between precision levels may also generalize between networks and datasets.
•	The description of the quantization scheme is not clear enough in order to reproduce the results:
o	Please give details about every step from FP32 to FPx values or cite appropriate literature.
o	Equation 4 and 5: How are the scaling factors SC determined?
o	Please clarify the difference/relation between n and WL.


Minor points:
•	Equation 3: What does “represent. range(q^i)” mean?
•	Text in Figure 1 and 2 is far too small and barely readable
•	Step 5 in Algorithm in Section 3.3: What does “p violates y more than gamma times” mean? What is y?
•	Please clarify “distribution approach”. Distribution of what?
•	Table 1: For the baseline experiments, the precision is switched from 8 to 32 bits, for MuPPET from 8 to 12 bits (see main text). What is the motivation behind these different choices?
•	Do you use any type of data augmentation?
•	Table 3: Please clarify “theoretical limit”. Does this limit include 12 and 14 bit quantisation. What do you mean by “optimized quantization implementation” in main text?
review_title: Official Blind Review #1

Meta-review: The submission presents an approach to speed up network training time by using lower precision representations and computation to begin with and then dynamically increasing the precision from 8 to 32 bits over the course of training. The results show that the same accuracy can be obtained while achieving a moderate speed up. 

The reviewers were agreed that the paper did not offer a signficant advantage or novelty, and that the method was somewhat ad hoc and unclear. Unfortunately, the authors' rebuttal did not clarify all of these points, and the recommendation after discussion is for rejection. 
Decision: Reject