Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This proposes two techniques to replace mixed-precision arithmetic with half-precision training for a large part of the training process. In the first approach, the authors simply switch all mixed-precision operations with half-precision operations, and can achieve performances slightly lower than SOTA. In the second approach, the authors propose to dynamically switch between mixed-operations and half-precision operations during training. The authors claim that this second approach can match SOTA results while using half-precision arithmetic for more than 94% of training.

Overall the paper is fairly well written, and easy to follow. The proposed techniques seem to empirically work well. However, I have a number of concerns about the paper, which explains my score. I list these concerns below.

1. The proposed approach has a number of additional hyperparameters, which makes it less likely for the algorithm to be widely used if the algorithm is very sensitive to the values of this. For very extreme values of these hyperparameters, I would expect the algorithm to start behaving quite poorly. But it would help a lot to provide some sensitivity analysis to these hyperparameters for reasonable values of these hyperparameters.

2. How much do the optimal hyperparameters (like numBatchesMP, numBatchesBF16, emaT) vary across problems?

3. How much do the above-mentioned optimal hyperparameters vary with mini batch size?

4. How are the other hyperparameters like learning rates selected? Is the learning rate tuned?

5. Are the experiments repeated multiple times?

6. It seems a bit weird to call a modification that simply uses half-precision arithmetic for most FMA operations a significant contribution of the paper, especially since it can't reach SOTA performance.

7. Algorithm 1 should be written out in a better way that shows the training loop. It is slightly confusing the way it is written up right now.

Overall I think the paper would significantly benefit from a more thorough empirical evaluation.

=============================

Edit after rebuttal:
I thank the reviewers for their response and for running the additional experiments. However, I find the updated version of the paper to be inadequate in fully answering my concerns. While the authors have included a hyperparameter sensititivity analysis, I find the experiment to be unconvincing. Only two of the three hyperparameters are swept over a very small range of values, and the results presented are only for the first 12 epochs, while the actual model is typically trained for 90 epochs. While I appreciate the added experiment and realize that 10 days is too short a time to put in a proper sensitivity analysis, based on the current draft of the paper, I cannot recommend accepting this paper. I am however raising my score to a weak reject.

review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: In this paper, the authors propose approaches to accelerate deep neural network training with mixed-precision arithmetic.
Observed that relying purely on half-precision arithmetic results in lower accuracy, the authors developed a method 
to dynamically switch between mixed-precision arithmetic (MP) and half-precision arithmetic (BF16 FMA).
Empirical results show that the dynamic approach can achieve similar accuracy as MP and FP32 algorithms.

Although this paper shows the possibility to accelerate DNN training without great loss in performance, there are many issues with the paper itself. First, the title is ambiguous. The dynamic approach could mean a lot of things while training 
DNNs and one cannot tell what the paper is about simply relying on the title.
Also, the dynamic algorithm itself is not well presented. For example, how to choose the hyperparameters? What is the
overhead to switch between MP and BF16FMA? 

Apart from the algorithm itself, I also have questions regarding the experimental results.
Is there a reason why the performance of the half-precision arithmetic 
varies across different neural networks (Inception > Resnet)?
Specifically, what is the key factor that influences the sensitivity of a neural network towards precision?

Overall I think this paper should be improved in its experiments and presentation.


review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: The author(s) propose to accelerate the training of deep neural networks while also maintain the performance of the trained model by switching between fully half-precision computation and mixed-precision computation. Compared to the commonly-used mixed-precision training strategy, the proposed method can accelerate the training speed. Besides, on an image classification task, models trained by the proposed method achieve comparable performance with those trained by mix-precision training or full-precision training.

Strength:
1.	Section 3 provides useful information for the workloads of deep neural networks.

Weakness:
1.	The overall idea is not novel. The proposed method simply switches between two existing training strategies, i.e., the mixed-precision training and half-precision training. The claim that "this paper is the first in demonstrating that  half-precision can be used for a very large portion of DNNs training and still reach state-of-the-art accuracy" may not correct, in fact, Nvidia's apex has already supported using mixed-precision or entirely half-precision to train DNNs, and there is no clear evidence that the proposed method is better than theirs due to the lack of experiments on more tasks and datasets.


2.	From Table 1, the Dynamic strategy outperforms BF16 in terms of classification accuracy. However, from the experiments, it’s unable to tell that the gains actually come from this Dynamic strategy. Maybe similar gains can be obtained once the same amount of MP iterations are executed at any period of the training process. For example, consider a simpler strategy: first train the model with BF16 for K% of the total training iterations, then for the last (100-K)% iterations, train it with MP. K can be tuned so that the proportion of BF16FMA is close to those in Table 1. Models trained with this strategy might achieve similar performance with the proposed Dynamic strategy. 


3.	It’s hard to apply the proposed method in real applications, since BF16 is only supported by very few kinds of hardware.


4.	To demonstrate the effectiveness of the proposed method more clearly, it could be better to provide the exact proportion of reductions in terms of memory/computation/bandwidth in the experiments. 


5.	From Table 1, there still exists a large performance gap in terms of accuracy (1.56% for ResNet-50) between the model trained by the proposed method and the model trained by state-of-the-art MP.


6.	The organization of this paper can be improved. For example, Section 5.2 spends too much space introducing the emulation of BF16, which I think is not very relevant to the topic of this paper. And Figure 3 takes too much space.

review_title: Official Blind Review #2

Meta-review: The submission proposes a dynamic approach to training a neural net which switches between half and full-precision operations while maintaining the same classifier accuracy, resulting in a speed up in training time. Empirical results show the value of the approach, and the authors have added additional sensitivity analysis by sweeping over hyperparameters. 

The reviewers were concerned about the novelty of the approach as well as the robustness of the claims that accuracy can be maintained even in the accelerated, dynamic regime. After discussion there were still concerns about the sensitivity analysis and the significance of the results.

The recommendation is to reject the paper at this time.
Decision: Reject