Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: The authors propose cyclic precision training (CPT), a method to train integer-quantized neural networks with high precision while saving bit operations. CPT alternates the numerical precision of the network during training between low (2-3 bits) and high (the final desired precision, e.g. 8 bits). A total of 32 cycles of low to high are used, and this is only applied to the weights and activations, the backward pass is always in high precision. CPT is able to achieve improved accuracy across a variety of models on CIFAR-10/100 as well as transformers on PTB and WikiText. On ImageNet, CPT achieves accuracy on part with regular quantized training but saves bit ops.

The authors include a large number of other interesting data to show that CPT is insensitive to hyperparameters. Regardless of what the low precision setting, CPT(a,b) achieves better accuracy than b-bit quantized training on CIFAR-100. Varying the number of cycles between 8 and 128  affects the results by at most 0.5%. These experiments and the loss visualizations show that CPT is robust on CIFAR.

The major thing holding the paper back for me is that the lack of a good theoretical explanation for the CPT phenomenon. The authors draw a parallel between cyclic learning rates and CPT, but the paper itself shows that cyclic LR is not effective with quantization. I believe that lower accuracy means higher noise which resembles a higher LR, but clearly the two are not equivalent. One other idea is that lower accuracy may resemble lower batch size (higher SGD noise), but not sure what that would mean.

In general, the paper presents overwhelming evidence that CPT is a real phenomenon which may help train quantized DNNs. Even though the mechanism is currently not well understood, the paper deserves to be seen by others.

Questions:
 - I assume the CNNs are all trained with SGD. Was the transformer trained with SGD or ADAM or another optimizer? 
 - What if you just trained in low precision until say, epoch 80, then use CPT? Does cyclic precision have to be used throughout training or just near the end?

EDIT: I managed to set this up on a small transformer and was able to replicate the paper results. The transformer was trained with ADAM. Definite accept from me.
review_title: Interesting empirical paper, unsure about theoretical explanation

final_score: rating: 7: Good paper, accept
review_content: ### Overview
In this paper, the authors proposed Cyclic Precision Training (CPT) for low precision training. CPT varies the precision between two boundary values cyclically during training. It improves the accuracy, converges to a wider minima, reduces training variance, and reduces the training cost.

### Clarity
The paper is well written and clearly organized. The figures and tables are informative and clearly organized.

### Pros
1. The paper provides an in-depth analysis of the advantages of varying precision. It shows the connection between precision and learning rate, and dynamic precision helps convergence to a better minima, verified by the visualization.
2. The proposed CPT is effective for improving training convergence and reducing training cost. As shown in Figure 5, the accuracy v.s. training cost trade-off is better compared to static low-precision training. CPT even slightly improves the final accuracy.
3. The author proposed an automatic method to determine the boundaries for cyclic precision at negligible overhead.
4. The experiments are solid. The advantage of CPT is consistent over different models and tasks. Ablation study shows the robustness of CPT w.r.t. different hyper-parameters.

### Cons
1. The results of CPT is impressive. However, the actual application scenario could be quite restricted at this time. Most of the hardware does not support flexible bit-precision like 3-8 in this paper. Supporting multiple bit-precision will require extra compute units, leading to higher cost and energy consumption. It is unclear if the trade-off is worthwhile. Using FPGA can support flexible bit like in this paper, but using ASIC supporting int8 training (like GPU) might be more efficient compared to the improvement from CPT (I think generally FPGA is not more efficient than GPU for NN training). 
2. The precision range test is not explained clearly. Is the threshold consistent for all experiments? In figure 4, the training accuracy for the switching point on two curves are not the same. Does it indicate we need to manually tune the threshold for each experiment?
3. The authors claim that CPT improves convergence for quantization training. Since the hypothesis is that the precision may work similarly to the learning rate, a baseline would be training with fixed precision (just like in this paper) while using cyclic *learning rate*.  Can CPT also outperform this baseline?

EDIT: The response is very helpful and it addresses my concerns. I raised my score to 7 after reading the response.
review_title: Review for CPT: Efficient Deep Neural Network Training via Cyclic Precision

final_score: rating: 7: Good paper, accept
review_content: The authors proposed an interesting low-precision training method using a dynamic precision schedule. Their proposed Cyclic Precision Training (CPT) cyclically varies the precision during the training and the boundary of precision values is determined by a precision range test (PRT). As shown in their empirical results, CPT largely reduces time/energy costs during training while maintaining comparable accuracy. While CPT does show a promising empirical performance, the motivation and reason behind the proposed method require further explanations. 
Specifically,

1. Motivation hypothesis in section 3.1 lacks explanation and justification. Authors point out that dynamic learning rate schedules help to improve the training and DNNs learn to fit different patterns at different training stages, but there is no strong connection to why dynamic learning rate could relate to dynamic precision schedule. 

2. Visualization examples aiming for supporting the hypothesis are not strong enough. The progressive precision case only achieves slightly higher accuracy and such benefit may be diminished given a large number of repeated experiments. Therefore I don't fully agree authors' statement "progressive precision schedule helps to converge to better local minima with wider contours, indicating a lower generalization error".

3. Authors do point out that low-precision with large quantization noise may have a similar effect as a high learning rate with large stochastic noise, which helps the exploration over loss surface during DNN training. This hypothesis is interesting and it would be better to support the hypothesis with empirical validation and theoretical justification.

In addition, I think the proposed precision range test (PRT) method is unfair and requires modification.  PRT starts from the lowest possible precision and gradually increases the precision while monitoring the difference in the training accuracy magnitude averaged over several consecutive iterations. However, such a difference could also be affected by the training stages at different iterations. A better test should be evaluating all precision settings from the same starting point (e.g. training from scratch), and then measuring the differences after reaching the convergence stage.  

The paper is mostly well-written. Section 4.4 seems redundant and should be put before the experimental section. The connections are also required to be explained in detail.
review_title: cyclic precision is very interesting but the paper requires more explanations and motivations behind the idea

final_score: rating: 7: Good paper, accept
review_content: A simple yet apparently effective technique. The paper is clear and well written.

The authors demonstrate that cyclically changing the precision of weights and activations during training leads to better results (both accuracy and training cost) than static quantisation methods. The method is simple and easy to implement . The  authors provide a method to automatically select the method's hyperparameters.

1. This work appears to be an another example of multiprecision training (with a precision-switching mechanism): https://arxiv.org/abs/2006.09049

2. The abstract claims that the variance of accuracy decreases when using CPT. This is briefly mentioned around Figure 7, but I would also expect to see these figures available for comparison in Tables 1-3, and some more discussion.

3. The original hypothesis was that cyclic precision would have a similar effect to cyclic learning rate, but there's little discussion of this in the paper. There's one paragraph in Section 4.2 which compares CPT against CLR.  Could both techniques be applied together?

Minor

* Table 1: Apologies if I am missing something obvious, it's unclear to me what "GBitOPs" and "latency" refer to. It appears that two separate models have been trained, one for CIFAR-10 and one for CIFAR-100, but only one GBitOPs/latency figure is reported?

* I find it quite hard to parse paragraph 2 of section 4.2 (Benchmark over CLR on top of SBM....)

* Section 4.2, paragraph 4, "staic" should be "static"

* There are several capitalisation issues in the references 

EDIT: Thanks to the authors' for their clear responses, I'm happy to raise my score to a 7.
review_title: technique to cyclically vary precision during training

Meta-review: All of the reviewers are impressed by this paper's empirical results and they agree that this is a good paper and should be accepted. Some questions about the theoretical justification of the proposed method and its potential practical impact remain open, but the empirical results are impressive and can result in more research in understanding Cyclic Precision Training (CPT) and improving quantized training of neural nets. I suggest acceptance as a spotlight presentation.
Decision: Accept (Spotlight)