Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper outlines a new method that allows using a variety of precision in the numerical representation of the network to increase performance (both in terms of accuracy and speed). They learn a threshold value for which all activation values above the threshold are learned at full precision, while all below are learned at reduced precision. This enables substantial performance gains. 

The authors summary of the contributions made in the paper is accurate.

The paper is well written and clearly articulates a contribution to the literature. As such, I think it should be accepted. However, the major question I had as I read the paper was the efficacy on GPU, which the paper discusses, but does not implement, nor show any empirical results for, which weakens the paper. Most deep learning happens on GPUs (or similar accelerators), so until this technique is implemented there, it is of limited use. It is still a contribution to the literature, but the paper would be significantly strengthened with a GPU implementation. Additionally, the experimental evidence is lacking. More experiments would also strengthen the paper.

If these changes were made I would change my score to 8 (accept). I do think that the work is slightly premature, and would benefit significantly from adding GPU results and additional experiments. The contribution is strong, however, and should be published in some form, either now, or at a future date.

For the experimental setup, I had a few questions: 

1) What happens with fixed thresholds? E.g. doing a sweep over fixed values. 

2) How do the results vary for different initialization schemes? 

3) How do the results vary with the 5 hyperparameters listed? How were they chosen?

4) How consistent are the results? i.e. what happens if the experiments are repeated N times? Do we see the same values?

In short, I would like to see more experiments. The results are encouraging, but brief. Evaluating on more architectures would strengthen the paper.

Overall questions:

- How well does the 1.2% improvement in perplexity compare to SOTA? Please add context for the numbers reported. It's not at all clear how good of an improvement is seen. 
- How do the results change with top-5 accuracy vs top-1 accuracy? 


Notes which did not affect the review score:

- There are some typos, e.g. “PG computes most features in a low precision and only a small proportion of important features in a higher precision.” Saying “PG computes most features using reduced precision and only a small proportion of important features using high precision” would be more correct. There are similar typos throughout that I have not listed.
- Tables 3 & 4, and Figure 4, are very cramped and hard to read.
Table 1 and 2 are quite crowded; can you rearrange them so they’re easier to read?


review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: This paper introduces Precision Gating, a novel mechanism to quantize neural network activations to reduce the average bitwidth, resulting in networks with fewer bitwise operations. The idea is to have a learnable threshold Delta that determines if an output activation should be computed in high or low precision, determined by the most significant bits of the value. Assuming that high activations are more important, these are computed at higher precision.

I agree that the following three key contributions listed in the paper are (slightly re-formulated):
1. Introducing Precision Gating (PG), the first end-to-end trainable method that enables dual-precision execution of DNNs and is applicable to a wide variety of network architectures.
2. Precision gating enables DNN computation with a better average bitwidth to accuracy tradeoff than other state-ofthe-
art quantization methods. Combined with its lightweight gating logic, PG demonstrates the potential to reduce DNN execution costs in both commodity and dedicated hardware.
3. Unlike prior works that focus only on inference, precision gating achieves the same sparsity during back-propagation as forward propagation, which reduces the computational cost for both passes.

These contributions are novel and experimental evidence is provided for multiple networks and datasets. The paper is well-written and provides insightful figures to showcase the strengths of the present method. Related work is adequately cited. The paper does not contain much theory, but wherever possible equations are provided to illustrate in detail how the method works.

Experimental results are shown for the datasets CIFAR-10 with ResNet-18 and ShiftNet-20, and ImageNet with ShuffleNet V2 0.5x. On both datasets, PG outperforms uniform quantization, PACT, Fix-Threshold and SeerNet in terms of top-1 accuracy and average bitwidth. 
What I am missing is information about the variability of results, since there are no error bars. Are the results averaged over multiple trials (if yes how many?), and is there a difference in variance between the methods? I realize that adding standard deviations to all results in the tables might be infeasible, but a qualitative statement would be interesting. In particular, the random initialization of the hb bits could play a bigger role than lb bits.

The two variants of PG, with and without sparse backpropagation are also investigated, showing that sparse backpropagation leads to more sparsity. To show that the resulting lower average bitwidth gained with PG leads to increased performance, the authors implement it in Python (running on CPU) and measure the wall clock time to execute the ResNet-18 model. Speedups $> 1$ are shown for every layer when using PG. Evidence from other papers is cited to argue that similar speedups are expected on GPUs.

Even though at the moment it is unclear to me how statistically significant the results are, and I strongly recommend commenting on this in the paper, I think the idea of PG and the demonstrated benefits make the paper interesting enough to be accepted at ICLR.

I also have a few questions that I could not get completely from the paper:
1. I am a bit confused by what you call features. Fig. 2 shows by example how the method works for an input $I$. Is $I$, a single number, i.e. a single entry of your input vector, or do you mean the complete input vector?
2. Could you give a bit more insight, how you tuned your hyperparameters, especially $\delta$ and $\sigma$?
3. What exactly does e.g. $\delta=-1$ mean? The network ideally should compute at high precision, when the result when only considering the most significant bits is above -1?

From a hardware point of view, the paper focuses on GPU implementations. I would have hoped for a discussion of suitable custom hardware that could support PG most efficiently.


Minor comments that I would be interested in but did not influence my score
- It seems to me that on the top-left image of Fig. 3, one blue circle (the second largest) is too much? First part shows 8 dots, middle and right only seven?
- Can you please cite a source that DNN activations have outliers (Sec. 3.4)?
- You could also define e.g. one $\delta$ and $\Delta$ per layer, couldn't you? Would be interesting to see if  e.g. thinning out the precision over depth is possible / has advantages.
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: This paper presents an interesting quantization technique that is, unusually, end-to-end trainable and not just an inference technique. According to the experiments, the method achieves better performance and computational savings as compared to other quantization method baselines. The results are admirably demonstrated on a variety of models, including CNN and RNN-based neural nets, as well as on several datasets in different domains, including ImageNet, CIFAR10, and PTB. We see the method seems to generalize across all of these.

Nevertheless, while I found this is very interesting work, I have a number of issues with the experiments, which I'll go into below. I feel this work is being released prematurely and could use some more polish to help sell the method better. Below are a few remarks and questions for the authors that would be helpful to be answered.

* Why only report on ResNet-18? It would be far more useful to show numbers against ResNet-50. It would also be useful to show the non-quantized best results on these models and datasets.
* I wish more effort had been spent to analyze the experiments. For example, I am not sure I understand the effects of the threshold on this method. What happens when it's set manually?
* How exactly is computation cost savings calculated so crudely? If it uses B_avg, why not calculate the bitwidth per layer and sum things up? Using B_avg strikes me as being quite crude.
* When the authors address runtime changes except in table 6, they changed their baseline to a vanilla ResNet-18 with dense weights. What are the runtime effects relative to ShuffleNet and ShiftNet?
review_title: Official Blind Review #3

Meta-review: The submission proposes an approach to accelerate network training by modifying the precision of individual weights, allowing a substantial speed up without a decrease in model accuracy. The magnitude of the activations determines whether it will be computed at a high or low bitwidth.

The reviewers agreed that the paper should be published given the strong results, though there were some salient concerns which the authors should address in their final revision, such as how the method could be implemented on GPU and what savings could be achieved.

Recommendation is to accept.
Decision: Accept (Poster)