Venue: ICLR 2018 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes to learn small and low-cost models by combining distillation and quantization. Two strategies are proposed and the ideas are reasonable and clearly introduced. Experiments on various datasets are conducted to show the effectiveness of the proposed method.

Pros:
(1) The paper is well written, the review of distillation and quantization is clear.
(2) Extensive experiments on vision and neural machine translation are conducted.
(3) Detailed discussions about implementations are provided.

Cons:
(1) The differentiable quantization strategy seems not to be consistently better than the straightforward quantized distillation which may need more research.
(2) The actual speedup is not clearly calculated. The authors claim that the inference times of 2xResNet18 and ResNet18 are similar which seems to be unreasonable. And it seems to need a lot of more work to make the idea really practical.

Finally, I am curious whether the idea will work on object detection task.

review_title: The authors try to compress models by combining distillation and quantization. The problem is important. The solution seems to work on large datasets and large models

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This paper presents a framework of using the teacher model to help the compression for the deep learning model in the context of model compression. It proposed both the quantized distillation and also the differentiable quantization. The quantized distillation method just simply adapt the distillation work for the task of model compression, and give good results to the baseline method. While the differentiable quantization optimise the quantization function in a unified back-propagation framework. It is interesting to see the performance improvements by using the one-step optimisation method. 

I like this paper very much as it is in good motivation to utilize the distillation framework for the task of model compression. The starting point is quite interesting and reasonable. The information from the teacher network is useful for constructing a better compressed model. I believe this idea is quite similar to the idea of Learning using Privileged Information, in which the information on teacher model is only used during training, but is not utilised during testing. 

Some minor comments:
In table 3, it seems that the results for 2 bits are not stable, and are there any explanations?
What will be the results if the student model performs the same with the teacher model (e.g., use the teacher model as the student model to do the compression) or even better (reverse the settings)?
What will be the prediction speed for each of models? We can also get the time of speedup for the compressed model.

It will be better if the authors could discuss the connections between distillation and the recent work for the Learning using Privileged Information setting:
Vladimir Vapnik, Rauf Izmailov:
Learning using privileged information: similarity control and knowledge transfer. Journal of Machine Learning Research 16: 2023-2049 (2015)
Xinxing Xu, Joey Tianyi Zhou, IvorW. Tsang, Zheng Qin, Rick Siow Mong Goh, Yong Liu: Simple and Efficient Learning using Privileged Information. BeyondLabeler: Human is More Than a Labeler, Workshop of the 25th International Joint Conference on Artificial Intelligence (IJCAI-16). New York City, USA. July, 2016.

review_title: A very interesting paper with good motivation and good results.

final_score: rating: 7: Good paper, accept
review_content: The paper proposes to combine two approaches to compress deep neural networks - distillation and quantization. The authors proposed two methods, one largely relying on the distillation loss idea then followed by a quantization step, and another one that also learns the location of the quantization points. Somewhat surprisingly, nobody has combined the two approaches before, which makes this paper interesting. Experiments show that both methods work well in compressing large deep neural network models for applications where resources are limited, like on mobile devices. 

Overall I am mostly OK with this paper but not impressed by it.  Detailed comments below.

1. Quantizing with respect to the distillation loss seems to do better than with the normal loss - this needs more discussion. 
2. The idea of using the gradient with respect to the quantization points to learn them is interesting but not entirely new (see, e.g., "Matrix Recovery from Quantized and Corrupted Measurements", ICASSP 2014 and "OrdRec: An Ordinal Model for Predicting Personalized Item Rating Distributions", RecSys 2011, although in a different context). I also wonder if it would work better if you can also allow the weights to move a little bit (it seems to me from Algorithm 2 that you only update the quantization points). How about learning them altogether? Also this differentiable quantization method does not really depend on distillation, which is kind of confusing given the title.
3. I am a little bit confused by how the bits are redistributed in the second method, as in the end it seems to use more than the proposed number of bits shown in the table (as recognized in section 4.2). This makes the comparison a little bit unfair (especially for the CIFAR 100 case, where the "2 bits" differentiable quantization is actually using 3.23 bits). This needs more clarification.
4. The writing can be improved. For example, the concepts of "teacher" and "student" is not clear at all in the abstract - consider putting the first sentence of Section 3 in there instead. Also, the first sentence of the paper reads as "... have showed tremendous performance", which is not proper English. At the top of page 3 I found "we will restrict our attention to uniform and non-uniform quantization". What are you not restricting to, then?

Slightly increased my rating after reading the rebuttal and the revision. 
review_title: Review of "Model compression via distillation and quantization"

Meta-review: The submission proposes a method for quantization.  The approach is reasonably straightforward, and is summarized in Algorithm 1.  It is the analysis which is more interesting, showing the relationship between quantization and adding Gaussian noise (Appendix B) - motivating quantization as regularization.

The submission has a reasonable mix of empirical and theoretical results, motivating a simple-to-implement algorithm.  All three reviewers recommended acceptance.
Decision: Accept (Poster)