Venue: ICLR 2018 conference

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: Summary:
The paper presents three different methods of training a low precision student network from a teacher network using knowledge distillation.
Scheme A consists of training a high precision teacher jointly with a low precision student. Scheme B is the traditional knowledge distillation method and Scheme C uses knowledge distillation for fine-tuning a low precision student which was pretrained in high precision mode.

Review:
The paper is well written. The experiments are clear and the three different schemes provide good analytical insights.
Using scheme B  and C student model with low precision could achieve accuracy close to teacher while compressing the model.

Comments:
Tensorflow citation is missing.
Conclusion is short and a few directions for future research would have been useful.
review_title: good paper

final_score: rating: 7: Good paper, accept
review_content: The authors investigate knowledge distillation as a way to learn low precision networks. They propose three training schemes to train a low precision student network from a teacher network. They conduct experiments on ImageNet-1k with variants of ResNets and multiple low precision regimes and compare performance with previous works

Pros:
(+) The paper is well written, the schemes are well explained
(+) Ablations are thorough and comparisons are fair
Cons:
(-) The gap with full precision models is still large 
(-) Transferability of the learned low precision models to other tasks is not discussed

The authors tackle a very important problem, the one of learning low precision models without comprosiming performance. For scheme-A, the authors show the performance of the student network under many low precision regimes and different depths of teacher networks. One observation not discussed by the authors is that the performance of the student network under each low precision regime doesn't improve with deeper teacher networks (see Table 1, 2 & 3). As a matter of fact, under some scenarios performance even decreases. 

The authors do not discuss the gains of their best low-precision regime in terms of computation and memory.

Finally, the true applications for models with a low memory footprint are not necessarily related to image classification models (e.g. ImageNet-1k). How good are the low-precision models trained by the authors at transferring to other tasks? Is it possible to transfer student-teacher training practices to other tasks?
review_title: speed & memory gains, transferability

final_score: rating: 7: Good paper, accept
review_content: The paper aims at improving the accuracy of a low precision network based on knowledge distillation from a full-precision network. Instead of distillation from a pre-trained network, the paper proposes to train both teacher and student network jointly. The paper shows an interesting result that the distilled low precision network actually performs better than high precision network.

I found the paper interesting but the contribution seems quite limited.

Pros:
1. The paper is well written and easy to read.
2. The paper reported some interesting result such as that the distilled low precision network actually performs better than high precision network, and that training jointly outperforms the traditional distillation method (fixing the teacher network) marginally.

Cons:
1. The name Apprentice seems a bit confusing with apprenticeship learning.
2. The experiments might be further improved by providing a systematic study about the effect of precisions in this work (e.g., producing more samples of precisions on activations and weights).
3. It is unclear how the proposed method outperforms other methods based on fine-tuning. It is also quite possible that after fine-tuning the compressed model usually performs quite similarly to the original model.
review_title: Interesting results but very limited contribution

Meta-review: Meta score: 7

The paper combined low precision computation with different approaches to teacher-student knowledge distillation.  The experimentation is good, with good experimental analysis.  Very clearly written.  The main contribution is in the different forms of teacher-student training combined with low precision.

Pros:
 - good practical contribution
 - good experiments
 - good analysis
 - well written
Cons:
 - limited originality
Decision: Accept (Poster)