Venue: ICLR 2017 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes to investigate attention transfers between a teacher and a student network. 

Attention transfer is performed by minimising the l2 distance between the teacher/student attention maps at different layers, in addition to minimising the classification loss and optionally a knowledge distillation term.
Authors define several activation based attentions (sum of absolute feature values raise at the power p or max of values raised at the power p). They also propose a gradient based attention (derivative of the Loss w.r.t. inputs). 

They evaluate their approaches on several datasets (CIFAR, Cub/Scene, Imagenet) showing that attention transfers  does help improving the student network test performance.  However, the student networks performs worst than the teacher, even with attention.

Few remarks/questions:
- in section 3 authors  claim that networks with higher accuracy have a higher spatial correlation between the object and the attention map. While Figure 4 is compelling, it would be nice to have quantitative results showing that as well.
- how did you choose the hyperparameter values, it would be nice to see what is the impact of $\beta$.
- it would be nice to report teacher train and validation loss in Figure 7 b)
- from the experiments, it is not clear what at the pros/cons of the different attention maps
- AT does not lead to better result than the teacher. However, the student networks have less parameters. It would be interesting to characterise the corresponding speed-up. If you keep the same architecture between the student and the teacher, is there any benefit to the attention transfer?

In summary:
Pros:
- Clearly written and well motivated.
- Consistent improvement of the student with attention compared to the student alone.
Cons:
- Students have worst performances than the teacher models.
- It is not clear which attention to use in which case?
- Somewhat incremental novelty relatively to Fitnet

review_title: Some nice results, but it is not clear what are the advantages/drawbacks of the different attention maps

final_score: rating: null
review_content: F is a mapping from a 3 tensor to a matrix defined in Eq. (1). So I assume I assume the Lp norm in Eq. (2) is in fact matrix norm. So the question is do you solve for the largest eigenvalue of F(A)^T F(A) in order to normalize the attention map F(A) ? 

Also are the absolute value notations in section 3.1 element-wise or matrix-wise? 
review_title: Notation question

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper proposes a new way of transferring knowledge.
I like the idea of transferring attention maps instead of activations.
However, the experiments don’t show a big improvement compared with knowledge distillation alone and I think more experiments are required in IMAGENET section.
I would consider updating the score if the authors extend the last section 4.2.2.
review_title: official review

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper presented a modified knowledge distillation framework that minimizes the difference of the sum of statistics across the a feature map between the teacher and the student network. The authors empirically demonstrated the proposed methods outperform the fitnet style distillation baseline. 

Pros:
+ The author evaluated the proposed methods on various computer vision dataset 
+ The paper is in general well-written

Cons:  
- The method seems to be limited to the convolutional architecture
- The attention terminology is misleading in the paper. The proposed method really just try to distill the summed squared(or other statistics e.g. summed lp norm) of  activations in a hidden feature map.
- The gradient-based attention transfer seems out-of-place. The proposed gradient-based methods are never compared directly to nor are used jointly with the "attention-based" transfer. It seems like a parallel idea added to the paper that does not seem to add much value.
- It is also not clear how the induced 2-norms in eq.(2) is computed. Q is a matrix \in \mathbb{R}^{H \times W}  whose induced 2-norm is its largest singular value. It seems computationally expensive to compute such cost function. Is it possible the authors really mean the Frobenius norm?

Overall, the proposed distillation method works well in practice but the paper has some organization issues and unclear notation.  

review_title: Review

final_score: rating: null
review_content: Here are a few short pre-review questions

1) Why don't you use the same settings for the activation-based and gradient-based attention transfer on the CIFAR-10, it would be nice to compare those two approaches as it not clear what are their relative advantages/drawbacks.

2) Intermediate hints in Fitnet have been introduced to ease the optimization of the student network. It would be nice to see if the attention transfer have similar effect, or if it provide more benefits in term of generalization.

Also, you mention that you have experimented with Fitnet hint. It would be nice to report the obtained results, and do a comparison with your activation-based attention transfer as the two approaches are close.

3) Gradient attention transfer does not seems to improve upon knowledge distillation in Table 3. Do you have an intuition explaining this result?

4) What value are you using for the hyperparameter \beta (controlling the attention transfer cost). Did you try decaying it during training?
review_title: Pre review questions

Meta-review: Important task (attention models), interesting distillation application, well-written paper. The authors have been responsive in updating the paper, adding new experiments, and being balanced in presenting their findings. I support accepting this paper.
Decision: Accept (Poster)